A double-branch multi-scale image defogging method based on high-quality codebook

Abstract

The present application relates to the technical field of image processing, and particularly relates to a double-branch multi-scale image defogging method based on a high-quality codebook, comprising: obtaining an original image super-resolution reconstruction dataset; training a VQGAN network model by using the original image super-resolution reconstruction dataset to obtain a codebook, a network structure of a VQ decoder and corresponding parameters thereof; obtaining an original image defogging dataset; training a double-branch multi-scale image defogging network model by using the original image defogging dataset; and inputting a foggy image into the trained double-branch multi-scale image defogging network model to obtain a generated clear and fog-free image. The present application reconstructs an original foggy image to realize an end-to-end image defogging process, and improves the clarity and recognizability of the foggy image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image restoration, and in particular relates to a double-branch multi-scale image defogging method based on a high-quality codebook. BACKGROUND

[0002] Fog is an aerosol system composed of a large number of small water droplets suspended in the air near the ground, and is one of the main reasons for image blurring, color distortion, and reduced contrast.

[0003] In recent years, extreme weather has occurred frequently, and many cities have been affected by fog. The accuracy of information obtained from images collected by outdoor visual systems such as automatic driving, video monitoring, military reconnaissance, and remote sensing images is reduced. With the increase in fog and the appearance of non-uniform haze, the quality of images will deteriorate rapidly, resulting in color distortion, feature blurring, reduced contrast, and other visual quality degradation, which makes it impossible to identify objects and backgrounds in the images, and has a serious impact on the execution and processing of subsequent visual tasks such as semantic segmentation and object detection in computer vision. Therefore, it is necessary to preprocess the images to reduce the impact of thick fog on image quality.

[0004] Current mainstream technical solutions for image defogging based on deep learning include:

[0005] Based on physical models and prior knowledge, some methods use deep learning, and some methods use image enhancement preprocessing operations. Many studies have begun to use convolutional neural networks to estimate parameters in the atmospheric scattering model. In order to avoid cumulative errors in the parameter estimation process, an end-to-end network is proposed to directly estimate the clear image from the hazy image. The hazy image is often obtained by processing the clear image through a physical model, mainly the atmospheric scattering model. The atmospheric scattering model itself cannot perfectly describe the formation process of all fog and haze, so the artificially processed image cannot well replace the real hazy image. Therefore, the trained model has poor generalization ability and often fails when used to process real images. Therefore, these methods can achieve excellent performance on synthetic data sets, but their performance on real data sets needs to be improved. SUMMARY

[0006] To solve the above technical problems, the present application provides a double-branch multi-scale image defogging method based on a high-quality codebook, comprising:

[0007] S1: Obtain an original image super-resolution reconstruction data set, wherein the original image super-resolution reconstruction data set comprises an original clear image;

[0008] S2: training the VQGAN network model using the original image super-resolution reconstruction dataset to obtain the codebook, the network structure of the VQ decoder and the corresponding parameters thereof;

[0009] The VQGAN network model comprises a VQ encoder, a codebook and a VQ decoder.

[0010] S3: obtaining an original image defogging dataset, wherein the original image defogging dataset comprises original foggy images and corresponding clear non-foggy images.

[0011] S4: training the double-branch multi-scale image defogging network model using the original image defogging dataset.

[0012] The double-branch multi-scale image defogging network model is divided into a prior matching branch and a channel attention branch; wherein the prior matching branch comprises a VQGAN network model with fixed parameters, a pyramid hollow neighborhood attention encoder and an enhanced decoder, and the structural components thereof are sequentially a VQ encoder, a pyramid hollow neighborhood attention encoder, a fixed-parameter codebook matching module, a fixed-parameter VQ decoder and an enhanced decoder; the channel attention branch comprises a 3x3 convolution and four residual channel attention layers; and finally, the results of the two branches are fused through a feature fusion structure.

[0013] S5: inputting the foggy image into the trained double-branch multi-scale image defogging network model to obtain a generated clear non-foggy image.

[0014] The present application has the following advantages compared with the prior art: a discrete codebook is trained from a clear non-foggy image, which encapsulates high-quality prior knowledge with original image color and structure, and then a double-branch neural network structure, i.e., a prior matching branch and a channel attention branch, is constructed, neighborhood attention based on Transformer and channel attention based on convolution are used to extract global features of the foggy image and learn complex interactive features between the heavy fog area and the underlying scene, and a feature fusion module is used to fuse the features extracted by the two branches, in the matching process of the high-quality prior constraint codebook and the foggy image features, a controllable distance recalculation operation is used to replace the areas affected by the fog in the image, the original foggy image is reconstructed to realize an end-to-end image defogging process, and the clarity and recognizability of the foggy image are improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A double-branch multi-scale image defogging method flowchart based on a high-quality codebook;

[0016] Figure 2 A double-branch multi-scale defogging network structure schematic diagram based on a high-quality codebook;

[0017] Figure 3 This is a schematic diagram of the VQ encoder and VQ decoder structure of the present invention;

[0018] Figure 4 This is a structural diagram of the pyramid cavity neighborhood attention encoder of the present invention;

[0019] Figure 5 This is a structural diagram of the enhanced decoder of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] This invention proposes a dual-branch, multi-scale image dehazing method based on a high-quality codebook, such as... Figure 1 As shown, the entire process requires two stages of training;

[0022] The first stage is to use VQGAN to pre-train a discrete codebook that reflects the general detailed texture features of a clear image. In this stage, the network consists of a VQ encoder, a codebook, and a VQ decoder. The training objective is to obtain a codebook that stores high-quality clear image features and its corresponding VQ decoder.

[0023] S1: Obtain the original image super-resolution reconstruction dataset;

[0024] The original image super-resolution reconstruction dataset includes the original, sharp images;

[0025] In the specific implementation, the dataset used is Flickr2K, which contains 1000 clear images, including people, buildings, animals, objects, etc. A low-resolution subset was used, which was downsampled by 4 times using the bicubic method. The specific resolution is about 512×349. 800 images were used for training, 100 images for validation, and 100 images for testing.

[0026] S2: Train the VQGAN network model using the original image super-resolution reconstruction dataset to obtain the codebook, the network structure of the VQ decoder, and its corresponding parameters;

[0027] Among them, such as Figure 2 As shown, the VQGAN network model includes a VQ encoder, a codebook, and a VQ decoder, where the structures of the VQ encoder and VQ decoder are as follows: Figure 3 As shown;

[0028] Preferably, the training of the VQGAN network model in S2 comprises:

[0029] S21: inputting the original clear image x into the VQ encoder to obtain a latent feature map z;

[0030] Specifically, the VQ encoder and decoder adopt a UNet-based network architecture, the former half is feature extraction, and the latter half is up-sampling. Unet has been proved to perform well in image classification, segmentation and other fields. Since the encoder down-sampling extracts features while losing some edge features, multiple residual structures are adopted to realize the recovery of edge features through feature splicing.

[0031] S22: matching the latent feature map z to the nearest element in the codebook to obtain a codebook-discrete feature map z q ;

[0032] The codebook is a discrete codebook. The discrete codebook compresses the detail texture features of the image and plays a key role in image reconstruction. The present application adopts the latest research achievement VQGAN in the field of image generation and makes certain modifications. For specific technical details, see ESSER P, ROMBACH R, OMMER B. Taming transformers for high-resolution image synthesis [C]. IEEE International Conference on Computer Vision and Pattern Recognition, Virtual Event, 2021: 12873-12883. By training on haze-free images, the semantic information contained in the haze-free images is compressed and a semantic-rich codebook is established through model self-supervised learning. The compression rate is improved while the perceptual quality is still good, thereby serving as prior knowledge for haze-free images, which is used as a constraint condition for the subsequent haze-free image generation model.

[0033] The mathematical representation of the codebook is Given a high-quality image x as the input of the VQ encoder E vq , a latent feature map z is outputted, and then each pixel z ij of z is matched to the nearest element in the codebook to obtain a codebook-discrete feature map Subsequently, the discretized features are inputted into the VQ decoder D vq to obtain the processed image y. The whole process can be represented as follows:

[0034] z ij = Evq (x ij )

[0035]

[0036] where Z ij represents the pixel at the position ij of the latent feature map Z, E vq () represents the VQ encoder, x ij represents the pixel at the position ij of the high-quality image x, M represents the codebook-discrete feature map, M() represents the codebook matching operation, and ||z-z k || represents the distance between the discrete feature Z of the foggy image and the codebook encoding z k , and argmin() represents the function of taking the minimum value of the distance, y ij represents the pixel at the position ij of the generated image,

[0037] It is worth noting that when training the network, the discrete operation in the middle of the encoder and the decoder cannot be back propagated through the gradient because the operation is not differentiable, so during back propagation, the gradient is only copied from the decoder to the encoder, so that the model can be trained end-to-end through the loss function.

[0038] The features of the clear image are compressed into short vectors by the discrete operation and stored in the codebook, so the discrete codebook compresses the texture features of the image, which plays a key role in image reconstruction. The reconstructed image is used to judge the true and false of the image by the discriminator to assist the network to further learn until the discriminator cannot judge the true and false of the image, and then the training is completed.

[0039] S23: Constructing a codebook-discrete loss function according to the latent feature map Z and the discrete feature map Z q , updating the parameters of the VQ encoder and the codebook of the model as the optimization objective of the minimum loss function;

[0040] where the codebook-discrete loss function is defined as follows:

[0041]

[0042] where L Z represents the codebook-discrete loss function, z q represents the discrete feature map, Z represents the latent feature map, β represents the weight factor, and sg[Z q ] represents the stop-gradient, and the loss function mainly measures the discrete loss generated between the output Z of the encoder and the discrete vector z q ;

[0043] S24: Updating the parameters of the VQ encoder and the codebook of the model as the optimization objective of the minimum loss function according to the discrete feature map z qThe clear image y is decoded by a VQ decoder to obtain a reconstructed clear image y;

[0044] S25: constructing a loss function of the VQ decoder according to the reconstructed clear image y and the original clear image x, and updating parameters of the VQ decoder of the model as an optimization target of minimizing the loss function;

[0045] Details of this step are shown in S46;

[0046] The second stage constructs a double-branch multi-scale neural network, uses a neighborhood attention based on a Transformer and a channel attention based on convolution to extract global features of the foggy image and learn complex interaction features between the dense fog area and the underlying scene, and uses the discrete codebook obtained from the clear image in the first stage to assist the reconstruction process of the foggy image. Specifically, a controllable distance recalculation operation is used to match the corresponding discrete codebook and replace the areas affected by the fog in the image, thereby achieving the effect of defogging.

[0047] S3: obtaining an original image defogging dataset;

[0048] The original image defogging dataset includes original foggy images and corresponding clear images without fog.

[0049] In specific implementation, the used dataset is O-HAZE, I-HAZE, DENSE-HAZE, NH-HAZE-20, NH-HAZE-21, and NH-HAZE-23. O-HAZE and I-HAZE respectively contain 45 pairs of outdoor scenes and 35 pairs of indoor scenes of foggy and corresponding clear images. DENSE-HAZE is characterized by dense and uniform fog scenes. This dataset contains 55 pairs of real dense fog images and corresponding clear images of various outdoor scenes. These dense fog images are generated by a professional fog machine, and it is almost impossible to distinguish the objects originally existing in the image. Compared with conventional datasets, the defogging difficulty is very large. The NH-HAZE dataset is a set of real non-uniform fog images and corresponding clear images. The non-uniform fog is introduced by a professional fog generator to simulate real fog conditions. According to the shooting date, it can be further divided into NH-HAZE-20, NH-HAZE-21, and NH-HAZE-23, which respectively have 40, 25, and 55 image pairs. For more detailed information, refer to Table 1.

[0050] Table 1:

[0051]

[0052]

[0053] S4: training the double-branch multi-scale image defogging network model using the original image defogging dataset;

[0054] The double-branch multi-scale image defogging network model is divided into a prior matching branch and a channel attention branch.

[0055] The prior matching branch includes a VQ encoder, a pyramid hole neighborhood attention encoder, a fixed-parameter codebook matching module, a fixed-parameter VQ decoder, and an enhanced decoder. Figure 4 The enhanced decoder is as shown in Figure 5 .

[0056] The channel attention branch includes a 3x3 convolution and four residual channel attention layers.

[0057] The results of the two branches are finally fused through a feature fusion structure.

[0058] Preferably, the training of the double-branch multi-scale image defogging network model in S4 includes:

[0059] S41: inputting the original foggy image x into the VQ encoder to roughly extract features and obtain preliminary features F1, and then inputting the preliminary features F1 into the pyramid hole neighborhood attention encoder to obtain advanced features F2.

[0060] The VQ encoder performs well when encoding clear images, but it is insufficient when encoding images with thick fog or non-uniform haze. This is mainly because in the defogging task, the encoder not only needs to extract general structural texture features in the image, but also needs to identify the haze area in the image. However, the VQ encoder network architecture is shallow and cannot complete this task well.

[0061] In order to fully extract the global features such as texture and structure of the foggy image, the present application designs an encoder based on hole neighborhood attention in the prior matching branch. The hole neighborhood attention is a variant of self-attention in Vision Transformer, which is an effective and scalable visual sliding window attention mechanism, and outperforms Vision Transformer and Swin Transformer in visual downstream performance.

[0062] The neighborhood attention Transformer is composed of a multi-layer perception (MLP), a LayerNorm (LN), a residual connection and a multi-head neighborhood attention (NA), when the neighborhood size is the smallest, each pixel only pays attention to the neighborhood of 1 pixel around itself, and when the neighborhood size reaches the maximum value, the output of the neighborhood attention is equal to the self-attention;

[0063] Compared with the self-attention, the neighborhood attention not only reduces the calculation cost, but also introduces a local inductive bias similar to the convolution, specifically, the NA is a pixel-by-pixel operation, and the self-attention (SA) is located to the nearest neighbor pixel, so compared with the quadratic complexity of the self-attention, the neighborhood attention has linear time and space complexity;

[0064] In the encoder of the designed hole neighborhood attention, four feature maps with different resolution sizes are obtained through a sequencer and twice down-sampling, a pyramid structure is used to aggregate the feature information of each layer as the input of the next layer through cascading operation, and different levels of features are reused to realize different scales of features;

[0065] Therefore, the neighborhood attention mechanism is adopted in the present application, the global feature of the image is focused, high-quality prior is used for feature matching, and the network generalization is improved;

[0066] Firstly, the VQ encoder is used to roughly extract the image position and structure information, then the shallow layer features are serialized by the overlapping tokenizer for input into the neighborhood attention Transformer block, and a down-sampler is connected after the second neighborhood attention Transformer block, the spatial size is reduced to half of the original, and the channel number is doubled, so different scale feature maps are generated;

[0067] In order to fuse the feature maps of different scales, a pyramid-shaped feature aggregation mode is designed in the present application, and many dense connection operations are added, since the direct addition operation will lose some original feature information in the feature fusion, and the cascading operation is a lossless operation in strict sense, therefore, in the process of gradually processing the feature maps, the dense connection is adopted to aggregate the feature information of each layer as the input of the next layer through addition and cascading operation, so that different levels of features are aggregated, different scales of features are reused, the global information in the features is optimized, and through multiple residual connections, different levels of features are allowed to be fused, so that the multi-scale features of the image fog distribution are extracted, which is beneficial to the subsequent high-quality prior matching;

[0068] S42: input the high-level feature F2 into the fixed-parameter codebook matching module for matching to obtain a matched feature F3;

[0069] In the image reconstruction using a high-quality codebook, the discrete code output by the encoder is difficult to match to the corresponding high-quality codebook, which is mainly due to the serious image degradation, the domain gap problem of the foggy image and the fog-free image, resulting in inconsistent data distribution, so it is necessary to match the distance between the encoder output and the codebook, and use the matching operation of the controllable distance recalculation method to reduce the problem caused by the domain gap, so as to achieve a better reconstruction effect;

[0070] The distance between the discrete code of the foggy image and each code in the codebook is calculated to find the codebook code with the minimum distance, and then a weight function F is used to adjust the finally calculated distance, so as to obtain the matching formula:

[0071]

[0072] F(f k ,α)=f k ×e α

[0073] Wherein, M(z) represents the matching process of the codebook matching module, F() represents the weight function generated according to the frequency difference, f k represents the frequency difference of the foggy image and the clear image on the codebook activation, and α represents the parameter for adjusting the degree of dehazing, and ||z-z k || represents the distance between the discrete feature z of the foggy image and the codebook code z k , and argmin() represents the minimum value function of the distance.

[0074] In the matching formula, the activation frequency difference f k of the codebook code and α need to be solved, for the activation frequency difference of the codebook code, at the beginning of the dehazing network training, the activation frequency difference of each code in the codebook is set to 0, if the current foggy image is not matched to the codebook code and the clear image is matched to the codebook code, the frequency difference on the code is updated, after multiple training, the network will learn a best value, for the value of α, the difference between the code obtained by the encoder and the code of the clear fog-free image is represented by the difference between two probability distributions, the Kullback-Leibler divergence (KL divergence) is used to measure the similarity of two probability distributions, the greater the similarity, the smaller the value of the KL divergence, and the probability distribution of the codebook activation of the clear image is P c , and the probability distribution corresponding to the foggy image is P hand the probability distribution of the foggy image can be adjusted by a, so that the adaptation problem of the two different domains, the domain where the haze-free image is located and the domain where the foggy image is located, is converted into solving an optimal parameter a, so that the KL divergence of P c (x = z k ) and P h (x = z k | a) is minimized.

[0075]

[0076]

[0077] S43: input the original clear haze-free image into the pre-trained VQGAN network model in S2, and sequentially pass through the VQ encoder and the codebook to obtain the intermediate feature F4;

[0078] The specific details of this step are described in S2.

[0079] S44: according to the preliminary feature F1, the high-level feature F2, the matched feature F3 and the intermediate feature F4, construct an encoder loss function of the double-branch multi-scale image dehazing network model, and update the parameters of the VQ encoder and the pyramid hole neighborhood attention encoder of the model as the optimization objective of the minimum loss function.

[0080] In order to help the output of the encoder to match the correct high-quality codebook prior in the subsequent steps, we need to make the features output by the encoder meet the consistent standard normal distribution as when training the high-quality prior.

[0081] Assuming that the input of the foggy image is x h , the input of the haze-free image is x gt , the dehazing network encoder is E, and the encoder used for training the codebook is E vq , we can get the intermediate feature z h of the foggy image processed by the encoder E, and the intermediate feature z h of the haze-free image processed by the encoder E vq .

[0082] In the process of controlling image generation, we also need to control the style difference between the generated image and the haze-free image, so we use the Gram matrix to measure the style loss, and use the discriminator D when training the codebook to judge whether the generated feature is real or not, so the final encoder loss is:

[0083]

[0084] Where, L VQan encoder loss function representing a double-branch multi-scale image defogging network model, z h an intermediate feature representing a foggy image, an intermediate feature representing a fog-free image, λ style and λ adv Ψ() represents a matrix for measuring the style loss, E[] represents an encoder, D() represents a discriminator, an i-th intermediate feature representing a foggy image.

[0085] S45: sequentially input the feature F3 into the VQ decoder with fixed parameters and the enhanced decoder for decoding to obtain an intermediate feature F5;

[0086] Specifically, the output result obtained only through the VQ decoder is prone to lack of detailed information in some deep haze places, and the image structure and texture are relatively blurred. In order to improve the decoding ability of the detailed features of the foggy image, the application designs an enhanced decoder based on multiple attentions in the prior matching branch, combines the channel attention and the pixel attention, and finally passes through an enhanced block based on pyramid pooling to ensure that the detailed features of different scales are embedded into the final result.

[0087] S46: input the original foggy image into the channel attention branch to obtain an intermediate feature F6;

[0088] The channel attention branch pays more attention to non-uniform haze and thick fog areas with significant brightness changes, avoids over-enhancement problems, improves the overall reconstruction performance of the image, and the attention mechanism can make the network flexibly pay attention to the haze features to reconstruct high-quality fog-free images. Non-uniform haze and thick fog will significantly increase the brightness of the occluded area, pay more attention to the restoration of areas with significant brightness changes other than the sky, snow, etc., can avoid over-enhancement problems, and thus improve the overall reconstruction performance of the image.

[0089] S47: add the intermediate features F5 and F6 through a channel addition operation, and then pass through a feature fusion module to obtain a generated defogging image y;

[0090] The final feature fusion part adopts a feature fusion module composed of a reflection padding layer, a convolution layer and a Tanh activation function, which fuses the outputs of the two branches.

[0091] S48: according to the clear image y after defogging and the original clear image x, construct a loss function of the VQ decoder, and update the parameters of the VQ decoder of the model with the minimum loss function as the optimization target;

[0092] In this stage, it is needed to judge whether the final generated image of the whole network is correctly completed the defogging task, it is noted that since the loss is calculated separately, the parameters of the encoder and the decoder are not updated in the same dimension, therefore the gradient of this stage cannot be back propagated to the encoder;

[0093] Therefore, the loss of other parts of the network is calculated by combining the following several losses;

[0094] Smooth L1 loss:

[0095] The smooth L1 loss function (also known as mean absolute error) combines the advantages of L1 and L2 losses, which measures the average error amplitude of the distance between the predicted value and the true value, the advantage of the smooth L1 loss function is that the derivative is continuous at 0, which makes the solving efficiency higher and the convergence speed faster;

[0096]

[0097] where x i and y i represent the i-th pixel of the clear image and the foggy image respectively, N is the total number of pixels, f θ (·) represents the defogging network, and f θ (x i ) represents the i-th pixel of the image reconstructed by the defogging network;

[0098] MS-SSIM loss:

[0099] The MS-SSIM loss is based on the assumption that the human eye can obtain image structure information, which can provide a reference on the image quality perception standard, let O and G represent two windows centered on the i-th pixel of the defogged image and the real image respectively, apply a Gaussian filter on the two windows to generate the corresponding mean (μ O , μ G ), standard deviation σ O , σ G and covariance σ OG , the MS-SSIM loss can be expressed as follows:

[0100]

[0101] where C1 and C2 are two very small values, which help to stabilize the fraction in the formula;

[0102] Perceptual loss:

[0103] The VGG16 pretrained on the ImageNet dataset, which is a computer vision dataset created by Professor Li Feifei of Stanford University, etc., containing more than 14 million pictures and more than 20,000 classifications, is used to measure the perceptual loss. The VGG16 pretrained on the ImageNet dataset has good perceptual effect and can help the model to reconstruct more fine details.

[0104] Let x and y represent the foggy image and the clean image respectively, f θ (x) represents the image reconstructed by the dehazing network, and Φ j (·) represents the feature map processed by the jth layer of VGG16. The L2 loss is used to measure the distance between the reconstructed image and the clean image feature map, and N represents the number of feature maps used to calculate the perceptual loss.

[0105]

[0106] Adversarial loss:

[0107] Since the pixel-based loss function cannot provide sufficient supervision on small datasets, an adversarial loss is added to make up for the shortcomings of the above loss.

[0108]

[0109] Where D represents the discriminator used when training the codebook, and M represents the number of sample data.

[0110] Total loss:

[0111] The weights of the smooth l1 loss function, MS-SSIM loss, perceptual loss, and adversarial loss are initially set to 1, 0.5, 0.01, and 0.0005 respectively, and will be adjusted according to the experimental results. Thus, the total image reconstruction loss is as follows:

[0112] L=L l1 +0.5L MS-SSIM +0.01L perc +0.0005L adv

[0113] S5: input the foggy image into the trained dual-branch multi-scale image dehazing network model to obtain the generated clear and fog-free image.

[0114] Preferably, during training, the input images are randomly cropped to a size of 256x256, and the data set is augmented by scaling, random rotation and flipping, the Adam optimizer is used, the default beta1 and beta2 are 0.9 and 0.99 respectively, the initial learning rate is 0.0001, the batch size is set to 1, and the model is implemented on an NVIDIA V100 Tensor Core based on Pytorch.

[0115] While embodiments of the present application have been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application. The scope of the application is defined by the appended claims and their equivalents.

Claims

1. A high-quality codebook-based dual-branch multi-scale image defogging method, characterized in that, The method comprises the following steps: S1: obtaining an original image super-resolution reconstruction data set, wherein the original image super-resolution reconstruction data set comprises an original clear image; S2: training a VQGAN network model by using the original image super-resolution reconstruction data set, and obtaining a codebook, a network structure of a VQ decoder and corresponding parameters thereof; The VQGAN network model comprises a VQ encoder, a codebook and a VQ decoder; S3: obtaining an original image defogging data set, wherein the original image defogging data set comprises an original foggy image and a corresponding clear image without fog; S4: training a double-branch multi-scale image defogging network model by using the original image defogging data set; The double-branch multi-scale image defogging network model is divided into a prior matching branch and a channel attention branch; wherein the prior matching branch comprises a VQGAN network model with fixed parameters, a pyramid hollow neighborhood attention encoder and an enhanced decoder, and the structure thereof comprises, in sequence, a VQ encoder, the pyramid hollow neighborhood attention encoder, a codebook matching module with fixed parameters, a VQ decoder with fixed parameters and the enhanced decoder; the channel attention branch comprises a 3*3 convolution and four residual channel attention layers; and finally, the results of the two branches are fused through a feature fusion structure; The pyramid hollow neighborhood attention encoder comprises a sequencer, a neighborhood attention Transformer block and a down-sampler; The processing process of the pyramid hollow neighborhood attention encoder comprises the following steps: Step 1: input data is obtained through the sequencer and twice down-sampling to obtain four feature maps with different resolution sizes; a pyramid structure is adopted to cascade the feature information of each layer as the input of the next layer, thereby aggregating the features of different levels; Step 2: a plurality of residual connections are adopted to allow the features of different levels to be fused, thereby extracting multi-scale features of the image fog distribution to obtain output features; S5: inputting a foggy image into the trained double-branch multi-scale image defogging network model to obtain a generated clear image without fog.

2. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 1, characterized in that, The step of training the VQGAN network model in step S2 is as follows: S21: input the original clear image x to the VQ encoder E of the UNet-based network architecture vq The latent feature map z is obtained by extracting and sampling in the UNet-based network architecture, and splicing edge features by using multiple residual structures during the sampling process. S22: match the latent feature map z to the closest element in the codebook, resulting in a codebook-discretized feature map z q ; S23: constructing a codebook discretization loss function according to the latent feature map z and the discrete feature map z q , and updating parameters of the VQ encoder of the model and the codebook with a minimum loss function as an optimization objective. S24: the discrete feature map z q is sent into a VQ decoder to obtain the reconstructed clear image y; S25: constructing a loss function of the VQ decoder according to the reconstructed clear image y and the original clear image x, and updating the parameters of the VQ decoder of the model with the minimum loss function as the optimization objective.

3. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 1 or 2, characterized in that, The mathematical expression of the codebook is as follows: wherein Z denotes a codebook, z k denotes a codebook encoding, K denotes a codebook encoding number, denotes a codebook encoding set.

4. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 2, characterized in that, The codebook discretization loss function is as follows: The method comprises the following steps: where L Z represents the codebook discretization loss function, sg[] represents stop-gradient, z q represents the discrete feature map, z represents the latent feature map, and β represents the weight factor.

5. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 1, characterized in that, The training steps of the double-branch multi-scale image defogging network model in S4 comprise the following steps: S41: inputting an original foggy image x into a VQ encoder to roughly extract features to obtain preliminary features F1, and then inputting the preliminary features F1 into a pyramid hollow neighborhood attention encoder to obtain advanced features F2; S42: inputting the advanced features F2 into a codebook matching module with fixed parameters for matching to obtain matched features F3; S43: inputting an original clear image without fog into the VQGAN network model pre-trained in S2, and sequentially passing through a VQ encoder, a codebook to obtain intermediate features F4; S44: According to the preliminary feature F1, the high-level feature F2 and the matched feature F3 and the intermediate feature F4, the encoder loss function of the double-branch multi-scale image defogging network model is constructed, and the parameters of the VQ encoder and the pyramid hollow neighborhood attention encoder of the model are updated with the minimum loss function as the optimization objective; S45: The feature F3 is sequentially sent to the VQ decoder with fixed parameters and the enhanced decoder for decoding to obtain the intermediate feature F5; S46: The original foggy image is input into the channel attention branch to obtain the intermediate feature F6; S47: The intermediate features F5 and F6 are added through a channel addition operation, and then passed through a feature fusion module to obtain a generated defogging image y; S48: According to the clear image y after defogging and the original clear image x, the loss function of the remaining part of the double-branch multi-scale image defogging network model is constructed, including the smooth l1 loss function, the MS-SSIM loss, the perception loss and the adversarial loss, and the parameters of the remaining part of the model are updated with the minimum loss function as the optimization objective; The parameters of the remaining part of the model include the parameters of the VQ decoder, the enhanced decoder and the codebook matching module.

6. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 1, characterized in that, The matching process of the codebook matching module is represented as: F(f k ,α) = f k × e α where M(z) represents the matching process of the codebook matching module, F() represents a weight function generated according to the frequency difference, f k represents the frequency difference of the foggy image and the clear image on the codebook activation, and a represents a parameter for adjusting the degree of defogging. ||z-z k represents the distance between the discrete features z of the foggy image and the codebook coding z k represents the distance between the discrete features z of the foggy image and the codebook coding z 7. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 1, characterized in that, The encoder loss function of the double-branch multi-scale image defogging network model includes: wherein, L VQ represents an encoder loss function of the double-branch multi-scale image defogging network model, z h represents the intermediate feature of the foggy image, represents the intermediate feature of the fog-free image, λ style and λ adv represent first and second hyperparameters for adjusting the weights of different losses, Ψ() represents a matrix for measuring the style loss, E[] represents an encoder, D() represents a discriminator, represents the i-th intermediate feature of the foggy image.

Images