Double-branch multi-scale image defogging method based on high-quality codebook

Abstract

The invention relates to the technical field of image processing, in particular to a double-branch multi-scale image defogging method based on a high-quality codebook, and the method comprises the steps: obtaining an original image super-resolution reconstruction data set; training the VQGAN network model by using the original image super-resolution reconstruction data set to obtain a codebook, a network structure of a VQ decoder and corresponding parameters; obtaining an original image defogging data set; training the dual-branch multi-scale image defogging network model by using the original image defogging data set; the foggy image is input into the trained double-branch multi-scale image defogging network model to obtain a generated clear fogless image, the original haze image is reconstructed, the end-to-end image defogging process is achieved, and the definition and recognizability of the foggy image are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image restoration, and in particular to a dual-branch multi-scale image defogging method based on a high-quality codebook. Background Art

[0002] Fog is an aerosol system composed of a large number of tiny water droplets suspended in the air near the ground. It is one of the main causes of image blur, color distortion and reduced contrast.

[0003] In recent years, extreme weather has occurred frequently. Many cities have suffered from fog, which occurs frequently seasonally. Affected by fog, the accuracy of image information acquired by outdoor visual systems such as autonomous driving, video surveillance, military reconnaissance, and remote sensing imaging will be reduced. As the fog worsens and uneven haze appears, the quality of the image will deteriorate sharply, resulting in color distortion, blurred features, reduced contrast, and other visual quality degradations, making it impossible to recognize objects and backgrounds in the image. This will have a serious impact on the execution and processing of various subsequent visual tasks, such as semantic segmentation and target detection in computer vision. Therefore, it is necessary to preprocess the image to reduce the impact of dense fog on image quality.

[0004] The current mainstream technology solutions for image dehazing based on deep learning include:

[0005] Based on physical models and prior knowledge, and using deep learning methods, some of them are supplemented by image enhancement preprocessing operations. Many studies have begun to use convolutional neural networks to estimate the parameters in the atmospheric scattering model. In order to avoid the cumulative errors in the parameter estimation process, people have proposed an end-to-end network to directly estimate the foggy image to generate the fog-free image. The foggy image is often obtained by degrading the clear fog-free image through the physical model, mainly the atmospheric scattering model. The atmospheric scattering model itself cannot perfectly describe the formation process of all haze. Therefore, the image obtained by this artificial processing cannot replace the real foggy image well. Therefore, the trained model has poor generalization ability and often fails when used to process real images. Therefore, these methods can finally achieve excellent performance on synthetic data sets, but their performance on real data sets needs to be improved. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a dual-branch multi-scale image defogging method based on a high-quality codebook, comprising:

[0007] S1: Acquire an original image super-resolution reconstruction dataset, where the original image super-resolution reconstruction dataset includes an original clear image;

[0008] S2: Use the original image super-resolution reconstruction dataset to train the VQGAN network model to obtain the network structure of the codebook and VQ decoder and their corresponding parameters;

[0009] The VQGAN network model includes a VQ encoder, a codebook, and a VQ decoder;

[0010] S3: Acquire an original image defogging dataset, where the original image defogging dataset includes: an original foggy image and a corresponding clear fog-free image;

[0011] S4: Use the original image dehazing dataset to train the dual-branch multi-scale image dehazing network model;

[0012] The dual-branch multi-scale image defogging network model is divided into a priori matching branch and a channel attention branch; wherein the priori matching branch includes a VQGAN network model with fixed parameters, a pyramid void neighborhood attention encoder, and an enhanced decoder, and its structure is composed of a VQ encoder, a pyramid void neighborhood attention encoder, a fixed parameter codebook matching module, a fixed parameter VQ decoder, and an enhanced decoder in sequence; the channel attention branch includes a 3×3 convolution and 4 residual channel attention layers; and finally the results of the two branches are fused through a feature fusion structure;

[0013] S5: Input the foggy image into the trained dual-branch multi-scale image defogging network model to obtain a clear and fog-free image.

[0014] The beneficial effects of the present invention are as follows: compared with the prior art, a discrete codebook trained from a clear and fog-free image is introduced, which encapsulates high-quality prior knowledge of the color and structure of the original image, and then a two-branch neural network structure is constructed, namely a prior matching branch and a channel attention branch, and the Transformer-based neighborhood attention and the convolution-based channel attention are used to extract the global features of the foggy image and learn the complex interaction features between the dense fog area and the underlying scene, and the features extracted by the two branches are fused through a feature fusion module. In the matching process of the high-quality prior constraint codebook and the foggy image features, the area affected by haze in the image is replaced by a controllable distance recalculation operation, and the present invention reconstructs the original haze image to realize an end-to-end image defogging process, thereby improving the clarity and recognizability of the foggy image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart of a dual-branch multi-scale image defogging method based on a high-quality codebook of the present invention;

[0016] Figure 2 Schematic diagram of the dual-branch multi-scale defogging network structure based on high-quality codebook of the present invention;

[0017] Figure 3 A schematic diagram of the structure of a VQ encoder and a VQ decoder of the present invention;

[0018] Figure 4 This is a structural diagram of the pyramid hole neighborhood attention encoder of the present invention;

[0019] Figure 5 This is a structural diagram of the enhanced decoder of the present invention. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] The present invention proposes a dual-branch multi-scale image defogging method based on a high-quality codebook, such as Figure 1 As shown, the whole needs to go through two stages of training;

[0022] The first stage is to use VQGAN to pre-train a discrete codebook that reflects the detailed texture features common to clear images. In this stage, the network consists of a VQ encoder, a codebook, and a VQ decoder. The training goal is to obtain a codebook that stores high-quality clear image features and its corresponding VQ decoder.

[0023] S1: Get the original image super-resolution reconstruction dataset;

[0024] The original image super-resolution reconstruction dataset includes the original clear image;

[0025] In the specific implementation, the dataset used is Flickr2K, which has a total of 1,000 clear images, including people, buildings, animals, objects, etc. A low-resolution subset of the images is used, which is 4 times downsampled by the bicubic method. The specific resolution is about 512×349, of which 800 are used for training, 100 for verification, and 100 for testing.

[0026] S2: Use the original image super-resolution reconstruction dataset to train the VQGAN network model to obtain the network structure of the codebook and VQ decoder and their corresponding parameters;

[0027] Among them, Figure 2 As shown, the VQGAN network model includes a VQ encoder, a codebook, and a VQ decoder, where the VQ encoder and VQ decoder structures are as follows Figure 3 As shown;

[0028] Preferably, training the VQGAN network model in S2 includes:

[0029] S21: Input the original clear image x into the VQ encoder to obtain the potential feature map z;

[0030] Specifically, the VQ encoder and decoder use a UNet-based network architecture. The first half is feature extraction, and the second half is upsampling. Unet has been proven to perform well in image classification, segmentation and other fields. Since the encoder will lose some edge features while downsampling to refine features, multiple residual structures are used to achieve edge feature retrieval through feature splicing.

[0031] S22: Match the potential feature map z to the nearest element in the codebook to obtain the discrete feature map z after the codebook q ;

[0032] Among them, the codebook is a discrete codebook. The discrete codebook compresses the detailed texture features of the image and plays a key role in image reconstruction. The present invention adopts VQGAN, the latest research result in the field of image generation, and makes certain modifications to it. For specific technical details, see ESSER P, ROMBACH R, OMMER B. Taming transformers for high-resolution image synthesis [C]. IEEE International Conference on Computer Vision and Pattern Recognition, Virtual Event, 2021: 12873-12883. By training on haze-free images, the semantics contained in the haze-free images are compressed and a semantically rich codebook is established through model self-supervised learning. While improving the compression rate, it can still maintain good perceptual quality, thereby serving as a priori knowledge of haze-free images and being used as a constraint condition for subsequent defogging models to generate haze-free images;

[0033] The mathematical representation of the codebook is Given a high-quality image x, as a VQ encoder E vq The input is a latent feature map z, and then each pixel z ij Match the nearest element in the codebook to obtain the discrete feature map of the codebook The discretized features are then input into the VQ decoder D vq , and then obtain the processed image y. The whole process can be expressed as follows:

[0034] z ij =Evq (x ij )

[0035]

[0036] Among them, Z ij represents the pixel at position ij in the potential feature map Z, E vq () represents VQ encoder, x ij represents the pixel at position ij in the high-quality image x, represents the discrete feature map after the codebook, M() represents the codebook matching operation, ||zz k || represents the discretized feature Z and codebook encoding z of the foggy image k The distance between, argmin() represents the minimum distance function, y ij represents the pixel at position ij of the generated image,

[0037] It is worth noting that the discrete operation between the encoder and the decoder during network training cannot be back-propagated through the gradient because the operation is not differentiable. Therefore, during back-propagation, the gradient is simply copied from the decoder to the encoder, so that the model can be trained end-to-end through the loss function.

[0038] The features of the clear image are compressed into short vectors through discretization operations and stored in the codebook. Therefore, the discrete codebook compresses the detailed texture features of the image and plays a key role in image reconstruction. The discriminator is used to judge the authenticity of the reconstructed image to assist the network in further learning. The training is completed until the discriminator cannot judge the authenticity of the image.

[0039] S23: According to the potential feature map z and the discrete feature map z q , construct a codebook discretization loss function, and update the parameters of the model's VQ encoder and codebook with the minimum loss function as the optimization goal;

[0040] The codebook discretization loss function is defined as follows:

[0041]

[0042] Among them, L Z represents the codebook discretization loss function, z q represents the discrete feature map, Z represents the potential feature map, β represents the weight factor, sg[Z q ] indicates stop-gradient. This loss function mainly measures the output Z of the encoder and the discrete vector z q Discretization loss between

[0043] S24: Discrete feature map z qSend it to the VQ decoder for decoding to obtain the reconstructed clear image y;

[0044] S25: constructing a loss function of the VQ decoder according to the reconstructed clear image y and the original clear image x, and updating the parameters of the VQ decoder of the model with the minimum loss function as the optimization goal;

[0045] The specific details of this step are shown in S46;

[0046] In the second stage, a dual-branch multi-scale neural network is constructed. The Transformer-based neighborhood attention and convolution-based channel attention are used to extract the global features of the foggy image and learn the complex interaction features between the dense fog area and the underlying scene. The discrete codebook obtained from the clear image in the first stage is used to assist the reconstruction process of the foggy image. Specifically, a controllable distance recalculation operation is used to match the corresponding discrete codebook and replace the area affected by haze in the image, thereby achieving the defogging effect.

[0047] S3: Get the original image dehazing dataset;

[0048] The original image defogging dataset includes: the original foggy image and the corresponding clear fog-free image;

[0049] In the specific implementation, the datasets used are O-HAZE, I-HAZE, DENSE-HAZE, NH-HAZE-20, NH-HAZE-21, and NH-HAZE-23. O-HAZE and I-HAZE contain 45 pairs of foggy and corresponding clear images of outdoor scenes and 35 pairs of indoor scenes, respectively. DENSE-HAZE is characterized by dense and uniform haze scenes. The dataset contains 55 pairs of real dense fog images and corresponding fog-free images of various outdoor scenes. These dense fog images are obtained by professional haze researchers. The machine-generated dense fog image is almost indistinguishable from the original objects in the image. Compared with conventional datasets, defogging is very difficult. The NH-HAZE dataset is a set of real non-uniform fog images and corresponding fog-free image pairs. The non-uniform fog is introduced by a professional fog generator to simulate real foggy conditions. According to the different shooting and release dates, it can be further divided into NH-HAZE-20, NH-HAZE-21, and NH-HAZE-23, with 40, 25, and 55 image pairs respectively. For more detailed information, refer to Table 1:

[0050] Table 1:

[0051]

[0052]

[0053] S4: Use the original image dehazing dataset to train the dual-branch multi-scale image dehazing network model;

[0054] A dual-branch multi-scale image dehazing network model, which is divided into a priori matching branch and a channel attention branch;

[0055] The prior matching branch includes a VQ encoder, a pyramid hole neighborhood attention encoder, a fixed parameter codebook matching module, a fixed parameter VQ decoder, and an enhanced decoder; the pyramid hole neighborhood attention encoder is as follows: Figure 4 As shown, the enhanced decoder is Figure 5 As shown;

[0056] The channel attention branch includes a 3×3 convolution and 4 residual channel attention layers;

[0057] Finally, the results of the two branches are fused through a feature fusion structure;

[0058] Preferably, the training of the dual-branch multi-scale image defogging network model in S4 includes:

[0059] S41: Input the original foggy image x into the VQ encoder to roughly extract features to obtain preliminary features F1, and then input the preliminary features F1 into the pyramid void neighborhood attention encoder to obtain high-level features F2;

[0060] The VQ encoder performs well in encoding clear images, but it is not capable of encoding images with dense fog or non-uniform haze. This is mainly because in the dehazing task, the encoder not only needs to extract the common structural texture features in the image, but also needs to identify the haze area in the image. The VQ encoder network architecture is shallow and cannot complete this task well.

[0061] In order to fully extract global features such as texture and structure of foggy images, this paper designs an encoder based on hole neighborhood attention in the prior matching branch. The hole neighborhood attention is a variant of the self-attention in the Vision Transformer. It is an effective and scalable visual sliding window attention mechanism, and it also surpasses the Vision Transformer and Swin Transformer in visual downstream performance.

[0062] The neighborhood attention Transformer consists of a multi-layer perception (MLP), a normalization layer (LayerNorm, LN), a residual connection and a multi-head neighborhood attention (NA). When the neighborhood size is the smallest, each pixel only pays attention to the neighborhood of one pixel around itself. When the neighborhood size reaches the maximum value, the output of the neighborhood attention is equal to the self-attention.

[0063] Compared with self-attention, neighborhood attention not only reduces the computational cost, but also introduces a local inductive bias similar to convolution. Specifically, NA is a pixel-by-pixel operation that localizes self-attention (SA) to the nearest neighbor pixel. Therefore, compared with the quadratic complexity of self-attention, neighborhood attention has linear time and space complexity.

[0064] In the hollow neighborhood attention encoder designed by the present invention, four feature maps of different resolutions are obtained through a serializer and two downsamplings, and a pyramid structure is used to take the feature information of each previous layer as the input of the next layer through a cascade operation, aggregate features of different levels, and realize feature reuse of different scales;

[0065] Therefore, the present invention adopts a neighborhood attention mechanism to focus on the global features of the image while using high-quality priors for feature matching, thereby improving the generalization of the network;

[0066] First, the VQ encoder is used to roughly extract the image position and structure information, and then the shallow features are serialized through the Overlapping Tokenizer to be input into the neighborhood attention Transformer block. A downsampler is connected after the second neighborhood attention Transformer block, which reduces the spatial size to half of the original size and doubles the number of channels, thus generating feature maps of different scales.

[0067] In order to fuse feature maps of different scales, the present invention designs a pyramid-shaped feature aggregation mode, in which many dense connection operations are added. Since direct addition operation will lose some original feature information in feature fusion, and cascade operation is strictly a lossless operation, dense connection is used in the process of step-by-step processing of feature maps to use the feature information of each previous layer as the input of the next layer through addition and cascade operations, thereby aggregating features of different levels and realizing feature reuse of different scales, thereby optimizing the global information in the features. Through multiple residual connections, features of different levels are allowed to be fused, thereby extracting multi-scale features of image fog distribution, which is conducive to subsequent high-quality prior matching;

[0068] S42: inputting the high-level feature F2 into a codebook matching module with fixed parameters for matching, and obtaining a matched feature F3;

[0069] When reconstructing an image using a high-quality codebook, the discrete code output by the encoder is difficult to match to the corresponding high-quality codebook. This is mainly due to the serious image degradation. There is a domain gap between foggy and non-fog images, resulting in inconsistent data distribution. Therefore, it is necessary to match the distance between the encoder output and the codebook, and use the matching operation of the controllable distance recalculation method to reduce the problem caused by the domain gap, thereby achieving a better reconstruction effect.

[0070] Calculate the distance between the discrete code of the foggy image and each code in the codebook to find the codebook code with the smallest distance, and then adjust the final calculated distance through a weight function F to obtain the matching formula:

[0071]

[0072] F(f k ,α)=f k ×e α

[0073] Among them, M(z) represents the matching process of the codebook matching module, F() represents the weight function generated according to the frequency difference, and f k represents the frequency difference between the foggy image and the clear image in the codebook activation, α represents the parameter used to adjust the degree of defogging, ||zz k || represents the discretized feature z and codebook encoding z of the foggy image k The distance between, argmin() represents the function of taking the minimum distance.

[0074] In the matching formula, the activation frequency difference f of the decoding code is required k and α, for the activation frequency difference of the codebook code, at the beginning of the defogging network training, the activation frequency difference of each code in the codebook is set to 0. If the current foggy image does not match the codebook code and the clear image matches the codebook code, the frequency difference on the code is updated. After multiple trainings, the network will learn an optimal value. For the value of α, the difference between the code obtained by the encoder and the code of the clear and fog-free image is expressed by the difference between the two probability distributions. The Kullback-Leibler Divergence (KL divergence) is an indicator used to measure the similarity of two probability distributions. The greater the similarity, the smaller the value of the KL divergence. Let the probability distribution of the codebook activation of the clear image be P c , the probability distribution corresponding to the foggy image is P h, and the probability distribution of foggy images can be adjusted by ɑ. In this way, the adaptation problem of two different domains, the domain where the fog-free image is located and the domain where the foggy image is located, is transformed into solving an optimal parameter ɑ so that P c (x=z k ) and P h (x=z k |ɑ) has the smallest KL divergence;

[0075]

[0076]

[0077] S43: Input the original clear and fog-free image into the VQGAN network model pre-trained in S2, and pass through the VQ encoder and codebook in sequence to obtain the intermediate feature F4;

[0078] The details of this step are detailed in S2;

[0079] S44: Based on the preliminary feature F1, the high-level feature F2, the matched feature F3 and the intermediate feature F4, the encoder loss function of the dual-branch multi-scale image dehazing network model is constructed, and the parameters of the VQ encoder and the pyramid void neighborhood attention encoder of the model are updated with the minimum loss function as the optimization goal;

[0080] In order to help the encoder output match the correct high-quality codebook prior in the subsequent steps, we need to specifically make the features of the encoder output satisfy the standard normal distribution consistent with that used when training the high-quality prior;

[0081] Assume the foggy image input is x h , the haze-free image input is x gt , the defogging network encoder is E, and the encoder used for training the codebook is E vq , we can get the intermediate feature z of the foggy image after being processed by encoder E h =E(x h ), the haze-free image is passed through the encoder E vq Processed intermediate features

[0082] In the process of controlling image generation, we also need to control the style difference between the generated image and the haze-free image. Therefore, Ψ, the Gram matrix, is used to measure the style loss, and the discriminator D during codebook training is used to determine whether the generated features are real. Therefore, the final encoder loss is:

[0083]

[0084] Among them, L VQrepresents the encoder loss function of the dual-branch multi-scale image dehazing network model, z h represents the intermediate features of the foggy image, represents the intermediate features of the haze-free image, λ style and λ adv represents the first and second hyperparameters used to adjust the weights of different losses, Ψ() represents the matrix used to measure style loss, E[] represents the encoder, D() represents the discriminator, represents the i-th intermediate feature of the foggy image.

[0085] S45: Send the feature F3 to the VQ decoder and the enhanced decoder with fixed parameters for decoding, and obtain the intermediate feature F5;

[0086] Specifically, the output result obtained only by the VQ decoder is prone to lack of detail information in some places with deep haze, and the image structure and texture are relatively blurred. In order to improve the decoding ability of the detail features of the foggy image, the present invention designs an enhanced decoder based on multiple attentions in the prior matching branch, combines channel attention and pixel attention, and finally uses an enhancement block based on pyramid pooling to ensure that feature details of different scales are embedded in the final result;

[0087] S46: Input the original foggy image into the channel attention branch to obtain the intermediate feature F6;

[0088] Adding channel attention branches pays more attention to non-uniform haze and dense fog areas with significant brightness changes, avoids over-enhancement problems, and improves the overall reconstruction performance of images. The attention mechanism enables the network to flexibly focus on haze features to reconstruct high-quality fog-free images. Non-uniform haze and dense fog will significantly increase the brightness of the occluded areas. Paying more attention to the restoration of areas with significant brightness changes, such as the sky and snow, can avoid over-enhancement problems, thereby improving the overall reconstruction performance of images.

[0089] S47: The intermediate features F5 and F6 are added through a channel, and then the defogging image y is generated through a feature fusion module;

[0090] The final feature fusion part uses a feature fusion module consisting of a reflection padding layer, a convolutional layer, and a Tanh activation function to fuse the outputs of the two branches;

[0091] S48: constructing a loss function of the VQ decoder according to the clear image y after defogging and the original clear image x, and updating the parameters of the VQ decoder of the model with the minimum loss function as the optimization goal;

[0092] At this stage, it is necessary to determine whether the image finally generated by the entire network has correctly completed the dehazing task. Note that since the loss is calculated separately, the parameters of the encoder and decoder are not updated in the same dimension, so the gradient at this stage cannot be back-propagated to the encoder;

[0093] Therefore, the losses of other parts of the network are calculated by combining the following losses:

[0094] Smoothed L1 loss:

[0095] The smooth L1 loss function (also known as the mean absolute error) combines the advantages of L1 and L2 losses. It measures the average error between the predicted value and the true value. The advantage of the smooth L1 loss function is that the derivative at point 0 is continuous, which makes the solution more efficient and converges faster.

[0096]

[0097] Among them, x i and i represents the i-th pixel of the clear image and the foggy image respectively, N is the total number of pixels, and f θ (~) represents the dehazing network, f θ (x i ) represents the i-th pixel of the image reconstructed by the dehazing network;

[0098] MS-SSIM loss:

[0099] MS-SSIM loss is based on the assumption that the human eye obtains image structure information and can provide a reference for image quality perception standards. Let O and G represent two windows centered on the i-th pixel in the dehazed image and the real image, respectively. Gaussian filters are applied to the two windows to generate the corresponding mean (μ O ,μ G ), standard deviation σ O , σ G and covariance σ OG , the MS-SSIM loss can be expressed as follows:

[0100]

[0101] Among them, C1 and C2 are two very small values, which help to stabilize the fraction in the formula;

[0102] Perceptual Loss:

[0103] The perceptual loss is measured using VGG16 pre-trained on the ImageNet dataset. ImageNet is a computer vision dataset created by Fei-Fei Li, a professor at Stanford University. The dataset contains more than 14 million images and more than 20,000 categories. VGG16 pre-trained on this dataset has a good perceptual effect and can help the model reconstruct finer details.

[0104] Let x and y represent the foggy image and the clean image respectively, f θ (x) represents the image reconstructed by the dehazing network, Φ j (·) represents the feature map after being processed by the jth layer of VGG16. The L2 loss is used to measure the distance between the feature map of the reconstructed image and the clean image. N represents the number of feature maps used to calculate the perceptual loss.

[0105]

[0106] Fighting Loss:

[0107] Since the pixel-based loss function cannot provide sufficient supervision on small datasets, adversarial loss is added to make up for the shortcomings of the above loss;

[0108]

[0109] Where D represents the discriminator used in training the codebook, and M represents the number of sample data;

[0110] Total loss:

[0111] The weights of the smooth l1 loss function, MS-SSIM loss, perceptual loss, and adversarial loss are initially set to 1, 0.5, 0.01, and 0.0005, respectively. They will be adjusted later based on experimental results, so the total image reconstruction loss is expressed as follows:

[0112] L=L l1 +0.5L MS-SSIM +0.01L perc +0.0005L adv

[0113] S5: Input the foggy image into the trained dual-branch multi-scale image defogging network model to obtain a clear fog-free image;

[0114] Preferably, during training, the input images are randomly cropped to a size of 256×256, and the dataset is augmented by scaling, random rotation, and flipping. The Adam optimizer is used, with default β1 and β2 of 0.9 and 0.99, respectively, an initial learning rate of 0.0001, and a batch size of 1. The model is implemented based on Pytorch on an NVIDIA V100 Tensor Core.

[0115] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dual-branch multi-scale image defogging method based on high-quality codebook, characterized in that: include: S1: Acquire an original image super-resolution reconstruction dataset, where the original image super-resolution reconstruction dataset includes an original clear image; S2: Use the original image super-resolution reconstruction dataset to train the VQGAN network model to obtain the network structure of the codebook and VQ decoder and their corresponding parameters; The VQGAN network model includes a VQ encoder, a codebook, and a VQ decoder; S3: Acquire an original image defogging dataset, where the original image defogging dataset includes: an original foggy image and a corresponding clear fog-free image; S4: Use the original image dehazing dataset to train the dual-branch multi-scale image dehazing network model; The dual-branch multi-scale image defogging network model is divided into a priori matching branch and a channel attention branch; wherein the priori matching branch includes a VQGAN network model with fixed parameters, a pyramid void neighborhood attention encoder, and an enhanced decoder, and its structure is composed of a VQ encoder, a pyramid void neighborhood attention encoder, a fixed parameter codebook matching module, a fixed parameter VQ decoder, and an enhanced decoder in sequence; the channel attention branch includes a 3×3 convolution and 4 residual channel attention layers; and finally the results of the two branches are fused through a feature fusion structure; S5: Input the foggy image into the trained dual-branch multi-scale image defogging network model to obtain a clear and fog-free image.

2. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 1 is characterized in that: The steps for training the VQGAN network model in step S2 are: S21: Input the original clear image x into the VQ encoder E based on the UNet network architecture vq Extraction and sampling are performed in the process, and multiple residual structures are used to splice edge features in the sampling process to obtain a potential feature map z; S22: Match the potential feature map z to the nearest element in the codebook to obtain the discrete feature map z after the codebook q ; S23: According to the potential feature map z and the discrete feature map z q , construct a codebook discretization loss function, and update the parameters of the model's VQ encoder and codebook with the minimum loss function as the optimization goal; S24: Discrete feature map z q Send it to the VQ decoder for decoding to obtain the reconstructed clear image y; S25: According to the reconstructed clear image y and the original clear image x, a loss function of the VQ decoder is constructed, and the parameters of the VQ decoder of the model are updated with the minimum loss function as the optimization goal.

3. The dual-branch multi-scale image defogging method based on a high-quality codebook according to claim 1 or 2, characterized in that: The mathematical expression of the codebook is: Where Z represents the codebook, z k represents the codebook encoding, K represents the number of codebook encodings, Represents the codebook encoding set.

4. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 2 is characterized in that: The codebook discretization loss function includes: Among them, L Z represents the codebook discretization loss function, sg[] represents the stop-gradient, z q represents the discrete feature map, z represents the latent feature map, and β represents the weight factor.

5. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 1, characterized in that: The training steps for the dual-branch multi-scale image dehazing network model in S4 include the following steps: S41: Input the original foggy image x into the VQ encoder to roughly extract features to obtain preliminary features F1, and then input the preliminary features F1 into the pyramid void neighborhood attention encoder to obtain high-level features F2; S42: input the high-level feature F2 into a codebook matching module with fixed parameters for matching, and obtain a matched feature F3; S43: Input the original clear and haze-free image into the pre-trained VQGAN network model described in S2, and pass through the VQ encoder and codebook in sequence to obtain the intermediate feature F4; S44: Based on the preliminary feature F1, the high-level feature F2, the matched feature F3 and the intermediate feature F4, the encoder loss function of the dual-branch multi-scale image dehazing network model is constructed, and the parameters of the VQ encoder and the pyramid void neighborhood attention encoder of the model are updated with the minimum loss function as the optimization goal; S45: Send the feature F3 to the VQ decoder and the enhanced decoder with fixed parameters for decoding, and obtain the intermediate feature F5; S46: Input the original foggy image into the channel attention branch to obtain the intermediate feature F6; S47: The intermediate features F5 and F6 are added through a channel, and then the defogging image y is generated through a feature fusion module; S48: According to the dehazed clear image y and the original clear image x, the loss function of the remaining part of the dual-branch multi-scale image dehazing network model is constructed, including smooth l1 loss function, MS-SSIM loss, perceptual loss, and adversarial loss. The parameters of the remaining part of the model are updated with the minimum loss function as the optimization goal.

6. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 5, characterized in that: The pyramid hole neighborhood attention encoder processing process includes the following steps: Step 1: The input data is serialized and downsampled twice to obtain four feature maps of different resolutions. The pyramid structure is used to aggregate features of different levels by cascading the feature information of each previous layer as the input of the next layer. Step 2: Use multiple residual connections to allow features at different levels to be fused, thereby extracting multi-scale features of the image fog distribution and obtaining output features.

7. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 5, characterized in that: The matching process of the codebook matching module is expressed as: F(f k ,α)=f k ×e α Among them, M(z) represents the matching process of the codebook matching module, F() represents the weight function generated according to the frequency difference, and f k represents the frequency difference between the foggy image and the clear image in the codebook activation, α represents the parameter used to adjust the degree of defogging, ||zz k || represents the discretized feature z and codebook encoding z of the foggy image k The distance between them, argmin() represents the function of taking the minimum distance.

8. The dual-branch multi-scale image defogging method based on high-quality codebook according to claim 5, characterized in that: The encoder loss function of the dual-branch multi-scale image dehazing network model includes: Among them, L VQ represents the encoder loss function of the dual-branch multi-scale image dehazing network model, z h represents the intermediate features of the foggy image, represents the intermediate features of the haze-free image, λ style and λ adv represents the first and second hyperparameters used to adjust the weights of different losses, Ψ( ) represents the matrix used to measure style loss, E[ ] represents the encoder, D( ) represents the discriminator, represents the i-th intermediate feature of the foggy image.

Images