A two-stage decoupled image defogging method based on zero-shot learning

Abstract

This invention discloses a two-stage decoupled image dehazing method based on zero-shot learning. This method employs three sub-networks to decouple observed haze images into several latent layers, including a haze-free image layer, a transmission layer, and an atmospheric light layer. The main focus of this method is to recover the haze-free image and transmission map. In short, the method uses an embedded dark channel prior (DCP) in the first stage to obtain coarse estimates of the haze-free image and transmission map. In the second stage, two sub-networks refine the results from the first stage to obtain more accurate haze-free images and transmission maps, while another sub-network directly estimates the atmospheric light. Specifically, we also propose a novel multi-scale Transformer block and apply it to the sub-network refining the haze-free image. It performs multi-scale token aggregation in the self-attention part, enabling it to capture features at different scales and effectively recover latent scene information in the haze map.

Description

Technical Field

[0001] This invention relates to the field of image dehazing technology, specifically to a two-stage decoupled image dehazing method based on zero-shot learning. Background Technology

[0002] Haze is a phenomenon caused by the accumulation and suspension of large amounts of particulate matter in the air. Due to atmospheric absorption and scattering, light is affected by particulate matter during propagation, resulting in reduced image clarity and contrast, and severe image quality degradation in images acquired by various image acquisition devices. This poses significant challenges to subsequent image processing operations. In severe cases, these factors can also distort color information in images, making it even more difficult to apply various computer vision technologies.

[0003] In recent years, with the widespread adoption of computer vision systems, these systems have played a vital role in roads, aviation, and other fields. However, low visibility due to foggy weather has caused significant challenges for visual equipment used in aviation and road traffic. Therefore, image dehazing has always been a focus of research in the field of computer vision.

[0004] The goal of image dehazing is to estimate potential haze-free images from observed haze images. For single-image dehazing, a commonly used model represents the degradation process of haze images:

[0005] I=J(x)t(x)+A(1-t(x))

[0006] Where I represents the hazy image, J represents the potentially hazy image, A represents the global atmospheric light, t represents the transport graph, and x represents pixel location information. The transport graph can be represented as:

[0007] t(x)=e -βd(x) ,

[0008] Where β is the atmospheric scattering coefficient and d is the scene depth.

[0009] As can be seen from the above, image dehazing is a typical ill-posed problem. Consequently, many image dehazing methods have been proposed, which can be broadly categorized into prior-based methods and learning-based methods. Prior-based methods, i.e., traditional dehazing methods, tend to use prior knowledge inherent in the image itself for dehazing. For example, Dark Channel Prior (DCP) is used to detect the haze distribution in an image, and Color Attenuation Prior (CAP) is used to estimate the transmission map. Although these prior-based methods have achieved certain results, the prior knowledge they use is easily refuted in practice and exhibits low robustness in complex real-world scenarios. Summary of the Invention

[0010] To address the shortcomings of existing technologies, the present invention aims to provide a two-stage decoupled image dehazing method based on zero-shot learning. This method is based on zero-shot learning and directly inputs hazy images into the network for dehazing without providing hazy-clean image pairs, thus avoiding large-scale data training. At the same time, in the process of restoring hazy images, this method can not only ensure the extraction of multi-scale features, but also ensure the extraction of original scale and neighborhood information.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a two-stage decoupling image dehazing method based on zero-shot learning, comprising the following steps:

[0012] (1) Obtain two test subsets from the realistic single image dehazing (RESIDE) dataset: the Synthetic Objective Testing Set (SOTS) and the Hybrid Subjective Testing Set (HSTS). Preprocess the original dataset and use it as the test set.

[0013] (2) Constructing a defogging network model: Based on the atmospheric scattering physical model I=J(x)t(x)+A(1-t(x)),

[0014] Where x represents the input haze image, I(x) represents the reconstructed hazy image, J(x) represents the original scene information, i.e. the clear, haze-free image, t(x) represents the transmittance, and A represents atmospheric scattered light.

[0015] Based on the idea of ​​decoupling, the network model is divided into three layers: a potential haze-free image layer, a transport layer, and a global atmospheric light layer. The potential haze-free image and the transport layer are estimated in two stages. The first stage uses the embedded dark channel prior for coarse estimation, and the second stage uses the haze-free image refinement network and the transport layer refinement network for fine estimation. For global atmospheric light, an encoder-decoder network (i.e., atmospheric light estimation network) is used for estimation. At the top of the network, the images estimated by the three sub-networks (haze-free image refinement network, transport layer refinement network, and atmospheric light estimation network) are reconstructed according to the atmospheric scattering physical model to obtain the reconstructed haze image.

[0016] (3) Input the haze images in the preprocessed dataset into the constructed dehazing network model, calculate the loss according to the designed loss function, continuously iterate and update the parameters, and directly dehaze the haze images.

[0017] As a preferred option: step (2), the first stage of rough estimation specifically includes the following steps:

[0018] (2.1) Calculate the dark channel of the input image x: First, take the minimum value among the RGB three channels to construct a grayscale image, then invert the grayscale image and perform max pooling. Invert the result of max pooling to obtain I. dark The specific formula is as follows Where I c This refers to one of the R, G, and B channels of I;

[0019] (2.2) Calculate a rough estimate T of the transmission graph DCP (x): I dark Substituting into the atmospheric scattering physics model, we obtain I dark (x)=J dark (x)t(x)+A(1-t(x)), where I dark (x) and J dark (x) represent the dark channels of images I and J, respectively; for atmospheric light A, in I dark The top 0.1% of pixels by brightness are selected, and their average value is taken as the atmospheric light A value; while for J... dark Based on the dark channel prior assumption, the darkness in the non-sky region of an outdoor, fog-free image J typically tends to zero. dark (x)→0, and thus according to Preliminary results of the transmission map were obtained; due to the uneven edge transitions in the dark channel image, guided filtering was used to improve the T... DCP The edges of (x) are smoother, and a guided filter is implemented using an average pooling operation with a kernel size of 19*19 and a stride of 1.

[0020] (2.3) Calculate a rough estimate of J for the fog-free map DCP (x): Based on the atmospheric scattering physics model, and given atmospheric light A and transmission diagram T. DCP (x), J DCP (x): Obtained through equation calculation.

[0021] As a preferred option: Step (2) involves constructing three subnets of the dehazing network model, specifically including the following steps:

[0022] (A) The architecture of the fog-free image thinning network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers. Soft reconstruction layers are used to obtain global residuals.

[0023] (B) The transmission graph refinement network is a non-degenerate network consisting of five convolutional layers. In the first four layers, each layer only includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer contains a convolutional layer and a sigmoid function that normalizes the output to [0,1].

[0024] (C) The atmospheric light estimation network is a symmetric encoder-decoder architecture network. The encoder and decoder each have four layers. Each layer of the encoder contains a convolutional layer, a ReLU activation function, and a max pooling layer. Each layer of the decoder contains an upsampling layer, a convolutional layer, a batch normalization layer, and a ReLU activation function. Between the encoder and decoder, a reparameterization trick module is used. This module transforms the encoder output into a latent Gaussian distribution, then resamples it, and then inputs it into the decoder.

[0025] Preferably, in step (A), the multi-scale Transformer block is constructed, which specifically includes the following steps:

[0026] (A1) The RescaleNorm layer is used as the normalization layer. The specific normalization process of RescaleNorm is as follows:

[0027] Where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block. These represent the mean and standard deviation, respectively. These are the learned scaling factor and the bias, respectively. and These are two linear layer weights and bias terms used to transform μ and σ, respectively. The transformation process is represented as {γ,β}={σW}. γ +B γ μW β +B β To accelerate convergence, B γ and B β Initialize to 1 and 0.

[0028] (A2) Multi-scale self-attention is used to compute self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregate multi-scale feature information within the window and preserve neighborhood information to a certain extent.

[0029] Preferably, in step (A2), multi-scale self-attention (MSSA) is calculated, which includes two branches:

[0030] (A21) In the branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is preserved, and a normal fully connected operation is performed on it. For key K and value V, scaling transformation (ST) is used in the same self-attention layer to obtain K and V of different sizes, specifically as follows:

[0031] Q = XW Q ,

[0032] K i V i =ST i (X)W i K ST i (X)W i V ,

[0033] V i =V i +LE i (V i ),

[0034] Where X represents the input feature sequence, W Q W i K W i V It is the linear projection parameter of the i-th scale layer onto the same head of the self-attention layer, LE i (·) represents the local augmentation component of depthwise convolution; while ST i (·) is the scaling operation for the i-th scale layer, which provides downsampling of X, and can be specifically expressed as:

[0035] ST i (X)=LN(Conv i (X)),

[0036] Where LN(·) refers to Layer Normalization; Conv i (·) refers to the convolution used in the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations. Therefore, in the same self-attention layer, keys and values ​​capture features at different scales. Then, the self-attention head can be calculated as:

[0037]

[0038] Where Softmax(·) is the activation function, T represents the transpose operation, and d is the dimension. The different heads are connected, and multi-head self-attention (MH) in the main branch is calculated using the following method:

[0039] MH(Q,K,V)=Concat(head0,…,head j W O ,

[0040] Where Concat(·) is the concatenation operation, head i W represents the i-th self-attention head. O These are linear projection parameters;

[0041] (A22) Another branch that preserves neighborhood information only performs linear projection and convolution operations on the input features sequentially. Multi-scale self-attention MSSA can be represented as:

[0042] MSSA = Concat(MH{Q,K) i V i} i＝1,…,m )+Conv0(XW 0 ),

[0043] Where Concat(·) is the join operation, MH represents multi-head self-attention, and Q, K i V i Let W represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively. Conv0(·) refers to the convolution operation, X is the input feature sequence, and W is the input feature sequence. O These are linear projection parameters;

[0044] Preferably, in step (3), the loss function consists of five loss functions:

[0045] (3.1) Reconstruction loss L Rec L is used to calculate the error between the input image and the reconstructed image, thereby constraining the entire network. Rec Defined as:

[0046] L Rec =||I(x)-x|| F ,

[0047] Among them, ||·|| F Let Frobinius norm be the given matrix, x be the input haze image, and I(x) be the haze image reconstructed from the outputs of the three subnets.

[0048] (3.2) Loss function L J Used to calculate the estimated J of a fog-free image. R The difference between the brightness and saturation of (x) is specifically expressed as:

[0049] L J =||V(J R(x))-S(J R (x))|| F ,

[0050] Where V(·) represents brightness, S(·) represents saturation, and ||·|| F Let Frobinius norm be the given matrix.

[0051] (3.3) Atmospheric light loss L H Used to calculate the estimated atmospheric light A(x) and A init The loss between (x), where A init (x) is the initial atmospheric light automatically estimated from the input image data, L H Specifically, it is expressed as follows:

[0052] L H =||A(x)-A init (x)|| F .

[0053] (3.4)L KL It is the variational inference term loss of the reparameterization module in the atmospheric estimation optical network, which makes the resampled latent code in the network... Minimizing the difference between z and the sampled values ​​is mathematically...

[0054]

[0055] Where KL(·) represents the Kullback-Leibler divergence between the two distributions, z i Let μ represent the i-th dimension of z. z ,σ z Let z represent the mean and standard deviation, respectively. They represent z respectively i The mean and standard deviation;

[0056] (3.5) To avoid network overfitting, the regularization term L is set to reduce the loss. Reg Specifically, it is expressed as:

[0057]

[0058] Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels in A(x), and A(x) represents atmospheric light;

[0059] The total loss is defined by combining the five loss functions mentioned above:

[0060] L = L Rec +L J +L H +L KL +λLReg ,

[0061] Among them, L Reg The parameter λ is a non-negative parameter used to balance regularization, and in practice it is 0.1.

[0062] By adopting the above technical solution, the present invention has the following beneficial effects:

[0063] This invention constructs an image dehazing network model: based on the atmospheric scattering physics model, the network model is divided into three layers based on the idea of ​​decoupling: a latent haze-free image layer, a transport layer, and a global atmospheric light layer. The latent haze-free image and transport layer are estimated using two stages: the first stage uses an embedded dark channel prior for coarse estimation, and the second stage uses a haze-free image refinement network and a transport layer refinement network for fine estimation, respectively. For global atmospheric light, an encoder-decoder network is used for estimation. At the top of the network, the images estimated by the three sub-networks are reconstructed according to the atmospheric scattering physics model to obtain the reconstructed haze image. A novel multi-scale Transformer block is introduced into the fine haze-free image sub-network of the dehazing network model. It performs multi-scale token aggregation in the self-focused part, enabling it to capture features at different scales and effectively recover latent scene information in the haze image.

[0064] This invention is based on zero-shot learning, which directly inputs haze images into the network for dehazing without providing haze-clean image pairs, thus avoiding large-scale data training. At the same time, in the process of restoring haze-free images, this method can not only ensure the extraction of multi-scale features, but also ensure the extraction of original scale and neighborhood information.

[0065] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the defogging network model according to an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of a multi-scale Transformer block according to an embodiment of the present invention. Detailed Implementation

[0068] See Figure 1 and Figure 2 This invention discloses a two-stage decoupling image dehazing method based on zero-shot learning, comprising the following steps:

[0069] (1) Obtain two test subsets from the realistic single image dehazing (RESIDE) dataset: the Synthetic Objective Testing Set (SOTS) and the Hybrid Subjective Testing Set (HSTS). Preprocess the original dataset and use it as the test set.

[0070] (2) Constructing a defogging network model: Based on the atmospheric scattering physical model I=J(x)t(x)+A(1-t(x)),

[0071] Where x represents the input haze image, I(x) represents the reconstructed hazy image, J(x) represents the original scene information, i.e. the clear, haze-free image, t(x) represents the transmittance, and A represents atmospheric scattered light.

[0072] Based on the idea of ​​decoupling, the network model is divided into three layers: a potential haze-free image layer, a transport layer, and a global atmospheric light layer. The potential haze-free image and the transport layer are estimated in two stages. The first stage uses the embedded dark channel prior for coarse estimation, and the second stage uses the haze-free image refinement network and the transport layer refinement network for fine estimation. For global atmospheric light, an encoder-decoder network (i.e., atmospheric light estimation network) is used for estimation. At the top of the network, the images estimated by the three sub-networks (haze-free image refinement network, transport layer refinement network, and atmospheric light estimation network) are reconstructed according to the atmospheric scattering physical model to obtain the reconstructed haze image.

[0073] (3) Input the haze images in the preprocessed dataset into the constructed dehazing network model, calculate the loss according to the designed loss function, continuously iterate and update the parameters, and directly dehaze the haze images.

[0074] As a preferred option: step (2), the first stage of rough estimation specifically includes the following steps:

[0075] (2.1) Calculate the dark channel of the input image x: First, take the minimum value among the RGB three channels to construct a grayscale image, then invert the grayscale image and perform max pooling. Invert the result of max pooling to obtain I. dark The specific formula is as follows Where I c This refers to one of the R, G, and B channels of I;

[0076] (2.2) Calculate a rough estimate T of the transmission graph DCP (x): I dark Substituting into the atmospheric scattering physics model, we obtain I dark(x)=J dark (x)t(x)+A(1-t(x)), where I dark (x) and J dark (x) represent the dark channels of images I and J, respectively; for atmospheric light A, in I dark The top 0.1% of pixels by brightness are selected, and their average value is taken as the atmospheric light A value; while for J... dark Based on the dark channel prior assumption, the darkness in the non-sky region of an outdoor, fog-free image J typically tends to zero. dark (x)→0, and thus according to Preliminary results of the transmission map were obtained; due to the uneven edge transitions in the dark channel image, guided filtering was used to improve the T... DCP The edges of (x) are smoother, and a guided filter is implemented using an average pooling operation with a kernel size of 19*19 and a stride of 1.

[0077] (2.3) Calculate a rough estimate of J for the fog-free map DCP (x): Based on the atmospheric scattering physics model, and given atmospheric light A and transmission diagram T. DCP (x), J DCP (x): Obtained through equation calculation.

[0078] As a preferred option: Step (2) involves constructing three subnets of the dehazing network model, specifically including the following steps:

[0079] (A) The architecture of the fog-free image thinning network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers. Soft reconstruction layers are used to obtain global residuals.

[0080] (B) The transmission graph refinement network is a non-degenerate network consisting of five convolutional layers. In the first four layers, each layer only includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer contains a convolutional layer and a sigmoid function that normalizes the output to [0,1].

[0081] (C) The atmospheric light estimation network is a symmetric encoder-decoder architecture network. The encoder and decoder each have four layers. Each layer of the encoder contains a convolutional layer, a ReLU activation function, and a max pooling layer. Each layer of the decoder contains an upsampling layer, a convolutional layer, a batch normalization layer, and a ReLU activation function. Between the encoder and decoder, a reparameterization trick module is used. This module transforms the encoder output into a latent Gaussian distribution, then resamples it, and then inputs it into the decoder.

[0082] Preferably, in step (A), the multi-scale Transformer block is constructed, which specifically includes the following steps:

[0083] (A1) The RescaleNorm layer is used as the normalization layer. The specific normalization process of RescaleNorm is as follows:

[0084] Where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block. These represent the mean and standard deviation, respectively. These are the learned scaling factor and the bias, respectively. and These are two linear layer weights and bias terms used to transform μ and σ, respectively. The transformation process is represented as {γ,β}={σW}. γ +B γ μW β +B β To accelerate convergence, B γ and B β Initialize to 1 and 0.

[0085] (A2) Multi-scale self-attention is used to compute self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregate multi-scale feature information within the window and preserve neighborhood information to a certain extent.

[0086] Preferably, in step (A2), multi-scale self-attention (MSSA) is calculated, which includes two branches:

[0087] (A21) In the branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is preserved, and a normal fully connected operation is performed on it. For key K and value V, scaling transformation (ST) is used in the same self-attention layer to obtain K and V of different sizes, specifically as follows:

[0088] Q = XW Q ,

[0089] K i V i =ST i (X)W i K ST i (X)W i V ,

[0090] V i=V i +LE i (V i ),

[0091] Where X represents the input feature sequence, W Q W i K W i V It is the linear projection parameter of the i-th scale layer onto the same head of the self-attention layer, LE i (·) represents the local augmentation component of depthwise convolution; while ST i (·) is the scaling operation for the i-th scale layer, which provides downsampling of X, and can be specifically expressed as:

[0092] ST i (X)=LN(Conv i (X)),

[0093] Where LN(·) refers to Layer Normalization; Conv i (·) refers to the convolution used in the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations. Therefore, in the same self-attention layer, keys and values ​​capture features at different scales. Then, the self-attention head can be calculated as:

[0094]

[0095] Where Softmax(·) is the activation function, T represents the transpose operation, and d is the dimension. Multi-head self-attention in the main branch is computed by concatenating different heads using the following method:

[0096] MH(Q,K,V)=Concat(head0,…,head j W O ,

[0097] Where Concat(·) is the concatenation operation, head i W represents the i-th self-attention head. O These are linear projection parameters; (A22) Another branch that preserves neighborhood information only performs linear projection and convolution operations on the input features sequentially. Multi-scale self-attention can be expressed as:

[0098] MSSA = Concat(MH{Q,K) i V i} i＝1,…,m )+Conv0(XW 0 ).

[0099] Where Concat(·) is the join operation, MH represents multi-head self-attention, and Q, K i V i Let W represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively. Conv0(·) refers to the convolution operation, X is the input feature sequence, and W is the input feature sequence. O These are linear projection parameters;

[0100] Preferably, in step (3), the loss function consists of five loss functions:

[0101] (3.1) Reconstruction loss L Rec L is used to calculate the error between the input image and the reconstructed image, thereby constraining the entire network. Rec Defined as:

[0102] L Rec =||I(x)-x|| F ,

[0103] Among them, ||·|| F Let Frobinius norm be the given matrix, x be the input haze image, and I(x) be the haze image reconstructed from the outputs of the three subnets.

[0104] (3.2) Loss function L J Used to calculate the estimated J of a fog-free image. R The difference between the brightness and saturation of (x) is specifically expressed as:

[0105] L J =||V(J R (x))-S(J R (x))|| F ,

[0106] Where V(·) represents brightness, S(·) represents saturation, and ||·|| F Let Frobinius norm be the given matrix.

[0107] (3.3) Atmospheric light loss L H Used to calculate the estimated atmospheric light A(x) and A init The loss between (x), where A init (x) is the initial atmospheric light automatically estimated from the input image data, L H Specifically, it is expressed as follows:

[0108] L H =||A(x)-A init (x)|| F .

[0109] (3.4)L KLIt is the variational inference term loss of the reparameterization module in the atmospheric estimation optical network, which makes the resampled latent code in the network... Minimizing the difference between z and the sampled values ​​is mathematically...

[0110]

[0111] Where KL(·) represents the Kullback-Leibler divergence between the two distributions, z i Let μ represent the i-th dimension of z. z ,σ z Let z represent the mean and standard deviation, respectively. They represent z respectively i The mean and standard deviation;

[0112] (3.5) To avoid network overfitting, the regularization term L is set to reduce the loss. Reg Specifically, it is expressed as:

[0113]

[0114] Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels in A(x), and A(x) represents atmospheric light;

[0115] The total loss is defined by combining the five loss functions mentioned above:

[0116] L = L Rec +L J +L H +L KL +λL Reg ,

[0117] Among them, L Reg The parameter λ is a non-negative parameter used to balance regularization, and in practice it is 0.1.

[0118] In practical applications:

[0119] This invention provides a two-stage decoupled image dehazing method based on zero-shot learning, which can perform dehazing in an end-to-end manner and mainly consists of two stages: The first stage uses an embedded dark channel prior to obtain a coarse estimate of the haze-free image and transmission map; the second stage uses two sub-networks to refine the results of the first stage to obtain a more accurate haze-free image and transmission map, while another sub-network directly estimates atmospheric illumination. Specifically, this invention introduces a novel multi-scale Transformer block into the refined haze-free image sub-network of the dehazing network model. After constructing the dehazing network model, the loss is calculated according to a pre-designed loss function, and the parameters are iteratively updated to directly dehaze the haze image.

[0120] I. Two-stage decoupled defogging network model

[0121] The first stage of constructing a two-stage decoupled dehazing network model includes the following steps:

[0122] Step 1: Calculate the dark channel of the input image x. First, take the minimum value from the RGB channels to construct a grayscale image. Then, invert the grayscale image and perform max pooling. Invert the result of max pooling to obtain I. dark The specific formula is:

[0123]

[0124] Where I c It refers to one of the R, G, and B channels of I.

[0125] Step 2: Calculate a rough estimate T of the transmission graph DCP (x). I dark Substituting into the atmospheric scattering physics model, we get:

[0126] I dark (x)=J dark (x)t(x)+A[1-t(x)], (2)

[0127] Where I dark (x) and J dark (x) represent the dark channels of images I and J, respectively. For atmospheric light A, in I... dark The top 0.1% of pixels by brightness are selected, and their average value is taken as atmospheric light A. For J... dark Based on the dark channel prior assumption, the darkness in non-sky regions of an outdoor, fog-free image J typically tends to zero, i.e.

[0128] J dark (x)→0. (3)

[0129] Combining Equation 2, we obtain the preliminary results of the transmission diagram based on the following formula:

[0130]

[0131] Because of the uneven edge transitions in dark channel images, guided filtering is used to improve T... DCP The edges of (x) are smoother. A guided filter is implemented using an average pooling operation with a kernel size of 19*19 and a stride of 1.

[0132] Step 3: Calculate a rough estimate J for the fog-free map. DCP (x). Based on the atmospheric scattering physics model, and given the atmospheric light A and transmission diagram T. DCP (x), J DCP(x) can be calculated using the following equation:

[0133]

[0134] The second stage of constructing the two-stage decoupled dehazing network model specifically includes the following three sub-networks:

[0135] (1) Haze-free image thinning network: The architecture of the haze-free image thinning network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers. Soft reconstruction layers are used to obtain global residuals.

[0136] (2) Transmission Graph Refinement Network: The transmission graph refinement network is a non-degenerate network consisting of five convolutional layers. Specifically, the first four layers of this subnet each contain only a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer contains a convolutional layer and a sigmoid function that normalizes the output to [0,1].

[0137] (3) Atmospheric Light Estimation Network: The atmospheric light estimation network is a symmetric encoder-decoder architecture. The encoder and decoder each have four layers. Each layer of the encoder contains a convolutional layer, a ReLU activation function, and a max-pooling layer. Each layer of the decoder contains an upsampling layer, a convolutional layer, a batch normalization layer, and a ReLU activation function. Between the encoder and decoder, a reparameterization trick module is used. This module transforms the encoder output into a latent Gaussian distribution, then resamples it before inputting it into the decoder.

[0138] II. Multi-scale Transformer Block

[0139] A two-stage decoupled dehazing network model is constructed for haze-free image thinning, including a RescaleNorm layer, multi-scale self-attention, and an MLP layer. The RescaleNorm layer and the multi-scale self-attention introduced in this invention are described in detail here:

[0140] (1) RescaleNorm layer. As a normalization layer, the specific normalization process of RescaleNorm can be expressed as:

[0141]

[0142] Where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block. These represent the mean and standard deviation, respectively. These are the learned scaling factor and the bias, respectively. and These are two linear layer weights and bias terms used to transform μ and σ, respectively. The transformation process is represented as {γ,β}={σW}. γ +B γ μW β +B β To accelerate convergence, B γ and B β Initialize to 1 and 0.

[0143] (2) Multi-scale self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregate multi-scale feature information within the window while preserving information from the neighborhood to a certain extent.

[0144] In the main branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is preserved, and a normal fully connected operation is performed. For the key K and value V, scaling transformation (ST) is used within the same self-attention layer to obtain K and V of different sizes, specifically as follows:

[0145]

[0146] Where X represents the input feature sequence, W Q W i K W i V It is the linear projection parameter of the i-th scale layer onto the same head of the self-attention layer, LE i (·) represents the local augmentation component of depthwise convolution. ST... i (·) is the scaling operation for the i-th scale layer, which provides downsampling of X, and can be specifically expressed as:

[0147] ST i (X)=LN(Conv i (X)), (8)

[0148] Here, LN(·) refers to Layer Normalization. Conv i (·) refers to the convolution used in the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations. Therefore, within the same self-attention layer, keys and values ​​can capture features at different scales. Then, the self-attention head can be calculated as:

[0149]

[0150] Where Softmax(·) is the activation function, T represents the transpose operation, and d is the dimension. Multi-head self-attention in the main branch is computed by concatenating different heads using the following method:

[0151] MH(Q,K,V)=Concat(head0,…,head j W O (10)

[0152] Where Concat(·) is the concatenation operation, head i W represents the i-th self-attention head. O These are linear projection parameters.

[0153] Another branch that preserves neighborhood information only performs linear projection and convolution operations on the input features sequentially. In summary, multi-scale self-attention can be represented as:

[0154] MSSA = Concat(MH{Q,K) i V i} i＝1,…,m )+Conv0(XW 0 (11)

[0155] Where Concat(·) is the join operation, MH represents multi-head self-attention, and Q, K i V i Let W represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively. Conv0(·) refers to the convolution operation, X is the input feature sequence, and W is the input feature sequence. O These are linear projection parameters.

[0156] III. Loss Function

[0157] The loss function consists of the following five loss functions:

[0158] (1) Reconstruction loss L Rec This is used to calculate the error between the input image and the reconstructed image, thereby constraining the entire network. L Rec Defined as:

[0159] L Rec =||I(x)-x|| F (12)

[0160] Among them, ||·|| F Let Frobinius norm be the given matrix, x be the input haze image, and I(x) be the haze image reconstructed from the outputs of the three subnets.

[0161] (2) Loss function L J Used to calculate the estimated J of a fog-free image.R The difference between the brightness and saturation of (x) is specifically expressed as:

[0162] L J =||V(J R (x))-S(J R (x))|| F (13)

[0163] Where V(·) represents brightness and S(·) represents saturation.

[0164] (3) Atmospheric light loss L H Used to calculate the estimated atmospheric light A(x) and A init The loss between (x), where A init (x) is the initial atmospheric light automatically estimated from the input image data. L H Specifically, it is expressed as follows:

[0165] L H =||A(x)-A init (x)|| F (14)

[0166] (4)L KL It is the variational inference term loss of the reparameterization module in the atmospheric estimation optical network. Its purpose is to improve the resampled latent code in the network. Minimize the difference between z and the sampled value. Mathematically,

[0167]

[0168] Where KL(·) represents the Kullback-Leibler divergence between the two distributions, z i Let μ represent the i-th dimension of z. z ,σ z Let z represent the mean and standard deviation, respectively. They represent z respectively i The mean and standard deviation.

[0169] (5) To avoid network overfitting, the regularization term loss L is set. Reg Specifically, it can be expressed as:

[0170]

[0171] Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels in A(x), and A(x) represents atmospheric light.

[0172] In summary, the total loss is defined by combining the five loss functions mentioned above:

[0173] L = L Rec +L J +L H +L KL +λL Reg (17)

[0174] Where L Reg The parameter λ is a non-negative parameter used to balance regularization, and in practice it is 0.1.

[0175] This invention is based on zero-shot learning, which directly inputs haze images into the network for dehazing without providing haze-clean image pairs, thus avoiding large-scale data training. At the same time, in the process of restoring haze-free images, this method can not only ensure the extraction of multi-scale features, but also ensure the extraction of original scale and neighborhood information.

[0176] Based on the atmospheric scattering physics model, the network model is divided into three layers using the decoupling concept: a latent haze-free image layer, a transport layer, and a global atmospheric light layer. The latent haze-free image and transport layer are estimated in two stages: the first stage uses an embedded dark channel prior for coarse estimation, and the second stage uses a haze-free image refinement network and a transport layer refinement network for fine estimation, respectively. For global atmospheric light, an encoder-decoder network is used for estimation. At the top of the network, the images estimated by the three sub-networks are reconstructed according to the atmospheric scattering physics model to obtain the reconstructed haze image. A novel multi-scale Transformer block is introduced into the fine haze-free image sub-network of the dehazing network model. This block performs multi-scale token aggregation in its self-focused part, enabling it to capture features at different scales and effectively recover latent scene information in the haze image.

[0177] The above embodiments, which describe the specific features of the present invention, are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-essential improvements and adjustments made to the present invention by those skilled in the art based on the above description of the invention shall fall within the scope of protection of the present invention.

Claims

1. A two-stage decoupling image dehazing method based on zero-shot learning, characterized in that: Includes the following steps: (1) Obtain two test subsets from the real single-image dehazing dataset: the synthetic target test set and the mixed subjective test set. Preprocess the original dataset and use it as the test set. (2) Constructing a defogging network model: Based on the atmospheric scattering physical model I(x)=J(x)t(x)+A[1-t(x)], Where x represents the input haze image, I(x) represents the reconstructed hazy image, J(x) represents the original scene information, t(x) represents the transmittance, and A represents atmospheric scattered light; Based on the idea of ​​decoupling, the network model is divided into three layers: a potential haze-free image layer, a transport layer, and a global atmospheric light layer. The potential haze-free image and the transport layer are estimated in two stages. The first stage uses the embedded dark channel prior for coarse estimation, and the second stage uses the haze-free image refinement network and the transport layer refinement network for fine estimation. For global atmospheric light, an encoder-decoder network is used for estimation. At the top of the network, the images estimated by the three sub-networks of haze-free image refinement network, transport layer refinement network, and atmospheric light estimation network are reconstructed according to the atmospheric scattering physical model to obtain the reconstructed haze image. (3) Input the haze images in the preprocessed dataset into the constructed dehazing network model, calculate the loss according to the designed loss function, continuously iterate and update the parameters, and directly perform image dehazing on the haze images; In step (A2), multi-scale self-attention (MSSA) is computed, which includes two branches: (A21) In the branch, the query vector Q, key vector K, and value vector V are processed differently. For Q, the original scale is preserved, and a normal fully connected operation is performed on it. For K and V, in the same self-attention layer, scaling transformation (ST) is used to obtain K and V of different sizes, specifically as follows: Q=XW Q , K i =ST i (X)W i k ,V i =ST i (X)W i V , V i =V i +LE i (V i ), Where X represents the input feature sequence, W Q W i K W i V It is the linear projection parameter of the i-th scale layer onto the same head of the self-attention layer, LE i (·) represents the local augmentation component of depthwise convolution; while ST i (·) is the scaling operation for the i-th scale layer, which provides downsampling of X, and can be specifically expressed as: ST i (X)=LN(Conv i (X)), Where LN(·) refers to Layer Normalization; Conv i (·) refers to the convolution used in the i-th scale layer. Different scale layers use different kernel sizes and strides, resulting in different scale transformations. Therefore, in the same self-attention layer, keys and values ​​capture features at different scales. Then, the self-attention head can be calculated as: Where Softmax(·) is the activation function, T represents the transpose operation, and d is the dimension. The different heads are connected, and multi-head self-attention (MH) in the main branch is calculated using the following method: MH(Q,K i ,V i )=Concat(head0,﹣﹣﹣,head i )W O , Where Concat(·) is the concatenation operation, head i W represents the i-th self-attention head. O These are linear projection parameters; (A22) Another branch that preserves neighborhood information only performs linear projection and convolution operations on the input features sequentially. Multi-scale self-attention MSSA can be expressed as: MSSA=Concat(MH{Q,K i ,V i } i＝1,---,m )+Conv0(XW O ), Where Concat(·) is the join operation, MH represents multi-head self-attention, and Q, K i V i Let W represent the query vector, the key vector of the i-th scale layer, and the value vector of the i-th scale layer, respectively. Conv0(·) refers to the convolution operation, X is the input feature sequence, and W is the input feature sequence. O These are linear projection parameters.

2. The two-stage decoupling image dehazing method based on zero-shot learning according to claim 1, characterized in that: Step (2), the first stage of rough estimation, specifically includes the following steps: (2.1) Calculate the dark channel of the input image x: First, take the minimum value among the RGB three channels to construct a grayscale image, then invert the grayscale image and perform max pooling. Invert the result of max pooling to obtain I. dark The specific formula is as follows Where I c This refers to one of the R, G, and B channels of I; (2.2) Calculate a rough estimate T of the transmission graph DCP (x): I dark Substituting into the atmospheric scattering physics model, we obtain I dark (x)=J dark (x)t(x)+A(1-t(x)), where I dark (x) and J dark (x) represent the dark channels of images I and J, respectively; for atmospheric light A, in I dark The top 0.1% of pixels by brightness are selected, and their average value is taken as the atmospheric light A value; while for J... dark Based on the dark channel prior assumption, the dark channel of the non-sky region in an outdoor fog-free image J tends to zero. dark (x)→0, and thus according to Preliminary results of the transmission map were obtained; due to the uneven edge transitions in the dark channel image, guided filtering was used to improve the T... DCP The edges of (x) are smoother, and a guided filter is implemented using an average pooling operation with a kernel size of 19*19 and a stride of 1. (2.3) Calculate a rough estimate of J for the fog-free map DCP (x): Based on the atmospheric scattering physics model, and given atmospheric light A and transmission diagram T. DCP (x), J DCP (x): Obtained through equation calculation.

3. The two-stage decoupling image dehazing method based on zero-shot learning according to claim 2, characterized in that: Step (2) involves constructing three subnets of the dehazing network model, specifically including the following steps: (A) The architecture of the fog-free image thinning network is an improved 5-level U-Net architecture, in which the convolutional blocks are replaced by the introduced multi-scale Transformer blocks, and the SK fusion module is used to connect and fuse different layers. Soft reconstruction layers are used to obtain global residuals. (B) The transmission graph refinement network is a non-degenerate network consisting of five convolutional layers. In the first four layers, each layer only includes a convolutional layer, a batch normalization layer, and a LeakyReLU activation function. The last layer contains a convolutional layer and a sigmoid function that normalizes the output to [0,1]. (C) The atmospheric light estimation network is a symmetric encoder-decoder architecture network. The encoder and decoder each have four layers. Each layer of the encoder contains a convolutional layer, a ReLU activation function, and a max pooling layer. Each layer of the decoder contains an upsampling layer, a convolutional layer, a batch normalization layer, and a ReLU activation function. Between the encoder and decoder, a reparameterization trick module is used. This module transforms the encoder output into a latent Gaussian distribution, then resamples it, and then inputs it into the decoder.

4. The two-stage decoupling image dehazing method based on zero-shot learning according to claim 3, characterized in that: In step (A), the multi-scale Transformer block is constructed, which specifically includes the following steps: (A1) The RescaleNorm layer is used as the normalization layer. The specific normalization process of RescaleNorm is as follows: Where F(·) represents the main part of the multi-scale self-attention in the multi-scale Transformer block. These represent the mean and standard deviation, respectively. These are the learned scaling factor and the bias, respectively. and These are two linear layer weights and bias terms used to transform μ and σ, respectively. The transformation process is represented as {γ,β}={σW}. γ +B γ μW β +B β To accelerate convergence, B γ and B β Initialize to 1 and 0; (A2) Multi-scale self-attention is used to compute self-attention. Multi-scale self-attention (MSSA) has two branches, which aggregate multi-scale feature information within the window and preserve neighborhood information to a certain extent.

5. The two-stage decoupling image dehazing method based on zero-shot learning according to claim 4, characterized in that: In step (3), the loss function consists of five loss functions: (3.1) Reconstruction loss L Rec L is used to calculate the error between the input image and the reconstructed image, thereby constraining the entire network. Rec Defined as: L Rec =||I(x)-x|| F , Among them, ||·|| F Let Frobinius norm be the given matrix, x be the input haze image, and I(x) be the haze image reconstructed from the outputs of the three subnets. (3.2) Loss function L J Used to calculate the estimated J of a fog-free image. R The difference between the brightness and saturation of (x) is specifically expressed as: L J =||V(J R (x))-S(J R (x))|| F , Where V(·) represents brightness, S(·) represents saturation, and ||·|| F Denote the Frobinius norm of a given matrix; (3.3) Atmospheric light loss L H Used to calculate the estimated atmospheric light A(x) and A init The loss between (x), where A init (x) is the initial atmospheric light automatically estimated from the input image data, L H Specifically, it is expressed as follows: L H =||A(x)-A init (x)‖ F , (3.4)L KL It is the variational inference term loss of the reparameterization technique module in the atmospheric estimation optical network, which makes the resampled latent code in the network... Minimizing the difference between the latent code z before sampling and the sampled code is mathematically... Where KL(·) represents the Kullback-Leibler divergence between the two distributions, z i Let μ represent the i-th dimension of z. z ,σ z Let z represent the mean and standard deviation, respectively. They represent z respectively i The mean and standard deviation; (3.5) To avoid network overfitting, the regularization term L is set to reduce the loss. Reg Specifically, it is expressed as: Where N(·) represents the second-order neighborhood, |N(·)| represents the size of the second-order neighborhood, n represents the number of pixels in A(x), and A(x) represents atmospheric light; The total loss is defined by combining the five loss functions mentioned above: L=L Rec +L J +L H +L KL +λL Reg , Among them, L Reg The parameter λ is a non-negative parameter used to balance regularization, and in practice it is 0.1.

Images