A real-time image defogging method based on a lightweight model

Abstract

The application discloses a real-time image defogging method based on a lightweight model, which takes a foggy image as input, processes through multiple modules in the model, and finally provides a defogging reference with high visual quality for downstream high-level visual tasks. The application uses convolution operation with increasing convolution kernel size to extract multi-level features of the image, uses a gating fusion module for feature interaction, and uses an attention mechanism for feature weighting, which has strong interpretability. And the improved atmospheric scattering model is used to jointly estimate the atmospheric light and the transmission map. Under the weather condition of fog, it can provide a defogging reference for the input image in real time and maintain good visual quality. The overall process of the method is relatively simple, the required calculation amount in actual application is small, the stability and generalization performance are relatively strong, and the method can be applied to automatic driving, video monitoring, remote sensing images and various scenes.

Description

Technical Field

[0001] This invention relates to the field of image dehazing technology, and in particular to a real-time image dehazing method based on a lightweight model. Background Technology

[0002] Improving accuracy has been achieved in various computer vision tasks, including object detection, segmentation, repair, and compression. However, in foggy weather, the performance of these algorithms can be limited if the input image contains noise, distortion, or corruption. This is because small particles such as water vapor, sand, dust, and smoke suspended in the air cause attenuation and scattering of light absorbed by the camera, thus reducing image quality. Therefore, image dehazing is an important low-level vision task. As a preprocessing step for downstream high-level vision tasks, it can improve the accuracy of algorithms and reduce application risks.

[0003] Currently, internationally, mainstream image dehazing methods fall into two main categories: the first category is based on prior knowledge. The earliest dehazing method used was based on dark channel priors (He, 2010). This method assumes that all image pixels affected by fog have an intensity approaching zero in at least one color channel, which is the dark channel. However, this prior condition may not be met in sky pixels, as the area is brighter and it's impossible to determine whether it's affected by fog. Furthermore, dark channel prior methods are computationally expensive. Additionally, methods based on semi-physical guided filtering recover image texture information by roughly estimating the fog thickness map. Generally, these methods are constrained by empirical assumptions or statistical laws, which limits their application scenarios to some extent.

[0004] The second category of methods is deep learning methods based on neural networks. Many deep learning-based methods employ symmetric encoder and decoder structures, with U-Net being the most common. The encoder extracts multi-level or multi-scale features from the image, while the decoder performs high-resolution image restoration, such as Wavelet-U-Net (Yang, 2019) and MSBDN (Dong, 2020). Secondly, multi-path networks have also been widely used. They typically extract different levels of information about the same feature across multiple parallel network branches, such as different color spaces, for example, Review-Net (Mehra, 2020) and U-Color (Li, 2021). Therefore, using residual and attention mechanisms to integrate and weight features is an essential step in most deep learning-based methods. However, because many methods employ a module stacking strategy, they cannot effectively balance accuracy and real-time performance, and are also prone to overfitting on the training set.

[0005] In summary, the above analysis shows that current methods based on prior assumptions and statistical analysis have low accuracy and limited applicability. Most deep learning-based image dehazing methods still have significant shortcomings in terms of complexity and efficiency. In foggy weather conditions, the processing time required for preprocessing downstream tasks such as object detection and segmentation is too long. Although they can mitigate the impact of fog on image quality to some extent, the system cannot respond to the input image in real time after adding the dehazing preprocessing step.

[0006] Therefore, most current image dehazing methods cannot meet the needs of practical applications and deployments. The main problem to be solved is how to effectively extract features from hazy images and use them to recover hazy-free images, while maximizing the system's response speed. Summary of the Invention

[0007] The purpose of this invention is to provide a real-time image dehazing method based on a lightweight model, which solves the problem that most image dehazing methods cannot meet the needs of practical applications and deployments.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] This invention provides a real-time image dehazing method based on a lightweight model, comprising the following steps:

[0010] Step A: Obtain the original foggy image to be processed, and use a sequence of convolution operations with increasing kernel size to obtain multi-level features of the original foggy image; wherein the convolution operation with smaller kernel size is used to extract low-level features of the image, and the convolution operation with larger kernel size is used to extract high-level features of the image.

[0011] Step B: Merge the multi-level features from Step A through the first connection layer;

[0012] Step C: Use the gated fusion module to perform feature interaction on the multi-level features merged in Step B, so as to complement and fuse the features of different levels;

[0013] Step D: Connect and merge the high-level features from step A with the output of step C again through the second connection layer;

[0014] Step E: Employ an attention mechanism to weigh the impact of the merged features from Step D on the output dehazing reference.

[0015] Step F: High-resolution reconstruction. The weighted features from step E are subjected to a series of convolution operations again to obtain the K values ​​of the three RGB channels of the image.

[0016] Step G: Substitute the original image into the improved atmospheric scattering model that includes the K value calculated in step F to obtain a dehazed reference image corresponding to the original hazy image.

[0017] Further, the convolution operation sequence in step A includes: convolutional layer 1, convolutional layer 2 and convolutional layer 3, with convolutional kernel sizes of 3×3, 5×5 and 7×7 respectively; after three consecutive convolutional layers, pixel-level residual connections are made between convolutional layer 1 and convolutional layer 3.

[0018] Furthermore, the first connection layer in step B is the residual summation of convolutional layer 1 and convolutional layer 3, and the output of convolutional layer 1 and convolutional layer 2.

[0019] Further, step C includes:

[0020] The three layers of information from the first connection layer are used as input to the gated fusion module, and the corresponding feature map sets are denoted as F1, F2, and F3, respectively. The j-th feature map of the i-th layer is denoted as F. i,j ;

[0021] All feature maps from the three sets are passed through convolutional layer 4 to obtain three feature maps, denoted as C1, C2, and C3 respectively;

[0022] Multiply the three feature maps by F1, F2 and F3 at the pixel level, respectively, and then add the feature maps at the corresponding positions of each layer at the pixel level to output the i-th layer feature map.

[0023] Further, step E includes:

[0024] An attention mechanism consisting of a channel attention module and a pixel attention module is adopted;

[0025] In this process, the channel attention module uses global average pooling on the merged feature maps in step D, and the evaluation pooling value M of the c-th feature map is... c The value of the element in the s-th row and t-th column of this feature map is denoted as Then M c Expressed by the following formula:

[0026]

[0027] Where H and W are the height and width of the image, respectively;

[0028] Then, two consecutive convolutional layers and activation layers are used for transformation to obtain a one-dimensional weight vector. The c-th element of this vector is multiplied by the c-th feature map, as expressed by the following formula:

[0029]

[0030] In the formula, and These are two convolutional layers, with δ(·) and σ(·) being the ReLU and Sigmoid activation functions, respectively. A one-dimensional weight vector at the channel level. For pixel-level multiplication, the one-dimensional weight vector is multiplied by each channel separately, resulting in a set of feature maps.

[0031] The pixel attention module outputs the feature map of channel attention. Perform a transformation at the pixel scale, and the output value is... The derivation formula is as follows:

[0032]

[0033] in, and These are two convolution operations, and σ(·) is the Sigmoid activation function.

[0034] Furthermore, the high-resolution reconstruction in step F employs the L2 loss function.

[0035] Furthermore, the formula for the improved atmospheric scattering model in step G is as follows:

[0036] J(θ)=K(θ)×I(θ)-K(θ)+b

[0037] In the formula, K(θ) represents the combined variable K after combining global atmospheric light A and transmission map t, and is expressed by the following formula:

[0038]

[0039] In the formula, A is a constant representing global atmospheric light; t is the transmission map of each element in the range (0,1); θ is the pixel coordinate of the H×W image; I and J are the input image with fog and the reference output image without fog, respectively; b is a default constant, which is set to 1 in the experiment.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) This method uses an improved atmospheric scattering model to jointly estimate atmospheric light and transmission maps, which can ensure the authenticity of the restored image and improve the real-time dehazing efficiency. In addition, this method mainly consists of convolutional layers and connection layers, with a simple architecture; it does not require excessive resources from the carrier.

[0042] (2) This method uses a gated fusion module and an attention mechanism to interact with features, which can effectively extract, fuse and weight multi-level features, avoid complex structural design and module stacking, and greatly reduce the number of parameters; further improve the real-time dehazing efficiency of images;

[0043] (3) This method is less prone to overfitting on the training set due to its low parameter count, and can better cope with various real-world scenarios. It can overcome common problems such as artificial artifacts and uneven color. It can improve the performance of target detection in scenarios such as driving in fog, video surveillance, and remote sensing. Attached Figure Description

[0044] Figure 1 A flowchart illustrating the real-time image dehazing method based on a lightweight model provided in this embodiment of the invention;

[0045] Figure 2 This is a schematic diagram of the attention mechanism provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0047] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," and "the other end," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0048] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installed," "equipped with," "connected," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0049] Reference Figure 1As shown, this invention provides a real-time image dehazing method based on a lightweight model. First, a sequence of consecutive convolutional layers with increasing kernel size is used to extract multi-level features of the image. Second, the extracted features are concatenated, and a gating fusion module is used for feature interaction. Then, the extracted high-level features are concatenated with the output of the gating fusion module, and an attention mechanism is used for feature weighting. Finally, high-resolution image restoration is achieved through convolution, and the reconstructed image is obtained by substituting it into an improved atmospheric scattering model.

[0050] Figure 1 This is a schematic diagram of the technical process of the present invention, which mainly includes seven steps:

[0051] Step A: Obtain the original fogged image to be processed, such as fogged images of autonomous driving scenes, surveillance videos, or remote sensing images, which need to be defogging processed.

[0052] By employing a convolutional sequence with increasing kernel size, multi-level features of the original foggy image are obtained, comprising convolutional layer 1, convolutional layer 2, and convolutional layer 3. Smaller convolutional kernels are used to extract local features, while larger kernels are used to extract global features. A convolutional layer sequence with kernel sizes increasing from 3×3, 5×5 to 7×7 is used, corresponding to convolutional layer 1, convolutional layer 2, and convolutional layer 3, respectively. After three consecutive convolutional layers, pixel-level residual connections are established between convolutional layer 1 and convolutional layer 3 to refine the semantic information of high-level features.

[0053] Table 1. Details of real-time image dehazing methods based on lightweight models.

[0054]

[0055]

[0056] Step B involves merging the multi-level features extracted in Step A using Connector Layer 1. The role of Connector Layer 1 is to merge low-level and high-level features of the image along the feature map dimension, compensating for information loss in the initial layers as the network deepens. For example... Figure 1 As shown in Table 1, the three layers of the connection layer 1 are the residual summation of convolutional layer 1 and convolutional layer 3, convolutional layer 1, and convolutional layer 2.

[0057] Step C involves using a gated fusion module to interact with the multi-level features merged in Step B, enabling the features from different levels to complement and fuse. This invention employs an effective feature-interaction-based strategy to replace densely connected residual blocks, achieving second-order interactions between multi-level features and effectively reducing computational load.

[0058] The three layers of information combined by the connection layer 1 (convolutional layer 1, convolutional layer 2, and convolutional layer 1 + convolutional layer 3) are used as the input to the gated fusion module. All feature maps are denoted as F, and the feature map sets corresponding to these three layers are denoted as F1, F2, and F3, respectively. The j-th feature map of the i-th layer is denoted as Fi. i,j First, all feature maps are passed through a convolutional layer 4, denoted by C(·), resulting in three feature maps: C1(F), C2(F), and C3(F). These three feature maps are then multiplied pixel-wise by F1, F2, and F3, respectively. Finally, the feature maps at corresponding positions in each of the three multiplied layers (k = 1, 2, 3) are summed pixel-wise, resulting in the i-th layer G in the output feature map. i It can be expressed by the following formula:

[0059]

[0060] in, The expression represents pixel-level multiplication, and summation represents pixel-level addition. As shown in the equation above, the input to the gated fusion module consists of three layers, each with the same number of feature maps, while the total number of output feature maps is reduced to one-third of the input, equal to the number of feature maps in each layer of the input. The gated fusion module enhances features in feature maps with adjacent pixels and introduces interactions by dynamically assigning weights to other feature maps through pixel-level multiplication. This strengthens the ability of convolution to preserve and utilize multi-level features in a dense and expanded manner.

[0061] Step D involves concatenating and merging the high-level features extracted in Step A with the output of Step C. The two layers connected by connection layer 2 are convolutional layer 3 and the output of the gated fusion module, respectively. Since the global high-level features from the feature extraction stage are redistributed to lower levels through feature interaction in the gated fusion module, and this high-level information is essential for image restoration, it is merged with the output of the gated fusion module into connection layer 2.

[0062] Step E, the attention mechanism, weighs the impact of the features merged in step D on the output dehazing reference. In cases of uneven fog distribution, the attention mechanism can evaluate the region and thickness of the fog distribution, thereby reducing complex shadows or dark spots in the restored image and further improving the image's visual quality.

[0063] Attention mechanisms can give different parts of information different levels of attention when processing large amounts of information. The attention mechanism used in this invention consists of channel attention and pixel attention. The channel attention mechanism selects important features related to the foggy region, while the pixel attention mechanism refines the selected foggy region. Its structure is as follows: Figure 2 As shown.

[0064] All convolution operations used in the attention mechanism have a kernel size of 1×1.

[0065] Channel attention first applies global average pooling to each feature map, then the average pooling value M of the c-th feature map is... c The value of the element in the s-th row and t-th column of this feature map is denoted as Then M c It can be expressed by the following formula:

[0066]

[0067] Where H and W represent the height and width of the image, respectively.

[0068] Then, a one-dimensional weight vector is obtained by transforming the vector using two consecutive convolutional layers and activation layers. Multiplying the c-th element of this vector by the c-th feature map can be expressed by the following formula:

[0069]

[0070] In the formula and These are two convolutional layers, with δ(·) and σ(·) being the ReLU and Sigmoid activation functions, respectively. A one-dimensional weight vector at the channel level. For pixel-level multiplication, the one-dimensional weight vector is multiplied by each channel separately, resulting in a set of feature maps.

[0071] Similarly, pixel attention will output the feature map of channel attention. The transformation is performed at the pixel scale, and the output value is... The derivation formula is as follows:

[0072]

[0073] in, and These are two convolution operations, and σ(·) is the Sigmoid activation function.

[0074] Step F, High-Resolution Reconstruction. Subsequently, all features are input into the high-resolution stage, which consists of two convolutional layers, namely convolutional layer 5 and convolutional layer 6.

[0075] While loss functions such as L1 loss, L2 loss, SSIM, or perceptual loss have been shown to achieve good performance, experiments demonstrate that the most widely used L2 loss (also known as mean squared error) is the most suitable loss function for this model. The definition of L2 loss is as follows:

[0076]

[0077] In the formula, I is the input foggy image, and J is the output fog-free image; H and W are the height and width of the image, respectively; s represents the row element value of the feature map, and t represents the column element value of the feature map. Since the network indirectly estimates the parameter K instead of directly generating the fog-free image, using loss functions such as perceptual loss is not interpretable. Furthermore, this method has fewer parameters and is less prone to overfitting; therefore, a regularization term is unnecessary. Experimental results also confirm that introducing L1 loss has a negative effect.

[0078] Step G involves substituting the original image into the improved atmospheric scattering model to calculate the dehazed image. The traditional ASM model can be reconstructed to jointly estimate global atmospheric light A and transmittance t, thereby reducing the number of parameters.

[0079] I(θ)=J(θ)×t(θ)+A(1-t(θ))

[0080] In the formula, A is a constant, t is the pixel transmittance of light in the range (0,1), θ represents the pixel coordinates of the H×W image, and I and J are the foggy input and fog-free output, respectively. Therefore, the expected value in this embodiment, which is the fog-free approximation, can be written as the following formula:

[0081]

[0082] To contribute to the efficiency of real-time vision tasks, this embodiment also employs this model. To integrate these two factors into a single variable, the revised formula for the ASM model is as follows:

[0083] J(θ)=K(θ)×I(θ)-K(θ)+b

[0084] In the formula, K(θ) represents the combined variable K after combining global atmospheric light A and transmission map t, and can be expressed by the following formula:

[0085]

[0086] In the formula, A is a constant representing global atmospheric light; t is the transmission map of each element in the range (0,1); θ is the pixel coordinate of the H×W image; I and J are the input image with fog and the reference output image without fog, respectively; b is a default constant, which is set to 1 in the experiment.

[0087] Specifically, K is an intermediate parameter of the network. This method transforms the input image to a size of 640×480, generating a separate K value for each RGB channel of the input. At the end of the network, substituting a K value of size 3×640×480 into the above formula, with b set to the default value of 1, yields a haze-free reference image.

[0088] For computer vision tasks such as autonomous driving, remote sensing imagery, and video surveillance, where computing and storage resources are limited in hardware deployment, this invention serves as a preprocessing step for high-level vision tasks such as target detection, segmentation, and image classification. It can effectively improve the visual quality of images in real time and ensure the implementation effect of subsequent tasks.

[0089] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A real-time image dehazing method based on a lightweight model, characterized in that, Includes the following steps: Step A: Obtain the original foggy image to be processed, and use a sequence of convolution operations with increasing kernel size to obtain multi-level features of the original foggy image; Convolution operations with smaller kernel sizes are used to extract low-level features of an image, while convolution operations with larger kernel sizes are used to extract high-level features of an image. Step B: Merge the multi-level features from Step A through the first connection layer to obtain three layers of information, namely: the sum of the residuals of convolutional layer 1 and convolutional layer 3, the output of convolutional layer 1, and the output of convolutional layer 2; Step C: Utilize the gated fusion module to perform feature interaction on the multi-level features merged in Step B, complementing and fusing features from different levels; specifically, this includes: using the three layers of information from the first connection layer as input to the gated fusion module, with the corresponding feature map sets denoted as follows: , and The j-th feature map of the i-th layer is denoted as The feature maps of the three sets are passed through convolutional layer 4 to obtain three feature maps, denoted as follows: , and The three feature maps are respectively compared with... , and Pixel-level multiplication, and then pixel-level addition of the feature maps at corresponding positions in each layer, outputs the i-th layer feature map; Step D: The high-level features from step A and the output from step C are connected and merged again through the second connection layer to obtain the merged feature map; Step E: Employ an attention mechanism to weight the impact of the merged features from Step D on the output dehazing reference; Step F: High-resolution reconstruction. The weighted features from step E are subjected to a series of convolution operations again to obtain the K values ​​of the three RGB channels of the image. Specifically, high-resolution reconstruction is performed through convolutional layers 5 and 6, which directly output a three-channel parameter K value with the same size as the original image. Step G: Substitute the original image into the improved atmospheric scattering model that includes the K value calculated in step F to obtain a dehazed reference image corresponding to the original hazy image; the formula for the improved atmospheric scattering model in step G is as follows: In the formula, Let K represent the combined variable after combining global atmospheric light A and transmission map t, i.e., K obtained in step F; θ is the pixel coordinate of the H×W image; I and J are the input image with fog and the reference output image without fog, respectively; b is a default constant, set to 1 in the experiment.

2. The real-time image dehazing method based on a lightweight model according to claim 1, characterized in that, The convolution operation sequence in step A includes: convolutional layer 1, convolutional layer 2, and convolutional layer 3, with kernel sizes of 3×3, 5×5, and 7×7, respectively; after three consecutive convolutional layers, pixel-level residual connections are made between convolutional layer 1 and convolutional layer 3.

3. The real-time image dehazing method based on a lightweight model according to claim 1, characterized in that, Step E includes: An attention mechanism consisting of a channel attention module and a pixel attention module is adopted; In this process, the channel attention module uses global average pooling on the merged feature maps in step D, and the evaluation pooling value of the c-th feature map is... The value of the element in the s-th row and t-th column of this feature map is denoted as... ,but Expressed by the following formula: Where H and W are the height and width of the image, respectively; Then, two consecutive convolutional layers and activation layers are used to transform the vector, resulting in a one-dimensional weight vector. The c-th element of this vector is multiplied by the c-th feature map, as expressed by the following formula: In the formula, and There are two convolutional layers, and These are the ReLU and Sigmoid activation functions, respectively. A one-dimensional weight vector at the channel level. For pixel-level multiplication, the one-dimensional weight vector is multiplied by each channel separately, resulting in a set of feature maps. ; The pixel attention module outputs the feature map of channel attention. Perform a transformation at the pixel scale, and the output value is... The derivation formula is as follows: in, and These are two convolution operations. This is the Sigmoid activation function.

4. The real-time image dehazing method based on a lightweight model according to claim 1, characterized in that, In step F, the high-resolution reconstruction uses the L2 loss function.

Images