Single image deep unfolding cycle dehazing method

Through the deep-expanded cyclic defog removal method, the adjustable network depth and multi-task framework are used, combined with the depth prediction task and the meta-channel self-attention layer, the past fog and under-fog removal problems of the defog removal method in the existing technology are solved, significantly improving the Image defog accuracy and quality.

CN115131238BActive Publication Date: 2024-11-05WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210735527.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-27
Publication Date
2024-11-05
Estimated Expiration
2042-06-27

AI Technical Summary

Technical Problem

Existing deep learning-based image defog removal methods are prone to the problems of past fog and lack of fog removal. The network's excessive dependence on data leads to poor generalization performance and cannot effectively deal with image dequality in complex scenarios.

Method used

Using the deep expansion loop defog removal method, the cyclic decoder and multi-task framework that can adjust the network depth, uses depth prediction tasks to assist in finding defog clues, combines the meta-channel self-attention layer and long and short-term memory units to iteratively decode to improve the image quality.

Benefits of technology

Effectively suppress past fog and under-fog removal phenomena in the fog removal results, improve image fog removal accuracy, improve fog-containing image quality, and provide better data support for subsequent visual tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115131238B_ABST
    Figure CN115131238B_ABST
Patent Text Reader

Abstract

The present invention provides a single image deep unfolding cycle defogging method. The multi-task framework uses the deep prediction task to assist in solving the defogging clues, and uses the cycle decoder to integrate the correlation between different features to iteratively solve the defogging image. The present invention applies the deep unfolding cycle defogging method to natural foggy images. The real image experiment shows that compared with the current image defogging network, the present invention is more effective in suppressing the past fog and under-defogging problems, and can greatly improve the image quality of foggy images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of images, and in particular relates to a technical solution for defogging using a cyclic decoder and a multi-task framework, which realizes image defogging by combining a depth prediction task and an image defogging task. Background Art

[0002] In recent years, with the promotion of policies such as smart transportation and safe cities, outdoor imaging equipment has been rapidly popularized across the country. According to a report by Comparitech, by 2022, the number of surveillance cameras deployed in China will reach 626 million. In addition, with the rapid popularization of face recognition and the rise of autonomous driving technology, digital image data has occupied an increasingly large proportion in people's lives, which has greatly promoted the development of outdoor computer vision technology. As the cost of imaging equipment continues to decrease and intelligent imaging equipment has been widely popularized, digital images have become the most common information carrier in today's society and are widely used in security monitoring, autonomous driving, road detection, and satellite detection. However, imaging equipment working outdoors is inevitably affected by the environment. For example, haze weather will reduce the clarity of the image, which is a fatal threat to computer vision systems that pursue high stability such as security and autonomous driving. Image defogging, as a technical solution to improve the quality of foggy images at the software level, has the excellent characteristics of fast, low cost, and good effect, and has become an indispensable preprocessing step in the field of image processing. Therefore, it is of great practical significance to develop image defogging technology and apply it to digital image defogging.

[0003] Generally speaking, haze is caused by the increase in particle size in the atmospheric medium, which leads to atmospheric absorption and scattering, weakening the radiation of scene points received by the camera. Images taken under such weather conditions become blurred due to the loss of detail information, color fidelity and contrast, which seriously hinders the efficiency and reliability of subsequent visual systems. The purpose of image dehazing is to fully exploit the redundant information in the image and restore a high-quality dehazed image from a low-quality hazy image.

[0004] In the past few decades, image defogging technology has made great progress. The image defogging methods proposed by early scholars are mainly based on hand-designed priors. Most of these priors are summarized based on some statistical observations of haze-free images. Using these priors and atmospheric scattering models, the corresponding haze-free image can be directly solved from a haze-containing image. In addition, with the continuous development of hardware computing power and large defogging datasets, more and more scholars are engaged in the research of using data-driven methods to solve the image defogging problem. On the one hand, the defogging method based on deep learning can directly learn defogging clues from the data, thereby reducing the dependence on hand-designed priors, and often can obtain performance far exceeding that of the defogging method based on priors, while also having lower time complexity. However, deep neural networks essentially fit parameters according to a pre-designed fixed model structure. The defogging strategy learned is the global optimal in the statistical sense of the dataset, but it cannot be guaranteed to be optimal for each independent sample. On the other hand, the degradation of fog on images is not only related to the fog itself, but also highly coupled with the scene content. The real scene is complex and changeable and cannot be exhaustively enumerated by a limited defogging dataset. The network's excessive reliance on data makes the generalization performance of the network unguaranteed. Therefore, although the deep learning architecture can extract more robust and effective features, it is still inevitably affected by past fog and under-defogging to a certain extent. Therefore, considering the loop structure that can dynamically adjust the network depth, it is very meaningful to propose an image defogging method (DURN) based on a multi-task framework. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the existing image defogging methods based on deep learning are prone to over-fogging and under-defogging. A deep unfolding cyclic defogging method is provided, which expands the dynamic expression ability of the network through a cyclic decoder with adjustable network depth, and uses depth prediction tasks to assist in finding defogging clues, thereby improving the quality of foggy images.

[0006] The technical method of the present invention provides a single image depth expansion cycle defogging method, comprising the following steps:

[0007] Step 1, extracting features of an input foggy image by using an encoder to obtain feature encoding based on the input foggy image;

[0008] Step 2, decoding the feature encoding based on the foggy image obtained in step 1 using a loop decoder to obtain a defogging image and a depth prediction image, specifically including the following sub-steps:

[0009] Step 2.1, channel concatenating the feature code based on the input foggy image output from the encoder in step 1 and the feature code output from the t-1th cyclic decoder;

[0010] Step 2.2: For the feature encoding obtained in step 2.1, the meta-channel self-attention layer is used to perform feature weighting, thereby obtaining weighted features adapted to the depth prediction task and the image defogging task respectively;

[0011] Step 2.3, the weighted features obtained in step 2.2 are aggregated using long short-term memory units to aggregate historical information, and the dehazed image and depth prediction image are output:

[0012] If t ≥ T meets the defogging requirement, the defogging image is output; if it does not meet the requirement, t = t + 1 and repeat step 2. Further, the processing process of the encoder in step 1 is:

[0013] F 0 =E(I|θ 1 ) (1)

[0014] Where I is the input foggy image, E is the encoder, and θ 1 are the parameters of the encoder, Encode the obtained features.

[0015] Furthermore, the processing of the loop decoder in step 2 is:

[0016]

[0017] In the formula, G t is the input feature of the recurrent decoder, is the dehazed image decoded for the tth time, is the depth prediction image decoded for the tth time, R is the loop decoder, θ 2 are the parameters of the loop decoder, is the processed feature of the t-th decoding, which can be decoded multiple times by the cyclic decoder with the same parameters, thereby achieving an explicitly adjustable network depth.

[0018] Furthermore, the channel splicing in step 2.1 is implemented as follows:

[0019] G t =[F 0 ,F t-1 ] (3)

[0020] In the formula, F 0 is the feature encoding based on the input foggy image output by encoder E in step 1, and F t-1 is the feature of the decoder’s t-1th output, The features after splicing.

[0021] Furthermore, the specific implementation of step 2.2 is as follows:

[0022]

[0023]

[0024] Among them, MCA is the meta-channel self-attention layer, is the feature distributed to the dehazing branch, is the feature distributed to the depth prediction branch, θ depth and θ dehaze are the parameters of the MCA of the two branches respectively.

[0025] Furthermore, the meta-channel self-attention layer MCA processes the features as follows:

[0026] Step A. For the input feature G, use the convolution layer to perform convolution to achieve linear transformation of the input feature, and use the batch normalization layer to normalize it, and use the linear correction unit with parameters to perform nonlinear transformation, so as to map the input feature G to a feature of the same size:

[0027]

[0028] Among them, Conv is a convolution layer with a convolution kernel size of 3×3. is the batch normalization layer, is a linear correction unit with parameters, is the feature of the same size as G;

[0029] Step B. Multiply W obtained in step A by the input feature G element by element:

[0030]

[0031] in, Represents the element-by-element multiplication operation, F Global Represents the characteristics of the output;

[0032] Step C. Perform global average pooling on the features obtained in step B:

[0033]

[0034] Where H and W are input features F Global The height and width in the spatial dimension, i and j represent the horizontal and vertical coordinates of the spatial position respectively;

[0035] Step D. Use the fully connected layer with shared weights to process g obtained in step C, interact with global information, and use the Sigmoid function to normalize and obtain the cross-channel feature importance weights:

[0036]

[0037] Where W SFC is the parameter matrix of the fully connected layer, e is the Euler number, and q is the obtained feature importance weight;

[0038] Step E: Multiply the feature weight q obtained in step D by the input feature G to obtain the weighted feature.

[0039] Furthermore, step 2.3 is implemented as follows:

[0040]

[0041]

[0042] In the formula, and They represent the cell states of the t-th cycle of the LSTM of the defogging branch and the depth prediction branch, respectively. and denotes the hidden state of the t-th cycle of the defogging branch and the depth prediction branch LSTM, θ dehaze represents the parameters of the LSTM of the defogging branch, θ depth represents the parameters of the LSTM of the depth prediction branch, is the dehazed image output by the tth cycle, It is the depth prediction image output by the tth cycle.

[0043] Furthermore, the value of T is 3.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The deep unfolding cycle defogging method proposed in the present invention uses a multi-task framework to use the deep prediction task to assist in solving the defogging clues based on the deep learning image defogging method, and uses the cycle decoder to integrate the correlation between different features to iteratively solve the defogging image. Starting from the common problems of past fog and under-defogging in current defogging methods, this method realizes dynamic network reasoning through an iterative decoder that can dynamically adjust the network depth, and establishes a deep unfolding cycle defogging network. Compared with the current image defogging network, this network is more effective in suppressing the problems of past fog and under-defogging.

[0046] The deep unfolding cycle defogging method proposed in this invention has a stronger expression ability than the current image defogging method based on deep learning, overcomes the problem that a single network is difficult to learn multiple mapping functions, can effectively suppress the past fog and under-defogging phenomenon in the defogging results, improves the defogging accuracy, greatly improves the impact quality of foggy images, and provides effective data for subsequent visual task applications. Therefore, the deep unfolding cycle defogging method not only has very important academic value, but also has important practical significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of an embodiment of the present invention;

[0048] Figure 2 It is the algorithm flow chart of step 2 of the embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clear, the following is a further detailed description of the deep expansion cycle defogging method according to an embodiment of the present invention in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] The purpose of the present invention is to solve the problem that the existing image defogging methods based on deep learning are prone to over-fogging and under-defogging. The dynamic expression ability of the network is expanded through a recurrent decoder with adjustable network depth, and the depth prediction task is used to assist in finding defogging clues, so as to improve the quality of foggy images.

[0051] like Figure 1 The implementation of a single image depth expansion cycle defogging method provided in the embodiment includes the following processing steps:

[0052] Step 1: Extract features of the input foggy image using an encoder to obtain feature encoding based on the input foggy image:

[0053] F 0 =E(I|θ 1 ) (1)

[0054] Where I is the input foggy image, E is the encoder, and θ 1 are the parameters of the encoder, Encode the obtained features.

[0055] In the embodiment, UNET++ is used as the encoder of the network.

[0056] Step 2: The feature encoding based on the foggy image obtained in step 1 is decoded using a loop decoder to obtain a defogging image and a depth prediction image:

[0057]

[0058] In the formula, G t is the input feature of the recurrent decoder, is the dehazed image decoded for the tth time, is the depth prediction image decoded for the tth time, R is the loop decoder, θ 2 are the parameters of the loop decoder, is the processed feature of the t-th decoding, which can be decoded multiple times by the cyclic decoder with the same parameters, thereby achieving an explicitly adjustable network depth.

[0059] The size of the foggy image in the embodiment is 256×256×3, and the number of channels in the defogging network is C=32.

[0060] Step 2.1, channel concatenate the feature code based on the input foggy image output by encoder E in step 1 and the feature code output by the t-1th (t=1, 2, 3, 4...) cyclic decoder:

[0061] G t =[F 0 ,F t-1 ] (3)

[0062] In the formula, F 0 is the feature encoding based on the input foggy image output by encoder E in step 1, and F t-1 is the feature of the decoder's t-1th (t=1, 2, 3, 4...) output, The features after splicing.

[0063] Step 2.2: For the feature code G obtained in step 2.1 t , the meta-channel self-attention layer is used to weight the features, so as to obtain weighted features adapted to the depth prediction task and image defogging task respectively:

[0064]

[0065]

[0066] Among them, MCA is the meta-channel self-attention layer, is the feature distributed to the dehazing branch, is the feature distributed to the depth prediction branch, θ depth and θ dehaze are the parameters of the MCA of the two branches respectively. The specific processing of MCA on features is:

[0067] Step A. For the input feature G, use the convolution layer to perform convolution to achieve linear transformation of the input feature, and use the batch normalization layer to normalize it, and use the linear correction unit with parameters to perform nonlinear transformation, so as to map the input feature G to a feature of the same size:

[0068]

[0069] Among them, Conv is a convolution layer with a convolution kernel size of 3×3. is the batch normalization layer, is a linear correction unit with parameters, is the feature with the same size as G.

[0070] Step B. Multiply W obtained in step A by the input feature G element by element:

[0071]

[0072] in, Represents the element-by-element multiplication operation, F Global Represents the characteristics of the output.

[0073] Step C. Perform global average pooling on the features obtained in step B:

[0074]

[0075] Where H and W are input features F Global The height and width in space. i and j represent the horizontal and vertical coordinates of the space position respectively.

[0076] Step D. Use the fully connected layer with shared weights to process g obtained in step C, interact with global information, and use the Sigmoid function to normalize and obtain the cross-channel feature importance weights:

[0077]

[0078] Where W SFC is the parameter matrix of the fully connected layer, e is the Euler number, and q is the obtained feature importance weight.

[0079] Step E: Multiply the feature weight q obtained in step D by the input feature G to obtain the weighted feature.

[0080] Step 2.3, use long short-term memory units to aggregate historical information for the weighted features obtained in step 2.2:

[0081]

[0082]

[0083] In the formula, and They represent the cell states of the t-th cycle of the LSTM of the defogging branch and the depth prediction branch, respectively. and denotes the hidden state of the t-th cycle of the defogging branch and the depth prediction branch LSTM, θ dehaze represents the parameters of the LSTM of the defogging branch, θ depth represents the parameters of the LSTM of the depth prediction branch, is the dehazed image output by the tth cycle, It is the depth prediction image output by the tth cycle.

[0084] If t≥T satisfies the defogging requirement, the defogging image is output; if it does not meet the requirement, t=t+1 and step 2 is repeated.

[0085] In the embodiment, the termination parameter T of the iteration is set to 3.

[0086] The embodiments of the present invention use natural images of size 256×256×3, but are not limited to images of this size or natural images. For other fogged images, no matter what sensor the image is collected by or what size it is, it has wide versatility and is less restricted by objective factors. The experimental results of real foggy images show that this method can better suppress the occurrence of past fog and under-defogging compared with other methods, and can effectively improve the quality of the image.

[0087] It should be noted and understood that various modifications and improvements can be made to the present invention described in detail above without departing from the spirit and scope of the present invention required by the appended claims. Therefore, the scope of the technical solution claimed is not limited by any specific exemplary teaching given.

Claims

1. A single image deep expansion cycle defogging method, characterized in that: The following steps are involved: Step 1, extracting features of an input foggy image by using an encoder to obtain feature encoding based on the input foggy image; Step 2, decoding the feature encoding based on the foggy image obtained in step 1 using a loop decoder to obtain a defogging image and a depth prediction image, specifically including the following sub-steps: Step 2.1, channel concatenating the feature code based on the input foggy image output from the encoder in step 1 and the feature code output from the t-1th cyclic decoder; The processing of the loop decoder is: In the formula, G t is the input feature of the recurrent decoder, is the dehazed image decoded for the tth time, is the depth prediction image decoded for the tth time, R is the loop decoder, θ2 is the parameter of the loop decoder, is the processed feature after the t-th decoding, which can be decoded multiple times by the cyclic decoder with the same parameters, thereby achieving an explicitly adjustable network depth; The implementation method of channel splicing is: G t =[F0,F t-1 ] (2) Where F0 is the feature encoding based on the input foggy image output by encoder E in step 1, and F t-1 is the feature of the decoder’s t-1th output, is the feature after splicing; Step 2.2: For the feature encoding obtained in step 2.1, the meta-channel self-attention layer is used to perform feature weighting, thereby obtaining weighted features adapted to the depth prediction task and the image defogging task respectively; The specific implementation of step 2.2 is as follows: Among them, MCA is the meta-channel self-attention layer, is the feature distributed to the dehazing branch, is the feature distributed to the depth prediction branch, θ depth and θ dehaze are the parameters of the MCA of the two branches respectively; Step 2.3, the weighted features obtained in step 2.2 are aggregated using long short-term memory units to aggregate historical information, and the dehazed image and depth prediction image are output: Step 2.3 is implemented as follows: In the formula, and They represent the cell states of the t-th cycle of the LSTM of the defogging branch and the depth prediction branch, respectively. and denotes the hidden state of the t-th cycle of the defogging branch and the depth prediction branch LSTM, θ dehaze represents the parameters of the LSTM of the defogging branch, θ depth represents the parameters of the LSTM of the depth prediction branch, is the dehazed image output by the tth cycle, is the depth prediction image output by the tth cycle; If t≥T satisfies the defogging requirement, the defogging image is output; if it does not meet the requirement, t=t+1 and step 2 is repeated.

2. The single image deep expansion cycle defogging method according to claim 1, characterized in that: The processing of the encoder in step 1 is: F0=E(I|θ1) (7) Where I is the input foggy image, E is the encoder, and θ1 is the encoder parameter. Encode the obtained features.

3. The single image deep expansion cycle defogging method according to claim 1, characterized in that: The meta-channel self-attention layer MCA processes the features as follows: Step A. For the input feature G, use the convolution layer to perform convolution to achieve linear transformation of the input feature, and use the batch normalization layer to normalize it, and use the linear correction unit with parameters to perform nonlinear transformation, so as to map the input feature G to a feature of the same size: Among them, Conv is a convolution layer with a convolution kernel size of 3×3. is the batch normalization layer, is a linear correction unit with parameters, is the feature of the same size as G; Step B. Multiply W obtained in step A by the input feature G element by element: in, Represents the element-by-element multiplication operation, F Global Represents the characteristics of the output; Step C. Perform global average pooling on the features obtained in step B: Where H and W are input features F Global The height and width in the spatial dimension, i and j represent the horizontal and vertical coordinates of the spatial position respectively; Step D. Use the fully connected layer with shared weights to process g obtained in step C, interact with global information, and use the Sigmoid function to normalize and obtain the cross-channel feature importance weights: Where W SFC is the parameter matrix of the fully connected layer, e is the Euler number, and q is the obtained feature importance weight; Step E: Multiply the feature weight q obtained in step D by the input feature G to obtain the weighted feature.

4. The single image deep expansion cycle defogging method according to claim 1, characterized in that: The value of T is 3.