Image dehazing methods
Through the training of the adversarial model, the image defog removal is optimized using generators and discriminators, and the problems of distortion and insufficient visibility of defog removal images in the prior art are solved, achieving efficient and real defog removal effect.
Patent Information
- Application Number
- CN202111196138.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-14
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-10-14
AI Technical Summary
Existing image defog removal methods cannot maintain the authenticity and visibility of the image while efficiently removing fog, especially in dynamic scenes, which are difficult to obtain effective defog removal effects.
Using an adversarial model, including generators and discriminators, by training foggy images, defogging images and fogging images of the same scene, using encoder, decoder and feature repair modules, repeated training until the generator and discriminators reach Nash equilibrium State, output optimized defogging image.
The effect of maintaining image authenticity and visibility while efficiently removing fog is achieved, and the problem of distortion of images in the prior art is solved.
Smart Images

Figure CN113947542B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an image defogging method and belongs to the field of deep learning. Background Art
[0002] Under weather conditions such as fog and haze, dust, smoke or other floating particles will greatly absorb and scatter light, causing serious degradation of the images acquired by the visual system, which in turn affects the detection, tracking and recognition of the target. This not only makes it difficult to obtain target information from the image, but also affects subsequent image processing.
[0003] At present, the main defogging methods include image defogging based on image restoration, image defogging based on image enhancement, and image defogging based on image fusion. Image defogging based on image restoration obtains multiple blurred images in the same scene, uses at least two polarization images to estimate the parameters of the physical model, and then inversely solves the physical model to restore the image. However, this method is only applicable to specific surveillance scenes, and it is difficult to obtain two polarization images at the same time for other dynamic scenes. Image defogging based on image enhancement enhances the image by redistributing the histogram. This method is simple and fast and can enhance the visibility of the image, but it will cause color distortion and noise amplification. In image defogging based on image fusion, the first image is used to restore color and reduce noise, and the second image is used to enhance the visibility of the image. However, if the second input image cannot effectively enhance the visibility of the blurred image, they will not be able to achieve good performance. Existing defogging methods all have certain problems, and cannot make the defogging image not distorted and have good visibility while defogging efficiently.
[0004] In view of this, it is necessary to propose a new image dehazing method to solve the above problems. Summary of the invention
[0005] The object of the present invention is to provide an image defogging method to solve the problem that the existing image defogging method cannot effectively defog the image while preventing the defogging image from being distorted and having good visibility.
[0006] To achieve the above object, the present invention provides an image defogging method, comprising the following steps:
[0007] Step 1: Obtain a public data set as training data, wherein the public data set includes foggy images, defogged images, and non-fog images taken of the same scene;
[0008] Step 2: inputting the training data into an adversarial model including a generator and a discriminator for training, wherein the generator includes an encoder, a decoder and a feature restoration module, the encoder includes at least one convolutional layer, and the decoder includes at least one deconvolutional layer;
[0009] Step 3: Repeat steps 1 and 2 until the generator and the discriminator reach a Nash equilibrium state, and obtain a trained adversarial model;
[0010] Step 4: Input the image to be dehazed into the trained adversarial model in step 3 to obtain an optimized dehazed image.
[0011] As a further improvement of the present invention, in step 2, the generator is used to perform a defogging operation on the foggy image to obtain the defogged image, and the discriminator is used to compare the defogged image with the haze-free image to guide the defogging operation of the generator and reduce the difference between the defogged image and the haze-free image.
[0012] As a further improvement of the present invention, the specific steps of training in step 2 are:
[0013] Step 21: input the foggy image into the encoder, and the encoder extracts features of the input foggy image through the convolution layer and generates a feature image;
[0014] Step 22: the feature restoration module uses a residual network to restore the feature image, obtains a feature restoration image and inputs it into the decoder;
[0015] Step 23: the decoder outputs the feature-restored image as a primary dehazed image by deconvolution;
[0016] Step 24: inputting the haze-free image into the discriminator, and the discriminator compares the haze-free image with the initial dehazed image obtained in step 23 to obtain a feature matching loss;
[0017] Step 25: Use the feature matching loss to perform loss repair on the feature repair image in step 22 to obtain a repaired feature repair image, and input it into the decoder;
[0018] Step 26: The decoder outputs the patched feature repaired image as a final dehazed image by deconvolution.
[0019] As a further improvement of the present invention, there are at least two discriminators, which are used to compare the haze-free image and the initial dehazed image at different image sampling sizes.
[0020] As a further improvement of the present invention, the discriminator includes a first discriminator, a second discriminator and a third discriminator. The first discriminator uses an image sampled at the original size of the haze-free image and the initial defogged image for comparison; the second discriminator uses an image downsampled by 1 / 2 of the haze-free image and the initial defogged image for comparison; the third discriminator uses an image downsampled by 1 / 4 of the haze-free image and the initial defogged image for comparison.
[0021] As a further improvement of the present invention, the judgment selection formula of the discriminator is:
[0022]
[0023] Where G is the generator, D k is the discriminator, L GAN (G,D k ) is the loss function.
[0024] As a further improvement of the present invention, in step 2, both the encoder and the decoder include a dense feature fusion module, and the dense feature fusion module is used to apply the feature information of each layer to the next layer when the convolution layer performs convolution and the deconvolution layer performs deconvolution.
[0025] As a further improvement of the present invention, the formula of the dense feature fusion module in the decoder is expressed as:
[0026]
[0027] Among them, i n is the feature map in the nth layer of the encoder, is the output of the n-th layer dense feature fusion module, It is the sum of the outputs of the dense feature fusion modules from the first layer to the n-1th layer in the encoder.
[0028] As a further improvement of the present invention, the decoder adopts an enhancement strategy, each deconvolution layer has an enhancement unit, and for the enhancement unit of the nth layer, the feature j extracted from the enhancement unit of the n+1th layer is n+1 Upsample and use the feature map i of the same layer corresponding to the encoder n Enhanced, finally the extracted feature j of the n-th layer enhancement unit is obtained n , extract feature j n It is expressed as:
[0029]
[0030] Among them, ↑2 means that the size difference between the n-th layer enhancement unit and the n+1-th layer enhancement unit is 2 times, Indicates that the nth layer parameter is θn Repair unit.
[0031] As a further improvement of the present invention, the feature matching loss in step 24 The expression formula is:
[0032]
[0033] Where T is the total number of layers of the discriminator, N i represents the number of elements in the i-th layer of the discriminator, represents the feature extractor of the i-th layer of the discriminator.
[0034] The beneficial effect of the present invention is that the image defogging method of the present invention repeatedly inputs a public data set including foggy images, defogged images and non-fog images taken of the same scene into an adversarial model including a generator and a discriminator for training until the generator and the discriminator reach a Nash equilibrium state, thereby obtaining a trained adversarial model that can use and output an optimized defogged image, thereby solving the problem that the existing image defogging method cannot efficiently defog while making the defogged image not distorted and having good visibility. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flow chart of the image defogging method of the present invention.
[0036] Figure 2 yes Figure 1 Flow chart of the specific steps of step 2 in .
[0037] Figure 3 It is a structural block diagram of the adversarial model in the present invention. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0039] like Figure 1 As shown, the present invention discloses an image defogging method, which specifically includes the following steps:
[0040] Step 1: Obtain a public dataset as training data. The public dataset includes foggy images, defogged images, and non-fog images taken of the same scene.
[0041] Step 2: Input the training data into the adversarial model including the generator G and the discriminator D for training;
[0042] Step 3: Repeat steps 1 and 2 until the generator G and the discriminator D reach a Nash equilibrium state and obtain the trained adversarial model;
[0043] Step 4: Input the image to be dehazed into the trained adversarial model described in step 3 to obtain an optimized dehazed image.
[0044] The generator G in the adversarial model of the present invention is used to perform a defogging operation on the input foggy image to obtain a defogging image. The generator G includes an encoder G Enc , Feature repair module G Res and decoder G Dec , where the encoder G Enc It includes at least one convolution layer, through which the input foggy image is convolved to extract feature information and obtain a feature image; a feature restoration module G Res Used to repair the feature image through the residual network; decoder G Dec It includes at least one deconvolution layer, and outputs a haze-free image through deconvolution.
[0045] The discriminator D is used to compare the dehazed image with the haze-free image, guide the dehazing operation of the generator G, and reduce the difference between the dehazed image and the haze-free image.
[0046] The specific steps for training in step 2 are:
[0047] Step 21: Feed the foggy image into encoder G Enc In the encoder G Enc The convolution layer is used to extract the features of the input foggy image and generate a feature image;
[0048] Step 22: Feature Repair Module G Res The feature image is repaired using the residual network to obtain the feature repaired image and input it into the decoder G Dec middle;
[0049] Step 23: Decoder G Dec The feature repaired image is output as the initial dehazed image by deconvolution;
[0050] Step 24: The haze-free image is input into the discriminator D. The discriminator D compares the haze-free image with the initial dehazed image obtained in step 23 to obtain the feature matching loss.
[0051] Step 25: Use feature matching loss to repair the feature repair image in step 22, obtain the repaired feature repair image, and input it into the decoder G Dec middle;
[0052] Step 26: Decoder G Dec The patched feature repaired image is output as the final dehazed image by deconvolution.
[0053] There are at least two discriminators D, which are used to compare the haze-free image and the initial dehazed image at different image sampling sizes.
[0054] The discriminator D includes a first discriminator D1, a second discriminator D2 and a third discriminator D3. The first discriminator D1 uses an image sampled at the original size of the fog-free image to compare with the initial defogging image; the second discriminator D2 uses an image downsampled to 1 / 2 of the fog-free image to compare with the initial defogging image; the third discriminator D3 uses an image downsampled to 1 / 4 of the fog-free image to compare with the initial defogging image. The downsampling sizes of the fog-free images used by the first discriminator D1, the second discriminator D2 and the third discriminator D3 can be set by the user as needed, as long as the three different downsampling sizes can distinguish the fog-free images.
[0055] In order to distinguish high-resolution real images from synthetic images, the discriminator D needs to have a larger receptive field, which requires a deeper network or a larger convolution kernel, which will increase the network capacity and require more memory, so a multi-scale discriminator D is used. In this embodiment, the haze-free image and the initial dehazed image are first sampled, and then a multi-scale image pyramid is created with three image scales: the original image, 1 / 2 downsampled of the original image, and 1 / 4 downsampled of the original image. The discriminators D1, D2, and D3 distinguish the haze-free image from the initial dehazed image at three scales. Although the structures of the discriminators D1, D2, and D3 are the same, due to their different receptive fields, the guidance of the generator G is different. The discriminator with the largest receptive field guides the generator G to generate a globally consistent image, and the discriminator with the smallest receptive field guides the generator G to generate finer details. In the discriminator D, the learning problem becomes a multi-scale learning problem, and the judgment selection formula of the discriminator D is:
[0056]
[0057] Where G is the generator, D k is the discriminator, L GAN (G,D k ) is the loss function.
[0058] In step 2, the encoder G Enc and decoder G Dec Both include a dense feature fusion module (Dense Feature Fusion Module), which is used to apply the feature information of each layer to the next layer when the convolution layer performs convolution and the deconvolution layer performs deconvolution.
[0059] The formula of the dense feature fusion module is expressed as:
[0060]
[0061] Among them, i n For encoder G EncThe feature map in the nth layer, is the output of the n-th layer dense feature fusion module, For encoder G Enc The sum of the outputs of the dense feature fusion modules from the first layer to the n-1th layer in . Calculate the tth iteration, and difference Calculated as in For the back-projection operation, Upsample to and Same dimensions. Then update for in For the projection operation, Downsample to and Same dimension. After iterating all enhanced features before the nth layer (that is, when t traverses the values in {0,1,…,n-2}), the final enhanced features are obtained.
[0062] In the decoder G Dec In the example, the dense feature fusion module has the same effect as in the encoder G. Enc The effect is similar to .
[0063] The decoder G adopts a strengthening strategy (Strengthen-Operate-Subtract boosting algorithm). Each deconvolution layer has a strengthening unit. For the strengthening unit of the nth layer, the feature j extracted from the strengthening unit of the n+1th layer is n+1 Upsample and use the same layer corresponding encoder G Enc The feature map i n Enhanced, finally the extracted feature j of the n-th layer enhancement unit is obtained n , extract feature j n It is expressed as:
[0064]
[0065] Among them, ↑2 means that the size difference between the n-th layer enhancement unit and the n+1-th layer enhancement unit is 2 times, Indicates that the nth layer parameter is θ n Repair unit.
[0066] The generated image is improved by introducing a feature matching loss based on the discriminator D, because the generator G needs to produce natural statistics at different scales, so the feature matching loss can make the training more stable. Feature information is extracted from multiple layers of the discriminator D, and then the intermediate representation of matching the haze-free image and the initial dehazed image is learned. The feature matching loss in step 24 The expression formula is:
[0067]
[0068] Where T is the total number of layers of the discriminator D, N i represents the number of elements in the i-th layer of the discriminator D, represents the feature extractor of the i-th layer of the discriminator D.
[0069] In summary, the image defogging method of the present invention repeatedly inputs a public data set including foggy images, defogged images and fog-free images taken of the same scene into an adversarial model including a generator and a discriminator for training until the generator and the discriminator reach a Nash equilibrium state, thereby obtaining a trained adversarial model that can use and output an optimized defogged image, thereby solving the problem that the existing image defogging method cannot efficiently defog while making the defogged image not distorted and having good visibility.
[0070] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. An image defogging method, characterized in that: The specific steps include: Step 1: Obtain a public data set as training data, wherein the public data set includes foggy images, defogged images, and non-fog images taken of the same scene; Step 2: inputting the training data into an adversarial model including a generator and a discriminator for training, wherein the generator includes an encoder, a decoder and a feature restoration module, the encoder includes at least one convolutional layer, and the decoder includes at least one deconvolutional layer; Step 3: Repeat steps 1 and 2 until the generator and the discriminator reach a Nash equilibrium state, and obtain a trained adversarial model; Step 4: input the image to be dehazed into the trained adversarial model in step 3 to obtain an optimized dehazed image; In the step 2, the encoder and the decoder both include a dense feature fusion module, and the dense feature fusion module is used to apply feature information of each layer to the next layer when the convolution layer performs convolution and the deconvolution layer performs deconvolution; The formula of the dense feature fusion module in the decoder is expressed as: in, For encoder The feature maps in the layer, For the The output of the layer-dense feature fusion module, The first to the second layers in the encoder -The sum of the outputs of the 1-layer dense feature fusion module; The decoder adopts an enhancement strategy. Each deconvolution layer has an enhancement unit. For the enhancement unit of the nth layer, the feature extraction of the enhancement unit of the n+1th layer is performed. Upsample and use the feature map of the corresponding encoder in the same layer Enhanced, finally the extracted features of the n-th layer enhancement unit are obtained , extract features It is expressed as: in, It means that the size difference between the n-th layer enhancement unit and the n+1-th layer enhancement unit is 2 times, Indicates that the nth layer parameter is Repair unit.
2. The image defogging method according to claim 1, characterized in that: In step 2, the generator is used to perform a defogging operation on the foggy image to obtain the defogged image, and the discriminator is used to compare the defogged image with the fog-free image to guide the defogging operation of the generator and reduce the difference between the defogged image and the fog-free image.
3. The image defogging method according to claim 1, characterized in that: The specific steps of training in step 2 are: Step 21: input the foggy image into the encoder, and the encoder extracts features of the input foggy image through the convolution layer and generates a feature image; Step 22: the feature restoration module uses a residual network to restore the feature image, obtains a feature restoration image and inputs it into the decoder; Step 23: the decoder outputs the feature-restored image as a primary dehazed image by deconvolution; Step 24: inputting the haze-free image into the discriminator, and the discriminator compares the haze-free image with the initial dehazed image obtained in step 23 to obtain a feature matching loss; Step 25: Use the feature matching loss to perform loss repair on the feature repair image in step 22 to obtain a repaired feature repair image, and input it into the decoder; Step 26: The decoder outputs the patched feature repaired image as a final dehazed image by deconvolution.
4. The image defogging method according to claim 3, characterized in that: There are at least two discriminators, which are used to compare the haze-free image and the initial dehazed image at different image sampling sizes.
5. The image defogging method according to claim 4, characterized in that: The discriminator includes a first discriminator, a second discriminator and a third discriminator. The first discriminator uses an image sampled at the original size of the haze-free image to compare with the initial defogging image; the second discriminator uses an image downsampled by 1 / 2 of the haze-free image to compare with the initial defogging image; the third discriminator uses an image downsampled by 1 / 4 of the haze-free image to compare with the initial defogging image.
6. The image defogging method according to claim 5, characterized in that: The judgment selection formula of the discriminator is: Where G is the generator, is the discriminator, is the loss function.
7. The image defogging method according to claim 3, characterized in that: The feature matching loss described in step 24 The expression formula is: Where T is the total number of layers of the discriminator, represents the number of elements in the i-th layer of the discriminator, (s,x) represents the discriminator The feature extractor of the layer.
