A progressive image defogging method and system based on CNN and convolutional LSTM network

Abstract

The application provides a progressive image defogging method based on a CNN and a convolutional LSTM network, and relates to the technical field of image processing. The method comprises the following steps: obtaining a haze-free image and a hazy image under the same scene; constructing an image defogging model comprising a down-sampling module, a plurality of defogging modules and an up-sampling module, wherein the defogging modules are embedded with LSTM modules; inputting the hazy image into the image defogging model to obtain a haze-free image estimated by the model; calculating a loss together with the haze-free image estimated by the model, the haze-free image and the hazy image, training the image defogging model, and obtaining a trained image defogging model; cutting the trained image defogging model, selecting the number of defogging modules to be used, constructing a prediction model, and realizing haze removal of a haze weather degraded image based on the prediction model. The progressive image defogging model can make the converged model have a certain cuttability to adapt to different application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a progressive image dehazing method and system based on CNN and convolutional LSTM networks. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Imaging systems in hazy or foggy weather are affected by fog, aerosols, dust, and other pollutants in the air, resulting in image degradation such as whitening, reduced contrast, color changes, and loss of detail. These degradation factors significantly reduce image quality and severely impact the performance of subsequent computer vision tasks such as object detection, image segmentation, and image understanding. Therefore, image dehazing is essential.

[0004] Current image dehazing methods are mainly divided into three types: dehazing methods based on traditional image enhancement techniques, image dehazing methods based on prior knowledge, and image dehazing methods based on deep learning. Among them:

[0005] Traditional image enhancement techniques for dehazing mainly rely on histogram equalization, homomorphic filtering, wavelet transform, and other techniques. However, because these techniques do not incorporate the imaging principles of foggy images or consider the texture features of the images, they suffer from poor dehazing effects, inability to effectively restore detail information, and color distortion.

[0006] Image dehazing methods based on prior knowledge combine atmospheric scattering models, using hand-designed prior knowledge or assumptions as additional constraints in the estimation process of atmospheric light component A(x) and transmittance distribution t(x). However, hand-designed prior knowledge may fail in some scenarios, leading to local color distortion. For example, the dark channel prior may not hold true in the sky region of the image, causing color distortion in the sky region after dehazing.

[0007] Deep learning-based image dehazing methods utilize deep neural networks to extract image features and learn the input-output mapping from a large number of hazy / clear image pairs. Early deep learning-based methods directly estimated the transmittance distribution t(x) and atmospheric light composition A(x) using neural networks, then obtained a clean output image based on an atmospheric scattering model. However, the estimation of transmittance distribution and atmospheric light composition is easily affected by noise, leading to errors. This non-end-to-end learning approach suffers from accumulated errors, resulting in a loss of dehazing performance. Subsequently, some end-to-end image dehazing methods based on deep neural networks were proposed. These methods no longer estimate parameters in the atmospheric scattering model but directly recover a clean image from the input hazy image, thus avoiding noise and accumulated errors. However, this method often relies on a non-clipping neural network structure and cannot adjust the model size according to different application scenarios. Summary of the Invention

[0008] To overcome the shortcomings of the prior art, this invention provides a progressive image dehazing method and system based on CNN and convolutional LSTM networks. By building a progressive image dehazing model, the converged model can have a certain degree of scalability to adapt to different application scenarios.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] The first aspect of this invention provides a progressive image dehazing method based on CNN and convolutional LSTM networks.

[0011] A progressive image dehazing method based on CNN and convolutional LSTM networks includes the following steps:

[0012] Acquire fog-free and foggy images of the same scene;

[0013] Construct an image dehazing model that includes a downsampling module, several dehazing modules, and an upsampling module, wherein the dehazing module embeds an LSTM module;

[0014] The foggy image is input into the image dehazing model to obtain the fog-free image estimated by the model;

[0015] The loss is calculated together from the fog-free image, the fog-free image, and the foggy image estimated by the model, and the image dehazing model is trained to obtain the trained image dehazing model.

[0016] The trained image dehazing model is cropped, the number of dehazing modules used is selected, a prediction model is constructed, and the dehazing process for de-fogging images is implemented based on the prediction model.

[0017] A second aspect of the present invention provides a progressive image dehazing system based on CNN and convolutional LSTM networks.

[0018] A progressive image dehazing system based on CNN and convolutional LSTM networks includes:

[0019] The image acquisition module is configured to acquire both fog-free and foggy images of the same scene.

[0020] The model building module is configured to build an image dehazing model that includes a downsampling module, several dehazing modules, and an upsampling module, wherein the dehazing module contains an LSTM module.

[0021] The model-estimated haze-free image acquisition module is configured to: input a hazy image into an image dehazing model to obtain a model-estimated haze-free image;

[0022] The training module is configured to calculate the loss together with the fog-free image, fog-free image, and foggy image estimated by the model, train the image dehazing model, and obtain the trained image dehazing model.

[0023] The cropping module is configured to: crop the trained image dehazing model, select the number of dehazing modules to use, build a prediction model, and perform dehazing processing on foggy and degraded images based on the prediction model.

[0024] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps of the progressive image dehazing method based on CNN and convolutional LSTM networks as described in the first aspect of the present invention.

[0025] The fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the progressive image dehazing method based on CNN and convolutional LSTM networks as described in the first aspect of the present invention.

[0026] The above one or more technical solutions have the following beneficial effects:

[0027] This invention combines an atmospheric scattering model with a dehazing module design, which effectively utilizes information from the input image to improve the color and detail recovery of the output image. Furthermore, this invention constructs a progressive dehazing model with multiple dehazing modules cascaded together, and embeds an LSTM module within each dehazing module. This allows the converged model to be tailored to adapt to different scenarios and performance requirements.

[0028] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0030] Figure 1 This is a flowchart of the method of the first embodiment of the present invention;

[0031] Figure 2 This is a structural diagram of the image dehazing model according to the first embodiment of the present invention;

[0032] Figure 3 This is a structural diagram of the defogging module according to the first embodiment of the present invention;

[0033] Figure 4 This is a structural diagram of the LSTM module in the first embodiment of the present invention;

[0034] Figure 5(a) is a fog-free image of the first embodiment of the present invention;

[0035] Figure 5(b) shows a foggy image of the first embodiment of the present invention;

[0036] Figure 5(c) shows the defogging effect when the number of defogging modules in the first embodiment of the present invention is 3;

[0037] Figure 5(d) shows the defogging effect when the number of defogging modules in the first embodiment of the present invention is 6;

[0038] Figure 6 This is a system structure diagram of the second embodiment. Detailed Implementation

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0041] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0042] Example 1

[0043] This embodiment discloses a progressive image dehazing method based on CNN and convolutional LSTM networks.

[0044] like Figure 1 As shown, a progressive image dehazing method based on CNN and convolutional LSTM networks includes the following steps:

[0045] S1: Obtain both fog-free and foggy images of the same scene;

[0046] S2: Construct an image dehazing model that includes a downsampling module, several dehazing modules, and an upsampling module, wherein the dehazing module contains an LSTM module;

[0047] S3: Input the foggy image into the image dehazing model to obtain the fog-free image estimated by the model;

[0048] S4: Calculate the loss together with the fog-free image, fog-free image, and foggy image estimated by the model, and train the image dehazing model to obtain the trained image dehazing model;

[0049] S5: Crop the trained image dehazing model, select the number of dehazing modules to use, build a prediction model, and implement dehazing processing for images degraded by fog on foggy days based on the prediction model.

[0050] Preferably, step S1 specifically includes:

[0051] Obtain a fog-free image of the scene, and then generate a foggy image of the same scene based on an atmospheric scattering model. The specific formula for the atmospheric scattering model is as follows:

[0052] I(x)=J(x)t(x)+A(x)(1-t(x)),

[0053] Where I(x) represents a foggy image, J(x) represents a fog-free image, t(x) represents the transmittance distribution, and A(x) represents the atmospheric light composition.

[0054] The obtained foggy and fog-free images are paired to form an image pair dataset, which is then divided into a training set and a test set in a 4:1 ratio.

[0055] Preferably, step S2 specifically includes:

[0056] like Figure 2 As shown in (a), the downsampling module first includes a boundary padding layer with a padding size of 3, followed by three combined blocks in series. Each combined block consists of a convolutional layer and a ReLU activation layer connected in series. The convolutional kernel sizes of the convolutional layers in the three combined blocks are 7×7, 3×3, and 3×3, respectively, with strides of 1, 2, and 2, and padding of 0, 1, and 1, respectively.

[0057] From the atmospheric scattering model, we can obtain:

[0058] J(x)=M(x)(I(x)-A(x))+I(x)

[0059] Among them, M(x)=(1-t(x)) / t(x);

[0060] In this embodiment, the number of defogging modules connected in series is set to 6; such as Figure 3 As shown, the dehazing module takes I(x) as input and uses channel-level attention modules and pixel-level attention modules to simulate M(x); where:

[0061] The channel-level attention module consists of a serially connected average pooling layer, a convolutional layer, a ReLU activation layer, a convolutional layer, and a Sigmoid activation layer. The convolutional kernel size of the convolutional layer is 1×1, the stride is 1, and the padding is 0.

[0062] The pixel-level attention module consists of a series of convolutional layers, ReLU activation layers, LSTM modules, convolutional layers, and Sigmoid activation layers. The convolutional kernels of the convolutional layers are 1×1 in size, with a stride of 1 and padding of 0.

[0063] The dehazing module uses a residual block to simulate (I(x)-A(x)) in the above model, and connects the module input to the module output to simulate I(x) at the end of the above model; the residual block consists of convolutional layers and ReLU activation layers, with the convolutional kernel size being 3×3, stride being 1, and padding being 1.

[0064] The dehazing module achieves progressive learning through an embedded LSTM module, such as... Figure 4 As shown, the working principle of the LSTM module can be represented by the following formula:

[0065] f=σ(W f [x,H t-1 ]+b f ),

[0066] i = σ(W i [x,H t-1 ]+b i ),

[0067] g = tanh(W) g [x,H t-1 ]+b g ),

[0068] o=σ(W o [x,H t-1 ]+b o ),

[0069]

[0070]

[0071] Where x is the input of the LSTM module, H t This is the output of the LSTM module, C t H is the state variable of the LSTM module.t-1 and C t-1 These are the output and state variables of the LSTM module in the previous dehazing module. f, i, o, and g represent the outputs of the forget gate, input gate, output gate, and input modulation gate, respectively. W f W i W g W o These are the weight matrices for the forget gate, input gate, output gate, and input modulation gate, respectively. f b i b g b o σ represents the bias vectors for the forget gate, input gate, output gate, and input modulation gate, respectively; tanh represents the sigmoid activation function; and tanh represents the tanh activation function.

[0072] The upsampling module first contains two serial combination blocks, which consist of a series of deconvolutional layers and ReLU activation layers. The deconvolutional layer has a kernel size of 3×3, a stride of 2, and padding of 1. Then, a padding layer, a convolutional layer, and a Tanh activation layer are connected in series. The convolutional layer has a kernel size of 7×7, a stride of 1, and padding of 0.

[0073] To fully utilize the invariant features in the input image, the outputs of the last two convolutional layers of the downsampling module are added to the inputs of the first two deconvolutional layers of the upsampling module to establish a fusion connection.

[0074] Steps S3 and S4 specifically include:

[0075] The training of the image dehazing model is constrained by a combination of L1 loss and feature contrast loss. The specific formula for L1 loss is as follows:

[0076]

[0077] Where Y represents the fog-free image estimated by the model, Y t Represents a true, fog-free image;

[0078] The model-estimated haze-free image, the real haze-free image, and the haze image are fed into a VGG19 network pre-trained on the ImageNet dataset. The output feature maps of layers 1, 3, 5, 9, and 13 are used to calculate the feature contrast loss. The specific formula is as follows:

[0079]

[0080] Where X represents the input foggy image, Φ iLet ω represent the output feature of the i-th layer of the VGG19 network. i Indicates the weighting coefficient;

[0081] The total loss function of the image dehazing model is:

[0082] L=L1+λL pc

[0083] Where λ is the equilibrium parameter, with a value of 0.1.

[0084] Step S5 specifically includes:

[0085] The trained image dehazing model is cropped according to the requirements for image restoration effect and real-time processing: the trained downsampling module, the first N dehazing modules, and the upsampling module are connected in series to form a prediction model, where N does not exceed the number of dehazing modules set during training.

[0086] The experimental results of this embodiment are given below:

[0087] This embodiment tests the impact of embedding an LSTM module in the dehazing module on the restoration of degraded images, and the restoration effect of degraded images when the number of dehazing modules is cropped to 3 and not cropped. The most common and widely used PSNR (Peak Signal-to-Noise Ratio) is introduced to evaluate the restoration effect of degraded images. The PSNR calculation formula is as follows:

[0088]

[0089]

[0090] Here, 255 refers to the maximum pixel value in an 8-bit grayscale image. MSE (Mean Square Error) is the mean square error between the estimated haze-free image and the true haze-free image, where M and N represent the image height and width, respectively. n and P n These are the nth pixel values ​​of the real haze-free image and the estimated haze-free image, respectively. PSNR is measured in dB, and a higher PSNR value indicates better image restoration.

[0091] Table 1 shows the PSNR values ​​for tests with and without LSTM modules embedded in the dehazing module, and with different numbers of dehazing modules N. It can be seen that embedding LSTM modules in the dehazing module allows the intermediate dehazing layers to output features similar to the final layer, thus giving the converged model a certain degree of scalability. In scenarios where model size is limited and dehazing performance is more acceptable, the number of dehazing modules can be reduced to obtain a lightweight model without retraining. The image dehazing effects using different numbers of dehazing modules are shown below. Figures 5(a)-5(b) As shown.

[0092] Table 1. PSNR values ​​for tests with and without LSTM embedded in the dehazing module and with different numbers of dehazing modules N.

[0093]

[0094]

[0095] Example 2

[0096] This embodiment discloses a progressive image dehazing system based on CNN and convolutional LSTM networks.

[0097] like Figure 6 As shown, a progressive image dehazing system based on CNN and convolutional LSTM networks includes:

[0098] The image acquisition module is configured to acquire both fog-free and foggy images of the same scene.

[0099] The model building module is configured to build an image dehazing model that includes a downsampling module, several dehazing modules, and an upsampling module, wherein the dehazing module contains an LSTM module.

[0100] The model-estimated haze-free image acquisition module is configured to: input a hazy image into an image dehazing model to obtain a model-estimated haze-free image;

[0101] The training module is configured to calculate the loss together with the fog-free image, fog-free image, and foggy image estimated by the model, train the image dehazing model, and obtain the trained image dehazing model.

[0102] The cropping module is configured to: crop the trained image dehazing model, select the number of dehazing modules to use, build a prediction model, and perform dehazing processing on foggy and degraded images based on the prediction model.

[0103] Example 3

[0104] The purpose of this embodiment is to provide a computer-readable storage medium.

[0105] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps in the progressive image dehazing method based on CNN and convolutional LSTM networks as described in Embodiment 1 of this disclosure.

[0106] Example 4

[0107] The purpose of this embodiment is to provide an electronic device.

[0108] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the progressive image dehazing method based on CNN and convolutional LSTM networks as described in Embodiment 1 of this disclosure.

[0109] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0110] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0111] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A progressive image dehazing method based on CNN and convolutional LSTM networks, characterized in that, Includes the following steps: Acquire fog-free and foggy images of the same scene; Construct an image dehazing model that includes a downsampling module, several dehazing modules, and an upsampling module, wherein the dehazing module embeds an LSTM module; The foggy image is input into the image dehazing model to obtain the fog-free image estimated by the model; The loss is calculated together from the fog-free image, the fog-free image, and the foggy image estimated by the model, and the image dehazing model is trained to obtain the trained image dehazing model. The trained image dehazing model is cropped, the number of dehazing modules used is selected, a prediction model is constructed, and the dehazing process of the de-fogging image is realized based on the prediction model. The downsampling module includes a filler layer, followed by three combined blocks in series. Each combined block consists of a convolutional layer and a ReLU activation layer connected in series. From the atmospheric scattering model, we can obtain: J(x) = M(x)(I(x) - A(x)) + I(x), Where M(x) = (1-t(x)) / t(x), the dehazing module takes the hazy image as input, uses channel-level attention and pixel-level attention modules to simulate M(x), uses a residual block to simulate (I(x)-A(x)), and connects the hazy image input to the dehazing module to the output to simulate I(x); I(x) represents the hazy image, J(x) represents the hazy image, t(x) represents the transmittance distribution, and A(x) represents the atmospheric light composition; The upsampling module contains two serially combined blocks, each consisting of a serially connected deconvolutional layer and a ReLU activation layer, followed by a serially connected padding layer, a convolutional layer, and a Tanh activation layer. The outputs of the last two convolutional layers of the downsampling module are added to the inputs of the first two deconvolutional layers of the upsampling module to establish a fusion connection. The LSTM module is embedded within the pixel-level attention module to achieve progressive learning.

2. The progressive image dehazing method based on CNN and convolutional LSTM networks as described in claim 1, characterized in that, Obtain a haze-free image, and then generate a corresponding haze image based on an atmospheric scattering model. The atmospheric scattering model is as follows: I(x)=J(x)t(x)+A(x)(1-t(x)).

3. The progressive image dehazing method based on CNN and convolutional LSTM networks as described in claim 1, characterized in that, The training of the image dehazing model is constrained by two loss methods. The first loss is L1 loss, specifically: Where Y represents the fog-free image estimated by the model, Y t Represents a true, fog-free image; The second method is feature contrast loss, which uses the feature maps output from layers 1, 3, 5, 9, and 13 of a VGG19 network pre-trained on the ImageNet dataset to calculate the contrast loss. The specific formula is as follows: Where X represents the input foggy image, Let represent the output features of the i-th layer of the VGG19 network. Indicates the weighting coefficient; The total loss function of the image dehazing model is: in, It is a balance parameter.

4. The progressive image dehazing method based on CNN and convolutional LSTM networks as described in claim 1, characterized in that, The maximum number of dehazing modules in the prediction model shall not exceed the number of dehazing modules set during the training of the image dehazing model.

5. A progressive image dehazing system based on CNN and convolutional LSTM networks, characterized in that: include: The image acquisition module is configured to acquire both fog-free and foggy images of the same scene. The model building module is configured to: build an image dehazing model that includes a downsampling module, several dehazing modules and an upsampling module, wherein the dehazing module embeds an LSTM module; the downsampling module includes a filler layer, followed by three serial combination blocks, each combination block consisting of a serially connected convolutional layer and a ReLU activation layer. From the atmospheric scattering model, we can obtain: J(x) = M(x)(I(x) - A(x)) + I(x), Where M(x) = (1-t(x)) / t(x), the dehazing module takes the hazy image as input, uses channel-level attention and pixel-level attention modules to simulate M(x), uses a residual block to simulate (I(x)-A(x)), and connects the hazy image input to the dehazing module to the output to simulate I(x); I(x) represents the hazy image, J(x) represents the hazy image, t(x) represents the transmittance distribution, and A(x) represents the atmospheric light composition; The upsampling module contains two serially combined blocks, each consisting of a serially connected deconvolutional layer and a ReLU activation layer, followed by a serially connected padding layer, a convolutional layer, and a Tanh activation layer. The outputs of the last two convolutional layers of the downsampling module are added to the inputs of the first two deconvolutional layers of the upsampling module to establish a fusion connection. The LSTM module is embedded within the pixel-level attention module to achieve progressive learning; The model-estimated haze-free image acquisition module is configured to: input a hazy image into an image dehazing model to obtain a model-estimated haze-free image; The training module is configured to calculate the loss together with the fog-free image, fog-free image, and foggy image estimated by the model, train the image dehazing model, and obtain the trained image dehazing model. The cropping module is configured to: crop the trained image dehazing model, select the number of dehazing modules to use, build a prediction model, and perform dehazing processing on foggy and degraded images based on the prediction model.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by a processor, the program implements the steps in the progressive image dehazing method based on CNN and convolutional LSTM networks as described in any one of claims 1-4.

7. An electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the progressive image dehazing method based on CNN and convolutional LSTM networks as described in any one of claims 1-4.

Images