Image defogging method and system based on high-quality coding dictionary prior

Abstract

The application discloses an image defogging method and system based on a high-quality coding dictionary prior, comprising: synthesizing a corresponding foggy image after image degradation processing of a given clear image, forming an image pair with the foggy image and the clear image, and training a defogging network, wherein the defogging network comprises an encoder and a decoder; obtaining a clear image by using the trained defogging network on a foggy image to be processed; wherein the encoder is used to extract image features on the foggy image to be processed, the image features are matched with a coding dictionary in a nearest neighbor feature matching manner, and the decoder is used to perform defogging processing according to the image features under the guidance of the nearest neighbor features. The foggy image synthesis method and the foggy network using the high-quality prior can achieve more reliable and effective defogging effects, and solve the problems of data artifacts generated in the existing defogging method and inherent defects of the prior method.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image dehazing method and system based on a high-quality coding dictionary prior. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Regarding image dehazing, early attempts primarily focused on estimating the parameters of the atmospheric scattering model proposed in the equations to achieve the desired dehazing effect. For example, the 2011 paper "Single image haze removal using dark channel prior" published in TPAMI proposed a dark channel prior, assuming that pixels in non-hazy areas have low intensity in at least one color channel, achieving impressive results. However, empirically based, manually designed priors are limited to specific scenarios.

[0004] With the development of deep learning technology, many studies have attempted to use convolutional neural networks (CNNs) to estimate the parameters of the degenerate model in the equations. In addition, to avoid the accumulation of errors in parameter estimation, a series of end-to-end networks were subsequently proposed to directly estimate fog-free images. The above learning-based methods have achieved excellent performance on synthetic datasets, however, their performance degrades significantly on real-world data.

[0005] Regarding dehazing of real images, some studies have utilized Generative Adversarial Networks (GANs) to generate blurred data suitable for real-world conditions; however, GANs are prone to producing artifacts in the generated results, which is detrimental to model training. Another research direction aims to introduce prior knowledge through loss functions or network architectures; however, directly using manually designed priors cannot avoid the inherent defects of prior-based methods. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes an image dehazing method and system based on a high-quality coding dictionary prior. By employing a hazy image synthesis method and utilizing a high-quality prior hazy network, a more reliable and effective dehazing effect is achieved, thus resolving the problems of data artifacts and inherent defects in prior methods in existing dehazing approaches.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides an image dehazing method based on a high-quality coding dictionary prior, comprising:

[0009] A given clear image is degraded and then synthesized into a corresponding hazy image. The hazy image and the clear image are paired to train the dehazing network, which includes an encoder and a decoder.

[0010] The trained dehazing network is used to obtain a clear image from the foggy image to be processed. Specifically, the encoder extracts image features from the foggy image to be processed, and performs nearest neighbor feature matching between the image features and the encoding dictionary. Guided by the nearest neighbor feature, the decoder performs dehazing processing based on the image features.

[0011] As an alternative implementation, the image degradation processing procedure includes:

[0012] The gamma adjustment is applied to a given sharp image, and noise is added to simulate insufficient lighting.

[0013] The transmission map of foggy weather is synthesized based on image depth, and atmospheric colors with color bias are randomly synthesized. Fog images are synthesized according to atmospheric scattering models, and JPEG compression is added to form the final foggy image.

[0014] As an alternative implementation, the encoder of the dehazing network includes a convolutional layer with a kernel size of 3, two residual blocks with downsampling steps, and four residual Transformer layers based on sliding windows.

[0015] As an alternative implementation, the decoder of the dehazing network includes two residual blocks with upsampling steps and a convolutional layer with a kernel size of 3.

[0016] As an alternative implementation, the nearest neighbor feature matching uses a trained VQGAN model. The matched nearest neighbor features are input into the decoder of the trained VQGAN, and the decoder of the dehazing network is guided by the decoder of the trained VQGAN to perform dehazing processing.

[0017] As an alternative implementation, the training process of the VQGAN model includes:

[0018] The VQGAN model's encoder extracts image features from a given foggy image.

[0019] The image features are matched with the nearest neighbor features of the encoding dictionary. The nearest neighbor features are then used to obtain the reconstruction result using the decoder of the VQGAN model. The VQGAN model is trained based on the reconstruction result, and the loss between the reconstruction result and the given foggy image is used as the optimization objective.

[0020] As an alternative implementation, the encoder of the VQGAN model includes a convolutional layer with a kernel size of 3 and two residual blocks with downsampling steps.

[0021] As an alternative implementation, the decoder of the VQGAN model includes two residual blocks with upsampling steps and a convolutional layer with a kernel size of 3.

[0022] As an alternative implementation, the nearest neighbor feature matching process is represented as: finding the feature most similar to the image feature in the encoding dictionary and replacing the original image feature.

[0023] Secondly, the present invention provides an image dehazing system based on a high-quality coding dictionary prior, comprising:

[0024] The training module is configured to synthesize a corresponding hazy image from a given sharp image after image degradation processing, and to form an image pair between the hazy image and the sharp image to train the dehazing network, which includes an encoder and a decoder.

[0025] The dehazing module is configured to use a trained dehazing network to obtain a clear image from the hazy image to be processed. Specifically, the encoder extracts image features from the hazy image to be processed, performs nearest neighbor feature matching between the image features and the encoding dictionary, and uses the nearest neighbor feature as a guide to perform dehazing processing based on the image features.

[0026] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0027] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] This invention proposes an image dehazing method and system based on a high-quality encoding dictionary prior. It re-formulates the degradation model of realistic hazy images, proposes a phenomenological degradation pipeline to simulate foggy images taken in the wild, and designs a novel image degradation processing pipeline to synthesize more realistic foggy images from given clear images. This significantly enhances the model's ability to learn dehazing, allowing the dehazing network to learn foggy images that are closer to reality, thus enabling it to better process real foggy images.

[0030] This invention proposes an image dehazing method and system based on high-quality coding dictionary priors. It utilizes an adversarial generative model based on coding dictionaries to learn high-quality coding dictionary priors and introduces these priors into the dehazing network. After performing nearest-neighbor feature matching between the features of the hazy image and the coding dictionary, the matched features are input into the decoder of the adversarial generative model to guide the decoder of the dehazing network to perform dehazing.

[0031] This invention proposes an image dehazing method and system based on a high-quality coding dictionary prior. In real image dehazing tasks, a high-quality coding dictionary is used to propose a controllable high-quality prior matching operation to bridge the gap between the synthetic domain and the real domain and produce an effect with adjustable enhancement intensity.

[0032] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0033] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0034] Figure 1 Here is a flowchart of the image dehazing method based on a high-quality coding dictionary prior provided in Embodiment 1 of the present invention;

[0035] Figure 2 (a)- Figure 2 (g) is a schematic diagram comparing the output results of the method of the present invention and other existing methods on the RTTS dataset.

[0036] Figure 3 (a)- Figure 3 (g) is a schematic diagram comparing the output results of the method of the present invention and other existing methods on the Fattal dataset. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0039] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0041] Example 1

[0042] This embodiment provides an image dehazing method based on a high-quality encoded dictionary prior, such as... Figure 1 As shown, it includes:

[0043] A given clear image is degraded and then synthesized into a corresponding hazy image. The hazy image and the clear image are paired to train the dehazing network, which includes an encoder and a decoder.

[0044] The trained dehazing network is used to obtain a clear image from the foggy image to be processed. Specifically, the encoder extracts image features from the foggy image to be processed, and performs nearest neighbor feature matching between the image features and the encoding dictionary. Guided by the nearest neighbor feature, the decoder performs dehazing processing based on the image features.

[0045] In this embodiment, the process of synthesizing a corresponding hazy image from a given clear image after image degradation processing includes:

[0046] (1) Adjust the gamma of the given clear image and add noise to simulate the insufficient lighting of the foggy image;

[0047] (2) Based on image depth, a transmission map of foggy weather is synthesized, and atmospheric colors with color bias are randomly synthesized. Fog images are synthesized according to the atmospheric scattering model.

[0048] (3) Add JPEG compression to form the final foggy image.

[0049] In this embodiment, a high-quality encoded dictionary prior is learned using a Vector-Quantized Generative Adversarial Network (VQGAN) model based on encoded dictionaries; specifically:

[0050] (1) Take the given foggy image as input and extract depth features through the encoder of VQGAN;

[0051] The encoder includes a convolutional layer with a kernel size of 3 and two residual blocks with downsampling steps.

[0052] (2) Match the deep features with the nearest neighbor features of the encoding dictionary to obtain the matched nearest neighbor features;

[0053] The encoding dictionary contains prior information about high-quality images. The dictionary has a capacity of 1024 and 512 channels. The nearest neighbor feature matching process is represented as follows:

[0054]

[0055] That is, features before matching In the encoding dictionary Find the nearest vector in And it was replaced by it.

[0056] (3) The nearest neighbor features obtained by matching are used with the decoder of VQGAN to obtain the reconstruction result. VQGAN is trained based on the reconstruction result, and the loss between the reconstruction result and the input foggy image is the optimization target.

[0057] The decoder includes two residual blocks with upsampling steps and a convolutional layer with a kernel size of 3.

[0058] In this embodiment, the process of dehazing the foggy image to be processed using the trained dehazing network includes:

[0059] First, the foggy image to be processed is encoded by the foggy image encoder to obtain image depth features that are easy to match a high-quality coding dictionary;

[0060] The encoder of the dehazing network includes a convolutional layer with a kernel size of 3, two residual blocks with downsampling steps, and four residual Transformer layers based on sliding windows.

[0061] Then, the extracted image depth features are matched with the nearest neighbor features of the trained VQGAN's encoded dictionary to obtain the nearest neighbor features;

[0062] In the inference phase, a controllable prior matching mechanism is designed to help the network match better priors through weights and generate a series of results whose enhancement level can be adjusted. This matching mechanism is expressed as:

[0063]

[0064] in The activation frequencies of the encoding dictionary on foggy and clear images.

[0065] Furthermore, the nearest neighbor features obtained from the matching are input into the decoder of the trained VQGAN, and the image depth features before matching are input into the decoder of the foggy image. The decoder of the trained VQGAN guides the decoder of the defogging network to perform defogging processing, and the decoder of the defogging network gradually eliminates the information loss caused by the encoding dictionary.

[0066] The decoder of the dehazing network consists of two residual blocks with upsampling steps and a convolutional layer with a kernel size of 3.

[0067] In this embodiment, the guidance process is completed by combining two decoders. Based on the characteristics of the decoders, the deformation in the decoder features of the trained VQGAN is corrected by a deformable convolutional layer, and the two are normalized to the same average value and then merged and input into the next feature processing layer.

[0068] In order to obtain high-quality results from high-quality priors, VQGAN needs to be trained on a large-scale dataset containing high-resolution and textured images. The datasets used in this embodiment are DIV2K and Flickr2K, which contain 4250 images to complete the pre-training of VQGAN. Both datasets are widely used for high-quality reconstruction tasks.

[0069] Meanwhile, the method of this embodiment is compared with five existing methods, including MSBDN (Multi-scale boosted dehazing network with dense feature fusion), Dehamer (Image dehazing transformer with transmission-aware 3D position embedding), PSD (Psd: Principled synthetic-to-real dehazing guided by physical priors), D4 (Self-augmented unpaired image dehazing via density and depth decomposition), and DAD (Domain adaptation for image dehazing). Specifically, the dehazing network is qualitatively and quantitatively evaluated on the RTTS dataset, which contains more than 4,000 real haze images with different scenes, resolutions, and degradation problems. The comparison results are as follows: Figure 2 (a)- Figure 2As shown in (g); the Fattal dataset, which includes 31 classic real-world haze cases, was used for further visual comparison, and the comparison results are as follows. Figure 3 (a)- Figure 3 As shown in (g).

[0070] Example 2

[0071] This embodiment provides an image dehazing system based on a high-quality encoded dictionary prior, including:

[0072] The training module is configured to synthesize a corresponding hazy image from a given sharp image after image degradation processing, and to form an image pair between the hazy image and the sharp image to train the dehazing network, which includes an encoder and a decoder.

[0073] The dehazing module is configured to use a trained dehazing network to obtain a clear image from the hazy image to be processed. Specifically, the encoder extracts image features from the hazy image to be processed, performs nearest neighbor feature matching between the image features and the encoding dictionary, and uses the nearest neighbor feature as a guide to perform dehazing processing based on the image features.

[0074] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0075] In further embodiments, the following is also provided:

[0076] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0077] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0078] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0079] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0080] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0081] Those skilled in the art will recognize that the units, i.e., algorithm steps, of the various examples described in connection with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0082] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An image dehazing method based on a high-quality encoding dictionary prior, characterized in that, include: A given clear image is degraded and then synthesized into a corresponding hazy image. The hazy image and the clear image are paired to train the dehazing network, which includes an encoder and a decoder. The trained dehazing network is used to obtain a clear image from the foggy image to be processed. Specifically, the encoder extracts image features from the foggy image to be processed, performs nearest neighbor feature matching between the image features and the encoding dictionary, and uses the nearest neighbor feature as a guide to perform dehazing processing based on the image features. The nearest neighbor feature matching uses a trained VQGAN model, and the process of dehazing the foggy image using a trained dehazing network specifically includes: The image depth features of the foggy image to be processed are obtained by the encoder of the defogging network; The image depth features are matched with the nearest neighbor features of the trained VQGAN's encoded dictionary to obtain the nearest neighbor features; The nearest neighbor features obtained from the matching are input into the decoder of the trained VQGAN, and the image depth features are input into the decoder of the dehazing network. The decoder of the trained VQGAN guides the decoder of the dehazing network to perform dehazing processing, and the decoder of the dehazing network gradually eliminates the information loss caused by the encoding dictionary.

2. The image dehazing method based on a high-quality coding dictionary prior as described in claim 1, characterized in that, Image degradation processing includes: The gamma adjustment is applied to a given sharp image, and noise is added to simulate insufficient lighting. The transmission map of foggy weather is synthesized based on image depth, and atmospheric colors with color bias are randomly synthesized. Fog images are synthesized according to atmospheric scattering models, and JPEG compression is added to form the final foggy image.

3. The image dehazing method based on a high-quality coding dictionary prior as described in claim 1, characterized in that, The encoder of the defogging network includes a convolutional layer with a kernel size of 3, two residual blocks with downsampling steps, and four residual Transformer layers based on sliding windows.

4. The image dehazing method based on a high-quality coding dictionary prior as described in claim 1, characterized in that, The decoder of the dehazing network includes two residual blocks with upsampling steps and a convolutional layer with a kernel size of 3.

5. The image dehazing method based on a high-quality coding dictionary prior as described in claim 1, characterized in that, The training process of the VQGAN model includes: The VQGAN model's encoder extracts image features from a given foggy image. The image features are matched with the nearest neighbor features of the encoding dictionary. The nearest neighbor features are then used to obtain the reconstruction result using the decoder of the VQGAN model. The VQGAN model is trained based on the reconstruction result, and the loss between the reconstruction result and the given foggy image is used as the optimization objective.

6. The image dehazing method based on a high-quality coding dictionary prior as described in claim 5, characterized in that, The encoder of the VQGAN model includes a convolutional layer with a kernel size of 3 and two residual blocks with downsampling steps; The decoder of the VQGAN model includes two residual blocks with upsampling steps and a convolutional layer with a kernel size of 3. The nearest neighbor feature matching process is represented as follows: find the feature that is most similar to the image feature in the encoding dictionary and replace the original image feature.

7. An image dehazing system based on a high-quality encoding dictionary prior, characterized in that, include: The training module is configured to synthesize a corresponding hazy image from a given sharp image after image degradation processing, and to form an image pair between the hazy image and the sharp image to train the dehazing network, which includes an encoder and a decoder. The dehazing module is configured to use a trained dehazing network to obtain a clear image from the hazy image to be processed. Specifically, the encoder extracts image features from the hazy image to be processed, performs nearest neighbor feature matching between the image features and the encoding dictionary, and uses the nearest neighbor feature as a guide to perform dehazing processing based on the image features. The nearest neighbor feature matching uses a trained VQGAN model, and the process of dehazing the foggy image using a trained dehazing network specifically includes: The image depth features of the foggy image to be processed are obtained by the encoder of the defogging network; The image depth features are matched with the nearest neighbor features of the trained VQGAN's encoded dictionary to obtain the nearest neighbor features; The nearest neighbor features obtained from the matching are input into the decoder of the trained VQGAN, and the image depth features are input into the decoder of the dehazing network. The decoder of the trained VQGAN guides the decoder of the dehazing network to perform dehazing processing, and the decoder of the dehazing network gradually eliminates the information loss caused by the encoding dictionary.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

Images