An image dehazing method, system, and embedded device

Abstract

This invention discloses an image dehazing method, system, and embedded device. The dehazing method of this invention adopts a two-branch dehazing model. The first branch of the dehazing model preserves the edge details of the hazy image by constructing bilateral grid data, thereby improving the clarity of the restored image. The fusion module fuses the bilateral grid data output by the first branch of the dehazing model and the depth information output by the second branch of the dehazing model, thereby improving the model's dehazing capability and effectively solving problems such as color distortion and incomplete dehazing.

Description

Technical Field

[0001] This invention relates to an image dehazing method, system, and embedded device, belonging to the field of image processing. Background Technology

[0002] In smoggy weather, the performance of monitoring systems, intelligent driving systems, and other similar systems can be significantly affected, and in some cases, they may even fail to function properly. Therefore, to improve the stability of monitoring systems and intelligent driving systems in smoggy weather, it is necessary to perform defogging processing on images acquired during such conditions.

[0003] In recent years, due to the rapid development of computer hardware and the continuous improvement of computing power, deep learning has also developed rapidly. However, current dehazing methods based on deep learning suffer from color distortion and incomplete dehazing. Summary of the Invention

[0004] This invention provides an image dehazing method, system, and embedded device, which solves the problems disclosed in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] An image dehazing method includes:

[0007] Capture images under hazy weather conditions;

[0008] Images of hazy weather are input into a pre-trained dehazing model to obtain a haze-free image. The dehazing model includes a first branch, a second branch, and a fusion module. The first branch extracts features from the input image and uses the extracted features to construct bilateral grid data containing image spatial information and pixel value information. The second branch extracts depth information from the input image. The fusion module fuses the bilateral grid data and depth information to output a haze-free image.

[0009] The first branch of the dehazing model uses the Unet network. The Unet network extracts shallow and deep features from the input image, fuses the shallow and deep features through skip connections, and performs affine transformation on the fused features to construct bilateral grid data.

[0010] The second branch of the dehazing model includes an encoder and a decoder. The encoder uses residual blocks to downsample the input image three times to obtain three feature maps of different sizes. The decoder uses a loop decoding method to decode the three feature maps of different sizes to obtain depth information.

[0011] The dehazing model is trained using contrastive learning.

[0012] An image dehazing system, comprising:

[0013] The acquisition module is used to acquire images under hazy weather conditions;

[0014] The dehazing module is used to input images from hazy weather into a pre-trained dehazing model to obtain a haze-free image. The dehazing model includes a first branch, a second branch, and a fusion module. The first branch extracts features from the input image and uses the extracted features to construct bilateral grid data containing image spatial information and pixel value information. The second branch extracts depth information from the input image. The fusion module fuses the bilateral grid data and depth information to output a haze-free image.

[0015] In the dehazing module, the first branch of the dehazing model uses the Unet network. The Unet network extracts shallow and deep features from the input image, fuses the shallow and deep features through skip connections, performs affine transformation on the fused features, and constructs bilateral grid data.

[0016] In the dehazing module, the second branch of the dehazing model includes an encoder and a decoder. The encoder uses residual blocks to downsample the input image three times to obtain three feature maps of different sizes. The decoder uses a loop decoding method to decode the three feature maps of different sizes to obtain depth information.

[0017] The dehazing model in the dehazing module is trained using contrastive learning.

[0018] The present invention also discloses an embedded device including one or more processors and one or more memories, wherein one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing an image dehazing method.

[0019] The beneficial effects achieved by this invention are as follows: The dehazing method of this invention adopts a two-branch dehazing model. The first branch of the dehazing model preserves the edge details of the hazy image by constructing bilateral grid data, thereby improving the clarity of the image restoration. The fusion module fuses the bilateral grid data output by the first branch of the dehazing model and the depth information output by the second branch of the dehazing model, thereby improving the dehazing capability of the model and effectively solving the problems of color distortion and incomplete dehazing. Attached Figure Description

[0020] Figure 1 A flowchart of an image dehazing method;

[0021] Figure 2 Flowchart for training the dehazing model;

[0022] Figure 3Images of foggy weather that have not been processed by image dehazing methods;

[0023] Figure 4 This is a foggy image after image dehazing. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0025] like Figure 1 As shown, an image dehazing method includes the following steps:

[0026] Step 1: Collect images under hazy weather conditions.

[0027] Step 2: Input the image under hazy weather into the pre-trained dehazing model to obtain a haze-free image. The dehazing model includes a first branch, a second branch, and a fusion module. The first branch extracts features from the input image and uses the extracted features to construct bilateral grid data containing image spatial information and pixel value information. The second branch extracts depth information from the input image. The fusion module fuses the bilateral grid data and depth information to output a haze-free image.

[0028] The above method adopts a two-branch dehazing model. The first branch of the dehazing model preserves the edge details of the hazy image by constructing bilateral grid data, thereby improving the clarity of the image restoration. The fusion module fuses the bilateral grid data output by the first branch of the dehazing model and the depth information output by the second branch of the dehazing model, thereby improving the model's dehazing ability and effectively solving problems such as color distortion and incomplete dehazing.

[0029] The aforementioned dehazing model is a neural network model that needs to be pre-built and trained, as detailed below. Figure 2 As shown, the process includes, in sequence, constructing paired datasets, building a neural network model, training the model, validating the model's performance, evaluating image quality, and finally obtaining the optimal dehazing model.

[0030] The elements in the paired dataset are pairs of images. These images can be acquired by multiple cameras in multiple scenes. The paired images include foggy images and fog-free images. The two images are taken by different cameras at nearby time points under different weather conditions, specifically in foggy and sunny weather. For example, the foggy image of scene A is the image of scene A taken at 8 o'clock on a foggy day, and the corresponding fog-free image is the image of scene A taken at 8 o'clock or 8:01 o'clock on a sunny day (the time difference between the foggy image and the image taken is no more than 2 minutes).

[0031] Before constructing paired datasets from the collected images, the images need to be screened for quality. Specifically, images with problems such as blurriness are removed, and high-quality paired datasets are constructed from the remaining images.

[0032] In order to improve the dehazing capability and the clarity of image restoration, as an embodiment of the present invention, the above-mentioned dehazing model adopts a dual-branch structure, specifically including a first branch of the dehazing model, a second branch of the dehazing model, and a fusion module.

[0033] Because existing methods often result in blurred edge information after image dehazing, the first branch of the dehazing model in this invention specifically employs the Unet network. The Unet network extracts shallow and deep features (features extracted from deep network layers) of the input image, fuses these features through skip connections, and performs an affine transformation on the fused features to construct bilateral grid data (i.e., the fused features are used as the input for the affine transformation). This is based on the edge-preserving capability of bilateral filtering to construct bilateral grid data with a bilateral filtering effect. The bilateral grid data is a three-dimensional tensor, obtained by introducing a third pixel value dimension into a tensor representing a two-dimensional spatial location. Specifically, the features are converted to a data format to obtain bilateral grid data containing image spatial information and pixel value information. The edge-preserving capability of the bilateral grid data is then used to retain edge details in the hazy image, improving the clarity of the restored image.

[0034] The second branch of the dehazing model uses a depth information extraction network, which includes an encoder and a decoder. The encoder uses residual blocks to downsample the input image three times to obtain three feature maps of different sizes. The decoder uses a cyclic decoding method to decode the three feature maps of different sizes to obtain depth information. The depth information represents the distance between the pixel and the camera.

[0035] The encoder in the second branch of the dehazing model differs from most feature extraction methods that use the full ResNet. Instead, the encoder here only uses the first three layers of ResNet as the feature extraction network. This ensures that shallow features of the image (such as contours and colors) are not destroyed, which is more conducive to generating depth information. In the decoder part, gated recurrent units are used for feature decoding, which can effectively avoid the gradient vanishing problem during network training. Compared with the currently used long short-term memory network, the gated recurrent unit has a simpler structure, fewer parameters, and is more suitable for building larger networks.

[0036] The fusion module combines bilateral mesh data and depth information to output a haze-free image, thus obtaining the final haze-free image. The fusion module combines the bilateral mesh data output from the first branch of the dehazing model with the depth information output from the second branch, improving the model's dehazing capability. Using this model effectively solves problems such as color distortion and incomplete dehazing.

[0037] To improve the dehazing model's ability to understand higher-level features, as an embodiment of the present invention, the aforementioned dehazing model is trained using contrastive learning. Contrastive learning allows the model-processed images to be closer to positive samples and further away from negative samples; specifically, during training, fog-free images from clear weather are used as positive samples, while foggy images are used as negative samples, thereby improving the model's dehazing capability.

[0038] The above paired datasets are divided into training samples and test samples. The training process for the above dehazing model using the training samples can be as follows:

[0039] 1) Fix the image size to a uniform 640*480. During training, increase the batch size to improve training efficiency.

[0040] 2) Perform normalization preprocessing on the images to remove the statistical mean in the image data, making each pair of training data more differentiated;

[0041] 3) Convert the image format and replace complex exponential operations with multiplication operations to improve training efficiency;

[0042] 4) Input the processed paired images into the constructed neural network model for training.

[0043] Model training involves multiple rounds, a continuous process of finding the optimal solution. Therefore, at the end of each round, the weight parameters of the current model are saved. The more training iterations there are, the more weight parameters there will be. Different weight parameters will lead to different model performance. To obtain the optimal model, test samples are used to validate the model, as follows:

[0044] S1) Input the foggy image into the trained dehazing model to obtain the processed fog-free image;

[0045] S2) Use objective evaluation metrics to evaluate the images processed by the model, and select the model with the highest score as the training result; among them, the objective evaluation metrics used include: peak signal-to-noise ratio for measuring image quality and metrics for measuring the structural similarity of attributes such as brightness between two images.

[0046] The trained dehazing model is then used for image dehazing, specifically for images taken during hazy weather (see...). Figure 3 Input a pre-trained dehazing model to obtain a haze-free image, see... Figure 4 This invention combines depth information and a bilateral grid for image dehazing, resulting in higher quality, haze-free images.

[0047] Based on the same technical solution, this invention also discloses a software system corresponding to the above method, an image dehazing system, comprising:

[0048] The acquisition module is used to acquire images under hazy weather conditions.

[0049] The dehazing module is used to input images from hazy weather into a pre-trained dehazing model to obtain a haze-free image. The dehazing model includes a first branch, a second branch, and a fusion module. The first branch extracts features from the input image and uses the extracted features to construct bilateral grid data containing image spatial information and pixel value information. The second branch extracts depth information from the input image. The fusion module fuses the bilateral grid data and depth information to output a haze-free image.

[0050] The dehazing model in the dehazing module is trained using contrastive learning. The first branch of the dehazing model employs a Unet network, which extracts shallow and deep features from the input image. These features are then fused through skip connections, and an affine transformation is performed on the fused features to construct a bilateral grid. The second branch of the dehazing model includes an encoder and a decoder. The encoder uses residual blocks to downsample the input image three times, obtaining three feature maps of different sizes. The decoder uses a cyclic decoding method to decode these three feature maps to obtain depth information.

[0051] The above system adopts a two-branch dehazing model. The first branch of the dehazing model preserves the edge details of the hazy image by constructing bilateral grid data, thereby improving the clarity of the image restoration. The fusion module fuses the bilateral grid data output by the first branch of the dehazing model and the depth information output by the second branch of the dehazing model, thereby improving the model's dehazing ability and effectively solving problems such as color distortion and incomplete dehazing.

[0052] Based on the same technical solution, the present invention also discloses an embedded device, including one or more processors and one or more memories, wherein one or more programs are stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing an image dehazing method.

[0053] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or embedded program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of an embedded device program product implemented on one or more embedded device-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing embedded device-usable program code.

[0054] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0055] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0056] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0057] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval.

Claims

1. An image defogging method, characterized by, include: Capture images under hazy weather conditions; Images of hazy weather are input into a pre-trained dehazing model to obtain a haze-free image. The dehazing model includes a first branch, a second branch, and a fusion module. The first branch extracts features from the input image and uses the extracted features to construct bilateral grid data containing image spatial information and pixel value information. The second branch extracts depth information from the input image. The fusion module fuses the bilateral grid data and depth information to output a haze-free image. The first branch uses a Unet network, which extracts shallow and deep features from the input image, fuses the shallow and deep features through skip connections, and performs affine transformations on the fused features to construct bilateral grid data. The second branch of the dehazing model includes an encoder and a decoder. The encoder uses residual blocks to downsample the input image three times to obtain three feature maps of different sizes. The decoder uses a loop decoding method to decode the three feature maps of different sizes to obtain depth information.

2. The image defogging method of claim 1, wherein, The dehazing model is trained using contrastive learning.

3. An image defogging system, characterized by, include: The acquisition module is used to acquire images under hazy weather conditions; The dehazing module is used to input images of hazy weather into a pre-trained dehazing model to obtain haze-free images. The dehazing model includes a first branch, a second branch, and a fusion module. The first branch extracts features from the input image and uses the extracted features to construct bilateral grid data containing image spatial information and pixel value information. The second branch extracts depth information from the input image. The fusion module fuses the bilateral grid data and depth information to output a haze-free image. The first branch of the dehazing model uses a Unet network, which extracts shallow and deep features from the input image, fuses the shallow and deep features through skip connections, and performs affine transformations on the fused features to construct bilateral grid data. The second branch of the dehazing model includes an encoder and a decoder. The encoder uses residual blocks to downsample the input image three times to obtain three feature maps of different sizes. The decoder uses a loop decoding method to decode the three feature maps of different sizes to obtain depth information.

4. The image defogging system of claim 3, wherein, The dehazing model in the dehazing module is trained using contrastive learning.

5. An embedded device, characterized in that, include: One or more processors and one or more memories, one or more programs stored in the one or more memories and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods according to claims 1 to 2.

Images