Double-branch image defogging algorithm based on multiple loss constraints

Abstract

The invention discloses a double-branch image defogging algorithm based on multiple loss constraints. The double-branch image defogging algorithm comprises a double-branch structure design and a triple loss constraint design. In the double-branch structure, the first branch is used for depth estimation, extracting depth information of the image and ensuring the geometric structure consistency of the defogged image; and the second branch is responsible for generating and processing a fog graph and simulating variable fog conditions in an actual scene. And estimating a transmission image and transmissivity through an atmospheric scattering model, and finally obtaining a defogged image. The triple loss constraints comprise image reconstruction loss, depth consistency loss and fog concentration estimation loss, the expressions of the defogged image in color, geometric structure and fog concentration estimation are respectively constrained, and the image quality and robustness are improved. Through triple loss constraint design, the common problems of color distortion and detail loss in the existing method are solved, it is ensured that the defogged image is consistent with the original clear image in geometric structure, and geometric distortion in the defogging process is avoided.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision and image processing, and in particular to a dual-branch image defogging algorithm based on multiple loss constraints. Background Art

[0002] In haze weather conditions, the contrast and clarity of images are usually severely reduced due to the scattering and absorption effects caused by particulate matter in the atmosphere (such as water droplets, dust, etc.). This effect not only reduces the visual quality of the image, but also has a negative impact on subsequent image analysis and computer vision tasks (such as object detection, recognition and tracking, etc.). Therefore, image dehazing technology has become a hot topic in recent years.

[0003] Traditional image dehazing methods can be roughly divided into two categories: image enhancement-based methods and physical model-based methods. The former improves visual effects by enhancing image contrast and details, but usually cannot accurately restore the real scene information of the image; the latter uses the atmospheric scattering model to reversely derive a haze-free image. Although it is more reliable in theory, its actual effect is often limited by inaccurate model assumptions and difficult parameter estimation.

[0004] With the development of deep learning, more and more researchers have begun to try to solve the image dehazing problem through deep learning models such as convolutional neural networks (CNN). These methods are usually trained with a large amount of labeled data, so that the model can learn complex nonlinear mapping relationships, so as to better restore the haze-free image. However, the existing deep learning methods still have some shortcomings, which are mainly manifested in the following aspects:

[0005] (1) Existing dehazing algorithms sometimes lead to the loss of image details, especially color distortion and blurring in the dehazed images;

[0006] (2) The depth information in the dehazing process is often not well preserved, which affects the geometric consistency of the image and leads to errors in the perception of three-dimensional scenes;

[0007] (3) Due to the complex changes in fog concentration, existing methods find it difficult to accurately estimate and compensate for fog of different concentrations, which in turn affects the consistency of the defogging effect. Summary of the invention

[0008] The technical problem to be solved by the present invention is to overcome the above technical defects and provide a dual-branch image defogging algorithm based on multiple loss constraints.

[0009] In order to solve the above problems, the technical solution of the present invention comprises the following steps:

[0010] S1, input a clear fog-free image;

[0011] S2. Processing the clear image through a depth estimation branch to obtain a depth map of the image;

[0012] S3, generating a simulated foggy image by adding fog of different concentrations to the clear image;

[0013] S4, using the transmission map estimation network and the atmospheric light estimation module to estimate the transmission map and the atmospheric light value respectively;

[0014] S5, combining the transmission map and the atmospheric light value, and restoring the fog-free image through the atmospheric scattering model;

[0015] S6. Design image reconstruction loss, depth consistency loss and fog concentration estimation loss, and train the network by optimizing these loss functions.

[0016] Furthermore, the foggy image generation module generates a simulated foggy image by adjusting the fog concentration coefficient of the clear image.

[0017] Furthermore, the atmospheric light estimation module adopts a UNet structure to estimate the atmospheric light value from the foggy image.

[0018] Furthermore, the depth consistency loss is used to compare the depth map of the dehazed image with the depth map initially estimated from the clear image.

[0019] Furthermore, the network training process is performed by minimizing a total loss function, which combines image reconstruction loss, depth consistency loss, and fog concentration estimation loss to optimize the dehazing effect.

[0020] Furthermore, the depth estimation branch includes a depth estimation network, which is used to receive the clear image as input and output a corresponding depth map.

[0021] Furthermore, the transmission map estimation network adopts the Swin Transformer architecture to capture the local and global features of the foggy image and estimate the transmission map.

[0022] Furthermore, the image reconstruction loss is used to constrain the defogged image to be consistent with the real image in color and details by comparing the defogged image with the synthesized foggy image.

[0023] Further, the fog concentration estimation loss is used to compare the fog concentration when generating the fog map with the fog concentration coefficient estimated from the defogged image.

[0024] The advantages of the present invention compared with the prior art are:

[0025] 1. The present invention provides a dual-branch image defogging algorithm based on multiple loss constraints. In the defogging process, an image reconstruction loss is designed to ensure that the defogging image is consistent with the original foggy image in color and detail. This innovation solves the common color distortion and detail loss problems in existing methods. The depth consistency loss is introduced to ensure that the defogging image is consistent with the original clear image in terms of geometry. By constraining the depth before and after defogging, the image reconstruction loss is used to ensure that the defogging image is consistent with the original clear image in terms of geometry. Figure 1 The present invention effectively avoids geometric distortion in the defogging process, thereby improving the authenticity of the defogging image. By simulating and estimating fog of different concentrations and introducing fog concentration estimation loss to ensure the accuracy of fog concentration, this innovation enables the network to adapt to foggy conditions of different concentrations, thereby improving the robustness and generalization ability of the defogging effect. By combining the three loss functions, the present invention realizes the joint optimization of multi-dimensional features in the image defogging process. This multiple loss constraint not only improves the comprehensive performance of the defogging effect, but also ensures the balance between color, depth and fog concentration, thereby achieving higher image quality.

[0026] 2. The present invention provides a dual-branch image defogging algorithm based on multiple loss constraints. In the transmission map estimation network, the present invention innovatively adopts the Swin Transformer architecture to replace the traditional convolutional neural network (CNN) structure. Specifically, the network uses 6 Patch Merging and Swin Transformer Blocks to process foggy images to better capture global and local features. Compared with the traditional convolution layer, Swin Transformer has stronger feature expression ability and adaptability, and can more accurately estimate the transmission map t in complex scenes, thereby improving the defogging effect. Swin Transformer organically combines local information and global information through the attention mechanism of the sliding window (Window), significantly improving the network's ability to process multi-scale information. This design is particularly suitable for processing complex details in foggy images, making the estimation of the transmission map more accurate, which helps to restore clearer and more realistic defogging images;

[0027] 3. The present invention provides a dual-branch image dehazing algorithm based on multiple loss constraints. Since the present invention comprehensively considers the change of fog concentration, the preservation of geometric depth information and the restoration of color details, the network shows high robustness and adaptability under different foggy conditions. This makes the dehazing algorithm widely applicable in practical applications and can cope with various complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flowchart of a dual-branch image defogging algorithm based on multiple loss constraints. DETAILED DESCRIPTION

[0029] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the appended claims.

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0031] A dual-branch image dehazing algorithm based on multiple loss constraints includes the following steps:

[0032] S1, input a clear fog-free image;

[0033] S2. Process the clear image through the depth estimation branch to obtain a depth map of the image;

[0034] S3, generating a simulated foggy image by adding fog of different concentrations to the clear image;

[0035] S4, using the transmission map estimation network and the atmospheric light estimation module to estimate the transmission map and the atmospheric light value respectively;

[0036] S5, combining the transmission map and the atmospheric light value, and restoring the fog-free image through the atmospheric scattering model;

[0037] S6. Design image reconstruction loss, depth consistency loss and fog concentration estimation loss, and train the network by optimizing these loss functions.

[0038] Furthermore, the depth estimation branch includes a depth estimation network, which is used to receive the clear image as input and output a corresponding depth map.

[0039] Furthermore, the foggy image generation module generates a simulated foggy image by adjusting the fog concentration coefficient of the clear image.

[0040] Furthermore, the transmission map estimation network adopts the Swin Transformer architecture to capture the local and global features of foggy images and estimate the transmission map.

[0041] Furthermore, the atmospheric light estimation module adopts the UNet structure to estimate the atmospheric light value from the foggy image.

[0042] Furthermore, the image reconstruction loss is used to constrain the dehazed image to be consistent with the real image in color and details by comparing the dehazed image with the synthesized foggy image.

[0043] Furthermore, a depth consistency loss is used to compare the depth map of the dehazed image with the depth map originally estimated from the clean image.

[0044] Furthermore, the fog concentration estimation loss is used to compare the fog concentration when generating the fog map with the fog concentration coefficient estimated from the defogged image.

[0045] Furthermore, the network training process is carried out by minimizing the total loss function, which combines the image reconstruction loss, depth consistency loss and fog concentration estimation loss to optimize the dehazing effect.

[0046] In specific use, according to Figure 1 As shown in , the input of the network is a clear fog-free image, denoted as . This image will serve as the basis for subsequent processing and perform multi-branch processing to generate synthetic foggy images and related estimation results.

[0047] This branch receives the clear image I clear As input, a depth estimation network (such as a convolutional neural network) is used to estimate the depth map of the image, denoted as D clear ,The function of the depth estimation branch is to extract the depth information of the image, which is an important geometric feature in the defogging process.,By comparing the depth information before and after defogging, the geometric structure of the defogging image can be ensured to be consistent with the original image, avoiding geometric distortion in the defogging process.

[0048] The second branch is from the clear image I clear At the beginning, different concentration coefficients β1 are applied to the image to simulate different concentrations of fog and generate synthetic foggy images, denoted as I hazy .

[0049] Synthesized foggy image I hazy After being input into the transmission map estimation network, the image will enter multiple layers of the network for feature extraction and transmission map estimation after preliminary processing.

[0050] At the same time, the foggy image is also input into the atmospheric light estimation module (for example, implemented by the UNet structure) to estimate the atmospheric light value A.

[0051] Using the transmission map t and atmospheric light A, the haze-free image J is restored through the atmospheric scattering model. dehazed The formula of the atmospheric scattering model is:

[0052]

[0053] This formula combines the transmission map t and the atmospheric light A to remove haze and restore a clear image.

[0054] Comparison of dehazed imagesJ dehazed Compared with the original clear image I clear , which is used to constrain the color and details of the dehazed image to be consistent with the original foggy image. The formula is:

[0055] Loss1=||J dehazed -I clear ||.

[0056] The dehazed image J dehazed Input the depth estimation network again to generate the defogging depth map D dehazed , and then compared with the depth map D of the original clear image clear Compare to ensure the consistency of depth information. The formula is:

[0057] Loss2=||D dehazed -D clear ||.

[0058] pass The fog concentration coefficient β2 is calculated and compared with the concentration coefficient β1 used when generating foggy images to ensure accurate estimation of fog concentration. The formula is:

[0059] Loss3=||β2-β1||.

[0060] The final total loss function Loss total =Loss1+Loss2+Loss3 combines the above three losses and is used to train the entire network to obtain the optimal dehazing effect.

[0061] The network training process minimizes the total loss function Loss total During the training process, the network learns how to accurately estimate the transmission map, transmittance, and atmospheric light under various fog conditions, thereby effectively removing fog and restoring clear images. By optimizing the depth consistency loss and fog concentration estimation loss, the network can not only generate dehazed images with good visual effects, but also ensure the accuracy of the geometric structure and the ability to adapt to different fog concentrations.

[0062] Embodiment 1

[0063] The network structure includes an input module, a dual-branch structure, an atmospheric scattering model, a triple loss constraint module, and a network training and optimization part.

[0064] A specific depth estimation module is used to extract the depth information of the input clear image. For example, a pre-trained ResNet-50 model is used as a basis, and several additional convolutional layers are added for fine-tuning to obtain a more accurate depth map.

[0065] Different concentrations of fog are added to the input clear images (for example, the fog concentration value β is set to 0.1, 0.3, 0.5) to generate simulated foggy images. These images are then processed using the Swin Transformer architecture, which contains 6 PatchMerging layers and Swin Transformer Blocks to estimate the transmission map and transmittance.

[0066] Image reconstruction loss (Loss1): Calculate the difference between the dehazed image and the original clear image, using the mean square error (MSE) as the loss function. For example, for an image of size H×W, the loss function can be defined as:

[0067] Loss1=H×W1 i＝1 ∑ Hj＝1 ∑ W (I dehazed (i,j)-I clear (i,j))2

[0068] Among them J dehazed represents the image after dehazing, I clear Indicates a clear image.

[0069] By calculating the difference between the depth map of the dehazed image and the depth map of the clear image, MSE is also used to measure the difference. For example, for the depth map D dehazed and D clear , the loss function can be expressed as:

[0070] Loss2=H×W1 i＝1 ∑ Hj＝1 ∑ W (D dehazed (i,j)-D clear (i,j))2

[0071] Calculate the difference between the fog density β used when generating the fog map and the fog density estimated from the dehazed image. If the estimated fog density is Then the loss function can be defined as:

[0072] Loss3=(β-β^)2

[0073] During the training process, Loss1, Loss2, and Loss3 are combined with certain weights to form the total loss function. For example, it can be set as:

[0074] Total_Loss=w1·Loss1+w2·Loss2+w3·Loss3

[0075] Among them, w1, w2, and w3 are pre-set weights, which can be adjusted according to experiments to achieve the best dehazing effect.

[0076] The Adam optimizer is used to minimize Total_Loss and the network parameters are iteratively updated through batch training data. The training dataset consists of pairs of clear images and their corresponding foggy images, with each batch containing 64 images.

[0077] After training is completed, the model performance is evaluated on the validation set, and the model parameters or the weights of the loss function are further adjusted based on the results.

[0078] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0079] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

[0080] The present invention and its embodiments are described above, which is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it and do not deviate from the purpose of the invention, they can creatively design a structure and embodiment similar to the technical solution, which should fall within the protection scope of the present invention.

Claims

1. A dual-branch image dehazing algorithm based on multiple loss constraints, characterized in that: The following steps are involved: S1, input a clear fog-free image; S2. Processing the clear image through a depth estimation branch to obtain a depth map of the image; S3, generating a simulated foggy image by adding fog of different concentrations to the clear image; S4, using the transmission map estimation network and the atmospheric light estimation module to estimate the transmission map and the atmospheric light value respectively; S5, combining the transmission map and the atmospheric light value, and restoring the fog-free image through the atmospheric scattering model; S6. Design image reconstruction loss, depth consistency loss and fog concentration estimation loss, and train the network by optimizing these loss functions.

2. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The depth estimation branch includes a depth estimation network, which is used to receive a clear image as input and output a corresponding depth map.

3. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The foggy image generation module generates a simulated foggy image by adjusting the fog concentration coefficient of the clear image.

4. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The transmission map estimation network adopts the Swin Transformer architecture to capture the local and global features of foggy images and estimate the transmission map.

5. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The atmospheric light estimation module adopts a UNet structure and is used to estimate the atmospheric light value from foggy images.

6. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The image reconstruction loss is used to constrain the defogging image to be consistent with the real image in color and details by comparing the defogging image with the synthesized foggy image.

7. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The depth consistency loss is used to compare the depth map of the dehazed image with the depth map originally estimated from the clean image.

8. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The fog concentration estimation loss is used to compare the fog concentration when generating the fog map with the fog concentration coefficient estimated from the defogged image.

9. The dual-branch image dehazing algorithm based on multiple loss constraints according to claim 1, characterized in that: The network training process is performed by minimizing the total loss function, which combines image reconstruction loss, depth consistency loss and fog concentration estimation loss to optimize the dehazing effect.

Images