A Two-Branch Image Dehazing Method Based on Edge Constraints

Abstract

This invention discloses a two-branch image dehazing method based on edge constraints, comprising the following steps: Step 1: Constructing an image dehazing dataset; Step 2: Obtaining edge maps of the images using the Canny edge detection operator, and constructing a training sample using the edge maps, corresponding fog maps, and clear maps; Step 3: Randomly segmenting the images in the training samples as input to the network, and using data augmentation methods to expand the sample size; Step 4: Constructing a two-branch dehazing network based on edge constraints, including a dehazing subnetwork and an edge subnetwork; Step 5: Initializing network parameters and constructing a target loss function to optimize the parameters of the two-branch dehazing network; Step 6: Using the trained model to dehaze the test fog map to obtain a clear, fog-free image. This invention can utilize edge constraints to recover clear, fog-free images, solving the problem of lost spatial details and semantic inconsistencies in the repaired images during image dehazing tasks.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a method for dehazing bi-branch images based on edge constraints. Background Technology

[0002] Haze not only affects people's daily travel but also severely impacts the normal operation of scene monitoring devices, which in turn stimulates research into image dehazing technology in the field of image processing. Particles such as smoke, dust, water droplets, fog, and haze in the air absorb or scatter atmospheric light, resulting in unclear images captured by imaging devices. Researching image dehazing algorithms is beneficial for reducing the impact of foggy weather on people's lives and has significant research and economic value.

[0003] Traditional image dehazing techniques rely on atmospheric scattering models, using artificially discovered prior knowledge to infer atmospheric light and transmittance, thereby obtaining a clear, haze-free image. The Dark Channel Prior observes that in a local region of a haze-free image, at least one color channel has a very low pixel value. Using the Dark Channel Prior, transmittance and atmospheric light can be estimated, leading to the inference of the haze-free image. However, haze images containing sky or white areas severely affect the estimation of atmospheric light, thus impacting the quality of the haze-free image. The Color-Line Prior observes that small image patches exhibit a one-dimensional distribution in GRB space, which is beneficial for recovering a clear, haze-free image. In recent years, deep learning has made significant progress in image super-resolution, image denoising, and image enhancement. Inspired by this, deep learning has also been used to improve image dehazing. Deep learning-based image dehazing methods fall into two categories: one uses deep learning to learn atmospheric light and transmittance from large amounts of data, and the other uses deep learning techniques to directly learn the mapping from haze images to clear, haze-free images. While deep learning-based image dehazing methods can produce clear, fog-free images, the resulting fog-free images often lose image details and sharp edges. Summary of the Invention

[0004] To address the issue of lost spatial details and semantic inconsistencies in image dehazing, this invention provides a bi-branch image dehazing method based on edge constraints, comprising the following steps:

[0005] Step 1: Construct an image dehazing dataset, which contains multiple pairs of images: a hazy image and its corresponding clear image;

[0006] Step 2: Use the Canny edge detection operator to obtain the edge map of the image, and construct a training sample by combining the edge map and the corresponding fog map and clear map;

[0007] Step 3: The images in the training samples are randomly divided into 256×256 image blocks as input to the network, and random cropping, random flipping and random rotation are used as data augmentation methods to expand the sample size;

[0008] Step 4: Construct a dual-branch dehazing network based on edge constraints. This network consists of two sub-networks: a dehazing sub-network and an edge sub-network. The dehazing sub-network is used to repair the hazy image and obtain a clear, haze-free image. The edge sub-network is used to extract the edge features of the image. The edge features can help the dehazing sub-network repair the detailed contour information of the image, and at the same time provide a spatial distribution constraint, enabling the dual-branch dehazing network to restore a haze-free image with clear details and semantic consistency.

[0009] Step 5: Initialize network parameters and construct the objective loss function to optimize the parameters of the edge-constrained dual-branch defogging network;

[0010] Step 6: Use the trained model obtained in Step 5 to dehaze the test fog image and obtain a clear, fog-free image.

[0011] For details of each of the above steps, please refer to the embodiment.

[0012] The advantages and beneficial effects of this invention are as follows: It provides a bi-branch image dehazing method based on edge constraints, which can utilize edge constraints to recover a clear, haze-free image and solve the problem of lost spatial details and semantic inconsistency in the image dehazing task.

[0013] The main innovations of this invention are: (1) A dual-branch image dehazing network based on edge constraints is proposed to restore the details and contours of a clear image; the dual-branch network includes a dehazing sub-network and an edge sub-network; the dehazing sub-network is dedicated to restoring the hazy image and obtaining a clear hazy image; the edge sub-network is dedicated to extracting the edge features of the image, which can help the dehazing sub-network restore the details and contours of the image, and at the same time provide a spatial distribution constraint, so that the dual-branch dehazing network can restore a hazy image with clear details and consistent semantics. (2) An edge information fusion module is proposed to progressively extract edge information from the image. The edge information of an image can reflect the spatial distribution and semantic content of the image. The extracted edge information is input into the dehazing sub-network in the shallow, intermediate and deep layers respectively, progressively guiding the dehazing and detail restoration of the image, while maintaining the semantic consistency and spatial distribution consistency of the image content. The edge information fusion module contains two branches. One branch uses a 3×3 convolution and a sigmoid function to generate an edge attention map, emphasizing the consistency of spatial distribution and semantic consistency. The other branch contains a stitching operation and a 3×3 convolution to fuse the hazy image and the edge map of the image, preserving the detail information of the image. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the structure of the present invention;

[0015] Figure 2 This is a schematic diagram of the residual dual attention module in this invention;

[0016] Figure 3 This is a schematic diagram of the edge information fusion module in this invention. Detailed Implementation

[0017] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and examples. The following examples are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0018] This invention provides a method for dehazing bi-branch images based on edge constraints, comprising the following steps:

[0019] Step 1: Construct an image dehazing dataset, which contains multiple pairs of images: a hazy image and its corresponding clear image;

[0020] Step 2: Use the Canny edge detection operator to obtain the edge map of the image, and construct a training sample by combining the edge map and the corresponding fog map and clear map;

[0021] Step 3: The images in the training samples are randomly divided into 256×256 image blocks as input to the network, and random cropping, random flipping and random rotation are used as data augmentation methods to expand the sample size.

[0022] Step 4: Construct a dual-branch dehazing network based on edge constraints. This network consists of two sub-networks: a dehazing sub-network and an edge sub-network, as follows: Figure 1 As shown; the dehazing sub-network is used to repair the hazy image and obtain a clear, haze-free image; the edge sub-network is used to extract the edge features of the image. Edge features help the dehazing sub-network repair the detailed contour information of the image and provide a spatial distribution constraint, enabling the two-branch dehazing network to restore a haze-free image with clear details and semantic consistency; specifically:

[0023] The construction steps of the dehazing sub-network are as follows: The dehazing sub-network first includes a 3×3 convolution to increase the number of channels in the input image to 64. Next, two residual dual attention modules extract basic image features for the reconstruction of the clear image. Following this is an edge information fusion module, which uses the edge information obtained from the edge sub-network to guide the restoration of the fog image and the reconstruction of the clear image. Edge information can provide spatial distribution information and semantic information of the scene, which helps the network eliminate fog and other blurring, and also helps to maintain the spatial distribution consistency and semantic consistency of the reconstructed image. The two residual attention modules and the edge information fusion module are used as a basic dehazing unit module. The dehazing sub-network can contain multiple dehazing unit modules, but considering the model's performance and complexity, this invention sets the dehazing sub-network to contain 3 dehazing unit modules. At the back end of the dehazing sub-network, a 3×3 convolution module is used to generate a clear, fog-free image.

[0024] The construction steps of the edge subnetwork are as follows: The dehazing subnetwork first includes a 3×3 convolution to increase the number of channels in the network to 64; then, the residual dual attention module is also used as the basic convolutional unit of the edge subnetwork. This module has strong information extraction capabilities and can extract the contours and edge structures of the image; to be consistent with the dehazing subnetwork, the edge subnetwork also includes 3 basic convolutional unit modules; at the back end of the edge subnetwork, a 3×3 convolutional module is used to generate the edge map of the image; the edge subnetwork is used to extract the edge information of the image and input the edge information at different stages into the dehazing subnetwork to guide image dehazing and image restoration.

[0025] Furthermore, the residual dual attention module included in the construction steps of the dehazing sub-network and the edge sub-network is the basic module of the edge-constrained dual-branch dehazing network proposed in this invention, used to extract structural and semantic information of the image; the residual dual attention module includes a residual convolution module, a dual attention module, and a residual connection, as shown below. Figure 2 As shown;

[0026] The residual convolution module contains two 3×3 convolutions and a residual connection. After the first convolution module, a ReLU activation function is used to enhance the network's nonlinear fitting ability, which can be represented as follows:

[0027] Y=Conv3(Relu(Conv3(X)))+X,

[0028] Where X represents the input of the module, Y represents the output of the module, Conv3 represents a 3×3 convolution, and ReLU represents a non-linear activation function;

[0029] The dual attention module consists of a channel attention module and a spatial attention module;

[0030] Specifically, the channel attention module learns the responses between channels, emphasizing the importance of different channels in the feature map. It mainly consists of a global average pooling, two 3×3 convolutions, a ReLU activation function, and a Sigmoid activation function, which can be represented as follows:

[0031] CA=Sig(Conv3(Relu(Conv3(Pool(X))))),

[0032] Where CA represents the output of the channel attention module, X is the input of the channel attention module, Conv3 represents a 3×3 convolution, ReLU is a non-linear activation function, Sig is the sigmoid activation function, and Pool is global average pooling.

[0033] Specifically, the spatial attention module generates spatial weights in the feature map to emphasize the importance of different pixels in the feature map. It mainly consists of two 3×3 convolutions, a ReLU activation function, and a Sigmoid activation function, which can be represented as follows:

[0034] SA=Sig(Conv3(Relu(Conv3(X)))),

[0035] Where SA represents the output of the spatial attention module, X is the input of the spatial attention module, Conv3 represents a 3×3 convolution, ReLU is a non-linear activation function, and Sig is the sigmoid activation function;

[0036] Furthermore, the edge information fusion module included in the construction step of the dehazing sub-network is used to fuse the edge information extracted from the edge sub-network, guiding the dehazing and detail restoration of the dehazing sub-network. Specifically, the edge information fusion module contains two inputs, one from the dehazing sub-network and the other from the edge sub-network, such as... Figure 3 As shown, the edge information from the edge sub-network is first processed by a 3×3 convolution and a sigmoid function to generate an edge attention map. The edge attention map contains the spatial distribution information and semantic information of the image, which can constrain the spatial distribution consistency and semantic consistency in the dehazing channel. Then, the edge information is concatenated into the input of the dehazing sub-network, and the last 3×3 convolution is used to fuse the edge information and dehazing information of the image, which is beneficial to recovering the detail information of the image.

[0037] Step 5: Initialize network parameters and construct the objective loss function to optimize the parameters of the edge-constrained dual-branch defogging network;

[0038] Specifically, the network parameters are initialized using the Kaiming initialization method, and the Adam optimizer is used with an initial learning rate of 1×10⁻⁴. The cosine annealing algorithm is then used to stably reduce the learning rate to 1×10⁻⁵. The objective loss function consists of two parts: a reconstruction loss for the sharp image and a reconstruction loss for the edge image, specifically expressed as follows:

[0039] Loss=||GT img -P img ||1+||GT edge -P edge ||1,

[0040] Where Loss represents the loss of the two-branch defogging network, and GT img Represents a clear, fog-free image, GT edge P represents the truth margin map. img P represents the predicted fog-free image. edge The edge graph representing the prediction;

[0041] Step 6: Use the trained model obtained in Step 5 to dehaze the test fog image and obtain a clear, fog-free image.

[0042] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A bi-branch image dehazing method based on edge constraints, characterized in that, Includes the following steps: Step 1: Construct an image dehazing dataset, which contains multiple pairs of images: a hazy image and its corresponding clear image; Step 2: Use the Canny edge detection operator to obtain the edge map of the image, and construct a training sample by combining the edge map and the corresponding fog map and clear map; Step 3: The images in the training samples are randomly divided into 256×256 image blocks as input to the network, and random cropping, random flipping and random rotation are used as data augmentation methods to expand the sample size; Step 4: Construct a dual-branch dehazing network based on edge constraints. This network consists of two sub-networks: a dehazing sub-network and an edge sub-network. The dehazing sub-network is used to repair the hazy image and obtain a clear, haze-free image. The edge sub-network is used to extract the edge features of the image. The edge features can help the dehazing sub-network repair the detailed contour information of the image, and at the same time provide a spatial distribution constraint, enabling the dual-branch dehazing network to restore a haze-free image with clear details and semantic consistency. The steps for constructing the defogging subnetwork are as follows: The dehazing subnetwork first includes a 3×3 convolution to increase the number of channels in the input image to 64. Next, two residual dual-attention modules extract basic image features for reconstructing the clear image. Following this is an edge information fusion module, which uses edge information obtained from the edge subnetwork to guide the restoration of the foggy image and the reconstruction of the clear image. Edge information provides spatial distribution and semantic information about the scene, helping the network eliminate fog and other blurring, and also helping to maintain the spatial distribution and semantic consistency of the reconstructed image. The two residual attention modules and one edge information fusion module are used as a basic dehazing unit module. The dehazing subnetwork contains three dehazing unit modules. At the back end of the dehazing subnetwork, a 3×3 convolution module is used to generate a clear, fog-free image. Step 5: Initialize network parameters and construct the objective loss function to optimize the parameters of the edge-constrained dual-branch defogging network; The network parameters are initialized using the Kaiming initialization method, and the Adam optimizer is used with an initial learning rate of 1×10⁻⁶. -4 The cosine annealing algorithm stably reduces the learning rate to 1×10. -5 The target loss function consists of two parts: one part is the reconstruction loss of the sharp image, and the other part is the reconstruction loss of the edge image, as shown in the following formula: ， Where Loss represents the loss of the two-branch defogging network. This indicates a clear, fog-free image. Represents a truth boundary map. This represents the predicted fog-free image. The edge map represents the prediction; Step 6: Use the trained model obtained in Step 5 to dehaze the test fog image and obtain a clear, fog-free image.

2. The edge-constrained bi-branch image dehazing method according to claim 1, characterized in that, In step 4, the construction steps of the edge sub-network are as follows: The dehazing subnetwork first includes a 3×3 convolution to increase the network's channel count to 64; then, the residual dual attention module is also used as the basic convolutional unit of the edge subnetwork, which has strong information extraction capabilities and can extract the image's contours and edge structures; in accordance with the dehazing subnetwork, the edge subnetwork also includes 3 basic convolutional unit modules; at the back end of the edge subnetwork, a 3×3 convolutional module is used to generate the image's edge map.

3. The edge-constrained bi-branch image dehazing method according to claim 2, characterized in that, The residual dual attention module in the construction steps of the dehazing subnetwork and the edge subnetwork is used to extract structural and semantic information of the image; the residual dual attention module contains a residual convolution module, a dual attention module and a residual connection.

4. The edge-constrained bi-branch image dehazing method according to claim 3, characterized in that, The residual convolution module contains two 3×3 convolutions and a residual connection. A ReLU activation function follows the first convolution module to enhance the network's non-linear fitting capability, as shown in the following formula: ， Where X represents the module's input and Y represents the module's output. represents a 3×3 convolution, and ReLU represents a non-linear activation function.

5. The edge-constrained bi-branch image dehazing method according to claim 4, characterized in that, The dual attention module consists of a channel attention module and a spatial attention module; The channel attention module learns the responses between channels, emphasizing the importance of different channels in the feature map. It includes a global average pooling, two 3×3 convolutions, a ReLU activation function, and a Sigmoid activation function, as shown in the following formula: ， Where CA represents the output of the channel attention module, and X is the input of the channel attention module. This represents a 3×3 convolution, where ReLU is a non-linear activation function, Sig is the Sigmoid activation function, and Pool is global average pooling. The spatial attention module generates spatial weights in the feature map to emphasize the importance of different pixels. It mainly consists of two 3×3 convolutions, a ReLU activation function, and a Sigmoid activation function, as shown in the following formula: ， Where SA represents the output of the spatial attention module, and X is the input of the spatial attention module. This represents a 3×3 convolution, where ReLU is a non-linear activation function and Sig is the Sigmoid activation function.

6. The edge-constrained bi-branch image dehazing method according to claim 1, characterized in that, In the construction steps of the defogging subnetwork: The edge information fusion module contains two inputs: one from the dehazing sub-network and the other from the edge sub-network. The edge information from the edge sub-network is first processed by a 3×3 convolution and a sigmoid function to generate an edge attention map. The edge attention map contains spatial distribution and semantic information of the image, which can constrain the spatial distribution consistency and semantic consistency in the dehazing channel. Then, the edge information is concatenated to the input of the dehazing sub-network, and the last 3×3 convolution is used to fuse the edge information and dehazing information of the image, which is beneficial for restoring the image's detailed information.

Images