Image defogging method based on hybrid architecture

Abstract

The present application belongs to the technical field of computer vision and image processing, and relates to an image defogging method based on a hybrid architecture, which is especially suitable for image defogging enhancement in real foggy scenes, and improves the defogging effect and adaptability of the method by jointly optimizing coarse-grained and fine-grained features through a deep learning model. The method comprises the following steps: coarse-grained features of an input image are extracted using a convolution structure; fine-grained features of the input image are extracted using a self-attention structure; the fine-grained and coarse-grained features are fused to obtain an enhanced defogging image; the above steps constitute a hybrid architecture defogging network; and the hybrid architecture defogging network learns parameters for defogging according to a designed loss function set.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, and relates to an image dehazing method based on a hybrid architecture, which is particularly suitable for image dehazing enhancement in real foggy scenes. It improves the dehazing effect and the adaptability of the method by jointly optimizing coarse-grained and fine-grained features through a deep learning model. Background Technology

[0002] Image dehazing is an important research area in computer vision, aiming to recover clear original scenes from hazy images affected by atmospheric scattering. Haze, as a common atmospheric degradation phenomenon, not only affects people's daily lives but also seriously interferes with the operation of machine vision systems.

[0003] From a human visual perspective, the presence of haze significantly reduces visibility, limiting people's ability to obtain information about their surroundings. In severe haze conditions, it can even lead to traffic accidents. For computer vision systems, haze degrades image quality, impacting the reliability of advanced vision tasks such as object detection and autonomous driving, and causing a range of misjudgments. Therefore, developing effective dehazing algorithms has significant theoretical and practical value. In fields such as smart city construction, autonomous driving, aerial remote sensing, and video surveillance, clear, haze-free images are a prerequisite for reliable system operation. Particularly in critical applications such as military reconnaissance and disaster relief, image dehazing technology can help extract valuable information from hazy images acquired under adverse weather conditions, providing support for decision-making.

[0004] Current image dehazing methods are mainly divided into two categories: physics-based dehazing methods and deep learning-based dehazing methods. Physics-based methods have achieved good results in specific scenarios, but due to the complexity of real-world haze scenes, these priors are not always valid. When the prior conditions are not met, artifacts such as halos and color distortion often occur in the dehazing results. Furthermore, these methods typically require complex optimization processes, resulting in low computational efficiency and making them difficult to meet the needs of real-time applications.

[0005] With the development of deep learning technology, data-driven deep learning dehazing methods have begun to emerge. Convolutional neural networks (CNNs) can extract overall information by utilizing inductive biases of images, but they lack the acquisition of detailed components. The newly emerging multi-head self-attention architecture (Transformer) performs multi-head self-attention computation on image patches, which is beneficial for extracting detailed information of images, but due to its large computational cost, it cannot meet the requirements of real-time processing.

[0006] Paper: GridDehazeNet: Attention-based multi-scale network for image dehazing, Conference: Computer Vision and Pattern Recognition (CVPR), Year: 2019. Liu et al. proposed an attention-based multi-scale image dehazing network. This algorithm extracts haze features at different levels by constructing a multi-scale feature pyramid structure and designs a spatial-channel dual attention module to dynamically adjust feature weights. Simultaneously, it introduces cross-scale skip connections to enhance the collaborative optimization of local details and global structure. GridDehazeNet uses an end-to-end training method, without relying on prior physical scattering models. It effectively eliminates non-uniform haze in both synthetic and real foggy datasets, especially demonstrating stronger detail recovery capabilities for sea fog images containing large areas of sky or low-texture regions. However, the model has limited effectiveness in handling motion blur or dense haze intertwined with shadows in dynamic scenes, and the computational cost of the multi-scale module is high.

[0007] Paper: FFA-NET: Feature Fusion Attention Network for Single Image Dehazing, Conference: AAAI Conference on Artificial Intelligence (AAAI), Year: 2020. Qin et al. proposed a single image dehazing network based on a feature fusion attention mechanism. This algorithm dynamically allocates weights during multi-level feature fusion to enhance effective features by designing channel attention modules and pixel attention modules, and combines a residual learning strategy to avoid gradient vanishing. FFA-NET uses an end-to-end training approach, does not rely on physical prior models, and can effectively recover clear images on both synthetic and real foggy datasets, especially showing significant improvement in dense fog areas and color distortion. However, the model has a large number of parameters, resulting in poor real-time performance on lightweight devices, and its generalization ability for mixed scenes of extreme lighting and fog is still insufficient.

[0008] Paper: Transformer for Single Image Dehazing, Conference: Computer Vision and Pattern Recognition (CVPR), Year: 2022. Chen et al. proposed a single-image dehazing method based on an improved Transformer architecture. This algorithm captures long-range dependencies and local detail features by introducing an adaptive feature enhancement module and a multi-scale attention mechanism, and combines the physical prior of an atmospheric scattering model for end-to-end training, achieving modeling of complex haze distributions. Compared to traditional methods, Dehazeformer exhibits stronger robustness in sea fog images containing large areas of sky, water, and other low-texture regions. However, the algorithm relies on large-scale synthetic datasets for training, and the model has a large number of parameters, requiring significant computational resources. Summary of the Invention

[0009] This invention addresses the respective advantages and disadvantages of convolutional and self-attention structures, providing a hybrid architecture-based image dehazing method. Convolutional structures are used to process coarse-grained information in hazy images, while self-attention structures are used to process fine-grained information, thus improving both the overall dehazing effect and efficiency. Coarse-grained information represents the global structure or low-frequency components of the image (such as overall contours, lighting, and general content), while fine-grained information focuses on local details or high-frequency components (such as textures, edges, and small objects).

[0010] The technical solution of the present invention: The image dehazing method based on a hybrid architecture includes the following steps: Step 1: Use a convolutional structure to extract coarse-grained features from the input image.

[0011] Step 2: Use a self-attention structure to extract fine-grained features from the input image.

[0012] Step 3: Fuse fine-grained and coarse-grained features to obtain the enhanced dehazed image.

[0013] Step 4, together with steps 1 to 3, constitutes a hybrid architecture defogging network.

[0014] Step 5: The hybrid architecture dehazing network learns the parameters for dehazing based on the designed loss function set.

[0015] Step 1 specifically involves: Step 1.1: First-level CNN coarse-grained feature extraction.

[0016] Original size obtained from convolutional blocks and First-level feature extraction at the scale.

[0017] For the input image Initial features are extracted through a convolutional layer. , ,in, Image height, Image width, It has three channels: RGB. Represents the number of channels; the convolutional layer contains 3×3 convolutions, normalization, and activation functions; initial features The expression is: (1) in, For convolution kernel, For bias, For normalization operations, This is the activation function.

[0018] For initial features Downsampling is performed to obtain downsampled features. Then, the first-level features are generated through two convolutional blocks, i.e., a set of multiple convolutional layers. The extraction expression is: in, and For convolution kernel, .

[0019] Step 1.2: Second-level CNN coarse-grained feature extraction.

[0020] Perform via convolutional blocks Second-level feature extraction at the scale.

[0021] For first-level features Downsampling was performed to obtain Then ConvBlock was used to capture second-level features at a 1 / 4 scale. The expression is: Step 2 specifically involves: Step 2.1: Add the second-level features Divided into windows of size The feature blocks are then converted into one-dimensional features and their positions are encoded.

[0022] Second-level features after block division The expression is: in, , , To determine the size of the split window, For shape transformation operations, For position encoding.

[0023] Step 2.2: Perform self-attention calculation within the window, and shift the window after each calculation to obtain global information modeling.

[0024] To reduce device latency and increase image processing speed, the second-level features after segmentation are... Perform attention calculations within the window to obtain self-attention features. The expression is: in, These are the query, key, and value, respectively, represented by a linear transformation of the window block; To scale the values ​​and prevent gradient vanishing, This is a relative position offset term. It is a linear transformation.

[0025] Next, the window will be shifted, moving it to the bottom right corner. The number of pixels is used to create overlapping areas during calculations in different windows, and the self-attention is repeatedly calculated according to equations (6) and (7) to obtain features. This enables cross-window information interaction.

[0026] Step 2.3: Features are restored to spatial structure, and the one-dimensional sequence is reshaped back into a two-dimensional feature map.

[0027] The features obtained in step 2.2 The features are reconstructed back to two dimensions, thus enabling the fusion process in step 3, which restores the two-dimensional features. The expression is: in, .

[0028] Step 3 specifically involves: Step 3.1: Extract the following from Step 1 Scale features and those extracted in step 2 Scale feature fusion yields fused features containing multi-scale information.

[0029] For the scale is Second-level features First, the scale is The two-dimensional features of the recovery Upsampling to The two are then fused to obtain the second-level fusion feature. , means as follows: in, Representative characteristics Features obtained after doubling upsampling This represents the adaptive channel attention calculation operation.

[0030] The definition of adaptive channel attention is: in, For input features, Represents average pooling operation. Represents max pooling operation. · represents a linear transformation, and · represents a pixel-wise product.

[0031] Step 3.2: Extract the following from Step 3.1 Scale features and steps in step 1 Scale feature fusion is performed to obtain fused features with global scale, and the dehazed image is reconstructed.

[0032] For the scale is initial features First, the feature map size is... Second-level fusion features Upsampling to Then, the features are fused using adaptive channel attention to obtain the first-level fused features. , means as follows: in, Representative characteristics Features obtained through double upsampling.

[0033] Finally, the first-level fusion features are... The final dehazed image is recovered through convolutional blocks. It is expressed as follows: in, This represents the restored dehazed image.

[0034] Step 5 specifically involves: Step 5.1: Guide the hybrid architecture dehazing network to perform the dehazing task by constraining the pixel-by-pixel similarity between the dehazed image and the ground truth image.

[0035] Pixel-level loss constraint limits the similarity of the dehazed image to the ground truth image in terms of pixel values, ensuring that the generated image is consistent with the ground truth image in terms of color and brightness. Pixel-level loss The expression is: in, and These represent the ground truth image and the dehazed image, respectively. This represents the calculation process of the L1 distance.

[0036] Step 5.2: Guide the hybrid architecture dehazing network to perform the dehazing task by constraining the similarity of the feature domains of the dehazed image and the ground truth image.

[0037] Feature-level loss extracts deep features from the image using a pre-trained VGG-19 network and calculates the distance between the dehazed image and the ground truth image in the feature space. The output of an intermediate layer of the VGG-19 network is selected to constrain the consistency between the generated image and the ground truth image. Feature-level loss The expression is: in, This is a pre-trained VGG-19 network.

[0038] Step 5.3: By comparing the feature distances between the dehazed image and the ground truth image and the input hazy image, the hybrid architecture dehazing network is guided to perform the dehazing task.

[0039] Contrast loss enhances the local adaptability of the dehazing effect by narrowing the feature distance between the dehazed image and the ground truth image, and widening the feature distance between the dehazed result and the input hazy image. Contrast loss The expression is: in, It is a weight constant. For selection Features at different scales, respectively and , It is a constant used to keep the denominator non-zero, set to .

[0040] Step 5.4: The total loss function is a combination of three loss functions, and hyperparameters are used to balance the weights of each loss function. The total loss function is integrated as follows: in, and These are the hyperparameters used to balance the loss.

[0041] The beneficial effects of this invention are: This invention integrates convolutional and self-attention structures, leveraging the stability and efficiency of convolutional networks in extracting low-frequency information such as global image structure and illumination, while also enhancing the ability to capture high-frequency details such as texture and edges through attention mechanisms. By combining cross-level feature fusion and adaptive channel attention, the model's ability to perceive and process haze regions with different concentrations is significantly improved. Finally, the excellent performance on a real non-uniform haze dataset verifies the comprehensive advantages of this method in improving the quality and detail recovery of dehazed images and enhancing model adaptability. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the defogging process of the present invention.

[0043] Figure 2 Foggy images input for real foggy scenes Figure 3 It is a dehazed image output from a real foggy scene. Detailed Implementation

[0044] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.

[0045] In this embodiment, when training on a real-world haze dataset, the network is trained for a total of 3000 epochs. With an initial learning rate of 4e-4, it is reduced to 1e-4 after 1000 epochs. When inputting haze images into the network, the images are randomly cropped into 800×800 patches to reduce memory usage. The network training architecture is PyTorch, and the optimizer used is Adam. The system is equipped with an Intel Core i7-9700K CPU @3.00 GHz, 32 GB of RAM, and an Nvidia GeForce RTX 3090 graphics card.

[0046] The network training part used NH-HAZE and NH-HAZE2 dehazing data to learn the dehazing mode. NH-Haze is the first dataset specifically designed for non-uniform haze scenarios. Traditional dehazing datasets are mostly based on the assumption of synthetic haze or uniform haze, while NH-Haze focuses on real-world scenarios with uneven haze distribution, making it more challenging. NH-Haze2 is an extended version of NH-Haze, further increasing data diversity and complexity, aiming to solve more extreme non-uniform haze problems. The haze in both datasets was artificially generated by mixing fog machines and humidifiers, thus simulating real non-uniform haze distribution. NH-HAZE contains a total of 55 paired images, and NH-HAZE2 contains 35 paired images. In this embodiment, 50 images were extracted from NH-HAZE and 30 images were extracted from NH-HAZE2, for a total of 70 images to form the training set, and the remaining images were used as the test set.

[0047] The specific steps of this embodiment are as follows: Step 1: Extracting input using convolutional structure Figure 2 The image shown has coarse-grained features.

[0048] Step 1.1, Figure 2 The input image, with a shape of 1200×1600×3, is fed into a convolutional layer with a kernel size of 3×3 to obtain initial features with a shape of 1200×1600×12. The initial features were then downsampled to obtain downsampled features with a shape of 600×800×24. Two repeated convolutional structures are used to obtain first-level features with a shape of 600×800×24. See formulas (1) to (3).

[0049] Step 1.2: Repeat the downsampling and subsequent convolution operations in Step 1.1 to obtain the second-level feature with a shape of 300×400×48. See formula (4).

[0050] Step 2: Using a multi-head self-attention structure based on the second-level features obtained in Step 1. Extract fine-grained features.

[0051] Step 2.1: First, the feature map is divided into non-overlapping windows of size 8. The pixels within each window are flattened into a one-dimensional sequence, and positional encoding is added to each sequence to obtain the second-level feature map with a block shape of 1875×64×48. See formula (5).

[0052] Step 2.2: Divide the second-level features into blocks. The input is fed into a multi-head self-attention module to calculate the attention weights between all positions within each window. Then, the window is shifted 4×4 pixels to the bottom right, and the self-attention calculation is repeated to achieve cross-window information interaction and obtain self-attention features. See formulas (6) and (7).

[0053] Step 2.3: Transform one-dimensional features Reconstructing back to two-dimensional features The shape is 300×400×48, see formula (8).

[0054] Step 3: Fuse coarse-grained and fine-grained features to obtain the enhanced dehazed image.

[0055] Step 3.1: The second-level feature with a shape of 600×800×24 obtained from the convolutional block in Step 1.2... The recovered two-dimensional feature with a shape of 300×400×48 is obtained by self-attention in step 2.3. Fusion is performed using channel attention to obtain a second-level fused feature with a shape of 600×800×24. See formulas (9) to (11).

[0056] Step 3.2: Apply the initial feature with a shape of 1200×1600×12 obtained in Step 1.1. This is combined with the second-level fusion feature with a shape of 600×800×24 obtained in step 3.1. The fusion process yields a first-level fusion feature with a shape of 1200×1600×12. The image is then restored to its final dehazed state using convolutional blocks. See formulas (12) to (14).

[0057] Step 4, and steps 1 to 3 together form a hybrid architecture defogging network.

[0058] Step 5: The hybrid architecture dehazing network learns the parameters for dehazing based on the designed loss function set.

[0059] The numerical values ​​of the three losses are obtained according to the formulas in steps 5.1, 5.2, and 5.3. After step 5.4 hyperparameters = 0.1, After balancing to 0.005, backpropagation is performed to update the network's learnable parameters, ultimately obtaining a clear, dehazed image, as shown below. Figure 3 As shown.

Claims

1. An image dehazing method based on a hybrid architecture, characterized in that, Includes the following steps: Step 1: Use a convolutional structure to extract coarse-grained features from the input image; Step 1.1: First-level CNN coarse-grained feature extraction; Original size obtained from convolutional blocks and First-level feature extraction at the scale; Step 1.2: Second-level CNN coarse-grained feature extraction; Perform via convolutional blocks Second-level feature extraction at the scale; Step 2: Use a self-attention structure to extract fine-grained features from the input image; Step 2.1: Segment the second-level features into window sizes of... The feature blocks are then converted into one-dimensional features and their positions are encoded. Step 2.2: Perform self-attention calculation within the window, and shift the window after each calculation to obtain global information modeling; Step 2.3: Features are restored to spatial structure, reshaping the one-dimensional sequence back into a two-dimensional feature map; Step 3: Fuse fine-grained and coarse-grained features to obtain the enhanced dehazed image; Step 3.1: Extract the following from Step 1 Scale features and those extracted in step 2 Scale feature fusion yields fused features containing multi-scale information; Step 3.2: Extract the following from Step 3.1 Scale features and steps in step 1 Scale feature fusion is performed to obtain fused features with global scale, and the dehazed image is reconstructed. Step 4, together with steps 1 to 3, constitutes a hybrid architecture defogging network; Step 5: The hybrid architecture dehazing network learns the parameters for dehazing based on the designed loss function set; Step 5.1: Guide the hybrid architecture dehazing network to perform the dehazing task by constraining the pixel-by-pixel similarity between the dehazed image and the ground truth image; Step 5.2: Guide the hybrid architecture dehazing network to perform the dehazing task by constraining the similarity of the feature domains of the dehazed image and the ground truth image; Step 5.3: By comparing the feature distances between the dehazed image and the ground truth image and the input hazy image, guide the hybrid architecture dehazing network to perform the dehazing task; Step 5.4: The total loss function is a combination of three loss functions, and hyperparameters are used to balance the weights of each loss function.

2. The image dehazing method based on a hybrid architecture according to claim 1, characterized in that, Step 1.1 is as follows: For the input image Initial features are extracted through a convolutional layer. , ,in, Image height, Image width, It has three channels: RGB. Represents the number of channels; the convolutional layer contains 3×3 convolutions, normalization, and activation functions; initial features The expression is: （1） in, For convolution kernel, For bias, For normalization operations, For activation functions; For initial features Downsampling is performed to obtain downsampled features. Then, the first-level features are generated through two convolutional blocks, i.e., a set of multiple convolutional layers. The extraction expression is: in, and For convolution kernel, ; Step 1.2 is as follows: Perform via convolutional blocks Secondary coarse-grained feature extraction at the scale; For first-level features Downsampling was performed to obtain Then ConvBlock was used to capture second-level features at a 1 / 4 scale. The expression is: 。 3. The image dehazing method based on a hybrid architecture according to claim 1, characterized in that, Step 2.1 is as follows: Second-level features Divided into windows of size Feature blocks, second-level features after block division The expression is: in, , , To determine the size of the split window, For shape transformation operations, For position encoding; Step 2.2 is as follows: The second-level features after block division Perform attention calculations within the window to obtain self-attention features. The expression is: in, These are the query, key, and value, respectively, represented by a linear transformation of the window block; To scale the values ​​and prevent gradient vanishing, This is a relative position offset term. It is a linear transformation; Next, the window will be shifted, moving it to the bottom right corner. The number of pixels is used to create overlapping areas during calculations in different windows, and the self-attention is repeatedly calculated according to equations (6) and (7) to obtain features. This enables cross-window information interaction; Step 2.3 is as follows: The features obtained in step 2.2 The features are reconstructed back to two dimensions, thus enabling the fusion process in step 3, which restores the two-dimensional features. The expression is: in, .

4. The image dehazing method based on a hybrid architecture according to claim 1, characterized in that, Step 3.1 is as follows: For the scale is Second-level features First, the scale is The two-dimensional features of the recovery Upsampling to The two are then fused to obtain the second-level fusion feature. , means as follows: in, Representative characteristics Features obtained after doubling upsampling This represents the adaptive channel attention calculation operation; The definition of adaptive channel attention is: in, For input features, Represents average pooling operation. Represents max pooling operation. · represents a linear transformation, and · represents a pixel-wise product; Step 3.2 is as follows: For the scale is initial features First, the feature map size is... Second-level fusion features Upsampling to Then, the features are fused using adaptive channel attention to obtain the first-level fused features. , means as follows: in, Representative characteristics Features obtained after doubling upsampling; Finally, the first-level fusion features are... The final dehazed image is recovered through convolutional blocks. It is expressed as follows: in, This represents the restored dehazed image.

5. The image dehazing method based on a hybrid architecture according to claim 1, characterized in that, Step 5.1 is as follows: Pixel-level loss constraint is used to constrain the similarity of the dehazed image and the ground truth image in pixel values, ensuring that the generated image is consistent with the ground truth image in color and brightness; pixel-level loss The expression is: in, and These represent the ground truth image and the dehazed image, respectively. This represents the calculation process for the L1 distance; Step 5.2 is as follows: Feature-level loss extracts deep features from the image using a pre-trained VGG-19 network and calculates the distance between the dehazed image and the ground truth image in the feature space; the output of the intermediate layers of VGG-19 is selected to constrain the consistency between the generated image and the real image; feature-level loss. The expression is: in, For the pre-trained VGG-19 network; Step 5.3 is as follows: Contrast loss enhances the local adaptability of the dehazing effect by narrowing the feature distance between the dehazed image and the ground truth image, and widening the feature distance between the dehazed result and the input hazy image. The expression is: in, It is a weight constant. For selection Features at different scales, respectively and , It is a constant used to keep the denominator non-zero; Step 5.4 is as follows: The total loss function is summarized as follows: in, and These are the hyperparameters used to balance the loss.

Images