A dynamic three-branch self-attention dehazing method

Abstract

The application discloses a dynamic three-branch self-attention defogging method, wherein a scheme firstly performs global and local feature processing on a haze image, and enhances the comprehensiveness and accuracy of defogging through a dynamic three-branch module; then, a self-attention mechanism is established, the bottom layer of the network is introduced, automatic integration and weighting of effective feature information are realized, and the expression capability and defogging effect of the network are improved; then, a high-frequency loss method is proposed, aiming at enhancing the texture detail restoration capability of the defogged image; then, the high-frequency information of the image is highlighted, and the detail restoration effect is improved; finally, the model is trained to obtain an optimal defogging model, the optimal model is used for image defogging, a defogging result is output, and the detection result is evaluated. The application effectively solves the problem that haze features are difficult to learn, and has better defogging effect in actual tests.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and more specifically to a dehazing method based on dynamic three-branch self-attention. Background Technology

[0002] Image dehazing is one of the core problems in computer vision and image processing, widely used in surveillance, automotive, aerospace, and intelligent transportation. Due to its significant improvement in visual effects, image dehazing technology holds an important position in the global image processing field. However, weather conditions such as fog, smoke, and rain can severely affect image quality, degrading the visual quality of objects in the image and thus hindering subsequent image analysis and understanding. Therefore, in practical applications, dehazing is a crucial step to ensure proper image processing.

[0003] Traditional image dehazing methods mainly rely on physical models, using prior information to estimate transmittance and atmospheric light values, and then inverting the dehazed image. Although these methods can achieve good dehazing results, they are highly sensitive to factors such as fog density and lighting conditions, and have poor adaptability to complex real-world environments. In addition, these methods have complex image processing algorithms and high computational load, making it difficult to meet the needs of real-time processing.

[0004] In recent years, deep learning technology has achieved significant breakthroughs in many fields, including image classification, object detection, and semantic segmentation. In the problem of image dehazing, deep learning methods can learn fog generation models by learning from a large amount of training data, thus effectively dehazing images. However, existing deep learning methods still face difficulties in recognizing and processing small fog particles, especially those in complex backgrounds and under different lighting conditions. This results in fog residue in distant scenes, and the restoration of color and texture information remains insufficient. Therefore, a dynamic three-branch self-attention dehazing method is needed to address these problems. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a dynamic three-branch self-attention dehazing method. This method solves the problems of existing technologies in image dehazing, such as poor adaptability to complex real-world environments, complex processing algorithms, large computational load, and difficulty in meeting real-time processing requirements. It also addresses the difficulties of deep learning technology in image dehazing, such as recognition and processing difficulties, resulting in residual haze in distant scenes, and insufficient restoration of color and texture information. This method features fast processing speed, good dehazing effect, and high image restoration accuracy.

[0006] To achieve the above-mentioned technical features, the technical solution adopted by the present invention is as follows:

[0007] A dynamic three-branch self-attention dehazing method includes the following steps:

[0008] S1. Obtain the publicly available dataset, then perform data preprocessing on foggy and fog-free images to establish the network framework;

[0009] S2, Construct a dynamic three-branch self-attention dehazing network. This network adopts a U-shaped structure to extract haze features and restore images.

[0010] S3 constructs a dynamic three-branch module, which dynamically extracts feature maps of different scales by assigning different weights to the three branches, thus solving the problem of difficult feature extraction in haze.

[0011] S4 adds a self-attention module to the bottom layer of the network;

[0012] S5, in the commonly used pixel matching loss L PM and perceived loss L P Adding high-frequency loss L HF This ensures the texture details of the image and improves clarity;

[0013] S6, Training the dynamic three-branch self-attention dehazing network model: Divide the dataset into training and validation sets in an 8:2 ratio and input them into the dynamic three-branch self-attention dehazing network model for pre-training, for a total of 200 rounds;

[0014] S7 uses the trained optimal weights for testing and evaluates the detection results to achieve end-to-end automated dehazing.

[0015] Preferably, in step S1, the data preprocessing step includes collecting and organizing the foggy and fog-free images to obtain foggy and fog-free datasets, and then performing standardization and normalization processing on the images to improve the generalization ability of the network model, as follows:

[0016]

[0017]

[0018] Where z is the standardized value, x is the original data value, μ is the mean of the dataset, and σ is the standard deviation of the dataset.

[0019] Preferably, in step S2, the dynamic three-branch self-attention dehazing network framework includes a backbone U-shaped network, a dynamic three-branch module, and a self-attention module. The backbone U-shaped network is used to extract haze features from the image and restore the image. The dynamic three-branch module is a strategy to enhance feature extraction. The self-attention module is used to enable the network to utilize features more selectively and accelerate the network's learning.

[0020] Preferably, the structure of the dynamic three-branch module in step S3 includes three parallel downsampling layers P1 to P3, each downsampling layer followed by a learnable weight, dynamically extracting feature maps of different scales.

[0021] Furthermore, in feature extraction, pooling downsampling and convolutional downsampling have different advantages and disadvantages. Pooling downsampling is computationally efficient, has few parameters, and possesses some translation invariance, but it cannot extract rich features. In contrast, convolutional downsampling can learn feature representations and extract richer features, but its computational complexity is higher. Similarly, 1x1 and 3x3 convolutional kernels also have their own advantages and disadvantages. 1x1 convolutional kernels are computationally efficient, have few parameters, and can reduce feature dimensions or channel number, but their receptive field is smaller. 3x3 convolutional kernels have a larger receptive field and can capture a wider range of contextual information, but their computational complexity is higher. Through downsampling operations with three branches, the module can extract fine-grained, medium-grained, and global features, improving its ability to perceive features at different scales. Finally, the module's weight parameters are learnable and can be updated and optimized through parameter learning. This allows the module to automatically learn the optimal weight combination, improving feature extraction performance and adapting to different data distributions. In summary, the dynamic three-branch module has the advantages of flexibility, multi-scale feature fusion, and parameter learning.

[0022] Preferably, in the self-attention module structure of step S4, the self-attention mechanism assigns higher weights to information with high relevance to itself by calculating the similarity between vectors, and assigns lower weights to information with low relevance. This is a powerful tool for capturing the interdependencies within a sequence. This mechanism achieves adaptive weighting at different positions in the input features, enabling the capture of long-distance dependencies in the sequence. In a U-shaped network, the low-level features typically contain image details and local structures, but their size is significantly reduced due to multiple downsampling operations. This results in a significant loss of spatial information in the low-level features, making the feature maps relatively abstract and lacking fine edge information. The self-attention mechanism works as follows:

[0023] S401 transforms 3D feature vectors into 2D feature vectors through view transformation. Then, for each input vector, it performs a linear transformation through three different fully connected layers to obtain three new vectors, which are called key(K), query(Q), and value(V).

[0024]

[0025] S402: For any pair of Q and K, a score is obtained by calculating their dot product; this score represents the validity of the corresponding two input vectors. To ensure the stability of the model, the score is normalized using the softmax function.

[0026]

[0027] S403, finally, the normalized scores are weighted and summed over the V vector to obtain the output of this self-attention module. This result is a weighted combination of the input vectors, where the weights represent the correlation between the input vectors, as shown in the following formula:

[0028] Attention(Q,K,V)=V·Similarity(Q,K);

[0029] In the formula, K, Q, and V are three sets of vectors obtained by linear transformation through three different fully connected layers.

[0030] Preferably, in step S5, due to the presence of haze, high-frequency information in the image is severely lost. To address this issue, a high-frequency loss function is proposed, and the calculation method for the high-frequency loss is as follows:

[0031] S501, the haze-free image and the dehazed image are respectively processed by the Canny algorithm for edge extraction, because the Canny algorithm has excellent performance in edge detection. By extracting edge information from the image, the loss of high-frequency information during the dehazing process can be effectively compensated;

[0032] S502, performs pixel matching calculations between the image processed by the Canny algorithm and the original image:

[0033] L HF =E x,k [||canny(y)-canny(k)||1];

[0034] In the formula, L HF This indicates high-frequency loss, canny represents the result image processed by the canny edge detection algorithm, and ||·||1 indicates that the difference between the two images is calculated point by point.

[0035] Preferably, the evaluation metrics in step S7 include peak signal-to-noise ratio and structural similarity.

[0036] The beneficial effects of the dynamic three-branch self-attention dehazing method provided by this invention are as follows:

[0037] This solution first performs global and local feature processing on hazy images, using a dynamic three-branch module to enhance the comprehensiveness and accuracy of dehazing. Then, a self-attention mechanism is established and introduced into the network's lower layer to automatically integrate and weight effective feature information, thereby improving the network's expressive power and dehazing effect. Next, a high-frequency loss method is proposed to enhance the texture detail restoration capability of the dehazed image. Then, by highlighting high-frequency information in the image, the detail restoration effect is improved. Finally, the model is trained to obtain the optimal dehazing model, which is used to perform image dehazing, outputting the dehazing result and evaluating the detection results. This invention effectively solves the problems of existing technologies in image dehazing, such as poor adaptability to complex real-world environments, complex processing algorithms, high computational load, and difficulty in meeting real-time processing requirements. It also addresses the difficulties of deep learning technology in image dehazing, such as recognition and processing difficulties leading to haze residue in distant areas and insufficient restoration of color and texture information. This invention features fast processing speed, good dehazing effect, and high image restoration accuracy. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the flowchart of the present invention;

[0039] Figure 2 This is a diagram of the dynamic three-branch self-attention network structure in this invention;

[0040] Figure 3 This is a structural diagram of the dynamic three-branch module in this invention;

[0041] Figure 4 This is a structural diagram of the self-attention module in this invention;

[0042] Figure 5 This is a schematic diagram of the high-frequency loss principle in this invention;

[0043] Figure 6 This is a diagram showing the neural network dehazing effect obtained in Embodiment 2 of the present invention. Detailed Implementation

[0044] Example 1:

[0045] like Figure 1 As shown, a dynamic three-branch self-attention dehazing method includes the following steps:

[0046] S1. Obtain the publicly available dataset, then perform data preprocessing on foggy and fog-free images to establish the network framework;

[0047] S2, Construct a dynamic three-branch self-attention dehazing network. This network adopts a U-shaped structure to extract haze features and restore images.

[0048] S3 constructs a dynamic three-branch module, which dynamically extracts feature maps of different scales by assigning different weights to the three branches, thus solving the problem of difficult feature extraction in haze.

[0049] S4 adds a self-attention module to the bottom layer of the network;

[0050] S5, in the commonly used pixel matching loss L PM and perceived loss L P Adding high-frequency loss L HF This ensures the texture details of the image and improves clarity;

[0051] S6, Training the dynamic three-branch self-attention dehazing network model: Divide the dataset into training and validation sets in an 8:2 ratio and input them into the dynamic three-branch self-attention dehazing network model for pre-training, for a total of 200 rounds;

[0052] S7 uses the trained optimal weights for testing and evaluates the detection results to achieve end-to-end automated dehazing.

[0053] Preferably, in step S1, the data preprocessing step includes collecting and organizing the foggy and fog-free images to obtain foggy and fog-free datasets, and then performing standardization and normalization processing on the images to improve the generalization ability of the network model, as follows:

[0054]

[0055]

[0056] Where z is the standardized value, x is the original data value, μ is the mean of the dataset, and σ is the standard deviation of the dataset.

[0057] like Figure 2 As shown, preferably, in step S2, the dynamic three-branch self-attention dehazing network framework includes a backbone U-shaped network, a dynamic three-branch module, and a self-attention module. The backbone U-shaped network is used to extract haze features from the image and restore the image. The dynamic three-branch module is a strategy to enhance feature extraction. The self-attention module is used to make the network utilize features more selectively and accelerate the learning of the network.

[0058] like Figure 3 As shown, preferably, the structure of the dynamic three-branch module in step S3 includes three parallel downsampling layers P1 to P3, each downsampling layer is followed by a learnable weight, and feature maps of different scales are dynamically extracted.

[0059] Furthermore, in feature extraction, pooling downsampling and convolutional downsampling have different advantages and disadvantages. Pooling downsampling is computationally efficient, has few parameters, and possesses some translation invariance, but it cannot extract rich features. In contrast, convolutional downsampling can learn feature representations and extract richer features, but its computational complexity is higher. Similarly, 1x1 convolutional kernels and 3x3 convolutional kernels also have their own advantages and disadvantages. 1x1 convolutional kernels are computationally efficient, have few parameters, and can reduce feature dimensions or the number of channels, but their receptive field is smaller. 3x3 convolutional kernels have a larger receptive field and can capture a wider range of contextual information, but their computational complexity is higher. Specifically:

[0060]

[0061] Where x is the input feature map, conv is a 3x3 convolution that only changes the number of channels, conv1 and conv2 are 1x1 and 3x3 convolutions respectively that only perform a 2x downsampling on the feature map, and pool is max pooling that also performs a 2x downsampling on the feature map.

[0062] Through downsampling operations along three branches, the module can extract fine-grained, medium-grained, and global features, improving its ability to perceive features at different scales. Finally, the module's weight parameters are learnable, updated and optimized through parameter learning. This allows the module to automatically learn the optimal weight combination, improving feature extraction performance and adapting to different data distributions; specifically as follows:

[0063]

[0064] In the formula, W i For adaptive weights, the initial weight coefficients are 0.333. This allows the module to dynamically adjust the contribution of different branches to feature extraction to adapt to task requirements. In summary, the dynamic three-branch module has the advantages of flexibility, multi-scale feature fusion, and parameter learning.

[0065] like Figure 4As shown, preferably, in the self-attention module structure in step S4, the self-attention mechanism assigns higher weights to information with high relevance to itself by calculating the similarity between vectors, and assigns lower weights to information with low relevance. This is a powerful tool that can capture the interdependencies within a sequence. This mechanism achieves adaptive weighting of different positions in the input features, and can capture long-distance dependencies in the sequence. In a U-shaped network, the low-level features usually contain the details and local structure of the image, but their size has been significantly reduced due to multiple downsampling operations. This results in a significant loss of spatial information in the low-level features, making the feature map relatively abstract and lacking fine edge information. To solve this problem, introducing a self-attention module can help the network refocus on details and local features, improving the network's perception ability at the low level.

[0066] The self-attention mechanism is as follows:

[0067] S401 transforms 3D feature vectors into 2D feature vectors through view transformation. Then, for each input vector, it performs a linear transformation through three different fully connected layers to obtain three new vectors, which are called key(K), query(Q), and value(V).

[0068]

[0069] S402: For any pair of Q and K, a score is obtained by calculating their dot product; this score represents the validity of the corresponding two input vectors. To ensure the stability of the model, the score is normalized using the softmax function.

[0070]

[0071] S403, finally, the normalized scores are weighted and summed over the V vector to obtain the output of this self-attention module. This result is a weighted combination of the input vectors, where the weights represent the correlation between the input vectors, as shown in the following formula:

[0072] Attention(Q,K,V)=V·Similarity(Q,K);

[0073] In the formula, K, Q, and V are three sets of vectors obtained by linear transformation through three different fully connected layers.

[0074] The total training loss in step S6 consists of three parts: Loss = λ1L PM +λ2L P +λ3L HF ;

[0075] like Figure 5As shown in the formula, L PM For pixel matching loss, L P To perceive loss, L HF For high-frequency loss; pixel matching loss; to obtain a realistic image, this paper improves the adversarial loss by comparing pixels one by one:

[0076] L p =E x,k [||yk||1];

[0077] Perceptual loss is used to address the problem that traditional mean squared error loss functions neglect high-level semantic information of images, leading to generated images that do not match human perception. Perceptual loss is employed to improve the visual perceptual similarity of dehazed images; specifically, we use the first n layers (denoted as φ) of the VGG-16 network. n We extract feature maps from the hazy image y and the dehazed image k. Then, we define the perceptual loss by calculating the mean squared error distance between the feature maps. By introducing the abstract information contained in the feature maps, we can better restore the high-level semantics of the image, making the generated image more in line with the intuitive perception of the human eye and improving the visual quality. The formula is as follows:

[0078]

[0079] In the formula, C, W, and H represent the number of image channels, width, and height, respectively.

[0080] Preferably, in step S5, due to the presence of haze, high-frequency information in the image is severely lost. To address this issue, a high-frequency loss function is proposed, and the calculation method for the high-frequency loss is as follows:

[0081] S501, the haze-free image and the dehazed image are respectively processed by the Canny algorithm for edge extraction, because the Canny algorithm has excellent performance in edge detection. By extracting edge information from the image, the loss of high-frequency information during the dehazing process can be effectively compensated;

[0082] S502 performs pixel matching calculations between the image processed by the Canny algorithm and the original image:

[0083] L HF =E x,k [||canny(y)-canny(k)||1];

[0084] In the formula, L HF This indicates high-frequency loss, canny represents the result image processed by the canny edge detection algorithm, and ||·||1 indicates that the difference between the two images is calculated point by point.

[0085] Preferably, the evaluation metrics in step S7 include peak signal-to-noise ratio and structural similarity.

[0086] Peak Signal-to-Noise Ratio (PSNR) evaluates an image by calculating the differences between individual pixels in two images. A higher PSNR value indicates higher image quality. The formula is:

[0087]

[0088] In the formula, m is the length of the image, n is the width of the image, I(i,j) is the dehazed image, and K(i,j) is the true haze-free image. Structural similarity (SSIM) judges image quality by analyzing three factors: brightness, contrast, and structure. Its design is more in line with the characteristics of the human visual perception system. The value range is 0 to 1; the higher the similarity, the higher the image quality. The formula is:

[0089]

[0090] In the formula: x and y are the dehazed image and the haze-free image, respectively; μ is the mean value; σ 2 For variance calculation, σ is for covariance calculation, and c1 and c2 are constants.

[0091] Example 2:

[0092] To verify the effectiveness of each strategy in the proposed network, the following ablation experiments were designed:

[0093] 1) Model a: Original U-shaped network, without adding any modules;

[0094] 2) Model b: Based on the original U-shaped network, a dynamic three-branch module (DTBM) is used for downsampling;

[0095] 3) Model c: Add a self-attention module (DTBM+SAM);

[0096] 4) Model d: The final network in this paper, with added high-frequency loss (DTBM+SAM+HF-Loss).

[0097] This paper evaluates various algorithms using the SOTS outdoor test set. All experiments were conducted under the same training equipment and parameters; the dehazing comparison results are as follows: Figure 6 As shown, the results of the objective evaluation indicators are summarized in Table 1.

[0098] from Figure 6It can be observed that the original U-shaped network did not effectively learn the characteristics of haze, and a large amount of haze residue still exists within the red box. By adding the DTBM module, the network can better understand haze, and the PSNR is improved by 1.92dB, but due to insufficient feature utilization, haze residue still exists.

[0099] from Figure 6 As can be seen, after adding the SAM module, the haze in the red box disappeared significantly; after adding FH-Loss, the haze component in the red area was completely removed, and the color was almost the same as the original image; in summary, the DTBM+SAM+HF-Loss network used in this paper performed the best in terms of dehazing effect, with the highest degree of dehazing completion and color reproduction.

[0100] Table 1: Objective Comparison of Ablation Experiments

[0101]

[0102] As can be seen from the objective data in Table 1, the DTBM+SAM+HF-Loss network achieves the highest level in all indicators; at the same time, each module proposed in this paper has a positive impact on the defogging effect.

Claims

1. A dehazing method based on dynamic three-branch self-attention, characterized in that, Includes the following steps: S1. Obtain the publicly available dataset, then perform data preprocessing on foggy and fog-free images to establish the network framework; S2. A dynamic three-branch self-attention dehazing network is constructed. This network adopts a U-shaped structure to extract haze features and restore images. The framework of the dynamic three-branch self-attention dehazing network includes a backbone U-shaped network, a dynamic three-branch module, and a self-attention module. The backbone U-shaped network is used to extract haze features from the image and restore the image. The dynamic three-branch module is a strategy to enhance feature extraction. The self-attention module is used to make the network utilize features more selectively and accelerate the network's learning. A dynamic three-branch module is set in each layer of the U-shaped network encoder. S3 constructs a dynamic three-branch module, which dynamically extracts feature maps of different scales by assigning different weights to the three branches; the structure of the dynamic three-branch module includes three parallel downsampling layers. P 1 ~P 3. Each downsampling layer is followed by a learnable weight to dynamically extract feature maps of different scales; S4 adds a self-attention module to the bottom layer of the network; S5, in pixel matching loss and perceived loss Adding high-frequency loss ; S6, Training the dynamic three-branch self-attention dehazing network model: Divide the dataset into training and validation sets in an 8:2 ratio and input them into the dynamic three-branch self-attention dehazing network model for pre-training; S7 uses the trained optimal weights for testing and evaluates the detection results to achieve end-to-end automated dehazing.

2. The dehazing method for dynamic three-branch self-attention according to claim 1, characterized in that: In step S1, the data preprocessing step includes collecting and organizing foggy and fog-free images to obtain foggy and fog-free datasets, and then standardizing and normalizing the images.

3. The dehazing method for dynamic three-branch self-attention according to claim 1, characterized in that: In step S4, the self-attention module structure calculates the similarity between vectors, assigning higher weights to information with high relevance and lower weights to information with low relevance. The self-attention mechanism works as follows: S401 transforms 3D feature vectors into 2D feature vectors through view changes. Then, for each input vector, it performs a linear transformation through three different fully connected layers to obtain three new vectors, which are called key (K), query (Q), and value (V). ； S402, for any pair of Q and K, obtain a score by calculating their dot product; this score represents the validity of the corresponding two input vectors, and the score is normalized using the softmax function: ； S403. Finally, the normalized scores are weighted and summed over the V vector. The result is the output of this self-attention module. This result is a weighted combination of the input vectors, where the weights represent the correlation between the input vectors, as shown in the following formula: ； In the formula, K, Q, and V are three sets of vectors obtained by linear transformation through three different fully connected layers.

4. The dehazing method for dynamic three-branch self-attention according to claim 1, characterized in that: In step S5, the high-frequency loss method is as follows: S501, the haze-free image and the dehaze image are respectively processed by the Canny algorithm for edge extraction; S502 performs pixel matching calculations between the image processed by the Canny algorithm and the original image: ； In the formula, Indicates high-frequency loss; canny represents the result image processed by the canny edge detection algorithm. This indicates that the difference between two images is calculated point by point.

5. The dehazing method for dynamic three-branch self-attention according to claim 1, characterized in that: The evaluation metrics in step S7 include peak signal-to-noise ratio and structural similarity.

Images