A training method and a dehazing method of an image dehazing model based on an external enhancer and an internal enhancer

Abstract

The application discloses a kind of training method and defogging method of image defogging model based on external enhancer and internal enhancer. Including the following steps: obtain fog-free image set, proportionally divide the image set for training and test image set;External enhancement strategy is carried out on the image set for training, and the training image set is obtained;Feature extraction is carried out on the training image set using defogging model, and feature map is obtained, feature fusion is carried out on feature map using feature fusion module, and defogging image is obtained, external loss is calculated, and model training is carried out;Internal enhancement strategy is carried out on the defogging image, internal loss is calculated, and model training is carried out. Repeat the above training steps until the entire training image set is trained, and the trained model is obtained. The application proposes a channel migration-based data enhancement method from the outside of the data set, so that the augmented data set approaches the target data set in the color channel, and the model performance is effectively improved through external and internal collaborative enhancement.

Description

Technical Field

[0001] This invention belongs to the field of image dehazing, and more specifically, relates to a training method and a dehazing method for an image dehazing model based on an external enhancer and an internal enhancer. Background Technology

[0002] In recent years, with the rapid development and widespread application of computer vision technology, high-quality image input has become a crucial foundation for many visual tasks. This has led to increasing attention being paid to image dehazing, especially in the fields of remote sensing and natural image processing. Haze often causes a significant decrease in image contrast and sharpness, blurring the boundaries of objects that were originally clearly defined in color and outline. This not only greatly affects the visual effect of the image but also poses many challenges to its further applications. For example, in the field of remote sensing, satellite and aerial photographs affected by haze lose a large amount of critical information, thus affecting the accuracy and reliability of tasks such as topographic mapping, environmental monitoring, and disaster assessment. Similarly, in the field of autonomous driving, images captured by vehicle cameras that are interfered with by haze may lead to misjudgments in the autonomous driving system's identification of roads and obstacles, thereby affecting driving safety.

[0003] Therefore, developing effective image dehazing techniques is of great significance for enhancing the application value of remote sensing and natural imagery. By removing the effects of haze, image clarity and contrast can be restored, making them more suitable for various computer vision tasks. This can not only improve the safety and reliability of autonomous driving systems but also enhance the application effectiveness of remote sensing images in environmental monitoring, urban planning, and resource management.

[0004] Image dehazing technology has attracted significant research attention in recent years due to its ability to improve image quality while enhancing performance in subsequent computer vision tasks. However, selecting appropriate dehazing strategies for different image features and haze conditions remains a challenging and popular research area. Therefore, researchers typically explore various approaches. On one hand, traditional image processing methods enhance visual clarity by increasing contrast and sharpening edges; on the other hand, physics-based methods attempt to simulate the interaction between light and atmospheric particles to inversely derive clear, haze-free images. Other researchers are dedicated to developing deep learning-based dehazing models, using large pairs of haze and clear images as training data to allow the model to learn the complex mapping relationship from haze to clear images.

[0005] The existing invention patent with publication number CN115330627A proposes an image dehazing method, apparatus, device, and medium based on self-supervised learning, including: acquiring multiple clear, haze-free images; processing each clear, haze-free image separately using an atmospheric scattering model according to multiple preset image processing angles to obtain multiple hazy images of the clear, haze-free images; for each hazy image, taking the hazy image and its corresponding clear, haze-free image as an image pair to obtain multiple image pairs; training the image dehazing model using multiple image pairs to obtain a trained image dehazing model; and using the trained image dehazing model to perform dehazing processing on the image to be dehazed. This scheme does not reduce the contrast differences between different hazy image sets, nor does it make the augmented image set closer to the target image set in terms of color channels, resulting in poor robustness of the model. Summary of the Invention

[0006] To reduce the contrast differences between different hazy image sets, make the augmented image set closer to the target image set in terms of color channels, and improve the robustness of the model, this invention provides a training method and a dehazing method for an image dehazing model based on external and internal enhancers.

[0007] The first aspect of this invention provides a method for training an image dehazing model based on an external enhancer and an internal enhancer, comprising the following steps:

[0008] Obtain a preset set of foggy and fog-free images, and divide the set into a training image set and a test image set according to a preset ratio; apply an external enhancement strategy to the training image set according to a preset method to obtain the enhanced image set as the training image set;

[0009] Training begins by using a dehazing model to extract features from the training image set, obtaining feature maps. The feature fusion module then fuses the features in the feature maps to obtain dehazed images. The external loss of the model is calculated, and the model is trained using a preset strategy.

[0010] An internal enhancement strategy is applied to the dehazed image, the internal loss of the model is calculated, and the model is trained using a preset strategy to end one training cycle.

[0011] Repeat the above training steps until the entire training image set is trained, and you will get the trained model.

[0012] The trained model was evaluated using a set of test images.

[0013] Furthermore, the step of applying an external enhancement strategy to the image set includes:

[0014] The image set is divided into augmented image set and target image set;

[0015] The augmented image set is generated by using a channel migration method, which calculates gamma values ​​and performs gamma calibration.

[0016] Furthermore, the image set is divided into two parts: the RSID dataset is used as the augmented image set, and the NID dataset is used as the target image set.

[0017] Furthermore, the formula for the gamma calibration is:

[0018]

[0019] Among them, O R,G，B and I R,G，B These represent the output and input pixel intensities, respectively, with γ being the gamma factor. R, G, and B represent the three color channels, with gamma values ​​γ. R,G,B It is obtained by calculating the gamma value transferred to the target image set.

[0020] Furthermore, the feature extraction using a dehazing model on the training image set is achieved by normalizing, padding, projecting, and calculating attention scores on the input feature map using the DehazeNetwork dehazing network.

[0021] Furthermore, the step of using the feature fusion module to perform feature fusion on the feature map includes:

[0022] Channel attention weights for different feature maps are learned by using global average pooling.

[0023] The feature map is projected using a linear layer, and the feature map is fused by combining the channel attention weights with the projection.

[0024] Furthermore, the step of applying an internal enhancement strategy to the dehazed image includes:

[0025] Perform a weak enhancement operation on the dehazed image to generate the weak enhancement result;

[0026] Perform a strong enhancement operation on the weak enhancement result to generate a strong enhancement result;

[0027] Calculate the difference between the two results to obtain the model's internal loss.

[0028] Furthermore, the formula for calculating the internal loss of the computational model is as follows:

[0029]

[0030]

[0031] Where α is a hyperparameter, s is the current training iteration, S is the total training iterations, and I... weak Represented as an image generated by weak enhancement from a real haze-free image, I aggr Based on the obtained I weak The image generated through strong enhancement.

[0032] Furthermore, the preset strategy for model training is a cosine annealing strategy.

[0033] A second aspect of the present invention provides an image dehazing method based on an external enhancer and an internal enhancer, which uses the trained model to dehaze the image to be dehazed, and obtains the dehazed image.

[0034] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0035] This invention enhances the dataset both externally and internally, further uncovering more effective local information. Specifically, it proposes a channel-transfer-based data augmentation method from outside the dataset, aiming to reduce contrast differences between different hazy datasets and make the augmented dataset more similar to the target dataset in terms of color channels. The augmented dataset obtained using this method better matches the distribution of the target dataset, and the model trained using these augmented datasets is more robust and performs better. Attached Figure Description

[0036] To make the objectives and technical solutions of this invention clearer, the following drawings are provided and described:

[0037] Figure 1 A flowchart provided for an embodiment of the present invention;

[0038] Figure 2 This is a method framework diagram provided for an embodiment of the present invention;

[0039] Figure 3 This is a flowchart illustrating an external augmentation strategy for a dataset, provided in an embodiment of the present invention.

[0040] Figure 4 This is a flowchart illustrating the internal enhancement strategy for a dataset provided in an embodiment of the present invention;

[0041] Figure 5 This is a dehazing effect diagram on the NID dataset provided in an embodiment of the present invention; Detailed Implementation

[0042] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0043] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0044] The first aspect of this invention provides a training method for an image dehazing model based on external and internal enhancers, such as... Figure 1 The diagram shows a training method for an image dehazing model based on external and internal enhancers. Figure 2 The diagram shown illustrates the training method framework provided by this invention. The specific steps are as follows:

[0045] S1: Obtain a preset set of foggy and fog-free images, and divide the set of images into a training set and a test set according to a preset ratio; perform an external enhancement strategy on the training set according to a preset method to obtain the enhanced image set as the training image set;

[0046] The specific process is as follows:

[0047] S1.1: Divide the image set to determine the augmented image set and the target image set;

[0048] More specifically, the RSID dataset was used as the augmented image set, and the NID dataset was used as the target image set.

[0049] In one specific embodiment, the fog-free dataset comprises the NID dataset and the RSID dataset. The NID dataset consists of synthetic natural fog images that rely on depth information estimation. Therefore, the NID dataset first estimates the depth map of the natural image using a depth prediction algorithm, and then synthesizes the natural fog image according to a fog simulation strategy. The NID dataset includes 1000 pairs of fog-real image pairs. The RSID dataset consists of synthetic remote sensing fog images, containing 1000 pairs of fog-real image pairs. RSID is based on a large amount of fog parameter statistics, where ambient light A∈[0.7,1], medium transmission t∈[0.25,0.65], and fog thickness s∈[0.35,0.75]. The RSID dataset is used as the augmented image set, and the NID dataset is used as the target image set. A channel transfer method is applied to the RSID dataset.

[0050] S1.2: The channel migration method is used on the augmented image set to generate an enhanced image set by calculating the gamma value and performing gamma calibration.

[0051] More specifically, the formula for the gamma calibration is:

[0052]

[0053] Among them, O R,G,B and I R,G,B These represent the output and input pixel intensities, respectively, with γ being the gamma factor. R, G, and B represent the three color channels, with gamma values ​​γ. R,G,B This method calculates the gamma value transferred to the target image set. Unlike other domain transfer methods, this invention achieves efficient domain transfer without requiring model training. The colors of the RSID dataset are adjusted to make it more similar to the NID dataset. After channel transfer processing, the RSID images more closely match the RGB distribution of the NID dataset. This channel transfer method effectively expands the dataset's scope. Furthermore, this method introduces external datasets, enriching the dataset's diversity.

[0054] In one specific embodiment, the NID dataset and the RSID dataset were selected. The former is a dataset of hazy natural images, and the latter is a dataset of hazy remote sensing images. The NID dataset was used as the target domain, and the RSID dataset was used for domain transfer. We first calculated the gamma values ​​γ of the three channels to be transferred to the target domain. R,G,B Next, we perform gamma calibration on the image RSID dataset that needs to be transferred, such as... Figure 3 As shown.

[0055] To reduce the contrast differences between different foggy datasets, the first 900 images from the NID dataset were selected as the training set, and the remaining 100 images were selected as the test set, which was named N100. The first 300 and first 900 images from the RSID dataset were transferred to the NID dataset for data augmentation. The first 900 images comprised the first 300 images. Channel transfer was performed on the first 300 RSID images to obtain a 300-image RSID augmented dataset. Following a 3:1 ratio of NID to RSID augmented datasets, the first 900 images from the NID dataset were combined with the 300 RSID augmented datasets to create the augmented dataset RSCT3:1, totaling 1200 images. Channel transfer was also performed on the first 900 RSID images to obtain a 900-image RSID augmented dataset. Following a 1:1 ratio of NID to RSID augmented datasets, the first 900 images from the NID dataset were combined with the 900 RSID augmented datasets to create the augmented dataset RSCT1:1, totaling 1800 images. RSCT3:1 and RSCT1:1 were used as the training sets.

[0056] S2: Training begins. Feature extraction is performed on the training image set using a dehazing model to obtain feature maps. Feature fusion is then performed on the feature maps using a feature fusion module to obtain dehazed images. The external loss of the model is calculated, and the model is trained using a preset strategy.

[0057] More specifically, the feature extraction is achieved by normalizing, padding, projecting, and calculating attention scores on the input feature map using a Dehaze Network.

[0058] More specifically, the feature fusion step in step S2 includes:

[0059] Channel attention weights for different feature maps are learned by using global average pooling.

[0060] The feature map is projected using a linear layer, and the feature map is fused by combining the channel attention weights with the projection.

[0061] In one specific embodiment, the Dehaze Network of the present invention is a network composed of a 5-layer U-Net structure, which trains the dehaze model by feeding the RSCT3:1 dataset and the RSCT1:1 dataset into the dehaze model respectively. The network includes two downsampling and upsampling operations. In the network, the input hazy image passes through the Dehaze Block module and the feature fusion module to finally obtain the output dehazed image.

[0062] In the Dehaze Module, the input feature maps are processed through normalization and padding layers, and then projected onto Q, K, and V (query, key, value) using linear layers to calculate attention scores. Furthermore, the Dehaze Block module applies multi-head self-attention within a window to handle relationships between image patches. This allows the dehazing network to consider the correlations between different locations while extracting features. Therefore, the model can more effectively learn structural information in the image, improving model performance.

[0063] Unlike the direct fusion of features in the traditional U-Net network structure, the feature fusion module learns channel attention weights for different feature maps using operations such as global average pooling. This allows the network to dynamically adjust the contributions of each feature map based on its importance. In the feature fusion module, assuming there are two feature maps f1 and f2, we use a linear layer to project f1 onto f'1. Then, we fuse f'1 using weights through the following additional short residuals, as shown below:

[0064] x = w1f′1 + w2f2 + f2

[0065] Here, w1 and w2 are feature weights, and x is the short residual fusion feature. The final short residual fusion stage fuses the two feature maps through linear combination, while introducing an additional short residual term. This design helps to preserve the information of the original input while incorporating the fused features. It allows the network to better utilize multi-scale and multi-level feature information, thereby improving the accuracy and robustness of the dehazing effect.

[0066] In the model's loss function, use the loss function. Train the model to obtain the loss outside the data:

[0067]

[0068] in, y is the dehazed image obtained during training, and y is the real haze-free image.

[0069] S3: Apply an internal enhancement strategy to the dehazed image, calculate the model's internal loss, train the model using a preset strategy, and end one training session;

[0070] More specifically, the preset strategy for model training in S2 and S3 is a cosine annealing strategy.

[0071] In one specific embodiment, the Adamw optimizer cosine annealing strategy is used to train the model, with the batch size and training period set to 8 and 3000, respectively.

[0072] More specifically, the steps of the internal enhancement strategy in step S3 include:

[0073] A weak enhancement operation is performed on the dehazed image obtained in S2 to generate the weak enhancement result.

[0074] Perform a strong enhancement operation on the weak enhancement result to generate a strong enhancement result;

[0075] Calculate the difference between the two results to obtain the model's internal loss.

[0076] More specifically, the formula for calculating the internal loss of the model in S2 is as follows:

[0077]

[0078]

[0079] Where α is a hyperparameter, s is the current training iteration, S is the total training iterations, and I... weak Represented as an image generated by weak enhancement from a real haze-free image, I aggr Based on the obtained I weak The image was generated through strong enhancement. (By...) Through self-supervised learning, we can obtain the similarity between data points, allowing the model to learn more relevant details and become more robust.

[0080] Regarding the internal data augmentation, to improve the model's ability to preserve detail and contrast in dehazed images, this invention proposes an internal data augmentation strategy. Building upon external data augmentation, it emphasizes attention to local image details, making the model more robust.

[0081] In one specific embodiment, the output result of external enhancement Internal enhancements are performed, specifically as follows:

[0082]

[0083] Where `weak()` represents a weaker data augmentation operation; in this example, weak augmentation `weak()` is random pruning. weak Indicated as according to The image is generated through weak enhancement. Next, we will... weak Performing a strong enhancement operation is specifically shown below:

[0084] I aggr =aggr(I weak )

[0085] I aggr Based on the previously obtained I weak The image generated by strong enhancement is represented by `aggr()`, where `aggr()` is a Gaussian blur operation in this example, with a Gaussian kernel size of (9, 21) and a standard deviation ranging from (0.1, 5). Finally, we calculate the Iggr() obtained after weak-strong enhancement. weak and, aggr The difference. This allows us to obtain the loss function within the dataset to optimize the model, such as... Figure 4 As shown.

[0086] S4: Repeat the above training steps until the entire training image set is trained, and the trained model is obtained;

[0087] S5: Evaluate the trained model using a test image set.

[0088] In one specific embodiment, to evaluate the performance of the present invention, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are used as evaluation metrics. The trained dehazing model obtained from the above steps is then tested on the N100 test set, and the test results are as follows. Figure 5As shown, the results are compared with other advanced dehazing methods. During the test, the input images to the test set were processed by the dehazing model to obtain clear, haze-free images, which showed improvements in image texture details and structural contrast. The images generated after processing by the above model on the test set exhibited high quality and superior visual effects.

[0089] A second aspect of the present invention provides an image dehazing method based on an external enhancer and an internal enhancer, which uses the trained dehazing model to dehaze the image to be dehazed, and obtains the dehazed image.

[0090] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A training method for an image dehazing model based on external and internal enhancers, characterized in that, Includes the following steps: Obtain a preset set of foggy and fog-free images, and divide the set into a training image set and a test image set according to a preset ratio; apply an external enhancement strategy to the training image set according to a preset method to obtain the enhanced image set as the training image set; Training begins by using a dehazing model to extract features from the training image set, obtaining feature maps. The feature fusion module then fuses the features in the feature maps to obtain dehazed images. The external loss of the model is calculated, and the model is trained using a preset strategy. An internal enhancement strategy is applied to the dehazed image, the internal loss of the model is calculated, and the model is trained using a preset strategy to end one training cycle. Repeat the above training steps until the entire training image set is trained, and you will get the trained model. The trained model was evaluated using a set of test images; The steps involved in applying external enhancement strategies to the image set include: The image set is divided into augmented image set and target image set; The channel migration method is used on the augmented image set to generate an enhanced image set by calculating the gamma value and performing gamma calibration. The steps for implementing internal enhancement strategies for dehazed images include: Perform a weak enhancement operation on the dehazed image to generate the weak enhancement result; Perform a strong enhancement operation on the weak enhancement result to generate a strong enhancement result; Calculate the difference between the two results to obtain the model's internal loss.

2. The training method for an image dehazing model based on an external enhancer and an internal enhancer according to claim 1, characterized in that, The image set is divided into two parts: the RSID dataset is used as the augmented image set, and the NID dataset is used as the target image set.

3. The training method for an image dehazing model based on external and internal enhancers according to claim 1, characterized in that, The formula for gamma calibration is: in, and These represent the output and input pixel intensities, respectively; γ is the gamma factor; and R, G, and B represent the three color channels and their gamma values, respectively. It is obtained by calculating the gamma value transferred to the target image set.

4. The training method for an image dehazing model based on external and internal enhancers according to claim 1, characterized in that, The feature extraction using a dehazing model on the training image set is achieved by normalizing, padding, projecting, and calculating attention scores on the input feature map using the DehazeNetwork dehazing network.

5. The training method for an image dehazing model based on external and internal enhancers according to claim 1, characterized in that, The step of using the feature fusion module to perform feature fusion on the feature map includes: Channel attention weights for different feature maps are learned by using global average pooling. The feature map is projected using a linear layer, and the feature map is fused by combining the channel attention weights with the projection.

6. The training method for an image dehazing model based on external and internal enhancers according to claim 1, characterized in that, The formula for calculating the internal loss of the computational model is as follows: in, It is a hyperparameter, where s is the current training iteration and S is the total training iterations. This represents an image generated by weak enhancement from a real, fog-free image. Based on what was obtained The image generated through strong enhancement.

7. The training method for an image dehazing model based on an external enhancer and an internal enhancer according to claim 1, characterized in that, The preset strategy for model training is the cosine annealing strategy.

8. An image dehazing method based on an external enhancer and an internal enhancer, wherein the method uses the model trained according to claims 1-7 to dehaze the image to be dehazed, thereby obtaining a dehazed image.

Images