Training method and defogging method of image defogging model based on external intensifier and internal intensifier
By adopting external and internal enhancement strategies in the image defogging model, the contrast difference of fogging image set is reduced, and combined with feature extraction and fusion modules, the problem of poor model robustness in the existing technology is solved, significantly improving the defogging effect and image quality.
Patent Information
- Application Number
- CN202410950703.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-16
AI Technical Summary
In the process of image defogging, the contrast difference between different fogging image sets is large, resulting in poor robustness of the model and the inability to effectively reduce the difference in color channels, affecting the defogging effect.
Using the image defogging model training method based on external enhancers and internal enhancers, the contrast difference is reduced through channel migration and gamma calibration, and combined with feature extraction and feature fusion modules, the model training is used to enhance the data set. Diversity and internal information.
The robustness and defogging performance of the model are improved, and the generated defogging images are improved in texture detail and contrast, suitable for applications of natural and remote sensing images.
Smart Images

Figure CN119068284A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image defogging, and more specifically, relates to a training method and a defogging method of an image defogging model based on an external enhancer and an internal enhancer. Background Art
[0002] In recent years, with the rapid development and widespread application of computer vision technology, high-quality image input has become the key foundation for many visual tasks. This has led to increasing attention to image dehazing tasks, especially in the fields of remote sensing image and natural image processing. Haze often causes a significant decrease in image contrast and clarity, making the boundaries of objects that were originally clear in color and outline blurred. This not only greatly affects the visual effect of the image, but also brings many challenges to the further application of the image. For example, in the field of remote sensing, satellite images and aerial photography images under the influence of haze will lose a lot of key information, thus affecting the accuracy and reliability of tasks such as terrain mapping, environmental monitoring and disaster assessment. Similarly, in the field of autonomous driving, if the images collected by the vehicle camera are interfered by haze, it may cause the autonomous driving system to misjudge the recognition of roads and obstacles, thereby affecting driving safety.
[0003] Therefore, developing effective image dehazing technology is of great significance to improving the application value of remote sensing images and natural images. By removing the impact of haze, the clarity and contrast of the image can be restored, making it more suitable for various computer vision tasks. This can not only improve the safety and reliability of the autonomous driving system, but also enhance the application effect of remote sensing images in environmental monitoring, urban planning and resource management.
[0004] Image dehazing technology has attracted a lot of research attention from scholars in recent years because it can improve the quality of images while improving the results of subsequent computer vision tasks. However, choosing different dehazing strategies to improve the dehazing effect according to different image features and haze conditions has always been a research difficulty and hotspot in the field of image dehazing. Therefore, researchers usually take a variety of methods to explore. On the one hand, traditional methods based on image processing improve visual clarity by enhancing the contrast of images and sharpening edges; on the other hand, methods based on physical models try to simulate the interaction between light and suspended particles in the atmosphere, thereby reversely deriving a clear image without haze. Other researchers are committed to developing a dehazing model based on deep learning, using a large number of haze images and clear image pairs as training data to allow the model to learn the complex mapping relationship from haze images to clear images.
[0005] The invention patent with the prior art publication number CN115330627A proposes an image defogging method, device, equipment and medium based on self-supervised learning, including: obtaining multiple clear and fog-free images; for each clear and fog-free image, using the atmospheric scattering model to process the clear and fog-free image according to multiple preset image processing angles to obtain multiple foggy images of the clear and foggy image; for each foggy image, using the foggy image and the clear and fog-free image corresponding to the foggy image as an image pair to obtain multiple image pairs; using the multiple image pairs to train the image defogging model to obtain the trained image defogging model; using the trained image defogging model to defog the image to be defogged. This scheme does not reduce the contrast difference between different foggy image sets, so that the augmented image set is close to the target image set in the color channel, and the robustness of the model is poor. Summary of the invention
[0006] The present invention provides a training method and a defogging method for an image defogging model based on an external enhancer and an internal enhancer in order to reduce the contrast difference between different foggy image sets, make the augmented image set close to the target image set in the color channel, and improve the robustness of the model.
[0007] A first aspect of the present invention provides a training method for an image defogging model based on an external enhancer and an internal enhancer, comprising the following steps:
[0008] Obtain a preset foggy-fog-free image set, and divide the image set for training and the test image set according to a preset ratio from the image set; perform an external enhancement strategy on the image set for training according to a preset method to obtain an enhanced image set as a training image set;
[0009] At the beginning of training, the defogging model is used to extract features from the training image set to obtain a feature map, and the feature fusion module is used to fuse the feature map to obtain a defogging image, and the model external loss is calculated, and the model training is performed using a preset strategy;
[0010] Performing an internal enhancement strategy on the dehazed image, calculating the internal loss of the model, performing model training using a preset strategy, and completing one training;
[0011] Repeat the above training steps until the entire training image set is trained to obtain a trained model;
[0012] The trained model is evaluated using the test image set.
[0013] Furthermore, the step of performing an external enhancement strategy on the image set includes:
[0014] Divide the image set to determine the augmented image set and the target image set;
[0015] The channel migration method is used on the augmented image set to generate the enhanced image set by calculating the gamma value and gamma calibration.
[0016] Furthermore, the image set is divided, the RSID data set is used as the augmented image set, and the NID data set is used as the target image set.
[0017] Furthermore, the formula for gamma calibration is:
[0018]
[0019] Among them, O R,G,B and I R,G,B Represents the output and input pixel intensities respectively, and γ is the gamma factor. R, G, and B represent the three color channels respectively, and the gamma value γ R,G,B It is obtained by calculating the gamma value of the image transferred to the target image set.
[0020] Furthermore, the feature extraction of the training image set using the dehazing model is achieved by normalizing, filling, projecting, and calculating the attention score of the input feature map using the DehazeNetwork dehazing network.
[0021] Furthermore, the step of using a feature fusion module to perform feature fusion on the feature map includes:
[0022] The channel attention weights of different feature maps are learned by using the global average pooling operation;
[0023] The feature map is projected using a linear layer, and the channel attention weight is combined with the projection to perform feature map fusion.
[0024] Furthermore, the step of performing an internal enhancement strategy on the dehazed image includes:
[0025] Perform a weak enhancement operation on the dehazed image to generate a weak enhancement operation result;
[0026] Performing a strong enhancement operation on the weak enhancement operation result to generate a strong enhancement operation result;
[0027] The gap between the two results is calculated to obtain the internal loss of the model.
[0028] Furthermore, the calculation formula for the internal loss of the calculation model is:
[0029]
[0030]
[0031] Among them, α is a hyperparameter, s is the current training number, S is the total training number, I weak It is represented as a picture generated by weak enhancement based on the real fog-free image, I aggr Based on the obtained I weak The image generated by strong enhancement.
[0032] Furthermore, the preset strategy for model training is a cosine annealing strategy.
[0033] The second aspect of the present invention provides an image defogging method based on an external enhancer and an internal enhancer, and utilizes the trained model to defog the defogged image to obtain a defogged image.
[0034] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0035] Based on the external dataset enhancement strategy, the present invention also enhances the internal part of the dataset, further mining more local effective information of the dataset. Among them, a data enhancement method based on channel migration is proposed from the outside of the dataset, aiming to reduce the contrast difference between different foggy datasets and make the augmented dataset close to the target dataset in color channel. The augmented dataset obtained by this method is more consistent with the distribution of the target dataset, and the model trained by using these augmented datasets is more robust and has better performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to make the purpose and technical solution of the present invention clearer, the present invention provides the following drawings and descriptions:
[0037] Figure 1 A flowchart provided for an embodiment of the present invention;
[0038] Figure 2 A method framework diagram provided for an embodiment of the present invention;
[0039] Figure 3 A flow chart of an external enhancement strategy for a data set provided by an embodiment of the present invention;
[0040] Figure 4 A flow chart of an internal enhancement strategy for a data set provided by an embodiment of the present invention;
[0041] Figure 5 A defogging effect diagram on a NID dataset provided by an embodiment of the present invention; DETAILED DESCRIPTION
[0042] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.
[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0044] The first aspect of the present invention provides a training method for an image defogging model based on an external enhancer and an internal enhancer, such as Figure 1 The flowchart of a training method of an image dehazing model based on an external enhancer and an internal enhancer is shown in FIG. Figure 2 The figure shows a framework diagram of the training method provided by the present invention, and the specific steps are as follows:
[0045] S1: obtaining a preset foggy-fog-free image set, dividing the image set for training and the test image set according to a preset ratio from the image set; performing an external enhancement strategy on the image set for training according to a preset method, and obtaining an enhanced image set as a training image set;
[0046] The specific process is:
[0047] S1.1: Divide the image set to determine the augmented image set and the target image set;
[0048] More specifically, the RSID dataset is used as the augmented image set, and the NID dataset is used as the target image set.
[0049] In a specific embodiment, the foggy-fog-free dataset includes the NID dataset and the RSID dataset. The NID dataset is a synthetic natural foggy image, which relies on the estimation of depth information. Therefore, the NID dataset first estimates the depth map of the natural image using a depth prediction algorithm, and then synthesizes the natural foggy image according to the fog simulation strategy. The NID dataset includes 1,000 pairs of foggy-real image pairs. The RSID dataset is a synthetic remote sensing foggy image, containing 1,000 pairs of foggy-real image pairs. RSID is based on a large amount of fog parameter statistics. Among them, the ambient light A∈[0.7,1], the medium transmittance t∈[0.25,0.65], and the fog thickness s∈[0.35,0.75]. The RSID dataset is used as the augmented image set, the NID dataset is used as the target image set, and the channel migration method is used for the RSID dataset.
[0050] S1.2: Use the channel migration method on the augmented image set, calculate the gamma value, and perform gamma calibration to generate the enhanced image set.
[0051] More specifically, the formula for the gamma calibration is:
[0052]
[0053] Among them, O R,G,B and I R,G,B Represents the output and input pixel intensities respectively, and γ is the gamma factor. R, G, and B represent the three color channels respectively, and the gamma value γ R,G,B It is obtained by calculating the gamma value of the image set migrated to the target image set. Unlike other domain migration methods, the present invention can achieve efficient domain migration without training models. The color of the RSID dataset is adjusted to make it more similar to the NID dataset. The RSID image after channel migration is more consistent with the NID dataset in RGB distribution. This channel migration method effectively broadens the number of datasets. In addition, this method introduces external datasets to enrich the diversity of datasets.
[0054] In a specific embodiment, the NID dataset and the RSID dataset are selected. The former is a natural image fog dataset, and the latter is a remote sensing image fog dataset. The NID dataset is used as the target domain, and the RSID dataset is used for domain migration. We first calculate the gamma values γ of the three channels that need to be migrated to the target domain. R,G,B ,Afterwards, we perform gamma calibration on the RSID dataset of images that need to be migrated, such as Figure 3 shown.
[0055] In order to reduce the contrast difference between different foggy datasets, the first 900 images of the NID dataset are selected as the training set, and the remaining 100 images are used as the test set, which is named N100. The first 300 images and the first 900 images in the RSID dataset are selected and migrated to the NID dataset for data augmentation. The first 900 images include the first 300 images. The first 300 RSID dataset images are migrated to obtain 300 RSID augmented datasets. According to the ratio of NID dataset to RSID augmented dataset 3:1, the first 900 images of the NID dataset are taken and synthesized with the 300 RSID augmented datasets to form an augmented dataset RSCT3:1, with a total of 1200 images. The first 900 RSID dataset images are migrated to obtain 900 RSID augmented datasets. According to the ratio of NID dataset to RSID augmented dataset 1:1, the first 900 images of the NID dataset are taken and synthesized with the 900 RSID augmented datasets to form an augmented dataset RSCT1:1, with a total of 1800 images. RSCT3:1 and RSCT1:1 are used as training sets.
[0056] S2: Training starts, using the defogging model to extract features from the training image set to obtain a feature map, using the feature fusion module to fuse the feature map to obtain a defogging image, calculating the model external loss, and training the model using a preset strategy;
[0057] More specifically, the feature extraction is achieved by normalizing, padding, projecting, and calculating the attention score of the input feature map using the Dehaze Network.
[0058] More specifically, the feature fusion step in step S2 includes:
[0059] The channel attention weights of different feature maps are learned by using the global average pooling operation;
[0060] The feature map is projected using a linear layer, and the channel attention weight is combined with the projection to perform feature map fusion.
[0061] In a specific embodiment, the dehaze network of the present invention is a network composed of a 5-layer U-Net structure, and the RSCT3:1 data set and the RSCT1:1 data set are respectively put into the dehaze model for training. The network includes two downsampling and upsampling. In the network, the input foggy image passes through the Dehaze Block module and the feature fusion module to finally obtain the output dehazed image.
[0062] The feature maps input in the Dehaze Module are respectively passed through the normalization layer and the padding layer, and then projected to Q, K, V (query, key, value) using a linear layer to calculate the attention score. In addition, the Dehaze Block module applies multi-head self-attention within the window to process the relationship between image blocks. It allows the dehazing network to take into account the association between different positions while extracting features. Therefore, the model can learn the structural information in the image more effectively and improve the model performance.
[0063] Unlike the direct fusion of the traditional U-Net network structure, the feature fusion module learns the channel attention weights of different feature maps by using operations such as global average pooling. In this way, the network can dynamically adjust the contribution of each feature map according to its importance. In the feature fusion module, assuming there are two feature maps, f1 and f2, we use a linear layer to project f1 to f'1. After that, we use the weights to fuse f'1 through the following additional short residuals, as shown below:
[0064] x=w1f′1+w2f2+f2
[0065] Among them, w1, w2 are feature weights, x is the short residual fusion feature, and the final short residual fusion stage fuses the two feature maps through linear combination, while introducing an additional short residual term. This design helps to retain the original input information while introducing the fused features. It enables the network to better utilize multi-scale and multi-level feature information, thereby improving the accuracy and robustness of the dehazing effect.
[0066] In the loss function of the model, use the loss function Train the model to get the loss external to the data:
[0067]
[0068] in, is the dehazed image obtained during training, and y is the real haze-free image.
[0069] S3: Performing an internal enhancement strategy on the dehazed image, calculating the internal loss of the model, performing model training using a preset strategy, and ending one training session;
[0070] More specifically, the preset strategy for model training in S2 and S3 is the cosine annealing strategy.
[0071] In a specific embodiment, the model is trained using the Adamw optimizer cosine annealing strategy, and the batch size and training epochs are set to 8 and 3000, respectively.
[0072] More specifically, the steps of the internal enhancement strategy in step S3 include:
[0073] Perform a weak enhancement operation on the dehazed image obtained by S2 to generate a weak enhancement operation result;
[0074] Performing a strong enhancement operation on the weak enhancement operation result to generate a strong enhancement operation result;
[0075] The gap between the two results is calculated to obtain the internal loss of the model.
[0076] More specifically, the calculation formula for calculating the internal loss of the model in S2 is:
[0077]
[0078]
[0079] Among them, α is a hyperparameter, s is the current training number, S is the total training number, I weak It is represented as a picture generated by weak enhancement based on the real fog-free image, I aggr Based on the obtained I weak The image generated by strong enhancement. Through self-supervised learning, we can obtain the similarities between the internal data, allowing the model to learn more relevant details and become more robust.
[0080] In terms of the internal data set, in order to improve the model's ability to preserve details and contrast in dehazed images, this paper proposes an internal data enhancement strategy, which emphasizes the focus on local details of the image on the basis of external enhancement of the data set, making the model more robust.
[0081] In a specific embodiment, the output result of the external enhancement Internal enhancements are performed, as follows:
[0082]
[0083] Wherein weak() represents a weak data enhancement operation. In the embodiment, weak enhancement weak() is random cropping. weak Expressed as based on Weakly enhance the generated image. Next, we will I weak Perform strong enhancement operation, which is specifically expressed as follows:
[0084] I aggr =aggr(I weak )
[0085] I aggr Based on the previously obtained I weak The image generated by the strong enhancement is aggr(), which represents the strong enhancement operation. In the embodiment, the strong enhancement operation aggr() is Gaussian blur, where the Gaussian kernel convolution kernel size is (9, 21) and the standard deviation range is (0.1, 5). Finally, we calculate the I obtained after weak-strong enhancement weak and, aggr The gap between the two is thus obtained. The loss function within the data set is used to optimize the model, such as Figure 4 shown.
[0086] S4: Repeat the above training steps until the entire training image set is trained to obtain a trained model;
[0087] S5: Evaluate the trained model using the test image set.
[0088] In a specific embodiment, in order to evaluate the performance of the present invention, the present invention uses peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as evaluation indicators. The trained defogging model obtained by the above steps is tested on the N100 test set. The test results are as follows: Figure 5As shown. Compared with other currently advanced defogging methods. During the test, the images input into the test set were processed by the defogging model to obtain fog-free and clear images, and the defogging images were improved in image texture details and structural contrast. The images generated by the test set after being processed by the above model have higher quality and better visual effects.
[0089] A second aspect of the present invention provides an image defogging method based on an external enhancer and an internal enhancer, wherein the trained defogging model is used to defog the defogged image to obtain a defogged image.
[0090] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A training method for an image dehazing model based on an external enhancer and an internal enhancer, characterized in that: The steps include: Obtain a preset foggy-fog-free image set, and divide the image set for training and the test image set according to a preset ratio from the image set; perform an external enhancement strategy on the image set for training according to a preset method to obtain an enhanced image set as a training image set; At the beginning of training, the defogging model is used to extract features from the training image set to obtain a feature map, and the feature fusion module is used to fuse the feature map to obtain a defogging image, and the model external loss is calculated, and the model training is performed using a preset strategy; Performing an internal enhancement strategy on the dehazed image, calculating the internal loss of the model, performing model training using a preset strategy, and completing one training; Repeat the above training steps until the entire training image set is trained to obtain a trained model; The trained model is evaluated using the test image set.
2. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 1, characterized in that: The step of performing an external enhancement strategy on the image set comprises: Divide the image set to determine the augmented image set and the target image set; The channel migration method is used on the augmented image set to generate the enhanced image set by calculating the gamma value and gamma calibration.
3. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 2, characterized in that: The image set is divided, the RSID data set is used as the augmented image set, and the NID data set is used as the target image set.
4. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 2, characterized in that: The formula for the gamma calibration is: Among them, O R,G,B and I R,G,B Represents the output and input pixel intensities respectively, γ is the gamma factor, R, G, B represent the three color channels respectively, and the gamma value γ R,G,B It is obtained by calculating the gamma value of the image transferred to the target image set.
5. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 1, characterized in that: The feature extraction of the training image set using the dehazing model is achieved by normalizing, padding, projecting, and calculating the attention score of the input feature map using the DehazeNetwork dehazing network.
6. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 1, characterized in that: The step of using a feature fusion module to perform feature fusion on the feature map comprises: The channel attention weights of different feature maps are learned by using the global average pooling operation; The feature map is projected using a linear layer, and the channel attention weight is combined with the projection to perform feature map fusion.
7. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 1, characterized in that: The step of performing an internal enhancement strategy on the dehazed image comprises: Perform a weak enhancement operation on the dehazed image to generate a weak enhancement operation result; Performing a strong enhancement operation on the weak enhancement operation result to generate a strong enhancement operation result; The gap between the two results is calculated to obtain the internal loss of the model.
8. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 1, characterized in that: The calculation formula for the internal loss of the calculation model is: Among them, α is a hyperparameter, s is the current training number, S is the total training number, I weak It is represented as a picture generated by weak enhancement based on the real fog-free image, I aggr Based on the obtained I weak The image generated by strong enhancement.
9. The training method of an image defogging model based on an external enhancer and an internal enhancer according to claim 1, characterized in that: The preset strategy for model training is the cosine annealing strategy.
10. An image defogging method based on an external enhancer and an internal enhancer, the method adopting the model trained according to claims 1-9, using the model to defog an image to be defogged to obtain a defogged image.
Citation Information
Patent Citations
Template-based image enhancement method and device
CN112991236A
Pancreatic tumor classification method based on local background augmentation and multichannel transfer learning
CN113052824A
Model training method and device, equipment and storage medium
CN113159329A
Aerial image defogging method based on boundary constraint and color correction
CN113313702A
Noise-containing label image identification method based on adaptive category equalization
CN116912568A
