Polarization image defogging method based on neural network

Abstract

The invention provides a polarization image defogging method based on a neural network, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a plurality of fogged polarization images, and generating a polarization image package; constructing a neural network defogging model; and based on the constructed neural network defogging model, carrying out defogging processing on the foggy polarization image in the generated polarization image packet to obtain a fogless image. According to the image defogging method, the deep learning and the polarization imaging technology are combined, the problem that the calculation complexity of the polarization imaging technology is high can be effectively solved, the overall generalization ability and defogging level are improved, and better robustness and high adaptability can be shown in the face of different types of fog and different degrees of haze.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for dehazing polarized images based on neural networks. Background Technology

[0002] In fields such as image analysis and computer vision, clear images are a prerequisite for accurate analysis. However, haze particles in the air have a strong scattering and absorption effect on light, significantly reducing image clarity and contrast, resulting in blurred and difficult-to-discern image information. This makes it difficult to effectively process and analyze the information in the image, severely restricting the accuracy and efficiency of subsequent processing and analysis. This problem not only hinders the effective operation of key applications such as intelligent traffic monitoring and environmental monitoring, but also has a wide-ranging impact on people's daily life quality and health and safety assessments. Therefore, it is crucial to restore and improve the optical imaging quality under the influence of haze to overcome the negative impact of haze on light, restore the true details of the image, and provide a guarantee for various image analysis tasks.

[0003] As a low-level computer vision task, image dehazing has been extensively studied in recent years, alongside high-level computer vision tasks such as classification and detection. Based on the methods and theories employed, image dehazing techniques can be divided into three types.

[0004] The first type is based on image enhancement. These methods use technical means to improve image contrast and color saturation to enhance image quality in hazy weather. This method typically does not rely on physical degradation models of the image but directly processes the image to achieve visual improvement.

[0005] The second method is based on a physical model. This method for image dehazing is primarily based on physical models of image degradation and atmospheric scattering models. In foggy weather conditions, light is scattered by particles as it travels through the atmosphere, leading to reduced image contrast and color shift. By simulating the reverse process, the original appearance of the image can be restored to some extent.

[0006] The third type is based on deep learning. Deep learning-based methods can be divided into two categories: one is to indirectly recover fog-free images by learning parameters in an atmospheric model through a network, and the other is to use a deep learning network to achieve end-to-end output of fog-free images from foggy images as input.

[0007] However, these methods still have shortcomings: 1. Among the aforementioned methods, the traditional polarization imaging dehazing algorithm is not fast enough in terms of processing speed, and cannot meet the requirements of real-time dehazing. Moreover, when the scene changes and weather conditions change significantly, the dehazing effect is unstable, i.e., the robustness is insufficient; 2. Most deep learning-based methods input a single image (RGB image) and use an atmospheric scattering model for training and testing. However, they rely excessively on the accuracy of the estimated medium transmission image and atmospheric light image. When the results are not accurate enough, the error will accumulate continuously during the training process, thus affecting the final result. That is, they have the problems of uncertainty and poor generalization ability. Summary of the Invention

[0008] To address the above problems, this invention provides a neural network-based method for dehazing polarized images. It acquires polarized images with haze and constructs a set of polarized image packets. By combining polarization imaging technology with convolutional neural network technology, it outputs dehazed images, restoring image information lost due to haze, improving image contrast, and enhancing the visibility of hazy images.

[0009] This invention provides a neural network-based method for dehazing polarization images, comprising: Acquire multiple fogged polarized images and generate a polarization image package; Construct a neural network dehazing model; Based on the constructed neural network dehazing model, the hazy polarized images in the generated polarization image package are dehazed to obtain hazy-free images.

[0010] As a further improvement of the present invention, the step of acquiring multiple foggy polarized images and generating a polarization image package includes acquiring multiple foggy polarized images with different polarization angles and generating a polarization image package from the acquired multiple foggy polarized images with different polarization angles.

[0011] As a further improvement of the present invention, after acquiring multiple foggy polarized images and generating a polarized image packet, the foggy polarized images in the polarized image packet are preprocessed using a polarization filter.

[0012] As a further improvement of the present invention, after acquiring multiple hazy polarized images and generating a polarization image packet, the degree of polarization and the polarization angle of the hazy polarized images are calculated:

[0013]

[0014] Where D is the degree of polarization and A is the polarization angle. Q ′、 U ′、 I ′ represents a component in the Stokes vector.

[0015] As a further improvement of the present invention, the step of acquiring multiple fogged polarized images with different polarization angles and generating a polarized image package from the acquired multiple fogged polarized images with different polarization angles includes acquiring three fogged polarized images of the same region, with polarization angles of 0°, 60° and 120° respectively.

[0016] As a further improvement of the present invention, the construction of the neural network dehazing model includes constructing a network model based on an improved AOD-Net.

[0017] As a further improvement of the present invention, the network model based on AOD-Net includes a K-estimating module and a dehazing image generation module, wherein the K-estimating module is responsible for generating images from... Medium estimate The parameters, the dehazing image generation module utilizes As its input adaptive parameter estimation ,in These are observed foggy images. It is the scene's radiation rate. Transmittance in the atmospheric scattering model and atmospheric light value A Integrated parameters.

[0018] As a further improvement of the present invention, the and It can be obtained through the following formula:

[0019]

[0020] Where b is a constant.

[0021] As a further improvement of the present invention, the network model based on AOD-Net adopts dynamic convolution.

[0022] As a further improvement of the present invention, the network model based on AOD-Net adopts dynamic convolution, including five convolutional layers and three connection layers. The Concat1 layer connects features from the DyConv1 and DyConv2 layers, the Concat2 layer connects features from the DyConv2 and DyConv3 layers, and the Concat3 layer connects features from the DyConv1, DyConv2, DyConv3 and DyConv4 layers.

[0023] As a further improvement of the present invention, the construction of the network model based on the AOD-Net improvement includes optimizing the loss function.

[0024] As a further improvement of the present invention, the optimized loss function is:

[0025] in, and This represents the average gray level of the two images. and The variance representing the gray levels of the two images. Represents covariance, and It is a constant.

[0026] As a further improvement of the present invention, the construction of the network model based on AOD-Net includes training the network model based on AOD-Net to obtain the trained network model based on AOD-Net, and based on the trained network model based on AOD-Net, performing dehazing processing on the hazy polarized images in the generated polarization image packet to obtain a hazy image.

[0027] As a further improvement of the present invention, the step of acquiring multiple foggy polarized images and generating a polarized image package includes acquiring foggy polarized images using a polarization camera mounted on a UAV.

[0028] This invention provides a neural network-based method for dehazing polarization images, which has at least one of the following advantages: 1. Combining deep learning with polarization imaging technology can effectively improve the high computational complexity of polarization imaging technology, enhance the overall generalization ability and dehazing level, and demonstrate better robustness and stronger adaptability when facing different types of fog and different degrees of haze. 2. Using multiple polarization images with different polarization angles can make full use of scene information, enrich feature information, and has a significant effect on solving uncertainty problems and improving generalization ability. 3. In the design of the neural network model, a multi-scale feature extraction method is adopted to capture the haze features at different scales, thereby improving the defogging effect. At the same time, a lightweight design is adopted to reduce the number of parameters and the amount of computation, thereby improving the computational efficiency. 4. Dynamic convolution is introduced to replace ordinary convolution in the network, enabling it to learn specific convolution kernel parameters for different input data, thereby giving the network a stronger feature representation ability. At the same time, the loss function is changed to improve the dehazing effect. Attached Figure Description

[0029] Figure 1 This is a schematic flowchart of a neural network-based polarization image dehazing method according to an embodiment of the present invention.

[0030] Figure 2 This is another schematic diagram of a neural network-based polarization image dehazing method according to an embodiment of the present invention.

[0031] Figure 3 This is a flowchart of the AOD-Net dehazing method for polarization image dehazing based on neural networks, according to an embodiment of the present invention.

[0032] Figure 4 This is a schematic diagram of the AOD-Net model structure.

[0033] Figure 5 This is a schematic diagram of the AOD-Net model structure after referencing the dynamic measuring tape.

[0034] Figure 6 This is a dehazing effect diagram of the polarization image dehazing method based on neural networks according to an embodiment of the present invention. Detailed Implementation

[0035] The following describes specific embodiments and appendices. Figure 1-6 The invention is described in detail so that those skilled in the art can more fully understand its purpose, features and effects.

[0036] Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. In the event of any discrepancy between the definitions of terms in this invention and their commonly understood meaning by one of ordinary skill in the art to which this invention pertains, the definitions set forth herein shall prevail.

[0037] This invention provides a neural network-based polarization image dehazing method, which improves upon existing dehazing methods and enhances the dehazing effect.

[0038] Example 1 As a specific embodiment of the present invention, this embodiment provides a polarization image dehazing method based on neural networks, referring to... Figure 1 The specific steps are as follows: S100: Acquire multiple fogged polarization images and generate a polarization image package; S200, Construct a neural network dehazing model; S300: Based on the constructed neural network dehazing model, the hazy polarized images in the generated polarization image package are dehazed to obtain a hazy image.

[0039] The present invention provides a neural network-based polarization image dehazing method that combines neural networks and polarization imaging technology to process foggy images. By combining polarization characteristics, it provides feature information for various complex scenarios and is applicable to a variety of scenarios, such as environmental monitoring in shallow sea areas, transportation, and public safety.

[0040] Example 2 As a specific embodiment of the present invention, this embodiment provides a polarization image dehazing method based on neural networks, referring to... Figure 1 , Figure 2 Based on Example 1, the following further applies: In step S100, a polarization camera mounted on a drone is used to acquire fog-covered polarization images. Specifically, when monitoring a certain area but it is affected by heavy fog, the polarization camera on the drone collects multiple fog-covered polarization images of the same area. These fog-covered polarization images are grayscale images. In this embodiment, three fog-covered polarization images of the same area are used for illustration. In other embodiments, the number of fog-covered polarization images of the same area can be greater than three, such as four or five; that is, there is no limit to the number of fog-covered polarization images acquired for each area to be monitored.

[0041] In one feasible embodiment, the drone is a DJI Matrice M300RTK and the polarization camera is an MS600Pro.

[0042] The three acquired hazy polarized images have polarization angles of 0°, 60°, and 120°, respectively, and are represented as I0°, I60°, and I120°. The three acquired hazy polarized images are combined into a polarization image package.

[0043] Polarization, as one of the fundamental physical properties of light waves, can provide information about the object under test that other optical properties cannot. Therefore, using polarization imaging technology to dehaze images has unique advantages over traditional methods. By employing three dehazed polarized images at different angles, the image can provide polarization data in multiple directions. This polarization information can provide rich information about the image structure and texture.

[0044] Furthermore, after generating the polarization image package in S100, the process also includes: preprocessing the hazy polarization images in the polarization image package to obtain a dataset for the network dehazing model constructed in S200.

[0045] Specifically, the foggy polarized images in the polarization image package are preprocessed using a polarization filter to reduce noise, enhance contrast, and calculate relevant parameters. In this embodiment, these parameters include the Stokes parameter, polarization angle, and degree of polarization. The dataset includes a training set, a validation set, and a test set, with a training set:validation set:test set ratio of 3:1:1. The polarization filter can improve the stability of the optical signal.

[0046] Furthermore, the Stokes vector (I,Q,U,V) is used to characterize the polarization information of light. In real natural environments, the circularly polarized component in polarized light is almost non-existent, so V=0 is set. Therefore, (I,Q,U) is used in the visible light polarization dehazing model.

[0047] With 0° as the reference direction, the light intensity I(a) at any polarization angle a is:

[0048] Commonly used polarization angles are 0°, 45°, 90°, and 135°, which can be represented by Stokes vectors as follows:

[0049]

[0050]

[0051] In this embodiment, the three foggy polarized images have polarization angles of 0°, 60°, and 120°, respectively, and need to be converted.

[0052] in, M The Mueller matrix is:

[0053] The last row and last column of the matrix represent non-linearly polarized components, both of which are 0. This is because non-linearly polarized components are extremely rare in nature and are easily masked by noise.

[0054] After transformation, the Stokes vector is:

[0055]

[0056]

[0057] Furthermore, the degree of polarization and the polarization angle are obtained:

[0058]

[0059] D is the degree of polarization (Dop), and A is the angle of polarization (Aop).

[0060] After calculation using the above formula, a fogged polarized image with polarization information is obtained, and the dataset contains polarization information.

[0061] The preprocessed training set is input into the neural network dehazing model constructed in S200 for training, resulting in a trained optimized neural network dehazing model. Then, the hazy polarized images from the polarization image packet are input into the trained optimized neural network dehazing model for dehazing, outputting a hazy-free image. Optionally, the hazy polarized images input into the trained optimized network dehazing model can be hazy polarized images from the test set.

[0062] This invention utilizes polarized images captured by a polarization camera mounted on a drone, which offers better scene applicability compared to traditional dehazing methods. Furthermore, by combining deep learning with polarization imaging technology, it can more effectively estimate scene depth and transmission rate, thereby improving the dehazing effect.

[0063] Example 3 As a specific embodiment of the present invention, this embodiment provides a polarization image dehazing method based on neural networks, referring to... Figure 1 , Figure 2 Based on Example 2, the following further applies: Building a neural network dehazing model in S200 includes constructing a network model based on an improved version of AOD-Net, specifically including: S201. Obtain the atmospheric scattering model The atmospheric scattering model in this embodiment is as follows:

[0064] in, These are observed foggy images. It is the scene radiometry, which can represent an ideal clean image, A and As a key parameter, A Indicates the global atmospheric light value. Transmittance is defined as:

[0065] in, β It is the atmospheric scattering coefficient. d ( x () is the distance between the object and the camera.

[0066] Due to separate estimation and A To recover a clean image, use the value. This will lead to the accumulation and amplification of errors, so further... and A Unify into one parameter To minimize the error, we have:

[0067]

[0068] Where b is a constant that defaults to 1.

[0069] By jointly estimating atmospheric light value and transmittance, the problem of the sky and white areas affecting the estimation of atmospheric light can be effectively avoided.

[0070] S202. Establish AOD-Net network Reference Figure 3 The AOD-Net network includes a K-estimating module and a dehazed image generation module. The K-estimating module is a crucial component of AOD-Net, responsible for estimating fog depth and relative level. In this embodiment, the K-estimating module is responsible for... Medium estimate The parameters are then processed by the dehazing image generation module, utilizing... As its input adaptive parameters are estimated .

[0071] Figure 4 This is a schematic diagram of the AOD-Net network structure. The model has five convolutional layers and three connection layers. The input image passes through a Conv1 layer with a 1x1 kernel and a Conv2 layer with a 3x3 kernel. The feature matrices obtained from these two layers are then fused into a Concat1 layer. Next, the image passes through a Conv3 layer with a 5x5 kernel, which fuses the feature matrices obtained from Conv2 and Conv3 layers into a Concat2 layer. Then, the image passes through a Conv4 layer with a 7x7 kernel, which fuses all the feature matrices obtained from the previous four convolutions into a Concat3 layer. Finally, the image passes through a Conv5 layer with a 3x3 kernel to obtain the output.

[0072] Convolutional layers with varying kernel sizes are used, and multi-scale features are formed by fusing filters of different sizes. Only three filters are used between each convolutional layer, reducing the number of parameters and computational cost while maintaining dehazing performance. Following the K-estimating module, the dehazed image generation module consists of an element-wise multiplication layer and multiple addition layers, finally processed by a formula... Generate a restored image.

[0073] S203, Optimize AOD-Net network This invention utilizes Dynamic Convolution (DynamicConv) instead of the ordinary convolution in the AOD-Net network. The main purpose of Dynamic Convolution is to improve network performance without significantly increasing computational load. Traditional Convolutional Neural Networks (CNNs) use the same kernel parameters for all inputs, while Dynamic Convolution can learn specific kernel parameters for different input data, thereby giving the network a stronger feature representation capability.

[0074] Traditional static sensor :

[0075] in, and These are the weight parameters and the bias vector. It is an activation function.

[0076] Dynamic sensor Aggregated linear functions :

[0077]

[0078]

[0079] Where π is the first... linear functions The weight vector, and as х It changes with the changes. Dynamic sensor It is an input linear model The combination of these factors results in a stronger fitting ability.

[0080] Figure 5 This is a schematic diagram of the model structure after adding dynamic convolution. The Concat1 layer connects the features from the DyConv1 and DyConv2 layers, the Concat2 layer connects the features from the DyConv2 and DyConv3 layers, and the Concat3 layer connects the features from the DyConv1, DyConv2, DyConv3, and DyConv4 layers.

[0081] Dynamic convolution provides more flexible and powerful feature extraction capabilities, which can improve the performance and adaptability of the model without significantly increasing the computational burden. Intermediate connections also compensate for the information loss during the convolution process.

[0082] S204. Optimize the loss function AOD-Net uses a single mean squared error loss function to measure the squared difference between predicted and true values, which can easily get trapped in local optima. This method uses SSIM+L2 as the loss function to improve dehazing performance.

[0083] Specifically, SSIM is a loss function widely used in computer vision. It evaluates image quality by comparing the brightness, contrast, and structural information of images. It is a metric used to measure the similarity between two images, and its expression is:

[0084] in, x and y There are two images; The focus is on brightness. c ( x , y The focus is on contrast. s ( x , y The focus is on the structure, which is represented in detail below:

[0085] in, and This represents the average gray level of the two images. The purpose of this is to prevent the denominator from being zero.

[0086]

[0087] in, and The variance representing the gray levels of the two images. Represents covariance, The purpose of this is to prevent the denominator from being zero.

[0088]

[0089] in, The purpose of this is to prevent the denominator from being zero.

[0090] Furthermore, If the value of is set to 1, then:

[0091] By using SSIM+L2 as the loss function, the accuracy of the model can be improved.

[0092] S205. Train the optimized model to obtain the trained network model based on AOD-Net. The preprocessed training set is input into the model for training. The network parameters are adjusted to minimize the difference between the predicted image output by the model and the real clear image, resulting in a trained network model based on AOD-Net.

[0093] Furthermore, it also includes: S206. Test the trained network model based on the improved AOD-Net and evaluate its performance. The preprocessed validation set is input into the trained network model based on AOD-Net. Through a series of feature extraction and fusion processes, the clear image after dehazing is finally output.

[0094] The performance of the model was evaluated by comparing the images before and after dehazing using PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and MSE (Mean Squared Error). The dehazing effects of the unmodified AOD-Net and the improved AOD-Net of this invention were also compared, as shown in Table 1. Figure 6 As shown.

[0095]

[0096] Among them, the higher the PSNR value, the better the quality of the restored image; the SSIM value is usually between 0 and 1, and the higher the value, the better the image quality; the lower the MSE value, the better the image quality.

[0097] As can be seen from Table 1, the PSNR, SSIM, and MSE indices of the present invention are all superior to those of the unmodified AOD-Net, demonstrating that the present invention has higher performance and effectiveness.

[0098] Figure 6 (a) The original image with fog. Figure 6 (b) is the dehazed image obtained using the unmodified AOD-Net. Figure 6 (c) is a dehazed image obtained by using AOD-Net. As can be seen from the comparison, the dehazed image obtained by using AOD-Net of the present invention is clearer, has higher fidelity, and has better effect.

[0099] Based on the evaluation structure, model fine-tuning and data augmentation can be used to optimize the model in order to achieve better defogging results.

[0100] Finally, the foggy polarized image is input into the trained AOD-Net-based improved network model to output a fog-free image.

[0101] Specifically, including: S301, Convert the fogged polarized image The feature maps are fed into the K-estimating module of the network model based on AOD-Net and then fed into it. ; S302. Using the dehazing image generation module, through the formula... Obtain and output a fog-free image.

[0102] The present invention provides a neural network-based polarization image dehazing method. This method uses a UAV equipped with a polarization camera to acquire multiple hazy polarization images of the same area. After image preprocessing, a set of polarization image packets is formed, and a neural network dehazing model is constructed. The polarization imaging technology is combined with the convolutional neural network technology. The model is trained and tested to achieve minimal image quality loss, outputs a dehazed image, restores the image information loss caused by haze, improves image contrast, and enhances the visibility of haze images.

[0103] The neural network-based polarization image dehazing method provided by this invention can be used to process foggy images to obtain clear images after dehazing. This method can be used in environmental monitoring and other scenarios to eliminate the interference and impact of heavy fog on monitoring results.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A method for dehazing polarization images based on neural networks, characterized in that, The method includes: Acquire multiple fogged polarized images and generate a polarization image package; Construct a neural network dehazing model; Based on the constructed neural network dehazing model, the hazy polarized images in the generated polarization image packet are dehazed to obtain hazy-free images.

2. The polarization image dehazing method based on neural networks according to claim 1, characterized in that, The step of acquiring multiple fogged polarized images and generating a polarized image package includes acquiring multiple fogged polarized images with different polarization angles and generating a polarized image package from the acquired multiple fogged polarized images with different polarization angles.

3. The polarization image dehazing method based on neural networks according to claim 2, characterized in that, After acquiring multiple foggy polarized images and generating a polarized image package, the foggy polarized images in the polarized image package are preprocessed using a polarization filter.

4. The polarization image dehazing method based on neural networks according to claim 2, characterized in that, After acquiring multiple hazy polarized images and generating a polarization image packet, the degree of polarization and the polarization angle of the hazy polarized images are calculated: Where D is the degree of polarization and A is the polarization angle. Q ′、 U ′、 I ′ represents a component in the Stokes vector.

5. The polarization image dehazing method based on neural networks according to claim 2, characterized in that, The process of acquiring multiple fogged polarized images with different polarization angles and generating a polarized image package from these images includes acquiring three fogged polarized images of the same region with polarization angles of 0°, 60°, and 120°, respectively.

6. The polarization image dehazing method based on neural networks according to claim 1, characterized in that, The construction of the neural network dehazing model includes building a network model based on an improved version of AOD-Net.

7. The polarization image dehazing method based on neural networks according to claim 6, characterized in that, The network model based on AOD-Net includes a K-estimating module and a dehazing image generation module. The K-estimating module is responsible for generating images from... Medium estimate The parameters, the dehazing image generation module utilizes As its input adaptive parameter estimation ,in These are observed foggy images. It is the scene's radiation rate. Transmittance in the atmospheric scattering model and atmospheric light value A Integrated parameters.

8. The polarization image dehazing method based on neural networks according to claim 7, characterized in that, The and It can be obtained through the following formula: Where b is a constant.

9. The polarization image dehazing method based on neural networks according to claim 7, characterized in that, The network model based on the improved AOD-Net uses dynamic convolution.

10. The method for dehazing polarization images based on neural networks according to claim 1, characterized in that, The process of acquiring multiple foggy polarized images and generating a polarized image package includes acquiring foggy polarized images using a polarization camera mounted on a UAV.

Images