Parameter adaptive image dust removal method based on degradation degree

Abstract

The present invention relates to the field of computer vision technology, and in particular to a parameter-adaptive image dust removal method based on degradation degree, which mainly solves the technical problem that the existing dust image restoration method cannot be applied to dust images with uneven degradation degree. The parameter-adaptive image dust removal method based on degradation degree includes: S1. Obtaining a degradation image t 0 ; S2. Construct the PAM module; S3. Insert the PAM module into the static dust removal model to obtain a dynamic model with adaptive parameters and train it until convergence; S4. Input the dust image and the corresponding degradation map into the trained dynamic model and output the dust removal image. This method uses the degradation map area to represent the non-uniform degradation degree of the dust image, and then uses the degradation map as a priori to adaptively adjust the parameters of the static dust removal model through the PAM module, so that the dust image can be restored according to the degradation distribution of different areas of the image. This can adapt to dust images with uneven concentrations and have a better dust removal effect.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision, and in particular to a parameter adaptive image dust removal method based on degradation degree. Background Art

[0002] There is a lot of dust in scenes such as mining, workshop processing, and construction. Dust scatters and absorbs light, causing complex degradation of scene images such as reduced contrast and clarity, color distortion, and loss of a large amount of scene feature information, resulting in poor imaging effects. This not only limits workers' work efficiency, but also easily causes safety accidents and threatens workers' lives. Therefore, dust image restoration, as a prerequisite for high-level computer vision such as dust environment target recognition and scene analysis, plays a vital role in realizing intelligent monitoring of dust environments and unmanned production.

[0003] Existing dust image restoration is generally based on static dust removal models, such as FFA-Net, DMPHN and DWT-FFCGAN. This static model is suitable for dust images with uniform concentration, but for dust images with uneven concentration, the processing performance is greatly reduced and it is difficult to meet actual needs. Summary of the invention

[0004] In order to overcome the technical defect that the existing dust image restoration method is not applicable to dust images with uneven concentration, the present invention provides a parameter adaptive image dust removal method based on degradation degree.

[0005] The parameter-adaptive image dust removal method based on degradation degree provided by the present invention comprises the following steps in sequence:

[0006] S1. Obtaining a degradation map t0 of the dust image;

[0007] S2. Construct a PAM module, wherein the PAM module takes the degradation map t0 and the static feature map F as input and uses a dynamic parameter block as the main structure, wherein the dynamic parameter block is composed of a convolutional layer conv and an activation function δ, and the expression of the dynamic parameter block is:

[0008] ;

[0009] Wherein, γ and β represent the dynamic parameters output by the dynamic parameter block;

[0010] The dynamic parameter block converts the input degradation map t0 into dynamic parameters γ and β through multi-layer convolution. The PAM module performs spatial feature transformation on the input static feature map F through the dynamic parameters γ and β to obtain the dynamic feature map F d , the calculation formula is:

[0011] ;

[0012] S3. Inserting the PAM module into the static dust removal model to obtain a dynamic model with adaptive parameters and training until convergence;

[0013] S4. Input the dust image and the corresponding degradation image into the trained dynamic model, and output the dust-free image.

[0014] Optionally, the static dust removal model is FFA-Net, the dynamic model formed in step S3 is PAM-FFA, and the data set is MC-NH-HAZE;

[0015] The definition of dust image in MC-NH-HAZE is:

[0016] ;

[0017] Among them, I hazy represents the dust image, and I hazy Two polarization images with perpendicular directions to each other;

[0018] The definition of the clear image in MC-NH-HAZE is:

[0019] ;

[0020] Among them, I gt Indicates a clear image. and I gt Two polarization images with perpendicular directions to each other.

[0021] Optionally, the expression of the degradation graph t0 is:

[0022] ;

[0023] Where x represents the pixel position;

[0024] I represents polarization and image, and the expression of I is: , I ∥ and I ⊥ Represents two polarization images with perpendicular directions acquired by the polarization camera;

[0025] ΔI represents the polarization difference image, and the expression of ΔI is: ;

[0026] P d Represents the target polarization degree, P d The expression is:

[0027] , C w represents the calculation of covariance on the pixel neighborhood w, and J represents the target reflected light without particle scattering;

[0028] P b Represents the polarization degree of backscattered light, P b The expression is:

[0029] , patch is the area with the largest dust concentration in the polarization image;

[0030] A ∞ Represents the backscattered light value at infinity, A ∞ The expression is:

[0031] .

[0032] Optionally, the degradation graph t0 is smoothed after being calculated to obtain t0′, and t0′ is used to replace t0 as the degradation graph in step S1.

[0033] Optionally, the solution of t0′ can be simplified to the optimal solution of the following formula according to the cutout formula:

[0034] ;

[0035] Where L represents the Laplace matrix and λ represents the regularization parameter.

[0036] Optionally, in step S3, the loss function used in training is:

[0037] ;

[0038] Among them, L Total is the loss function during PAM-FFA training, λ RL , PL and λ TV is the calculation coefficient, L RL is the reconstruction loss function, L PL is the perceptual loss function, L TV is the total change loss function;

[0039] ;

[0040] , Φ represents the VGG-16 network, i represents the pixel of the image, N represents the total number of pixels in the image, j represents the number of layers in the VGG-16 network, and K represents the total number of layers in the VGG-16 network;

[0041] .

[0042] Optionally, the static dust removal model is DMPHN, the dynamic model formed in step S3 is PAM-DMPHN, and the data set is NH-HAZE.

[0043] Optionally, the static dust removal model is DWT-FFC GAN, the dynamic model formed in step S3 is PAM-DWT-FFC, and the data set is Combined.

[0044] Optionally, DehazeNet is first used to estimate the degradation map to form an estimated value, and then the estimated value is normalized, and the pixel values ​​of the degradation map are mapped to the [0, 1] interval, and finally the normalized result is multiplied by 255 to obtain the degradation map t0.

[0045] Optionally, the activation function δ is a Leaky ReLU function.

[0046] The technical solution provided by the present invention has the following advantages compared with the prior art:

[0047] The parameter adaptive image dust removal method based on degradation degree provided by the present invention represents the non-uniform degradation degree of the dust image through the degradation map area, and then uses the degradation map as a priori to adaptively adjust the parameters of the static dust removal model through the PAM module, so that the dust image can be restored according to the degradation distribution of different areas of the image. In this way, it can adapt to dust images with uneven concentration and have a better dust removal effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 A structural block diagram showing a PAM module in an embodiment of the present invention;

[0051] Figure 2 A flow chart showing the method of obtaining a degradation map t0 by using polarization information in an embodiment of the present invention;

[0052] Figure 3 A flow chart showing how to obtain a degradation graph t0 using a peer-to-peer network in an embodiment of the present invention;

[0053] Figure 4A schematic diagram showing the structure after the PAM module is inserted into three static dust removal models in an embodiment of the present invention;

[0054] Figure 5 A diagram showing the qualitative comparison results of PAM-FFA in the embodiments of the present invention;

[0055] Figure 6 A diagram showing the qualitative comparison results of PAM-DMPHN in an embodiment of the present invention;

[0056] Figure 7 A diagram showing the qualitative comparison results of PAM-DWT-FFC in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0059] Combine the following Figures 1 to 7 The specific embodiments of the present invention are described in detail.

[0060] This embodiment provides a parameter adaptive image dust removal method based on degradation degree, including steps S1 to S4.

[0061] S1. Obtain the degradation map t0 of the dust image and the static feature map F output by the static dust removal model.

[0062] Two methods for obtaining the degradation graph t0 are provided here.

[0063] The first one uses polarization information to obtain the degradation map t0.

[0064] Specifically, the expression of the degradation graph t0 is:

[0065] ;

[0066] Where x represents the pixel position;

[0067] I represents polarization and image, and the expression of I is: , I ∥ and I ⊥ Represents two polarization images with perpendicular directions acquired by the polarization camera;

[0068] ΔI represents the polarization difference image, and the expression of ΔI is: ;

[0069] P d Represents the target polarization degree, P d The expression is:

[0070] , C w represents the calculation of covariance on the pixel neighborhood w, and J represents the target reflected light without particle scattering;

[0071] P b Represents the polarization degree of backscattered light, P b The expression is:

[0072] , patch is the area with the largest dust concentration in the polarization image;

[0073] A ∞ Represents the backscattered light value at infinity, A ∞ The expression is:

[0074] .

[0075] It should be noted that in order to obtain P d , assuming that the local area P of the object d Since the degradation map t0 depends on the scene depth and the atmospheric attenuation coefficient, the target reflected light J without particle scattering is determined by the characteristics of the scene target itself, so let w represent a certain pixel area, then t0 and J are statistically independent when they are within the pixel area w, that is, Since there is noise in practical applications, P d The solution can be converted into solving an optimization problem.

[0076] It should be noted that the derivation process of the expression of the degradation graph t0 is as follows:

[0077] The degradation process of dust image can be expressed as formula:

[0078] (1);

[0079] Among them, I d Represents the direct transmission component, I b represents the backscattering component;

[0080] From formula (1), we can get:

[0081] (2);

[0082] By transforming formula (2), we can get:

[0083] (3);

[0084] By transforming formula (3), we can get:

[0085] (4);

[0086] The image polarization degree DOP can be expressed as the formula:

[0087] (5);

[0088] According to formula (5), we can get:

[0089] (6);

[0090] From formula (6), we can get:

[0091] (7);

[0092] Substituting formula (7) and (1) into formula (4), we can obtain the expression of the degradation graph t0.

[0093] Furthermore, the degradation map t0 is smoothed after being calculated to obtain t0′, and t0′ is used to replace t0 as the degradation map in step S1. Since the degradation map t0 calculated by formula (1) has a rough edge and has obvious block effects, it can be optimized by smoothing.

[0094] Specifically, according to the cutout formula, the solution of t0′ is simplified to the optimal solution of the following formula:

[0095] ;

[0096] Where L represents the Laplace matrix and λ represents the regularization parameter.

[0097] The final t0′ is as follows Figure 2 As shown in the figure, it can be seen that due to the influence of dust concentration and imaging distance changes, the more serious the degradation area is, the smaller the value of t0′ is, and the darker the color of the corresponding area of ​​the image is.

[0098] The second method is to use an end-to-end network to obtain the degradation graph t0.

[0099] Specifically, DehazeNet is first used to estimate the degradation map to form an estimated value, and then the estimated value is normalized and the pixel value of the degradation map is mapped to the [0,1] interval. Finally, the normalized result is multiplied by 255 to obtain the degradation map t0. DehazeNet is an existing algorithm proposed by Cai et al., such as Figure 3As shown, the degradation map estimated by the algorithm can reasonably estimate different degradation areas. In order to amplify the differences between different degradation areas, this embodiment performs the aforementioned feature scaling on the estimated value. The processed degradation map t0 enables the adaptive mechanism to better adaptively adjust the restoration model parameters according to the degree of degradation of different areas of the image.

[0100] S2. Construct the PAM module. The PAM module takes the degradation map t0 and the static feature map F as input and uses the dynamic parameter block as the main structure. The dynamic parameter block consists of the convolution layer conv and the activation function δ. The expression of the dynamic parameter block is:

[0101] ;

[0102] Among them, γ and β represent the dynamic parameters output by the dynamic parameter block;

[0103] The dynamic parameter block converts the input degradation map t0 into dynamic parameters γ and β through multi-layer convolution. The PAM module transforms the input static feature map F through the dynamic parameters γ and β to obtain the dynamic feature map F. d , the calculation formula is:

[0104] .

[0105] Specifically, Figure 3 As shown in Figure 1, the input of the PAM module consists of two parts: the static feature map and the degradation map t0. The main structure of the PAM module is the dynamic parameter block.

[0106] Specifically, the activation function δ is a Leaky ReLU function.

[0107] S3. Inserting the PAM module into the static dust removal model to obtain a dynamic model with adaptive parameters and training it until convergence.

[0108] It is easy to understand that, corresponding to different static dust removal models, the data sets used for training are different, the loss functions used for training are also different, and the methods of obtaining the degradation graphs are also different.

[0109] The following describes the three most commonly used static dust removal models: FFA-Net, DMPHN and DWT-FFC GAN.

[0110] The first one is that the static dust removal model is FFA-Net, the dynamic model formed is PAM-FFA, the data set is MC-NH-HAZE, the polarization information is used to obtain the degradation map t0, and the loss function used is L Total .

[0111] It should be noted that MC-NH-HAZE is a dataset created by this scheme.Total It is the loss function created by this scheme.

[0112] The definition of dust image in MC-NH-HAZE is:

[0113] ;

[0114] Among them, I hazy represents the dust image, and I hazy Two polarization images with perpendicular directions to each other;

[0115] The definition of the clear image in MC-NH-HAZE is:

[0116] ;

[0117] Among them, I gt Indicates a clear image. and I gt Two polarization images with perpendicular directions to each other.

[0118] Specifically, in order to construct MC-NH-HAZE, professional smoke-making equipment can be used in an outdoor area with good lighting to simulate the uneven dust scene under natural conditions, and a polarization camera can be used to capture polarization images containing linear polarization information. The captured image pixels are 588×488, and the camera pixel array surface is covered with a polarization array consisting of four polarizers at 0°, 45°, 90°, and 135°. According to experience, the polarization images at 0° and 90° are used to represent I ∥ and I ⊥ Two variables.

[0119] Specifically, Figure 4 As shown in the figure, in FFA-Net, the existing model is embedded in two ways, namely PAM-FFA-1 and PAM-FFA-2. PAM-FFA-1 performs adaptive adjustment of parameters after the entire G structure extracts and processes features. PAM-FFA-2 embeds the dynamic parameter layer interval after the B module.

[0120] Specifically, the loss function L used Total The expression is:

[0121] ;

[0122] Among them, λ RL , PL and λ TV is the calculation coefficient, L RL is the reconstruction loss function, LPL is the perceptual loss function, L TV is the total change loss function;

[0123] ;

[0124] , Φ represents the VGG-16 network, i represents the pixel of the image, N represents the total number of pixels in the image, j represents the number of layers in the VGG-16 network, and K represents the total number of layers in the VGG-16 network;

[0125] .

[0126] When implementing, select , , , and Set to 0.6 and 0.4 respectively.

[0127] The second one is that the static dust removal model is DMPHN, the dynamic model formed is PAM-DMPHN, the data set is NH-HAZE, and an end-to-end network is used to obtain the degradation map t0 of the dust image.

[0128] The third type is that the static dust removal model is DWT-FFC GAN, the dynamic model formed is PAM-DWT-FFC, the data set is Combined, and an end-to-end network is used to obtain the degradation map t0 of the dust image.

[0129] S4. Input the dust image and the corresponding degradation image into the trained dynamic model and output the dust-free image.

[0130] The following table compares the experimental results of the dynamic model and the existing static dust removal model:

[0131]

[0132] It should be noted that this embodiment uses the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) to quantitatively evaluate the difference between the inference result and the ground truth image. The higher the score, the better the restoration fidelity of the ground truth image. In addition, we also deploy the LPIPS perception indicator to evaluate the visual quality of the generated results. The lower the LPIPS, the closer it is to the real result.

[0133] In this embodiment, the PAM module is inserted into FFA-Net in two ways, and two loss functions are used as the optimization strategy during training, as shown in the table above. Figure 4We train the four models in end-to-end. Specifically, the initial learning rate is 0.0001 in all trainings. To speed up the training process, we choose Adam as the optimizer to train the model, where 𝛽1 and 𝛽2 are set to the default values ​​of 0.9 and 0.99. We train our models using Pytorch on a server equipped with NVIDIA RTX 3090. We test the performance of the models using the given test data in the MC-NH-HAZE, NH-HAZE, and Combined datasets.

[0134] As shown in the table above, it is obvious that the dynamic model with the proposed PAM module inserted has achieved significant improvement over the static model, indicating that the introduction of PAM has successfully improved the recovery performance of the dust removal model. The results of PAM-FFA and FFA-Net trained using different loss functions are given in the table above. The results show that the proposed loss function L Total The test results of show significant improvements for both static and dynamic models, and the performance of the two dynamic strategies is slightly different when using different loss functions. Total The qualitative comparison of Figure 5 As shown in Figure 2, we zoom in on the dust removal details of the blurred area of ​​the image to more intuitively demonstrate the advantages of our adaptive mechanism PAM. It can be seen that FFA-Net fails to completely remove the dust in these two images and brings serious artifacts to the last image. In contrast, our dust removal results are visually closer to the actual scene.

[0135] As shown in the above table, PAM-DMPHN outperforms DMPHN in all evaluation metrics, especially in the PSNR metric (improved by 0.401 dB). Figure 6 Quantitative comparison of the NH-HAZE dataset. To highlight the difference in details between PAM-DMPHN and DMPHN, we zoom in on the dust removal results in the heavily dusty area (as shown in the red box). After DMPHN dust removal, the bricks suffer from severe detail loss and the grass color is distorted, while our results are closer to the real situation.

[0136] As shown in the table above, PAM-DWT-FFC outperforms DWT-FFC GAN in both PSNR and SSIM metrics, which is considered to be the most advanced method in recent dust removal methods. PAM-DWT-FFC and DWT-FFC GAN have similar LPIPS values. Both methods have good dust removal effects, but the color fidelity of PAM-DWT-FFC is closer to the actual scene, such as Figure 7 shown.

[0137] The above is only a specific implementation of the present invention, which enables those skilled in the art to understand or implement the present invention. Although detailed descriptions are given with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments, and they should all be covered by the protection scope of the claims.

Claims

1. A parameter-adaptive image dust removal method based on degradation degree, characterized in that: The steps are as follows: S1. Obtaining a degradation map t0 of the dust image; The expression of the degradation graph t0 is: ； Where x represents the pixel position; I represents polarization and image, and the expression of I is: , I ∥ and I ⊥ Represents two polarization images with perpendicular directions acquired by the polarization camera; ΔI represents the polarization difference image, and the expression of ΔI is: ; P d Represents the target polarization degree, P d The expression is: , C w represents the calculation of covariance on the pixel neighborhood w, and J represents the target reflected light without particle scattering; P b Represents the polarization degree of backscattered light, P b The expression is: , patch is the area with the largest dust concentration in the polarization image; A ∞ Represents the backscattered light value at infinity, A ∞ The expression is: ； S2. Construct a PAM module, wherein the PAM module takes the degradation map t0 and the static feature map F as input and uses a dynamic parameter block as the main structure, wherein the dynamic parameter block is composed of a convolutional layer conv and an activation function δ, and the expression of the dynamic parameter block is: ； Wherein, γ and β represent the dynamic parameters output by the dynamic parameter block; The dynamic parameter block converts the input degradation map t0 into dynamic parameters γ and β through multi-layer convolution. The PAM module performs spatial feature transformation on the input static feature map F through the dynamic parameters γ and β to obtain the dynamic feature map F d , the calculation formula is: ； S3. Inserting the PAM module into the static dust removal model to obtain a dynamic model with adaptive parameters and training until convergence; The static dust removal model is FFA-Net, the dynamic model formed is PAM-FFA, and the dataset used is MC-NH-HAZE; The definition of dust image in MC-NH-HAZE is: ； Among them, I hazy represents the dust image, and I hazy Two polarization images with perpendicular directions to each other; The definition of the clear image in MC-NH-HAZE is: ； Among them, I gt Indicates a clear image. and I gt Two polarization images with perpendicular directions to each other; S4. Input the dust image and the corresponding degradation image into the trained dynamic model, and output the dust-free image.

2. The parameter-adaptive image dust removal method based on degradation degree according to claim 1, characterized in that: The degradation graph t0 is smoothed after being calculated to obtain t0′, and t0′ is used to replace t0 as the degradation graph in step S1.

3. The parameter-adaptive image dust removal method based on degradation degree according to claim 2, characterized in that: According to the cutout formula, the solution of t0′ is simplified to the optimal solution of the following formula: ； Where L represents the Laplace matrix and λ represents the regularization parameter.

4. The parameter-adaptive image dust removal method based on degradation degree according to claim 3, characterized in that: In step S3, the loss function used in training is: ； in, is the loss function during PAM-FFA training, , and To calculate the coefficient, To reconstruct the loss function, is the perceptual loss function, is the total change loss function; , and is the scale parameter, is the mean absolute error loss, is the mean square error loss; , Φ represents the VGG-16 network, i represents the pixel of the image, N represents the total number of pixels in the image, j represents the number of layers in the VGG-16 network, and K represents the total number of layers in the VGG-16 network. is the i-th pixel value of the image after dust removal output by the model, is the i-th pixel value of the clear image; , This is the dust-free image output by the model.

Images