Deep learning defogging method based on texture brightness feature enhancement

Abstract

The invention discloses a deep learning defogging method based on texture brightness feature enhancement, and the method comprises the steps: firstly constructing a deep learning defogging network based on texture brightness feature enhancement based on U-Net, inputting a fog image into the deep learning defogging network for texture brightness feature enhancement, and sequentially carrying out the down-sampling and up-sampling operations, and a generated defogged image result, namely a clear fogless image with higher quality, is obtained. According to the method, the deep learning defogging network filters important information in a frequency domain by using multi-scale feature fusion, a visual transformation module, a discrete wavelet transform upper sampling module and a discrete wavelet transform lower sampling module, enhances interactive fusion of underlying features, and provides a new brightness texture guiding module, so that the brightness texture guiding effect is improved. The feature fusion process of the encoder is guided by combining detail information and brightness information in the image, so that richer semantic information is reserved in the output image without depending on a specific fog image imaging relation, the defogged image can be reconstructed, and the defogged image is higher in quality and clearer.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical image defogging, and in particular relates to a deep learning defogging method based on texture brightness feature enhancement. Background Art

[0002] With the rapid development of artificial intelligence technology, drones have been used in agriculture, surveying and mapping, regional logistics, environmental monitoring, disaster relief, intelligent transportation and other fields. Drones can carry visual sensors such as digital cameras and video cameras to obtain real scene images and videos through visual perception. However, under severe weather conditions such as haze, drone aerial images have prominent problems such as low image quality and missing target details, which is not conducive to later processing of images such as target detection, target behavior analysis and scene semantic analysis. Therefore, optical image defogging technology is a current research hotspot.

[0003] At present, there are three main types of image defogging methods: (1) defogging methods based on prior information; (2) defogging methods based on image enhancement; (3) defogging methods based on deep learning. Among them, the defogging method based on prior information attempts to discover some prior laws between foggy images and clear images, and use these laws to guide the defogging method; when the quality of the foggy image is severely degraded, this method cannot be used because it cannot provide accurate prior information. The defogging method based on image enhancement enhances the performance of the haze image by enhancing the fog-free area in the image, but it will cause greater color distortion.

[0004] There are two methods for defogging technology based on deep learning: one is to calculate the fog-free image by learning the transmission map and atmospheric light intensity components in the fog image imaging relationship; the other is to directly estimate the defogging image through end-to-end design. The latter can avoid the shortcomings of inaccurate description of the fog image imaging relationship, does not depend on the specific fog image imaging relationship, and uses the deep learning network to first extract the different scale features of the image through downsampling, and then generate the defogging image through upsampling. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a deep learning defogging method based on texture brightness feature enhancement, which introduces enhanced information into the deep learning defogging network to obtain higher quality clear and fog-free images.

[0006] The technical solution adopted by the present invention is: a deep learning defogging method based on texture brightness feature enhancement, the specific steps are as follows:

[0007] S1. Build a deep learning dehazing network based on texture brightness feature enhancement based on U-Net;

[0008] The deep learning dehazing network includes: 1 texture brightness feature enhancement module, 6 visual transformation modules ViT-Block, 2 discrete wavelet transform downsampling modules DWT-Down, 1 neck network module BottleNeck, and 2 discrete wavelet transform upsampling modules DWT-Up.

[0009] S2, inputting the fog image into the texture brightness feature enhancement module, and outputting the fog image after texture brightness enhancement;

[0010] S3. Based on step S2, down-sampling and up-sampling operations are performed on the fog image and the texture brightness fusion feature map obtained after being processed by the texture brightness feature enhancement module in sequence to obtain a generated defogging image result.

[0011] Furthermore, the step S2 is specifically as follows:

[0012] S21, extracting texture features from the fog image input to the texture brightness feature enhancement module;

[0013] Set the input image F in It is a three-channel color image H×W×3. The improved adaptive threshold Canny algorithm is used to extract the edge texture information of the input image and obtain the texture mapping feature image F1 of the original image.

[0014] Among them, H represents the image height and W represents the image width.

[0015] First, a Gaussian kernel convolution operation is performed on the input image to smooth the input image and eliminate some abnormal high-frequency noise. The expression is as follows:

[0016]

[0017] Among them, I G represents the image after Gaussian smoothing, (x, y) represents the input image coordinates, (u, v) represents the coordinates of the Gaussian kernel, σ controls the size of the Gaussian convolution kernel, satisfying the kernel radius k≈3σ.

[0018] Then, the Sobel operator is used to calculate the gradient information of the smoothed image. The expression is as follows:

[0019]

[0020] Among them, * represents the convolution operation, D x and D y Represents the Sobel operator in the horizontal and vertical directions respectively. Using the gradient maps in the xy directions, the amplitude information of the gradient map is calculated. Phase information

[0021] Then, the strength of edge pixels is judged. First, for each pixel, its gradient direction is checked, and the amplitude information of the gradient of adjacent pixels is compared in the gradient direction, that is, the gradient amplitude of the current pixel is compared with the gradient amplitude of adjacent pixels in the gradient direction; if the gradient amplitude of the current pixel is not the local maximum, it is set to 0. The non-maximum suppression method is used for processing, retaining the local maximum value of the gradient and suppressing the gradient of the non-edge part. The expression is as follows:

[0022]

[0023] Among them, Z represents the gradient amplitude map after maximum suppression processing, and G represents the original gradient amplitude map.

[0024] Then double threshold division is used to divide the edge of the retained pixel area. The expression is as follows:

[0025]

[0026] Among them, T low and T high Respectively represent the minimum and maximum thresholds for determining whether the gradient amplitude is a strong or weak edge. If the current pixel gradient amplitude is greater than T high , it is determined as a strong edge; if the current pixel gradient amplitude is less than T high But it is greater than T low , it is judged as a weak edge; if the current pixel gradient amplitude is less than T low , it is judged as non-edge.

[0027] A method for local adaptive threshold adjustment is designed to refine the original global threshold setting through windowing operation. The expression is as follows:

[0028]

[0029] Among them, (m,n) represents the coordinates in the local window, g represents the gradient magnitude of each pixel in the window, and Dirac represents the Dirac function.

[0030] In the divided local window, the gradient amplitude histogram H(g) is calculated and the number of occurrences of each pixel amplitude in the current window is counted.

[0031] According to the gradient amplitude histogram H(g), a cumulative amplitude histogram C(g) is constructed, which represents the number of pixels with gradient amplitude values ​​greater than or equal to g in the current window. The expression is as follows:

[0032]

[0033] Based on the statistical information of the local histogram, the high and low thresholds used to judge the strength of the texture in the current window are calculated. The expression is as follows:

[0034] T high (i,j)=max{g|C(g)>η·C(0)} (7)

[0035] T low (i,j)=max{g|C(g)>η′·C(0)} (8)

[0036] Among them, T high (i,j) and T low (i, j) represents the high threshold and low threshold to be applied in the window of row i and column j in the image. C(0) represents the sum of all gradient histograms in the current window, and η and η′ represent proportional constants.

[0037] Starting from the strong edge pixel, trace the edge along the gradient direction. If the weak edge pixel is connected to the strong edge pixel, it is marked as an edge; if the weak edge pixel is not connected to any strong edge pixel, it is discarded.

[0038] Finally, a texture mapping feature map F1 with a size of H×W×1 is obtained.

[0039] S22, extracting brightness features from the fog image input to the texture brightness feature enhancement module;

[0040] The input image is processed to obtain the brightness prior information F2 of the three channels of the image.

[0041] First, multiple Gaussian kernels with different kernel sizes are convolved on the input image to obtain a Gaussian pyramid composed of local brightness information at different scales. The brightness maps of each layer are weighted fused to obtain the global brightness information L combined with Gaussian smoothing filters of different scales. multi , the expression is as follows:

[0042]

[0043] Among them, red, green, and blue represent the three channels of the input image. s Represents the sth Gaussian convolution kernel. s represents the fusion weight of the sth convolution kernel. k represents the Gaussian kernel radius.

[0044] According to the color characteristics of the image, a processing method is designed to use the color variance as the adjustment basis to adaptively adjust the weights of the RGB channels. The expression is as follows:

[0045]

[0046] Among them, Var(·) represents the variance of RGB color channels, [ω R ,ω G ,ωB ] represents the weight coefficient of RGB three channels.

[0047] Calculate the color contrast in the image and adjust the brightness. The expression is as follows:

[0048] L color (i,j)=ω R ·R(i,j)+ω G ·G(i,j)+ω B ·B(i,j) (11)

[0049] Among them, L color (i, j) represents the result of color channel information integration, R(i, j), G(i, j) and B(i, j) represent F in The point-by-point pixel values ​​of the R, G and B channels are obtained, and the channel information is integrated based on the weight coefficients of formula (10).

[0050] At the same time, considering the initial brightness information and the brightness adjustment based on color information, the brightness image generated by equation (11) is gamma corrected to enhance the image details, and the pixel value of the output image is normalized to [0, 255], as shown below:

[0051]

[0052] Among them, L gamma represents the result after Gamma correction, β represents the weight coefficient, γ represents the Gamma correction coefficient, L min and L max That is L gamma F2 represents the brightness prior feature map, and the size of the brightness prior feature map is H×W×1.

[0053] S23, fusing the texture features and brightness features obtained in steps S21 and S22, and splicing them with the input fog image, and then performing convolution block processing to output the fog image with enhanced texture brightness;

[0054] Integrate the edge and brightness feature maps of the input image, and use two learnable parameters ω1 and ω2 to obtain the fused feature map F3 = ω1·F1+ω2·F2.

[0055] Subsequently, the input image is concatenated with F3 in the channel dimension, and first undergoes a 1×1 convolution to resize the number of channels to 3, and then undergoes a deep convolution with a convolution kernel size of 5, a step size of 1, and a padding of 2 to generate a fusion information feature map F as the color information and texture brightness information. out , the expression is as follows:

[0056] Fout =DepthwiseConv 5×5 (Conv 1×1 (Concat(F3,F in ))) (14)

[0057] Furthermore, the step S3 is specifically as follows:

[0058] The down-sampling operation process includes: 3 ViT-Blocks and 2 DWT-Downs; the up-sampling operation includes: 3 ViT-Blocks and 2 DWT-Ups.

[0059] During the downsampling operation, the fog image and the texture brightness fusion feature map obtained after processing by the texture brightness feature enhancement module are first input into the first ViT-Block in the downsampling process to extract feature information of different scales, and then input into the first DWT-Down to reduce the image size after downsampling at a ratio of 1 / 2; the image processed by the first DWT-Down and the texture brightness fusion feature map are simultaneously input into the second ViT-Block to extract feature information of different scales, and then input into the second DWT-Down to reduce the image size after downsampling at a ratio of 1 / 2; then the image processed by the second DWT-Down and the texture brightness fusion feature map are simultaneously input into the third ViT-Block, and after three ViT-Blocks, feature information of different scales is extracted.

[0060] Then, the features extracted by the third ViT-Block in the downsampling process are input into BottleNeck for processing. Through frequency domain auxiliary processing, the output image features are obtained and enter the upsampling process.

[0061] During the upsampling process, the image features output after BottleNeck are first input into the first ViT-Block in the upsampling process to extract feature information of different scales, and then input into the first DWT-Up together with the image processed by the second DWT-Down in the downsampling process to increase the image size by a factor of 2; the image processed by the first DWT-Up is then input into the second ViT-Block to extract feature information of different scales, and then input into the second DWT-Up together with the image processed by the first DWT-Down in the downsampling process to increase the image size by a factor of 2; finally, the image processed by the second DWT-Up is input into the third ViT-Block to extract feature information of different scales, and the image size is restored to the original input size.

[0062] Finally, the upsampled image is fused with the original fog image to obtain the generated dehazed image result.

[0063] Furthermore, in step S1, the deep learning dehazing network is trained using a public data set, and a loss function is used for model training, as follows:

[0064] In order to make the predicted image closer to the real clear image at the pixel level, one of the loss functions L s The expression is as follows:

[0065]

[0066] in, and Y represent the predicted image and true label respectively.

[0067] Aligned in the spatial domain and introducing the learning of frequency signals in the network, the frequency domain L1 loss function is introduced to train the network, and the expression is as follows:

[0068]

[0069]

[0070] Where M and N represent the width and height of the amplitude spectrum respectively. out and A represent the amplitude spectra of the network predicted image and the target image respectively, P out and P represent the phase spectra of the network predicted image and the target image respectively. amp and L pha represent the amplitude and phase loss respectively.

[0071] The input image is introduced into the loss function to calculate the contrast loss. By comparing the features of the network output and the ground truth, the contrast loss function expression is as follows:

[0072]

[0073] Where a, p, and n represent the network prediction image, the true value image, and the input image, respectively. i represents the i-th layer output of feature extraction, ω i represents the weight coefficient of the i-th layer, d L1 Represents the L1 loss calculation result, and ε represents an arbitrarily small constant to prevent division by zero errors.

[0074] The total loss function expression is as follows:

[0075]

[0076] in, They represent the corresponding loss function weights respectively.

[0077] Furthermore, in step S1, the evaluation indicators of the defogging processing effect of the deep learning defogging network are PSNR and SSIM, which are as follows:

[0078] (1) Peak signal-to-noise ratio (PSNR);

[0079]

[0080] Among them, MAX represents the maximum possible value of the image pixel value, and MSE represents the mean square error of the pixel values ​​between the original image and the reconstructed image.

[0081] (2) Structural Similarity Index (SSIM);

[0082]

[0083] Among them, μ x and μ y Represent the average values ​​of the original image and the reconstructed image, σ x and σ y Represent the standard deviation of the original image and the reconstructed image, σ xy represents the covariance between the original image and the reconstructed image, and C1 and C2 represent constants used for stabilization.

[0084] Beneficial effects of the present invention: The method of the present invention first constructs a deep learning defogging network based on texture brightness feature enhancement based on U-Net, inputs the fog image into the deep learning defogging network for texture brightness feature enhancement, and performs downsampling and upsampling operations in sequence to obtain a generated defogging image result, that is, a clear and fog-free image of higher quality. The deep learning defogging network described in the method of the present invention utilizes multi-scale feature fusion, visual transformation module, discrete wavelet transform up- and downsampling modules to filter important information in the frequency domain and enhance the interactive fusion of underlying features. By proposing a new brightness texture guidance module, the feature fusion process of the encoder is guided by combining the detail information and brightness information in the image, thereby retaining richer semantic information in the output image, and is independent of the specific fog image imaging relationship, and can reconstruct the defogging image, and the defogging image is of higher quality and clearer. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 This is a flow chart of a deep learning dehazing method based on texture brightness feature enhancement of the present invention.

[0086] Figure 2 This is a structural diagram of the deep learning defogging network described in an embodiment of the present invention.

[0087] Figure 3 The flowchart of the fog image texture brightness feature enhancement in the texture brightness feature enhancement module in the embodiment of the present invention.

[0088] Figure 4 This is a visual comparison diagram of the defogging effect of the method of the present invention and the existing method in the embodiment of the present invention. DETAILED DESCRIPTION

[0089] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0090] like Figure 1 As shown, a flow chart of a deep learning defogging method based on texture brightness feature enhancement of the present invention is shown, and the specific steps are as follows:

[0091] S1. Build a deep learning dehazing network based on texture brightness feature enhancement based on U-Net;

[0092] like Figure 2 As shown, the deep learning dehazing network includes: 1 texture brightness feature enhancement module, 6 vision transformation modules (Vision Transformer Block, ViT-Block), 2 discrete wavelet transform down-sampling modules (Discrete Wavelet Transform Down-sampling, DWT-Down), 1 neck network module BottleNeck, and 2 discrete wavelet transform up-sampling modules (Discrete Wavelet Transform Up-Sampling, DWT-Up).

[0093] Among them, during the upsampling and downsampling process of the feature map, the ViT-Block module is used to extract feature information of different scales; during downsampling, the DWT-Down module separates the low-frequency frequency domain and high-frequency frequency domain information, and the low-frequency frequency domain feature information is passed to the next level ViT-Block; during upsampling, the high-frequency frequency domain features separated by the DWT-Down module are introduced to retain the high-frequency frequency domain information in the image sampling process.

[0094] S2, inputting the fog image into the texture brightness feature enhancement module, and outputting the fog image after texture brightness enhancement;

[0095] S3. Based on step S2, down-sampling and up-sampling operations are performed on the fog image and the texture brightness fusion feature map obtained after being processed by the texture brightness feature enhancement module in sequence to obtain a generated defogging image result.

[0096] like Figure 3 As shown, in this embodiment, the step S2 is specifically as follows:

[0097] S21, extracting texture features from the fog image input to the texture brightness feature enhancement module;

[0098] Set the input image F inIt is a three-channel color image H×W×3. The improved adaptive threshold Canny algorithm is used to extract the edge texture information of the input image and obtain the texture mapping feature image F1 of the original image.

[0099] Among them, H represents the image height and W represents the image width.

[0100] First, a Gaussian kernel convolution operation is performed on the input image to smooth the input image and eliminate some abnormal high-frequency noise. The expression is as follows:

[0101]

[0102] Among them, I G represents the image after Gaussian smoothing, (x, y) represents the input image coordinates, (u, v) represents the coordinates of the Gaussian kernel, σ controls the size of the Gaussian convolution kernel, satisfying the kernel radius k≈3σ. In this embodiment, three Gaussian kernels with σ of 1, 2, and 3 are used for multi-scale smoothing, and the weighted addition is performed to obtain the final result of the Gaussian kernel convolution operation.

[0103] Then, the Sobel operator is used to calculate the gradient information of the smoothed image. The expression is as follows:

[0104]

[0105] Among them, * represents the convolution operation, D x and D y Represents the Sobel operator in the horizontal and vertical directions respectively. Using the gradient maps in the xy directions, the amplitude information of the gradient map is calculated. Phase information

[0106] In order to refine the texture, remove pixels that are not real edges or have weak edge information, and judge the strength of edge pixels, first check the gradient direction of each pixel, and compare the amplitude information of the gradient of adjacent pixels in the gradient direction, that is, compare the gradient amplitude of the current pixel with the gradient amplitude of adjacent pixels in the gradient direction; if the gradient amplitude of the current pixel is not the local maximum, it is set to 0. In this way, the local maximum value of the gradient is retained and the gradient of the non-edge part is suppressed. The non-maximum suppression method is used for processing, and the expression is as follows:

[0107]

[0108] Among them, Z represents the gradient amplitude map after maximum suppression processing, and G represents the original gradient amplitude map.

[0109] Then double threshold division is used to divide the edge of the retained pixel area. The expression is as follows:

[0110]

[0111] Among them, T low and T high Respectively represent the minimum and maximum thresholds for determining whether the gradient amplitude is a strong or weak edge. If the current pixel gradient amplitude is greater than T high , it is determined as a strong edge; if the current pixel gradient amplitude is less than T high But it is greater than T low , it is judged as a weak edge; if the current pixel gradient amplitude is less than T low , it is judged as non-edge.

[0112] Since the setting of the threshold will greatly affect the judgment result of edge texture, different high and low thresholds need to be set for different types and sizes of input images for judgment. However, if the input is an image with obvious regional differences, the set global judgment threshold may lose its original effect in some areas of the image. To address this problem, a method of local adaptive threshold adjustment is designed. Through windowing operations, the original global threshold setting is refined to better adapt to more types of input images. The expression is as follows:

[0113]

[0114] Where (m,n) represents the coordinates in the local window, g represents the gradient magnitude of each pixel in the window, and Dirac represents the Dirac function.

[0115] In the divided local window, the gradient amplitude histogram H(g) is calculated and the number of occurrences of each pixel amplitude in the current window is counted.

[0116] According to the gradient amplitude histogram H(g), a cumulative amplitude histogram C(g) is constructed, which represents the number of pixels with gradient amplitude values ​​greater than or equal to g in the current window. The expression is as follows:

[0117]

[0118] According to the statistical information of the local histogram, the high and low thresholds used to judge the strength of the texture in the current window are calculated. C(0) represents the sum of all gradient histograms in the current window, and η and η′ are proportional constants.

[0119] T high (i,j)=max{g|C(g)>η·C(0)} (7)

[0120] T low (i,j)=max{g|C(g)>η′·C(0)} (8)

[0121] Among them, Thigh (i,j) and T low (i, j) represents the high threshold and low threshold to be applied in the window of the i-th row and j-th column divided in the image. In this embodiment, η and η′ are set to 0.1 and 0.05 respectively.

[0122] Starting from the strong edge pixel, trace the edge along the gradient direction. If the weak edge pixel is connected to the strong edge pixel, it is marked as an edge; if the weak edge pixel is not connected to any strong edge pixel, it is discarded.

[0123] Finally, a texture mapping feature map F1 with a size of H×W×1 is obtained.

[0124] S22, extracting brightness features from the fog image input to the texture brightness feature enhancement module;

[0125] In order to make the processed image retain the brightness distribution information of the original image as much as possible, the input image is processed to obtain the brightness prior information F2 of the three channels of the image.

[0126] First, convolution is performed on the input image using multiple Gaussian kernels with different kernel sizes, which can effectively reduce high-frequency noise while better retaining the edge information of the image. A Gaussian pyramid composed of local brightness information at different scales is obtained, and the brightness maps of each layer are weighted fused to obtain the global brightness information L combined with Gaussian smoothing filters of different scales. multi The expression is as follows:

[0127]

[0128] Among them, red, green, and blue represent the three channels of the input image. s represents the sth Gaussian convolution kernel. In this embodiment, S is 3, that is, three convolution kernels with σ values ​​of 8, 16, and 32 are used to perform large-scale smoothing on the input image. s represents the fusion weight of the sth convolution kernel. k represents the Gaussian kernel radius.

[0129] Considering that the color information of the image will also affect the contribution of brightness in different regions, a processing method is designed based on the color characteristics of the image. Using the variance of the color as the basis for adjustment, the weights of the RGB channels can be adaptively adjusted. The expression is as follows:

[0130]

[0131] Where Var(·) represents the variance of the RGB color channels. R ,ω G ,ω B ] represents the weight coefficient of RGB three channels.

[0132] In order to emphasize the brightness contribution of a specific color, the color contrast in the image can be calculated and the brightness can be adjusted. The expression is as follows:

[0133] L color (i,j)=ω R ·R(i,j)+ω G ·G(i,j)+ω B ·B(i,j) (11)

[0134] Among them, R(i,j), G(i,j) and B(i,j) represent F in The pixel values ​​of the R, G and B channels are integrated based on the weight coefficients of formula (10). color (i, j) represents the result after integrating the color channel information.

[0135] At the same time, considering the initial brightness information and the brightness adjustment based on color information, the brightness image generated by equation (11) is gamma corrected to enhance the image details, and the pixel value of the output image is normalized to [0, 255], as shown below:

[0136]

[0137] Among them, L gamma represents the result after Gamma correction, β represents the weight coefficient, γ represents the Gamma correction coefficient, L min and L max That is L gamma F2 represents the brightness prior feature map, and the size of the brightness prior feature map is H×W×1.

[0138] S23, fusing the texture features and brightness features obtained in steps S21 and S22, and splicing them with the input fog image, and then performing convolution block processing to output the fog image with enhanced texture brightness;

[0139] In order to integrate the edge and brightness feature maps of the input image, two learnable parameters ω1 and ω2 are used to obtain the fused feature map F3 = ω1·F1+ω2·F2.

[0140] Subsequently, the input image is concatenated with F3 in the channel dimension, and first undergoes a 1×1 convolution to resize the number of channels to 3, and then undergoes a deep convolution with a convolution kernel size of 5, a step size of 1, and a padding of 2 to generate a fusion information feature map F as the color information and texture brightness information. out , the expression is as follows:

[0141] F out =DepthwiseConv5×5 (Conv 1×1 (Concat(F3,F in ))) (14)

[0142] like Figure 2 As shown, in this embodiment, step S3 is specifically as follows:

[0143] The down-sampling operation process includes: 3 ViT-Blocks and 2 DWT-Downs; the up-sampling operation includes: 3 ViT-Blocks and 2 DWT-Ups.

[0144] During the downsampling operation, the fog image and the texture brightness fusion feature map obtained after processing by the texture brightness feature enhancement module are first input into the first ViT-Block in the downsampling process to extract feature information of different scales, and then input into the first DWT-Down to reduce the image size after downsampling at a ratio of 1 / 2; the image processed by the first DWT-Down and the texture brightness fusion feature map are simultaneously input into the second ViT-Block to extract feature information of different scales, and then input into the second DWT-Down to reduce the image size after downsampling at a ratio of 1 / 2; then the image processed by the second DWT-Down and the texture brightness fusion feature map are simultaneously input into the third ViT-Block, and after three ViT-Blocks, feature information of different scales is extracted.

[0145] Then, the features extracted by the third ViT-Block in the downsampling process are input into BottleNeck for processing. Through frequency domain auxiliary processing, the output image features are obtained and enter the upsampling process.

[0146] During the upsampling process, the image features output after BottleNeck are first input into the first ViT-Block in the upsampling process to extract feature information of different scales, and then input into the first DWT-Up together with the image processed by the second DWT-Down in the downsampling process to increase the image size by a factor of 2; the image processed by the first DWT-Up is then input into the second ViT-Block to extract feature information of different scales, and then input into the second DWT-Up together with the image processed by the first DWT-Down in the downsampling process to increase the image size by a factor of 2; finally, the image processed by the second DWT-Up is input into the third ViT-Block to extract feature information of different scales, and the image size is restored to the original input size.

[0147] Finally, the upsampled image is fused with the original fog image to obtain the generated dehazed image result.

[0148] In this embodiment, in step S1, the deep learning defogging network is trained using a public data set, and a loss function is used for model training, which is specifically as follows:

[0149] In order to restore the real high-quality fog-free image, one of the simplest methods is to make the predicted image closer to the real clear image at the pixel level. Then one of the loss functions L s The expression is as follows:

[0150]

[0151] in, and Y represent the predicted image and true label respectively.

[0152] In addition to alignment in the spatial domain, since the learning of frequency signals is introduced in the network, the frequency domain L1 loss function is considered to be introduced for network training. The expression is as follows:

[0153]

[0154]

[0155] Where M and N represent the width and height of the amplitude spectrum (phase spectrum) respectively. out and A represent the amplitude spectra of the network predicted image and the target image respectively, P out and P represent the phase spectra of the network predicted image and the target image respectively. amp and L pha represent the amplitude and phase loss respectively.

[0156] In order to improve the generalization ability of the model and avoid overfitting of the original input during the learning process, the input image is introduced into the loss function to calculate the contrast loss. By comparing the features of the network output and the ground truth, the network can be effectively guided to generate high-quality images that are closer to the ground truth, which makes the generated images visually closer to the real situation and richer in details.

[0157] The contrast loss function expression is as follows:

[0158]

[0159] Where a, p, and n represent the network prediction image, the true value image, and the input image, respectively. i represents the i-th layer output of feature extraction, ω i represents the weight coefficient of the i-th layer, d L1 Represents the L1 loss calculation result, and ε represents an arbitrarily small constant to prevent division by zero errors.

[0160] By minimizing L c, which can make the network effectively close to the true value and increase the distinction from the input image, thereby more effectively guiding the network to generate high-quality images that are closer to the ground truth.

[0161] The total loss function expression is as follows:

[0162]

[0163] in, They represent the corresponding loss function weights, which are set to 0.1, 0.1 in this embodiment.

[0164] In this embodiment, in step S1, the evaluation indicators of the defogging processing effect of the deep learning defogging network are PSNR and SSIM, which are as follows:

[0165] In order to compare the dehazing effects of different methods, the following two indicators are used:

[0166] (1) Peak Signal-to-Noise Ratio (PSNR);

[0167]

[0168] Where MAX represents the maximum possible value of the image pixel value (255 for an 8-bit image), and MSE represents the mean square error of the pixel values ​​between the original image and the reconstructed image.

[0169] (2) Structural Similarity Index (SSIM);

[0170]

[0171] Among them, μ x and μ y Represent the average values ​​of the original image and the reconstructed image, σ x and σ y Represent the standard deviation of the original image and the reconstructed image, σ xy represents the covariance between the original image and the reconstructed image, and C1 and C2 represent constants used for stabilization.

[0172] PSNR focuses more on the overall error level and is suitable for quantifying the global reconstruction error of the image, but it lacks in terms of perceptual quality. SSIM emphasizes the structural similarity of the image and can better reflect the human visual system's perception of image quality, so it is suitable for evaluating the perceptual quality of the image.

[0173] This embodiment also conducted a comparative verification and conducted a large number of experiments on the SOTS-Indoor and Dense-haze datasets to compare the dehazing effects of the method of the present invention with those of the existing methods. The results show that compared with other advanced dehazing methods, the method of the present invention has higher performance indicators such as PSNR and SSIM. The details are as follows:

[0174] (1) Using the SOTS-Indoor dataset, the training set contains 13,990 pairs of foggy and non-fog images, and the test set contains 500 pairs of foggy and non-fog images. The comparison results of the dehazing effect are shown in Table 1.

[0175] Table 1

[0176]

[0177] (2) Using the DenseHaze dataset, the comparison of dehazing effects on dense fog images is shown in Table 2.

[0178] Table 2

[0179]

[0180] The defogging effect of the method of the present invention and the main method can be compared as shown in the figure Figure 4 As shown in the figure, GT represents the real fog-free image.

[0181] In summary, the deep learning defogging network described in the method of the present invention utilizes multi-scale feature fusion, visual transformation module, discrete wavelet transform up- and down-sampling modules to filter important information in the frequency domain and enhance the interactive fusion of underlying features. By proposing a new brightness texture guidance module, the detail information and brightness information in the image are combined to guide the feature fusion process of the encoder, thereby retaining richer semantic information in the output image, and is independent of the specific fog image imaging relationship, so that the defogged image can be reconstructed, and the defogged image quality is higher and clearer.

[0182] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A deep learning dehazing method based on texture brightness feature enhancement, the specific steps are as follows: S1. Build a deep learning dehazing network based on texture brightness feature enhancement based on U-Net; The deep learning defogging network includes: 1 texture brightness feature enhancement module, 6 visual transformation modules ViT-Block, 2 discrete wavelet transform downsampling modules DWT-Down, 1 neck network module BottleNeck, 2 discrete wavelet transform upsampling modules DWT-Up; S2, inputting the fog image into the texture brightness feature enhancement module, and outputting the fog image after texture brightness enhancement; S3. Based on step S2, down-sampling and up-sampling operations are performed on the fog image and the texture brightness fusion feature map obtained after being processed by the texture brightness feature enhancement module in sequence to obtain a generated defogging image result.

2. According to the deep learning defogging method based on texture brightness feature enhancement according to claim 1, it is characterized in that: The step S2 is specifically as follows: S21, extracting texture features from the fog image input to the texture brightness feature enhancement module; Set the input image F in For a three-channel color image H×W×3, the improved adaptive threshold Canny algorithm is used to extract the edge texture information of the input image and obtain the texture mapping feature image F1 of the original image; Where H represents the image height and W represents the image width; First, a Gaussian kernel convolution operation is performed on the input image to smooth the input image and eliminate some abnormal high-frequency noise. The expression is as follows: Among them, I G represents the image after Gaussian smoothing, (x, y) represents the input image coordinates, (u, v) represents the coordinates of the Gaussian kernel, σ controls the size of the Gaussian convolution kernel, satisfying the kernel radius k≈3σ; Then, the Sobel operator is used to calculate the gradient information of the smoothed image. The expression is as follows: Among them, * represents the convolution operation, D x and D y Represents the Sobel operator in the horizontal and vertical directions respectively; using the gradient maps in the xy directions obtained respectively, the amplitude information of the gradient map is calculated Phase information Then, the strength of edge pixels is judged. First, for each pixel, its gradient direction is checked, and the amplitude information of the gradient of adjacent pixels is compared in the gradient direction, that is, the gradient amplitude of the current pixel is compared with the gradient amplitude of adjacent pixels in the gradient direction; if the gradient amplitude of the current pixel is not the local maximum, it is set to 0; the non-maximum suppression method is used for processing, the local maximum value of the gradient is retained and the gradient of the non-edge part is suppressed. The expression is as follows: Among them, Z represents the gradient amplitude map after maximum value suppression processing, and G represents the original gradient amplitude map; Then double threshold division is used to divide the edge of the retained pixel area. The expression is as follows: Among them, T low and T high Respectively represent the minimum and maximum thresholds for determining whether the gradient amplitude is a strong or weak edge. If the current pixel gradient amplitude is greater than T high , it is determined as a strong edge; if the current pixel gradient amplitude is less than T high But it is greater than T low , it is judged as a weak edge; if the current pixel gradient amplitude is less than T low , it is judged as non-edge; A method for local adaptive threshold adjustment is designed to refine the original global threshold setting through windowing operation. The expression is as follows: Where (m,n) represents the coordinates in the local window, g represents the gradient magnitude of each pixel in the window, and Dirac represents the Dirac function; In the divided local window, the gradient amplitude histogram H(g) is calculated, and the number of occurrences of each pixel amplitude in the current window is counted; According to the gradient amplitude histogram H(g), a cumulative amplitude histogram C(g) is constructed, which represents the number of pixels with gradient amplitude values ​​greater than or equal to g in the current window. The expression is as follows: Based on the statistical information of the local histogram, the high and low thresholds used to judge the strength of the texture in the current window are calculated. The expression is as follows: T high (i,j)=max{g|C(g)>η·C(0)} (7) T low (i,j)=max{g|C(g)>η′·C(0)} (8) Among them, T high (i,j) and T low (i, j) represents the high threshold and low threshold to be applied in the window of row i and column j in the image; C(0) represents the sum of all gradient histograms in the current window, and η and η′ represent proportional constants; Starting from the strong edge pixel, trace the edge along the gradient direction. If the weak edge pixel is connected to the strong edge pixel, mark it as an edge; if the weak edge pixel is not connected to any strong edge pixel, discard it. Finally, a texture mapping feature map F1 with a size of H×W×1 is obtained; S22, extracting brightness features from the fog image input to the texture brightness feature enhancement module; Process the input image to obtain the brightness prior information F2 of the three channels of the image; First, multiple Gaussian kernels with different kernel sizes are convolved on the input image to obtain a Gaussian pyramid composed of local brightness information at different scales. The brightness maps of each layer are weighted fused to obtain the global brightness information L combined with Gaussian smoothing filters of different scales. multi , the expression is as follows: Among them, red, green, and blue represent the three channels of the input image respectively; guass s represents the sth Gaussian convolution kernel; α s represents the fusion weight of the sth convolution kernel; k represents the Gaussian kernel radius; According to the color characteristics of the image, a processing method is designed to use the color variance as the adjustment basis to adaptively adjust the weights of the RGB channels. The expression is as follows: Among them, Var() represents the variance of the RGB color channel, [ω R ,ω G ,ω B ] represents the weight coefficient of RGB three channels; Calculate the color contrast in the image and adjust the brightness. The expression is as follows: L color (i,j)=ω R ·R(i,j)+ω G ·G(i,j)+ω B ·B(i,j) (11) Among them, L color (i, j) represents the result of color channel information integration, R(i, j), G(i, j) and B(i, j) represent F in The pixel values ​​of the R, G and B channels are calculated point by point, and the channel information is integrated based on the weight coefficients of formula (10); At the same time, considering the initial brightness information and the brightness adjustment based on color information, the brightness image generated by equation (11) is gamma corrected to enhance the image details, and the pixel value of the output image is normalized to [0, 255], as shown below: Among them, L gamma represents the result after Gamma correction, β represents the weight coefficient, γ represents the Gamma correction coefficient, L min and L max That is L gamma The minimum and maximum values ​​in ; F2 represents the brightness prior feature map, and the size of the brightness prior feature map obtained is H×W×1; S23, fusing the texture features and brightness features obtained in steps S21 and S22, and splicing them with the input fog image, and then performing convolution block processing to output the fog image with enhanced texture brightness; Integrate the edge and brightness feature maps of the input image, and use two learnable parameters ω1 and ω2 to obtain the fused feature map F3 = ω1·F1+ω2·F2; Subsequently, the input image is concatenated with F3 in the channel dimension, and first undergoes a 1×1 convolution to resize the number of channels to 3, and then undergoes a deep convolution with a convolution kernel size of 5, a step size of 1, and a padding of 2 to generate a fusion information feature map F as the color information and texture brightness information. out , the expression is as follows: F out =DepthwiseConv 5×5 (Conv 1×1 (Concat(F3,F in ))) (14)。 3. The deep learning defogging method based on texture brightness feature enhancement according to claim 1 is characterized in that: The step S3 is specifically as follows: The downsampling operation process includes: 3 ViT-Blocks and 2 DWT-Down; the upsampling operation includes: 3 ViT-Blocks and 2 DWT-Up; During the downsampling operation, the fog image and the texture brightness fusion feature map obtained after processing by the texture brightness feature enhancement module are first input into the first ViT-Block in the downsampling process to extract feature information of different scales, and then input into the first DWT-Down to reduce the image size after downsampling at a ratio of 1 / 2; the image processed by the first DWT-Down and the texture brightness fusion feature map are simultaneously input into the second ViT-Block to extract feature information of different scales, and then input into the second DWT-Down to reduce the image size after downsampling at a ratio of 1 / 2; then the image processed by the second DWT-Down and the texture brightness fusion feature map are simultaneously input into the third ViT-Block, and feature information of different scales is extracted after three ViT-Blocks; Then, the features extracted by the third ViT-Block in the downsampling process are input into BottleNeck for processing. Through frequency domain auxiliary processing, the output image features are obtained and enter the upsampling process. During the upsampling process, the image features output after BottleNeck are first input into the first ViT-Block in the upsampling process to extract feature information of different scales, and then input into the first DWT-Up together with the image processed by the second DWT-Down in the downsampling process to increase the image size by 2 times the ratio of upsampling; the image processed by the first DWT-Up is then input into the second ViT-Block to extract feature information of different scales, and then input into the second DWT-Up together with the image processed by the first DWT-Down in the downsampling process to increase the image size by 2 times the ratio of upsampling; finally, the image processed by the second DWT-Up is input into the third ViT-Block to extract feature information of different scales, and the image size is restored to the original input size; Finally, the upsampled image is fused with the original fog image to obtain the generated dehazed image result.

4. The deep learning defogging method based on texture brightness feature enhancement according to claim 1 is characterized in that: In step S1, the deep learning dehazing network is trained using a public data set and a loss function is used for model training, as follows: In order to make the predicted image closer to the real clear image at the pixel level, one of the loss functions L s The expression is as follows: in, and Y represent the predicted image and true label respectively; Aligned in the spatial domain and introducing the learning of frequency signals in the network, the frequency domain L1 loss function is introduced to train the network, and the expression is as follows: Where M and N represent the width and height of the amplitude spectrum respectively; A out and A represent the amplitude spectra of the network predicted image and the target image respectively, P out and P represent the phase spectra of the network predicted image and the target image respectively; L amp and L pha denote the amplitude and phase losses respectively; The input image is introduced into the loss function to calculate the contrast loss. By comparing the features of the network output and the ground truth, the contrast loss function expression is as follows: Among them, a, p, n represent the network prediction image, the true value image and the input image respectively; f i represents the i-th layer output of feature extraction, ω i represents the weight coefficient of the i-th layer, d L1 represents the result of L1 loss calculation, ε represents an arbitrarily small constant to prevent division by zero error; The total loss function expression is as follows: in, They represent the corresponding loss function weights respectively.

5. The deep learning defogging method based on texture brightness feature enhancement according to claim 1 is characterized in that: In step S1, the evaluation indicators of the defogging processing effect of the deep learning defogging network are PSNR and SSIM, which are as follows: (1) Peak signal-to-noise ratio (PSNR); Among them, MAX represents the maximum possible value of the image pixel value, and MSE represents the mean square error of the pixel values ​​between the original image and the reconstructed image; (2) Structural Similarity Index (SSIM); Among them, μ x and μ y Represent the average values ​​of the original image and the reconstructed image, σ x and σ y Represent the standard deviation of the original image and the reconstructed image, σ xy represents the covariance between the original image and the reconstructed image, and C1 and C2 represent constants used for stabilization.

Images