A deep learning image defogging method, system, device and storage medium based on enhanced curve learning

Abstract

The present invention discloses a deep learning image defogging method, system, device and storage medium based on enhanced curve learning, which is applied to the field of image processing technology, including: obtaining the brightness value of foggy image data in YUV color space; cutting the foggy image data into blocks of different scales, and fitting the Y channel histogram of the blocks to the GT transformation curve using polynomials, obtaining the polynomial coefficients of the blocks under the minimum fitting error based on the least squares method, as the learning labels of the foggy image data at different scales; based on the learning labels, using deep learning to learn the Y channel histogram transformation curve of the foggy image data at different scales, and multi-scale fusion of the learning results at different scales, and outputting the image defogging results. The present invention effectively improves the defogging efficiency, accuracy and stability.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a deep learning image defogging method, system, device and storage medium based on enhancement curve learning. Background Art

[0002] In real scenes, fog is composed of tiny water droplets or ice crystals suspended in the air near the ground. These particles scatter and reflect light, which has a significant impact on the propagation of light and visual effects. As a result, the data obtained by the collection equipment will have problems such as low contrast and color saturation.

[0003] Common dehazing algorithms are mainly divided into traditional dehazing algorithms and deep learning dehazing algorithms. Traditional dehazing algorithms are mainly divided into solutions based on histogram enhancement, dark channel model, and Retinex theory. The method based on histogram enhancement is fast, but the pixels are globally mapped and some detailed features are easily lost; the method based on dark channel enhancement has poor adaptability to sky scenes; based on Retinex theory, it is difficult to obtain reflectivity, so it is impossible to obtain good results; deep learning dehazing algorithms can generally obtain good results through higher-dimensional parameter fitting, but they are generally difficult to train and have insufficient speed. Specifically, the histogram equalization based on restricted contrast enhancement in the traditional dehazing algorithm divides the image into blocks and performs restricted contrast enhancement histogram equalization on each block of the image. Then, the histogram equalization result of each block is interpolated by linear interpolation to smooth the difference between blocks, thereby obtaining a histogram enhancement result with smooth blocks. However, there are obvious problems: first, it is difficult to select the limiting threshold, and there are obvious differences in the limiting contrast thresholds between pictures of different scenes. Second, there are phenomena such as halos. Although smoothing is used, it will lead to a large range of obvious transition zones. Compared with traditional dehazing algorithms, deep learning algorithms can effectively avoid the problem of halos and remove fog to a certain extent. However, the existing method networks are relatively complex, inefficient, and not very practical. Most networks are not stable enough on data outside the training set, and even show unrealistic colors or texture distortions, resulting in distorted results.

[0004] Therefore, how to provide a deep learning image defogging method, system, device and storage medium based on enhanced curve learning that can effectively avoid the halo phenomenon in defogging and effectively improve the defogging efficiency, accuracy and stability is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention proposes a deep learning image defogging method, system, device and storage medium based on enhancement curve learning.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A deep learning image defogging method based on enhancement curve learning, comprising:

[0008] Step 1: Obtain the brightness value of the foggy image data in the YUV color space;

[0009] Step 2: Crop the foggy image data into blocks of different sizes, and use polynomial fitting to convert the Y channel histogram of the block to the GT. Based on the least squares method, obtain the polynomial coefficients of the block with the minimum fitting error as the learning labels of the foggy image data at different scales.

[0010] Step 3: Based on the learning labels, use deep learning to learn the Y channel histogram transformation curve of the foggy image data at different scales, and perform multi-scale fusion on the learning results at different scales to output the image defogging result.

[0011] Optionally, in step 1, the brightness value of the foggy image data in the YUV color space is obtained as follows:

[0012]

[0013] in, is the brightness value; is a constant; It is the RGB value of the foggy image data.

[0014] Optionally, in step 2, the foggy image data is cropped into blocks of different sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, specifically:

[0015] The foggy image data is cropped into blocks of 3*3, 5*5, and 7*7 sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, as follows:

[0016]

[0017] in, The first blocks are 3*3, 5*5, and 7*7 in size. The polynomial corresponding to the transformation curve from the Y channel histogram of each block to GT; for The corresponding scale size of the block The polynomial coefficients corresponding to the Y channel histogram transformation curve of each block.

[0018] Optionally, in step 2, the polynomial coefficients of the blocks with the minimum fitting error are obtained based on the least squares method, as follows:

[0019]

[0020] in, for The corresponding scale size of the block The fitting error of the polynomial corresponding to the Y channel histogram transformation curve of each block compared with the corresponding GT; for The corresponding scale size of the block The corresponding GT of each block; for The corresponding scale size of the block The polynomial corresponding to the Y channel histogram transformation curve of each block.

[0021] Optionally, in step 3, based on the learning labels, deep learning is used to learn the Y channel histogram transformation curve of the foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging result, specifically:

[0022] Step 3.1: Use 10 3*3*1 convolution kernels to convolve the foggy image data, extract the feature F1, and use the polynomial coefficients of the 3*3 scale block with the minimum fitting error as the learning label. Through 5 3*3*10 convolutions with a step size of 3, the learning result A representing the 3*3 Y channel histogram transformation curve is obtained;

[0023] Step 3.2: Use 10 3*3*1 convolution kernels to convolve F1 to obtain feature F2, and use the polynomial coefficients of the 5*5 scale block with the minimum fitting error as the learning label. Through 5 5*5*5 convolutions with a step size of 5, the learning result B representing the 5*5 Y channel histogram transformation curve is obtained;

[0024] Step 3.3: Use three 3*3*5 convolution kernels to convolve F2 to obtain feature F3, and use the polynomial coefficients of the 7*7 scale block with the minimum fitting error as the learning label. Through five 7*7*3 convolutions with a step size of 7, the learning result C representing the 7*7 Y channel histogram transformation curve is obtained;

[0025] Step 3.4: Perform quick table lookup transformation on learning results A, B, and C respectively, and perform multi-scale fusion with the foggy image data to obtain F4, and output the image defogging result through 3*3*4 convolution.

[0026] Optionally, in step 3.4, the fast table lookup is transformed into a power of 2 to 5 of 0 to 255.

[0027] Optionally, in step 3.4, after outputting the image defogging result, the step further includes: comparing the image defogging result with the GT and optimizing it using L1_loss.

[0028] The present invention also provides a deep learning image defogging system based on enhanced curve learning using a deep learning image defogging method based on enhanced curve learning, comprising:

[0029] Brightness value acquisition module: used to obtain the brightness value of foggy image data in YUV color space;

[0030] Pre-training initial weight determination module: used to crop the foggy image data into blocks of different sizes, and use polynomial fitting to transform the Y channel histogram of the block to the GT curve. Based on the least squares method, the polynomial coefficients of the block with the minimum fitting error are obtained as the learning labels of the foggy image data at different scales.

[0031] Image defogging module: Based on the learning labels, deep learning is used to learn the Y channel histogram transformation curve of foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging results.

[0032] The present invention also provides an electronic device, comprising:

[0033] Memory for storing computer programs;

[0034] A processor is used to implement the steps of a deep learning image defogging method based on enhancement curve learning when executing a computer program.

[0035] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a deep learning image defogging method based on enhancement curve learning are implemented.

[0036] It can be seen from the above technical solutions that compared with the prior art, the present invention proposes a deep learning image defogging method, system, device and storage medium based on enhanced curve learning. By using polynomial fitting histogram curves and using polynomial coefficients as deep learning network learning labels, the overall learning difficulty of the network is effectively reduced and the defogging accuracy is improved; by learning blocks of different scales and performing multi-scale fusion, the defogging accuracy and stability are effectively improved; when using a deep network to learn a histogram transformation curve, a step size of 3 is used when learning a 3*3 histogram transformation curve, a step size of 5 is used when learning a 5*5 histogram transformation curve, and a step size of 7 is used when learning a 7*7 histogram transformation curve, which greatly reduces the amount of calculation and further improves the defogging efficiency. In summary, the present invention effectively improves the defogging efficiency, accuracy and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0038] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0039] Figure 2 This is a schematic diagram of the deep learning training network structure of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1:

[0042] Embodiment 1 of the present invention discloses a deep learning image defogging method based on enhancement curve learning, such as Figure 1 As shown, including:

[0043] Step 1: Get the brightness value of the foggy image data in the YUV color space.

[0044] The foggy image data is taken in the evening or morning of a cloudy day. The real scene is taken as input, and the fog maker is used to simulate the real foggy scene to create smoke as the network input picture. In order to more comprehensively cover the foggy scene, the fog maker is controlled to simulate light fog, medium fog and heavy fog by controlling the fog release time and fog release size.

[0045] Get the brightness value of the foggy image data in the YUV color space as follows:

[0046]

[0047] in, is the brightness value; are constants, set to 0.299, 0.587, and 0.114 respectively; It is the RGB value of the foggy image data.

[0048] Step 2: Crop the foggy image data into blocks of different sizes, and use a polynomial to fit the transformation curve from the Y channel histogram of the block to the GT. Based on the least squares method, the polynomial coefficients of the block with the minimum fitting error are obtained as the learning labels of the foggy image data at different scales.

[0049] The foggy image data is cropped into blocks of different sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, specifically:

[0050] The foggy image data is cropped into blocks of 3*3, 5*5, and 7*7 sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, as follows:

[0051]

[0052] in, The first blocks are 3*3, 5*5, and 7*7 in size. The polynomial corresponding to the transformation curve from the Y channel histogram of each block to GT; for The corresponding scale size of the block The polynomial coefficients corresponding to the Y channel histogram transformation curve of each block.

[0053] Based on the least squares method, the polynomial coefficients of the blocks with the minimum fitting error are obtained as follows:

[0054]

[0055] in, for The corresponding scale size of the block The fitting error of the polynomial corresponding to the Y channel histogram transformation curve of each block compared with the corresponding GT; for The corresponding scale size of the block The corresponding GT of each block; for The corresponding scale size of the block The polynomial corresponding to the Y channel histogram transformation curve of each block; 255 is the number of gray levels of the discrete value of the curve used by the image histogram.

[0056] Step 3: Based on the learning labels, use deep learning to learn the Y channel histogram transformation curve of the foggy image data at different scales, and perform multi-scale fusion on the learning results at different scales to output the image defogging result.

[0057] After some data enhancement steps such as flipping and cropping, the foggy image data is input as follows Figure 2 The deep learning training network shown is trained.

[0058] Based on the learning labels, deep learning is used to learn the Y channel histogram transformation curve of foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging results, specifically:

[0059] Step 3.1: Use 10 3*3*1 convolution kernels to convolve the foggy image data, extract the feature F1, and use the polynomial coefficients of the 3*3 scale block with the minimum fitting error as the learning label. Through 5 3*3*10 convolutions with a step size of 3, the learning result A representing the 3*3 Y channel histogram transformation curve is obtained;

[0060] Step 3.2: Use 10 3*3*1 convolution kernels to convolve F1 to obtain feature F2, and use the polynomial coefficients of the 5*5 scale block with the minimum fitting error as the learning label. Through 5 5*5*5 convolutions with a step size of 5, the learning result B representing the 5*5 Y channel histogram transformation curve is obtained;

[0061] Step 3.3: Use three 3*3*5 convolution kernels to convolve F2 to obtain feature F3, and use the polynomial coefficients of the 7*7 scale block with the minimum fitting error as the learning label. Through five 7*7*3 convolutions with a step size of 7, the learning result C representing the 7*7 Y channel histogram transformation curve is obtained;

[0062] Step 3.4: Perform quick table lookup transformation on learning results A, B, and C respectively, and perform multi-scale fusion with the foggy image data to obtain F4, and output the image defogging result through 3*3*4 convolution.

[0063] The fast table lookup transforms the value from 0 to 255 to the power of 2 to 5.

[0064] After outputting the image defogging result, it also includes: comparing the image defogging result with GT and optimizing it using L1_loss.

[0065] The advantages of the present invention using polynomial fitting histogram transformation curve are:

[0066] The coefficients of the polynomial can be obtained through traditional algorithms, which facilitates the formation of correct learning labels, effectively reduces the overall learning difficulty of the network, and improves the dehazing accuracy.

[0067] Compared with the number of pixels, the amount of calculation of the histogram is greatly reduced, and the polynomial can effectively learn the curve transformation between images with fewer parameters, effectively improving the defogging efficiency and defogging accuracy;

[0068] Using smaller blocks of 3*3 makes it easier to increase the transformation of local variables, so that each block can be adaptively transformed, increase the adaptability to surrounding pixels, and effectively improve the stability of defogging;

[0069] Adopt multi-scale fusion, use 5*5 blocks, limit 3*3 blocks to prevent unevenness between blocks; similarly, 7*7 is limited to 5*5 blocks, and finally multi-scale fusion is performed to effectively improve the defogging accuracy through GT constraints;

[0070] When using a deep network to learn the histogram transformation curve, because only the pixel features of the corresponding block need to be learned, a step size of 3 is used when learning the 3*3 histogram transformation curve, a step size of 5 is used when learning the 5*5 histogram transformation curve, and a step size of 7 is used when learning the 7*7 histogram transformation curve. This can greatly reduce the amount of calculation, increase the network speed, and effectively improve the dehazing efficiency.

[0071] Embodiment 2:

[0072] Embodiment 2 of the present invention discloses a deep learning image defogging system based on enhancement curve learning using a deep learning image defogging method based on enhancement curve learning, comprising:

[0073] Brightness value acquisition module: used to obtain the brightness value of foggy image data in YUV color space;

[0074] Pre-training initial weight determination module: used to crop the foggy image data into blocks of different sizes, and use polynomial fitting to transform the Y channel histogram of the block to the GT curve. Based on the least squares method, the polynomial coefficients of the block with the minimum fitting error are obtained as the learning labels of the foggy image data at different scales.

[0075] Image defogging module: Based on the learning labels, deep learning is used to learn the Y channel histogram transformation curve of foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging results.

[0076] Embodiment 3:

[0077] Embodiment 3 of the present invention discloses an electronic device, including:

[0078] Memory for storing computer programs;

[0079] A processor is used to implement the steps of a deep learning image defogging method based on enhancement curve learning when executing a computer program.

[0080] Embodiment 4:

[0081] Embodiment 4 of the present invention discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a deep learning image defogging method based on enhancement curve learning are implemented.

[0082] The embodiment of the present invention discloses a deep learning image defogging method, system, device and storage medium based on enhanced curve learning. By using a polynomial to fit the histogram curve and using the polynomial coefficients as the deep learning network learning labels, the overall learning difficulty of the network is effectively reduced and the defogging accuracy is improved; by learning blocks of different scales and performing multi-scale fusion, the defogging accuracy and stability are effectively improved; when using a deep network to learn the histogram transformation curve, a step size of 3 is used when learning a 3*3 histogram transformation curve, a step size of 5 is used when learning a 5*5 histogram transformation curve, and a step size of 7 is used when learning a 7*7 histogram transformation curve, which greatly reduces the amount of calculation and further improves the defogging efficiency. In summary, the present invention effectively improves the defogging efficiency, accuracy and stability.

[0083] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0084] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep learning image defogging method based on enhancement curve learning, characterized in that: include: Step 1: Obtain the brightness value of the foggy image data in the YUV color space; Step 2: The foggy image data is cut into blocks of different sizes, and a polynomial is used to fit the transformation curve of the Y channel histogram of the block to the GT. The polynomial coefficients of the block with the minimum fitting error are obtained based on the least squares method as the learning labels of the foggy image data at different scales. Step 3: Based on the learning label, use deep learning to learn the Y channel histogram transformation curve of the foggy image data at different scales, and perform multi-scale fusion on the learning results at different scales to output the image defogging result; In step 2, the foggy image data is cut into blocks of different sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, specifically: The foggy image data is cropped into blocks of 3*3, 5*5, and 7*7 sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, as follows: f ij (x)=z ij +a ij x+b ij x 2 +c ij x 3 +d ij x 4 +e ij x 5 ; Where i = 1, 2, 3, f 1j (x), f 2j (x), f 3j (x) is the polynomial corresponding to the transformation curve from the Y channel histogram of the jth block to GT in the blocks of 3*3, 5*5, and 7*7 scales respectively; z ij 、a ij , b ij 、c ij ,d ij 、e ij is the polynomial coefficient corresponding to the Y channel histogram transformation curve of the jth block in the block with the corresponding scale size of i; In step 3, based on the learning label, deep learning is used to learn the Y channel histogram transformation curve of the foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging result, specifically: Step 3.1: Use 10 3*3*1 convolution kernels to convolve the foggy image data, extract the feature F1, and use the polynomial coefficients of the 3*3 scale block with the minimum fitting error as the learning label, and obtain the learning result A representing the 3*3 Y channel histogram transformation curve through 5 3*3*10 convolutions with a step size of 3; Step 3.2: Use 10 3*3*1 convolution kernels to convolve F1 to obtain feature F2, and use the polynomial coefficients of the 5*5 scale block with the minimum fitting error as the learning label. Through 5 5*5*5 convolutions with a step size of 5, the learning result B representing the 5*5 Y channel histogram transformation curve is obtained; Step 3.3: Use three 3*3*5 convolution kernels to convolve F2 to obtain feature F3, and use the polynomial coefficients of the 7*7 scale block with the minimum fitting error as the learning label. Through five 7*7*3 convolutions with a step size of 7, the learning result C representing the 7*7 Y channel histogram transformation curve is obtained; Step 3.4: Perform quick table lookup transformation on the learning results A, B, and C respectively, perform multi-scale fusion with the foggy image data to obtain F4, and output the image defogging result through 3*3*4 convolution.

2. The deep learning image defogging method based on enhancement curve learning according to claim 1 is characterized in that: In step 1, the brightness value of the foggy image data in the YUV color space is obtained as follows: I lum =C r ×I R +C g ×I G +C b ×I B ; Among them, I lum is the brightness value; C r , C g , C b is a constant; I R ,I G ,I B It is the RGB value of the foggy image data.

3. The deep learning image defogging method based on enhancement curve learning according to claim 1 is characterized in that: In step 2, the polynomial coefficients of the block with the minimum fitting error are obtained based on the least squares method, as follows: Among them, ε ij is the fitting error of the polynomial corresponding to the Y channel histogram transformation curve of the jth block in the block of the corresponding scale size i compared with the corresponding GT; q ij is the corresponding GT of the jth block in the block of the corresponding scale size of i; f ij (x) is the polynomial corresponding to the Y channel histogram transformation curve of the jth block in the block with the corresponding scale size of i.

4. The deep learning image defogging method based on enhancement curve learning according to claim 1, characterized in that: In step 3.4, the fast table lookup is transformed into a 2-5 power change of 0-255.

5. The deep learning image defogging method based on enhancement curve learning according to claim 1, characterized in that: In step 3.4, after outputting the image defogging result, the process further includes: comparing the image defogging result with the GT, and optimizing it using L1_l oss.

6. A deep learning image defogging system based on enhanced curve learning using the deep learning image defogging method based on enhanced curve learning according to any one of claims 1 to 5, characterized in that: include: Brightness value acquisition module: used to obtain the brightness value of foggy image data in YUV color space; Pre-training initial weight determination module: used to crop the foggy image data into blocks of different scales, and use polynomial fitting to transform the Y channel histogram of the block to the GT curve, and obtain the polynomial coefficient of the block with the minimum fitting error based on the least squares method as the learning label of the foggy image data at different scales; Image defogging module: Based on the learning labels, deep learning is used to learn the Y channel histogram transformation curve of the foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging result.

7. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of a deep learning image defogging method based on enhancement curve learning as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the deep learning image defogging method based on enhancement curve learning as described in any one of claims 1 to 5 are implemented.

Images