Deep learning image defogging method, system and device based on enhanced curve learning, and storage medium

Abstract

The invention discloses a deep learning image defogging method, system and device based on enhanced curve learning, and a storage medium, and is applied to the technical field of image processing, and the method comprises the steps: obtaining a brightness value of foggy image data in a YUV color space; respectively cutting the foggy image data into blocks with different scales, fitting Y-channel histograms of the blocks to a transformation curve of GT by using a polynomial, and obtaining a polynomial coefficient of the blocks under a minimum fitting error based on a least square method as learning labels of the foggy image data under different scales; and based on the learning label, learning Y-channel histogram transformation curves of the foggy image data under different scales by using deep learning, carrying out multi-scale fusion on learning results under different scales, and outputting an image defogging result. According to the invention, the defogging efficiency, precision and stability are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a deep learning-based image dehazing method, system, device, and storage medium based on enhancement curve learning. Background Technology

[0002] In real-world scenarios, fog is composed of tiny water droplets or ice crystals suspended in the air near the ground. These particles scatter and reflect light, significantly impacting light propagation and visual effects. Consequently, the data acquired by the monitoring equipment may suffer from issues such as low contrast and low color saturation.

[0003] Common dehazing algorithms are mainly divided into traditional dehazing algorithms and deep learning dehazing algorithms. Traditional dehazing algorithms are further divided into histogram enhancement-based, dark channel model-based, and Retinex theory-based schemes. Histogram enhancement-based methods are fast, but pixels are globally mapped, which can easily lose some detailed features; dark channel enhancement-based methods have poor adaptability to sky scenes; Retinex theory-based methods are difficult to obtain reflectivity, so they cannot achieve good results; deep learning dehazing algorithms, through higher-dimensional parameter fitting, generally achieve good results, but they are generally difficult to train and have speed limitations. Specifically, traditional dehazing algorithms based on histogram equalization with limited contrast enhancement divide the image into blocks and perform limited contrast enhancement histogram equalization on each block. Then, linear interpolation is used to interpolate the histogram equalization results of each block to smooth the differences between blocks, thereby obtaining a smooth histogram enhancement result between blocks. However, there are some obvious problems: First, it is difficult to select the appropriate threshold, as the threshold for limiting contrast varies significantly between images in different scenes. Second, phenomena such as halo effects occur; although smoothing is used, it results in a large and obvious transition band. Compared with traditional dehazing algorithms, deep learning-based algorithms can effectively avoid the halo problem and remove fog to some extent. However, existing methods have relatively complex networks, low efficiency, and limited practicality. Most networks are not stable enough on data outside the training set, and may even display unrealistic colors or texture distortions, resulting in distorted results.

[0004] Therefore, how to provide a deep learning-based image dehazing method, system, device, and storage medium that can effectively avoid the halo phenomenon in dehazing and effectively improve the efficiency, accuracy, and stability of dehazing is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention proposes a deep learning-based image dehazing method, system, device and storage medium based on enhancement curve learning.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A deep learning-based image dehazing method based on enhancement curve learning includes: Step 1: Obtain the brightness value of the hazy image data in the YUV color space; Step 2: The foggy image data is cropped into blocks of different scales, and the transformation curve from the Y-channel histogram of the block to the ground plane is fitted using a polynomial. The polynomial coefficients of the block with the minimum fitting error are obtained based on the least squares method and used as the learning labels of the foggy image data at different scales. Step 3: Based on the learned labels, use deep learning to learn the Y-channel histogram transformation curve of the foggy image data at different scales, and perform multi-scale fusion of the learning results at different scales to output the image dehazing result.

[0007] Optionally, in step 1, the brightness values ​​of the hazy image data in the YUV color space are obtained, as follows: in, This refers to the brightness value. It is a constant; These are the RGB values ​​of the foggy image data.

[0008] Optionally, in step 2, the foggy image data is cropped into blocks of different sizes, and the transformation curve from the Y-channel histogram to the ground plane is fitted using a polynomial, specifically as follows: The foggy image data was cropped into blocks of 3*3, 5*5, and 7*7 sizes, respectively, and the transformation curves from the Y-channel histogram to the ground plane were fitted using a polynomial, as shown below: in, In blocks of sizes 3*3, 5*5, and 7*7 respectively, the first... The polynomials corresponding to the transformation curves from the Y-channel histograms of each block to the ground truth (GT); for In the block corresponding to the scale size, the first The polynomial coefficients corresponding to the Y-channel histogram transform curves of each block.

[0009] Optionally, in step 2, the polynomial coefficients of the blocks with the minimum fitting error are obtained based on the least squares method, as follows: in, for In the block corresponding to the scale size, the first The fitting error of the polynomial corresponding to the Y-channel histogram transform curve of each block compared to the corresponding GT; for In the block corresponding to the scale size, the first The corresponding GT of each block; for In the block corresponding to the scale size, the first The polynomials corresponding to the Y-channel histogram transformation curves of each block.

[0010] Optionally, in step 3, based on the learned labels, deep learning is used to learn the Y-channel histogram transformation curves of the hazy image data at different scales, and the learning results at different scales are fused at multiple scales to output the image dehazing result, specifically: Step 3.1: Convolve the foggy image data using 10 3*3*1 convolution kernels to extract the feature F1. Use the polynomial coefficients of the 3*3 scale blocks with the minimum fitting error as the learning labels. Through 5 3*3*10 convolutions with a stride of 3, obtain the learning result A representing the 3*3 Y channel histogram transformation curve. Step 3.2: Convolve F1 with 10 3*3*1 convolution kernels to obtain feature F2, and use the polynomial coefficients of 5*5 scale blocks with the minimum fitting error as learning labels. Through 5 5*5*5 convolutions with a stride of 5, obtain the learning result B representing the 5*5 Y channel histogram transformation curve. Step 3.3: Convolve F2 with three 3*3*5 convolution kernels to obtain feature F3, and use the polynomial coefficients of 7*7 scale blocks with the minimum fitting error as learning labels. Through five 7*7*3 convolutions with a stride of 7, obtain the learning result C representing the 7*7 Y channel histogram transformation curve. Step 3.4: Perform fast lookup table transformation on learning results A, B, and C respectively, and fuse them with the foggy image data at multiple scales to obtain F4. Output the dehazing result of the image through 3*3*4 convolution.

[0011] Optionally, in step 3.4, the quick lookup table transformation is changed to powers 2-5 of 0-255.

[0012] Optionally, in step 3.4, after outputting the image dehazing result, the following steps are also included: comparing the image dehazing result with the ground truth (GT) and optimizing it using L1 loss.

[0013] This invention also provides a deep learning image dehazing system based on enhancement curve learning, which utilizes a deep learning image dehazing method based on enhancement curve learning, comprising: Brightness value acquisition module: used to acquire the brightness value of foggy image data in the YUV color space; Pre-training initial weight determination module: It is used to crop the foggy image data into blocks of different scales, and use polynomial fitting to fit the transformation curve of the Y channel histogram of the block to the ground plane. Based on the least squares method, it obtains the polynomial coefficients of the block with the minimum fitting error, which are used as the learning labels of the foggy image data at different scales. Image dehazing module: Based on learned labels, it uses deep learning to learn the Y-channel histogram transformation curve of hazy image data at different scales, and performs multi-scale fusion of the learning results at different scales to output the image dehazing result.

[0014] The present invention also provides an electronic device, comprising: Memory, used to store computer programs; A processor, used to implement a deep learning-based image dehazing method based on reinforcement curve learning when executing a computer program.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a deep learning image dehazing method based on reinforcement curve learning.

[0016] As can be seen from the above technical solutions, compared with the prior art, this invention proposes a deep learning image dehazing method, system, device, and storage medium based on enhanced curve learning. By utilizing polynomial fitting of histogram curves and using polynomial coefficients as learning labels for the deep learning network, the overall learning difficulty of the network is effectively reduced, and the dehazing accuracy is improved. By learning from blocks of different scales and performing multi-scale fusion, the dehazing accuracy and stability are effectively improved. When using the deep network to learn histogram transformation curves, a step size of 3 is used when learning 3*3 histogram transformation curves, a step size of 5 is used when learning 5*5 histogram transformation curves, and a step size of 7 is used when learning 7*7 histogram transformation curves, which greatly reduces the computational load and further improves the dehazing efficiency. In summary, this invention effectively improves the dehazing efficiency, accuracy, and stability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0019] Figure 2 This is a schematic diagram of the deep learning training network structure of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example 1: Embodiment 1 of this invention discloses a deep learning-based image dehazing method based on enhancement curve learning, such as... Figure 1 As shown, it includes: Step 1: Obtain the brightness value of the foggy image data in the YUV color space.

[0022] The foggy image data was captured on a cloudy evening or morning, using real-world scenes as input. A fog generator was used to simulate real foggy conditions, creating fog as the input image for the network. To more comprehensively cover foggy scenes, the fog generator's fog release time and size were controlled to simulate light, medium, and heavy fog.

[0023] The brightness values ​​of the hazy image data in the YUV color space are obtained as follows: in, This refers to the brightness value. These are constants, set to 0.299, 0.587, and 0.114 respectively; These are the RGB values ​​of the foggy image data.

[0024] Step 2: The foggy image data is cropped into blocks of different sizes, and the transformation curve from the Y-channel histogram of the block to the ground plane is fitted using a polynomial. The polynomial coefficients of the block with the minimum fitting error are obtained based on the least squares method and used as the learning labels of the foggy image data at different sizes.

[0025] The foggy image data is cropped into blocks of different sizes, and the transformation curve from the Y-channel histogram to the ground plane is fitted using a polynomial. Specifically: The foggy image data was cropped into blocks of 3*3, 5*5, and 7*7 sizes, respectively, and the transformation curves from the Y-channel histogram to the ground plane were fitted using a polynomial, as shown below: in, In blocks of sizes 3*3, 5*5, and 7*7 respectively, the first... The polynomials corresponding to the transformation curves from the Y-channel histograms of each block to the ground truth (GT); for In the block corresponding to the scale size, the first The polynomial coefficients corresponding to the Y-channel histogram transform curves of each block.

[0026] The polynomial coefficients of the block division with the minimum fitting error are obtained using the least squares method, as follows: in, for In the block corresponding to the scale size, the first The fitting error of the polynomial corresponding to the Y-channel histogram transform curve of each block compared to the corresponding GT; for In the block corresponding to the scale size, the first The corresponding GT of each block; for In the block corresponding to the scale size, the first The polynomial corresponding to the Y-channel histogram transformation curve of each block; 255 is the gray level of the discrete value of the curve used in the image histogram.

[0027] Step 3: Based on the learned labels, use deep learning to learn the Y-channel histogram transformation curve of the foggy image data at different scales, and perform multi-scale fusion of the learning results at different scales to output the image dehazing result.

[0028] After performing data augmentation steps such as flipping and cropping on the foggy image data, the input is as follows: Figure 2 The deep learning training network shown is trained.

[0029] Based on learned labels, deep learning is used to learn the Y-channel histogram transformation curves of hazy image data at different scales. The learning results at different scales are then fused at multiple scales to output the image dehazing result, specifically: Step 3.1: Convolve the foggy image data using 10 3*3*1 convolution kernels to extract the feature F1. Use the polynomial coefficients of the 3*3 scale blocks with the minimum fitting error as the learning labels. Through 5 3*3*10 convolutions with a stride of 3, obtain the learning result A representing the 3*3 Y channel histogram transformation curve. Step 3.2: Convolve F1 with 10 3*3*1 convolution kernels to obtain feature F2, and use the polynomial coefficients of 5*5 scale blocks with the minimum fitting error as learning labels. Through 5 5*5*5 convolutions with a stride of 5, obtain the learning result B representing the 5*5 Y channel histogram transformation curve. Step 3.3: Convolve F2 with three 3*3*5 convolution kernels to obtain feature F3, and use the polynomial coefficients of 7*7 scale blocks with the minimum fitting error as learning labels. Through five 7*7*3 convolutions with a stride of 7, obtain the learning result C representing the 7*7 Y channel histogram transformation curve. Step 3.4: Perform fast lookup table transformation on learning results A, B, and C respectively, and fuse them with the foggy image data at multiple scales to obtain F4. Output the dehazing result of the image through 3*3*4 convolution.

[0030] Quick table lookup transforms the values ​​from 0 to 255 to the powers of 2 to 5.

[0031] After outputting the image dehazing result, the process also includes: comparing the image dehazing result with the ground truth (GT) and optimizing using L1 loss.

[0032] The advantages of using polynomial fitting for histogram transformation curves in this invention are: The coefficients of the polynomial can be obtained through traditional algorithms, which facilitates the formation of correct learning labels, effectively reduces the overall learning difficulty of the network, and improves the dehazing accuracy.

[0033] Histograms significantly reduce computational complexity compared to the number of pixels, and polynomials can effectively learn curve transformations between images with fewer parameters, thus improving dehazing efficiency and accuracy. Using smaller 3x3 blocks makes it easier to increase the transformation of local variables, enabling each block to undergo adaptive transformation, increasing its adaptability to surrounding pixels, and effectively improving dehazing stability. Multi-scale fusion is employed, using 5x5 blocks and restricting 3x3 blocks to prevent unevenness between blocks; similarly, 7x7 blocks are restricted to 5x5 blocks, and finally multi-scale fusion is performed. Through GT constraints, the dehazing accuracy is effectively improved. When using deep networks to learn histogram transformation curves, since only the pixel features of the corresponding blocks need to be learned, a step size of 3 is used when learning 3*3 histogram transformation curves, a step size of 5 is used when learning 5*5 histogram transformation curves, and a step size of 7 is used when learning 7*7 histogram transformation curves. This can greatly reduce the amount of computation, improve network speed, and effectively improve dehazing efficiency.

[0034] Example 2: Embodiment 2 of the present invention discloses a deep learning image dehazing system based on enhancement curve learning, comprising: Brightness value acquisition module: used to acquire the brightness value of foggy image data in the YUV color space; Pre-training initial weight determination module: It is used to crop the foggy image data into blocks of different scales, and use polynomial fitting to fit the transformation curve of the Y channel histogram of the block to the ground plane. Based on the least squares method, it obtains the polynomial coefficients of the block with the minimum fitting error, which are used as the learning labels of the foggy image data at different scales. Image dehazing module: Based on learned labels, it uses deep learning to learn the Y-channel histogram transformation curve of hazy image data at different scales, and performs multi-scale fusion of the learning results at different scales to output the image dehazing result.

[0035] Example 3: Embodiment 3 of the present invention discloses an electronic device, comprising: Memory, used to store computer programs; A processor, used to implement a deep learning-based image dehazing method based on reinforcement curve learning when executing a computer program.

[0036] Example 4: Embodiment 4 of the present invention discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of a deep learning image dehazing method based on reinforcement curve learning.

[0037] This invention discloses a deep learning-based image dehazing method, system, device, and storage medium based on enhanced curve learning. By utilizing polynomial fitting of histogram curves and using polynomial coefficients as learning labels for the deep learning network, the overall learning difficulty of the network is effectively reduced, and the dehazing accuracy is improved. By learning from blocks of different scales and performing multi-scale fusion, the dehazing accuracy and stability are effectively improved. When using the deep network to learn histogram transformation curves, a step size of 3 is used when learning 3*3 histogram transformation curves, a step size of 5 is used when learning 5*5 histogram transformation curves, and a step size of 7 is used when learning 7*7 histogram transformation curves, which greatly reduces the computational load and further improves the dehazing efficiency. In summary, this invention effectively improves the dehazing efficiency, accuracy, and stability.

[0038] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0039] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep learning image defogging method based on enhancement curve learning, characterized in that: include: Step 1: Obtain the brightness value of the foggy image data in the YUV color space; Step 2: The foggy image data is cut into blocks of different sizes, and a polynomial is used to fit the transformation curve of the Y channel histogram of the block to the GT. The polynomial coefficients of the block with the minimum fitting error are obtained based on the least squares method as the learning labels of the foggy image data at different scales. Step 3: Based on the learning label, use deep learning to learn the Y channel histogram transformation curve of the foggy image data at different scales, and perform multi-scale fusion on the learning results at different scales to output the image defogging result.

2. The deep learning image defogging method based on enhancement curve learning according to claim 1 is characterized in that: In step 1, the brightness value of the foggy image data in the YUV color space is obtained as follows: in, is the brightness value; is a constant; It is the RGB value of the foggy image data.

3. The deep learning image defogging method based on enhancement curve learning according to claim 1 is characterized in that: In step 2, the foggy image data is cut into blocks of different sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, specifically: The foggy image data is cropped into blocks of 3*3, 5*5, and 7*7 sizes, and a polynomial is used to fit the transformation curve from the Y channel histogram of the block to the GT, as follows: in, They are In the block of size The polynomial corresponding to the transformation curve from the Y channel histogram of each block to GT; for The corresponding scale size of the block The polynomial coefficients corresponding to the Y channel histogram transformation curve of each block.

4. The deep learning image defogging method based on enhancement curve learning according to claim 1, characterized in that: In step 2, the polynomial coefficients of the block with the minimum fitting error are obtained based on the least squares method, as follows: in, for The corresponding scale size of the block The fitting error of the polynomial corresponding to the Y channel histogram transformation curve of each block compared with the corresponding GT; for The corresponding scale size of the block The corresponding GT of each block; for The corresponding scale size of the block The polynomial corresponding to the Y channel histogram transformation curve of each block.

5. The deep learning image defogging method based on enhancement curve learning according to claim 4 is characterized in that: In step 3, based on the learning label, deep learning is used to learn the Y channel histogram transformation curve of the foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging result, specifically: Step 3.1: Use 10 3*3*1 convolution kernels to convolve the foggy image data, extract the feature F1, and use the polynomial coefficients of the 3*3 scale block with the minimum fitting error as the learning label, and obtain the learning result A representing the 3*3 Y channel histogram transformation curve through 5 3*3*10 convolutions with a step size of 3; Step 3.2: Use 10 3*3*1 convolution kernels to convolve F1 to obtain feature F2, and use the polynomial coefficients of the 5*5 scale block with the minimum fitting error as the learning label. Through 5 5*5*5 convolutions with a step size of 5, the learning result B representing the 5*5 Y channel histogram transformation curve is obtained; Step 3.3: Use three 3*3*5 convolution kernels to convolve F2 to obtain feature F3, and use the polynomial coefficients of the 7*7 scale block with the minimum fitting error as the learning label. Through five 7*7*3 convolutions with a step size of 7, the learning result C representing the 7*7 Y channel histogram transformation curve is obtained; Step 3.4: Perform quick table lookup transformation on the learning results A, B, and C respectively, perform multi-scale fusion with the foggy image data to obtain F4, and output the image defogging result through 3*3*4 convolution.

6. The deep learning image defogging method based on enhancement curve learning according to claim 5, characterized in that: In step 3.4, the fast table lookup is transformed into a 2-5 power change of 0-255.

7. The deep learning image defogging method based on enhancement curve learning according to claim 5, characterized in that: In step 3.4, after outputting the image defogging result, it also includes: comparing the image defogging result with GT and optimizing it using L1_loss.

8. A deep learning image defogging system based on enhanced curve learning using the deep learning image defogging method based on enhanced curve learning according to any one of claims 1 to 7, characterized in that: include: Brightness value acquisition module: used to obtain the brightness value of foggy image data in YUV color space; Pre-training initial weight determination module: used to crop the foggy image data into blocks of different scales, and use polynomial fitting to transform the Y channel histogram of the block to the GT curve, and obtain the polynomial coefficient of the block with the minimum fitting error based on the least squares method as the learning label of the foggy image data at different scales; Image defogging module: Based on the learning labels, deep learning is used to learn the Y channel histogram transformation curve of the foggy image data at different scales, and the learning results at different scales are multi-scale fused to output the image defogging result.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of a deep learning image defogging method based on enhancement curve learning as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the deep learning image defogging method based on enhancement curve learning as described in any one of claims 1 to 7 are implemented.

Images