Low-illumination underwater image enhancement method based on Zero-shot learning and improved white balance

Abstract

The invention discloses a low-illumination underwater image enhancement method based on Zero-shot learning and improved white balance, and the method comprises the following steps: S1, inputting an image into a three-branch full convolutional network, decomposing the image into a reflection image, an illumination image and a noise image through the three-branch full convolutional network, then constructing a loss function composed of reconstruction loss, texture detail loss and noise loss, carrying out iterative fusion on the reflection image by using the loss function, carrying out noise prediction and denoising by using a noise image in the iteration process, and carrying out illumination recovery and iterative convergence on the reflection image by using an illumination image to obtain a fused image; s2, carrying out secondary iterative calculation on the fused image through a gray world algorithm; the image is processed through the three-branch full convolutional network, data set training is not needed, then white balance processing is carried out on the processed image through the secondary gray world algorithm and the perfect reflection algorithm, the problems of color cast, chromatic aberration and contrast of the image are reduced, and meanwhile the method has the advantage of being high in processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly relates to a low-light underwater image enhancement method based on Zero-shot learning and improved white balance. Background Art

[0002] Humans rely on precious underwater resources. However, due to the underwater environment usually facing problems such as underwater turbulent diffusion, severe water body absorption and scattering, various types of noise, low contrast, monotonous color, and complex background, in order to effectively detect the underwater environment and reduce potential safety hazards, developing an underwater image enhancement method for high turbidity and low light has great significance for practical applications.

[0003] Currently, the two main methods for underwater image processing technology are restoration-based methods and enhancement-based methods; restoration-based methods rely heavily on the underwater optical imaging model, while enhancement-based methods have a weaker dependence on the model. The purpose of underwater image enhancement is to improve the visual effect, applicability, contrast, and clarity of underwater images for specific application scenarios, highlight image details, and achieve images under normal atmospheric illumination; existing underwater image enhancement algorithms are divided into traditional image enhancement methods, underwater imaging model methods, and deep learning methods. Traditional image enhancement methods are further divided into spatial domain, transform domain, and comprehensive methods. Deep learning methods are divided into attenuation-based and non-attenuation-based models. Since it is difficult to obtain underwater low-light images and the dataset is scarce, it is relatively difficult to train neural networks, and they lack flexibility. For underwater imaging model methods such as the Retinex model, according to the characteristics of underwater images and various priors, a regularization term and weight components are constructed. Then, a suitable solution method is selected to solve the model to obtain the illumination component and reflectance component. Finally, the illumination component is adjusted through gamma correction and fused with the reflectance component to obtain the final output image. However, due to the lack of image correction processing, there are problems of color offset and overcompensation of attenuation color. Summary of the Invention

[0004] The purpose of the present invention is to provide a low-light underwater image enhancement method based on Zero-shot learning (or called zero-sample learning) and improved white balance. The present invention processes images through a three-branch fully convolutional network without dataset training, and then performs white balance processing on the processed images through the quadratic gray world algorithm and the perfect reflection algorithm, reducing color deviation, color difference, and contrast problems of the images, and at the same time having the characteristics of high processing efficiency.

[0005] The technical solution of the present invention: A low-light underwater image enhancement method based on Zero-shot learning and improved white balance is carried out according to the following steps:

[0006] Step S1: The image is input into a three-branch fully convolutional network, which decomposes the image into a reflection map, an illumination map, and a noise map. Subsequently, a loss function composed of a reconstruction loss, a texture detail loss, and a noise loss is constructed, and the reflection map is iteratively fused using the loss function. During the iterative process, the noise map is used for noise prediction and denoising, and the illumination map is used to restore the illumination of the reflection map. After the iteration converges, a fused image is obtained;

[0007] Step S2: First, the fused image is subjected to a secondary iterative calculation through the gray world algorithm to correct the R, G, and B channel values of the fused image, and then the perfect reflection algorithm is used to perform white balance processing on the fused image;

[0008] Step S3: The fused image obtained in Step S2 is subjected to gamma correction as a grayscale image, and finally an enhanced image is output.

[0009] In the above low-light underwater image enhancement method based on Zero-shot learning and improved white balance, the decomposition process of the three-branch fully convolutional network for the reflection map, illumination map, and noise map is carried out according to the following formula:

[0010] I(x) = R(x)·S(x) + N(x);

[0011] where I is the input image, R is the reflection map, S is the illumination map, and N is the noise map.

[0012] In the aforementioned low-light underwater image enhancement method based on Zero-shot learning and improved white balance, the loss function is shown as follows:

[0013] L = L r + λ t L t + λ n L n ;

[0014] where L r is the reconstruction loss, L t is the texture detail loss, L n is the noise loss, and λ t and λ n are both weight coefficients.

[0015] In the aforementioned low-light underwater image enhancement method based on Zero-shot learning and improved white balance, the reconstruction loss is expressed as:

[0016] Lr = ||I - (R·S + N)||1 + ||S - S0||1 + ||R - I / S||1;

[0017] Among them, S0 is the maximum value of the R, G, and B channels and serves as the initial estimate of illumination.

[0018] In the aforementioned low-light underwater image enhancement method based on Zero-shot learning and improved white balance, the texture detail loss is obtained by segmentally smoothing the low-light region, as shown in the following formula:

[0019]

[0020] Among them, is the partial derivative in the horizontal x direction, is the partial derivative in the vertical y direction, w x and w y are respectively expressed as:

[0021]

[0022]

[0023] Among them, G is the Gaussian filter, is the convolution, and I g is the input grayscale version.

[0024] In the aforementioned low-light underwater image enhancement method based on Zero-shot learning and improved white balance, the noise loss is expressed as the weighted sum of the light-guided noise loss and the segmentally smoothed reflection loss, as shown in the following formula:

[0025]

[0026] Among them, w n and w r are the weights of light guidance and are respectively expressed as:

[0027] w n (x) = I(x)

[0028]

[0029] Among them, normalize is the minimum-maximum normalization.

[0030] In the aforementioned low-light underwater image enhancement method based on Zero-shot learning and improved white balance, the process of performing secondary iterative calculation on the fused image through the gray world algorithm is as follows:

[0031] The first gray world algorithm calculation:

[0032]

[0033] Among them, They are the average values of the R, G, and B channels respectively, and K is the gain coefficient of the gray world algorithm;

[0034] The gain coefficients K of the R, G, and B channels R , K G , K B are expressed as:

[0035]

[0036] where R', G', and B' are the R, G, and B channel values under another light source; remap R', G', and B' into [0, 255] to obtain

[0037]

[0038] The second gray world algorithm is shown as follows:

[0039]

[0040] R” = K R 'R * ;

[0041]

[0042] B” = K B 'B * ;

[0043]

[0044] G” = K G 'G * ;

[0045]

[0046] where is the mean value of α1, α2, and α3, is the mean value of β1, β2, and β3, and K' is the gain coefficient of the quadratic gray world algorithm.

[0047] In the aforementioned low-light underwater image enhancement method based on Zero-shot learning and improved white balance, the perfect reflection algorithm is shown as follows:

[0048]

[0049] where R MAX , G MAX , B MAX are the maximum values of R, G, and B respectively, They are the average values of the R, G, and B channels respectively, λ R , λ G , λ B is the gain coefficient, and λ is the average value of the maximum values of the R, G, and B channels respectively. Its function is to perform mapping after correction. R 0 , B 0 , G 0 are the values of the R, G, and B channels after gain.

[0050] In the aforementioned low-light underwater image enhancement method based on Zero-shot learning and improved white balance, mapping correction is performed on the gray world algorithm and the perfect reflection algorithm to prevent the gray world algorithm from overflowing and improve the accuracy of the perfect reflection algorithm;

[0051] The mapping correction process is shown in the following formula:

[0052] uR” 2 +vR” = K'

[0053] u(R 0 ) 2 +vR 0 = λ;

[0054] After calculating u and v, further obtain through the orthogonal combination algorithm and coefficient distribution:

[0055]

[0056]

[0057] Among them, are the values of the R, G, and B channels obtained after the orthogonal combination algorithm.

[0058] Compared with the prior art, the present invention first inputs the image into a three-branch fully convolutional network, decomposes it into a reflection map, an illumination map, and a noise map, constructs a loss function composed of a reconstruction loss, a texture detail loss, and a noise loss, and iteratively fuses the reflection map with the illumination map and the noise map according to the loss function until convergence to obtain a fused image, which is updated with the loss function. It does not require any prior image examples or prior training, and has high flexibility in use; first performs secondary iterative calculation on the fused image through the gray world algorithm to correct the R, G, and B channel values of the fused image, and then uses the perfect reflection algorithm to perform white balance processing on the fused image to solve problems such as color difference and color cast; finally, uses the fused image after white balance processing as a grayscale image for gamma correction to further enhance the visuality of the image while improving the processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1This is the flowchart of the present invention;

[0060] Figure 2 It is the structural diagram of a three-branch fully convolutional network. Specific implementation manner

[0061] The present invention will be further described below in conjunction with the accompanying drawings and embodiments, but it shall not be used as a basis for limiting the present invention.

[0062] Embodiment: A low-light underwater image enhancement method based on Zero-shot learning (or zero-sample learning) and improved white balance, as shown in the attached Figure 1 figure, and is carried out according to the following steps:

[0063] Step S1: Input the image into the three-branch fully convolutional network (RRDNet). As shown in the attached Figure 2 figure, the three-branch fully convolutional network has three branches. The first branch performs multiple convolutions plus rectified linear units, and then performs convolution and activation to decompose the reflection map. The second branch performs multiple convolutions plus rectified linear units, and then performs convolution and activation to decompose the illumination map. The third branch performs multiple convolutions plus rectified linear units, and then performs convolution and hyperbolic tangent to decompose the noise map. The three-branch fully convolutional network decomposes the image into a reflection map, an illumination map, and a noise map. The reflection map is preprocessed and improved, and then a loss function composed of a reconstruction loss, a texture detail loss, and a noise loss is constructed. The loss function is used to iteratively fuse the reflection map. During the iterative process, the noise map is used for noise prediction and denoising, and the illumination map is used to restore the illumination of the reflection map. After the iteration converges, a fused image is obtained;

[0064] The decomposition process of the three-branch fully convolutional network for the reflection map, illumination map, and noise map is carried out according to the following formula:

[0065] I(x) = R(x)·S(x) + N(x);

[0066] where, I is the input image, R is the reflection map, S is the illumination map, and N is the noise map; in order to ensure the rationality of image decomposition, the Retinex theory needs to be satisfied, and the maximum value S0 of the R, G, and B channels is used as the initial estimate of illumination;

[0067] The loss function is shown in the following formula:

[0068] L = L r + λ t L t + λ n L n ;

[0069] where, L r is the reconstruction loss, L t is the texture detail loss, L nis the noise loss, λ t and λ n are both weight coefficients;

[0070] The reconstruction loss is expressed as:

[0071] Lr = ||I - (R·S + N)||1 + ||S - S0||1 + ||R - I / S||1;

[0072] Since under normal illumination, the surface illumination is usually relatively flat, using piecewise smoothing can effectively improve the low-illumination area, and the formula is as follows:

[0073]

[0074] Among them, is the partial derivative in the horizontal x direction, is the partial derivative in the vertical y direction, w x and w y are respectively expressed as:

[0075]

[0076] Among them, G is the Gaussian filter, is the convolution, I g is the input grayscale version;

[0077] Since stretching the contrast of the low-illumination dark area will enlarge the noise, in order to suppress the noise, the already estimated illumination map is used to estimate the dark area noise through weighted focusing, and the noise loss is expressed as the weighted sum of the illumination-guided noise loss and the piecewise-smooth reflection loss, which is expressed as follows:

[0078]

[0079] Among them, w n and w r are the illumination-guided weights, and are respectively expressed as:

[0080] w n (x) = I(x)

[0081]

[0082] Among them, normalize is the minimum-maximum normalization, aiming to limit the values of the noise map to a fixed range, and at the same time ensure that the smoothed reflection component suppresses the real noise rather than the edges. Therefore, weighting the above two items is dependent on the illumination map;

[0083] The underwater dataset UIEB was tested for the perceptual quality index (PCQI) and peak signal-to-noise ratio (PSNR) using a three-branch fully convolutional network, the Retinex algorithm, and the LIME algorithm. The results are shown in Table 1 below

[0084]

[0085] Table 1

[0086] As can be seen from Table 1, the three-branch fully convolutional network achieved the highest average PCQI and the second-highest average PSNR, showing good performance. The automatic Retinex algorithm, due to its built-in correction algorithm, also introduced additional noise, causing image distortion and resulting in a high PSNR value

[0087] Step S2: First, perform a secondary iterative calculation on the fused image through the gray world algorithm to correct the R, G, and B channel values of the fused image. Then, perform white balance processing on the fused image using the perfect reflection algorithm

[0088] The first gray world algorithm is shown as follows

[0089]

[0090] where are the average values of the R, G, and B channels respectively, and K is the gain coefficient of the gray world algorithm

[0091] The gain coefficients K R , K G , K B are expressed as

[0092]

[0093] where R', G', and B' are the R, G, and B channel values under another light source. Since there is a possibility of overflow in this algorithm, to prevent this situation, the calculated data is linearly remapped back to [0, 255] using the maximum value of the channel to obtain

[0094]

[0095] The second gray world algorithm is shown as follows

[0096]

[0097] R”=K R 'R * ;

[0098]

[0099] B” = K B 'B * ;

[0100]

[0101] G” = K G 'G * ;

[0102]

[0103] Wherein, is the average value of α1, α2, α3, is the average value of β1, β2, β3, and K' is the gain coefficient of the quadratic gray world algorithm;

[0104] The perfect reflection algorithm is as shown in the following formula:

[0105]

[0106] Wherein, R MAX , G MAX , B MAX are the maximum values of R, G, B respectively, are the average values of the three channels of R, G, B respectively, λ R , λ G , λ B are gain coefficients, λ is the average value of the maximum values of the three channels of R, G, B respectively, and its function is the mapping after correction. R 0 , B 0 , G 0 are the values of the three channels of R, G, B after gain.

[0107] On the basis of the original quadratic gray world algorithm and the linear mapping correction of the perfect reflection method, in order to prevent the gray world algorithm from overflowing and improve the accuracy of the perfect reflection algorithm, the mapping correction method is modified to the following form

[0108] uR” 2 + vR” = K'

[0109] u(R 0 ) 2 + vR 0 = λ;

[0110] Calculate u and v, and further obtain through the orthogonal combination algorithm and coefficient distribution:

[0111]

[0112] Wherein, They are the R, G, and B channel values obtained after the orthogonal combination algorithm.

[0113] The white balance processing proposed in this embodiment is compared with the average value method, perfect reflection, gray world, dynamic range, and depth white balance algorithms, and the results are shown in Table 2 below.

[0114]

[0115] Table 2

[0116] The information entropy reflects the richness of image information; the UCIQE index uses a linear combination of chromaticity, saturation, and contrast for quantitative evaluation, quantifying uneven color cast, blur, and low contrast respectively; the UIQM refers to the linear combination of the color measurement index (UICM), sharpness measurement index (UISM), and contrast measurement index (UIConM), and the larger the value, the better the color balance, sharpness, and contrast of the image; it can be seen from Table 2 that the white balance processing in this embodiment obtains the highest information entropy and UCIQE values, as well as the second highest UIQM value, with high processing reliability.

[0117] Step S3: Gamma correct the fused image obtained in step S2 as a grayscale image, reducing the operation of graying the image and improving the efficiency. An ablation experiment is carried out on the fused image and the grayscale image of the input image, and the results are shown in Table 3 below.

[0118]

[0119] Table 3

[0120] It can be seen from the above table that the fused image is based on the reflection map and is basically similar to the grayscale image of the input image, with an error of only 0.2%.

[0121] Step S4: Output the enhanced image; compare the output image of this embodiment with other algorithms, and the results are shown in Table 4 below.

[0122]

[0123]

[0124] Table 4

[0125] This embodiment has higher visibility and a more natural visual effect, conforming to real illumination images.

[0126] In summary, the present invention decomposes the input image into three components: illumination, reflection, and noise through a three-branch fully convolutional network, and effectively estimates the noise and restores the illumination through iterative operations of a loss function, thereby clearly predicting the noise for the purpose of denoising, and having the characteristics of not requiring any previous image examples or previous training; the white balance processing of the present invention performs a secondary gray world algorithm on the low-illumination underwater image and combines it with the perfect reflection algorithm. The algorithm principle of the secondary gray world is to re-iterate the three-channel values after re-mapping the gray world algorithm into the gray world algorithm formula, so that the output three-channel values are more visually appealing and can better solve the color cast and color difference caused by the low-illumination underwater environment; in gamma correction, the fused image after white balance processing is directly used as a grayscale image, reducing the operation of graying the image and improving the efficiency.

Claims

1. A low-light underwater image enhancement method based on zero-shot learning and improved white balance, characterized in that: The steps are as follows: Step S1: The image is input into a three-branch fully convolutional network. The three-branch fully convolutional network decomposes the image into a reflection map, an illumination map, and a noise map. Subsequently, a loss function composed of a reconstruction loss, a texture detail loss, and a noise loss is constructed. The reflection map is iteratively fused using the loss function. During the iterative process, the noise map is used for noise prediction and denoising, and the illumination map is used to restore the illumination of the reflection map. After the iterative convergence, a fused image is obtained; Step S2: First, the fused image is subjected to a secondary iterative calculation through the gray world algorithm to correct the R, G, and B channel values of the fused image. Then, the perfect reflection algorithm is used to perform white balance processing on the fused image; Step S3: The fused image obtained in Step S2 is subjected to gamma correction as a grayscale image, and finally, an enhanced image is output.

2. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 1, characterized in that: The decomposition process of the three-branch fully convolutional network for the reflection map, illumination map, and noise map is carried out according to the following formula: I(x) = R(x)·S(x) + N(x); where, I is the input image, R is the reflection map, S is the illumination map, and N is the noise map.

3. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 2, characterized in that: The loss function is shown as follows: L = L r + λ t L t + λ n L n ; Among them, L r is the reconstruction loss, L t is the texture detail loss, L n is the noise loss, λ t and λ n are both weight coefficients.

4. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 3, characterized in that: The reconstruction loss is expressed as: Lr = ||I - (R·S + N)||1 + ||S - S0||1 + ||R - I / S||1; where, S0 is the maximum value of the R, G, and B channels and serves as the initial estimate of illumination.

5. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 4, wherein: The texture detail loss smooths the low-illumination area through piecewise smoothing, as shown in the following formula: Among them, is the partial derivative in the horizontal x direction, is the partial derivative in the vertical y direction, w x and w y are respectively expressed as: Among them, G is a Gaussian filter, is convolution, and I g is the grayscale version of the input.

6. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 5, characterized in that: The noise loss is expressed as the weighted sum of the illumination-guided noise loss and the piecewise smoothed reflection loss, as shown in the following formula: where, w n and w r are light-guided weights, respectively expressed as: w n (x) = I(x) where, normalize is the minimum-maximum normalization.

7. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 1, characterized in that: The process of performing a secondary iterative calculation on the fused image through the gray world algorithm is as follows: The first gray world algorithm calculation: Among them, are the average values of the R, G, and B channels respectively, and K is the gain coefficient of the gray world algorithm; Express the gain coefficients K of the R, G, and B channels R , K G , K B as follows: where, R', G', and B' are the R, G, and B channel values under another light source; R', G', and B' are remapped into [0, 255] to obtain The second gray world algorithm is as shown in the following formula: R” = K R 'R * ; B” = K B 'B * ; G” = K G 'G * ; Among them, is the mean value of α1, α2, and α3, is the mean value of β1, β2, and β3, and K' is the gain coefficient of the quadratic gray world algorithm.

8. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 7, wherein: The perfect reflection algorithm is as shown in the following formula: Among them, R MAX , G MAX , B MAX are respectively the maximum values of R, G, and B, are respectively the average values of the R, G, and B channels, λ R , λ G , λ B are gain coefficients, λ is the average value of the respective maximum values of the R, G, and B channels, and its function is the mapping after correction. R 0 , B 0 , G 0 are the values of the R, G, and B channels after gain.

9. The low-light underwater image enhancement method based on Zero-shot learning and improved white balance according to claim 8, characterized in that: Mapping correction is performed on the gray world algorithm and the perfect reflection algorithm to prevent the gray world algorithm from overflowing and improve the accuracy of the perfect reflection algorithm; The mapping correction process is as shown in the following formula: uR” 2 +vR” = K' u(R 0 ) 2 +vR 0 = λ; u and v are calculated, and further obtained through the orthogonal combination algorithm and coefficient assignment: Among them, are the R, G, and B channel values obtained after the orthogonal combination algorithm.

Images