An underwater image enhancement method based on the fusion of a dehydration model and naturalness details

Abstract

The present invention discloses an underwater image enhancement method based on the fusion of a dewatering model and naturalness details. The method includes: inputting an underwater image into an inter-channel attenuation difference model to output the inter-channel attenuation difference; inputting the underwater image into an underwater environmental illumination model to output the background light value of the underwater environment; outputting an underwater image with underwater effects removed after being processed by the dewatering model; performing contrast enhancement; inputting each channel into a fusion weight model to output a fusion weight for increasing naturalness; performing image fusion to obtain a weight-fused underwater image; and performing two convolutional fusions to obtain an enhanced underwater image, thereby realizing the enhancement of the underwater image. The method of the present invention can continuously produce stable enhancement results, effectively improve various characteristics of underwater images, endow the generated images with a comfortable and natural tone, and improve the visual perception of underwater operations by on-board remote operators; in addition, the extremely simple dewatering model is beneficial to real-time processing and scalability, providing a new solution for the dewatering process of deep-sea images.

Description

Technical Field

[0001] The present invention relates to an underwater image enhancement method, belonging to the technical field of underwater image processing, and particularly to an underwater image enhancement method based on the fusion of a dewatering model and naturalness details. Background Art

[0002] High-definition and high-quality images are crucial for providing an immersive visual experience for shipboard operators. In the marine environment, unmanned underwater devices with advanced imaging systems are required to meet the need to replace human divers to venture into the deep sea to obtain information. These imaging systems are usually equipped with sonar and visual sensors. Among them, sonar sensors are good at contour detection and target recognition, providing macroscopic detection capabilities. Visual sensors, on the other hand, help to obtain rich-content images, including colors, contours, and other relevant information, thus achieving precise close-range perception. As a key tool for perceiving complex details of the underwater environment, visual sensors are widely used in fields such as marine resource exploration, marine ecological research, and marine autonomous monitoring. Due to the inherent properties of the water medium and the presence of suspended particles, deep-sea images exhibit degradation phenomena such as reduced brightness, color distortion, weakened contrast, and blurriness, which greatly affect the quality and visual fidelity of the captured images.

[0003] In deep learning solutions for underwater image processing, there is usually a lack of high-quality underwater datasets to train supervised learning methods. And the underwater datasets generated by style transfer methods based on generative adversarial networks cannot guarantee reliability in practical applications. In terms of enhancing underwater image processing, traditional methods actually work, and many pixel-based and model-based methods already exist. However, most existing traditional underwater image enhancement methods often do not start from the important property of the attenuation rate of light propagating in water, and those methods that mention attenuation dependence lack wavelength dependence. Summary of the Invention

[0004] In order to solve the problems existing in the background art, the present invention provides an underwater image enhancement method based on the fusion of a dewatering model and naturalness details.

[0005] The technical solution adopted by the present invention is as follows:

[0006] The underwater image enhancement method based on the fusion of a dewatering model and naturalness details of the present invention includes:

[0007] 1) Establish an inter-channel attenuation difference model for underwater images, input the underwater image in the RGB three channels into the inter-channel attenuation difference model, and the inter-channel attenuation difference model outputs the inter-channel attenuation difference.

[0008] 2) Establish an underwater environmental illumination model for the underwater image. Input the underwater image in the RGB three channels into the underwater environmental illumination model, and the underwater environmental illumination model outputs the water environmental background light value.

[0009] 3) Establish a dewatering model for the underwater image. Input the underwater image in the RGB three channels, the inter-channel attenuation difference, and the water environmental background light value into the dewatering model, and the dewatering model outputs an underwater image with the underwater effect preliminarily removed.

[0010] 4) Enhance the contrast of the underwater image with the underwater effect preliminarily removed to obtain the L* channel in the HSL color space after enhancement of the L channel in the HSL color space of the underwater image.

[0011] 5) Establish a fusion weight model for the underwater image. Input the S channel in the HSB color space, the L channel in the HSL color space, and its enhanced L* channel in the HSL color space of the underwater image into the fusion weight model, and the fusion weight model outputs a naturalness-increasing fusion weight; According to the underwater image and the underwater image with the underwater effect removed, use the naturalness-increasing fusion weight for image fusion to obtain a weight-fused underwater image.

[0012] 6) Perform two convolutional fusions on the weight-fused underwater image to obtain an enhanced underwater image, realizing the enhancement of the underwater image.

[0013] The RGB three channels are represented by the three-primary color mode based on red (Red), green (Green), and blue (Blue); the L channel in the HSL color space is represented by hue (Hue), saturation (Saturation), and brightness (Brightness); the S channel in the HSB color space is represented by hue (Hue), saturation (Saturation), and lightness (Lightness).

[0014] In the said step 1), the inter-channel attenuation difference model of the underwater image is specifically as follows:

[0015] S c =(I c -Z)·Z

[0016] where c represents the RGB three channels, and S c represents the inter-channel attenuation difference of the c channel; I c represents the underwater image under the c channel; Z represents the optimal color channel, and the initial difference is instantiated for the attenuation difference of the picture by multiplying by the optimal color channel Z;

[0017] Select one channel with a higher channel mean from the blue channel B and the green channel G of the c channel as the optimal color channel Z.

[0018] The inter-channel attenuation difference model is used to reflect the relationship between the attenuation coefficient and the visible wavelength in seawater, and is defined as a three-dimensional matrix of size h×w×3 used to capture the pixel value difference between each channel and the least attenuated channel, where h represents the height of the image and w represents the width of the image.

[0019] In step 2), the underwater environmental illumination model of the underwater image is specifically as follows:

[0020] A = max(B mean , G mean )

[0021]

[0022]

[0023] Among them, A represents the water environment background light value; P and |P| respectively represent the set of pixel points in the top one-thousandth of the dark channel values and its size, B mean and G mean respectively represent the average values of the blue channel B and the green channel G of the pixel point x in the pixel point set P; B(x) and G(x) respectively represent the blue channel B and the green channel G of the pixel point x.

[0024] Take the maximum value of the blue-green channel B from the set of points in the top one-thousandth of the dark channel values of the image as the water environment background light value of the underwater image.

[0025] The pixel point set P in the top one-thousandth of the dark channel values is specifically as follows:

[0026] P = {x|CDF(D(x)) > 0.999}

[0027]

[0028] Among them, CDF() represents the cumulative distribution function; D(x) represents the dark channel value of the pixel point x; Ω(x) represents the smallest operation area centered on the pixel point x; I c () represents the underwater image in the c channel; y represents all pixels in Ω(x).

[0029] The cumulative distribution function CDF can obtain the distribution probability of the dark channel value corresponding to the x pixel point, and the values with a cumulative distribution probability greater than 0.999 are taken and added to the point set P. The dark channel matrix D(x) is a two-dimensional matrix of size h×w, where h represents the height of the image and w represents the width of the image. The above operation is performed on each pixel position x of the image to obtain the dark channel matrix. Ω(x) is a minimum operation area, and the choice of the area size will affect the dark channel result. Selecting a larger area can obtain a darker dark channel, but in order to avoid using the same value for the entire image, our choice cannot be too large. Since the original underwater illumination is low, in order to retain some details, we specifically select a patch size of 7×7.

[0030] In the step 3) described above, the underwater image dewatering model is specifically as follows:

[0031]

[0032] Among them, c represents the RGB three channels, and E c (x) represents the underwater image after preliminary removal of the underwater effect of the underwater image at the pixel point x in the c channel processed by the dewatering model; I c (x) represents the underwater image at the pixel point x in the c channel; S c represents the inter-channel attenuation difference of the c channel; D(x) represents the dark channel value of the pixel point x; A represents the water environment background light value; α represents a preset coefficient.

[0033] During the processing of the RGB image, the values of each channel will be normalized, that is, E∈[0,1], which is of the same order of magnitude as the attenuation difference. The value of α can be selected as 1 to simplify the operation, which is computationally efficient and can achieve good visual effects.

[0034] In the step 4) described above, enhancing the contrast of the underwater image after preliminary removal of the underwater effect is specifically to perform image enhancement using the adaptive histogram method with contrast limited, specifically, performing contrast limited adaptive histogram equalization on the output image E after preliminary dewatering treatment to obtain E′, and converting E′ to the HSL space to obtain the L* channel.

[0035] In the step 5) described above, the underwater image fusion weight model is specifically as follows:

[0036] weight = ω1·weight HSL +ω2·weight HSB

[0037] Among them, weight represents the fusion weight for increasing naturalness; ω1 and ω2 respectively represent the first and second weight coefficients. During specific implementation, ω1 = 1 and ω2 = 0.875; weight HSL represents the HSL space weight component; weightHSB Represents the HSB space weight component.

[0038] The HSL space weight component weight mentioned above HSL Specifically as follows:

[0039] weight HSL = |L - L * |

[0040] Wherein, L and L * respectively represent the L-channel matrix of the HSL color space of the underwater image and its enhanced L*-channel matrix of the HSL color space.

[0041] The HSB space weight component weight mentioned above HSB Specifically as follows:

[0042]

[0043] Wherein, S represents the S-channel matrix of the HSB color space of the underwater image.

[0044] In the said step 6), performing two convolutional fusions specifically means using two convolutional layers to fuse the weighted underwater image. Specifically in implementation, the convolutional kernels of the two convolutional layers are specifically as follows:

[0045]

[0046]

[0047] Wherein, the c1 convolutional kernel is mainly used for image sharpening, and the c2 convolutional kernel is mainly used for enhancing image details.

[0048] The novel dewatering model proposed by the present invention better adapts to the attenuation characteristics of underwater light by introducing the attenuation difference between channels. The method of the present invention can improve the degradation phenomena such as reduced brightness, color distortion, weakened contrast and blurring of deep-sea images, and can continuously generate stable enhancement results, adjust to a comfortable and natural tone and has high robustness for images with different degrees of color shift and quality degradation for deep-sea image enhancement.

[0049] The beneficial effects of the present invention are:

[0050] The enhancement method of the present invention is significantly effective in improving multiple characteristics of underwater images, capable of continuously generating stable enhancement results, and effectively improving various characteristics of underwater images, including saturation, chroma, color balance, contrast, sharpness, etc. The method can continuously generate stable enhancement results, making the generated images present a comfortable and natural tone, and improving the visual perception of underwater operations by remote operators on the ship. In addition, the method shows good adaptability to images with different degrees of color shift and quality attenuation. Moreover, the dewatering model in the present invention is relatively simple in form, which is conducive to real-time processing and scalability, providing an innovative solution for the dewatering treatment of deep-sea images. Brief Description of the Drawings

[0051] Figure 1 is the flowchart of the method of the present invention;

[0052] Figure 2 is an example diagram of the present invention in detail enhancement;

[0053] Figure 3 is a schematic diagram of the test results of representative pictures selected in the embodiment of the present invention, where Figure 3 (a) is the original underwater image, Figure 3 (b) is the benchmark truth Ground Truth diagram given in the dataset, Figure 3 (c) is the processing result diagram of the histogram equalization HE method, Figure 3 (d) is the processing result diagram of the method based on fusion Fusion, Figure 3 (e) is the processing result diagram of the rank-one matrix prior Rank method, Figure 3 (f) is the processing result diagram of the color correction RDPS method based on the theory of retinal cortex image processing, Figure 3 (g) is the processing result diagram of the generated underwater dark channel prior GUDCP method, Figure 3 (h) is the processing result diagram of the underwater adversarial generation network UWGAN method for the conversion between the real world and the underwater world, Figure 3 (i) is the processing result diagram of the data-driven underwater image enhancement network WaterNet method, Figure 3 (j) is the processing result diagram of the deep learning network WaveNet for underwater image restoration based on wavelength, Figure 3 (k) is the processing result diagram of the present invention. Detailed Embodiment

[0054] To more clearly describe the objectives, technical solutions, and advantages of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific examples. It should be understood that the specific examples described below are only used to explain the present invention, and the scope of the present invention is not limited by the following examples. In addition, the technical features involved in each embodiment of the present invention can be combined with each other as long as they do not conflict with each other.

[0055] As Figure 1 shown, the underwater image enhancement method based on the combination of a dewatering model and naturalness details of the present invention includes:

[0056] 1) Establish an inter-channel attenuation difference model for underwater images. Input the underwater image in the RGB three channels into the inter-channel attenuation difference model, and the inter-channel attenuation difference model outputs the inter-channel attenuation difference.

[0057] The RGB three channels are represented by the three primary color modes based on Red, Green, and Blue; the L channel in the HSL color space is represented by Hue, Saturation, and Brightness; the S channel in the HSB color space is represented by Hue, Saturation, and Lightness.

[0058] In step 1), the inter-channel attenuation difference model for the underwater image is specifically as follows:

[0059] S c =(I c -Z)·Z

[0060] where c represents the RGB three channels, and S c represents the inter-channel attenuation difference of the c channel; I c represents the underwater image in the c channel; Z represents the optimal color channel, and the initial difference is instantiated for the attenuation difference of the picture by multiplying the optimal color channel Z.

[0061] Select one channel with a higher channel mean from the blue channel B and the green channel G of the c channel as the optimal color channel Z.

[0062] The inter-channel attenuation difference model is used to reflect the relationship between the attenuation coefficient and the visible wavelength in seawater, and is defined as a three-dimensional matrix of size h×w×3 for capturing the pixel value difference between each channel and the least attenuated channel, where h represents the height of the image and w represents the width of the image.

[0063] 2) Establish an underwater environmental illumination model for underwater images. Input the underwater image in the RGB three channels into the underwater environmental illumination model, and the underwater environmental illumination model outputs the water environment background light value.

[0064] In step 2), the underwater environmental illumination model of the underwater image is specifically as follows:

[0065] A = max(B mean , G mean )

[0066]

[0067]

[0068] where A represents the water environment background light value; P and |P| respectively represent the set of the top one-thousandth of the pixel points in the dark channel value and its size, and G mean and G mean respectively represent the mean values of the blue channel B and the green channel G of the pixel point x in the pixel point set P; B(x) and G(x) respectively represent the blue channel B and the green channel G of the pixel point x.

[0069] Take the maximum value of the blue-green channel B from the set of the top one-thousandth of the dark channel values of the image as the water environment background light value of the underwater image.

[0070] The set P of the top one-thousandth of the pixel points in the dark channel value is specifically as follows:

[0071] P = {x|CDF(D(x)) > 0.999}

[0072]

[0073] where CDF() represents the cumulative distribution function; D(x) represents the dark channel value of the pixel point x; Ω(x) represents a minimum operation area centered on the pixel point x; I c () represents the underwater image in the c channel; y represents all pixels in Ω(w).

[0074] The cumulative distribution function CDF can obtain the distribution probability of the dark channel value corresponding to the x pixel point, and take the values with a cumulative distribution probability greater than 0.999 and add them to the point set P. The dark channel matrix D(x) is a two-dimensional matrix with a size of h×w, where h represents the height of the image and w represents the width of the image. The above operations are performed on each pixel position x of the image to obtain the dark channel matrix. Ω(x) is a minimum operation area, and the selection of the area size will affect the dark channel result. Selecting a larger area can obtain a darker dark channel, but in order to avoid using the same value for the entire image, our selection cannot be too large. Since the original illumination underwater is low, in order to retain some details, we specifically select a patch size of 7×7.

[0075] 3) Establish a dewatering model for the underwater image, input the underwater image in the RGB three channels, the inter-channel attenuation difference, and the water environment background light value into the dewatering model, and the dewatering model outputs an underwater image with the underwater effect preliminarily removed.

[0076] In step 3), the de-watering model for underwater images is specifically as follows:

[0077]

[0078] Among them, c represents the three RGB channels, and E c (x) represents the underwater image after preliminary removal of underwater effects processed by the de-watering model for the underwater image in channel c where pixel point x is located; I c (x) represents the underwater image in channel c where pixel point x is located; S c represents the inter-channel attenuation difference of channel c; D(x) represents the dark channel value of pixel point x; A represents the background light value of the water environment; α represents a preset coefficient.

[0079] During the processing of RGB images, the values of each channel will be normalized, that is, E ∈ [0, 1], which is of the same order of magnitude as the attenuation difference. The value of α can be selected as 1 to simplify the operation, with high calculation efficiency and good visual effects.

[0080] 4) Enhance the contrast of the underwater image after preliminary removal of underwater effects to obtain the enhanced L* channel of the L channel in the HSL color space of the underwater image.

[0081] In step 4), enhancing the contrast of the underwater image after preliminary removal of underwater effects is specifically to perform image enhancement using the adaptive histogram method with contrast limited, specifically, perform contrast-limited adaptive histogram equalization on the output image E after preliminary de-watering processing to obtain E′, and convert E′ to the HSL space to obtain the L* channel.

[0082] 5) Establish a fusion weight model for the underwater image, input the S channel of the HSB color space, the L channel of the HSL color space, and its enhanced L* channel of the HSL color space of the underwater image into the fusion weight model. The fusion weight model outputs a fusion weight for increasing naturalness; according to the underwater image and the underwater image after removing underwater effects, use the fusion weight for increasing naturalness to perform image fusion to obtain a weight-fused underwater image.

[0083] In step 5), the fusion weight model for the underwater image is specifically as follows:

[0084] weight = ω1 · weight HSL + ω2 · weight HSB

[0085] Among them, weight represents the fusion weight for increasing naturalness; ω1 and ω2 respectively represent the first and second weight coefficients. In specific implementation, ω1 = 1 and ω2 = 0.875; weight HSLRepresents the weight component in the HSL color space; weight HSB Represents the weight component in the HSB color space.

[0086] The weight component in the HSL color space, weight HSL Specifically as follows:

[0087] weight HSL = |L - L * |

[0088] where, L and L * respectively represent the L-channel matrix of the HSL color space of the underwater image and its enhanced L*-channel matrix of the HSL color space.

[0089] The weight component in the HSB color space, weight HSB Specifically as follows:

[0090]

[0091] where, S represents the S-channel matrix of the HSB color space of the underwater image.

[0092] 6) After performing two convolutional fusions on the weight-fused underwater image, an enhanced underwater image is obtained, realizing the enhancement of the underwater image.

[0093] In step 6), performing two convolutional fusions specifically means using two convolutional layers on the weight-fused underwater image. Specifically, when implemented, the convolutional kernels of the two convolutional layers are specifically as follows:

[0094]

[0095]

[0096] where, the c1 convolutional kernel is mainly used for image sharpening, and the c2 convolutional kernel is mainly used for enhancing image details.

[0097] As Figure 2 shown, it is a schematic diagram of the pixel-by-pixel fusion process in the image. From top to bottom, they are: no fusion, fusion using Conv1, and fusion using Conv2. On the right is an enlargement of the framed part on the left.

[0098] As Figure 3 shown, this invention selects 7 representative pictures from the underwater image enhancement benchmark dataset UIEB (Underwater Image Enhancement Benchmark Dataset) for testing, which is a comparative experiment on the comprehensive performance of this invention when used on deep-sea pictures; as Figure 3 in (a), Figure 3 in (b), Figure 3 in (c),Figure 3 of (d), Figure 3 of (e), Figure 3 of (f), Figure 3 of (g), Figure 3 of (h), Figure 3 of (i), Figure 3 of (j) and Figure 3 of (k), it can be seen from the comparison diagrams that the enhanced images of this method have a comfortable and natural tone, and can provide uniform and reliable restoration in terms of contrast, brightness, and color deviation even for different pictures, significantly improving the visual perception of underwater images by humans. Moreover, the visual clarity of both the foreground and the background is enhanced, showing clear details.

[0099] The above content only expresses the technical idea of the present invention and cannot be used to limit the protection scope of the present invention. Any modifications made on the basis of the technical solution according to the technical idea proposed by the present invention fall within the protection scope of the claims of the present invention. It should be understood that the protection scope of the present invention is not limited by the above content.

Claims

1. An underwater image enhancement method based on the fusion of a dehydration model and naturalness details, characterized in that Including: 1) Establish an inter-channel attenuation difference model for underwater images. Input the underwater image in the RGB three channels into the inter-channel attenuation difference model, and the inter-channel attenuation difference model outputs the inter-channel attenuation difference; 2) Establish an underwater environmental illumination model for underwater images. Input the underwater image in the RGB three channels into the underwater environmental illumination model, and the underwater environmental illumination model outputs the water environmental background light value; 3) Establish a dewatering model for underwater images. Input the underwater image in the RGB three channels, the inter-channel attenuation difference, and the water environmental background light value into the dewatering model, and the dewatering model outputs an underwater image with underwater effects preliminarily removed; 4) Enhance the contrast of the underwater image with underwater effects preliminarily removed to obtain the enhanced L* channel in the HSL color space of the L channel of the underwater image in the HSL color space; 5) Establish a fusion weight model for underwater images. Input the S channel in the HSB color space, the L channel in the HSL color space, and its enhanced L* channel in the HSL color space of the underwater image into the fusion weight model, and the fusion weight model outputs a naturalness-increasing fusion weight; According to the underwater image and the underwater image with underwater effects removed, use the naturalness-increasing fusion weight to perform image fusion to obtain a weight-fused underwater image; 6) After performing two convolutional fusions on the weight-fused underwater image, an enhanced underwater image is obtained, realizing the enhancement of the underwater image; In step 3) described above, the dewatering model of the underwater image is specifically as follows: Among them, c represents the three RGB channels, and E c (x) represents the underwater image with the preliminary removal of the underwater effect after being processed by the dewatering model for the underwater image in channel c where pixel point x is located; I c (x) represents the underwater image in channel c where pixel point x is located; S c represents the inter-channel attenuation difference of channel c; D(x) represents the dark channel value of pixel point x; A represents the water environment background light value; α represents a preset coefficient; In step 5) described above, the fusion weight model of the underwater image is specifically as follows: weight = ω1·weight HSL + ω2·weight HSB Among them, weight represents the weight for enhancing naturalness fusion; ω1 and ω2 respectively represent the first and second weight coefficients; weight HSL represents the weight component in the HSL space; weight HSB represents the weight component in the HSB space; The HSL space weight component weight mentioned above HSL is as follows: weight HSL = |L - L * | Among them, L and L * respectively represent the L-channel matrix of the HSL color space of the underwater image and its enhanced L*-channel matrix of the HSL color space; The HSB space weight component weight mentioned above HSB is as follows: Where S represents the S channel matrix in the HSB color space of the underwater image.

2. The underwater image enhancement method based on the dehydration model and the fusion of naturalness details according to claim 1, characterized in that: The RGB three channels are represented by the three-primary color mode based on red (Red), green (Green), and blue (Blue); the L channel in the HSL color space is represented by hue (Hue), saturation (Saturation), and brightness (Brightness); the S channel in the HSB color space is represented by hue (Hue), saturation (Saturation), and lightness (Lightness).

3. The underwater image enhancement method based on the combination of dewatering model and naturalness details according to claim 1, characterized in that: In step 1) described above, the inter-channel attenuation difference model of the underwater image is specifically as follows: S c = (I c - Z) · Z Among them, c represents the three RGB channels, and S c represents the inter-channel attenuation difference of the c channel; I c represents the underwater image under the c channel; Z represents the optimal color channel; Select the channel with a higher channel mean from the blue channel B and the green channel G of the c channel as the best color channel Z.

4. The underwater image enhancement method based on the combination of dewatering model and naturalness details according to claim 1, characterized in that: In step 2) described above, the underwater environmental illumination model of the underwater image is specifically as follows: A = max(B mean , G mean ) Among them, A represents the background light value of the water environment; P and |P| respectively represent the set of pixel points in the top one-thousandth of the dark channel values and its size, B mean and G mean respectively represent the average values of the blue channel B and the green channel G of the pixel point x in the pixel point set P; B(x) and G(x) respectively represent the blue channel B and the green channel G of the pixel point x.

5. The underwater image enhancement method based on the fusion of dewatering model and naturalness details according to claim 4, characterized in that: The set P of the top one-thousandth pixel points in terms of the dark channel value of the underwater image is specifically as follows: P = {x|CDF(D(x)) > 0.999} Among them, CDF() represents the cumulative distribution function; D(x) represents the dark channel value of pixel point x; Ω(x) represents the smallest operation area centered on pixel point x; I c () represents the underwater image in the c channel; y represents all pixels in Ω(x).

6. The underwater image enhancement method based on the combination of a dewatering model and naturalness details according to claim 1, characterized in that: In step 4) described above, enhancing the contrast of the underwater image with underwater effects preliminarily removed is specifically to perform image enhancement using the adaptive histogram method with contrast limited.

Images