Underwater image enhancement method integrating deep learning and traditional image enhancement technology

Abstract

The present invention discloses an underwater image enhancement method that integrates deep learning and traditional image enhancement technology. First, the average value difference between each channel of the input underwater image and its corresponding natural image is analyzed. From the difference, it can be seen that the red channel needs to be compensated and the green channel needs to be attenuated. Therefore, color compensation is performed on the R channel and G channel of the input underwater image using an attention-guided residual network. The motivation of this strategy is to observe that most underwater images are composed of relatively single and uniform color distributions. For scene contrast enhancement and scene deblurring, a multi-scale convolutional neural network is developed, in which CLAHE (Contrast-Limited Adaptive Histogram Equalization) and Gamma Correction Algorithm are introduced as supplements to handle complex and changeable underwater imaging environments. Experimental results show that the underwater image enhancement method of the present invention achieves relatively good results.

Description

Technical Field

[0001] The present invention relates to the field of image enhancement technology, and in particular to an underwater image enhancement method that integrates deep learning and traditional image enhancement technology. Background Art

[0002] The visual quality of underwater images is of great significance for the understanding and recognition of underwater scenes. However, due to the selective absorption and scattering of light by water, underwater images collected by underwater imaging equipment usually produce serious color distortion, blur, low contrast and brightness, which seriously affects people's understanding and perception of underwater scenes.

[0003] In the past few years, many different methods have been proposed to improve the visual quality of underwater images for underwater image enhancement. These methods can be roughly divided into underwater image enhancement methods based on traditional image processing, image restoration methods based on underwater image imaging optical models, and underwater image enhancement and restoration methods based on deep learning. Underwater image enhancement methods based on traditional image processing, such as histogram equalization, homomorphic filtering, and retina-based enhancement methods, do not consider the image degradation mechanism, directly enhance the feature pixels in the image, and suppress some unimportant features. However, since the image degradation mechanism is not considered, color distortion and oversaturation may still exist, resulting in the loss of some image information. Unlike underwater image enhancement methods based on traditional image processing, methods based on underwater image imaging optical models consider the optical imaging physical model, such as underwater image enhancement methods based on wavelength compensation, blue-green channel, and red channel, which can achieve better enhancement effects. Unlike traditional image enhancement and restoration methods, methods based on deep learning explore constraints from a large amount of training data to restore clear images. Despite significant progress, deep learning-based underwater image enhancement methods remain challenging due to the complexity of underwater imaging degradation mechanisms, which include local detail loss and global color distortion of pixels across the entire image.

[0004] As we all know, red light attenuates the most underwater and disappears the fastest, while green and blue light attenuate less underwater and are relatively well preserved. Underwater images generally appear blue-green, and the pixel values ​​of each channel vary, while the pixel values ​​of the corresponding natural images tend to be consistent. Therefore, this feature can be used to more clearly analyze the differences between underwater images and their corresponding natural images. By calculating the mean difference between the natural image and its corresponding underwater image, it can be found that the mean difference of the red channel is above 0, indicating that the red channel of the underwater image is severely attenuated, and pixel compensation should be performed on its channel; the mean difference of the green channel is below 0, indicating that the pixel values ​​of the green channel of the underwater image are relatively rich, and they should be appropriately attenuated; while the mean difference of the blue channel fluctuates around 0, without a certain regularity, so only a simple pixel enhancement is performed on the pixel values ​​of the blue channel. Summary of the invention

[0005] The purpose of the present invention is to provide an underwater image enhancement method that integrates deep learning and traditional image enhancement technology. The method combines color correction and contrast enhancement methods to obtain high-quality clear underwater images.

[0006] The technical solution adopted by the present invention is an underwater image enhancement method that integrates deep learning and traditional image enhancement technology, which specifically includes the following steps:

[0007] Step 1, input an underwater image dataset synthesized by cycleGAN, which includes pairs of degraded underwater images and clear underwater images, and divide the image dataset into a training set and a test set;

[0008] Step 2, preprocessing the training set and test set obtained in step 1;

[0009] Step 3, analyzing the mean difference between the paired degraded underwater images and the clear underwater images in the image data set of step 1;

[0010] Step 4: Based on the analysis results of step 3, an enhanced network model of the underwater image is constructed;

[0011] Step 5, using the training set preprocessed in step 2 to train the network model constructed in step 4 to obtain the optimal enhanced network model;

[0012] Step 6: Put the test set preprocessed in step 2 into the enhanced network model trained in step 5, and output the enhanced underwater image.

[0013] The present invention is also characterized in that:

[0014] In step 2, the process of preprocessing the test set and the training set is: normalizing the pixel values ​​of the image to [0, 1].

[0015] The specific process of step 3 is: according to the mean difference between the paired degraded underwater images and the clear underwater images, a mean difference map of three channels is drawn; the three channels are: red channel mean difference R, green channel mean difference G, and blue channel mean difference B; the mean difference of the red channel is above 0, indicating that the mean of the natural underwater image is greater than the mean of the degraded underwater image, so the red channel of the underwater image needs to be compensated; the mean difference of the green channel is below 0, indicating that the mean of the natural underwater image is less than the mean of the degraded underwater image, so the green channel of the underwater image needs to be attenuated; the mean difference of the blue channel fluctuates around 0, so the blue channel is enhanced.

[0016] The specific process of step 4 is:

[0017] Step 4.1, combine the convolution operation, instance normalization, and activation function ReLU into a convolution layer:

[0018] F=ReLU(Instance_norm(Conv(x))) (1);

[0019] Step 4.2, using deep learning technology to build a color correction network model;

[0020] Step 4.3, construct an attention-guided red channel color compensation module;

[0021] Step 4.4, constructing a blue channel network model;

[0022] Step 4.5, construct the green channel color attenuation module guided by attention;

[0023] Step 4.6, performing channel stitching on the results obtained in steps 4.3 to 4.5, and outputting a corrected underwater image;

[0024] In step 4.7, a contrast enhancement subnet composed of multi-scale convolutions is constructed to process the image obtained in step 4.6 and output the enhanced underwater image.

[0025] The specific process of step 5 is: use the L1 loss function and the Perceptual loss function to constrain the network model obtained in step 4, and then back-propagate to update the parameters to obtain the optimal underwater image enhancement network model;

[0026] The L1 loss function is shown in the following formula (2):

[0027]

[0028] The perceptual loss function is shown in the following formula (3):

[0029]

[0030] The total loss function is:

[0031] L=λ l L l +λ2L per (4).

[0032] The beneficial effect of the present invention is that the present invention combines the two methods of color correction and contrast enhancement, first obtains a color-balanced image, and then performs contrast enhancement on the color-balanced image, thereby obtaining a high-quality underwater image to provide high-quality data for subsequent underwater image processing work. Under this premise, it has a better enhancement effect and a higher evaluation index. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of the process of an underwater image enhancement method of the present invention that integrates deep learning and traditional image enhancement technology;

[0034] Figure 2 It is a mean difference map between the underwater image analyzed in the underwater image enhancement method of the present invention that integrates deep learning and traditional image enhancement technology and its corresponding natural image;

[0035] Figure 3 A schematic diagram of a color correction network structure in an underwater image enhancement method that integrates deep learning and traditional image enhancement technology in the present invention;

[0036] Figure 4 A schematic diagram of a contrast enhancement structure in an underwater image enhancement method that integrates deep learning and traditional image enhancement technology in the present invention;

[0037] Figure 5 (a) to (f) are comparison diagrams of the enhancement results of the underwater image enhancement method of the present invention that combines deep learning and traditional image enhancement technology. DETAILED DESCRIPTION

[0038] The technical solution of the present invention will be fully and systematically described below in conjunction with the accompanying drawings in the examples of the present invention.

[0039] The underwater image enhancement method of the present invention combines deep learning and traditional image enhancement technology, such as Figure 1 As shown, the specific steps include:

[0040] Step 1: Input the underwater image dataset synthesized by cycleGAN, which includes a total of 6128 pairs of underwater images (including 6128 degraded underwater images and corresponding 6128 clear underwater images). Select 4000 pairs of images for training, and select 100 pairs of images from the remaining images for testing.

[0041] Step 2: Preprocess the test set and training set, normalize the pixel values ​​of the images to [0, 1], and crop them to 256x256x3.

[0042] Step 3: The specific steps of analyzing the mean difference between the underwater image and its corresponding natural image are as follows:

[0043] Step 3: If Figure 2 As shown in the figure, the mean difference between the underwater image and its corresponding natural image is analyzed, and the mean difference graph of the three channels is plotted. The horizontal axis represents the number of samples, and the selected sample number is 3128 degraded underwater images and their corresponding 3128 clear natural underwater images. The vertical axis represents the mean difference between the natural image and its corresponding underwater image. Mean difference R represents the mean difference of the red channel of the natural image and its corresponding underwater image, meandifference G represents the mean difference of the green channel of the natural image and its corresponding underwater image, and mean difference B represents the mean difference of the blue channel of the natural image and its corresponding underwater image. From Figure 2 It can be seen that the mean difference of the red channel is above 0, indicating that the mean of the natural underwater image is greater than the mean of the degraded underwater image, so it is necessary to compensate the red channel of the underwater image; Figure 2 It can be seen that the mean difference of the green channel is below 0, indicating that the mean of the natural underwater image is smaller than the mean of the degraded underwater image, so the green channel of the underwater image needs to be attenuated. Figure 2 It can be seen that the mean difference of the blue channel fluctuates around 0, so appropriate enhancement operations are performed on the blue channel.

[0044] Step 4: Based on the analysis of the results of step 3, build an enhanced network model for underwater images. The specific steps are as follows:

[0045] Step 4.1, combine the convolution operation, instance normalization, and activation function ReLU into a convolution layer:

[0046] F=ReLU(Instance_norm(Conv(x)))

[0047] Step 4.2, use deep learning technology to build a color correction network model, such as Figure 3 As shown in the figure, Input represents the input image, Output represents the output image, and I1 represents the underwater image after color correction. GAP represents the global average pooling layer, FC represents the fully connected layer, sigmoid represents the activation function, CES represents the contrast enhancement subnet, c3x3 represents the convolution of size 3x3, 1x1 represents the convolution of size 1x1, and Rb represents the residual block RES_BLOCK. First, the underwater image is separated into three channels of RGB, and then each of the three channels passes through a convolution layer with a convolution kernel size of 3x3 and a stride of 1.

[0048] O1=F1(R)

[0049] O2=F2(G)

[0050] O3=F3(B)

[0051] The specific steps for constructing the red channel compensation network model in step 4 are as follows:

[0052] Step 4.3, construct the attention-guided red channel color compensation module, such as Figure 3 As shown, the red channel compensation module contains 5 branches. In the first branch, the input feature O1 passes through the GAP (global average pooling) layer, the FC (fully connected) layer, and the sigmoid (activation function) layer to obtain the channel attention feature map O4; then the learned channel weight is assigned to O1, that is, a multiplication operation is performed to complete the channel attention weighting and obtain the channel weighted feature O5; then the feature O1 passes through two convolution layers with a convolution kernel size of 3x3 and a step size of 1, and a sigmoid (activation function) layer to obtain the spatial attention feature map O6, and the channel weighted feature O5 passes through a convolution layer with a convolution kernel size of 1x1 and a step size of 1 to obtain feature O7. Feature O7 is multiplied with the spatial attention feature map O6 to obtain feature O8, completing the channel spatial attention weighting operation; 1-O1 is used to constrain the weights to prevent the weights from being over-weighted, and feature O9 is obtained; finally, feature O1 and feature O9 are element-wise added to compensate for some color information lost in the attention weighting process, and feature O 10 :

[0053] O4=sigmoid(FC(GAP(O1)))

[0054] O5=O1 x O4

[0055] O6=sigmoid(F4(O1))

[0056] O7=F5(O5)

[0057] O8=O7 x O6

[0058] O9=(1-O1)x O8

[0059] O 10 =O1+O9

[0060] Step 4.3.1: Output feature O 10 After a convolution layer with a kernel size of 3x3 and a stride of 1:

[0061] O 11 =F6(O 10 )

[0062] The specific steps for building the blue channel network model in step 4 are as follows:

[0063] Step 4.4, the output feature O2 is passed through four convolutional residual blocks RES_BLOCK1, RES_BLOCK2, RES_BLOCK3, and RES_BLOCK4 composed of jump connections in sequence:

[0064] O 12 =RES_BLOCK1(O2)

[0065] O 13 =RES_BLOCK2(O 12 )

[0066] O 14 =RES_BLOCK3(O 13 )

[0067] O 15 =RES_BLOCK4(O 14 )

[0068] Step 4.4.1: Output feature O 15 After a convolutional layer with a kernel size of 3x3 and a stride of 1:

[0069] O 16 =F7(O 15 )

[0070] The specific steps for constructing the green channel attenuation network model in step 4 are as follows:

[0071] Step 4.5, construct the green channel color attenuation module guided by attention, Figure 3 As shown in the figure, the green channel compensation module contains 4 branches. The first branch passes the input feature O3 through the GAP (global average pooling) layer, FC (fully connected) layer, and sigmoid (activation function) layer to obtain the channel attention feature map O 17 , and then assign the learned channel weight to O3, that is, perform a multiplication operation to complete the channel attention weighting and obtain the channel weighted feature O 18 ; Then the feature O3 passes through two convolution layers with a convolution kernel size of 3x3 and a step size of 1, and a sigmoid (activation function) layer to obtain the spatial attention feature map O 19 , the channel weighted feature O 18 Through a convolution layer with a convolution kernel size of 1x1 and a stride of 1, feature O is obtained. 20 , feature O 20 With the spatial attention feature map O 19 Perform multiplication operation to obtain feature O 21 , complete the weighted operation of channel space attention; finally, use the input feature O3 and feature O 21 Perform weight subtraction operation to obtain the green channel attenuation feature O22 :

[0072] O 17 =sigmoid(FC(GAP(O3)))

[0073] O 18 =O3 x O 17

[0074] O 19 =sigmoid(F8(O3))

[0075] O 20 =F9(O 18 )

[0076] O 21 =O 20 xO 19

[0077] O 22 =O3-O 21

[0078] Step 4.5.1: Output feature O 22 After a convolution layer with a kernel size of 3x3 and a stride of 1:

[0079] O 23 =F 10 (O 21 )

[0080] Step 4.6, the output features O of step 4.3.1, step 4.4.1, and step 4.5.1 11 , O 16 , O 23 Channel splicing is performed, and a convolution with a convolution kernel size of 3x3 and a stride of 1 is performed to output the color-corrected underwater image I1:

[0081] O 24 =F 11 (Concat(O 11 ,O 16 ,O 23 ))

[0082] The specific steps for constructing the contrast enhancement subnet in step 4 are as follows:

[0083] Step 4.7, construct a contrast enhancement subnet composed of multi-scale convolutions, such as Figure 4As shown in the figure, I1 is the underwater image after color correction, ICE represents the CLAHE (Contrast-Limited Adaptive Histogram Equalization) algorithm, and IGC represents the Gamma Correction Algorithm. WCE, WGC, ZCE, and ZGC represent weight values ​​respectively. relu represents the activation function, Convolutional layer represents the convolution layer, c3x3 represents the convolution of size 3x3, c5x5 represents the convolution of size 5x5, c7x7 represents the convolution of size 7x7, 1x1 represents the convolution of size 1x1, and sigmoid represents the activation function. Output represents the output enhanced underwater image. First, the input O 24 The features are processed by CLAHE algorithm and Gamma correction algorithm respectively:

[0084] O 25 =CLAHE(O 24 )

[0085] O 26 =Gamma(O 24 )

[0086] Step 4.7.1, feature O 25 After three convolution layers with kernel sizes of 3x3, 5x5, and 7x7:

[0087] O 27 =F 12 (O 25 )

[0088] Step 4.7.2, feature O 24 , O 25 , O 26 For channel connection, first pass through 3x3, 5x5, 7x7, 1x1 convolution layers, then pass through 3x3, 5x5, 7x7 convolution layers, perform feature extraction, and finally pass through the sigmoid activation function to normalize the feature weights to [0,1]:

[0089] O 28 =sigmoid(F 13 (F 12 (Concat(O 24 ,O 25 ,O 26 ))))

[0090] Step 4.7.3, feature O 26 After three convolution layers with kernel sizes of 3x3, 5x5, and 7x7:

[0091] O 29 =F 14 (O 26 )

[0092] Step 4.7.4, output feature O 27 , O 28 , O 29 Perform weight multiplication and then addition operations:

[0093] O 30 =O 27 xO 28 +O 28 xO 29

[0094] Step 4.7.5, output feature O 30 After a convolution layer with a convolution kernel size of 3x3 and a stride of 1, the enhanced underwater image is finally output:

[0095] O 31 =F 15 (O 30 )

[0096] Step 5 represents the training process, and the specific steps are as follows:

[0097] Step 5: Use the training set processed in step 2 to train the model; use the L1 loss function and the Perceptual loss function to constrain the network model obtained in step 4, and then backpropagate to update the parameters. After 50 trainings, a well-trained underwater image enhancement network model is finally obtained. The loss functions used are as follows:

[0098] L1 loss function, using L1 loss function is also called minimizing the absolute error (Least AbosuloteError). LAE is to minimize the true value y i and the predicted value f(x i ) is the sum of the absolute values ​​of the differences L1 between:

[0099]

[0100] Perceptual loss function: Perceptual loss is defined as the distance between the feature representation of the predicted RGB image and the real image, which is used to represent color and semantic information. Since the deep layers have a better understanding of semantic information and can fully preserve the image content and overall spatial structure, and the shallow layers are more sensitive to color and texture, we select the 4_3 layer from VGG-19net to make it more sensitive to both color and semantic information. represents the jth layer of the network, J represents the reconstruction result, Represents the ground truth value. It can be expressed mathematically as:

[0101]

[0102] Therefore, the total loss function of the present invention is:

[0103] L=λ1L1+λ2L per

[0104] L is the total loss function, λ1 and λ2 represent the set parameters, where λ1 is set to 1 and λ2 is set to 0.1.

[0105] Step 6: Put the test set processed in step 2 into the model trained in step 5, and finally output the enhanced underwater image results. The comparison results are shown in Figure 5 As shown in the figure. The first column is the real underwater image, the middle is the enhanced image obtained by the model, and the last column is the corresponding reference image. Water bodies of different colors and types were selected to test our method. (a) and (d) represent degraded underwater images, (b) and (e) represent the enhanced result images of the present invention, and (c) and (f) represent the reference images of underwater images.

Claims

1. An underwater image enhancement method integrating deep learning and traditional image enhancement technology, characterized by: The specific steps include: Step 1, input an underwater image dataset synthesized by cycleGAN, which includes pairs of degraded underwater images and clear underwater images, and divide the image dataset into a training set and a test set; Step 2, preprocessing the training set and test set obtained in step 1; Step 3, analyzing the mean difference between the paired degraded underwater images and the clear underwater images in the image data set of step 1; The specific process of step 3 is as follows: a mean difference map of three channels is drawn according to the mean difference between the paired degraded underwater images and the clear underwater images; the three channels are: red channel mean difference R, green channel mean difference G, and blue channel mean difference B; the mean difference of the red channel is above 0, indicating that the mean of the natural underwater image is greater than the mean of the degraded underwater image, so the red channel of the underwater image needs to be compensated; the mean difference of the green channel is below 0, indicating that the mean of the natural underwater image is less than the mean of the degraded underwater image, so the green channel of the underwater image needs to be attenuated; the mean difference of the blue channel fluctuates around 0, so the blue channel is enhanced; Step 4: Based on the analysis results of step 3, an enhanced network model of the underwater image is constructed; The specific process of step 4 is as follows: Step 4.1, combine the convolution operation, instance normalization, and activation function ReLU into a convolution layer: （1）； Step 4.2, using deep learning technology to build a color correction network model; Step 4.3, construct an attention-guided red channel color compensation module; Step 4.4, constructing a blue channel network model; Step 4.5, construct the green channel color attenuation module guided by attention; Step 4.6, performing channel stitching on the results obtained in steps 4.3 to 4.5, and outputting the corrected underwater image; Step 4.7, construct a contrast enhancement subnet composed of multi-scale convolutions, process the image obtained in step 4.6, and output the enhanced underwater image; Step 5, using the training set preprocessed in step 2 to train the network model constructed in step 4 to obtain the optimal enhanced network model; Step 6: put the test set preprocessed in step 2 into the enhanced network model trained in step 5, and output the enhanced underwater image.

2. The underwater image enhancement method integrating deep learning and traditional image enhancement technology according to claim 1, characterized in that: In step 2, the process of preprocessing the test set and the training set is: normalizing the pixel values ​​of the image to [0, 1].

3. The underwater image enhancement method integrating deep learning and traditional image enhancement technology according to claim 1, characterized in that: The specific process of step 5 is: using L1 loss function and Perceptual loss function to constrain the network model obtained in step 4, and then back-propagating to update parameters to obtain the optimal underwater image enhancement network model; The L1 loss function is shown in the following formula (2): （2）； The perceptual loss function is shown in the following formula (3): （3）； The total loss function is: （4）。

Images