Underwater image enhancement method based on ray-enhanced pixel fusion network

Abstract

The application relates to an underwater image enhancement method based on a light enhancement pixel fusion network. The method first proposes a light enhancement block (LEB) which can enhance image colors without reference to break through the color limitation of a traditional deep learning method; then, a pixel fusion subnetwork (PF SubNet) is proposed which can provide an adaptive feature weight function before feature fusion, and the recovery capability of the model to image fogging and high-frequency regions is improved. A large number of experiments prove the superiority of the method over existing methods.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to an underwater image enhancement method based on a light enhancement pixel fusion network. BACKGROUND

[0002] Deep learning is a data-driven method, which can avoid the selection of various prior parameters, and a convolutional neural network (CNN), which is one of the representatives of deep learning, has strong feature extraction capability. Therefore, many scholars have proposed various underwater image enhancement methods based on CNN. The core idea of these methods is to learn the mapping relationship between underwater images and clear images in pairs. However, there is a limitation in the traditional deep learning method, that is, the enhanced clear image can only be approximately referenced to the reference image. SUMMARY

[0003] The purpose of the present application is to break through the color limitations of traditional deep learning methods and provide an underwater image enhancement method based on a light enhancement pixel fusion network.

[0004] To achieve the above purpose, the technical scheme of the present application is as follows: an underwater image enhancement method based on a light enhancement pixel fusion network, first, a light enhancement block is proposed, which is responsible for learning and obtaining a feature mapping from an original image, and repairing the color deviation of an enhanced image; then, a pixel fusion subnetwork is proposed, which provides an adaptive feature weight function, and enhances the receptive field of the model through the fusion of feature mapping in different stages.

[0005] Compared with the prior art, the present application has the following beneficial effects: in order to break through the color limitations of traditional deep learning methods, the present application proposes a light enhancement block based on a residual block and a light enhancement curve, and the light enhancement curve contained therein is applied to underwater image processing tasks for the first time. In order to enhance the restoration ability of image fogging and high-frequency regions, a pixel fusion subnetwork based on a pixel attention mechanism and a feature fusion idea is also proposed, which can provide an adaptive feature weight function before feature fusion, so that the model focuses on image fogging and high-frequency region information. The performance of the method of the present application is better than that of most existing underwater image enhancement methods. In addition, the method of the present application not only exceeds the reference images provided by the UIEB[3] in the image enhancement field, but also exceeds the reference images provided by the UIEB[3] in multiple no-reference image quality evaluation algorithms, such as Entropy, Blind Image Quality Index (BIQI)[5], Naturalness Image Quality Evaluator (NIQE)[6], Average Gradient (AG) and JPEG compressed image NR quality measurement algorithm (JPEG)[7]. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1The network model architecture of the present application.

[0007] Figure 2 The model structure of the light enhancement block (LEB) is composed of a residual block and a light enhancement curve (LE-Curve) [1], the former is composed of two convolution blocks, an instance normalization layer and a ReLU activation function, which takes the output of the encoder as input to construct the initial feature map as output, and the latter is composed of two convolution blocks and a ReLU activation function, which takes the original image as input to learn the curve parameter mapping The output.

[0008] Figure 3 The curve parameter mapping Examples. Wherein (a) and (c) represent the input image and the clear image respectively, (b) represents the pixel-level curve parameter mapping The value of which determines the degree of exposure required by the pixel.

[0009] Figure 4 Loss, PSNR and SSIM curves with / without skip connection, where orange represents non-skip connection and blue represents skip connection.

[0010] Figure 5 Examples of pixel attention maps and channel attention maps, (a) represents an underwater image with a large number of dense blue pixels in the background, (b) represents its pixel attention map, and (c) represents the channel attention weight map of each pixel, the larger the value, the greater the attention.

[0011] Figure 6 Qualitative comparison results of the NYU-v2 RGB test set.

[0012] Figure 7 Qualitative comparison of the UIEB test set.

[0013] Figure 8 The results of Table 6 are normalized in the range of 0-5, and the larger the orange area, the better.

[0014] Figure 9 YOLO-V4

[14] detection results, the top row is the result of the original image, and the bottom row is the result of the present application.

[0015] Figure 10 SIFT feature matching

[15] results, the top row is the result of the original image, and the bottom row is the result of the present application.

[0016] Figure 11 Significance detection

[16] results, the top row is the result of the original image, and the bottom row is the result of the present application.

[0017] Figure 12 For Canny edge detection

[17] results, the top line is the result of the original image, and the bottom line is the result of the present application. DETAILED DESCRIPTION

[0018] The technical solutions of the present application will be specifically described below with reference to the drawings.

[0019] The underwater image enhancement method based on the light enhancement pixel fusion network provided by the present application first proposes a light enhancement block responsible for learning and obtaining a feature mapping from an original image and repairing color deviation of an enhanced image; then, a pixel fusion subnetwork is proposed to provide an adaptive feature weight function and improve the receptive field of the model through fusion of feature mappings at different stages.

[0020] The following is the specific implementation process of the present application.

[0021] 1. Method overview

[0022] The network architecture of the method of the present application is shown in Figure 1 . The processing flow is as follows: first, the encoder is used to extract features from the input image to construct an initial feature mapping; second, the light enhancement block (LEB) is iterated to learn the feature mapping and the curve parameter mapping required by the light enhancement curve (LE-Curve) [1] The curve parameter mapping will be input into the illumination smoothness loss function [2] to ensure that the enhanced image does not have a problem of too large difference in adjacent pixel values; then, the pixel fusion subnetwork (PF-SubNet) is applied to adaptively fuse feature mappings at different levels to obtain a combined feature mapping. Finally, the decoder is used to parse the combined feature mapping to obtain a clear image.

[0023] 2. Light enhancement block

[0024] As shown in Figure 2 , the LEB proposed in this paper is composed of a residual block (Residual Block) and a light enhancement curve (LE-Curve) [1]. The former is used to learn the mapping relationship between pairs of degraded and clear underwater images, and the latter is used to repair the color deviation existing in the clear image after enhancement. It is worth noting that the input of the residual block is the initial feature mapping constructed by the encoder, and the input of the light enhancement curve (LE-Curve) [1] is the original underwater image. The light enhancement curve (LE-Curve) [1] is a single-parameter and differentiable pixel-level high-order curve, which can be expressed by the following formula:

[0025]

[0026] where x represents the pixel coordinate, is the curve parameter mapping to be learned, n is the iteration number, which controls the curvature of the curve. When n is 1, formula 1 degenerates into a two-parameter form. As follows:

[0027] LE(I(x); a) = I(x) + aI(x)(1-I(x)) (2)

[0028] where I(x) represents the input underwater image, a e [-1, 1] is the curve parameter, which is used to control the curve size and exposure. In order to better understand the curve parameter mapping we give an example of curve parameter mapping Figure 3 in . Figure 3 (b) accurately shows the value change of different regions, and the clear image obtained after pixel-level curve mapping is shown in Figure 3 (c). Figure 3 From (c), it can be observed that the method improves the visual effect of the image to some extent. It is worth noting that the single-parameter form of the light enhancement curve (LE-Curve) [1] can reduce the computational cost and improve the calculation speed. The differentiability makes it possible to use a convolutional neural network to learn the adjustable parameters of the curve and ensure the smooth back propagation of the network. Pixel-level means that each pixel has a light enhancement curve, and high-order curve means that it can obtain better performance through multiple iterations, so the algorithm sets the iteration number to 7. Since the increase of the iteration number will increase the difficulty of model training, in order to optimize this situation, we introduce skip connection in the 7 light enhancement blocks (LEB). Figure 4 shows the difference between the loss function (Loss), peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) curves before and after the skip connection. Figure 4 From , it can be observed that the Loss curve converges faster after the skip connection, and the PSNR and SSIM curves rise faster and more.

[0029] 3、Pixel fusion subnetwork

[0030] Fusing feature maps of different levels can usually improve the receptive field of the model and thus improve the performance of deep learning models for different types of tasks. In order to improve the receptive field of the model and enable the model to perceive the fogging and high-frequency image regions of the underwater image, the present application proposes a pixel fusion subnetwork (PFSubNet) based on the pixel attention mechanism (PAM) [2]. The processing flow of this subnetwork can be divided into two stages.

[0031] The first stage is to use the pixel attention mechanism (PAM) [2] to adaptively assign weights to different levels of feature maps. For high-frequency and dense pixel regions in underwater images, the pixel attention mechanism (PAM) [2] will give more weight. Specifically, the feature maps F h ,F m from three different levels will be concatenated first, because in the underwater image enhancement task, the weight of each pixel point is equivalent, then the concatenated feature maps will be input into the global average pooling (GAP), which is to obtain the channel tensor, finally the channel tensor will be input into the pixel fusion subnetwork P to obtain the corresponding weight factors M h ,M m .

[0032] (M l ,M h ,M m )=P(GAP(F l ,F h ,F m )) (3)

[0033] The second stage is to linearly superimpose the feature maps F l ,F h ,F m and the weight factors M h ,M m obtained from formula 3 to obtain the combined feature map F o .

[0034] F o =M l *F l +M m *F m +M h *F h (8)

[0035] After obtaining the combined feature map F o , the decoder will parse it to obtain a clear image. It is worth noting that the pixel attention mechanism (PAM) [2] contains the channel attention mechanism (CAM) [2], which is responsible for assigning weights to different color channels in the image, and the weight size is determined by the pixel attention mechanism (PAM) [2]. Figure 5 Examples of pixel attention mechanism (PAM) and channel attention (CAM) weight maps are given in the following figures: Figure 5 (a) is an underwater image with a large number of dense blue pixel points, Figure 5 (b) is the corresponding pixel attention mechanism (PAM) map, Figure 5 (c) is the weight change of different color channels, the weight gradually increases from left to right. Since, Figure 5The background of (a) fully complies with the pixel attention mechanism (PAM) [2] which gives more weight to the areas of high frequency and dense pixels, so from Figure 5 As can be seen in (b), the weight values of the background area are significantly higher than those of other areas, which also coincides with the above-mentioned characteristics.

[0036] Embodiments.

[0037] 1. Experimental data and evaluation

[0038] The UIEB [3] and NYU-v2RGB-D [4] datasets are used to evaluate the performance of the algorithm of the present application. The UIEB [3] dataset is an underwater real image dataset, which includes 950 real-world underwater real images, of which 890 have corresponding reference images, and the remaining 60 do not. The NYU-v2RGB-D [4] dataset is an underwater synthetic image dataset, which contains 1449 real-world underwater synthetic images, which can be divided into two major categories according to different marine and coastal waters, and each major category contains 5 subcategories, i.e. different marine environments have I, IA, IB, II and III types, and different coastal waters have 1, 3, 5, 7 and 9 types. Among them, the turbidity of I type to III type gradually increases. Similarly, the turbidity of 1 type to 9 type gradually increases.

[0039] The present application adopts the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) as the full-reference evaluation indicators of the UIEB [3] and NYU-v2RGB-D [4] datasets. For the no-reference evaluation indicators, the present application adopts the Entropy, the blind image quality index (BIQI) [5], the naturalness image quality evaluator (NIQE) [6], the average gradient (AG) and the no-reference quality measurement algorithm of JPEG compressed images (JPEG) [7].

[0040] 2. Ablation study

[0041] Table 1. Quantitative evaluation results of the ablation experiment with / without LEBG and PF-SubNet

[0042]

[0043]

[0044] The core of the present application is the light enhancement block group (LEBG) and the PFSubNet. Therefore, we carried out the ablation experiment with / without LEBG and PFSubNet. The results are shown in Table 1, and both LEBG and PF-SubNet can effectively improve the performance, and the use of both can bring better performance improvement.

[0045] 3. Qualitative and quantitative comparison

[0046] Tables 2 and 3 present the quantitative evaluation results on the NYU-v2RGB-D[4] and UIEB[3] datasets, respectively. In the NYU-v2RGB-D[4] test set, the algorithm of this invention improved the PSNR index by 3.6% and the SSIM index by 2.5% compared with the UIEC^2-Net method. In the UIEB test set, the PSNR index was improved by 2%. Among them, HE, WB, UDCP[8] and ULAP[9] are traditional underwater image enhancement techniques, while UWGAN

[10] , UGAN

[11] , UWCNN

[12] , DUIENet[2] and UIEC^2-Net

[13] are representative underwater image enhancement techniques based on deep learning in the past 5 years. Figure 6 and Figure 7 Qualitative results for the NYU-v2RGB-D[4] and UIEB[3] datasets are presented respectively.

[0047] Table 2. Full-reference quantitative evaluation results of the NYU-v2 RGB-D dataset.

[0048]

[0049] Table 3. Full-reference quantitative evaluation results of the UIEB dataset

[0050]

[0051] To demonstrate that the image quality obtained by the algorithm of this invention is better than that of the reference image, the present invention presents the results in Table 4. Figure 7 The algorithm of this invention achieves the best scores in BIQI[5], NIQE[6], AG and JPEG[7] metrics compared to other methods, and even outperforms the reference image. To better understand the differences between the methods in Table 4, we normalize the results in Table 4 to the range of 0 to 5, and then display them using a radar chart. The results are as follows. Figure 8 As shown, the larger the orange area, the better.

[0052] Table 4 Figure 7 No reference assessment results

[0053]

[0054] 4. Application Testing

[0055] To test the practicality of the present invention, we used YOLO-V4

[14] , SIFT feature matching

[15] , saliency detection technology

[16] and Canny edge detection operator

[17] to test the original underwater image and the enhanced image of the present invention. The results are as follows: Figure 9 , Figure 10, Figure 11 and Figure 12 As shown in FIGS. 6, 7, 8, 9, and 10, for the original underwater images, the results of the algorithm of the present application can detect more and more accurate people and fish in YOLO-V4

[14] , match more and more accurate feature points in SIFT feature matching

[15] , detect more saliency information in saliency detection

[16] , and extract more edge features in Canny edge extraction

[17] .

[0056] References:

[0057] [1] Li C, Guo C, Chen C L. Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation [J]. IEEE Transactions on Software Engineering, 2021, PP(99).

[0058] [2] Qin X, Wang Z, Bai Y, et al. FFA-Net: Feature Fusion Attention Network for Single Image Dehazing [C] / / 2019.

[0059] [3] Li C, Guo C, Ren W, et al. An Underwater Image Enhancement Benchmark Dataset and Beyond [J]. IEEE Transactions on Image Processing, 2019.

[0060] [4] Anwar S, Li C, Porikli F. Deep Underwater Image Enhancement [J]. 2018.

[0061] [5] Moorthy A K, Bovik A C. A Two-Step Framework for Constructing Blind Image Quality Indices [J]. IEEE Signal Processing Letters, 2010, 17(5): 513-516.

[0062] [6] Mittal A, Fellow, IEEE, et al. Making a 'Completely Blind' Image Quality Analyzer [J]. IEEE Signal Processing Letters, 2013, 20(3): 209-212.

[0063] [7] Zhou W, Sheikh H R, Bovik A C. No-reference perceptual quality assessment of JPEG compressed images [C] / / Proceedings. International Conference on Image Processing.

[0064] [8] Drews P, Nascimento E R, Botelho S, et al. Underwater Depth Estimation and Image Restoration Based on Single Images [J]. IEEE Computer Graphics & Applications, 2016, 36(2): 24-35.

[0065] [9] Song W, Wang Y, Huang D, et al. A Rapid Scene Depth Estimation Model Based on Underwater Light Attenuation Prior for Underwater Image Restoration [C] / / 2018.

[0066]

[10] Wang N, Zhou Y, Han F, et al. UWGAN: Underwater GAN for Real-world Underwater Color Restoration and Dehazing [J]. 2019.

[0067]

[11] Fabbri C, Jahidul Islam M, Sattar J. Enhancing Underwater Imagery using Generative Adversarial Networks [C] / / 2018 IEEE International Conference on Robotics and Automation(ICRA). arXiv, 2018.

[0068]

[12] Anwar S, Li C, Porikli F. Deep Underwater Image Enhancement [J]. 2018.

[0069]

[13] Wang Y, Guo J, Gao H, et al. UIEC^2-Net: CNN-based Underwater Image Enhancement Using Two Color Space [J]. 2021.

[0070]

[14] Wang C Y, Bochkovskiy A, Liao H. Scaled-YOLOv4: Scaling Cross Stage Partial Network [C] / / Computer Vision and Pattern Recognition. IEEE, 2021.

[0071]

[15] Low D G. Distinctive Image Features from Scale-Invariant Keypoints [J]. International Journal of Computer Vision, 2004.

[0072]

[16] Liu J J, Hou Q, Liu Z A, et al. Poolnet+: Exploring the potential of pooling for salient object detection [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.

[0073]

[17] Canny J. A Computational Approach to Edge Detection [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1986, PAMI-8(6): 679-698.

[0074] The above is the preferred embodiment of the present application, any changes made in accordance with the technical solutions of the present application, the resulting functional effects do not exceed the scope of the technical solutions of the present application, all belong to the protection scope of the present application.

Claims

1. An underwater image enhancement method based on a light-enhancing pixel fusion network, characterized in that, A light enhancement block is constructed to learn and obtain feature maps from the original image and repair color cast in the enhanced image; a pixel fusion subnetwork is constructed to provide adaptive feature weighting and improve the receptive field of the model by fusing feature maps at different stages. The input underwater image needs to be extracted using an encoder to construct an initial feature map, and then the light enhancement block is input; the combined feature map output by the pixel fusion subnetwork needs to be parsed by a decoder to obtain a clear image; The light enhancement block consists of a residual block and a light enhancement curve. The input to the residual block is the initial feature map constructed by the encoder, and the input to the light enhancement curve is the underwater image. The light enhancement curve is a single-parameter and differentiable pixel-level high-order curve, which, when its order is greater than or equal to 2, is expressed by the following formula: Where the subscript n represents the order, x represents the pixel coordinate, and LE n (x) represents the light enhancement curve of the current order, LE n-1 (x) represents the previous order of light enhancement curve. It is the curve parameter mapping that needs to be learned for the current order. It controls the curvature of the curve. When the order n is 1, formula (1) will degenerate into a two-parameter form, as shown below: LE(I(x);α)=I(x)+αI(x)(1-I(x)) (2) Where I(x) represents the input underwater image, and α∈[-1,1] is the curve parameter, which is used to control the curve size and exposure level; The pixel fusion sub-network processing flow is divided into two stages: The first stage uses a pixel attention mechanism to adaptively assign weights to feature maps at different levels. For high-frequency and dense pixel regions in underwater images, the pixel attention mechanism assigns more weights. Specifically, it assigns weights to feature maps F from three different levels. l ,F h ,F m First, the features are concatenated. Then, the concatenated feature maps are fed into the global average pooling (GAP) to obtain channel tensors. Finally, the channel tensors are fed into the pixel fusion subnetwork P to obtain the corresponding weight factors M. l M h M m : (M l ,M h ,M m )=P(GAP(F l ,F h ,F m )) (3) The second stage involves feature maps F at three different levels. l ,F h ,F m The weighting factor M obtained from formula (3) l M h M m Perform linear superposition to obtain the combined feature map F o : F o =M l *F l +M m *F m +M h *F h (8) Obtain the combined feature map F o Then, the decoder will parse it to obtain a clear image.

2. The underwater image enhancement method based on a light-enhancing pixel fusion network according to claim 1, characterized in that, The pixel attention mechanism includes the channel attention mechanism, which is responsible for assigning weights to different color channels in the image. The weights are determined by the pixel attention mechanism.

Images