An underwater image enhancement method guided by a convolutional network-guided model mapping

Abstract

The present invention discloses an underwater image enhancement method guided by a convolutional network-guided model mapping. The underwater image enhancement method guided by a convolutional network-guided model mapping includes the following steps: Step 1: Build an underwater image enhancement network structure; Step 2: Use a deep dense residual module to extract and fuse features at different levels; Step 3: Use an attention mechanism to focus on representative feature channels. The advantages compared with the prior art are as follows: The present invention proposes an underwater image enhancement method of a convolutional network-guided model mapping network. With the help of improving the underwater imaging model, it fully combines the underwater scene understanding ability of the imaging model and the data-driven ability of the convolutional network. Visual perception comparison and image quality evaluation in the experimental results show that this network has flexible adaptive adjustment ability and can accurately perform color correction. In addition, the introduction of the attention mechanism can highlight the detailed features of the main body and further improve the image enhancement quality.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image enhancement processing research, and specifically refers to an underwater image enhancement method guided by a convolutional network to map a model. Background Art

[0002] The ocean contains countless precious natural resources. In the process of human exploration and development of the ocean, it is challenging to complete underwater tasks. The acquisition and perception of underwater scenes are crucial for completing marine biological recognition, underwater scene reconstruction, underwater archaeology, underwater environmental monitoring, etc. Usually, to obtain better underwater scene information, it often depends on a camera sensor to capture clear and complete underwater images or videos. However, due to the attenuation and scattering effects of light signals in water, the images obtained by the camera usually have problems such as color distortion, low contrast, and blurred details. Therefore, it is necessary to convert underwater images into a form that is more suitable for humans or machines to analyze and process.

[0003] Underwater image enhancement aims to improve the image quality without losing any information and is an image processing technology with high efficiency and low cost. By adjusting the pixels of the image to improve its visual effect, common methods include histogram equalization, color correction, and image fusion. These image enhancement methods can improve the visual quality to a certain extent, but due to the lack of underwater scene knowledge, they may aggravate noise, introduce artifacts, and cause color distortion. Deep learning technology, with its excellent feature expression ability and data-driven ability, has demonstrated impressive performance in the field of underwater image enhancement. For example, convolutional neural networks (CNNs) and generative adversarial networks (GANs) are typical methods. These data-driven methods achieve visual reconstruction by learning features such as color, structure, and texture. However, in most cases, they are only applicable to specific underwater scenes. The main reason is that deep networks ignore the knowledge in the field of underwater imaging. When faced with unfamiliar waters, the prior knowledge and fixed features learned from the training dataset cannot be well generalized to the constantly changing actual scenes, which limits the practical value of deep learning methods.

[0004] Model-based methods and learning-based methods are two different paradigms for processing underwater images. The imaging model method restores images based on prior knowledge of underwater scenes, while the deep learning method enhances images based on pre-trained prior parameters. By integrating these two modules, there is a possibility of benefiting from the complementarity of the two paradigms. Early studies explored the combination of imaging models and convolutional networks from the perspectives of underwater scene knowledge injection and using network optimization parameters. Some research works focused on injecting underwater scene knowledge for synthesizing datasets or enhancing network performance. For example, Chen et al. proposed a hybrid underwater image synthesis network that combines physical priors and data-driven approaches to synthesize a high-quality underwater image training dataset. Li et al. proposed an underwater image enhancement network (Ucolor) with multi-color space embedding guided by medium transmission, where the transmission map estimated by the model is used as the attention feature of the CNN to further improve the visual quality of underwater images. Another approach is to focus on using the network to optimize parameters. For example, Liu et al. proposed a unified paradigm for transmission-based image enhancement tasks, using a data-driven CNN as a prior set to automatically identify and set the attributes and data distributions of different scenes. Li et al. proposed an underwater image super-resolution enhancement method (AOA-U-Net), where the CNN network and the imaging model are used in series, and the data-driven ability of the CNN network is used to solve some complex parameters. In fact, the model-based method and the learning-based method are complementary, and their potential relationship has not been fully exploited.

[0005] To quantitatively and qualitatively study the performance of underwater image enhancement techniques in different scenarios, the present invention proposes an underwater image enhancement method with model mapping guided by a convolutional network. The model parameters are guided by a neural network, and this method can restore the true image color in different underwater scenes, improving the color restoration performance and generalization performance. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the above technical difficulties and provide an underwater image enhancement method with model mapping guided by a convolutional network (IMM-Net).

[0007] To solve the above technical problem, the technical solution provided by the present invention is as follows:

[0008] An underwater image enhancement method with model mapping guided by a convolutional network, the underwater image enhancement method with model mapping guided by a convolutional network includes the following steps;

[0009] Step 1: Build the underwater image enhancement network structure

[0010] The working process of this network can be divided into three steps: feature extraction, mapping, and recombination, by end-to-end combining the feature extraction network and the imaging model.

[0011] Specifically: First, a 3-layer deep dense residual module and a feature transformation module are used as the encoding network (Encoder) to represent features at different levels. Then, the generated feature maps are fed into the K c (x) prediction module, which consists of a channel attention and a deep dense residual module, emphasizes representative feature channels, and uses the Softmax function in the last layer to output a feature transfer map K c (x) with global context understanding. Additionally, the feature maps generated by the Encoder are concatenated with the original image and fed into the L c (x) prediction. Finally, the mapped feature maps of the image to the model are fed into the improved underwater imaging model to reconstruct and generate the enhanced image J(x).

[0012] Step 2: Extract and fuse features at different levels using the deep dense residual module

[0013] Specifically: The deep dense residual module (DRDB) is as shown in Figure 2 . Each DRDB consists of 4 layers of residual unit blocks, and the output feature layer of each layer is the input of all subsequent layers, aiming to alleviate the vanishing gradient problem and enhance the forward propagation of features, and reduce the overfitting phenomenon. Fusing features at different levels can retain the spatial information of the underlying features while using the semantic information of the high-level features for image understanding.

[0014] Step 3: Use the attention mechanism to focus on representative feature channels

[0015] Specifically: The details of the channel attention mechanism (CAM) are as shown in Figure 3 . The receptive field of the convolutional kernel is limited, and capturing global image information requires accumulating more network layers, resulting in reduced learning efficiency. The self-attention mechanism calculates the self-correlation in the features and uses the method of non-equivalent weight assignment to emphasize representative feature channels.

[0016] This module consists of two parts: global context modeling and channel feature transformation. The context modeling module generates a weight vector through a 1×1 convolutional kernel and the Softmax function, then performs pixel multiplication with another weight matrix to obtain the global context features. After feature transformation of the global context features, they are added to the input pixel by pixel, and the output dimension is the same as the input dimension. The channel attention formula is:

[0017]

[0018] where N p is the total number of pixels N p= H·W, where y and z represent the input and output of channel attention, i represents the number of channel indices, j represents the number of all possible pixel positions, and α j represents the weight at position j, and δ represents feature transformation.

[0019] Step 4: Feed the feature maps K c (x) and L c (x) into the improved underwater imaging model to generate the enhanced image J(x);

[0020] Specifically: The mapping of the imaging model has the expression of underwater prior knowledge, which has better generalization ability than simply using a deep network and can be effectively trained to adapt to visual adjustments under different underwater scene conditions;

[0021] The expression of the improved underwater imaging model is:

[0022] J c (x) = K c (x)I c (x) - K c (x) + L c (x)

[0023]

[0024]

[0025] In this formula, x represents the pixel point coordinates, c represents the color channel, I c (x) represents the picture taken by the camera in the underwater environment, J c (x) represents the radiated light of the object itself, that is, the clear image expected to be obtained, t(x) represents the direct transmission and backscattering attenuation coefficient, B c (x) represents the background light, K c (x) is a model dependent on the input image, L c (x) is an adaptive color compensation to eliminate the color deviation of the underwater background.

[0026] Compared with other underwater image enhancement algorithms, the present invention has the following advantages:

[0027] In the present invention, an underwater image enhancement method of a model mapping network guided by a convolutional network is proposed. With the help of the improved underwater imaging model, the underwater scene understanding ability of the imaging model and the data-driven ability of the convolutional network are fully combined. The experimental results show through visual perception comparison and image quality evaluation that the network has flexible adaptive adjustment ability and can accurately perform color correction. In addition, the introduction of the attention mechanism can highlight the detailed features of the main body and further improve the image enhancement quality. Brief Description of the Drawings

[0028] To clearly illustrate the technical solution of the present invention and at the same time show the results of comparison with the processing effects of other existing algorithms, the following briefly introduces the drawings required in the description of the invention.

[0029] Figure 1 It is a network structure diagram of an underwater image enhancement algorithm described in the present invention.

[0030] Figure 2 It is a network structure diagram of a deep dense residual network for feature extraction described in the present invention.

[0031] Figure 3 It is a network structure diagram of a channel attention network for feature extraction described in the present invention.

[0032] Figure 4 It is the experimental results of the algorithm of the present invention and other image enhancement algorithms on the UIEB dataset.

[0033] Figure 5 It is the experimental results of the algorithm of the present invention and other image enhancement algorithms on the Color-Checker7 dataset. Detailed Description of the Preferred Embodiments

[0034] The following details the embodiments of the present invention with reference to the accompanying drawings: These embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation schemes and specific operation processes are given. Specific Embodiment 1

[0036] The network training of an underwater image enhancement method guided by a convolutional network model includes the following steps:

[0037] Step 1: Setting of network training data

[0038] Randomly select 800 pairs of underwater images from the UIEB underwater image enhancement dataset, including real underwater images and corresponding reference images. We resize the input images to 128x128 by downsampling, and set relevant hyperparameters to meet the memory limitations required for subsequent feature extraction. Use ADAM to train our model, set the learning rate to 0.0001, the momentum to 0.9, the batch size to 1, and the epoch to 350. Use Pytorch as the deep learning framework, and use an Inter(R) i9-10900x CPU, 32GB RAM, and an Nvidia GTX3090 GPU.

[0039] Step 2: Verification of training results

[0040] For testing, we extracted 90 pairs of real images from UIEB as the test set, denoted as Test-R90, and used evaluation metrics for quantitative analysis. For the Test-R90 dataset, the mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS) were selected as full-reference evaluation metrics. A lower MSE and a higher PSNR indicate that the result is closer to the reference image in terms of image content. The higher the SSIM score, the more similar the result is to the reference image in terms of image structure and texture. LPIPS is the same as the perceptual loss and judges the perceptual similarity by measuring the high-level semantic features between images, which is more in line with human perception when judging image similarity.

[0041] As can be seen from Table 1, our IMM-Net outperforms other competing methods in the reference evaluation metrics. On the Test-R90 dataset, the PSNR and SSIM are increased by 11.9% and 2.2% respectively, and the MSE and LPIPS metrics are decreased by 42.3% and 0.09% respectively (compared with Ucolor). Through quantitative comparison, it is found that using deep training to learn the transmission mapping of images can better understand underwater scene knowledge and finally obtain enhanced images that are more in line with visual perception, as Figure 4 shown.

[0042] Table 1 uses PSNR (dB), MSE (×10 3 ), SSIM, and LPIPS as reference evaluation metrics, and uses UIQM and UCIQE as non-reference evaluation metrics.

[0043]

[0044] Step 3: Analysis of color restoration accuracy

[0045] The images of the Color-Checker7 dataset were sequentially input into the network (the people in the dataset were photographed holding a standard color card in an underwater environment about 1 m away from the camera), and enhanced underwater images were obtained through processing, as Figure 5 shown.

[0046] The UIQM metric was used to quantitatively evaluate the color, sharpness, and contrast of underwater images, and the specific evaluation details are shown in Table 2. All visual comparisons show that IMM-Net is not only more in line with human eye perception visually, but also exceeds or approaches the highest level in terms of metrics (the average UIQM is up to 3.67), indicating that the method in this paper has good color calibration ability. In addition, IMM-Net can well understand underwater scenes taken by different cameras, indicating that the method has flexible scene adaptation ability.

[0047] Table 2 Comparison using the UIQM evaluation metric on the Color-Checker7 dataset

[21] , with the best results marked in bold font.

[0048]

[0049] In this study, an underwater image enhancement method guided by a convolutional network mapping model is proposed. With the help of an improved underwater imaging model, the underwater scene understanding ability of the imaging model and the data-driven ability of the convolutional network can be fully combined. When dealing with waters with different degradation degrees, the network has flexible adaptive adjustment ability and can accurately perform color correction. Thus, the image is more in line with the form for human or machine analysis and processing, facilitating the acquisition of good underwater scene information.

[0050] The embodiments described above are only descriptions of the preferred example modes of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. An underwater image enhancement method guided by a convolutional network-guided model mapping, characterized in that: The underwater image enhancement method guided by the convolutional network-guided model mapping includes the following steps; S1. Obtain an underwater image, detect the pixel values of the RGB channels of the image, and perform pixel normalization to obtain the image I(x); S2. Use a convolutional neural network to extract features from the input RGB image I(x), and predict the feature transfer prediction maps K c (x) and L c (x); S3. Feature maps K c (x) and L c (x) are fed into the improved underwater imaging model to generate the enhanced image J(x); S4. During training guidance, we use a combination of multiple loss functions to guide the training from aspects of color content, perceptual information, and color smoothness; The improved underwater imaging model includes; The mapping of the imaging model has a simple expression of underwater prior knowledge, has better generalization ability than simply using a deep network, and can be effectively trained to adapt to visual adjustments under different underwater scene conditions; The expression of the improved underwater imaging model is J c (x) = K c (x)I c (x) - K c (x) + L c (x) In this formula, x represents the pixel point coordinates, c represents the color channel, and I c (x) represents the image captured by the camera in the underwater environment, and J c (x) represents the radiated light of the object itself, that is, the clear image expected to be obtained. t(x) represents the direct transmission and backscattering attenuation coefficient, and B c (x) represents the background light, and K c (x) is a model dependent on the input image, and L c (x) is the adaptive color compensation to eliminate the color deviation of the underwater background; The loss function includes In the lab space, adjusting the a and b spaces can balance the color without changing the brightness. The color loss quantifies the colors of the a and b spaces respectively through cosine similarity, and then sums and takes the average to obtain the color loss value. This loss function encourages the generated image to match the color in the reference image. The expression of the color loss function is where i represents a pixel, N represents the total number of pixels involved, at this time N = H×W, ∠(,) represents the cosine similarity value of two pixel points, and the subscripts a and b represent the pixel values in the a space and b space respectively.

2. The underwater image enhancement method guided by a convolutional network-guided model mapping according to claim 1, characterized in that: The feature extraction uses the feature fusion of different levels of the deep dense residual network to retain the spatial information of the underlying features while using the semantic information of the high-level features for image understanding.

3. The underwater image enhancement method guided by a convolutional network-guided model mapping according to claim 1, characterized in that: The feature extraction uses a channel attention mechanism to emphasize the representative feature channels and capture the global information of the image.

Images