An underwater image enhancement method based on deep learning

Abstract

The present invention discloses an underwater image enhancement method based on deep learning. The method first obtains underwater image data and performs preprocessing, adjusts all images to a uniform size and then enhances them to obtain an input image. Secondly, the input image is simultaneously input into a primary feature extraction network and a secondary feature extraction network, and the outputs of the two networks are added to obtain a background scattering feature map. Then, the input image and the background scattering feature map are added as the input feature map, and a direct transmission feature map is extracted based on dilated convolution. Finally, based on the input image, the background scattering feature map and the direct transmission feature map, synthesis is performed through a physical model synthesis module to obtain an enhanced image. The present invention uses a small number of parameters, which reduces the computational burden, can improve the enhancement effect when facing complex and low-quality underwater images, and accurately complete underwater image enhancement.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an underwater image enhancement method based on deep learning. Background Art

[0002] Underwater image enhancement technology plays a crucial role in the complex process of ocean exploration. In marine environmental monitoring, underwater image enhancement technology helps us monitor changes in the ocean environment in real time and promptly detect pollution and ecological issues. In underwater facility inspection, it helps accurately locate damage and ensures safe operation. And in seabed biological research, high-quality underwater imagery reveals the diversity and mysteries of marine life. Whether it's cutting-edge scientific underwater exploration, a single, or a series of, clear, accurate seafloor images can provide intuitive information unattainable by other means, providing strong support for decision-making.

[0003] However, the real underwater environment is complex and ever-changing. Water scatters and absorbs light. The absorption of light by water molecules weakens the light's energy, and the degree of light attenuation varies depending on the color of the light, causing color casts and low brightness in underwater images. Furthermore, scattering can lead to blurred image details and low contrast. Consequently, underwater image degradation is extremely severe. Typical low-quality underwater images often exhibit bluish or greenish hues, or insufficient illumination, along with blurred details and low contrast. These issues urgently need to be addressed to improve the quality of underwater images and better serve various ocean exploration and development activities.

[0004] In order to solve these problems, underwater image enhancement technology came into being. In particular, the enhancement algorithm combined with deep learning has developed rapidly and has achieved good results. Therefore, the research on underwater image enhancement technology not only has long-term strategic significance for national development, but also has important practical significance for facilitating people's lives.

[0005] Algorithms for underwater image processing can be divided into two categories, image enhancement and image restoration, based on whether they rely on underwater imaging models. Image enhancement algorithms primarily enhance the pixels of acquired underwater images to improve their visual quality, without considering the specific image formation and degradation processes. Image restoration algorithms, on the other hand, strive to restore the image's true state based on the underlying principles of underwater imaging. To achieve this, image restoration algorithms rely on a series of parameters, such as the optical properties of the water, camera settings, and the distance between the camera and the target object. These key parameters typically require precise estimation through manual measurement or other methods. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides an underwater image enhancement method based on deep learning.

[0007] The method of the present invention comprises the following steps:

[0008] Step (1). Obtain underwater image data and perform preprocessing, adjust all images to a uniform size and then enhance them to obtain input images.

[0009] Step (2). Extraction of background scattering feature map: Input the input image into a primary feature extraction network and a secondary feature extraction network at the same time. Finally, the outputs of the two networks are added together to obtain the background scattering feature map, which is used as the input of the next module. The specific method is as follows:

[0010] The primary feature extraction network: the input image first passes through three groups of 3×3 convolution layers with 8 channels, and the output feature map passes through global average pooling and 1×1 convolution kernel in turn, and the dimension is reduced to output a 3-channel feature map.

[0011] The secondary feature extraction network: the input image is added to the primary feature extraction network as the input feature map of the secondary feature extraction network, first passing through six groups of 3×3 size and 16 channels of convolutional layers to extract the features of the image, and the output of the secondary feature extraction network is jump-connected to the fourth group of convolutional layers; finally, dimensionality reduction is performed in sequence through global average pooling and 1×1 convolution kernel, and the feature map after dimensionality reduction is output.

[0012] The secondary feature extraction network uses the same activation function, global average pooling layer and 1×1 convolution kernel as the primary network, but its network structure is deeper. The features extracted by the primary network are used as auxiliary information and integrated into the input of the secondary network to achieve feature fusion.

[0013] Step (3). Extraction of direct transmission feature map: The input image and the background scattering feature map are added as the input feature map. Based on the dilated convolution, the direct transmission feature map is extracted. Similar to the previous module, this module also adopts a multi-subnet splicing architecture. The specific method is as follows:

[0014] Step 3.1: Add the input image and the background scattering feature map as input information F.

[0015] Step 3.2: Input F into four groups of 3×3 convolutional layers. The first three groups are 8-channel dilated convolutions, and the convolution kernels use dilation rates of 1, 2, and 5 respectively; the fourth group of convolutional layers is 3×3 convolution, which performs dimensionality reduction and obtains a 3-channel feature map G1.

[0016] Step 3.3: Add the input information F to the feature map G1, and input seven groups of 3×3, 8-channel convolutional layers. The first six groups are dilated convolutions, and the feature map G1 is connected to the fourth group of convolutional layers through jump connections; the feature map G2 is obtained by dimensionality reduction through the seventh group of convolutional layers.

[0017] Step 3.4: Add the feature map G1 and the feature map G2 to obtain the direct transmission feature.

[0018] Step (4). Based on the input image, background scattering feature map and direct transmission feature map, the physical model synthesis module is used to synthesize and obtain an enhanced image. The specific method is as follows:

[0019] This module aims to integrate the results of the first two functional blocks and generate an enhanced image D based on the input underwater image. C (x):

[0020]

[0021] The ambient light B C and direct transfer parameters are background scattering feature map and direct transmission feature map, I C (x) is the input image. First, the B estimated by the background scattering module is deducted from the original underwater image information. C Then, multiply this result with the information estimated by the direct transmission module; Finally, the above product result is combined with the output information B of the background scattering module C Add together to get the final enhanced image.

[0022] The beneficial effects of the present invention are as follows:

[0023] The main advantages of the method presented in this paper lie in three aspects: It introduces a physical imaging model into the deep learning network and aligns the abstract features extracted by the network with actual physical meaning according to physical formulas. This imbues the design of the network model with practical physical meaning, providing solid theoretical support for subsequent network training and optimization, and improving the network's enhancement effect when dealing with complex, low-quality underwater images. The present invention designs corresponding network modules for extracting two types of underwater imaging information. After fully considering the characteristics of the corresponding features, the module networks are further refined and layered, enriching the layers of feature extraction as much as possible. The model used in this paper requires a relatively small number of parameters, thus potentially allowing for further lightweighting of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A framework diagram of the method of the present invention;

[0025] Figure 2 This is a result diagram of the method of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, the method of the present invention is specifically as follows

[0028] In response to the shortcomings of the existing technology, the present invention provides an underwater image enhancement method based on deep learning.

[0029] The method of the present invention comprises the following steps:

[0030] Step (1). Obtain underwater images and preprocess them, unify all images to 224×224 size and then perform enhancement processing.

[0031] Step (2). Extraction of background scattering feature map: Input the input image into a primary feature extraction network and a secondary feature extraction network at the same time, and finally add the outputs of the two networks as the input of the next module. The specific method is as follows:

[0032] The input image first passes through three sets of 3×3 convolutional kernels with 8 channels, outputting an H×W×8 feature map. This is followed by a global average pooling layer, which produces eight 1×1 feature maps. Finally, a 1×1 convolutional kernel is used for dimensionality reduction to 3 channels. This network uses the PReLU activation function. The secondary feature extraction network uses the same activation function, global average pooling layer, and 1×1 convolutional kernel as the primary network, but its network structure is deeper, consisting of six sets of 3×3 convolutional kernels with 16 channels, to extract high-dimensional features from the image. The features extracted by the primary network are used as auxiliary information and incorporated into the input of the secondary network, resulting in an H×W×6 input and an H×W×16 output feature map. Finally, the outputs of the two networks are summed and used as the input to the next module. Furthermore, to balance network performance with computational time and improve learning efficiency, this module incorporates skip connections from the residual mechanism, skip-connecting the output of the secondary feature extraction network to the fourth layer. Since the secondary network has reached 6 layers, deeper networks are more susceptible to the vanishing gradient problem, so the residual connection mechanism is introduced, which allows information to be directly transferred "jump-like" between different layers of the network. In this way, the network no longer needs to directly learn the complex mapping relationship between input and target output, but instead learns the residual between input and output. The application of residual learning makes the transmission of gradient information smoother and more effective when training deep neural networks. The network can now complete learning by simply adding input information directly to the output, thus avoiding the difficulty of learning complex nonlinear transformations. Deeper neural networks can be constructed that are not easily affected by the vanishing gradient problem. This design significantly improves the training efficiency and performance of the network.

[0033] Step (3). Direct transmission feature map extraction. The input of the direct transmission feature extraction network combines the input image and the output of the background scattering feature extraction module, so the network input feature map size is H×W×6. Similar to the previous functional block, this network also adopts a multi-subnet splicing architecture. The specific method is as follows:

[0034] The first layer of the network consists of four sets of 3×3 convolution kernels, the first three of which are 8-channel dilated convolutions. Compared to the original normal convolution operation, dilated convolution has an additional parameter: the dilation rate, which refers to the number of intervals between the convolution kernel points. Dilated convolution expands the receptive field without reducing the size of the feature map. The three sets of dilated convolution kernels in the network use dilation rates of 1, 2, and 5, respectively. After three dilated convolutions, the output is an H×W×8 feature map. The fourth set uses a standard 3×3 convolution kernel, which performs dimensionality reduction and converts the feature map into a 3-channel image. Due to its simple structure, this network can quickly extract direct features from the input data, laying the foundation for subsequent work.

[0035] Similar to the previous background scattering feature extraction network, the above uses PReLU as the activation function. The input of the secondary feature extraction network combines the input of the first layer network and the intermediate feature map information, so the number of input channels is 9. The second layer network consists of seven groups of 3×3 convolution kernels with 8 channels. The first six groups are dilated convolutions, and the seventh group is a standard 3×3 convolution kernel with 3 channels, also using the PReLU activation function.

[0036] Given that the network size of the secondary feature extraction network has reached seven layers, a residual mechanism is also introduced to connect its input to the fourth set of dilated convolutions through jumps to improve the training efficiency and performance of the network.

[0037] Step (4). Physical model synthesis module, the specific method is as follows:

[0038] This module aims to integrate the results of the first two functional blocks and generate an enhanced image D based on the underwater image improvement model. C (x):

[0039]

[0040] The ambient light B C and direct transfer parameters It is unknown information and needs to be obtained from the first two feature extraction modules, namely the background scattering feature map and the direct transmission feature map. C (x) is the original image captured by the camera, D C (x) is an ideal underwater image. First, the B estimated by the background scattering module is deducted from the original underwater image information.C Then, multiply this result with the information estimated by the direct transmission module; Finally, the above product result is combined with the output information B of the background scattering module C Add together to get the final enhanced image.

[0041] Step (5). Experiment, train and test the network. The specific method is as follows:

[0042] All training set images need to be unified to 224×224 before processing. The experiment uses the UIEB dataset, which has a total of 890 real underwater images. The experiment uses pytorch as the network model training framework and uses the Adam optimizer to train the network. The test results are as follows Figure 2 As shown in the result image, we can intuitively see the enhancement effect of the algorithm. For degraded images in various situations, there is an enhancement effect, and color correction and contrast enhancement are obvious.

Claims

1. A method for underwater image enhancement based on deep learning, characterized in that: The following steps are involved: Step 1. Obtain underwater image data and preprocess it, resize all images to a uniform size, and then enhance them to obtain the input image; Step 2. Extraction of background scattering feature map: The input image is fed into a primary feature extraction network and a secondary feature extraction network at the same time, and the outputs of the two networks are added together to obtain the background scattering feature map; Step 3. Extraction of direct transmission feature map: The input image and the background scattering feature map are added as the input feature map, and the direct transmission feature map is extracted based on the dilated convolution. The specific implementation process is as follows: Step 3.1, add the input image and the background scattering feature map as input information F; Step 3.2: Input F into four groups of 3×3 convolutional layers. The first three groups are 8-channel dilated convolutions, and the convolution kernels use dilation rates of 1, 2, and 5 respectively. The fourth group of convolutional layers is 3×3 convolution, which performs dimensionality reduction and obtains a 3-channel feature map G1. Step 3.3: Add the input information F to the feature map G1 and input seven groups of 3×3, 8-channel convolutional layers. The first six groups are dilated convolutions, and the feature map G1 is connected to the fourth group of convolutional layers through a jump connection. After the seventh convolutional layer, the feature map G2 is obtained; Step 3.4, add the feature map G1 and the feature map G2 to obtain the direct transmission feature; Step 4. Based on the input image, background scattering feature map and direct transmission feature map, the physical model synthesis module is used to synthesize and obtain an enhanced image; The specific implementation process of the physical model synthesis module is as follows: Generate an enhanced image D based on the input underwater picture C (x): The ambient light B C and direct transfer parameters are background scattering feature map and direct transmission feature map, I C (x) is the input image.

2. The underwater image enhancement method based on deep learning according to claim 1, characterized in that: The primary feature extraction network: the input image first passes through three groups of 3×3 convolution layers with 8 channels, and the output feature map passes through global average pooling and 1×1 convolution kernel in turn, and the dimension is reduced to output a 3-channel feature map; The secondary feature extraction network: the input image is added to the primary feature extraction network as the input feature map of the secondary feature extraction network, first passing through six groups of 3×3 size and 16 channels of convolutional layers to extract the features of the image, and the output of the secondary feature extraction network is jump-connected to the fourth group of convolutional layers; finally, dimensionality reduction is performed in sequence through global average pooling and 1×1 convolution kernel, and the feature map after dimensionality reduction is output.

Images