A method for underwater image enhancement processing

Abstract

The present application provides an underwater image enhancement processing method, which constructs a RetinexNet image enhancement model. First, the Decom network model is used to decompose the input image into a reflection component and an illumination component. Then, the double inversion dehazing algorithm is integrated into the Relight network to adjust the illumination component of the image. After the improvement of the underwater image enhancement algorithm based on the improved RetinexNet with double inversion dehazing, the image details are more prominent, the colors are more vivid, and the detection targets are more obvious, which is particularly suitable for underwater image enhancement.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an underwater image enhancement processing method. Background Art

[0002] With the rapid development of smart cars, wading vehicles are an inevitable choice for professionals who often travel on waterways or frequently handle underwater work. Vehicles equipped with various underwater target detection functions will become a competitive advantage, and the combination of underwater target detection and automobiles will become an indispensable part of the future. Because the underwater environment is interfered by factors such as light attenuation, scattering, and absorption, the target objects in the underwater images will be blurred. In the underwater environment, different wavelengths of light are absorbed by water to different degrees, which will cause color distortion in the image and make the texture information of the underwater target objects blurred. This will affect the effect and accuracy of underwater target detection for underwater images. Therefore, in the underwater target detection method, it is particularly important to enhance the underwater image.

[0003] With the development of technology, deep learning algorithms have been introduced into the enhancement algorithms for underwater image acquisition. After searching, in the prior art, some technicians use the following methods to improve image enhancement processing:

[0004] 1. Rapid object detection using a boosted cascade of simple features (P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simplefeatures," Proceedings of the 2001 IEEE Computer Society Conference on Computer Visionand Pattern Recognition. CVPR 2001, Kauai, HI, USA, 2001, pp.II.) proposed a dual generative adversarial network for underwater image enhancement. The network uses two discriminators to complete adversarial learning of different areas of the underwater image using different training strategies, thereby obtaining an enhanced image.

[0005] 2. An effective framework for underwater image enhancement (Zhang H, Sun L, Wu L, et al. Du GAN: An effective framework for underwater image enhancement[J]. IET ImageProcessing, 2021, 15(9): 2010-2019.) proposed an underwater image enhancement algorithm based on illumination domain analysis. The contrast of the enhanced image is still relatively low; the rise of generative adversarial networks has introduced new methods for image enhancement.

[0006] However, in actual use, it was found that the above two technical solutions still have room for optimization. There are problems such as the inability to effectively correct the color of underwater images, the brightness enhancement of underwater images is not obvious, the color shift reduction is small, the contrast increase is low, and the details of the target objects are not highlighted enough. Summary of the invention

[0007] In order to solve the problem that the image enhancement effect of the underwater image enhancement processing method based on the deep learning algorithm in the prior art is still insufficient, the present invention provides an underwater image enhancement processing method, which can make the details of the enhanced target image more prominent, the color more vivid, and the detected target more obvious.

[0008] The technical solution of the present invention is as follows: an underwater image enhancement processing method, characterized in that it comprises the following steps:

[0009] S1: build a double inversion dehazing module;

[0010] The double inversion defogging module includes: a negation operation, a dark channel priori defogging operation and a negation operation connected in sequence;

[0011] The dark channel priori defogging operation includes: calculating dark channel, atmospheric light estimation, transmittance estimation and transmission optimization;

[0012] The calculation formula of the dark channel prior defogging operation is:

[0013] J(x)=((I(x)-A) / max(t(x),t0))+A;

[0014] Among them, J(x) is the image obtained after the dark channel prior dehazing operation, I(x) represents the image to be processed, t(x) represents the transmittance, A represents the global atmospheric light value; t0 is the transmittance of the image to be processed;

[0015] S2: Build the brightness adjustment module RelightNet;

[0016] The brightness adjustment module RelightNet enhances the brightness of the input image;

[0017] S3: Construct component enhancement module DecomNet;

[0018] The component enhancement module DecomNet optimizes the specified components of the input image;

[0019] S4: Build the RetinexNet image enhancement model;

[0020] The RetinexNet image enhancement model includes: two parallel component enhancement modules DecomNet, and a parallel brightness adjustment module RelightNet and a double inversion defogging module;

[0021] The input image is simultaneously sent to the two parallel component enhancement modules DecomNet; the two component enhancement modules DecomNet respectively optimize the brightness component and the reflection component of the input image, and the two component enhancement modules DecomNet share weights, and finally output the input image after the brightness component and the reflection component are optimized at the same time, and the image is sent to the parallel brightness adjustment module RelightNet and the double inversion defogging module respectively;

[0022] The brightness adjustment module RelightNet optimizes the brightness of the input image; the double inversion defogging module performs a defogging operation on the input image; the outputs of the brightness adjustment module RelightNet and the double inversion defogging module are combined after being optimized through a convolution operation to obtain a final output image.

[0023] It is further characterized by:

[0024] The inversion operation is to invert the original image at the pixel level. The main operation is to extract the red (R), green (G), blue (B) channels of the RGB image respectively, and invert them into the corresponding pixel values ​​R'G'B'. The formula is:

[0025] R'=255-R;

[0026] G'=255-G;

[0027] B'=255-B;

[0028] The transmittance t(x) in the dark channel prior dehazing operation is learned through a convolutional neural network;

[0029] The network model of the brightness adjustment module RelightNet includes 5 consecutive convolutional layers, 3 consecutive hierarchical adjustment modules and 1 1×1 convolutional layer connected in sequence; the hierarchical adjustment module includes: a nearest difference Resize operation and a convolution operation;

[0030] The input image is optimized for brightness through 5 convolutional layers, and then the brightness of the image is hierarchically adjusted through 3 consecutive nearest difference resize operations. Finally, it is connected to the same channel feature map through 1*1 convolution to achieve the effect of uniform global brightness enhancement.

[0031] The component enhancement module DecomNet includes: 5 consecutive convolutional layers and a sigmoid activation function, and an activation function ReLU is set after each convolutional layer;

[0032] For the input image, specify the component that needs to be enhanced, learn the specified component through 5 consecutive convolutional layers using key constraints, and optimize the specified component of the input image by minimizing the difference between the decomposed specified component and the original image;

[0033] The component enhancement module DecomNet learns the designated component, including: learning the consistent reflectivity shared by low-brightness and normal-light images, and learning the smoothness weight of illumination.

[0034] The present application provides an underwater image enhancement processing method, which constructs a RetinexNet image enhancement model. First, the Decom network model is used to decompose the input image into a reflection component and an illumination component. Then, the double inversion dehazing algorithm is integrated into the Relight network to adjust the illumination component of the image. After the improvement of the underwater image enhancement algorithm based on the improved RetinexNet with double inversion dehazing, the image details are more prominent, the colors are more vivid, and the detection targets are more obvious, which is particularly suitable for underwater image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic diagram of the structure of the RetinexNet image enhancement model in this application;

[0036] Figure 2 This is a diagram of the underwater image after the double inversion dehazing algorithm process;

[0037] Figure 3 Example of double inversion dehazing effect;

[0038] Figure 4 This is a comparison chart of the D-RetinexNet image enhancement algorithm and different algorithms;

[0039] Figure 5 This is a comparison chart of the thermal map visualization results before and after image enhancement. DETAILED DESCRIPTION

[0040] The present application includes an underwater image enhancement processing method, which includes the following steps.

[0041] S1: Construct a dual reverse dehaze module.

[0042] Underwater optical images have color distortion and low contrast because light decays when propagating in water and the interference of complex backgrounds can also have an adverse effect on underwater target detection tasks. In this method, the pixels of the RGB three-channel color underwater image are inverted and then the defogging algorithm is performed. Then the defogging image is inverted in the same way to obtain a relatively clear, brightness-enhanced and color-restored enhanced image.

[0043] The double inversion defogging module includes: a reverse operation Reverse Image, a dark channel prior defogging operation, and a reverse operation Reverse Image connected in sequence. That is, after a single inversion of the dark image, a bright image is generated, and then the dark channel prior defogging operation is used to remove the bright part, and then another inversion is performed to obtain an image with better contrast, image quality, color saturation, and visual effect than the original image, which can improve the subsequent recognition algorithm.

[0044] The dark channel prior dehazing operations include: calculate dark channel, estimation of atmospheric light, transmittance estimation and transmission optimization.

[0045] The inversion operation is to invert the original image at the pixel level. The main operation is to extract the red, green, blue and blue channels of the RGB image respectively, and invert them into the corresponding pixel values ​​​​R'G'B'. The formula is expressed as:

[0046] R'=255-R;

[0047] G'=255-G;

[0048] B'=255-B.

[0049] The calculation formula of the dark channel prior dehazing operation is:

[0050] J(x)=((I(x)-A) / max(t(x),t0))+A;

[0051] Among them, J(x) is the image obtained after the dark channel prior dehazing operation, I(x) represents the image to be processed, t(x) represents the transmittance, A represents the global atmospheric light value; t0 is the transmittance of the image to be processed.

[0052] The transmittance t(x) in the dark channel prior dehazing operation is learned through a convolutional neural network. In specific applications, a transmittance learning model is constructed based on a convolutional neural network, and a training data set and a validation data set are constructed using labeled image data that can show transmittance changes. The convolutional neural network is trained using the training data set; the transmittance is obtained by repeatedly calculating each image during the training of the convolutional neural network. In this method, the transmittance is estimated by the convolutional neural network to ensure the consistency of the authenticity of the input and output images. The convolutional neural network is used to iteratively learn the transmittance value in the double inversion dehazing algorithm, so that the network can more prominently process the details of the image when adjusting the brightness and color of the underwater image, so that the target to be detected can be more prominent than the background and the color is more vivid.

[0053] like Figure 2 In the figure, A is the original image, A1 is the effect of image A after the first inversion operation, A2 is the effect of image A1 after the dark channel prior defogging operation, and A3 is the effect of image A2 after the inversion operation. B is the original image, B1 is the effect of image B after the first inversion operation, B2 is the effect of image B1 after the dark channel prior defogging operation, and B3 is the effect of image B2 after the inversion operation. Comparing A with A3, and comparing B with B3, it can be seen that the image after the double inversion defogging algorithm has been improved in color and brightness.

[0054] The pixel values ​​of the original image and the image enhanced by the double inversion defogging algorithm and the total number of pixels corresponding to the pixel values ​​are digitized and then plotted, as shown in Figure 3 As shown in the comparison chart in . It can be found that the pixel points of the original image a are distributed around 0-100, and referring to the pixel value image c of the original image, the number of pixel points with a pixel value of 30 in c is obviously larger, while there is no pixel value distribution at the pixel value of 100-250. The overall pixel value is low and the image quality is poor. The original image a is enhanced by the double inversion dehazing algorithm to obtain the algorithm-enhanced image b, and the pixel point data image d corresponding to image b. It can be seen that the distribution of each pixel value in the enhanced image becomes more uniform. This shows that the image enhanced by the double inversion dehazing algorithm has been improved in both color and brightness, which helps to improve the accuracy of underwater target detection. Figure 3The horizontal axis of c and d is the pixel value, and the vertical axis represents the number of pixels with the intensity value in the image at a specific intensity value.

[0055] S2: Construct a brightness adjustment module RelightNet; the brightness adjustment module RelightNet enhances the brightness of the input image.

[0056] The network model of the brightness adjustment module RelightNet includes 5 consecutive convolutional layers, 3 consecutive hierarchical adjustment modules and 1 1×1 convolutional layer connected in sequence; the hierarchical adjustment module includes: a nearest difference Resize operation and a convolution operation;

[0057] The input image is optimized for brightness through 5 convolutional layers; then the brightness of the image is adjusted hierarchically through 3 consecutive nearest difference resize operations, that is, different local illumination distributions are adjusted while maintaining global illumination consistency; finally, it is connected to the same channel feature map through 1*1 convolution to achieve the effect of uniform global brightness enhancement.

[0058] S3: Construct a component enhancement module DecomNet; the component enhancement module DecomNet optimizes the specified components of the input image.

[0059] The component enhancement module DecomNet includes: 5 consecutive convolutional layers and a sigmoid activation function, and an activation function ReLU is set after each convolutional layer; for the input image, the component to be enhanced is specified, and the specified component is learned by using key constraints through 5 consecutive convolutional layers, and the specified component of the input image is optimized by minimizing the difference between the decomposed specified component and the original image. Specifically, the learning of the specified component by the component enhancement module DecomNet in this embodiment includes two parts: learning the consistent reflectivity shared by low-brightness and normal-light images, and learning the smoothness weight of the illumination.

[0060] That is, instead of using the ground truth of reflectance and illumination, DecomNet learns using key constraints, including the consistent reflectance shared by low-light and normal-light images, and the smoothness weight of the illumination, represented by the double-headed arrows in the figure; during training, the network optimizes by minimizing the difference between the decomposed illumination and reflectance components and the original image.

[0061] S4: Build the RetinexNet image enhancement model;

[0062] The RetinexNet image enhancement model includes: two parallel component enhancement modules DecomNet, a parallel brightness adjustment module RelightNet and a double inversion dehazing module. The two parallel component enhancement modules DecomNet are used to decompose the input image into reflection components and shadow components, and the brightness adjustment module RelightNet is used to readjust the contrast and color lighting of the image according to these components.

[0063] The input image is sent to two parallel component enhancement modules DecomNet at the same time; the two component enhancement modules DecomNet optimize the brightness component and reflection component of the input image respectively, and the two component enhancement modules DecomNet share weights, and finally output the input image with both the brightness component and the reflection component optimized, and send the image to the parallel brightness adjustment module RelightNet and the double inversion dehazing module respectively.

[0064] The brightness adjustment module RelightNet optimizes the brightness of the input image; the double inversion defogging module performs defogging on the input image; the outputs of the brightness adjustment module RelightNet and the double inversion defogging module are optimized and merged after convolution operation to obtain the final output image.

[0065] The main function of the brightness adjustment module RelightNet is to adjust the brightness of the image. In this application, the double inversion dehazing algorithm is integrated into the RelightNet network. The double inversion dehazing algorithm is learned through a convolutional neural network, so that the network can make the details of the image more prominent when adjusting the brightness and color of underwater images, making the target to be detected more prominent compared to the background and the color more vivid. In specific applications, the double inversion dehazing algorithm is integrated into the RetinexNet network by modifying the code.

[0066] Subsequent experiments have shown that the RetinexNet network with the double inversion dehazing algorithm can enhance underwater images in terms of color and brightness, and compared with traditional image enhancement algorithms, the improved RetinexNet with the double inversion dehazing algorithm is more adaptable to complex underwater backgrounds. At the same time, the RetinexNet (D-RetinexNet) algorithm model that integrates the double inversion dehazing algorithm can generate more positive samples through this improved image enhancement method before feeding underwater images into the target detection model for training, so that the subsequent target detection network can produce a greater gain effect when the backbone network extracts features of the target detection object.

[0067] In order to confirm the effect of the image enhancement processing method of the present application, three different image enhancement algorithms are used to perform image enhancement on the same set of images for comparison. Figure 4 The three image enhancement algorithms are: white balance enhancement algorithm, histogram equalization enhancement algorithm and the image enhancement method of this application (labeled as D-RetinexNet in the text).

[0068] The clarity enhancement degree is defined as Q, and the contrast enhancement degree is defined as D;

[0069] The clarity enhancement degree Q is equal to the enhanced image clarity minus the original underwater image clarity multiplied by 100%.

[0070] The contrast enhancement degree D is equal to the enhanced contrast minus the original underwater image contrast multiplied by 100%.

[0071] Figure 4 There are three original images in the figure: A, B, and C; the enhanced images based on the white balance enhancement algorithm are A1, B1, and C1; the enhanced images based on the histogram equalization enhancement algorithm are A2, B2, and C2; the enhanced images based on D-RetinexNet are A3, B3, and C3.

[0072] The clarity enhancement degree was defined as Q and the contrast enhancement degree was defined as D to evaluate the images of group A (A1, A2, A3), group B (B1, B2, B3), and group C (C1, C2, C3) enhanced by different enhancement algorithms, and the following Table 1 was obtained.

[0073] Table 1 Comparison of evaluation indicators of different enhancement algorithms

[0074]

[0075] Based on Table 1, it can be seen that the white balance enhancement algorithm has little effect on enhancing the clarity and contrast of underwater images, and the histogram equalization enhancement algorithm is not as good as the D-RetinexNet enhancement effect in terms of clarity and contrast. This proves that traditional physical methods cannot meet the requirements of underwater image enhancement. When faced with complex backgrounds and low brightness, the image enhancement algorithm based on deep learning in this application is more advantageous.

[0076] Figure 4It can be seen that group A images are underwater images of small target fish with low brightness, group B images are underwater images of normal fish with complex backgrounds at low brightness, and group C images are underwater images of normal fish with normal brightness but complex backgrounds. From the three groups of underwater images A1, B1, and C1 enhanced by the white balance enhancement algorithm, it can be found that when the brightness is too low, the image enhancement effect is not obvious. When the background brightness is normal, the color of the underwater image is restored more evenly after white balance, but the enhancement effect on the target is still not obvious. The three groups of underwater images A2, B2, and C2 enhanced by the histogram equalization enhancement algorithm do not save the R, G, and B three-channel pixels of the original underwater image but are converted into grayscale images. Therefore, when the background is not complex, it can play a certain role in target detection. However, when the background is complex and there are many targets (such as groups A2 and C2), the enhancement effect on the target and background is not good and it is difficult to distinguish. After the three groups of underwater images A3, B3, and C3 enhanced by the D-RetinexNet enhancement algorithm, it is found that the algorithm can improve the contrast while restoring the color, which can increase the detection accuracy of the subsequent target detection network.

[0077] Heatmap is a visualization technology that uses color to represent the density of data. It displays different colors according to the distribution of data on the image to show the distribution of image focus points, thereby visually showing the degree of attention paid by the neural network to the data in the image. Figure 5 , the images enhanced by D-RetinexNet are visualized in heat maps and compared with the original images. It can be found that the underwater fish images of the AD group without D-RetinexNet image enhancement and the underwater fish images of the EH group after D-RetinexNet image enhancement are compared in heat map visualization results. It is found that the enhanced images can cover more parts of the target detection object. At the same time, the image enhanced images show in the heat map that the target detection network will pay attention to more target locations, which means that the target detection network can pay more attention to the target feature information required in the image and increase the probability of final object identification.

[0078] from Figure 5 From the comparison of the heat maps before and after image enhancement, it can be seen that the AD group of images are original underwater images with color distortion and low contrast. From the comparison of the A' and E' heat maps, it can be found that the image attention area that has not been enhanced by the image enhancement algorithm is less than the image attention area after the image enhancement algorithm. From the three groups of images F, G, and H, it can be found that the images enhanced by the D-RetinexNet image enhancement algorithm have significant performance in contrast enhancement and color restoration. After visualizing the heat maps of the enhanced E', F', G' and H' images, it is found that the attention area becomes wider, which can enable the subsequent target detection network to extract more feature information when performing feature extraction.

Claims

1. A method for underwater image enhancement processing, characterized in that: It includes the following steps: S1: build a double inversion dehazing module; The double inversion defogging module includes: a negation operation, a dark channel priori defogging operation and a negation operation connected in sequence; The dark channel priori defogging operation includes: calculating dark channel, atmospheric light estimation, transmittance estimation and transmission optimization; The calculation formula of the dark channel prior defogging operation is: J(x)=((I(x)-A) / max(t(x),t0))+A; Among them, J(x) is the image obtained after the dark channel prior dehazing operation, I(x) represents the image to be processed, t(x) represents the transmittance, A represents the global atmospheric light value; t0 is the transmittance of the image to be processed; S2: Build the brightness adjustment module RelightNet; The brightness adjustment module RelightNet enhances the brightness of the input image; S3: Construct component enhancement module DecomNet; The component enhancement module DecomNet optimizes the specified components of the input image; S4: Build the RetinexNet image enhancement model; The RetinexNet image enhancement model includes: two parallel component enhancement modules DecomNet, and a parallel brightness adjustment module RelightNet and a double inversion defogging module; The input image is simultaneously sent to the two parallel component enhancement modules DecomNet; the two component enhancement modules DecomNet respectively optimize the brightness component and the reflection component of the input image, and the two component enhancement modules DecomNet share weights, and finally output the input image after the brightness component and the reflection component are optimized at the same time, and the image is sent to the parallel brightness adjustment module RelightNet and the double inversion defogging module respectively; The brightness adjustment module RelightNet optimizes the brightness of the input image; the double inversion defogging module performs a defogging operation on the input image; the outputs of the brightness adjustment module RelightNet and the double inversion defogging module are combined after being optimized through a convolution operation to obtain a final output image.

2. The underwater image enhancement processing method according to claim 1, characterized in that: The inversion operation is to invert the original image at the pixel level. The main operation is to extract the red (R), green (G), blue (B) channels of the RGB image respectively, and invert them into the corresponding pixel values ​​R'G'B'. The formula is: R'=255-R; G'=255-G; B'=255-B.

3. The underwater image enhancement processing method according to claim 1, characterized in that: The transmittance t(x) in the dark channel prior dehazing operation is learned through a convolutional neural network.

4. The underwater image enhancement processing method according to claim 1, characterized in that: The network model of the brightness adjustment module RelightNet includes 5 consecutive convolutional layers connected in sequence, 3 consecutive hierarchical adjustment modules and 1 1×1 convolutional layer; The hierarchical adjustment module includes: a nearest difference resize operation and a convolution operation; The input image is optimized for brightness through 5 convolutional layers, and then the brightness of the image is hierarchically adjusted through 3 consecutive nearest difference resize operations. Finally, it is connected to the same channel feature map through 1*1 convolution to achieve the effect of uniform global brightness enhancement.

5. The underwater image enhancement processing method according to claim 1, characterized in that: The component enhancement module DecomNet includes: 5 consecutive convolutional layers and a sigmoid activation function, and an activation function ReLU is set after each convolutional layer; For the input image, the components that need to be enhanced are specified, and the specified components are learned through 5 consecutive convolutional layers using key constraints. The specified components of the input image are optimized by minimizing the difference between the decomposed specified components and the original image.

6. The underwater image enhancement processing method according to claim 5, characterized in that: The component enhancement module DecomNet learns the designated component, including: learning the consistent reflectivity shared by low-brightness and normal-light images, and learning the smoothness weight of illumination.

Images