End-to-end underwater image restoration method based on ambient light perception

Underwater image processing through convolutional neural network based on ambient light perception, the problem of image clarity and color offset correction in the prior art is solved, more natural color presentation and detail retention are achieved, and objective indicators of image quality are improved.

CN113935916BActive Publication Date: 2024-11-22XIDIAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111195071.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-12
Publication Date
2024-11-22
Estimated Expiration
2041-10-12

AI Technical Summary

Technical Problem

Existing underwater image processing methods are difficult to effectively improve image clarity and completely correct color shifts, especially at different degrees of degradation, resulting in poor restoration effect.

Method used

Using a convolutional neural network based on ambient light perception, the ambient light perception network and the restoration subject network are constructed, and the adaptive moment estimation algorithm is used for training, combining multi-scale codec structures and residual modules to realize end-to-end underwater image restoration. .

Benefits of technology

It improves the color nature and detail retention of underwater images, reduces training errors, and significantly improves the visual effect of the images. Objective indicators such as PSNR, SSIM and CIEDE2000 are better than the existing technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113935916B_ABST
    Figure CN113935916B_ABST
Patent Text Reader

Abstract

The present invention discloses an end-to-end underwater image restoration method based on ambient light perception, which mainly solves the problem that the existing technology has poor color cast correction and clarity processing effects when processing underwater images. The scheme is: using the Pytorch framework to respectively construct an ambient light perception network and a restoration subject network, and respectively construct training sets B and C for the two networks; using an adaptive moment estimation algorithm to respectively use B and C to train the ambient light perception network and the restoration subject network, and convert the image I to be processed c Input the trained ambient light perception network and output the ambient light value A c ; A c and I c Input the trained restoration subject network and output a clear image J c The present invention improves the contrast of underwater images with different degradation levels, can effectively correct color deviation, and has better peak signal-to-noise ratio, structural similarity, color difference formula, reference-free image space quality assessment and underwater color image quality evaluation than the existing technologies, and can be used for underwater image clarity processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an underwater image restoration method, which can be used to process a single underwater image taken by an imaging system. Background Art

[0002] Affected by the absorption and scattering of light by water in real conditions, underwater optical images generally have quality degradation problems such as low contrast, color distortion, and image blur. These degraded images not only affect the subjective perception of the human eye, but also seriously restrict the performance of various intelligent visual information processing systems. Therefore, reconstructing clear underwater optical images has very important practical application value. At present, the key issue of underwater image processing methods is how to improve image clarity and correct color deviation. There are four main methods: traditional image enhancement, traditional image restoration, deep learning image restoration, and deep learning image enhancement. Among them:

[0003] Traditional image enhancement methods select suitable image enhancement techniques to improve image quality based on the specific manifestations of underwater image degradation. Typical methods include the underwater image enhancement algorithm based on Retinex and histogram equalization proposed by Zhang et al., see Zhang W, Li G, Ying Z, and et al. A New Underwater Image Enhancing Method via Color Correction and Illumination Adjustment [C]. / / IEEE International Conference on Visual Communications and Image Processing. 2017, DOI: 10.1109 / VCIP.2017.8305027; the underwater image enhancement algorithm based on fusion strategy proposed by Ancuti et al., see C. Ancuti, CO Ancuti, T. Haber, and P. Bekaert. Enhancing underwater images and videos by fusion [C]. / / IEEE Conference on Computer Vision and Pattern Recognition. 2012, DOI: 10.1109 / CVPR.2012.6247661. This type of method is simple in principle and can effectively improve the visual effect of the image. However, since it does not take into account the principle of underwater image degradation and ignores the relationship between the degree of degradation and depth, its enhancement results cannot correctly reflect the true color of the image.

[0004] Traditional image restoration methods are based on the physical imaging model of underwater degraded images, using different priors or assumptions to extract image features, and then using these features to design effective ambient light and transmittance estimation methods to achieve image restoration. Typical methods include the red channel prior proposed by Galdran et al., where the difference between 1 and the red channel value is used instead of the red channel value when calculating the dark channel, see Galdran A, Alvarez-Gila A, Alvarez-Gila A. Automatic Red-Channel underwater image restoration [J]. Journal of Visual Communication & Image Representation, 2015, 26 (C): 132-145. Although this method improves the effect of dark channel prior applied to underwater images, it violates the original statistical significance of the dark channel and reduces the effectiveness of the prior in clear images; Li et al. estimate the transmittance by reducing the red channel information loss, and estimate the ambient light value by taking advantage of the high brightness and large difference between the red and blue channels, see Li C, Guo J, Cong R, et al. Underwater image enhancement by dehazing with minimum information loss and histogram distribution prior [J]. IEEE Transactions on Image Processing. 2016: 5664-5677. This method effectively enhances the contrast of the image, but cannot completely remove the color cast. The image restoration effect of this type of method is highly dependent on the reliability of the prior. If the prior fails, large estimation errors will occur, resulting in poor image restoration effect.

[0005] The deep learning image restoration method is based on the physical imaging model and uses a neural network to automatically learn the nonlinear mapping relationship between underwater degraded images and imaging model parameters. Hou et al. proposed using the residual convolutional neural network URCNN to estimate the transmittance and ambient light of underwater images, and used the scene residual calculation method to balance underwater lighting, see Hou M, Liu R, Fan X and Luo Z. Joint residual learning for underwater image enhancement [C] / / IEEE International Conference on Image Processing (ICIP). 2018: 4043–4047. Wang et al. proposed a method of using a parallel convolutional neural network to estimate transmittance and ambient light respectively, and then used the underwater imaging model to obtain clear underwater images, see Wang K, Hu Y, Chen J, Wu X, Zhao X, Li Y. Underwater Image Restoration Based on a Parallel Convolutional Neural Network[J] / / RemoteSensing.2019;11(13):1951. The image restored by this network is clearer and more natural, but it still cannot eliminate color cast on images with a high degree of degradation, and the network model cannot avoid error transmission, so that small model parameter estimation errors may be amplified during the restoration process, resulting in large deviations in the restoration results.

[0006] The deep learning image enhancement method directly learns the mapping relationship between underwater degraded images and corresponding clear images "end-to-end" without the help of physical models. Li et al. borrowed the idea of ​​image fusion and proposed the WaterNet network. First, the original image is processed with white balance, histogram equalization and gamma correction respectively, input into the network, and the network is used to learn three confidence maps, which are then weighted fused with the image to obtain the enhanced underwater image, see Li C, Guo C, Ren W, and et al. An underwater image enhancement benchmark dataset and beyond [J] / / IEEE Transactions on Image Processing. 2020: 4376-4389; Islam et al. proposed the FUnIE-GAN network to enhance underwater images in real time, see Islam MJ, Xia Y and Sattar J. Fast underwater image enhancement for improved visual perception [J] / / IEEE Robotics and Automation Letters. 2020: 3227-3234. The network design of this type of method is relatively complex, the learning difficulty is relatively large, and it is easy to encounter problems of difficulty in fitting. In addition, this type of method is heavily dependent on the distribution of training data, has insufficient generalization ability, and has poor restoration effect on severely degraded underwater images. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose an underwater image restoration method based on ambient light perception, so as to utilize a convolutional neural network to restore underwater images end-to-end, reduce the color distortion and image blur of underwater images under different degrees of degradation, and improve the image restoration effect.

[0008] To achieve the above object, the technical solution of the present invention includes the following:

[0009] 1) Constructing underwater image restoration network ECCN in Pytorch framework:

[0010] 1a) Establish an ambient light perception network consisting of eleven convolutional layers, one upsampling layer, and one pooling layer;

[0011] 1b) Establishing a restoration main network including a shallow feature extraction module, a multi-scale encoding and decoding module and three feature integration and reconstruction modules;

[0012] 1c) Cascading the ambient light sensing network and the restoration main network to form an underwater image restoration network;

[0013] 2) Download the indoor dataset NYU containing depth information, the real single image dehazing dataset RESIDE containing depth information, and the paired underwater dataset EUVP synthesized based on the generative network, and filter, scale and crop these datasets in turn to obtain the indoor clear image set J, indoor depth atlas D, underwater clear image set U, and underwater degraded image set R1;

[0014] 3) Construct training image sets for the ambient light perception network and the restoration subject network respectively:

[0015] Set the ambient light value A and the blue channel transmittance T b , using the indoor clear image set J and the indoor depth atlas D, synthesize the underwater degraded image set R2, and pair the underwater degraded image set R2 with its corresponding ambient light value A as the training image set B of the ambient light perception network;

[0016] Pair the indoor clear image set J with the underwater degraded image set R2, and add the underwater clear image set U and the underwater degraded image set R1 as the training image set C for restoring the main network;

[0017] 4) Using the training image set B of the ambient light perception network, adopting the adaptive moment estimation algorithm, and aiming at minimizing the Euclidean distance loss value, the ambient light perception network is trained to obtain a trained ambient light perception network;

[0018] 5) Using the training image set C of the restoration subject network and the trained ambient light estimation network, the adaptive moment estimation algorithm is used to train the restoration subject network with the goal of minimizing the weighted total loss value to obtain a trained restoration subject network;

[0019] 6) Take an underwater image I to be restored c Input to the trained ambient light perception network and output the ambient light value A of the RGB three channels c ;

[0020] 7) Set the ambient light value A c With underwater image I c Input them together into the trained restoration subject network, and output the restored clear image J c .

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] First, the present invention uses a simple regression network to perceive the prior information of ambient light related to color, which assists the training and restoration of the main network, so that the restoration results of underwater images with different degrees of degradation can have more natural and accurate color presentation;

[0023] Second, the restoration main network of the present invention adopts a multi-scale codec structure and integrates feature information of three scales from coarse to fine, which can restore the global structure and local details of the degraded image; at the same time, due to the introduction of the residual module in the codec structure, the information transmission efficiency of each layer of the network during the training process is improved, the training error of the network is reduced, and it is more sensitive to detail transformation;

[0024] Third, the weighted loss training network designed by the present invention comprehensively considers the global structure, local texture and color factors, so that the detail information and edge texture information of the restored image are better preserved and the color restoration is more accurate.

[0025] The simulation results show that, under the premise of maintaining the contrast of the restored image, the present invention can better correct the image color cast and improve the visual effect; compared with other existing algorithms, the processing results have clearer boundaries, more natural colors, and higher overall image quality, and the objective indicators of peak signal-to-noise ratio PSNR, structural similarity SSIM and color difference formula CIEDE2000 are better than the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a general flow chart for realizing the present invention;

[0027] Figure 2 A structural diagram of the ambient light sensing network constructed in the present invention;

[0028] Figure 3 A structural diagram of the restoration subject network constructed in the present invention;

[0029] Figure 4 A comparison diagram of the processing effects of the present invention and the existing underwater image processing algorithm on underwater simulation images;

[0030] Figure 5 The figure is a comparison chart of the processing effects of the present invention and the existing underwater image processing algorithm on real underwater images. DETAILED DESCRIPTION

[0031] The specific implementation manner and effects of the present invention will be further described below in conjunction with the accompanying drawings.

[0032] Reference Figure 1 The specific implementation steps of the present invention are as follows:

[0033] Step 1: Build an ambient light perception network under the Pytorch framework.

[0034] like Figure 2As shown, the ambient light perception network constructed by the present invention includes eleven convolutional layers, one upsampling layer and one pooling layer, and its structural relationship is: the first convolutional layer → the second convolutional layer → the third convolutional layer → the fourth convolutional layer → the upsampling layer → the fifth convolutional layer → the sixth convolutional layer → the seventh convolutional layer → the eighth convolutional layer → the pooling layer → the ninth convolutional layer → the tenth convolutional layer → the eleventh convolutional layer;

[0035] The parameters of each layer are set as follows:

[0036] The convolution kernel size of the first convolution layer and the second convolution layer is 9*9.

[0037] The convolution kernel size of the third convolution layer, the fourth convolution layer, the fifth convolution layer, and the sixth convolution layer are all 7*7.

[0038] The convolution kernel sizes of the seventh, eighth, ninth, and tenth convolutional layers are all 5*5.

[0039] The convolution kernel size of the eleventh convolutional layer is 3*3.

[0040] The stride of all convolutional layers is 1. All convolutional layers include convolution operations and ReLU activation function layers.

[0041] The scaling factor of the upsampling layer is 2,

[0042] The pooling layer has a scaling factor of 2.

[0043] Step 2: Build the restoration subject network under the Pytorch framework.

[0044] like Figure 3 As shown in the figure, the restoration main network constructed in this example includes a shallow feature extraction module, a multi-scale encoding and decoding module, and three feature integration and reconstruction modules, among which:

[0045] The shallow feature extraction module is composed of a cascade of the first convolution layer → the second convolution layer → the third convolution layer;

[0046] The multi-scale encoding and decoding module is composed of three branches connected in parallel, where:

[0047] The first branch is: 1st pooling layer → 4th convolutional layer → 1st encoding unit → 2nd encoding unit → 1st decoding unit → 2nd decoding unit;

[0048] The second branch is: the second pooling layer → the fifth convolution layer → the third encoding unit → the fourth encoding unit → the third decoding unit → the fourth decoding unit; the third decoding unit is also connected to the second encoding unit;

[0049] The third branch is: the 3rd pooling layer → the 6th convolutional layer → the 5th encoding unit → the 6th encoding unit → the 5th decoding unit → the 6th decoding unit; the 5th decoding unit is also connected to the 4th encoding unit;

[0050] The first pooling layer, the second pooling layer, and the third pooling layer are all connected to the third convolutional layer;

[0051] Each coding unit is: I convolution layer → I residual module → II residual module → III residual module;

[0052] Each decoding unit is sequentially: IVth residual module → Vth residual module → VIth residual module → bilinear interpolation → IIth convolution layer;

[0053] The feature integration and reconstruction module is sequentially: upsampling layer → first convolution layer → second convolution layer → third convolution layer;

[0054] The connection relationship between each module is as follows:

[0055] The second decoding unit, the fourth decoding unit, and the sixth decoding unit in the multi-scale encoding and decoding module are respectively connected to the three feature reconstruction modules, and the third convolution layer of the shallow feature extraction module is simultaneously connected to the first convolution layer of the feature reconstruction module;

[0056] The output of the main restoration network is the terminal output of the feature reconstruction module connected to the third branch;

[0057] The parameters of each layer are set as follows:

[0058] The convolution kernel size of the first convolution layer is 7*7, the convolution kernel size of the second convolution layer is 5*5, and the convolution kernel size of the remaining convolution layers is 3*3; the convolution step size of all convolution layers is 1, and all convolution layers include convolution operations and ReLU activation function layers;

[0059] The scaling factor of the upsampling layer is 2;

[0060] The scaling factor of the first pooling layer is 8.

[0061] The scaling factor of the second pooling layer is 4.

[0062] The scaling factor of the third pooling layer is 2;

[0063] The bilinear interpolation operation is scaled by a factor of 2.

[0064] Step 3: Construct training image sets for the ambient light perception network and the restoration subject network respectively.

[0065] 3.1) Download the indoor dataset NYU containing depth information, the real single image dehazing dataset RESIDE containing depth information, and the paired underwater dataset EUVP synthesized based on the generative network from the Internet, and filter, scale and crop these datasets in turn, unify the size to 160×160, and obtain the indoor clear image set J, indoor depth atlas D, underwater clear image set U, and underwater degraded image set R1;

[0066] 3.2) Use the random function to randomly generate the red channel ambient light value A between 0.1 and 0.6 r , and then use the random function in A r -1.0 Randomly generate green channel ambient light value A g and the ambient light value A of the blue channel b ;

[0067] 3.3) Use the random function to randomly generate the blue channel transmittance parameter β between 0.5 and 2.5 b , the blue channel transmittance of each image is calculated using the indoor depth atlas D Red channel transmittance Green channel transmittance

[0068] 3.4) According to the indoor clear image set J and the ambient light value A and transmittance T generated in 3.2), the underwater degraded image set R2 = JT + A (1-T) is calculated;

[0069] 3.5) After scaling the size of the underwater degraded image set R2 to 49×49, it is paired with the ambient light value A and used together as the training image set B of the ambient light perception network;

[0070] 3.6) Pair the indoor clear image set J with the underwater degraded image set R2, add the underwater clear image set U and the underwater degraded image set R1 to the paired image set, and use them together as the training image set C for restoring the main network.

[0071] Step 4: Train the neural network.

[0072] 4a) Training the ambient light perception network:

[0073] 4a1) Use the Euclidean distance formula as the loss function of the ambient light perception network:

[0074]

[0075] Among them, ||·||2 is the operation of finding the two-norm of the matrix, c is the color channel of the input image, B c is the output of the ambient light perception network, B' c is the corresponding artificial synthetic ambient light;

[0076] 4a2) inputting the underwater degraded images in the training image set B of the ambient light perception network into the ambient light perception network in sequence, and outputting the estimated ambient light value A';

[0077] 4a3) Substituting the set ambient light value A and the estimated ambient light value A' into the Euclidean distance formula at the same time, and calculating the Euclidean distance loss value between A and A';

[0078] 4a4) using an adaptive moment estimation algorithm to update the weights and bias values ​​of each convolution operation in the ambient light perception network;

[0079] 4a5) Repeat steps 4a2)-4a4) 10,000 times until the Euclidean distance loss value is minimized, and a trained ambient light perception network is obtained.

[0080] 4b) Train the restoration subject network:

[0081] 3b1) The weighted total loss formula is used as the loss function LossC of the restored main network, which is expressed as follows:

[0082] LossC=0.75×Loss t +0.25×Loss c

[0083] Among them, Loss t is the boundary preservation loss, Loss c is the color cast correction loss, calculated as:

[0084]

[0085] Among them, ||·||2 is the operation of finding the matrix norm, |·| is the operation of finding the matrix norm, N is the number of pixels of the input image, J n , J n1 , J n2 are the terminal outputs of the three feature integration and reconstruction modules of the main network, J' n , J' n1 , J' n2 is the corresponding clear image, J' n1 J' is reduced by 16 times n , J' n2 J' is reduced by 64 times n ; represents the average value of the red channel, Δr n , Δb n , Δg n Represent the differences between the network output and the clear image in the red, green, and blue channels respectively;

[0086] 4b2) Inputting the underwater degraded images in the restoration subject network training set C into the trained ambient light estimation network in sequence, and outputting the estimated ambient light value A”;

[0087] 4b3) combining the underwater degraded image in the restoration subject network training set C with the estimated ambient light value A'', inputting the result into the restoration subject network, and outputting the estimated restoration image I';

[0088] 4b4) Substituting the clear image I and the estimated restored image I' in the restoration subject network training set C into the weighted total loss function LossC at the same time, and calculating the weighted total loss value between the clear image I and the restored image I';

[0089] 4b5) using an adaptive moment estimation algorithm to update the weights and bias values ​​of each convolution operation in the restoration main network;

[0090] 4b6) Repeat 4b3)-4b5) for a total of 20,000 times until the weighted total loss value is minimized, and the trained restoration subject network is obtained.

[0091] Step 5: Restore the image.

[0092] 5a) Take an underwater image I to be restored c Input to the trained ambient light perception network in step 4a) and output the ambient light value A c ;

[0093] 5b) The underwater image I c and ambient light value A c Combine the input to the trained restoration subject network in step 4b) and output a high-quality clear image J c ,c∈{r,g,b}, complete the restoration of underwater degraded images.

[0094] The effect of the present invention is further illustrated by the following simulation:

[0095] 1. Test conditions and methods

[0096] 1. Test images: 406 sets of underwater simulated images and 126 real underwater images with different degrees of degradation were selected;

[0097] 2. Test method: Nine methods in total are used, including the existing Anwar's algorithm, Zhang's algorithm, Galdran's algorithm, Li's algorithm, Wang's algorithm, WaterNet, FUnIE-GAN, CWR and the present invention.

[0098] 3. Simulation test content:

[0099] Simulation test 1: Use the above nine methods to restore four underwater simulation images. The results are as follows: Figure 4 As shown, where:

[0100] Figure 4 a is four synthetic underwater images;

[0101] Figure 4 b is the algorithm using Anwar Figure 4 a The result of processing underwater simulation image;

[0102] Figure 4 c is the algorithm used by Zhang Figure 4 a The result of restoring underwater simulated image;

[0103] Figure 4 d is the algorithm pair using RCP Figure 4 a The result of restoring underwater simulated image;

[0104] Figure 4 e is the method using Li Figure 4 a The result of restoring underwater simulated image;

[0105] Figure 4 f is the method used by Wang Figure 4 a The result of restoring underwater simulated image;

[0106] Figure 4 g is the use of WaterNet Figure 4 a The result of restoring underwater simulated image;

[0107] Figure 4 h is the use of FUnIE-GAN Figure 4 a The result of restoring underwater simulated image;

[0108] Figure 4 i is used for CWR Figure 4 a The result of restoring underwater simulated image;

[0109] Figure 4 j is the method of the present invention used for Figure 4 a The result of restoring underwater simulated image;

[0110] from Figure 4 The following different recovery effects can be seen:

[0111] When the existing methods of Anwar, Zhang, RCP, and Li are used to process images, the restoration results of color information are poor, and almost all images retain the original blue / green color cast;

[0112] The images restored using the existing Anwar method and RCP algorithm are sometimes still blurry; the existing algorithm proposed by Wang et al. can remove color cast, but due to inaccurate transmittance estimation, the color transition in local areas is unnatural;

[0113] The images processed by the existing WaterNet have a yellowish color cast overall, and the colors are not realistic and natural enough;

[0114] The processing results of the existing CWR algorithm are unstable, and the color cast removal of some images will fail;

[0115] After processing with FunIEGAN, the image clarity is good and the color cast can be removed, but the restored image still has inconsistent coloring;

[0116] The image restored by the method of the present invention has better effects than those of several other algorithms. The restored image has clear boundaries and high saturation, and also has a good restoration effect on images with rich colors.

[0117] Simulation test 2: Use the above nine methods to restore six real underwater images. The results are as follows: Figure 5 As shown, where:

[0118] Figure 5 a are six real underwater images;

[0119] Figure 5 b is the algorithm using Anwar Figure 5 a The result of processing the real underwater image;

[0120] Figure 5 c is the algorithm used by Zhang Figure 5 a The result of restoring the real underwater image;

[0121] Figure 5 d is the algorithm pair using RCP Figure 5 a The result of restoring the real underwater image;

[0122] Figure 5 e is the method using Li Figure 5 a The result of restoring the real underwater image;

[0123] Figure 5 f is the method used by Wang Figure 5 a The result of restoring the real underwater image;

[0124] Figure 5 g is the use of WaterNet Figure 5 a The result of restoring the real underwater image;

[0125] Figure 5h is the use of FUnIE-GAN Figure 5 a The result of restoring the real underwater image;

[0126] Figure 5 i is used for CWR Figure 5 a The result of restoring the real underwater image;

[0127] Figure 5 j is the method of the present invention used for Figure 5 a The result of restoring the real underwater image;

[0128] from Figure 5 The following recovery effects of different methods can be seen:

[0129] The results of processing using the existing Anwar method show that the overall contrast is improved, but a purple color cast is introduced in the local area; the images processed using the existing Zhang algorithm and Li algorithm show oversaturation of colors, which is not realistic and natural enough;

[0130] After being processed by the existing WaterNet, the image saturation is reduced; the existing RCP and the method proposed by Wang et al. do not process the image color deviation sufficiently, resulting in under-processing, and many images still show severe color deviation;

[0131] Using the existing FUnIE-GAN eliminates most of the blue-green color cast, but loses a lot of texture details and main background information;

[0132] The existing CWR method obviously corrects the color shift and improves the visibility of the restored image. However, CWR shows low contrast or oversaturation in some local areas.

[0133] The image restoration effect of the method of the present invention is better than that of the other eight existing methods, and has good visual processing effect on images with different degrees of degradation.

[0134] Simulation test 3: 406 groups of underwater simulation images were processed using the above nine methods, and the structural similarity SSIM index, peak signal-to-noise ratio PNSR index and color difference formula CIEDE2000 index were calculated. The results are shown in Table 1.

[0135] Table 1

[0136] index Anwar Zhang RCP Li Wang WaterNet FunIEGAN CWR The present invention PSNR 17.578 18.020 20.541 17.526 20.534 22.203 22.102 18.775 23.767 SSIM 0.873 0.883 0.950 0.812 0.936 0.954 0.927 0.866 0.969 CIEDE2000 17.123 14.964 15.992 14.436 16.283 9.004 7.637 12.318 6.075

[0137] It can be seen from Table 1 that the PSNR, SSIM and color difference index values ​​of the method of the present invention are better than those of other algorithms, which are consistent with the subjective results, and can effectively correct the color cast and improve the image clarity.

[0138] Simulation test 4: 126 real underwater images were processed using the above nine methods, and the BRISQUE index of the reference-free image spatial quality evaluator and the UCIQE index of the underwater color image quality evaluation were calculated. The results are shown in Table 2.

[0139] Table 2

[0140] index Original image Anwar Zhang RCP Li Wang WaterNet FunIEGAN CWR The present invention BRISQUE 20.756 28.989 15.842 17.640 15.843 19.925 19.047 16.649 17.609 14.945 UCIQE 0.452 0.448 0.494 0.465 0.484 0.496 0.436 0.458 0.494 0.540

[0141] It can be seen from Table 2 that the BRISQUE and UCIQE values ​​of the method of the present invention are better than those of the other eight methods, which is consistent with the subjective results, indicating that the images processed by the present invention are clearer and more natural.

[0142] In summary, the method of the present invention is superior to the other eight methods in underwater image processing.

Claims

1. An end-to-end underwater image restoration method based on ambient light perception, comprising: 1) Constructing underwater image restoration network ECCN in Pytorch framework: 1a) Establish an ambient light perception network consisting of eleven convolutional layers, one upsampling layer, and one pooling layer; 1b) Establish a restoration main network including a shallow feature extraction module, a multi-scale encoding and decoding module and three feature integration and reconstruction modules; the structure of each module is as follows: The shallow feature extraction module is composed of a cascade of the first convolution layer → the second convolution layer → the third convolution layer; The multi-scale encoding and decoding module is composed of three branches connected in parallel, where: The first branch is: 1st pooling layer → 4th convolutional layer → 1st encoding unit → 2nd encoding unit → 1st decoding unit → 2nd decoding unit; The second branch is: the second pooling layer → the fifth convolutional layer → the third encoding unit → the fourth encoding unit → the third decoding unit → the fourth decoding unit; the third decoding unit is also connected to the second encoding unit; The third branch is: the 3rd pooling layer → the 6th convolutional layer → the 5th encoding unit → the 6th encoding unit → the 5th decoding unit → the 6th decoding unit; the 5th decoding unit is also connected to the 4th encoding unit; The first pooling layer, the second pooling layer, and the third pooling layer are all connected to the third convolutional layer; The encoding units are: I convolution layer → I residual module → II residual module → III residual module; The decoding units are: IV residual module → V residual module → VI residual module → bilinear interpolation → II convolution layer; The feature integration and reconstruction module is sequentially: upsampling layer → first convolution layer → second convolution layer → third convolution layer; The second decoding unit, the fourth decoding unit, and the sixth decoding unit of the multi-scale encoding and decoding module are respectively connected to the three feature reconstruction modules, and the third convolution layer of the shallow feature extraction module is simultaneously connected to the first convolution layer of the feature reconstruction module; The output of the main restoration network is the terminal output of the feature reconstruction module connected to the third branch; 1c) Cascading the ambient light sensing network and the restoration main network to form an underwater image restoration network; 2) Download the indoor dataset NYU containing depth information, the real single image dehazing dataset RESIDE containing depth information, and the paired underwater dataset EUVP synthesized based on the generative network, and filter, scale and crop these datasets in turn to obtain the indoor clear image set J, indoor depth atlas D, underwater clear image set U, and underwater degraded image set R1; 3) Construct training image sets for the ambient light perception network and the restoration subject network respectively: Set the ambient light value A and the blue channel transmittance T b , using the indoor clear image set J and the indoor depth atlas D, synthesize the underwater degraded image set R2, and pair the underwater degraded image set R2 with its corresponding ambient light value A as the training image set B of the ambient light perception network; Pair the indoor clear image set J with the underwater degraded image set R2, and add the underwater clear image set U and the underwater degraded image set R1 as the training image set C for restoring the main network; 4) Using the training image set B of the ambient light perception network, adopting the adaptive moment estimation algorithm, and aiming at minimizing the Euclidean distance loss value, the ambient light perception network is trained to obtain a trained ambient light perception network; 5) Using the training image set C of the restoration subject network and the trained ambient light estimation network, the adaptive moment estimation algorithm is used to train the restoration subject network with the goal of minimizing the weighted total loss value to obtain a trained restoration subject network; 6) Take an underwater image I to be restored c Input to the trained ambient light perception network and output the ambient light value A of the RGB three channels c ; 7) Set the ambient light value A c With underwater image I c Input them together into the trained restoration subject network, and output the restored clear image J c .

2. The method according to claim 1, characterized in that: The structural relationship and parameters of each layer of the ambient light sensing network in 1a) are as follows: The first convolution layer → the second convolution layer → the third convolution layer → the fourth convolution layer → the upsampling layer → the fifth convolution layer → the sixth convolution layer → the seventh convolution layer → the eighth convolution layer → the pooling layer → the ninth convolution layer → the tenth convolution layer → the eleventh convolution layer; The convolution kernel sizes of the first and second convolution layers are both 9*9; The convolution kernel sizes of the third, fourth, fifth, and sixth convolutional layers are all 7*7; The convolution kernel sizes of the seventh, eighth, ninth, and tenth convolutional layers are all 5*5; The convolution kernel size of the eleventh convolutional layer is 3*3; The stride of all convolutional layers is 1; all convolutional layers include convolution operations and ReLU activation function layers; The scaling factor of the upsampling layer is 2; The pooling layer has a scaling factor of 2.

3. According to the method of claim 1, the parameters of each layer of the restored main network are set as follows: The convolution kernel size of the first convolution layer is 7*7, the convolution kernel size of the second convolution layer is 5*5, and the convolution kernel size of the remaining convolution layers is 3*3; The convolution step size of all convolutional layers is 1, and all convolutional layers include convolution operations and ReLU activation function layers; The scaling factor of the upsampling layer is 2; The scaling factor of the first pooling layer is 8, the scaling factor of the second pooling layer is 4, and the scaling factor of the third pooling layer is 2; The bilinear interpolation operation is scaled by a factor of 2.

4. The method according to claim 1, characterized in that: In the above 3), the indoor clear image set J and the indoor depth atlas D are used to synthesize the underwater degraded image set R2, which is implemented as follows: 3a) Use the random function to randomly generate the red channel ambient light value A between 0.1 and 0.6 r , and then use the random function in A r -1.0 Randomly generate green channel ambient light value A g and the ambient light value A of the blue channel b ; 3b) Use the random function to randomly generate the blue channel transmittance parameter β between 0.5 and 2.5 b , the blue channel transmittance of each image is calculated using the indoor depth atlas D Red channel transmittance Green channel transmittance 3c) Based on the indoor clear image set J and the generated ambient light value A and transmittance T, the underwater degraded image set R2 = JT + A (1-T) is calculated.

5. The method according to claim 1, characterized in that The training of the ambient light perception network in 4) is implemented as follows: 4a) The Euclidean distance formula is used as the loss function LossB of the ambient light perception network, which is expressed as follows: Among them, ||·||2 is the operation of finding the two-norm of the matrix, c is the color channel of the input image, B c is the output of the ambient light perception network, B' c is the corresponding artificial synthetic ambient light; 4b) inputting the underwater degraded images in the training image set B of the ambient light perception network into the ambient light perception network in sequence, and outputting the estimated ambient light value A'; 4c) Substituting the set ambient light value A and the estimated ambient light value A' into the Euclidean distance formula at the same time, and calculating the Euclidean distance loss value between A and A'; 4d) Use the adaptive moment estimation algorithm to update the weights and bias values ​​of each convolution operation in the ambient light perception network, and repeat steps 4b)-4c) 10,000 times until the Euclidean distance loss value is minimized to obtain a trained ambient light perception network.

6. The method according to claim 1, characterized in that In the above 5), the restoration main network is trained as follows: 5a) The weighted total loss formula is used as the loss function LossC of the restored main network, which is expressed as follows: LossC=0.75×Loss t +0.25×Loss c Among them, Loss t is the boundary preservation loss, Loss c is the color cast correction loss, calculated as: Among them, ||·||2 is the operation of finding the matrix norm, |·| is the operation of finding the matrix norm, N is the number of pixels of the input image, J n , J n1 , J n2 are the terminal outputs of the three feature integration and reconstruction modules of the main network, J' n , J' n1 , J' n2 is the corresponding clear image, J' n1 J' is reduced by 16 times n , J' n2 J' is reduced by 64 times n ; r n represents the average value of the red channel, Δr n , Δb n , Δg n Represent the differences between the network output and the clear image in the red, green, and blue channels respectively; 5b) Input the underwater degraded images in the restoration subject network training set C into the trained ambient light estimation network in sequence, and output the estimated ambient light value A”; 5c) combining the underwater degraded image in the restoration subject network training set C with the estimated ambient light value A'', inputting the result into the restoration subject network, and outputting the estimated restoration image I'; 5d) Substituting the clear image I and the estimated restored image I' in the restoration subject network training set C into the weighted total loss function LossC at the same time, and calculating the weighted total loss value between the clear image I and the restored image I'; 5e) using an adaptive moment estimation algorithm to update the weights and bias values ​​of each convolution operation in the restoration subject network; 5f) Repeat 5c)-5e) for a total of 20,000 times until the weighted total loss value is minimized, and the trained restoration subject network is obtained.