A color restoration method for severely color-distorted underwater images

Abstract

A color restoration method for severely color-distorted underwater images, belonging to the field of underwater image enhancement; Step 1: Estimation of the intermediate transmission image of the underwater image; Step 2: Use the underwater image and the intermediate transmission image as the input and label respectively to train the encoder-decoder network self-supervised network. After the training is completed, the fixed parameters of the encoder-decoder network self-supervised network are obtained; Step 3: Insert the fixed parameters of the encoder-decoder network self-supervised network in Step 2 into the underwater image color restoration network for training. The method of the present invention can eliminate the green hue in underwater images with severe green hues, restore the color of underwater images and conform to human visual senses; the underwater images used are more in line with the actual situation, and can ensure rapidity and accuracy, so it has more practical application significance and value.

Description

Technical Field

[0001] The present invention belongs to the field of underwater image enhancement, and relates to an underwater image processing method and an unsupervised learning method in deep learning technology. Specifically, it relates to a color restoration method for underwater images with severe color distortion. Background Art

[0002] Underwater image enhancement is an important foundation for other underwater vision tasks, such as underwater target recognition and underwater surrounding situation awareness. It has broad application prospects. In traditional deep learning methods, a large amount of paired data is required for training. However, it is difficult and costly to obtain labeled data for underwater images. In order to solve the need for labeled data in deep learning technology and meet the requirements of rapidity and accuracy, an underwater image enhancement method based on unsupervised learning must be able to restore the color of underwater images with severe color distortion without paired data training, and it is necessary to ensure that the restored underwater images conform to the visual perception of the human eye. In order to ensure the accuracy of the restored image color, an encoder-decoder neural network is pre-trained using an underwater image and an intermediate transmission image estimated by its red channel prior, and the parameters of this neural network are fixed to provide intermediate transmission image attention to the unsupervised network in a non-local manner during underwater color restoration. In order to improve the brightness and contrast of underwater images, style transfer loss and degenerate structural similarity loss are introduced during the training of the unsupervised underwater image color restoration network. Summary of the Invention

[0003] The purpose of the present invention is to overcome the need for paired training data in deep learning technology, rely on the powerful learning ability of the neural network, combine intermediate transmission image attention, and fuse style transfer loss and degenerate structural similarity loss to achieve accurate color restoration of underwater images with severe color distortion.

[0004] The present invention uses the intermediate transmission image of the underwater image to provide attention to the unsupervised network, thereby being able to more accurately restore the color of the underwater image. This method can well restore the color of underwater images from underwater pictures with severe green tones. Through learning by Cycle Generative Adversarial Networks (cycleGAN), accurate restoration of the color of underwater images is achieved. Through this method, the dependence on paired training data in deep learning technology can be eliminated, and rapidity and accuracy can also be ensured. It meets the requirements for tasks such as underwater target recognition using underwater images.

[0005] The purpose of the present invention is achieved as follows:

[0006] A color restoration method for underwater images with severe color distortion, comprising the following steps:

[0007] Step 1: Estimate the intermediate transmission image of the underwater image;

[0008] Step 2: Use the underwater image and the intermediate transmission image as input and label respectively to train the encoder-decoder network self-supervised network. After training is completed, obtain the fixed parameters of the encoder-decoder network self-supervised network;

[0009] Step 3: Insert the fixed parameters of the encoder-decoder network self-supervised network in Step 2 into the underwater image color restoration network for training;

[0010] The underwater color restoration network includes two generators and four discriminators: Generator G is responsible for generating an image of the land from the underwater image. The upper network of Generator G is the encoder-decoder network self-supervised network with fixed parameters, and the lower network is the encoder-decoder network self-supervised network with fixed parameters with three layers of depthwise separable convolutions added in the middle layer; Generator F is responsible for generating an underwater image from the land image. Name the discriminator responsible for identifying whether it is a real land image and its auxiliary discriminator as D Y , and name the discriminator responsible for identifying whether it is a real underwater image and its auxiliary discriminator as D X , and the training process is as follows:

[0011] 3.1: Training of the discriminator;

[0012] First, fix the parameters of the generator, open the parameters of the discriminator. The underwater image and the land image respectively pass through G and F to generate a fake land image and a fake underwater image, and then input them to D Y and D X respectively to obtain the identification results. Take the absolute value loss of these results with the label 'fake' to obtain the 'fake' loss value of the discriminator; then directly input the underwater image and the land image into D X and D Y to obtain the identification results. Take the absolute value loss of these results with the label 'true' to obtain the 'true' loss value of the discriminator; add the 'true' loss value and the 'fake' loss value to obtain the total loss value of the discriminator, and use the total loss value to update the parameters of the discriminator. The training of the discriminator is completed;

[0013] 3.2: Training of the generator:

[0014] Fix the parameters of the discriminator, open the parameters of the generator. Input the underwater image and the land image into G and F to generate a fake land image and a fake underwater image, and then input them back to F and G to obtain their restored images. Calculate the losses of the generators respectively through the following formula using the underwater image, the land image, the fake land image, the fake underwater image and their restored images, and then update the parameters of the two generators to complete the training of Generator G;

[0015]

[0016] Among them, λ1 = 1, λ2 = 10, λ3 = 9, λ4 = 6e - 9, λ5 = 1e - 6, and λ6 = 1e - 7 are the weight coefficients of each loss respectively;

[0017] The calculation formula of the degradation structural similarity loss is as follows:

[0018]

[0019] In the formula, x and y respectively represent the underwater image in the source domain and the land image in the target domain, G and F respectively represent the forward generator and the inverse generator, and μ and σ represent the mean and standard deviation of the image;

[0020] 3.3: Continuously loop steps 3.1 and 3.2 until the discriminator can no longer distinguish whether the input image is real or fake, and the underwater image color restoration network training is completed;

[0021] Step 4: Extract the generator G in the underwater image color restoration network and it can be applied to the restoration of the underwater image color.

[0022] The specific content of step one is as follows: The estimation of the intermediate transmission image of the underwater image is obtained by the following formula:

[0023]

[0024]

[0025] In the formula, D(x) represents the subtraction of the maximum intensity of the red channel from the maximum intensities of the blue and green channels, I c (x) represents the pixel value of channel c ∈ {r, g, b}, and Ω is a small block in the image, which is taken as 15 during calculation.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] The method of the present invention can remove the green tint in the underwater image with a severe green tint, restore the color of the underwater image and conform to the human visual sense;

[0028] The underwater image used in the method of the present invention is more in line with the actual situation, and can ensure rapidity and accuracy, so it has more practical significance and value in actual application;

[0029] The intermediate transmission image attention and degradation structural similarity loss used in the method of the present invention make the underwater image after color restoration closer to the land image, providing a clearer and more accurate image for underwater tasks that rely on images. Description of the Drawings

[0030] Figure 1 Schematic diagram of the encoding and decoding network structure;

[0031] Figure 2 Schematic diagram of the generator structure in the underwater image restoration network;

[0032] Figure 3 Schematic diagram of non-local attention;

[0033] Figure 4a -b Schematic diagram of the discriminator structure. DETAILED DESCRIPTION

[0034] The present invention will be described in further detail below with reference to the accompanying drawings and specific embodiments.

[0035] This method first estimates the intermediate transmission image of the underwater image using a red channel prior. This intermediate transmission image contains important information about underwater image attenuation. When light propagates through water, red light, due to its longest wavelength, attenuates the fastest, followed by green and blue light. Therefore, underwater images often appear green or blue. After obtaining the underwater image and its intermediate transmission image, an encoder-decoder neural network is trained. The encoder-decoder neural network is fixed in parameters and used to provide attention to the intermediate transmission image in an unsupervised underwater image color restoration network. During unsupervised network training, a style transfer loss and a degraded structural similarity loss are applied to improve image brightness and contrast.

[0036] Due to factors such as the time, location, temperature, water depth, and turbidity of the image, underwater images exhibit varying degrees of color distortion, low contrast, blurred details, and insufficient illumination. This limits the applicability of underwater image color restoration methods designed based on physical models or conventional image enhancement methods. Conventional deep learning-based underwater image color restoration methods require paired training data. To address this issue, unsupervised learning-based recurrent generative adversarial networks (GANs) have been introduced. They were first used for image style transfer tasks, transforming source images into the style of target images while preserving details and other features. The advantage of GANs is that they can achieve image transfer without the need for paired training data, bringing new insights to tasks such as image denoising, image deraining, image dehazing, and underwater image enhancement. A GAN is a cycle formed by two GANs. Its recurrent network structure allows source images transformed to the target domain to be fully transformed back to the source domain.

[0037] The intermediate transmission image can reflect important information about the attenuation of underwater images. Most underwater image enhancement methods based on the physical model of underwater imaging use various methods to obtain the intermediate transmission image, and then enhance the underwater image based on the obtained intermediate transmission image. The red-channel prior is an underwater image enhancement method inspired by the dark-channel prior for haze removal in land images. In underwater images, the attenuation rate of the red channel is greater than that of the green and blue channels. This results in a significantly smaller number of red pixels than green and blue pixels in underwater images, and the distribution of red pixels is also very concentrated in the histogram. The attenuation information contained in the intermediate transmission image indicates how to restore the underwater image.

[0038] Self-supervised learning is a deep learning method that has emerged in recent years. Its purpose is to fully exploit the information in the data itself. Self-supervised learning has been widely applied in fields such as natural language processing and computer vision. When there is a large amount of training data but only a small number of labels, using self-supervised learning for pre-training can effectively improve the performance of neural networks in real tasks. Common self-supervised tasks in the field of computer vision include image coloring, image puzzle, image inpainting, and some pseudo-label tasks generated in a hard-coded form. The intermediate transmission image of underwater images can be obtained using the red-channel prior. Self-supervision is performed on the underwater image and its intermediate transmission image in the encoder-decoder network. At this time, the encoder-decoder network performs a simple image generation task. Therefore, the L1 loss is used to limit the difference between the intermediate transmission image generated by the encoder-decoder network and the intermediate transmission image obtained by the red-channel prior. To ensure that underwater image information is not lost as much as possible during the calculation of the neural network, an encoder-decoder network with deep residuals and skip connections is designed. The addition of residual modules and skip connections makes the network have the characteristics of fast convergence, small error, and high accuracy. Adding an Upsampling layer to the cycle generative adversarial network will cause network instability, and the network is extremely prone to crashing during training. The generator in the underwater image color restoration network should be basically consistent with the encoder-decoder network structure, so a transposed convolutional layer is used to upsample the tensor for the network.

[0039] The underwater image color restoration cycle generative adversarial network follows the basic cycle generative adversarial network structure, including two generators and four discriminators. The structure of the generator is basically consistent with the encoder-decoder network. To improve the detail quality of the generated images, an auxiliary discriminator is added. Compared with the main discriminator, the auxiliary discriminator reduces the size of the perceptual field of view and only gives a true / false judgment on a small block of the image.

[0040] Due to the low brightness and contrast of underwater images, style transfer loss and degenerate structural similarity loss are used to assist in the training of the underwater image color restoration network. Style transfer loss was first proposed in the image style transfer task. It believes that at the high level of the VGG19 network, the image retains high-level features such as texture. By reducing the difference between the Gram matrices of these high-level features, the generated image can have the style features of the target domain image. The structural similarity index is an important indicator for measuring the similarity of image brightness, contrast, and structure. The structural similarity loss can make the gap between two images smaller and has a better convergence effect than the mean square error loss. For the underwater image restoration task, we only hope that the underwater image has the brightness and contrast features of the target domain image. Therefore, the degenerate structural similarity loss is proposed, which only focuses on the difference between image brightness and contrast.

[0041] The main steps of this method include: estimating the intermediate transmission image of the underwater image, self-supervised training of the encoding-decoding network, designing the degenerate structural similarity loss function, and training the underwater image color restoration network. Specifically as follows:

[0042] 1. Estimating the intermediate transmission image of the underwater image

[0043] Under the assumption that the ambient light is known, in the underwater environment, since red light attenuates the fastest, in order to determine the maximum difference between the red channel and the blue and green channels, the maximum intensity of the red channel can be subtracted from the maximum intensities of the blue and green channels, as shown in Equation 1

[0044]

[0045] where I c (x) represents the pixel value of channel c ∈ {r, g, b}, and Ω is a small block in the image, which is taken as 15 during calculation.

[0046] The intermediate transmission image can then be calculated by Equation 2

[0047]

[0048] Since the intermediate transmission image is calculated by taking a small block of the image, there are halos and blocky shadows in the obtained intermediate transmission image. Guided filtering is performed on the obtained intermediate transmission image to smooth it.

[0049] 2. Self-supervised training of the encoding-decoding network

[0050] After obtaining the above intermediate transmission image obtained by hard coding, it is used as label data for the self-supervised training of the encoding-decoding network. The encoding-decoding network is a deep residual structure (see Appendix Figure 1) To preserve information as much as possible and accelerate convergence, skip connections are added. Transposed convolution is used as the upsampling layer to increase network stability. The L1 loss function is used to narrow the gap between the generated image and the intermediate transmitted image.

[0051] 3. Design of Degraded Structural Similarity Loss Function

[0052] The structural similarity loss function can measure the difference between two images in terms of brightness, contrast, and structural similarity degree, with a value ranging from 0 to 1, where 1 represents complete identity and 0 represents complete difference. Based on the structural similarity loss function, its measurement of the structural similarity degree is ignored, and only the brightness and contrast of the images are concerned because our purpose is to make the enhanced underwater image have good brightness and contrast of the target domain image. The calculation of the degraded structural similarity loss is shown in Equation 3:

[0053]

[0054] In the formula, x and y represent the underwater image in the source domain and the land image in the target domain respectively, G and F represent the forward generator and the backward generator respectively, and μ and σ represent the mean and standard deviation of the image.

[0055] 4. Training of Underwater Image Color Restoration Network

[0056] The structure of the generator of the underwater image restoration network is basically the same as that of the encoder-decoder network (see Appendix Figure 2 ). Three layers of depthwise separable convolutions are added to the middle layer of the lower network of the underwater image restoration network generator to improve the independent processing ability of each channel. The upper network in the generator is an encoder-decoder network with fixed parameters after being trained with the underwater image and the intermediate transmitted image, which is used to provide the intermediate transmitted image attention to each layer of the lower network in a non-local manner (see Appendix Figure 3 ). The non-local method can generate attention globally for the image, but the disadvantage is that it occupies a large amount of computing resources. Therefore, only three non-local attentions are introduced. The structure of the discriminator adopts a fully convolutional structure (see Appendix Figure 4a -b). The convolutional layer of the main discriminator is more than that of the auxiliary discriminator because the receptive field of the auxiliary discriminator is relatively small, only a small piece in the image. A normalization layer is added to the auxiliary discriminator to improve its performance.

[0057] The loss function of the entire underwater image color restoration network includes adversarial loss, cycle consistency loss, style transfer loss, degraded structural similarity loss, and total variation loss, as shown in Equation 4:

[0058]

[0059] Among them, λ1 = 1, λ2 = 10, λ3 = 9, λ4 = 6e-9, λ5 = 1e-6, λ6 = 1e-7 are the weight coefficients of each loss respectively.

[0060] Before training the overall network, the datasets to be prepared are the underwater image dataset and the land image dataset. The intermediate transmission images are generated for the underwater image dataset by the method described in step 1. The underwater image and the intermediate transmission image at this time are used as the input and label respectively for the self-supervised network training described in step 2. After the self-supervised encoding and decoding network training is completed, the parameters are fixed and inserted into the designed underwater image color restoration network. At this time, the underwater image and the land image are used as the input and label respectively for the underwater image color restoration network training described in step 4. The generator parameters and discriminator parameters of this network are updated alternately. For the convenience of description, in the underwater image color restoration network, the image generator responsible for generating land images from underwater images is named G, the generator responsible for generating underwater images from land images is named F, the discriminator responsible for identifying whether it is a real land image and its auxiliary discriminator are named D Y , and the discriminator responsible for identifying whether it is a real underwater image and its auxiliary discriminator are named D X . Then the whole training process is as follows: First, fix the generator parameters and open the discriminator parameters. The underwater image and the land image pass through G and F respectively to generate fake land images and fake underwater images, and then they are input to D Y and D X respectively to obtain the identification results. The absolute value loss is calculated between these results and the label 'fake' to obtain the 'fake' loss value of the discriminator. Then, the underwater image and the land image are directly input into D X and D Y to obtain the identification results. The absolute value loss is calculated between these results and the label 'true' to obtain the 'true' loss value of the discriminator. The 'true' loss value and the 'fake' loss value are added together to obtain the total loss value of the discriminator, and the parameters of the discriminator are updated using the total loss value. At this time, the discriminator training is completed. Fix the discriminator parameters and open the generator parameters. Input the underwater image and the land image into G and F to generate fake land images and fake underwater images, and then input them back into F and G to obtain their restored images. The losses of the generators are calculated respectively through formula 4 using the underwater image, the land image, the fake land image, the fake underwater image, and their restored images, and then the parameters of the two generators are updated, which completes the training of the generators. After continuously looping through the above training steps until the accuracy of the discriminator is about 50%, at this time, the discriminator cannot distinguish whether the input image is real or fake, and the underwater image color restoration network training is completed. Extract the generator G in the network and it can be applied to the restoration of the underwater image color.

Claims

1. A color restoration method for severely color-distorted underwater images, characterized in that, Including the following steps: Step 1: Estimation of the intermediate transmission image of the underwater image; Step 2: Using the underwater image and the intermediate transmission image as input and label respectively to train the encoder-decoder network self-supervised network. After training is completed, the fixed parameters of the encoder-decoder network self-supervised network are obtained; Step 3: Inserting the fixed parameters of the encoder-decoder network self-supervised network in Step 2 into the underwater image color restoration network for training; The underwater color restoration network includes two generators and four discriminators: Generator G is responsible for generating land images from underwater images. The upper network of Generator G is a self-supervised network of an encoder-decoder network with fixed parameters, and the lower network is a self-supervised network of an encoder-decoder network with fixed parameters with three layers of depthwise separable convolutions added to the middle layer; Generator F is responsible for generating underwater images from land images. The discriminator responsible for identifying whether it is a real land image and its auxiliary discriminator are named D Y , and the discriminator responsible for identifying whether it is a real underwater image and its auxiliary discriminator are named D X , and the training process is as follows: Step 3.1: Training of the discriminator; First, fix the generator parameters and open the discriminator parameters. The underwater image and the land image respectively pass through G and F to generate a fake land image and a fake underwater image, and then input them into D. Y and D X Separate identification results are obtained. Take the absolute value loss of these results with the label 'fake' to get the 'fake' loss value of the discriminator. Then directly input the underwater image and the land image into D X and D Y to obtain the identification result. Take the absolute value loss of this result with the label 'true' to get the 'true' loss value of the discriminator. Add the 'true' loss value and the 'fake' loss value to get the total loss value of the discriminator. Use the total loss value to update the parameters of the discriminator. The training of the discriminator is completed. Step 3.2: Training of the generator: Fixing the discriminator parameters and opening the generator parameters. Inputting the underwater image and the land image into G and F to generate fake land images and fake underwater images, and then inputting them back into F and G to obtain their restored images. Using the underwater image, land image, fake land image, fake underwater image and their restored images to calculate the losses of the generator respectively through the following formula, and then updating the parameters of the two generators, thus completing the training of the generator G; Among them, λ1 = 1, λ2 = 10, λ3 = 9, λ4 = 6e-9, λ5 = 1e-6, λ6 = 1e-7 are the weight coefficients of each loss respectively; The calculation formula of the degraded structural similarity loss is as follows: In the formula, x and y represent the underwater image in the source domain and the land image in the target domain respectively, G and F represent the forward generator and the reverse generator respectively, and μ and σ represent the mean and standard deviation of the image; Step 3.3: Continuously looping Step 3.1 and 3.2 until the discriminator can no longer distinguish whether the input image is real or fake, and the underwater image color restoration network training is completed; Step 4: Extracting the generator G in the underwater image color restoration network can be applied to the restoration of the underwater image color.

2. The color restoration method for severely color-distorted underwater images according to claim 1, wherein, The specific content of Step 1 is: The estimation of the intermediate transmission image of the underwater image is obtained by the following formula: where D(x) represents the subtraction of the maximum intensity of the red channel from the maximum intensities of the blue and green channels, and I c (x) represents the pixel value of channel c ∈ {r, g, b}, and Ω is a small patch in the image, which is taken as 15 during calculation.

Images