Turbid underwater close-range observation image enhancement method

Abstract

The invention relates to the technical field of image enhancement methods, provides a turbid underwater close-range observation image enhancement method, and aims to perform turbid image enhancement based on image attenuation coefficient guidance so as to realize real-time close-range monitoring of construction operation in a turbid water area. Comprising the following steps that firstly, image samples of a training data set are constructed, the image samples are preprocessed to obtain training samples, the training samples comprise a source domain image sample and a target domain image sample, then a generator is constructed, and a target domain image sample is obtained; the generator comprises a parameter estimation module, an underwater image degradation model and an edge information guided detail restoration network, the parameter estimation module comprises a scene depth estimation network, an attenuation coefficient estimation network, a transmission image estimation network and a background light estimation network, and then the generator is trained through a training sample. After training is completed, an image enhancement model is obtained, and finally image enhancement is conducted on the turbid underwater close-range observation image through the image enhancement model.

Description

Technical Field

[0001] The invention relates to the technical field of image enhancement methods, and in particular to a turbid underwater close-range observation image enhancement method. Background Art

[0002] With the increasing depletion of land resources on the earth, the ocean has become a strategic choice for mankind to obtain key resources such as new energy and minerals. Underwater close-range observation equipment based on optical visual imaging is widely used in marine engineering. The development of underwater optical imaging technology is of great significance to building underwater work collaborative systems, optimizing task management, accelerating the overall progress of marine engineering projects, and improving the level of marine engineering intelligence.

[0003] However, due to the absorption and scattering of light by water media, underwater images usually have degradation problems such as low contrast, color distortion, and blurred details. Especially in most offshore areas, the water is relatively turbid, which further aggravates image degradation and increases the difficulty of underwater detection. More and more researchers are beginning to explore how to obtain high-quality underwater images. The current mainstream image enhancement algorithms can be divided into traditional algorithms and learning-based algorithms. Many traditional UIE methods are based on physical models, and their focus is on estimating all parameters in the physical model and inverting clear underwater images. These methods rely on prior knowledge, such as dark channel prior, maximum intensity prior, minimum information loss prior, and blur prior, etc. However, these prior knowledge are prone to failure in some cases, resulting in a decrease in the effect of image restoration. Based on data-driven methods, with the rapid development of deep models, learning-based methods are used. In more and more neighborhoods, researchers have achieved good results through traditional CNN models. As the Transformer model has shown excellent performance in many computer vision tasks, the Transformer model was introduced into the task and achieved good results. There are also some methods that are improved based on GAN to narrow the domain gap between underwater images and images in the air to improve restoration performance. In recent years, more and more researchers have begun to pay attention to the combination of the advantages of traditional methods and deep learning models. Cong et al. proposed PUGAN, combined with the simplified underwater image degradation model formula, and constructed a deep learning model based on the physical model. The deep model is used to estimate physical parameters such as transmission maps and background light, which further promotes the performance improvement of the image restoration model. The previous physical estimation-based methods generated inaccurate intermediate physical estimates.

[0004] However, in previous physical estimation-based methods, the intermediate transmission map and background light have no labels for learning. The inaccuracy of the generated intermediate physical estimation limits the performance of the deep learning model based on the physical model. Based on the existing data that is easily obtained in the air, a more accurate physical information estimation model is performed on the underwater image through physical parameter-guided learning and underwater image degradation model, and a detail-guided encoding and decoding structure is designed to achieve the restoration of underwater turbid images, so that the construction process can be monitored in real time and at close range in turbid waters. Therefore, we propose a visual enhancement method for underwater close-range observation of offshore wind power installation to solve the above technical problems. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a method for enhancing turbid water underwater close-range observation images, the technical purpose of which is to enhance turbid images based on the guidance of image attenuation coefficient, so that construction operations can be monitored in real time and at close range in turbid waters.

[0007] (II) Technical solution

[0008] To achieve the above object, the present invention provides the following technical solution: A method for enhancing turbid underwater close-range observation images, comprising the following steps: Step S1: constructing image samples of a training data set, and preprocessing the image samples to obtain training samples, wherein the training samples include source domain image samples and target domain image samples;

[0009] Step S2: constructing a generator, which includes a parameter estimation module, an underwater image degradation model, and a detail restoration network guided by edge information, wherein the parameter estimation module includes a scene depth estimation network, an attenuation coefficient estimation network, a transmission map estimation network, and a background light estimation network;

[0010] Step S3: training the generator using the training samples, and obtaining an image enhancement model after the training is completed;

[0011] Step S4: performing image enhancement on the turbid underwater close-range observation image using the image enhancement model.

[0012] Preferably, the step S1 comprises:

[0013] Step S11: extracting labeled image samples from the public underwater dataset as source domain dataset samples;

[0014] Step S12: extracting image samples from the public air RGBD dataset as target domain dataset samples;

[0015] Step S13: scaling the sizes of source domain dataset samples and target domain dataset samples to obtain source domain image samples and target domain image samples.

[0016] Preferably, step S3 comprises:

[0017] Step S31: Transform the source domain image sample X∈R B×3×H×W Input to the first generator G, get the intermediate estimated parameter output background light β1∈R B×3×H×W , transmission map t1∈R B×3×H×W And the final output of the generator

[0018] Step S32: Transform the target domain image sample Y∈R B×3×H×W , background light β1 and transmission map t1 are input to the second generator F, and the output is The parameter estimation module of the second generator does not work;

[0019] Step S33: and Input to the first discriminator and the second discriminator respectively, and calculate the generated adversarial loss L GAN_1 , L GAN_2 and L pix ;

[0020] Among them, L GAN_1 (G,D Y , X, Y) = E y～Pdata(y) [log(D Y (y))]+E x～Pdata(x) [log(1-D Y (G(x)))];

[0021] L GAN_2 (F,DX , X, Y) = E x～Pdata(x) [log(D X (x))]+E y～Pdata(y) [log(1-D X (F(y)))]; L GAN =L GAN_1 +L GAN_2 ; G represents the first generator; F represents the second generator; D Y represents the first discriminator; D X represents the second discriminator; CT X A clear reference image representing domain X;

[0022] Step S34: The output of the second generator Input to the first generator G, get the intermediate estimated parameter output depth map d, background light β2∈RB×3×H×W , transmission map t2∈R B×3×H×W And the generator finally outputs the output

[0023] Step S35: Calculate the total loss function L = L GAN1 +L GAN2 +L cycle +L pix +L d +L β +L t ,in L d =||d2-GT d ||1;L β =||β2-β1||1; L t =||t2-t1||1;||·||1 represents the L1 loss function;

[0024] Step S36: Perform back propagation according to the loss function L and update the weights;

[0025] Step S37: Repeat steps S31 to S36 until the trained first generator, i.e., the image enhancement model, is obtained.

[0026] Preferably, step S4 comprises:

[0027] Step S41: Transform the source domain image sample X∈R B×3×H×W Input the scene depth estimation module to obtain a rough relative depth map d∈R of the scene B×3×H×W as well as;

[0028] Step S42: Input the source domain image sample X and the attenuation coefficient estimation module at the same time to obtain the attenuation coefficient λ∈R B ×3×1×1 ;

[0029] Step S43: Multiply the depth map d and the attenuation coefficient λ and input them into the transmission map estimation module t∈R B×3×H×W ;

[0030] Step S44: Input the source domain image sample X into the background light estimation module to obtain the background light β∈R B×3×H×W ;

[0031] Step S45: The background light β, transmission map t and input X estimated by the network are used to calculate the underwater image degradation model. Get the initial restored image

[0032] Step S46: The initially restored image Input the detail enhancement network to obtain the final restored image

[0033] Preferably, in step S4, the scene depth estimation module, the background light estimation module and the transmitted light estimation module include 5 layers of 3×3 convolutions.

[0034] Preferably, in step S41, the attenuation coefficient estimation module specifically includes a 4×4 flat pooling operation with a step size of 4; 3 convolution operations with a step size of 2 and a convolution kernel size of 5×5; a global average pooling operation; and a dimensionality reduction operation with a step size of 1 and a convolution kernel size of 1×1.

[0035] Preferably, in step S46, the detail enhancement network consists of a convolution module, a deconvolution module, an edge extraction module and a connection module, wherein the edge extraction module AMEE performs wavelet transform on each layer of features and extracts vertical, horizontal and diagonal edge information through a high-pass filter.

[0036] (III) Beneficial effects

[0037] Compared with the prior art, the present invention provides a method for enhancing turbid underwater close-range observation images, which has the following beneficial effects:

[0038] 1. In the present invention, a more accurate physical information estimation model for underwater images is performed through physical parameter guided learning and underwater image degradation model.

[0039] 2. In the present invention, a detail-guided codec structure is designed to achieve the restoration of underwater turbid images, so that the construction process can be monitored in real time and at close range in turbid waters. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the structure of the encoding and decoding model of multi-physical feature estimation of the present invention;

[0041] Figure 2 is a schematic diagram of a detail restoration network guided by edge information of the present invention;

[0042] Figure 3 is a schematic diagram of an edge extraction module of the present invention;

[0043] Figure 4 Schematic diagram of the multi-task generative adversarial framework of the present invention. DETAILED DESCRIPTION

[0044] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings. The embodiments are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0045] Example

[0046] See also Figure 1-Figure 4 , Step S1: construct image samples of the training data set, and preprocess the image samples to obtain training samples, wherein the training samples include source domain image samples and target domain image samples;

[0047] Specifically, step S1 includes:

[0048] Step S11: extracting labeled image samples from the public underwater dataset as source domain dataset samples;

[0049] Step S12: extracting image samples from the public air RGBD dataset as target domain dataset samples;

[0050] Step S13: scaling the sizes of source domain dataset samples and target domain dataset samples to obtain source domain image samples and target domain image samples.

[0051] Step S2: constructing a generator, which includes a parameter estimation module, an underwater image degradation model, and a detail restoration network guided by edge information, wherein the parameter estimation module includes a scene depth estimation network, an attenuation coefficient estimation network, a transmission map estimation network, and a background light estimation network;

[0052] Step S3: Train the generator through training samples, and obtain an image enhancement model after training;

[0053] Specifically, step S3 includes:

[0054] Step S31: Transform the source domain image sample X∈R B×3×H×W Input to the first generator G, get the intermediate estimated parameter output background light β1∈R B×3×H×W , transmission map t1∈R B×3×H×W And the final output of the generator

[0055] Step S32: Transform the target domain image sample Y∈R B×3×H×W , background light β1 and transmission map t1 are input to the second generator F, and the output is The parameter estimation module of the second generator does not work;

[0056] Step S33: and Input to the first discriminator and the second discriminator respectively, and calculate the generated adversarial loss L GAN_1 , L GAN_2 and L pix ;

[0057] Among them, L GAN_1 (G,D Y , X, Y) = E y～Pdata(y) [log(DY (y))]+E x～Pdata(x) [log(1-D Y (G(x)))];

[0058] L GAN_2 (F,D X , X, Y) = E x～Pdata(x) [log(D X (x))]+E y～Pdata(y) [log(1-D X (F(y))]; L GAN =L GAN_1 +L GAN_2 ; G represents the first generator; F represents the second generator; D Y represents the first discriminator; D X represents the second discriminator; CT X A clear reference image representing domain X;

[0059] Step S34: The output of the second generator Input to the first generator G, get the intermediate estimated parameter output depth map d, background light β2∈R B×3×H×W , transmission map t2∈R B×3×H×W And the generator finally outputs the output

[0060] Step S35: Calculate the total loss function L = L GAN1 +L GAN2 +L cycle +L pix +L d +L β +L t ,in L d =||d2-GT d ||1;L β =||β2-β1||1; L t =||t2-t1||1;||·||1 represents the L1 loss function;

[0061] Step S36: Perform back propagation according to the loss function L and update the weights;

[0062] Step S37: Repeat steps S31 to S36 until the trained first generator, i.e., the image enhancement model, is obtained.

[0063] Step S4: Perform image enhancement on the turbid underwater close-range observation image using an image enhancement model.

[0064] Specifically, step S4 includes:

[0065] Step S41: Transform the source domain image sample X∈R B×3×H×W Input the scene depth estimation module to obtain a rough relative depth map d∈R of the scene B×3×H×W as well as;

[0066] In the embodiment of the present application, the scene depth estimation module includes 5 layers of 3×3 convolutions;

[0067] Step S42: Input the source domain image sample X and the attenuation coefficient estimation module at the same time to obtain the attenuation coefficient λ∈R B ×3×1×1 ;

[0068] In an embodiment of the present application, the attenuation coefficient estimation module specifically includes a 4×4 flat pooling operation with a step size of 4; 3 convolution operations with a step size of 2 and a convolution kernel size of 5×5; a global average pooling operation; and a dimensionality reduction operation with a step size of 1 and a convolution kernel size of 1×1.

[0069] Step S42: Multiply the depth map d and the attenuation coefficient λ and input them into the transmission map estimation module t∈R B×3×H×W ;

[0070] In the embodiment of the present application, the transmission map estimation module includes 5 layers of 3×3 convolutions;

[0071] Step S43: Input the source domain image sample X into the background light estimation module to obtain the background light β∈R B×3×H×W ;

[0072] In the embodiment of the present application, the background light estimation module includes 5 layers of 3×3 convolutions;

[0073] Step S44: The background light β, transmission map t and input X estimated by the network are used to calculate the underwater image degradation model. Get the initial restored image

[0074] Step S45: The initially restored image Input the detail enhancement network to obtain the final restored image

[0075] In the embodiment of the present application, the detail enhancement network consists of a convolution module, a deconvolution module, an edge extraction module and a connection module. The specific structure is as follows Figure 2 As shown in the figure, the edge extraction module AMEE performs wavelet transform on each layer of features and extracts vertical, horizontal and diagonal edge information through a high-pass filter. The specific structure is as follows Figure 3 shown.

[0076] The above specific implementation methods are only for illustrating the technical concept and structural features of the present invention, and the purpose is to enable relevant persons familiar with this technology to implement it. However, the above content does not limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for enhancing turbid underwater close-range observation images, characterized in that: The method comprises the following steps: Step S1: constructing image samples of a training data set, and preprocessing the image samples to obtain training samples, wherein the training samples include source domain image samples and target domain image samples; Step S2: constructing a generator, which includes a parameter estimation module, an underwater image degradation model, and a detail restoration network guided by edge information, wherein the parameter estimation module includes a scene depth estimation network, an attenuation coefficient estimation network, a transmission map estimation network, and a background light estimation network; Step S3: training the generator using the training samples, and obtaining an image enhancement model after the training is completed; Step S4: performing image enhancement on the turbid underwater close-range observation image using the image enhancement model.

2. The method for enhancing turbid underwater close-range observation images according to claim 1, characterized in that: The step S1 comprises: Step S11: extracting labeled image samples from the public underwater dataset as source domain dataset samples; Step S12: extracting image samples from the public air RGBD dataset as target domain dataset samples; Step S13: scaling the sizes of source domain dataset samples and target domain dataset samples to obtain source domain image samples and target domain image samples.

3. The method for enhancing turbid underwater close-range observation images according to claim 2, characterized in that: The step S3 comprises: Step S31: Transform the source domain image sample X∈R B×3×H×W Input to the first generator G, get the intermediate estimated parameter output background light β1∈R B×3×H×W , transmission map t1∈R B×3×H×W And the final output of the generator Step S32: Transform the target domain image sample Y∈R B×3×H×W , background light β1 and transmission map t1 are input to the second generator F, and the output is The parameter estimation module of the second generator does not work; Step S33: and Input to the first discriminator and the second discriminator respectively, and calculate the generated adversarial loss L GAN_1 , L GAN_2 and L pix ; where, L GAN_1 (G, D Y , X, Y) = E y～Pdata(y) [log(D Y (y))] + E x～Pdata(x) [log(1 - D Y (G(x)))]; L GAN_2 (F,D X ,X,Y)=E x～Pdata(x) [log(D X (x))]+E y～Pdata(y) [log(1-D X (F(y)))]; L GAN =L GAN_1 +L GAN_2 ; G represents the first generator; F represents the second generator; D Y represents the first discriminator; D X represents the second discriminator; GT X A clear reference image representing domain X; Step S34: The output of the second generator Input to the first generator G, get the intermediate estimated parameter output depth map d, background light β2∈R B×3×H×W , transmission map t2∈R B×3×H×W And the generator finally outputs the output Step S35: Calculate the total loss function L = L GAN1 +L GAN2 +L cycle +L pix +L d +L β +L t ,in L d =||d2-GT d ||1;L β =||β2-β1||1; L t =||t2-t1||1;||·||1 represents the L1 loss function; Step S36: Perform back propagation according to the loss function L and update the weights; Step S37: Repeat steps S31 to S36 until the trained first generator, i.e., the image enhancement model, is obtained.

4. The method for enhancing turbid underwater close-range observation images according to claim 3, characterized in that: The step S4 comprises: Step S41: Transform the source domain image sample X∈R B×3×H×W Input the scene depth estimation module to obtain a rough relative depth map d∈R of the scene B×3×H×W as well as; Step S42: Input the source domain image sample X into the attenuation coefficient estimation module at the same time to obtain the attenuation coefficient λ∈R B ×3×1×1 ; Step S43: Multiply the depth map d and the attenuation coefficient λ and input them into the transmission map estimation module t∈R B×3×H×W ; Step S44: Input the source domain image sample X into the background light estimation module to obtain the background light β∈R B×3×H×W ; Step S45: The background light β, transmission map t and input X estimated by the network are used to calculate the underwater image degradation model. Get the initial restored image Step S46: The initially restored image Input the detail enhancement network to obtain the final restored image 5. The method for enhancing turbid underwater close-range observation images according to claim 4, characterized in that: In step S4, the scene depth estimation module, the background light estimation module and the transmitted light estimation module include 5 layers of 3×3 convolutions.

6. The method for enhancing turbid underwater close-range observation images according to claim 5, characterized in that: In the step S41, the attenuation coefficient estimation module specifically includes a 4×4 flat pooling operation with a step size of 4; 3 convolution operations with a step size of 2 and a convolution kernel size of 5×5; a global average pooling operation; and a dimensionality reduction operation with a step size of 1 and a convolution kernel size of 1×1.

7. The method for enhancing turbid underwater close-range observation images according to claim 6, characterized in that: In step S46, the detail enhancement network consists of a convolution module, a deconvolution module, an edge extraction module and a connection module, wherein the edge extraction module AMEE performs wavelet transform on each layer of features and extracts vertical, horizontal and diagonal edge information through a high-pass filter.

Images