Underwater image restoration method based on polarization scattering and deep learning

Abstract

The invention discloses an underwater image restoration method based on polarization scattering and deep learning, which is characterized in that background region scattered light of an underwater image is removed based on a polarization scattering physical model, the color of the image is restored by utilizing machine learning, and the images of preliminary restoration and secondary restoration are fused to obtain a final image. And a final underwater image recovery image is obtained. Compared with the prior art, the method has the advantages that a mixed underwater image recovery algorithm is formed by combining the advantages of a polarization scattering model and deep learning on the basis of machine enhanced vision, the fidelity is effectively enhanced, detail information is recovered, and the problem of accuracy pixel-level matching of machine learning data set acquisition is effectively solved; the comprehensive application problems of large color loss, high scene demand, high recovery upper limit of machine learning and difficult model training of a physical model are solved, the method is simple and convenient, the use effect is good, and the method has good application prospects and commercial development value.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image restoration, and in particular to an underwater image restoration method based on polarization scattering and deep learning. Background Art

[0002] Underwater optical imaging is now widely used in the fields of marine resource exploration, underwater archaeology, maritime rescue, and marine life fishing. Due to the refractive index, scattering, and absorption characteristics of water, underwater images usually have problems such as blurring, fading, and reduced contrast. The details of underwater images are easily lost, which makes it difficult to identify and monitor underwater targets. Due to the scattering and absorption effects caused by turbid media, underwater images often lack detailed information, have more noise, and have low contrast. Enhancing underwater vision is a very challenging task.

[0003] In underwater optical imaging, due to the turbidity of water in real life, there are a large number of suspended particles, and light has a significant scattering effect when propagating underwater. Underwater images are usually affected by scattering, absorption and dispersion, resulting in reduced image quality, blurred details and color distortion. For shooting instruments that receive natural light, most of the natural light is scattered by suspended particles in the water, so it cannot be well captured by the instrument; for detection that actively gives a light source, most of the light is scattered by particles in the medium, so more of the backscattering of these particles is obtained. Therefore, the captured image is not clear, there will be a lot of noise or insufficient image details of the target object, etc. Due to the absorption and scattering of light by water and underwater turbid substances, underwater images often have serious color deviation, low contrast, blurred details and other problems during the acquisition process. These problems make it difficult for underwater images to meet application requirements and limit the development of underwater detection and other technologies. Underwater image restoration technology has a wide range of application needs in many fields such as marine military, marine environmental protection, marine engineering, underwater archaeology, and underwater fishing robots. Domestic and foreign scholars have conducted extensive and in-depth research in the field of underwater image restoration and enhancement. The current research methods mainly include the following categories:

[0004] 1) Traditional filtering methods: Early underwater image restoration methods did use traditional filtering techniques, such as average filtering, median filtering, and Gaussian filtering. These methods reduce noise and blur by smoothing the pixel values ​​in the image, but may also cause loss of image details. Although mean filtering, median filtering, and Gaussian filtering can effectively smooth images and reduce noise, they often cause blurring and loss of image details. While smoothing noise, these methods also smooth edges and details in the image and cannot handle complex noise. Traditional filtering methods are not effective in dealing with complex and non-uniform noise. Traditional filtering methods have difficulty dealing with color noise and scattering effects that are common in underwater images. Traditional filtering methods rely on simple statistical characteristics of local pixels and lack an understanding of the global structure and semantic information of the image, resulting in a failure to distinguish between noise and details when smoothing noise.

[0005] 2) Histogram equalization: Histogram equalization is a classic image enhancement technique and is also widely used in underwater image restoration. It stretches the image histogram, enhances the image contrast, and makes the grayscale distribution of the image more uniform, thereby improving the visual effect of the image. The disadvantage is that histogram equalization may cause over-enhancement of the image, produce excessive contrast and noise, especially in the case of uneven lighting. In color image processing, performing histogram equalization on each color channel separately may cause color distortion and unnatural color changes. Histogram equalization improves contrast by globally adjusting the grayscale distribution of the image, but it does not take into account the local characteristics and color information of the image, which may lead to over-enhancement and color distortion.

[0006] 3) Image defogging algorithm: Dark channel prior (DCP) was originally proposed for defogging of outdoor scenes. It assumes that the brightness of an object in a natural environment is very small in at least one color component, so it takes the minimum value of a small local area of ​​the foggy image as the superposition component of the fog, and then restores the foggy image based on the atmospheric scattering model. Kaiming He et al. proposed the dark channel prior (DCP) algorithm. In a fog-free image, the color of each local area in the image is relatively bright. According to the principle of the three primary colors of the image, there is at least one color channel with a very low value in each area. The image composed of the minimum values ​​of each channel in the image is called a dark channel image. If there is fog in the image, since the fog is grayish white, the values ​​of each channel are high, that is, the value in the dark channel is also high. Therefore, the concentration of the fog can be estimated by the grayish white degree of the dark channel image, so as to perform defogging. Due to the scattering and absorption characteristics of the underwater environment, underwater images often appear blurred and foggy. Drawing on the methods of defogging images in the atmosphere, researchers have proposed a variety of underwater image defogging algorithms, such as the Dark Channel Prior and Color Attenuation Prior. These methods aim to eliminate scattered light in images and restore image clarity and details. Prior assumptions have certain limitations. For example, methods such as Dark Channel Prior (DCP) and Color Attenuation Prior (CAP) rely on specific prior assumptions, which may not hold in some cases, resulting in poor defogging effects.

[0007] 4) Physics-based methods: Physics-based methods restore underwater images by establishing an optical propagation model and taking into account factors such as light scattering, absorption and reflection. Typical methods include algorithms based on simplified light propagation models (such as the Jaffe-McGlamery model) and algorithms based on complex light propagation models (such as the radiation transfer equation). These methods can more accurately simulate the underwater imaging process, thereby achieving higher quality underwater image restoration. However, accurate physical models such as the radiation transfer equation (RTE) are very complex, and solving these models usually requires a lot of computing resources and time. Moreover, these methods rely on multiple environmental parameters (such as water quality, light source characteristics, etc.), which are often difficult to obtain, and parameter estimation errors will affect the restoration effect. The reason is that the physical model needs to accurately describe the propagation process of light in water, which involves complex mathematical calculations and accurate measurements of a large number of environmental parameters. In practical applications, the uncertainty and variability of environmental parameters limit the adaptability and accuracy of the model.

[0008] 5) Deep learning methods: In recent years, deep learning methods have made significant progress in the field of underwater image restoration. Researchers have applied deep neural network models such as Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) to solve the problem of underwater image restoration. These methods can learn the mapping relationship of image restoration from a large amount of underwater image data, but it is difficult to train a network with sufficient generalization ability, and the authenticity of the restored images and the customization of the training network data set are also difficult. Deep learning methods require a large amount of training data. The quality of the training data set directly determines the quality of the network's restored images. It is difficult to train a network with sufficient generalization ability. The diversity and representativeness of the training data directly affect the generalization ability of the model, and the authenticity of the information restored by the deep network cannot be guaranteed. Deep learning relies on big data and high-performance computing, and its performance depends largely on the quality and quantity of training data. Due to the complexity and variability of the underwater environment, it is challenging to obtain a data set that comprehensively covers various situations.

[0009] Polarization imaging has been widely used, such as using polarized light for navigation using the sky polarization model. In underwater dark light environments, more than 70 species of marine organisms use polarization vision. The polarization of light and the scattering of underwater light are important characteristics of underwater light propagation. Establishing a physical model of underwater light propagation to obtain the desired target light information and reduce the useless scattered information in the water is an important method for underwater image restoration. Restoring underwater image details by reducing background scattered light has practical value in the field of underwater target recognition and monitoring. Image restoration technology based on polarization technology and scattering principles is of great significance in the field of computer vision and image processing. It can improve image quality and visualization, restore hidden details, and improve the accuracy of image processing and analysis. In the field of underwater image restoration, there are few studies combining polarization imaging with deep learning, but they have all proved their effectiveness. Therefore, it is also very important to study how to combine polarization imaging with deep learning to achieve good underwater image restoration effects.

[0010] The problems with the existing technology are: the physical model needs to solve complex mathematical equations, especially when it comes to solving inverse problems, the amount of calculation is large and the calculation complexity is high. In real-time applications, it is difficult to meet the needs of efficient processing, especially when processing high-resolution images. For example, the DCP method requires complex calculations for each pixel, which significantly increases the calculation time. The generalization ability of the model is weak, the adaptability is poor, and the restoration effect is unstable under different water quality conditions. For example, the performance of the inverse scattering model in turbid waters and clear waters is quite different. The physical model has large color loss and high scene requirements. Machine learning has a high recovery upper limit and difficult model training. Although the polarization scattering model enhances the image restoration details, the color will be quite different from the real image, which is a major problem for some application fields with color restoration requirements. When deep learning restores images, it restores the image color to be more natural and close to the real image, but there will be a slight secondary loss of details. Summary of the invention

[0011] The purpose of the present invention is to provide an underwater image restoration method based on polarization scattering and deep learning in response to the deficiencies of the prior art. The method adopts a polarization scattering model combined with a method based on deep learning, establishes a scattering model of light propagating in water, reversely derives the original image, uses transmittance to restore the image, and introduces machine learning to further optimize the polarization restored photos to improve the effect of underwater image restoration. The method removes scattered light from the background area of ​​underwater images based on the polarization scattering physical model, uses machine learning to restore the color of the picture, and finally outputs the final image after the initial restoration and secondary restoration of the image fusion, thereby achieving the best restoration effect, greatly improving the quality of underwater image restoration, and is particularly suitable for underwater robots, submersibles and other fields that need to improve underwater vision, and has good application prospects and commercial development value.

[0012] The specific technical scheme for achieving the purpose of the present invention is: an underwater image restoration method based on polarization scattering and deep learning, which is characterized by using a polarization scattering physical model to remove scattered light from the background area of ​​the underwater image, using machine learning to restore the color of the picture, and fusing the initial restoration and secondary restoration images to obtain the final underwater image restoration image, which specifically includes the following steps:

[0013] Step 1: Use the polarization camera module to acquire images of the scene target in the underwater image in three different polarization directions, and obtain polarization plate images of 0 degree, 45 degree and 90 degree respectively.

[0014] Step 2: Use the MATLAB program to calculate the three polarizer images using the descattering physical model to obtain images with restored details;

[0015] Step 3: Perform deep learning training on the Pix2pix generative adversarial network. The dataset uses images that have been initially restored by the polarization scattering physical model. The trained deep learning model is used to perform secondary restoration on the images that have been initially restored by the polarization scattering physical model of the scene target.

[0016] Step 4: Perform image fusion calculation on the image restored by deep learning again with the image initially restored by the polarization scattering physical model to obtain the final underwater image restoration image.

[0017] Compared with the prior art, the present invention is based on machine enhanced vision, and by combining the respective advantages of the polarization scattering model and deep learning, a hybrid underwater image restoration algorithm is formed, which effectively enhances its fidelity and restores detail information, and effectively solves the problem of obtaining accurate pixel-level matching of machine learning data sets, as well as the comprehensive application problems of large color loss of physical models, high scene requirements, high restoration upper limit of machine learning, and difficult model training. The method is simple, the use effect is good, and it has good application prospects and commercial development value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the present invention;

[0019] Figure 2 This is the flow chart of polarization scattering model;

[0020] Figure 3 This is a flowchart of underwater blurred image restoration based on Pix2Pix GANs network;

[0021] Figure 4 This is the loss function diagram of the initial model generator and discriminator;

[0022] Figure 5 This is the loss function diagram of the model generator and discriminator after parameter adjustment;

[0023] Figure 6 A comparison chart of the underwater image restoration effects. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below through specific implementations.

[0025] Example 1

[0026] See also Figure 1 The present invention performs underwater image restoration according to the following steps:

[0027] Step 1: Use the polarization scattering model to obtain polarized images of the scene targets in the underwater image at three different polarization directions, namely 0 degree, 45 degree and 90 degree.

[0028] Step 2: Use the MATLAB program to calculate the three polarizer images using the descattering physical model to obtain images with restored details.

[0029] The specific principle and process of step 2 using the MATLAB program to calculate the three images using the descattering physical model to obtain the image after detail restoration are as follows:

[0030] (1) Currently, most underwater polarization imaging methods assume that underwater backscatter is polarized, while the reflected light of the target is unpolarized. In fact, this assumption is unreliable because both backscatter and reflected light are polarized. The light intensity received by the detector mainly comes from two sources. The forward scattering signal is ignored because the forward scattering is extremely weak in natural scenes. The imaging model can be expressed by the following formula (a):

[0031] I(x,y)=D(x,y)+B(x,y)(a).

[0032] Among them, I(x,y) represents the total light intensity; D(x,y) represents the information of the target light received by the optical sensor, that is, the reflected light of the target; B(x,y) represents backscattering, which is generated by the scattering of the medium and other suspended particles in the scene, which is called the "background area". Problems such as increased noise, shallow depth of field and decreased clarity are mainly caused by B(x,y), so in order to restore the underwater degraded image, it is necessary to descatter the received light, that is, to reduce the impact of the background area on the target light D(x,y) information.

[0033] (2) A differential image can be obtained based on the Stokes vector. By rotating the polarizer, polarization images at three different angles, 0°, 45°, and 90°, can be obtained. More information can be obtained by including them in the Stokes vector representation shown in the following formula (b):

[0034]

[0035] Among them, M0 is the total intensity of the scene light, M1 represents the intensity difference between the horizontal and vertical polarization components, M2 represents the intensity difference when the polarizer is rotated 45° and -45° along the horizontal axis, and M3 represents the intensity difference between the left and right circularly polarized light. Since the circularly polarized signal is inherently quite weak, only the linearly polarized signal is considered.

[0036] There are two types of polarization information: degree of polarization DOP (denoted by ρ λ ) and the polarization angle AOP (denoted by α λ denoted), the polarization degree ρ λ and polarization angle α λ The following expressions (c) to (d) are:

[0037]

[0038] (3) By using the images of three different polarization angles and mathematical calculations based on the Stokes vector, we can obtain the information of the image at any polarization angle, the image with the maximum average light intensity, and the image with the minimum average light intensity.

[0039] The information of the image at the polarization angle is expressed by the following formula (e):

[0040]

[0041] The picture with the largest average light intensity and the picture with the smallest average light intensity are expressed by the following formulas (f) to (g):

[0042]

[0043] (4) The background area B(x, y) is judged by the EME evaluation function (Enhancement Measure Evaluation, abbreviated as EME, meaning regional contrast). The area with low contrast is the "background area".

[0044] The EME evaluation function is expressed by the following formula (h):

[0045]

[0046] (5) Calculate the average value of the background area as the base value. In order to reduce the impact of the background area on the target light D(x,y) information, it is necessary to design an Adaptive Local Optimization (ALO) method to optimize the base value, that is, to obtain a better solution B(x,y) information. The specific ALO method is: calculate the average value of the maximum light intensity map and the minimum light intensity map of the background area as the base value Use the base value to make the best estimate for the background area.

[0047] 5-1: First, divide the background area of ​​the entire image into n blocks, for example, divide a 512*512 image into 16*16 and 32*32 sub-areas. Then define the linear optimization coefficient k of the area i , so the optimization coefficient matrix of the background area is expressed by the following formula (i):

[0048]

[0049] By optimizing this matrix, we can get the optimized background area

[0050] 5-2: Optimize the backscatter polarization according to the following formula (j)

[0051]

[0052] in, (x,y) is the polarization degree of the global backscattered light at position (x,y), Imax and Imin are the brightest and darkest polarized images respectively (they are calculated by rotating the polarizer and then calculating the Stokes vector after taking the image at different angles), Ptarget is the polarization degree of the target, Imax

[0053] +Imin is the sum of the orthogonal polarization images, representing the total light intensity. Represent the estimated values ​​of backscattered light in the maximum and minimum polarization states, respectively.

[0054] From the above formula (j), we can see that when K, P target The optimal backscattering polarization can be calculated when Then, through physical model analysis and mathematical derivation, the mathematical expression of D(x,y) expressed by the following formula (k) can be obtained:

[0055]

[0056] Where D(x,y) represents the intensity of the target reflected light at position (x,y) in the image, Imax and Imin are the brightest and darkest polarized images, respectively, E is a unit vector with the same size as the image, which is used to adjust the mathematical form of the formula, PΘscat is the polarization of the global backscattered light, Ptarget is the polarization of the target light, ∈ and ξ are the parameters introduced to be optimized, which are used to balance the different terms in the formula.

[0057] From the above formula (k), we can see that when given When the optimal backscattering polarization D can be calculated, K, P can be adjusted by the following formula (1): target ,ε, and get D with the highest EME value.

[0058]

[0059] Where γ is the RGB channel (red, green, blue), r1 and r2 represent the number of blocks into which the image D is divided in the horizontal and vertical directions, respectively, and Dmax,i,j(x,y,z) and Dmin,i,j(x,y,z) represent the maximum and minimum light intensity values ​​in the block, respectively.

[0060] 5-3: There are many parameters to be optimized. For example, there are 32*32 parameters in the K global background area optimization coefficient matrix. For such a large number of parameters, traversal is not an advisable method. Therefore, this embodiment refers to the particle swarm optimization algorithm, and obtains the optimal solution through global exploration, historical memory, and particle swarm memory between particles, which can greatly reduce the optimization time. The specific i-th particle iteration is expressed by the following (m)~(n) formula:

[0061] V i (T+1)=ωV i (T)+C 1 r 1 (p i -X i )+C 2 r 2 (g i -X i ) (m);

[0062] Among them, Vi(T+1) is the velocity vector of particle i at iteration T+1, Vi(T) is the velocity vector of particle i at iteration T, ω is the inertia weight, which controls the influence of the current velocity of the particle on the next velocity, C1 and C2 are learning factors, which respectively adjust the step size of the particle approaching the individual optimal solution and the global optimal solution, r1 and r2 are random numbers in the range of [0,1], which are used to increase the randomness of the search, pi is the position of the individual historical optimal solution of particle i, Xi is the current position of particle i, and gi is the position of the historical optimal solution of the group (that is, the optimal solution among all particles).

[0063] X i (T+1)=V i (T+1)+X i (T) (n).

[0064] Where Xi(T+1) is the position vector of particle i at iteration T+1, Vi(T+1) is the velocity vector of particle i at iteration T+1 (calculated by formula (33)), and Xi(T) is the position vector of particle i at iteration T.

[0065] The empirical value requires 20 particles and 30 iterations to achieve the ideal effect. The attenuation coefficient is expressed as t = 1-B (x, y) / A∞, where A∞ represents the light intensity at infinite distance underwater, and the average value of the first 10% brightest pixels in the backscatter estimation image B (x, y) is taken. Based on the optimization parameters obtained in the previous article and the mathematical inversion calculation, the information of D (x, y) can be obtained. The restored target information can be expressed by the following formula (o):

[0066] L(x,y)=D(x,y) / t(o).

[0067] Among them, L(x,y) is the target light intensity after removing the influence of backscattering, D(x,y) is the target reflected light intensity (this is the target signal light obtained after removing backscattering from the total received light intensity), and t is the transmission coefficient, which describes the attenuation of light when it propagates in the medium.

[0068] 5-4: Displaying the restored L(x,y) information obtains an image with enhanced fidelity and restored details.

[0069] (6) To facilitate the derivation of the formula, the following are ignored or assumed:

[0070] 6-1: Since circularly polarized signals are inherently weak, only linearly polarized signals are considered;

[0071] 6-2: Ignore the forward scattering signal because forward scattering is very weak in natural scenes;

[0072] 6-3: The attenuation coefficient is spatially invariant under the same environment;

[0073] 6-4: The target DOP value (Ptarget) is spatially invariant;

[0074] 6-5: There is no light from the target in the background area;

[0075] 6-6: The average value of the top 10% of the brightest pixels in the backscattered light in a small background area is A∞.

[0076] (7) The light intensity value at each position of an image is expressed by the following formula (p):

[0077] I(x,y)=D(x,y)+B(x,y)(p).

[0078] Among them, I(x,y) is the total light intensity received by the underwater camera at the pixel point (x,y); D(x,y) is the target reflected light at the pixel point (x,y), which is the contribution of the target reflection to the light intensity of each pixel in the image; B(x,y) is the scattered light intensity of the floating particles and the medium at the pixel point (x,y) (the backscattering caused by the scattering of the water medium and other suspended particles can also be called the backscattered light intensity). Due to the existence of the medium, there will be a loss of light intensity, and this loss is universal. Therefore, the propagation coefficient t is defined to define the ratio of the received light intensity to the original light intensity. In addition to the attenuated light itself, the light at a certain pixel in the background also has the same A ∞ The attenuated light of other background pixels can be expressed by the following formula (q):

[0079]

[0080] Among them, A∞ is the intensity of the backscattered light at infinity (that is, the light intensity no longer changes with distance), and t is the transmission coefficient, which describes the attenuation of light when it propagates in the medium.

[0081] Since it is assumed that the target does not scatter light, the image of the target area is the image after L attenuation. Finally, D(x, y) and B(x, y) are expressed by the following (r) to (s) respectively:

[0082] D(x,y)=L(x,y)·t(x,y) (r);

[0083] B(x,y)=A ∞ (1-t(x,y)) (s).

[0084] Among them, L is the image without attenuation and scattering from the object, A ∞ is the intensity of light emitted from the background.

[0085] According to the hypothesis A ∞ It is defined by the following (t) formula:

[0086]

[0087] Where A∞ is the intensity of backscattered light at infinity, Λ is the set of the top 10% of pixels with the highest brightness in the image, and BΘmax(x,y) and BΘmin(x,y) represent the maximum and minimum backscattered light intensity values ​​corresponding to the pixel point (x,y) in the global backscattered light estimate BΘ, respectively.

[0088] Thus, the attenuation coefficient is obtained as shown in the following formula (u):

[0089]

[0090] Among them, t is the transmission coefficient, which describes the attenuation of light when it propagates in the medium, and A∞ is the intensity of the backscattered light at infinity (that is, the light intensity no longer changes with distance).

[0091] When polarization images of 0°, 45° and 90° are taken from water, the specific Stokes vector of the image can be obtained as shown in the following formula (v):

[0092]

[0093] Among them, M0(x,y) is the total light intensity, that is, the total intensity of the scene radiation light, without distinguishing the polarization direction, M1(x,y) is the intensity difference between horizontally polarized light and vertically polarized light, corresponding to the intensity difference between 0° and 90° linear polarized light, M2(x,y) is the intensity difference between 45° and -45° linear polarized light, which is used to describe another dimension of polarized light information, M3(x,y) is the intensity difference between left-handed circularly polarized light and right-handed circularly polarized light, I(x,y,0°), I(x,y,90°), I(x,y,45°) represent the light intensity measured at the coordinate (x,y) when the polarizer is rotated to 0°, 90°, and 45°, respectively, IL(x,y), IR(x,y): represent the intensity of left-handed circularly polarized light and right-handed circularly polarized light measured at the coordinate (x,y), respectively.

[0094] According to the Stokes vector, the light intensity diagram at any angle of the polarizer can be expressed by the following formula (w):

[0095]

[0096] Where I(x,y,θ) represents the light intensity measured at the coordinate (x,y) when the polarizer is rotated to the angle θ. M0(x,y), M1(x,y) and M2(x,y) are the three components of the Stokes vector, representing the total light intensity; the intensity difference between horizontally polarized light and vertically polarized light; and the intensity difference between 45° and -45° linearly polarized light.

[0097] θ is the rotation angle of the polarizer.

[0098] Thus, the average brightest light intensity map and the darkest light intensity map and the corresponding polarization angle θ are obtained by the following formula (x): max ,θ min :

[0099] I max =I(x,y,θ max )

[0100] I min =I(x,y,θ min ) (x).

[0101] Among them, Imax and Imin are the polarization angles θ corresponding to the polarization images with the brightest and darkest average light intensity, respectively. max ,θ min , that is, in θ max ,θ min The average light intensity is strongest at θ max ,θ min The average light intensity is the weakest.

[0102] (8) Decompose the light intensity image, target image, and background image into the strongest and weakest parts represented by the following equations (I) to (III):

[0103] I=I max +I min (I);

[0104] D=D max +D min (II);

[0105] B=B max +B min (III).

[0106] Among them, I is the sum of the orthogonal polarization images, representing the total light intensity of the image, D is the sum of the orthogonal polarization images of the target, representing the total light intensity of the target, and B is the sum of the orthogonal polarization images of the background, representing the total light intensity of the background.

[0107] Similarly, the DOP of D(x, y) and B(x, y) can be defined by the following formulas (IV) to (V): target , P scat :

[0108]

[0109] Among them, Ptarget is the polarization degree of the target reflected light, Dmax is the maximum intensity of the target reflected light, Dmin is the minimum intensity of the target reflected light, Pscat is the polarization degree of the backscattered light, Bmax is the maximum intensity of the backscattered light, and Bmin is the minimum intensity of the backscattered light.

[0110] Therefore, the following formulas (VI) to (VIII) can be derived:

[0111]

[0112] Among them, D is the total light intensity of the target, B is the total light intensity of the background, I is the total light intensity of the image, Ptarget is the polarization degree of the target reflected light, Pscat is the polarization degree of the backscattered light, and Imax and Imin are the brightest and darkest polarization images, respectively.

[0113] According to assumption 6-5, there is no light from the target in the background area. Select an area Ω and find I max , I min The corresponding Ω area, the average light intensity of this area is taken as the basic value of the background, and the maximum and minimum basic values ​​of the background are expressed by the following (IX) to (X):

[0114]

[0115] Among them, IΩmax is the maximum light intensity image measured in the background area Ω, IΩmin is the minimum light intensity image measured in the background area Ω, Δ is the total number of pixels in the background area, BΨmax is the average strongest light intensity in the background area, and BΨmin is the average weakest light intensity in the background area.

[0116] (9) In order to enhance the detail information, this basic value and the guided filter of the original image are used as the background area B(x, y) for information processing, which is specifically expressed by the following formulas (XI) to (XII):

[0117]

[0118] Among them, Bmax is BΨmax after guided filtering, and Bmin is BΨmin after guided filtering.

[0119] According to Assumption 6-4, the target DOP value (Ptarget) is space-invariant, and an optimization parameter P is defined target , P of formula (VIII) can be scat It is represented by the following formula (XIII):

[0120]

[0121] Among them, B is the total background light intensity, Ptarget is the polarization degree of the target reflected light, Pscat is the polarization degree of the backscattered light, and Imax and Imin are the brightest and darkest polarization images, respectively.

[0122] Similarly, by introducing the optimization parameter ε, equation (VI) can be expressed as follows (XIV):

[0123]

[0124] Where D(x,y) is the intensity of the target reflected light at the pixel position (x,y), Imin and Imax are the light intensities of the darkest and brightest orthogonal polarization images in the image, respectively, E is a unit vector of the same size as image I and is used for numerical calculations, PΘscat is the polarization degree of the backscattered light estimated by the adaptive local optimization (ALO) method, Ptarget is the polarization degree of the target reflected light, ε and ξ are the introduced coefficients to be optimized, which are used to adjust the numerical scale in the formula.

[0125] (10) The most suitable optimization parameter group (P) is found through the EME evaluation function of formula (h). target , ε), and is expressed by the following formula (XV):

[0126] (P target , ε)=argmax(EME(D))(XV).

[0127] The best optimization parameter set is brought back to D(x, y), and the D(x, y) parameter represented by the following formula (XVI) is used to obtain the target image L(x 1 y):

[0128] D(x,y)=L(x,y)·t(x,y) (XVI);

[0129]

[0130] Where D(x,y) represents the target information light intensity received at the pixel point (x,y), also known as the target reflected light, L(x,y) represents the unattenuated target signal light intensity, that is, the light intensity directly reflected from the target surface, and t(x,y) is the transmission coefficient, which describes the attenuation of light when it propagates in the medium.

[0131] Step 3: Deep Learning Training of Pix2pix Generative Adversarial Network

[0132] See also Figure 2 , the data set uses images that are initially restored by the polarization scattering physical model, and uses the trained deep learning model to perform secondary restoration on the images that are initially restored by the polarization scattering physical model of the scene target. Since the image restored by the polarization physical model is intended to enhance details to the extreme, but the color loss is large, and it is found that machine learning is a complex mapping between pictures, this complex mapping can meet the requirements for color authenticity. Therefore, the present invention uses a generative adversarial network to further restore the color of the initially restored image. The conditional generative adversarial network (CGAN) is a neural network architecture comprising two parts, consisting of a generator (G for short) and a discriminator (D for short), in which the two networks confront each other and learn at the same time.

[0133] See also Figure 3 , the steps of underwater blurred image restoration based on Pix2Pix GANs network are as follows:

[0134] 1) Shooting and Dataset Preparation

[0135] First, you need to simulate a blurry underwater environment and take blurry underwater photos and clear underwater photos. During the shooting process, first take a photo of the clear water target, then simulate a blurry underwater environment by clamping milk, keep the shooting angle and the shooting target unchanged, rotate the polarizer in front of the lens, and take underwater photos at three different polarization angles of 0°, 45°, and 90°. Process the three blurry underwater images through the above descattering physical model to obtain an underwater target picture with significantly enhanced details. Put this picture together with the clear water photo to obtain a group of photos. Repeat this process to obtain multiple groups of photos, which constitute a data set.

[0136] 2) Automatic cropping and interception

[0137] In order to make full use of shooting resources and reduce redundant data in the training process, this embodiment writes code to crop the image by automatically generating a screenshot frame and the user double-clicking to take a screenshot, generating multiple sets of small-sized image pairs. These images correspond to the same scene area, but are smaller in size, which is convenient for model processing and accelerates the training process.

[0138] 3) Image matching and dataset management

[0139] Due to the different refractive indices of clear water and blurred water, some groups of clear water photos and some groups of blurred photos may have obvious displacement during the shooting process. This module is used to screen high-quality data sets. The code is an image processing technology matching method based on Hadamard transform. The photo folders that are cropped and image matched each time are integrated to form a deep learning data set.

[0140] 4) Model training

[0141] The Pix2Pix GANs network is trained using a prepared high-quality dataset. During the training process, as the training cycle gradually increases, the generator loss gradually decreases, indicating that the images generated by the generator are getting closer and closer to the real images. The discriminator loss first decreases and then fluctuates slightly, indicating that the discriminator's ability to distinguish between generated images and real images is getting stronger and stronger. However, as the generator's ability improves to the point where the discriminator can no longer distinguish between generated images and real images well, there is a certain increase in jitter. Normally, after the generator loss stabilizes, the model is basically converged.

[0142] 5) Deep Learning Recovery

[0143] Applying the trained model to new blurry underwater images can generate corresponding clear images.

[0144] Pix2pix is ​​used, a conditional generative adversarial network (GAN). The main logic of its code is divided into the following parts:

[0145] 1) Data preprocessing

[0146] load(image_file): reads an image and splits it into an input image and a target image;

[0147] resize(input_image,real_image,height,width): adjust the image size;

[0148] random_crop(input_image,real_image): randomly crops an image;

[0149] normalize(input_image,real_image): Normalize the image pixel values ​​to the range of [-1,1];

[0150] random_jitter(input_image, real_image): Randomly jitter the image, including random scaling and flipping.

[0151] load_image_train(image_file) and load_image_test(image_file): used to load training and test data, respectively, including the data augmentation and preprocessing steps defined previously. Use tf.data.Dataset API to create and batch train and test datasets.

[0152] 2) Model Architecture

[0153] 2-1: Generator: downsample(filters, size, apply_batchnorm = True), defines the downsampling layer, including convolution, batch normalization and LeakyReLU activation.

[0154] upsample(filters, size, apply_dropout = False) : Defines an upsampling layer including transposed convolution, batch normalization, ReLU activation, and optional Dropout.

[0155] Generator(): Constructs a generator network using the U-Net architecture, including symmetrical downsampling and upsampling layers, and uses skip connections to connect corresponding layers.

[0156] 2-2: Discriminator: Discriminator(): Build a discriminator network and use convolutional layers to determine whether the input image pair is real.

[0157] 3) Loss Function

[0158] generator_loss(disc_generated_output,gen_output,target): The loss function of the generator, including GAN loss and L1 loss. The L1 loss is used to ensure the similarity between the generated image and the target image at the pixel level.

[0159] discriminator_loss(disc_real_output, disc_generated_output): The loss function of the discriminator, including the loss of real image pairs and the loss of generated image pairs.

[0160] 4) Optimizer and Checkpoint

[0161] generator_optimizer and discriminator_optimizer: Use Adam optimizer to optimize the generator and discriminator.

[0162] tf.train.Checkpoint: used to save and restore training checkpoints.

[0163] 5) Training and Evaluation

[0164] train_step(input_image,target,epoch): Single-step training function, calculates the gradients of the generator and discriminator and applies them.

[0165] fit(train_ds,epochs,test_ds): training function that performs multiple epoch training, evaluates model performance after each epoch and saves model checkpoints.

[0166] See also Figure 4 , the image display of the initial model training, the left image is the loss function of the generator, and the right image is the loss function of the discriminator.

[0167] See also Figure 5 , The image display of the model training after parameter adjustment. The left picture is the loss function of the generator, and the right picture is the loss function of the discriminator.

[0168] From the above Figure 4 and Figure 5It can be seen that as the training cycle gradually increases, the generator loss gradually decreases, which means that the images generated by the generator are getting closer and closer to the real images. The discriminator loss first decreases and then fluctuates slightly, which means that the discriminator's ability to distinguish between generated images and real images is getting stronger and stronger. However, as the generator's ability improves to the point where the discriminator can no longer distinguish between generated images and real images well, there is a certain increase in jitter. By increasing the number of training rounds and the number of learning rate decay rounds, increasing image rotation, and improving the diversity of training data, the initial learning rate is adjusted from 0.0002 to 0.0001, and the momentum parameter beta1 of the Adam optimizer is adjusted from 0.5 to 0.9; Figure 4 The initial model obtained Figure 5 A better model after parameter adjustment. Normally, after the generator loss is stable, the model has basically converged. This can only give a rough idea of ​​the model. The specific results also depend on the effect of the model on the test data set to generate pictures, and then judge the ability of the model based on the generated pictures. The data set for training the model is generally around 200 groups. Finally, the trained model is called to predict the input image, and the prediction result and the input image are visualized and saved.

[0169] The specific structure of the Pix2pix network is analyzed as follows:

[0170] 1) The U-net structure extracts and reconstructs image features for downsampling and upsampling layers, and retains detail information through skip connections).

[0171] 1-1: Encoder: Extracts and abstracts features from input images. Each layer reduces the resolution of the input image and extracts more features for downsampling. The downsampling part uses a series of convolutional layers (downsample). Each layer reduces the size of the feature map and increases the depth of the feature map (each convolutional layer maps the depth of the input feature map to a deeper feature map, which increases the dimension of the feature and thus the depth of the image. A convolutional layer has multiple convolution kernels. Each convolution kernel generates a corresponding feature map. A convolutional layer outputs multiple feature maps, i.e., the depth of the layer. The stacking of convolutional layers increases the depth of the image. Increasing the depth of the image is to increase the richness and abstractness of the features, so that the model can better understand the content and features of the input image, thereby improving the representation and generalization capabilities of the model. Deeper networks can learn more feature maps, which can capture more complex structures and patterns in the image, thereby improving the model's representation capabilities for the input image.

[0172] 1-2: Decoder: It consists of a series of deconvolution layers, each of which upsamples the feature map to a higher resolution and gradually reconstructs the original image. For each upsampling layer, an upsample layer is taken from the upsampling stack (up_stack), and the output of the corresponding skip connection layer is taken from the skip connection (skips). The output of the upsampling layer and the output of the skip connection layer are concatenated together using tf.keras.layers.Concatenate(). Finally, the low-resolution feature map is further restored to a higher resolution and output by using the transposed convolution layer defined by tf.keras.layers.Conv2DTranspose. The batch normalization layer is implemented by tf.keras.layers.BatchNormalization(), which is located after each upsampling layer (transposed convolution layer) to stabilize the training process (standardization to stabilize the input distribution) and accelerate model convergence. If the apply_dropout parameter is set to True, a Dropout layer is applied after upsampling to prevent overfitting. Finally, the ReLU activation function is used to introduce nonlinearity to help the network learn complex features.

[0173] The Conv2DTranspose layer works similarly to a standard convolution operation, but it performs a transposed convolution (deconvolution) operation. In a transposed convolution, the weights of the convolution kernel are transposed (rotated 180 degrees) and then convolved with the input feature map. Skip-connect is especially introduced in the upsampling process. A skip connection refers to a direct connection from a certain layer of the encoder to the corresponding layer of the decoder in a neural network, combining the features extracted by the encoder with the features generated by the decoder, so that the decoder can use more low-level and high-level feature information, thereby improving the performance and effect of the generated model. The feature map generated by the decoder is connected to the feature map of the corresponding layer of the encoder through the tf.keras.layers.Concatenate() function.

[0174] 2) The discriminator neural network Discriminator uses the idea of ​​PatchGAN. The discriminator of the native GAN outputs a single scalar value, which represents the authenticity probability of the entire input image. The discriminator outputs a two-dimensional feature map, in which each pixel corresponds to the authenticity probability of a local area (Patch) of the input image. The purpose of this is to enable the discriminator to perform fine-grained authenticity evaluation on the local area of ​​the image. The discriminator of PatchGAN outputs the authenticity probability of the local area, which enables the generator to better control the details and local structure of the image, and to perform authenticity evaluation at a finer scale, which is suitable for processing high-resolution images. The generator can gradually adjust the generation strategy according to the feedback of the discriminator to generate more realistic and detailed images. In the code, the two images inp and tar are first connected and stacked together through the tf.keras.layers.Concatenate() function, and then a series of convolutional layers are used to extract, process and transform the image features, and finally a two-dimensional feature map with a channel number of 1 is output through the convolutional layer last. Each pixel of this feature map represents the authenticity probability of the corresponding local area. The closer the value is to 1, the more likely the discriminator thinks this local area is from the real image, and the closer the value is to 0, the more likely the discriminator thinks this local area is from the generated image.

[0175] 3) Loss function (binary cross entropy, L1 loss)

[0176] 3-1: Generator loss function: The generator loss function consists of two parts: adversarial loss (GAN Loss) and content loss (L1Loss). The adversarial loss uses a binary cross entropy loss function, which is characterized by promoting adversarial between generated images and real images, and improving the realism of generated images. By making it more difficult for the discriminator to distinguish between generated images and real images, the generated images are made more realistic; the content loss uses the L1 loss function, which is characterized by calculating the pixel-level difference between the generated image and the target image. This helps to maintain the structure and details of the image, making the generated image closer to the target image in content. The adversarial loss and content loss are weighted and added, and multiplied by a hyperparameter λ to obtain the total loss function of the generator.

[0177] 3-2: Discriminator loss function: The discriminator loss function consists of two parts: real image loss and generated image loss, and uses a binary cross entropy loss function. The real image loss is used to measure the discriminator's ability to recognize real images, that is, the loss of the discriminator correctly classifying real images as real images; the generated image loss is used to measure the discriminator's ability to recognize generated images, that is, the loss of the discriminator correctly classifying generated images as generated images. Add the real image loss and the generated image loss to get the total loss function of the discriminator, which is expressed by the following formula (XVIII):

[0178]

[0179] Among them, the first term is the real image loss part, and the second term is the generated image loss part; LD is the loss function of the discriminator, which aims to distinguish between real images and generated images; N is the batch size, that is, the number of samples processed each time training; ∑i=1N is the sum of all samples in the batch to calculate the total loss; xreal(i) is the i-th real data sample, which is used to train the discriminator to recognize real images; D(xreal(i)) is the discriminator’s judgment result on the i-th real data sample, which should ideally be close to 1 (indicating real); yfake(i) is the label of the i-th generated data sample, which is used to indicate whether the sample is real (1) or generated (0); G(z(i)) is the generator’s generation result of the i-th random noise z(i), which is used to train the discriminator to recognize generated images; D(G(z(i))) is the discriminator’s judgment result on the i-th generated data sample G(z(i)) generated by the generator, indicating the probability that the sample is real.

[0180] 3-3: Binary Cross Entropy Loss Function

[0181] The purpose of the adversarial loss is to encourage the discriminator to accurately distinguish between generated images and real images. Therefore, for the discriminator, the generated images should be labeled as "fake" (0), while the real images should be labeled as "real" (1). By using the binary cross entropy loss function on the predicted values ​​of the generated images, the loss values ​​of the generated images being judged as "real" and "fake" can be calculated, and the parameters of the generator and discriminator can be updated based on these loss values.

[0182] 3-4: L1 loss function:

[0183] The L1 loss function measures the mean absolute error between two tensors, that is, the average of the absolute values ​​of the differences between the corresponding positions of the two tensors. The L1 loss function is used to measure the pixel-level difference between the generated image and the target image, that is, the content similarity of the image. By minimizing the L1 loss between the generated image and the target image, the generated image can be closer to the target image in content. The L1 loss function is expressed by the following formula (XIX):

[0184] L L1 (G) = E x,y,z [yG(x,z) 1 ](XIX).

[0185] Among them, LL1 is the L1 loss function of the generator, also known as pixel-level loss, which aims to make the generated image close to the real image at the pixel level; Ex,y,z is the expected value of the input data x, the real image data y and the random noise z; ∥yG(x,z)∥1 is the L1 distance between the generated image G(x,z) and the real image y, that is, the sum of the absolute errors of all pixel values. The goal of this loss function is to minimize this distance and make the generated image as close to the real image as possible.

[0186] Step 4: The image restored by deep learning is fused again with the image restored by the polarization scattering physical model, that is, the detail enhanced image obtained by the polarization scattering model and the color enhanced image obtained by machine learning are fused to obtain the final underwater image restoration image.

[0187] See also Figure 6 By comparing the underwater images restored by the present invention with those restored by the prior art, it can be clearly seen that the present invention is based on machine enhanced vision and combines the respective advantages of the polarization scattering model and deep learning to form a hybrid underwater image restoration algorithm, which effectively enhances its fidelity and restores detail information, solves the problem of obtaining accurate pixel-level matching of machine learning data sets, and solves the comprehensive application problems of physical models: large color loss and high scene requirements; machine learning: high recovery limit and difficult model training.

[0188] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. An underwater image restoration method based on polarization scattering and deep learning, characterized in that: The polarization scattering physical model is used to remove the scattered light in the background area of ​​the underwater image, and machine learning is used to restore the color of the image. The initial restoration and secondary restoration images are fused to obtain the final underwater image restoration image. The specific steps include: Step 1: Use the polarization camera module to acquire images of the scene target in the underwater image in three different polarization directions, and obtain polarization plate images of 0 degrees, 45 degrees and 90 degrees respectively; Step 2: For the three polarized film images obtained in step 1, use the MATLAB program to de-scatter the physical model and calculate the image after detail restoration; Step 3: Perform deep learning training on the Pix2pix generative adversarial network. The dataset used is the image initially restored by the polarization scattering physical model. The trained deep learning model is used to perform secondary restoration on the image initially restored by the polarization scattering physical model of the scene target. Step 4: Perform image fusion calculation on the image restored by deep learning again with the image initially restored by the polarization scattering physical model to obtain the final underwater image restoration image.

Images