A Fast Underwater Video Enhancement Processing Method and Device Based on DCP

Abstract

The present invention discloses a fast underwater video enhancement processing method and device based on DCP. By receiving the input underwater image and estimating the background light of the original underwater image based on the background light estimation model; according to the underwater imaging model, using the dark channel prior to obtain a rough transmission map; performing optimization on the transmission map according to the transmission map optimizer and performing smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map; obtaining a first enhanced image according to the background light and the optimized transmission map; performing color correction according to the first enhanced image to determine the final enhanced image. Compared with the prior art, the present invention mainly solves two problems of the performance and processing speed of underwater image and video enhancement and restoration, and realizes a real-time underwater video enhancement method applicable to various scenarios to meet the actual needs.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular, to a fast underwater video enhancement processing method and device based on DCP. Background Art

[0002] Image enhancement technology was proposed in the early 20th century and is an important component in the field of image processing. With the development of computer technology, image enhancement technology has made rapid progress.

[0003] Image processing refers to adding some information or transforming data to the original image by certain means, selectively highlighting the features of interest in the image or suppressing (masking) some unwanted features in the image, so that the image matches the visual response characteristics. Image enhancement and restoration technologies can be roughly divided into two categories: traditional methods and deep learning methods, mainly including non-physical model-based, physical model-based, deep learning methods, and fusion-based methods. Among them, the non-physical model-based method performs enhancement technology in the image space with pixels as the unit, and repairs the contrast, color, and clarity of the image at the pixel level according to the characteristics of underwater images. Since it does not consider the imaging characteristics and content of the image, it is easy to cause information loss and introduce artifacts. The enhancement and restoration methods based on physical methods establish a model according to the degradation principle of underwater images, and inversely deduce the original image before degradation from the mathematical process. This method is limited by the estimation of scene parameters and the application scenarios are also limited. However, if the scene parameters can be accurately obtained, the restoration effect is relatively remarkable. The deep learning-based method requires obtaining a large number of underwater image training samples to obtain algorithm parameters, with a large amount of computation and time-consuming. Due to the lack of real underwater samples, the generalization ability of the trained network is low. The fusion-based image enhancement algorithm can fuse two or more images into one image, and the obtained image has the maximum amount of information and does not generate details that do not exist in the image. Its advantage is that it can fuse multiple algorithms together, but it will increase the computational complexity of the algorithm and the operation processing speed is slow.

[0004] Currently, the existing underwater image enhancement algorithm based on DCP (Dark Channel Prior) is based on the underwater imaging model (IFM). This method only needs to obtain two scene parameters: background light (BL) and transmission map (TM), and then can inversely deduce the enhanced clear underwater image. Most of the above methods have the following problems: First, the underwater environment is complex and changeable. The existing enhanced and restored algorithms based on DCP improvement have limited application scenarios or only enhance a certain type of underwater image. Second, the enhancement methods based on the underwater imaging model mostly rely on the estimation of two parameters, background light and transmission map. If the parameter estimation is inaccurate, the enhancement effect is not obvious. Finally, most of the DCP-based algorithms have a slow processing speed and a large amount of computation, and are not suitable for real-time enhancement of underwater videos. Summary of the Invention

[0005] In view of this, the purpose of the embodiments of the present invention is to provide a fast underwater image enhancement processing method based on DCP, which fully considers the complexity of underwater imaging, proposes a background light estimation method applicable to multiple scenarios, and at the same time, designs an optimizer to solve the technical problem of inaccurate transmission map estimation in some scenarios.

[0006] The first aspect of the present invention provides a fast underwater video enhancement processing method based on DCP, and the method includes:

[0007] Receiving an input underwater image and estimating the background light of the original underwater image based on a background light estimation model;

[0008] According to the underwater imaging model, using the dark channel prior to obtain a rough transmission map;

[0009] Optimizing the transmission map according to a transmission map optimizer and performing smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map; obtaining a first enhanced image according to the background light and the optimized transmission map;

[0010] Performing color correction according to the first enhanced image to determine a final enhanced image.

[0011] Further, the receiving the input underwater image includes: receiving an input underwater video or image;

[0012] The background light estimation model is expressed as:

[0013]

[0014] Wherein,

[0015] Wherein, I C is the input image, BL represents the estimated background light; BL1 and BL2 respectively represent the background light estimated by different methods, σ is a preset threshold, m represents the color channels with channel mean greater than σ, and n represents the color channels with channel mean less than σ; mean(I C ) represents the mean of the C color channels of the input image, where the C color channels include three color channels of R, G, and B; Avg m , Std m and Med m respectively represent the mean, standard deviation and median of the color channels of the input image; Ω(x, y) represents a local region block centered on (x, y). Here z represents the pixel value of the coordinate (x, y), so I C (x, y) is essentially the same as I C (z), so, in the subsequent formulas, I C (z) = IC (x, y).

[0016] Furthermore, according to the underwater imaging model, obtaining a rough transmission map using dark channel prior includes:

[0017]

[0018] where t R (z) is the rough transmission map of the R channel, I C is the input underwater image, B C represents the background light, Ω(x, y) is a local region block; z represents the pixel value of the coordinate (x, y).

[0019] Furthermore, the transmission map optimizer performs optimization on the transmission map and smooths the transmission map based on guided filtering to obtain an optimized transmission map, including:

[0020] Obtain the scene depth map d(x, y), and the expression is as follows:

[0021] d(x, y) = θ0 + θ1 × v(x, y) + θ2 × s(x, y) + τ(x, y)

[0022] where v(x, y) represents the brightness of the image, s(x, y) represents the saturation of the image, θ0, θ1, θ2 are correlation coefficients, τ is a random variable of the model error, and follows a Gaussian distribution;

[0023] Perform a minimum filtering operation on the scene depth map to obtain the filtered depth map d f (x, y):

[0024]

[0025] The transmission maps t C (x, y) of the three color channels can be obtained by the following formula:

[0026] t C (x, y) = exp(-β C d f (x, y)), C ∈ {R, G, B}

[0027] where β C represents the attenuation coefficient of the light wave of the channel image in water. Only considering the red channel, taking β C = β R = 1, we get:

[0028]

[0029] The color attenuation theory holds that for the near-background area with a relatively small intensity value, its brightness value and saturation are both relatively small, and the obtained depth of field is also small, which is used to compensate for and modify the transmission map:

[0030]

[0031] The image saturation map Sat(I C (x, y)) can be expressed by the following formula:

[0032]

[0033] According to the HSV model, when there is no light in the scene, the image is fully saturated. As the white light increases, the color channels will lose saturation. The area illuminated by artificial light sources forces the pixels to have very close brightness values on all three color channels. The area lacking saturation in the image can be explained as being illuminated by a large number of artificial light sources. Especially for underwater images, the saturation of the scene without artificial light sources is much greater than that of the artificially illuminated area. This phenomenon is expressed by the following formula:

[0034] Sat p (I C (x, y)) = 1 - α × Sat(I C (x, y))

[0035] where α is a coefficient related to the average saturation, α = 1 – Avg(Sat(I C (x, y)));

[0036] Obtain the transmission map of the final R channel

[0037]

[0038] According to the transmission map of the R channel Further determine the transmission maps of the G and B color channels

[0039] Furthermore, the said according to the transmission map of the R channel Further determine the transmission maps of the G and B color channels Includes:

[0040] Obtain the transmission maps of the R, G, and B color channels according to the following formula

[0041] where (x, y) represents the horizontal and vertical coordinates of each point in the image; β R 、β G 、β Brespectively represent the attenuation coefficients of light waves in the R, G, and B channels in water; b(R), b(G), and b(B) represent the wavelengths of red, green, and blue light waves; B R 、B G 、B B represent the values corresponding to the three color channels in the background light.

[0042] Generate an optimized transmission map according to the transmission maps of the R, G, and B color channels. Generate an optimized transmission map.

[0043] Furthermore, obtaining a first enhanced image according to the background light and the optimized transmission map includes:

[0044]

[0045] where J C (x, y) is the first enhanced image; B C is the background light corresponding to each color channel; t C (x, y) represents the transmission map corresponding to channel C, where channel C includes the R, G, and B color channels.

[0046] Furthermore, the color correction includes color cast and contrast correction;

[0047] Performing color correction according to the first enhanced image to determine the final enhanced image includes:

[0048]

[0049] where I in and I out represent the input restored image and the output corrected image; m R 、m G and m B represent the average values of the color channels of image I in ; m represents any one of m R 、m G and m B for the corrected corresponding color channel; V p is the maximum intensity value of image I in ; the parameter λ as is a gain factor for adjusting the color of the input image, and its range is from 0 to 0.5;

[0050] where,

[0051] where the function tanh() represents the hyperbolic tangent function; m1 and m2 represent the minimum and maximum values of m R 、m G and m B 。

[0052] In addition, the second aspect of the present invention provides a fast underwater video enhancement processing device based on DCP, and the device includes:

[0053] A background light estimation module, which receives an input underwater image and estimates the background light of the original underwater image based on a background light estimation model;

[0054] A transmission map estimation module, which obtains a rough transmission map using the dark channel prior according to an underwater imaging model;

[0055] A first optimization module, which optimizes the transmission map according to a transmission map optimizer and performs smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map; and obtains a first enhanced image according to the background light and the optimized transmission map;

[0056] A second optimization module, which performs color correction according to the first enhanced image to determine a final enhanced image.

[0057] In addition, the third aspect of the present invention provides an electronic device, and the electronic device includes: one or more processors, a memory, and the memory is used to store one or more computer programs; the computer programs are configured to be executed by the one or more processors, and the programs include steps for implementing the above-mentioned fast underwater video enhancement processing method based on DCP.

[0058] In addition, the fourth aspect of the present invention provides a computer-readable storage medium, and at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the steps of the above-mentioned fast underwater video enhancement processing method based on DCP.

[0059] In the solution of the present invention, by receiving an input underwater image and estimating the background light of the original underwater image based on a background light estimation model; obtaining a rough transmission map using the dark channel prior according to an underwater imaging model; optimizing the transmission map according to a transmission map optimizer and performing smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map; obtaining a first enhanced image according to the background light and the optimized transmission map; performing color correction according to the first enhanced image to determine a final enhanced image. Compared with the prior art, the present invention mainly solves two problems of the performance and processing speed of underwater image and video enhancement and restoration. By improving the estimation methods of two parameters, the background light and the transmission map, the present invention proposes an estimation of the background light parameter and designs a transmission map optimizer, and corrects the brightness and contrast of the restored underwater image based on a white balance color correction method for automatically selecting a gain factor, thereby realizing a real-time underwater video enhancement method applicable to multiple scenarios to meet actual needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0061] Figure 1 is a flowchart of a fast underwater video enhancement processing method based on DCP disclosed in Embodiment 1 of the present invention;

[0062] Figure 2 is a schematic diagram of the overall steps of a fast underwater video enhancement processing based on DCP disclosed in Embodiment 1 of the present invention;

[0063] Figure 3 is a schematic diagram of the structure of a fast underwater video enhancement processing device based on DCP disclosed in Embodiment 2 of the present invention. Detailed implementation manners

[0064] Now, example embodiments will be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art.

[0065] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to give a full understanding of the embodiments of this application. However, those skilled in the art will realize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of this application.

[0066] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0067] The flowcharts shown in the drawings are only illustrative and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined, so the actual execution order may change according to the actual situation.

[0068] Glossary of terms:

[0069] ① Dark Channel Prior (DCP): A classic image dehazing algorithm. This method is based on a statistical hypothesis that in a haze-free image, there are some pixels in a local area, and at least one of the brightness values of the color channels in these pixels is very, very low, approaching 0.

[0070] ② Background Light (BL): It is mainly the light that illuminates the surrounding environment and background of the object being photographed, and it can be used to adjust the ambient and background tones around the person.

[0071] ③ Transmission Map (TM): A medium transmittance map used to describe the part of the light that is directly captured by the camera without scattering.

[0072] ④ Scene Depth Map: An image that takes the distance (depth) from the image acquisition device to each point in the scene as the pixel value, which directly reflects the geometric shape of the visible surface of the scene.

[0073] ⑤ Saturation Map: A map used to describe the color purity of underwater images.

[0074] ⑥ UDCP and NUDCP are two classic underwater image enhancement and restoration algorithms.

[0075] It should be noted that: "multiple" mentioned in this embodiment refers to two or more.

[0076] The implementation details of the technical solution of the embodiment of the present application are elaborated in detail below:

[0077] Underwater images and videos are important carriers for obtaining underwater scene information in this embodiment. Due to the complex physical and chemical properties underwater, underwater images often exhibit color deviation, low blurry contrast, and "bright spot" and other degradation phenomena. Existing image enhancement methods can be divided into two categories, one is based on traditional methods, and the other is based on deep learning. Most existing enhancement algorithms have limited application ranges on the one hand, only applicable to specific situations, and on the other hand, have high computational complexity and slow processing speeds. To meet actual needs, not only the performance and generalization ability of the algorithm need to be considered, but also the processing speed of the algorithm needs to be improved.

[0078] The underwater image enhancement algorithm based on DCP is based on the underwater imaging model (IFM). This method only needs to obtain two scene parameters: background light (BL) and transmission map (TM), and then the enhanced clear underwater image can be inversely deduced. The principle is simple. Optimizing the parameter estimation method can greatly improve the speed of the algorithm, and it is also easy to combine with the color correction algorithm to solve the problems of color deviation and low contrast of underwater images. However, the performance of this method is too dependent on scene parameters and is mostly applicable to underwater images obtained in a uniform illumination scene.

[0079] To address the above technical problems, this embodiment proposes a fast underwater image enhancement algorithm based on DCP, which fully considers the complexity of underwater imaging and proposes a background light estimation method applicable to multiple scenarios. At the same time, to solve the inaccuracy of transmission map estimation in some scenarios, an optimizer is specifically designed to overcome these problems. The purpose of the present invention is to propose a real-time underwater video enhancement method applicable to multiple scenarios to meet actual needs.

[0080] Embodiment 1

[0081] Please refer to Figure 1 , Figure 1 which is a schematic flowchart of a fast underwater video enhancement processing method based on DCP disclosed in an embodiment of the present invention. As Figure 1 shown, a fast underwater video enhancement processing method based on DCP in an embodiment of the present invention includes:

[0082] S1, receiving an input underwater image and estimating the background light of the original underwater image based on a background light estimation model.

[0083] Specifically, in this embodiment, the background light estimation method occupies a large amount of time in the entire algorithm processing. From the perspective of algorithm real-time performance, using an existing statistical model-based method for fast background light estimation can speed up the solution. To address the problem that this method has a significant effect only on images with red channel attenuation, it is improved to be applicable to images with severe attenuation in other color channels. At the same time, by combining it with the background light estimation method based on DCP, the applicable scenario range of the proposed background light estimation can be greatly expanded. Through the above estimation method, the background light can be estimated quickly, and its accuracy can also be guaranteed compared with other more time-consuming and accurate estimation methods, such as the quadtree hierarchical search algorithm and background light candidate regions. At the same time, from the practicality requirement, it can meet the actual real-time requirement of this embodiment.

[0084] Further, in S1, receiving the input underwater image includes: receiving an input underwater video or image;

[0085] In this embodiment, the improved background light estimation model is expressed as:

[0086]

[0087] wherein,

[0088] wherein, I C is the input image, BL represents the estimated background light; BL1 and BL2 respectively represent the background lights estimated using different methods, σ is a preset threshold, m represents the color channels with channel means greater than σ, n represents the color channels with channel means less than σ; mean(IC ) represents the mean value of the C color channel of the input image, where the C color channel includes three color channels: R, G, and B; Avg m , Std m and Med m respectively represent the mean value, standard deviation, and median of the color channels of the input image; Ω(x, y) represents a local region block centered at (x, y).

[0089] S2, according to the underwater imaging model, use the dark channel prior to obtain a rough transmission map.

[0090] According to the underwater imaging model:

[0091] I C (x, y) = J C (x, y)t C (x, y) + B C (1 - t C (x, y))

[0092] Among them, the parameters: I C represents the original underwater image input; J C represents the restored clear underwater image; B C represents the background light of the input underwater image, with a dimension of 1×3, corresponding to the three color channels of B, G, and R; t C represents the transmission map of the C color channel; C represents one of the BGR color channels; (x, y) represents the position coordinates of a certain pixel point in the image.

[0093] Perform a minimum value operation on the above formula on the local block Ω(x, y) to obtain:

[0094]

[0095] B C is a value greater than zero. Divide both sides by B C to get:

[0096]

[0097] t C (z) is continuous and invariant within the small local block, so there is:

[0098]

[0099] Perform a minimum value filtering operation on the three color channels of BGR to obtain:

[0100]

[0101] According to the dark channel prior theory it can be obtained:

[0102]

[0103] Finally, it can be obtained that:

[0104]

[0105] Because at the same point (x, y), the transmitted value of the red channel is smaller than those of the green and blue channels, it can be obtained that:

[0106]

[0107] Therefore, in this embodiment, based on the above derivation formula, further, in S2, the step of obtaining a rough transmission map using the dark channel prior according to the underwater imaging model includes:

[0108]

[0109] where t R (z) is the rough transmission map of the R channel, I C is the input underwater image, B C represents the background light, Ω(x, y) is a local region block; z represents the pixel value at the coordinate (x, y).

[0110] In S3, optimize the transmission map according to the transmission map optimizer, and perform smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map; obtain a first enhanced image according to the background light and the optimized transmission map.

[0111] Specifically, in this embodiment, due to the complex and changeable underwater environment, underwater images or videos may be acquired in various scenarios, such as turbid, clear, with artificial lighting, dim, etc. Relying solely on the above background light estimation method and transmission map estimation method for restoration has poor effects on some special scene images and may even introduce artifacts, which does not meet the actual requirements. Considering these factors, a TM optimizer is designed on this basis for optimization and improvement, which can not only be used to eliminate the influence of scene factors on the accuracy of transmission map estimation, but also improve the quality and effect of the restored underwater images or videos.

[0112] The design of the TM optimizer is mainly divided into two steps. The first step is to use the scene depth map to remove the inaccurate parts in the TM, and the second step mainly solves the influence of artificial light sources introduced by the underwater scene on image enhancement and restoration.

[0113] Further, in S3, optimizing the transmission map according to the transmission map optimizer and performing smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map further includes:

[0114] The scene depth d(x, y) is obtained based on the color attenuation theory, and the expression is as follows:

[0115] d(x,y) = θ0 + θ1×v(x,y) + θ2×s(x,y) + τ(x,y)

[0116] where v(x, y) represents the brightness of the image, s(x, y) represents the saturation of the image, θ0, θ1, θ2 are correlation coefficients, and τ is a random variable of the model error and follows a Gaussian distribution; these parameters can be obtained through supervised learning, and the paper "Zhu, Qingsong, Jiaming Mai, and Ling Shao. 'A fast single image haze removal algorithm using color attenuation prior.' IEEE transactions on image processing 24.11 (2015): 3522 - 3533." can be referred to. To simplify the problem, the optimal solutions in the paper are directly used, θ0 = 0.121779, θ1 = 0.959710, θ2 = -0.780245, and τ = 0.041337.

[0117] For some special underwater images containing artificial lighting or white objects, the scene depth map of the underwater image obtained by the scene depth linear model may not match the actual situation in the corresponding area. To overcome the above problems, the obtained scene depth map needs to be subjected to minimum filtering operation to avoid the influence of white areas in the image on the scene depth estimation, and the filtered depth map d f (x,y):

[0118]

[0119] The transmission map t C (x,y) of the three color channels can be obtained by the following formula

[0120] t C (x,y) = exp(-β C d f (x,y)), C ∈ {R, G, B}

[0121] where β C represents the attenuation coefficient of the light wave of the channel image in water.

[0122] Only considering the red channel first, taking β R = 1, we get:

[0123]

[0124] The color attenuation theory holds that for the close - range background area with relatively small intensity values, its brightness and saturation values are relatively small, and the obtained depth of field is also small, which can be used to compensate for and modify TM.

[0125]

[0126] As the depth increases, the underwater vision becomes darker and darker. Underwater imaging devices will introduce artificial light sources to increase the scene brightness, resulting in "bright spots" in the central area of the image. However, a high - brightness pixel area does not necessarily indicate the presence of artificial light in that area. It is found that the saturation in the HSV color space of the area illuminated by artificial light sources is relatively low. The image saturation map Sat(I C (x, y)) can be expressed by the following formula:

[0127]

[0128] According to the HSV model, when there is no light in the scene, the image is fully saturated. As the white light increases, the color channels will lose saturation. The area illuminated by artificial light sources forces the pixels to have very close brightness values in all three color channels. The area lacking saturation in the image can be interpreted as the area being illuminated by a large amount of artificial light sources. Especially for underwater images, the saturation of the scene without artificial light sources is much greater than that of the artificially illuminated area. This phenomenon can be expressed by the following formula as

[0129] Sat p (I C (x, y)) = 1 - α×Sat(I C (x, y))

[0130] α is a coefficient related to the average saturation value, which can be expressed as α = 1 – Avg(Sat(I C (x, y))).

[0131] Furthermore, by further optimizing the transmission map, the influence of the artificial light source area can be reduced while having little impact on other areas. Thus, the final transmission map of the red channel is obtained

[0132]

[0133] After the TM optimizer compensates for and modifies TM, a more accurate TM can be obtained. It is also necessary to use guided filtering to smooth TM.

[0134] Furthermore, according to the transmission map of the R channel The transmission maps of the G and B color channels are further determined

[0135] The transmission maps on each color channel are obtained using the following formula. These three matrices are not linearly independent. In this embodiment, only one matrix and two scale coefficients need to be obtained.

[0136] The transmission maps of the three color channels R, G, and B are obtained according to the following formula

[0137] where (x, y) represents the horizontal and vertical coordinates of each point in the image; β represents the attenuation coefficient of light waves in water;;β R 、β G 、β B respectively represent the attenuation coefficients of light waves in the R, G, and B channels in water; b(R), b(G), and b(B) represent the wavelengths of red, green, and blue light waves; B R 、B G 、B B represent the values corresponding to the three color channels in the background light.

[0138] According to the transmission maps of the three color channels R, G, and B Generate an optimized transmission map.

[0139] Further, in S3, the obtaining of the first enhanced image according to the background light and the optimized transmission map includes:

[0140]

[0141] where, J C (x, y) is the first enhanced image; B C is the background light corresponding to each color channel; t C (x, y) represents the transmission map corresponding to channel C, where channel C includes the three color channels R, G, and B.

[0142] S4. Perform color correction on the first enhanced image to determine the final enhanced image.

[0143] The brightness of the above enhanced underwater fog-free image is relatively dark, and color correction processing needs to be performed on the image to improve the image brightness, correct color deviation, and increase the contrast.

[0144] Further, the color correction includes color deviation and contrast correction; in S4, the performing of color correction on the first enhanced image to determine the final enhanced image includes:

[0145]

[0146] where, I in and I out represent the input restored image and the output corrected image; m R 、mG and m B represents the image I in the average value of the color channels, and m represents m R , m G and m B corrects any corresponding color channel among them; V p is the maximum intensity value of the image I in ; the parameter λ as is a gain factor for adjusting the color of the input image, and its range is from 0 to 0.5. According to the color correction effect selected by the restored image λ as a self - selected gain factor is proposed to obtain the required λ as . The principle of this embodiment is as follows: when the maximum value of m R , m G and m B is greater than 0.45, the image I in is brighter and λ as is close to 0.5; in other words, if the maximum value is less than 0.45, the image is dark and λ as tends to 0, which can be expressed as:

[0147] where,

[0148] where the function tanh() represents the hyperbolic tangent function; m1 and m2 represent the minimum and maximum values of m R , m G and m B among them.

[0149] According to the above - calculated results of color correction, the final enhanced image is determined, and the enhanced and restored underwater image is output.

[0150] In summary, this embodiment proposes a fast underwater image and video enhancement algorithm based on DCP. Aiming at the problem that the parameter estimation of the background light and the transmission map in the original DCP is inaccurate, this embodiment proposes a parameter estimation method applicable to various environments. Such as Figure 2As shown, in this embodiment, the first step is to execute the input of underwater images or videos. The second step: estimate the background light of the original underwater image. Among them, in the second step, an improved background light (BL) estimation method is proposed to estimate the BL quickly and accurately. The third step is the rough transmission map estimation. Among them, a rough estimation of the transmission map (TM) of the red channel based on DCP is performed. The fourth step: design of the transmission map optimizer. Among them, a TM optimizer integrating the scene depth map and the adaptive saturation map is designed to refine and optimize the above TM. Then, the TM of the green and blue channels is estimated by the attenuation ratio of the green and blue channels to the red channel. The fifth step: color correction. An improved color correction algorithm is used to improve the contrast and brightness. The sixth step is to output the enhanced and restored underwater image. In this embodiment, the proposed method can not only obtain clear underwater images, but also be used for real-time enhancement of underwater videos. Compared with the prior art, this embodiment mainly solves two problems, namely the performance and processing speed of underwater image and video enhancement and restoration. By improving the estimation methods of two parameters, the background light and the transmission map, the proposed background light parameter estimation and the design of the transmission map optimizer are carried out, and the brightness and contrast of the restored underwater image are corrected based on the white balance color correction method of automatically selecting the gain factor, realizing a real-time underwater video enhancement method applicable to various scenarios to meet the actual needs.

[0151] In this embodiment, further, an experiment on real-time enhancement and restoration of underwater videos is carried out.

[0152] The purpose of the present invention is to perform underwater image and video restoration on an underwater mobile operation platform (such as an autonomous underwater vehicle and a remotely operated underwater vehicle). The hardware resources and computing capabilities of these devices are quite poor, and real-time performance is challenging and significant. The proposed underwater image restoration method is tested in real time on a fin-propelled underwater vehicle manipulator system, and the size of the underwater image obtained by the network camera of this system is 312×554×3. In the marine environment, a two-minute 3000-frame underwater video is enhanced and restored by different methods, including UDCP, NUDCP, and our method. The image quality evaluation metrics (averages) and real-time measurement results are listed in Tables 1 and 2.

[0153] Table 1. Image quality evaluation metrics for enhancement and restoration by different methods

[0154]

[0155] As can be seen from Table 1, our method has achieved good results. A high peak signal-to-noise ratio indicates that the enhanced image is distortion-free, a high structural similarity index value indicates that the restored underwater image is very close to the reference high-quality underwater image in structure, and the UIQM metric is related to the image brightness, clarity, and contrast, and a large value indicates that good results have been achieved in these three aspects.

[0156] Table 2. Real-time processing speed of underwater video restoration and enhancement

[0157]

[0158] The UDCP method is only applicable to enhancing and restoring a single image. Its average processing speed for underwater videos is much slower and is not used for real-time enhancement and restoration. The NUDCP method can improve the speed of underwater video enhancement, but the algorithm of the present invention has an average processing time of 90 ms - 100 ms for each underwater image, and the average restoration speed exceeds 10 FPS, almost four times that of NUDCP. Because some parameter estimation methods have been improved and optimized, the parameters can be quickly solved to meet our actual needs. The algorithm designed by the present invention can greatly improve the processing speed of underwater video restoration and can be applied to engineering tasks in real time.

[0159] Example 2

[0160] As Figure 3 shown, the second aspect of this embodiment provides a fast underwater video enhancement processing device based on DCP. The device includes:

[0161] A background light estimation module 10, which receives the input underwater image and estimates the background light of the original underwater image based on the background light estimation model;

[0162] A transmission map estimation module 20, which obtains a rough transmission map using the dark channel prior according to the underwater imaging model;

[0163] A first optimization module 30, which optimizes the transmission map according to the transmission map optimizer and performs smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map; and obtains a first enhanced image according to the background light and the optimized transmission map;

[0164] A second optimization module 40, which performs color correction according to the first enhanced image to determine the final enhanced image.

[0165] Furthermore, the background light estimation module 10, which receives the input underwater image, is further configured to: receive the input underwater video or image;

[0166] The background light estimation model in the background light estimation module 10 is expressed as:

[0167]

[0168] Wherein,

[0169] Wherein, I CLet \(I\) be the input image, \(BL\) denote the estimated background light; \(BL1\) and \(BL2\) respectively denote the background light estimated using different methods, \(\sigma\) be a preset threshold, \(m\) denote the color channels whose channel means are greater than \(\sigma\), \(n\) denote the color channels whose channel means are less than \(\sigma\); \(mean(I C )\) represents the mean of the \(C\) color channel of the input image; \(Avg m \), \(Std m \) and \(Med m \) respectively represent the mean, standard deviation and median of the color channels of the input image; \(\Omega(x,y)\) represents a local region block centered at \((x,y)\).

[0170] Furthermore, the transmission map estimation module 20 is further configured to execute:

[0171]

[0172] where \(t R (z)\) is the rough transmission map of the \(R\) channel, \(I C \) is the input underwater image, \(B C \) represents the background light, \(\Omega(x,y)\) is a local region block; \(z\) represents the pixel value of the coordinate \((x,y)\).

[0173] Furthermore, the first optimization module 30 optimizes the transmission map according to the transmission map optimizer and performs smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map, and is further used for:

[0174] Obtain the scene depth map \(d(x,y)\), and the expression is as follows:

[0175] \(d(x,y)=\theta_0+\theta_1\times v(x,y)+\theta_2\times s(x,y)+\tau(x,y)\)

[0176] where \(v(x,y)\) represents the brightness of the image, \(s(x,y)\) represents the saturation of the image, \(\theta_0\), \(\theta_1\), \(\theta_2\) are correlation coefficients, \(\tau\) is a random variable of the model error and follows a Gaussian distribution;

[0177] Perform a minimum filtering operation on the scene depth map to obtain the filtered depth map \(d f (x,y)\):

[0178]

[0179] The transmission maps \(t C (x,y)\) of the three color channels can be obtained by the following formula:

[0180] \(t C (x,y)=\exp(-\beta C d f (x,y))\), \(C\in\{R,G,B\}\)

[0181] Among them, β C represents the attenuation coefficient of the light wave of the channel image in water.

[0182] Prioritize considering only the red channel, take β R = 1, and obtain:

[0183]

[0184] The color attenuation theory holds that for the near-background area with a relatively small intensity value, its brightness value and saturation are relatively small, and the obtained depth of field is also small, which is used to compensate and modify the transmission map:

[0185]

[0186] The image saturation map Sat(I C (x, y)) can be expressed by the following formula:

[0187]

[0188] According to the HSV model, when there is no light in the scene, the image is fully saturated. As the white light increases, the color channels will lose saturation. The area illuminated by artificial light sources forces the pixels to have very close brightness values on the three color channels. The area lacking saturation in the image can be interpreted as being illuminated by a large number of artificial light sources. Especially for underwater images, the saturation of the scene without artificial light sources is much greater than that of the artificially illuminated area. This phenomenon is expressed by the following formula:

[0189] Sat p (I C (x, y)) = 1 - α × Sat(I C (x, y))

[0190] Among them, α is a coefficient related to the average saturation, α = 1 – Avg(Sat(I C (x, y)));

[0191] Obtain the transmission map of the final R channel

[0192]

[0193] According to the transmission map of the R channel Further determine the transmission maps of the G and B color channels

[0194] Furthermore, the said according to the transmission map of the R channel Further determine the transmission maps of the G and B color channels Includes:

[0195] Obtain the transmission diagrams of the R, G, and B color channels according to the following formula

[0196] where (x, y) represents the horizontal and vertical coordinates of each point in the image; β represents the attenuation coefficient of light waves in water;

[0197] b(R), b(G), b(B) represent the wavelengths of red, green, and blue light waves; B R 、B G 、B B represent the values corresponding to the three color channels in the background light.

[0198] Generate an optimized transmission diagram according to the transmission diagrams of the R, G, and B color channels

[0199] Furthermore, the first optimization module 30, which obtains the first enhanced image according to the background light and the optimized transmission diagram, is further configured to:

[0200]

[0201] where J C (x, y) is the first enhanced image; B C is the background light corresponding to each color channel;

[0202] Furthermore, the color correction includes color cast and contrast correction;

[0203] The second optimization module 40 is further configured to execute:

[0204]

[0205] where I in and I out represent the input restored image and the output corrected image; m R 、m G and m B represent the average values of the color channels of the image I in ; V p is the maximum intensity value of the image I in ; the parameter λ as is a gain factor for adjusting the color of the input image, and its range is from 0 to 0.5;

[0206] where,

[0207] where the function tanh() represents the hyperbolic tangent function; m1 and m2 represent m R 、m G and m B ​The minimum and maximum values in

[0208] In addition, this embodiment also provides an electronic device, which includes: one or more processors, and a memory for storing one or more computer programs; the computer programs are configured to be executed by the one or more processors, and the programs include steps for implementing the above-mentioned fast underwater video enhancement processing method based on DCP.

[0209] In addition, this embodiment also provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the steps of the above-mentioned fast underwater video enhancement processing method based on DCP.

[0210] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this embodiment can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0211] In several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections to each other can be indirect couplings or communication connections through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0212] The unit described as a separation component may or may not be physically separated. As can be realized by those of ordinary skill in the art, the units and algorithm steps of each example described in combination with the embodiments disclosed in this embodiment can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0213] In addition, the functional units in each embodiment of the present invention can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0214] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a grid device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0215] The specific embodiments described above further elaborate on the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A fast underwater video enhancement processing method based on DCP, characterized in that, The method includes: Receiving an input underwater image and estimating the background light of the original underwater image based on a background light estimation model; According to the underwater imaging model, using the dark channel prior to obtain a rough transmission map; Optimizing the transmission map according to a transmission map optimizer and performing smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map; obtaining a first enhanced image according to the background light and the optimized transmission map; The optimizing the transmission map according to a transmission map optimizer and performing smoothing processing on the transmission map based on guided filtering to obtain an optimized transmission map includes: Obtaining a scene depth map d(x,y), the expression being as follows: d(x,y) = θ0 + θ1×v(x,y) + θ2×s(x,y) + τ(x,y) Among them, v(x, y) represents the brightness of the image, and s(x, y) represents the saturation of the image. is the correlation coefficient, τ is a random variable of the model error, and it follows a Gaussian distribution. Perform a minimum filtering operation on the scene depth map to obtain the filtered depth map d f (x, y): The transmission map t of the three color channels is obtained by the following formula C (x, y): t C (x, y) = exp(-β C d f (x, y)), C ∈ {R, G, B} Among them, β C represents the attenuation coefficient of the light wave of the channel image in water; Only consider the red channel preferentially and take β R = 1, and obtain: The color attenuation theory holds that for a close-range background area with a small intensity value, its brightness value and saturation are both small, and the obtained depth of field is also small, which is used to compensate for and modify the transmission map: Image saturation graph Sat(I C (x, y)) is expressed by the following formula: According to the HSV model, when there is no light in the scene, the image is fully saturated. As the white light increases, the color channels will lose saturation. The area illuminated by artificial light sources forces the pixels to have very close brightness values on all three color channels. The area lacking saturation in the image is interpreted as being illuminated by a large number of artificial light sources. Especially for underwater images, the saturation of a scene without artificial light sources is much greater than that of the artificially illuminated area. This phenomenon is expressed by the following formula: Sat p (I C (x,y)) = 1 - α × Sat(I C (x,y)) where α is a coefficient related to the average saturation, and α = 1 – Avg(Sat(I C (x,y))); Obtain the transmission graph of the final R channel According to the transmission diagram of the R channel Further determine the transmission diagrams of the G and B color channels Performing color correction according to the first enhanced image to determine a final enhanced image; The color correction includes color cast and contrast correction; The performing color correction according to the first enhanced image to determine a final enhanced image includes: Among them, I in and I out represent the input restored image and the output corrected image; m R 、m G and m B represent the average value of the color channels of the image I in , m represents m R 、m G and m B in any one of the corrected corresponding color channels; V p is the maximum intensity value of the image I in ; the parameter λ as is a gain factor used to adjust the color of the input image, and its range is from 0 to 0.5; Among them, Among them, the function tanh() represents the hyperbolic tangent function; m1 and m2 represent the minimum and maximum values among m R , m G and m B .

2. The fast underwater video enhancement processing method based on DCP according to claim 1, characterized in that, The receiving an input underwater image includes: receiving an input underwater video or image; The background light estimation model is expressed as: Among them, Among them, I C is the input image, BL represents the estimated background light; BL1 and BL2 respectively represent the background lights estimated using different methods as above, σ is a preset threshold, m represents the color channels with channel means greater than σ, and n represents the color channels with channel means less than σ; mean(I C ) represents the mean of the C color channel of the input image; among them, the C color channel includes three color channels R, G, and B; Avg m , Std m , and Med m respectively represent the mean, standard deviation, and median of the color channels of the input image; Ω(x, y) represents a local region block centered on (x, y).

3. The fast underwater video enhancement processing method based on DCP according to claim 2, wherein, The according to the underwater imaging model, using the dark channel prior to obtain a rough transmission map includes: Among them, t R (z) is the rough transmission map of the R channel, and I C is the input underwater image, and B C represents the background light, and Ω(x, y) is a local region block; z represents the pixel value of the coordinate (x, y).

4. The fast underwater video enhancement processing method based on DCP according to claim 3, wherein, The transmission graph according to the R channel Further determine the transmission graphs of the G and B color channels including: Obtain the transmission diagrams of the R, G, and B color channels according to the following formula Among them, (x, y) represents the horizontal and vertical coordinates of each point in the image; β R , β G , β B respectively represent the attenuation coefficients of light waves in the R, G, and B channels in water; b(R), b(G), and b(B) represent the wavelengths of red, green, and blue light waves; B R , B G , B B represent the values corresponding to the three color channels in the background light; According to the transmission diagrams of the three color channels of R, G, and B Generate an optimized transmission diagram.

5. The fast underwater video enhancement processing method based on DCP according to claim 4, wherein, The obtaining a first enhanced image according to the background light and the optimized transmission map includes: Among them, J C (x, y) is the first enhanced image; B C is the corresponding background light of each color channel; t C (x, y) represents the transmission map corresponding to channel C, where channel C includes three color channels: R, G, and B.

6. An electronic device, the electronic device comprising: One or more processors, a memory, the memory being used to store one or more computer programs; characterized in that the computer programs are configured to be executed by the one or more processors, and the program includes steps for implementing the fast underwater video enhancement processing method based on DCP according to any one of claims 1 to 5.

7. A computer-readable storage medium, wherein, At least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement the steps of the fast underwater video enhancement processing method based on DCP according to any one of claims 1 to 5.

Images