A method for real-time underwater image enhancement using curve iterative estimation
By combining iterative curve estimation and a lightweight network structure with a multi-loss function underwater image enhancement method, the problems of model complexity and long running time in existing technologies are solved, achieving real-time underwater image enhancement and improving image quality.
Patent Information
- Application Number
- CN202310675798.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-08
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-06-08
AI Technical Summary
Existing underwater image enhancement algorithms suffer from high model complexity, numerous parameters, long running time, and unstable results, making it difficult to effectively improve image quality in practical applications.
An iterative curve estimation method is adopted. By training a curve parameter network and combining content, perception, sharpness and color loss functions, the mapping relationship between the original image and the enhanced image is simulated. Image enhancement is performed using a lightweight convolution-activation function stacked block structure.
The model parameters have been simplified, the running time has been reduced, and the image enhancement effect has been improved, realizing real-time underwater image enhancement. It is suitable for small equipment and for seabed environment detection and underwater vision tasks.
Smart Images

Figure CN116777773B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of underwater image enhancement technology, and more specifically to a method for real-time underwater image enhancement using curve iterative estimation. Background Technology
[0002] In recent years, the development and utilization of marine resources has attracted increasing attention. Due to the complexity of the underwater environment, underwater images are often affected by various factors such as light attenuation, scattering, and absorption, leading to a decline in image quality and making it difficult to accurately obtain the required information. This poses challenges to subsequent underwater operations. Underwater image enhancement improves the visual perception quality of images, providing a foundation for subsequent tasks. Effective enhancement of underwater images can help people better explore and develop the marine world.
[0003] Current underwater image enhancement algorithms can be categorized into underwater imaging model-driven enhancement methods, model-free traditional underwater image enhancement methods, and data-driven deep learning-based image enhancement methods. Due to the complexity of the underwater environment, for underwater imaging model-driven enhancement methods, the more accurate the model, the more parameters need to be computed, which poses significant challenges to their practical application. The biggest problem with model-free traditional underwater image enhancement methods is their tendency to over-enhance or under-enhance images, making it difficult to achieve satisfactory and stable results. In contrast, deep learning-based methods generally achieve more stable results. However, current research focuses on designing complex network models, incorporating more loss functions, or integrating traditional prior knowledge into deep learning networks. These strategies undoubtedly increase the complexity of the methods, even complicating simple low-level vision tasks to some extent, leading to numerous limitations and long runtimes in application.
[0004] Therefore, when enhancing underwater images, how to reduce the number of model parameters, simplify the steps, improve the image enhancement effect, and reduce the running time is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of the above problems, the present invention provides a method for underwater real-time image enhancement using curve iterative estimation, so as to at least solve some of the technical problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A method for real-time underwater image enhancement using curve iterative estimation includes the following steps:
[0008] S1. Use an iterative curve to simulate the mapping relationship between the original image and the enhanced image, and obtain the curve parameters of the iterative curve;
[0009] S2. Estimate the curve parameters of the iterative curve using the trained curve parameter estimation network;
[0010] S3. Apply the estimated curve parameters iteratively to the color channels of the original image to obtain the final underwater enhanced image.
[0011] Furthermore, the function corresponding to the iterative curve is monotonic and can achieve gradient backpropagation; when the input is normalized to [0, 1], the output of the function is also kept constant in [0, 1].
[0012] Furthermore, in step S1, an iterative curve is used to simulate the mapping relationship between the original image and the enhanced image, specifically including:
[0013] S11. Use curves to simulate the low-order mapping between the original image and the enhanced image, represented as:
[0014] E θ (x)=α θ (x)R θ (x) 2 +(1-α θ (x))R θ (x)
[0015] Where θ∈{R, G, B}; R θ (x) represents the pixel value of the original image; E θ (x) represents the pixel value of the enhanced image; α θ (x) represents the curve parameters;
[0016] S12. Iterate the curve to achieve a higher-order mapping between the simulated original image and the enhanced image, as shown below:
[0017] E n (x)=α n (x)E n-1 (x) 2 +(1-α n (x))E n-1 (x)
[0018] Where n represents the number of iterations; E n (x) represents the pixel value of the image after the nth iteration of enhancement; α n (x) represents the curve parameters obtained from the nth iteration of the parameter estimation network.
[0019] Furthermore, the curve parameter estimation network consists of multiple convolutional-activation function stacks; these multiple convolutional-activation function stacks are connected by skip connections to form a symmetrical network structure.
[0020] Furthermore, the convolution-activation function stack incorporates an attention mechanism.
[0021] Furthermore, the curve parameter estimation network is trained using a set of loss functions, which includes a content loss function, a perceptual loss function, a sharpness loss function, and a color loss function.
[0022] Furthermore, each loss function in the loss function group is constructed as follows:
[0023] (1) The content loss function is constructed as follows:
[0024] L gt =||YE||1
[0025] Among them, L gt Y represents the content loss function; E represents the pixel value of the enhanced image; Y represents the pixel value of the ground truth (GT), and GT represents the label referenced to the enhanced image.
[0026] (2) The construction method of the perceptual loss function is expressed as follows:
[0027] L perce =||Φ(Y)-Φ(E)||2
[0028] Among them, L perce Φ(U) represents the perceptual loss function; Φ(U) represents the features extracted by GT based on the relu5_4 layer of the VGG-19 network; Φ(E) represents the features extracted by the augmented image based on the relu5_4 layer of the VGG-19 network.
[0029] (3) The sharpness loss function is constructed as follows:
[0030]
[0031] Among them, L sharp This represents the sharpness loss function; This represents the gradient value of the ground truth (GT). This represents the gradient value of the enhanced image;
[0032] (4) The color loss function is constructed as follows:
[0033] L color =(I R -I G ) 2 +(I R -IB ) 2 +(I G -I B ) 2
[0034] Among them, L color Represents the color loss function; I R I represents the average pixel intensity of the R channel of the image; G I represents the average pixel intensity of the G channel of the image; B This represents the average pixel intensity of the B channel of the image.
[0035] Furthermore, the curve parameters include R-channel curve parameters, G-channel curve parameters, and B-channel curve parameters.
[0036] Furthermore, in step S3, based on the R-channel curve parameters, G-channel curve parameters, and B-channel curve parameters, iterations are performed on the corresponding R-channel, G-channel, and B-channel, respectively.
[0037] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method for underwater real-time image enhancement using curve iterative estimation, which has the following beneficial effects:
[0038] This invention uses an iterative curve to simulate the mapping relationship between the original image and the enhanced image, and obtains the final underwater enhanced image based on the trained parameters of the iterative curve. The process is simple and easy to use. This method can reduce the number of model parameters and reduce the running time while improving the enhancement effect.
[0039] The curve parameter estimation network of this invention consists of multiple stacked blocks of convolutional-activation functions. Its lightweight network structure can be easily generalized to any small device, enabling real-time underwater image enhancement. This facilitates the development of seabed environment detection and underwater vision missions.
[0040] This invention constrains the training of a curve parameter estimation network using a content loss function, a perceptual loss function, a sharpness loss function, and a color loss function. The content loss function effectively prevents image blurring; the perceptual loss function ensures that the texture of the enhanced image is more refined; the sharpness loss function improves the image's clarity and contrast; and the color loss function constrains the generation of visually perceptually close-to-natural images. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0042] Figure 1 This is a schematic flowchart of a method for underwater real-time image enhancement using curve iterative estimation, provided in an embodiment of the present invention.
[0043] Figure 2 This is a schematic diagram of the curve parameter estimation network provided in an embodiment of the present invention.
[0044] Figure 3 This is a schematic diagram illustrating the usage process of the underwater real-time image enhancement method provided in an embodiment of the present invention.
[0045] Figure 4 This is a schematic diagram comparing the original underwater image and the enhanced image provided in an embodiment of the present invention. Detailed Implementation
[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0047] See Figure 1 As shown, this embodiment of the invention discloses a method for real-time underwater image enhancement using curve iterative estimation, comprising the following steps:
[0048] S1. Use iterative curves to simulate the mapping relationship between the original image and the enhanced image, and obtain the curve parameters of the iterative curves;
[0049] S2. Estimate the curve parameters of the iterative curve using the trained curve parameter estimation network;
[0050] S3. Iteratively apply the estimated curve parameters to the color channels of the original image to obtain the final underwater enhanced image.
[0051] The steps described above will now be explained in detail.
[0052] In step S1 above, the iterative curve used should satisfy the following condition:
[0053] 1. The curve should be as simple as possible, with few parameters, and the function corresponding to the curve should be monotonic and able to achieve gradient backpropagation, so as to enable the training of the parameter estimation network.
[0054] 2. This ensures that the function corresponding to the curve is monotonic, making the learning of the mapping more effective and simpler, while also maintaining image contrast. When the input is normalized to [0, 1], the output also remains constant in [0, 1], thus preventing pixel value overflow and the generation of invalid information.
[0055] In this step, an iterative curve is used to simulate the mapping relationship between the original image and the enhanced image, specifically including:
[0056] S11. Use curves to simulate the low-order mapping between the original image and the enhanced image, represented as:
[0057] E θ (x)=α θ (x)R θ (x) 2 +(1-α θ (x))R θ (x)
[0058] Where θ∈{R, G, B}; R θ (x) represents the pixel value of the original image; E θ (x) represents the pixel value of the enhanced image; α θ (x) represents the curve parameter to be solved;
[0059] S12. Iterate the curve to achieve a higher-order mapping between the simulated original image and the enhanced image, represented as:
[0060] E n (x)=α n (x)E n-1 (x) 2 +(1-α n (x))E n-1 (x)
[0061] Where n represents the number of iterations; E n (x) represents the pixel value of the image after the nth iteration of enhancement; α n (x) represents the curve parameters of the nth iteration obtained from the parameter estimation network; theoretically, higher-order polynomials can infinitely approximate any curve.
[0062] In step S2 above, the curve parameter estimation network consists of multiple convolutional activation function stacks. These stacks form a symmetrical network structure through skip connections, and the stacks incorporate an attention mechanism. In this embodiment, the network has only 70,767 parameters, a very small order of magnitude. For a 240*320*3 input image, flops = 5.42G. This lightweight network significantly reduces the limitations on application platforms, enabling real-time underwater image enhancement on embedded devices. A schematic diagram of the curve parameter estimation network can be found here. Figure 2 As shown;
[0063] The above curve parameter estimation network is trained using a set of loss functions; this set of loss functions includes content loss, perceptual loss, sharpness loss, and color loss; expressed as:
[0064] L total =λ1L gt +λ2L perce +λ3L sharp +λ4L color
[0065] Among them, L gt L represents the content loss function; perce L represents the perceptual loss function; sharp L represents the sharpness loss function; color λ1 represents the color loss function; λ2 represents the control factor of the content loss function; λ3 represents the control factor of the sharpness loss function; λ4 represents the control factor of the color loss function; based on these four control factors, the overall loss function can be balanced.
[0066] The loss function in the loss function group is constructed as follows:
[0067] (1) The content loss function is constructed as follows:
[0068] L gt =||YE||1
[0069] Among them, L gt The content loss function is represented by E; the pixel value of the enhanced image is represented by Y; the pixel value of GT is represented by GT, which represents the label referenced by the enhanced image. This content loss function is constructed based on the existing L1 loss function and can effectively prevent image blurring.
[0070] (2) The construction method of the perceptual loss function is expressed as follows:
[0071] L perce =||Φ(Y)-Φ(E)||2
[0072] Among them, L perce Let represent the perceptual loss function; Φ(Y) represent the features extracted by the ground truth based on the ReLU5_4 layer of the pre-trained VGG-19 network; Φ(E) represent the features extracted by the augmented image based on the ReLU5_4 layer of the pre-trained VGG-19 network. L perce It is constructed based on the constraint that the enhanced image and the ground truth should have similar feature representations. By constructing this perceptual loss function, it is possible to ensure that these features are preserved during the enhancement process, so that the enhanced image can generate more refined results in terms of features such as texture.
[0073] (3) The sharpness loss function is constructed as follows:
[0074]
[0075] Among them, L sharp This represents the sharpness loss function; This represents the gradient value of the ground truth (GT). This represents the gradient value of the enhanced image; its specific construction is as follows:
[0076]
[0077] in P represents the gradient value of the image; x P represents the gradient of the image in the x-direction obtained using the Sobel operator. y This represents the gradient of the image in the y-direction obtained using the Sobel operator. This sharpness loss function measures the blurriness of an image by calculating the gradient, and it can improve the sharpness and contrast of an image to a certain extent.
[0078] (4) The color loss function is constructed as follows:
[0079] L color =(I R -I G ) 2 +(I R -I B ) 2 +(I G -I B ) 2
[0080] Among them, L color Represents the color loss function; I R I represents the average pixel intensity of the R channel of the image; G I represents the average pixel intensity of the G channel of the image; BThis represents the average pixel intensity of the B channel of the image. The color loss function is constructed based on the color constancy theory, which assumes that a normal, natural image with good visual perception quality has similar mean and histogram distributions across all its color channels. This color loss function can constrain the generation of visually perceptually close-to-natural images.
[0081] The embodiments of the present invention constrain the training of the curve parameter estimation network by using the above-described loss function set.
[0082] In step S3 above, during the underwater image imaging process, due to the absorption of light by water, the light energy attenuates during transmission. Under normal circumstances, red light attenuates the fastest in water, while blue and green light attenuate more slowly. Therefore, the red channel (i.e., the R channel) is generally the channel with the most severe underwater image attenuation and should be the channel that receives the most compensation.
[0083] The range of parameter variation for the red channel should be wider. In this embodiment of the invention, based on the curve parameters of the R channel, G channel, and B channel, iteration is performed on the corresponding R channel, G channel, and B channel respectively, thereby applying different curve parameters to achieve the mapping from the original image to the enhanced image, thus completing the underwater image enhancement; and the image enhanced by the method provided in this embodiment of the invention can achieve key compensation of the red channel.
[0084] Figure 3 This is a schematic diagram illustrating the usage flow of the underwater real-time image enhancement method provided in this embodiment of the invention. First, the original underwater image is input into a curve parameter estimation network, which outputs curve parameters for the R, G, and B channels, respectively. Then, different parameterized curves are applied through three iterations, one for each color channel, to obtain the final enhanced underwater image. The curve parameter estimation network is trained by constructing a combination of content loss, perceptual loss, gradient loss, and color loss.
[0085] In one embodiment, such as Figure 4 (a) is the original underwater image. Figure 4 (b) is the image enhanced by the method of the present invention, by... Figure 4 As can be seen, the underwater image enhancement method proposed in this invention can fully restore the natural tone of the image without over-enhancing, and the texture is significantly clearer, effectively eliminating the blurriness in the original underwater image.
[0086] In one embodiment, the underwater real-time image enhancement method using curve iterative estimation proposed in this invention is real-time. As can be seen from Table 1, the method provided by this invention has a very fast processing speed, which is orders of magnitude faster than the current mainstream image enhancement algorithms. The method of this invention can achieve true real-time underwater image enhancement.
[0087] Table 1
[0088] method Runtime (seconds) Based on the minimum color loss method 0.050 Based on the hyperLaplace prior method 0.655 Convolutional Network-based Methods 1.153 Generative Adversarial Networks 0.0275 Image quality assessment method 0.601 This invention 0.002
[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0090] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for real-time image enhancement underwater using curve iterative estimation, characterized in that, Includes the following steps: S1. Use an iterative curve to simulate the mapping relationship between the original image and the enhanced image, and obtain the curve parameters of the iterative curve; S2. Estimate the curve parameters of the iterative curve using the trained curve parameter estimation network; S3. Apply the estimated curve parameters iteratively to the color channels of the original image to obtain the final underwater enhanced image; The curve parameter estimation network is trained using a set of loss functions, including a content loss function, a perceptual loss function, a sharpness loss function, and a color loss function. The content loss function, perceptual loss function, and sharpness loss function are all constructed based on the difference between the enhanced image and the label GT referenced by the enhanced image; The loss function in the loss function group is constructed as follows: (1) The content loss function is constructed as follows: ; wherein, represents the content loss function; E represents the pixel value of the enhanced image; Y represents the pixel value of GT, and GT represents the label to which the enhanced image refers. (2) The construction method of the perceptual loss function is expressed as follows: ; wherein, represents a perceptual loss function; represents GT based features extracted from relu5 4 layer of VGG-19 network; represents enhanced image based features extracted from relu5 4 layer of VGG-19 network; (3) The sharpness loss function is constructed as follows: ; wherein, denotes a sharpness loss function; denotes a gradient value of the GT; denotes a gradient value of the enhanced image; (4) The construction method of the color loss function is expressed as follows: ; wherein, represents a color loss function; represents the average pixel intensity of the image R channel; represents the average pixel intensity of the image G channel; represents the average pixel intensity of the image B channel.
2. The method of real-time image enhancement under water using curve iterative estimation of claim 1, wherein, The function corresponding to the iteration curve is monotonic and can achieve gradient backpropagation; when the input is normalized to [0,1], the output of the function is also kept constant in [0,1].
3. The method of real-time image enhancement under water using curve iterative estimation of claim 1, wherein, In step S1, an iterative curve is used to simulate the mapping relationship between the original image and the enhanced image, specifically including: S11. Use curves to simulate the low-order mapping between the original image and the enhanced image, represented as: ; wherein, ; represents a pixel value of the original image; represents a pixel value of the enhanced image; represents a curve parameter; S12. Iterate the curve to achieve a higher-order mapping between the simulated original image and the enhanced image, as shown below: ; wherein n represents the number of iterations; represents the pixel value of the image after the n-th iteration of enhancement; represents the curve parameters of the n-th iteration obtained by the parameter estimation network.
4. The method of real-time image enhancement under water using curve iterative estimation of claim 1, wherein, The curve parameter estimation network consists of multiple convolutional-activation function stacks; these multiple convolutional-activation function stacks are connected by skip connections to form a symmetrical network structure.
5. The method of real-time image enhancement under water using curve iterative estimation of claim 4, wherein, The stacked blocks of convolutional activation functions incorporate an attention mechanism.
6. The method of real-time image enhancement under water using curve iterative estimation of claim 1, wherein: The curve parameters include R-channel curve parameters, G-channel curve parameters, and B-channel curve parameters.
7. The method of real-time image enhancement under water using curve iterative estimation of claim 6, wherein: In step S3, based on the R-channel curve parameters, G-channel curve parameters, and B-channel curve parameters, iterations are performed on the corresponding R-channel, G-channel, and B-channel, respectively.
Citation Information
Patent Citations
Low-illumination video enhancement method and system and storage medium
CN115619674A
