An underwater halo image sharpening method based on radial gradient iterative network

Abstract

The application discloses a kind of underwater halo image sharpening method based on radial gradient iterative network, belong to underwater image processing technical field, mainly include: constructing underwater halo image sharpening model based on radial gradient iterative network, including halo extraction network and enhancement network;Based on the first model training dataset, the halo extraction network is trained, and loss function is constructed based on radial gradient constraint iterative weight regularization to guide the extraction halo map during training;The second model training dataset is based on the enhancement network is trained;Acquire the underwater halo image to be processed, the underwater halo image to be processed is input based on radial gradient iterative network underwater halo image sharpening model, to obtain clear underwater image.The application can effectively remove the halo effect caused by artificial light source, reduce the serious interference caused by subsequent computer vision task, and significantly improve underwater image quality.

Description

Technical Field

[0001] The present invention relates to the technical fields of ocean environment perception, digital image processing, and underwater image clarity, and in particular to an underwater halo image clarity method based on a radial gradient iterative network. Background Art

[0002] With the development of human exploration of the ocean, underwater robots have become an important tool for obtaining seabed information. However, due to the strong absorption and scattering of light by water and the complex underwater environment, the underwater environment is severely low in light. Therefore, underwater robots often need to carry artificial lighting devices for auxiliary illumination to improve visibility of the surrounding environment. However, the introduction of artificial light sources can cause problems such as brightness vignetting and low contrast in the captured images, resulting in poor image quality, which greatly affects subsequent visual tasks such as feature extraction and edge detection. Although existing underwater image enhancement methods have developed rapidly, few have considered the serious impact of artificial light sources in real underwater operating environments, resulting in poor practicality. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the present invention provides a method for clearing underwater halo images based on radial gradient iterative network. According to the similarity of the brightness distribution characteristics of land halo images and drawing on the formation principle of land halo images, the underwater halo image is defined as a low-quality underwater image without halo. low Combined with the multiplicative properties of the halo layer V, it can effectively remove the halo effect caused by strong artificial light sources, reduce its serious interference to subsequent computer vision tasks, and significantly improve the quality of underwater images.

[0004] The technical means adopted in the present invention are as follows:

[0005] A method for clearing underwater halo images based on a radial gradient iterative network comprises the following steps:

[0006] S1. Constructing an underwater halo image clarity model based on a radial gradient iterative network, wherein the underwater halo image clarity model based on the radial gradient iterative network includes a halo extraction network and an enhancement network. The halo extraction network is used to decompose an underwater image with halo into a pixel product of a halo map and a low-quality underwater image. The enhancement network is used to enhance the low-quality underwater image output by the halo extraction network.

[0007] S2. Construct a first model training data set, where the first model training data set includes underwater images with halos and real halos layers;

[0008] S3. Training the halo extraction network based on the first model training data set, constructing a loss function based on radial gradient constrained iterative weight regularization to guide the extraction of the halo image during training;

[0009] S4. Construct a second model training data set, where the second model training data set includes low-quality underwater images and clear underwater images output by the halo extraction network;

[0010] S5. Training the enhancement network based on the second model training data set, calculating a loss value between an image output by the enhancement network and a clear underwater image during training, performing error backpropagation based on the loss value, and updating the weights of the enhancement sub-network;

[0011] S6. Obtain an underwater halo image to be processed, and input the underwater halo image to be processed into an underwater halo image clearing model based on a radial gradient iterative network, so as to obtain a clear underwater image.

[0012] Furthermore, during training, a loss function is constructed based on radial gradient constrained iterative weight regularization to guide the extraction of halo images, including constructing the following loss function to guide model training:

[0013]

[0014] s2=p(s1) p-1

[0015] in, A radial gradient representing the true halo, Represents the radial gradient of the halo in the underwater halo image obtained in the kth iteration, v gt Represents the generated halo, v low represents the halo obtained from the underwater halo image, λ represents the constant coefficient that controls the importance of different items, and p represents the constraint coefficient.

[0016] Furthermore, training the halo extraction network based on the first model training data set also includes calculating a reconstruction loss, performing error back propagation according to the reconstruction loss value, and updating the weights of the halo extraction network. The calculation formula of the reconstruction loss is as follows:

[0017]

[0018] Among them, μ1 represents the constant coefficient that controls the importance of different items, It represents finding the radial gradient of a low-quality underwater image without halo. Radial gradient of an underwater image representing normal lighting.

[0019] Furthermore, training the halo extraction network based on the first model training data set also includes calculating a smoothing loss, performing error back propagation according to the smoothing loss value, and updating the weights of the halo extraction network. The calculation formula of the smoothing loss is as follows:

[0020]

[0021] in, represents the gradient of the extracted halo, A gradient representing a true glow.

[0022] Furthermore, the loss between the image output by the enhanced network and the clear underwater image is calculated during training according to the following formula:

[0023] L EF =λ1||S pre -S normal ||2

[0024]

[0025] Among them, S pre is the image output by the enhanced network, S normal is a clear underwater image, and the reconstruction loss L EF ,λ1λ2λ3 are constant coefficients that control the importance of different items, SSIM(S pre ,S normal ) structural similarity loss, represent the radial gradients of the predicted underwater image and the real underwater image, respectively.

[0026] Furthermore, the halo extraction network goes through a five-layer network structure in each iteration. The first to fourth network layers are all convolution layers with a convolution kernel size of 3*3 and a Relu layer, and the fifth layer is a convolution layer with a convolution kernel size of 3*3; the input of the first network layer is a low-quality underwater image with halo, and the output result of the first network layer will be used as the input of the second, third, fourth, and fifth network layers. The input of the second network layer is the output of the first layer, the input of the third network layer is the sum of the output results of the first and second network layers, the input of the fourth network layer is the sum of the output results of the first, second and third network layers, and the input of the fifth network layer is the sum of the output results of the first, second, third and fourth network layers.

[0027] Furthermore, the enhanced network includes a shallow feature extraction block, an independent dense block, a global residual learning block and a global feature fusion block. Except for the convolution layer with a convolution kernel size of 1*1 in the global feature fusion block, the rest are convolution layers with a convolution kernel size of 3*3. In the network, the features are downsampled to different scales, and then processed, desampled and fused from small scale to large scale.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] This paper proposes a radial gradient underwater halo image sharpening method based on an iterative optimization network. This method effectively avoids the common drawbacks of existing underwater image sharpening methods, such as overexposure of halo areas and underenhancement of dark background areas caused by artificial light sources. While removing haloes, it also improves underwater image quality. This method is both effective and robust, laying a theoretical and technical foundation for subsequent visual tasks such as panoramic seafloor observation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0031] Figure 1 The figure is a flow chart of a method for clearing underwater halo images based on a radial gradient iterative network in an embodiment of the present invention.

[0032] Figure 2 This is a diagram of the architecture of an underwater halo image clarity model based on a radial gradient iterative network in an embodiment of the present invention.

[0033] Figure 3 This is a halo image extracted by the underwater halo image clarity model based on the radial gradient iterative network in an embodiment of the present invention.

[0034] Figure 4 This is a clear underwater image output by the underwater halo image clearing model based on the radial gradient iterative network in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0036] It is to be understood that the terminology "first", "second" and the like used in the specification and the claims of the application as well as the foregoing drawings is merely intended to distinguish between similar objects and not necessarily for describing a special sequential order. It is to be understood that the use of the singular herein includes the plural unless specifically stated otherwise. The use of the term "including", "comprising" and variations thereof as well as the terms "consisting of" and variations thereof are intended to cover the meanings as taken in the art, for example, the process, method, system, product or apparatus including a series of steps or units is not necessarily limited to those clearly listed steps or units, but can include other steps or units not clearly listed or inherent to such process, method, product or apparatus.

[0037] As shown in Figure 1 The present application provides an underwater halo image sharpening method based on a radial gradient iterative network, mainly comprising the following steps:

[0038] S1, constructing an underwater halo image sharpening model based on a radial gradient iterative network, the model comprising a halo extraction network and an enhancement network, the halo extraction network being used to decompose the underwater image with halo into a halo image and a low-quality underwater image with uniform illumination, and the enhancement network being used to perform color and detail enhancement processing on the low-quality underwater image to obtain a restored clear underwater image.

[0039] According to the similarity of the brightness distribution characteristics of the terrestrial vignetting image and the underwater halo image, that is, the gradually decreasing characteristics in the radial direction from the center of the light source to the surrounding, and by referring to the formation principle of the terrestrial vignetting image and the low-illumination image, the underwater halo image is defined as a halo layer and a low-quality underwater image in the present application. Therefore, there are:

[0040] S low =I low ·V low

[0041] Wherein, S low represents the underwater halo image, I low represents the low-quality underwater image without halo, V low represents the underwater halo image.

[0042] The halo extraction network in the application comprises: the first layer network to the fourth layer network are all convolution layers with a convolution kernel size of 3*3 and a Relu layer; the fifth layer is a convolution layer with a convolution kernel size of 3*3; the input of the first layer network is a low-quality underwater image with halo, and the output result of the first layer network is used as the input of the second layer network, the third layer network, the fourth layer network and the fifth layer network; the input of the second layer network is the output of the first layer network. The input of the third layer network is the sum of the output results of the first layer network and the second layer network. The input of the fourth layer network is the sum of the output results of the first layer network, the second layer network and the third layer network. The input of the fifth layer network is the sum of the output results of the first layer network, the second layer network, the third layer network and the fourth layer network.

[0043] The enhancement network adopts a multi-scale residual dense network as a backbone network, can aggregate multi-scale information, and obtain greater range information. The basic structure is a residual dense network mainly comprising four parts: a shallow feature extraction network, an independent dense block, global residual learning and global feature fusion. Except that the convolution kernel size of the convolution layer for local and global feature fusion is 1*1, the rest is a convolution layer with a convolution kernel size of 3*3. In the network, the features are down-sampled to different scales (1 / 8, 1 / 4, 1 / 2 and 1), and then processed, de-sampled and fused from small scale to large scale.

[0044] In the application, the halo extraction network and the enhancement network need to be trained, the synthesized underwater image data set with halo is divided into a training set and a test set, the underwater image with uniform illumination, the real halo image and the underwater halo image are used as the data set for training, and the network model of the application is trained. Specifically, the underwater image data with halo for training in the training set and the underwater image with uniform illumination are input into the halo extraction network for processing, and the halo image V low , the low-quality underwater image without halo I low The input of the overall network model of the application comprises: (1) a real underwater image data set without halo, denoted as S normal ; (2) a random halo image V gt simulated by matlab; (3) an underwater halo image S low ; (4) inputting the underwater image with halo into the halo extraction sub-network to output the halo image V low , the low-quality underwater image without halo I low The specific content of network training is described by the following steps.

[0045] S2, a first model training data set is constructed, and the first model training data set comprises underwater images with halo and real halo layers.

[0046] Specifically, the input of the halo extraction network is an underwater image with halo Slow , real underwater image S normal and simulated halo map V gt Therefore, the first model training data set should include the above data.

[0047] S3. Training the halo extraction network based on the first model training data set. During training, a loss function is constructed based on radial gradient constrained iterative weight regularization to guide the extraction of the halo image.

[0048] In this embodiment, the specific steps of training the halo extraction network include:

[0049] S301, constructing an underwater halo extraction sub-network, setting parameters for the network model, and constructing a training set and a test set. The training set includes underwater images S with halo. low And the real underwater image S normal , simulated halo image V gt ;

[0050] S302: Input the input image of the training set into the halo extraction network for training to obtain a halo image V low and low-quality underwater images without halo I low .

[0051] S303, calculate and output halo image V low and the real target image V gt The loss value between the two is used, and the error is back-propagated according to the loss value to update the weights of the halo extraction network.

[0052] S304. Determine whether the halo extraction network has been trained. If so, obtain the trained network model and execute S305. Otherwise, return to S302.

[0053] S305 , input the test set into the trained halo extraction network model for testing, and determine whether the trained network model meets the expected requirements based on the test results. If so, execute S306 , otherwise return to S302 .

[0054] S306: Input the underwater image with halo into the halo extraction network after the test to obtain a halo image V low , low-quality underwater image without halo I low .

[0055] During the halo extraction network training process, radial gradient constrained iterative weight regularization is used to improve the gradient sparsity of the convolution output image.

[0056] The radial gradient refers to the gradient of a pixel in an image along the radial direction relative to the center of the image light source. The radial gradient at any pixel (x, y) in the image can pass through the center of the light source (x0, y0). The radial gradient expression is as follows:

[0057]

[0058] In the above formula, Ψ(x,y) represents the radial gradient value of the image S at the pixel (x,y), D represents the gradient operator, and D*V is the gradient of the halo image. represents the vector formed by the pixel (x, y) and the light source center (x0, y0), * represents convolution, and · represents the inner product operator.

[0059] Based on the above, we constructed the loss function for extracting halo as follows:

[0060]

[0061] Inspired by the solution of non-convex optimization subproblems by iterative reweighted least squares method, we introduce radial gradient constrained iterative weight regularization to improve the gradient sparsity of CNN output images. We transform the above formula into the following iterative optimization process:

[0062]

[0063] s2=p(s1) p-1 (3)

[0064] in A radial gradient representing the true halo, Represents the radial gradient of the halo in the underwater halo image obtained in the kth iteration, v gt Represents the generated halo, v low represents the halo obtained from the underwater halo image, λ represents the constant coefficient that controls the importance of different items, and p represents the constraint coefficient, which is empirically set to p = 0.8.

[0065] In this application, if Figure 2 As shown, L rg Represents the radial gradient loss between the desired halo image and the true halo image, which is the value in the above formula. W k ,W k+1 ,W k+2 Represents the weight factor obtained by iterative formula, W k represents the weight factor of the Kth iteration. Ψ is the radial gradient obtained according to formula (1).

[0066] The above formula uses the Iterative Reweighted Least Squares (IRLS) technique. IRLS treats the optimization as a series of standard least squares problems, each of which uses a weight factor based on the solution of the previous iteration, where w 1 =1, V gt This represents the ground truth of the halo image for the constructed dataset. We used a fully convolutional network, iteratively constructed through a cascaded residual network, to extract the desired halo image by inputting underwater images with haloes into the network and guiding training with the aforementioned loss function. After multiple experiments, we found that the best halo image was obtained when the number of iterations was 3 and p = 0.8, and these parameters were used in this application.

[0067] In order to preserve the details and colors of the low-quality underwater image without halo after removing halo, we also designed a reconstruction loss to make the underwater image after removing halo closer to the underwater image with uniform illumination. The reconstruction loss formula is as follows:

[0068]

[0069] Among them, μ1 represents the constant coefficient that controls the importance of different items, It represents finding the radial gradient of a low-quality underwater image without halo. Radial gradient of an underwater image representing normal lighting.

[0070] The halo image extracted from the halo extraction network should be smooth and not contain too many details, so we introduce a smoothing loss. The smoothing loss formula is as follows:

[0071]

[0072] in, represents the gradient of the extracted halo, A gradient representing a true glow.

[0073] S4. Construct a second model training dataset, where the second model training dataset includes low-quality underwater images and clear underwater images output by the halo extraction network.

[0074] Specifically, the second model training dataset includes low-quality underwater images S obtained by the halo extraction network. low , and GroundTruth's clear underwater images S normal .

[0075] S5. Train the enhancement network based on the second model training data set. During training, calculate the loss value between the image output by the enhancement network and the clear underwater image, perform error backpropagation according to the loss value, and update the weight of the enhancement sub-network.

[0076] Specifically, the training steps of the enhanced network include:

[0077] S501: Input the input image of the training set into the enhancement sub-network for training to obtain an enhanced and clear underwater image S pre .

[0078] S502, calculate the image S output by the enhancement sub-network pre and the real target image S normal The loss value between the two networks is used, and error back propagation is performed according to the loss value to update the weight of the enhanced sub-network.

[0079] S504: Determine whether the enhanced sub-network is trained. If so, obtain the trained enhanced sub-network and execute S505; otherwise, return to S502.

[0080] S505: Input the test set into the trained reflectance map restoration subnetwork for testing, and determine whether the trained reflectance map restoration subnetwork meets the expected requirements based on the test results. If so, execute S506; otherwise, return to S502;

[0081] S506, the low-quality underwater image S obtained by the halo extraction sub-network low Input into the trained enhancement sub-network to obtain the enhanced underwater image S pre .

[0082] During the training process, the output image S is calculated using the following loss function: pre Between the target image S normal The loss value is:

[0083] L EF =λ1||S pre -S normal ||2 (6)

[0084]

[0085] Among them, S pre is the image output by the enhanced network, S normal is a clear underwater image, and the reconstruction loss L EF ,λ1λ2λ3 are constant coefficients that control the importance of different items, SSIM(S pre ,S normal ) structural similarity loss, represent the radial gradients of the predicted underwater image and the real underwater image, respectively.

[0086] Formula (6) is used to constrain the enhanced underwater image S pre Compared with the real underwater image S normalGlobal feature difference, formula (7) uses SSIM structural similarity loss function to calculate the enhanced underwater image S pre Compared with the real underwater image S normal The SSIM index between them is used as the loss and in order to maintain the consistency between the enhanced underwater image and the Ground Truth, a loss function of the radial gradient distribution is constructed, and the L2 norm of the radial gradient distribution of the enhanced underwater image and the clear underwater image is calculated.

[0087] S6. Obtain an underwater halo image to be processed, and input the underwater halo image to be processed into an underwater halo image clearing model based on a radial gradient iterative network, so as to obtain a clear underwater image.

[0088] The scheme and effects of the present invention are further illustrated below through specific application examples.

[0089] S100, select and construct an existing synthetic underwater image S with halo low ;

[0090] S200, the underwater image S with halo low and the real underwater image S normal Input the halo extraction network to obtain the estimated halo map V low , low-quality underwater image S without halo low The steps are as follows:

[0091] The halo map is obtained by using radial gradient and iterative network with iterative reweighted least squares (IRLS) technique. low and the real halo map V gt The loss between the two will result in a low-quality underwater image I low and uniformly illuminated underwater image S normal Extract halo using radial gradient loss constraint. Obtain halo image V through network learning. low and low-quality underwater image S without halo low .

[0092] S300: detail enhancement and color restoration, converting the low-quality underwater image S200 without halo low The input is sent to the enhancement and restoration sub-network, where the underwater image is enhanced to obtain the underwater image S after detail enhancement and color correction. pre The steps are as follows:

[0093] The enhanced underwater image S pre and underwater image S normal Make loss function and calculate enhanced underwater image S pre and underwater image S normalStructural similarity loss.

[0094] The underwater halo image correction network is implemented based on the halo extraction network and the enhancement sub-network. The training and testing process includes the following steps:

[0095] S001. Build the above network model based on the pytorch deep learning framework, set the parameters of the network model, and construct a training set and a test set. 890 underwater images containing halos are synthesized in the training set, and 800 of them are set as the training set, and the remaining 90 are set as the test set.

[0096] S002. Input the input image of the training set into the network model for training, calculate the loss value between the output image and the true target image, perform error back propagation according to the loss value, and update the weight of the network model.

[0097] S003. Determine whether the network model has been trained. If so, obtain the trained network model and execute S004. Otherwise, return to S002.

[0098] S004, input the test set into the trained network model for testing, and judge whether the trained network model meets the expected requirements based on the test results. If so, execute S005, otherwise return to S002;

[0099] S005. Input the captured image with artificial light source into the trained network model to obtain a clear underwater image with uniform lighting distribution.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for clearing underwater halo images based on radial gradient iterative network, characterized in that: The following steps are involved: S1. Constructing an underwater halo image clarity model based on a radial gradient iterative network, wherein the underwater halo image clarity model based on the radial gradient iterative network includes a halo extraction network and an enhancement network. The halo extraction network is used to decompose an underwater image with halo into a pixel product of a halo map and a low-quality underwater image. The enhancement network is used to enhance the low-quality underwater image output by the halo extraction network. S2. Construct a first model training data set, where the first model training data set includes underwater images with halos and real halos layers; S3. Training the halo extraction network based on the first model training data set, constructing a loss function based on radial gradient constrained iterative weight regularization to guide the extraction of the halo image during training, including constructing the following loss function to guide model training: s2=p(s1) p-1 in, A radial gradient representing the true halo, Represents the radial gradient of the halo in the underwater halo image obtained in the kth iteration, v gt Represents the generated halo, v low represents the halo obtained from the underwater halo image, λ represents the constant coefficient that controls the importance of different items, and p represents the constraint coefficient; S4. Construct a second model training data set, where the second model training data set includes low-quality underwater images and clear underwater images output by the halo extraction network; S5. Training the enhancement network based on the second model training data set, calculating a loss value between an image output by the enhancement network and a clear underwater image during training, performing error backpropagation based on the loss value, and updating the weights of the enhancement sub-network; S6. Obtain an underwater halo image to be processed, and input the underwater halo image to be processed into an underwater halo image clearing model based on a radial gradient iterative network, so as to obtain a clear underwater image.

2. The underwater halo image clearing method based on radial gradient iterative network according to claim 1, characterized in that: The halo extraction network is trained based on the first model training data set, further comprising calculating a reconstruction loss, performing error back propagation according to the reconstruction loss value, and updating the weights of the halo extraction network. The calculation formula of the reconstruction loss is as follows: Among them, μ1 represents the constant coefficient that controls the importance of different items, It represents finding the radial gradient of a low-quality underwater image without halo. Radial gradient of an underwater image representing normal lighting.

3. The underwater halo image clearing method based on radial gradient iterative network according to claim 1, characterized in that: The halo extraction network is trained based on the first model training data set, further comprising calculating a smoothing loss, performing error back propagation according to the smoothing loss value, and updating the weights of the halo extraction network. The calculation formula of the smoothing loss is as follows: in, represents the gradient of the extracted halo, A gradient representing a true glow.

4. The underwater halo image clearing method based on radial gradient iterative network according to claim 1, characterized in that: Each iteration of the halo extraction network goes through a five-layer network structure. The first to fourth layers of the network are all convolution layers with a convolution kernel size of 3*3 and a Relu layer. The fifth layer is a convolution layer with a convolution kernel size of 3*3; the input of the first layer of the network is a low-quality underwater image with halo, and the output result of the first layer of the network will be used as the input of the second, third, fourth, and fifth layers of the network. The input of the second layer of the network is the output of the first layer, the input of the third layer of the network is the sum of the output results of the first and second layers of the network, the input of the fourth layer of the network is the sum of the output results of the first, second and third layers of the network, and the input of the fifth layer of the network is the sum of the output results of the first, second, third and fourth layers of the network.

5. The underwater halo image clearing method based on radial gradient iterative network according to claim 1, characterized in that: The enhanced network includes a shallow feature extraction block, an independent dense block, a global residual learning block and a global feature fusion block. Except for the convolution layer with a convolution kernel size of 1*1 in the global feature fusion block, the rest are convolution layers with a convolution kernel size of 3*3. In the network, the features are downsampled to different scales and then processed, desampled and fused from small scale to large scale.

Images