An underwater image sharpening method for UUV real operation lighting scenes

Abstract

The present invention provides an underwater image sharpening method for UUV real operation lighting scenes, comprising the following steps: constructing an imaging model capable of describing UUV operation lighting scenes; synthesizing an underwater image data set with a single-point light spot, and using the data set to train a sharpening network model; proposing a loss function based on the radial gradient distribution characteristics of the light spot, effectively separating the halo layer and the low-quality underwater image with uniform illumination distribution; introducing a network based on the Retinex theory to decompose the low-quality underwater image to obtain an illumination map and a reflection map; finally, respectively obtaining a reflection map with rich colors and details and an illumination map with brightness restoration through a reflection map restoration network and a brightness adjustment network, and obtaining the final clear image after the two are fused. The present invention can effectively remove the problem of non-uniform illumination caused by artificial light sources, well restore the contrast and color information of the image, significantly improve the quality of underwater images, and has excellent robustness, accuracy and universality in complex marine environments.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater image processing, and in particular to an underwater image clearing method for UUV real operation lighting scenes. Background Art

[0002] The underwater environment is different from that on land. The absorption and scattering of light by the water medium and a large number of suspended particles causes the underwater environment to present the characteristics of low illumination. In order to improve the perception of the surrounding environment, UUVs often need to be equipped with artificial light sources for auxiliary lighting. The introduction of strong light sources will produce obvious light spots in the foreground of the camera, resulting in obvious non-uniform illumination in the captured images. This phenomenon has a great impact on subsequent visual tasks, but it is rarely considered. In view of this, an underwater image sharpening method for UUV real operating lighting scenes is proposed. It has been verified that the algorithm proposed in the present invention has excellent effect and robustness for underwater image sharpening processing with a single-point artificial light source. Summary of the invention

[0003] The present invention provides an underwater image sharpening method for UUV real operation lighting scenes, which can solve the following technical problems: the auxiliary light source produces obvious light spots in the foreground of the camera, resulting in obvious non-uniform lighting phenomenon in the captured image. The present invention can effectively remove the non-uniform lighting problem caused by artificial light sources, well restore the contrast and color information of the image, and significantly improve the underwater image quality.

[0004] The technical means adopted by the present invention are as follows:

[0005] An underwater image clarity method for UUV real operation lighting scenes includes the following steps:

[0006] Construct an underwater imaging model to describe the brightness distribution characteristics of a single spot caused by an artificial light source carried by a UUV;

[0007] According to the underwater imaging model, an underwater image dataset with a single spot is synthesized on a real underwater image dataset;

[0008] An underwater image sharpening network model is trained based on the underwater image data set with a single light spot, and the underwater image sharpening network model includes:

[0009] The spot estimation subnetwork is used to process underwater images with single-point spots. A loss function with radial gradient distribution characteristics is introduced to decompose the halo layer and obtain low-quality underwater images with uniform illumination distribution.

[0010] Retinex-Net decomposition sub-network, which is used to decompose the low-quality underwater image into a reflectance map and an illumination map,

[0011] A reflection map restoration subnetwork is used to enhance the reflection map to obtain a restored reflection map.

[0012] A brightness adjustment subnetwork, which is used to adjust the brightness of the illumination map to obtain a restored illumination map;

[0013] The underwater image data with a single light spot to be processed is obtained, and the data is input into a trained underwater image sharpening network model, and the restored reflection map and the restored illumination map output by the underwater image sharpening network model are multiplied pixel by pixel to output a clear underwater image.

[0014] Furthermore, the training steps of training the underwater image sharpening network model based on the underwater image dataset with a single light spot include:

[0015] The synthetic underwater image dataset with single-point light spots is divided into a training set and a test set;

[0016] The training underwater image data in the training set is input into the spot estimation subnetwork for processing to generate a low-quality underwater image with a training halo layer and uniform illumination distribution;

[0017] Processing the low-quality underwater image for training based on the Retinex-Net decomposition sub-network to generate a reflection map for training and an illumination map for training;

[0018] Based on the reflection map restoration sub-network, the training image is enhanced to obtain a restored reflection map for training;

[0019] Performing brightness adjustment based on the illumination map for training in the brightness adjustment subnetwork to obtain a restored illumination map for training;

[0020] Multiplying the restored reflection image for training and the restored illumination image for training pixel by pixel, thereby outputting a clear underwater image for training;

[0021] The loss value between the output image of each sub-network and the true target image is calculated, and the error is back-propagated based on the loss value to update the weights of the spot estimation sub-network, Retinex-Net decomposition sub-network, reflectance map recovery sub-network and brightness adjustment sub-network.

[0022] Furthermore, the training steps of the light spot estimation sub-network include:

[0023] Acquire training set data, wherein the training set data includes underwater images with light spots, light spot images corresponding to the underwater images with light spots, and underwater images with uniform illumination;

[0024] The light spot estimation sub-network model is trained by using an underwater image with light spots as input data of the light spot estimation sub-network and using the light spot image and an underwater image with uniform illumination as output data;

[0025] The radial gradient distribution loss value of the uniformly illuminated underwater image output by the calculation model and the uniformly illuminated underwater image in the training set is calculated, and error back propagation is performed according to the radial gradient distribution loss value to update the weight of the spot estimation subnetwork.

[0026] Furthermore, the light spot estimation subnetwork includes a six-layer network and two branches. The first to fifth network layers are all convolution layers with a convolution kernel size of 3×3 and a Relu layer; in the sixth network layer, the part of the underwater image with uniform illumination is estimated using a Sigmoid activation function to obtain an output, and the branch for estimating the light spot image uses a TanH activation function as the last layer.

[0027] Furthermore, the training steps of the Retinex-Net decomposition sub-network include:

[0028] Acquire training set data, wherein the training set data includes an underwater image, and an illumination map and a reflectance map corresponding to the underwater image;

[0029] The decomposition network model is trained by using low-quality underwater images as input data of a Retinex-Net decomposition sub-network and using illumination maps and reflectance maps as output data; the Retinex-Net decomposition sub-network includes a reflectance map branch part and an illumination map branch part;

[0030] The loss values ​​of the illumination map and reflectance map output by the calculation model and the illumination map and reflectance map of GroundTruth in the training set are compared, and the error is back-propagated according to the loss value to update the weight of the Retinex-Net decomposition sub-network.

[0031] Furthermore, the training steps of the reflection map recovery sub-network include:

[0032] Acquire training set data, wherein the training set data includes a reflection map output by a Retinex-Net decomposition subnetwork and a reflection map of GroundTruth;

[0033] The reflection map output by the Retinex-Net decomposition sub-network is used as input data of the reflection map recovery sub-network to train the reflection map recovery sub-network;

[0034] The loss value between the restored reflection map output by the reflection map restoration subnetwork and the reflection map of GroundTruth in the training set is calculated, and error backpropagation is performed according to the loss value to update the weight of the reflection map restoration subnetwork.

[0035] Furthermore, the training step of the brightness adjustment sub-network includes:

[0036] Acquire training set data, wherein the training set data includes an illumination map output by a Retinex-Net decomposition subnetwork and an illumination map of GroundTruth;

[0037] The illumination map output by the Retinex-Net decomposition sub-network is used as input data of the brightness adjustment sub-network to train the brightness adjustment sub-network;

[0038] The loss value between the restored illumination map output by the brightness adjustment subnetwork and the illumination map of GroundTruth in the training set is calculated, and error backpropagation is performed according to the loss value to update the weight of the brightness adjustment subnetwork.

[0039] The underwater image sharpening method for UUV real operation lighting scene disclosed in the present invention has the following advantages:

[0040] An optical imaging model suitable for operating scenarios with strong artificial light sources is proposed. Based on the radial gradient distribution characteristics of the light spot brightness, the light spot image can be accurately separated, thereby realizing further restoration of the underwater image after removing the influence of the artificial light source. At the same time, based on the Retinex theory, the underwater image is decomposed into a reflection map and an illumination map. The brightness is enhanced by using a brightness adjustment network in the illumination map, and the details are restored by using an enhancement network in the reflection map. This can effectively improve the quality of underwater images with artificial light sources, and has excellent robustness, accuracy and effectiveness in complex marine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 The present invention is a flow chart of an underwater image clearing method for UUV real operation lighting scenes.

[0043] Figure 2 This is a processing flow chart of an underwater image clarity method for UUV real operation lighting scenes according to the present invention.

[0044] Figure 3 This is a network framework structure diagram of an underwater image clearing method for UUV real operation lighting scenes according to the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] like Figure 1 As shown, the present invention provides an underwater image clarity method for UUV real operation lighting scenes, comprising the following steps:

[0047] S1. Construct an underwater imaging model that can describe the brightness distribution characteristics of the local light spot caused by the artificial light source carried by the UUV.

[0048] Specifically, the underwater imaging model is used to simulate the brightness distribution characteristics of the local light spot caused by the artificial light source. According to the halo correction technology theory, the underwater image S with the light spot low Defined as the underwater image U low With the spot layer V low Multiply pixel by pixel to get the underwater image U low According to the Retinex theory, the reflection graph R low And illumination diagram I low Doing the multiplication operation, we get:

[0049]

[0050] S2. Synthesize an underwater image dataset with a single light spot on a real underwater image dataset according to the underwater imaging model.

[0051] S3. An underwater image sharpening network model proposed based on training of an underwater image dataset with a single-point light spot. The underwater image sharpening network model includes a light spot estimation subnetwork, a Retinex-Net decomposition subnetwork, a reflection map recovery subnetwork, and a brightness adjustment subnetwork. The light spot estimation subnetwork is used to process underwater images with a single-point light spot, and introduces a loss function with a radial gradient distribution characteristic, so as to decompose and obtain a low-quality underwater image with a halo layer and uniform illumination distribution. The Retinex-Net decomposition subnetwork is used to decompose the low-quality underwater image into a reflection map and an illumination map. The reflection map recovery subnetwork is used to enhance the reflection map to obtain a restored reflection map. The brightness adjustment subnetwork is used to adjust the brightness of the illumination map to obtain a restored illumination map.

[0052] The method of the present invention is mainly based on the spot estimation sub-network, the Retinex-Net decomposition sub-network, the reflectance map recovery sub-network and the brightness adjustment sub-network to achieve the function. The training steps of the above network structures include:

[0053] S31, dividing the synthesized underwater image data set with a single light spot into a training set and a test set to train the network model of the present invention.

[0054] S32, inputting the training underwater image data in the training set into the spot estimation sub-network model for processing, and generating a training halo layer and a low-quality underwater image with uniform illumination distribution.

[0055] Specifically, the input of the overall network model of the present invention includes: (1) a real underwater image dataset, denoted as U low ; (2) Random single spot image V simulated using matlab low According to the underwater imaging model proposed above, the simulated spot image V low With underwater image U low Multiply pixel by pixel. Then the underwater image S with local strong light introduced by a single point artificial light source can be output. low .

[0056] In this embodiment, the spot estimation sub-network is trained, and the network model is a Judge-V Net network. Specifically, it includes: low It is used as the training set to input into the spot estimation subnetwork, and the loss function of radial gradient distribution characteristics is introduced to train it, and the output is (1) a low-quality underwater image layer U′ with uniform illumination distribution low ; (2) Spot image V′ low .

[0057] The specific steps include:

[0058] S32a, constructing a light spot estimation sub-network model, setting parameters for the network model, and constructing a training set and a test set, wherein the training set includes underwater images with light spots, corresponding light spot images during synthesis, and underwater images with uniform illumination;

[0059] S32b, inputting the images of the training set into the light spot estimation sub-network model for training, to obtain an estimated light spot image and an underwater image with uniform illumination.

[0060] S32c, calculate the loss value between the image output by the spot estimation subnetwork and the real target image in the training set, perform error back propagation according to the loss value, and update the weight of the spot estimation subnetwork.

[0061] S32d, determine whether the spot estimation sub-network has been trained. If so, obtain the trained network model and execute S32e, otherwise return to S32b.

[0062] S32e, input the test set into the trained spot estimation sub-network model for testing, and judge whether the trained network model meets the expected requirements according to the test results, if so, execute S32f, otherwise return to S32b;

[0063] S32f, inputting the image with the light spot into the tested light spot estimation subnetwork for processing to obtain an estimated light spot image and an underwater image with uniform illumination.

[0064] The light spot estimation subnetwork model used to decompose the light spot layer and the underwater image layer includes a six-layer network and two branches. The first to fifth layers of the network are all convolution layers with a convolution kernel size of 3×3 and a Relu layer; the sixth layer of the network estimates the underwater image part with uniform illumination using the Sigmoid activation function to obtain the output, and the branch estimating the light spot image uses the TanH activation function as the last layer. According to the formula:

[0065]

[0066] in is the gradient of a pixel (x, y) in the image. According to formula (2), Multiply the vector from the pixel to the center of the image to calculate the radial gradient distribution of the image with spot light and the radial gradient distribution of the image with uniform illumination. Based on this, the radial gradient loss function (3) is designed to perform L1 loss on the radial gradient distribution of the two images.

[0067]

[0068] Furthermore, a series of reconstruction loss functions (4) to (6) are designed to make the spot estimation image and the underwater image with uniform illumination obtained by network decomposition conform to the pixel distribution of the ground truth:

[0069]

[0070] Among them, S low represents an underwater image with light spots, U low Represents an underwater image with light spots removed.

[0071] In addition to the reconstruction loss, smoothing loss and L1 loss function are introduced in the branch of spot estimation. We believe that the decomposed spot image should be smooth and should not contain too many details. Therefore, smoothing loss is introduced. The role of L1 loss function is to control the artificial light source layer learned by the network to be as close to the real light source distribution as possible:

[0072]

[0073] L equal_v =‖V low -V higt ‖1 (8)

[0074] S33. Processing the training underwater image with uniform illumination based on the Retinex-Net decomposition sub-network to generate a training reflection map and a training illumination map.

[0075] The Retinex-Net decomposition sub-network of the present invention first needs to estimate the low-quality underwater image U′ with uniform illumination output by the spot estimation sub-network model. low Specifically, the low-quality underwater image U′ low Input Retinex-Net decomposition sub-network model, output (1) reflection graph R′ low ; (2) Illuminance diagram I′ low .

[0076] In this embodiment, the steps of training the Retinex-Net decomposition sub-network include:

[0077] S33a, constructing a network model for underwater image decomposition, setting parameters for the network model, and constructing a training set and a test set, wherein the training set includes underwater images obtained by the spot estimation subnetwork and real underwater images;

[0078] S33b, inputting the input image of the training set into the underwater image decomposition network model for training to obtain an illumination map and a reflectance map.

[0079] S33c, calculating the loss value between the output image and the true target image, performing error back propagation according to the loss value, and updating the weights of the underwater image decomposition network.

[0080] S33d, determine whether the underwater image decomposition network has been trained. If so, obtain the trained network model and execute S33e, otherwise return to S33b.

[0081] S33e, input the test set into the trained underwater image decomposition network model for testing, and judge whether the trained network model meets the expected requirements according to the test results, if so, execute S33f, otherwise return to S33b;

[0082] S33f, input the underwater image obtained by the spot estimation sub-network into the tested underwater image decomposition network to obtain a reflection map and an illumination map.

[0083] Furthermore, the Retinex-Net decomposition subnetwork includes estimating the reflection map R′ low and estimated illumination map I′ low There are two branches, and the input of the network is the output U′ of the above light spot estimation subnetwork low . For the estimated reflection map branch, the first layer of the network uses a convolution layer with a convolution kernel of 3×3, a Relu layer, and a maximum pooling layer for downsampling. The second layer has the same structure as the first layer. The third layer uses a convolution layer with a convolution kernel of 3×3, a Relu layer, and an upsampling layer. The fourth layer has the same structure as the third layer. The fifth layer outputs the estimated reflection map after a sigmoid function. For the estimated illumination map branch, the first layer of the network uses a convolution layer with a convolution kernel size of 3×3 and a Relu layer. The second layer of the network also uses a convolution layer with a convolution kernel size of 3×3 and a Relu layer. The third layer uses the output of the second layer and the output of the fourth layer of the network of the estimated reflection map branch as input, and finally the estimated illumination map is obtained after a sigmoid activation function.

[0084] Furthermore, in this network, the reconstruction loss and L1 loss are designed to guide the direction of gradient descent of network training, where Represents pixel-by-pixel multiplication. The loss in formula (9) is used to constrain the decomposition of uniformly illuminated underwater images, which should satisfy the Retinex theory, that is, the input image is equal to the pixel-by-pixel multiplication of the decomposed illumination map and reflectance map. Formula (10) is used to constrain the decomposition of real underwater images, and its theoretical basis is the same as formula (9). Formulas (11) and (12) construct the L1 loss function constraint so that the decomposition of the estimated underwater image conforms to the distribution of the decomposition of the real underwater image.

[0085]

[0086] L I_loss =|I′ low -I high | (11)

[0087] L equal_r =|R′ low -R high | (12)

[0088] S34. Based on the reflectance map restoration subnetwork, detail enhancement and color correction processing are performed on the training reflectance map to generate a restored reflectance map for training; based on the brightness adjustment subnetwork, brightness adjustment processing is performed on the training illumination map to generate a restored illumination map for training.

[0089] In this step, the reflection map R′ needs to be low and illumination diagram I′ lowThe two are input into the reflection map restoration network and the brightness adjustment network respectively to obtain the restored reflection map R′ (1) high ; (2) Illuminance diagram I′ high .

[0090] In this embodiment, the training steps of the reflection map recovery sub-network include:

[0091] S34a, constructing a reflection map recovery subnetwork, setting parameters for the reflection map recovery subnetwork, and constructing a training set and a test set, wherein the training set includes a reflection map obtained by the Retinex-Net decomposition subnetwork and a reflection map of GroundTruth;

[0092] S34b, inputting the input image of the training set into the reflectance map recovery sub-network for training to obtain an enhanced reflectance map.

[0093] S34c, calculating the loss value between the image output by the reflection image restoration subnetwork and the true target image, performing error back propagation according to the loss value, and updating the weight of the reflection image restoration subnetwork.

[0094] S34d, determine whether the reflection image recovery sub-network has been trained. If so, obtain the trained reflection image recovery sub-network and execute S34e, otherwise return to S34b.

[0095] S34e, input the test set into the trained reflection map restoration subnetwork for testing, and determine whether the trained reflection map restoration subnetwork meets the expected requirements based on the test results, if so, execute S34f, otherwise return to S34b;

[0096] S34f. Input the reflection map obtained by the Retinex-Net decomposition sub-network into the reflection map recovery sub-network after the test to obtain the restored reflection map.

[0097] The reflection map recovery subnetwork consists of a 6-layer network structure: the first layer uses two convolutional layers with a convolution kernel size of 3×3 and uses the Relu function for activation, followed by a maximum pooling layer; the second layer has the same structure as the first layer, and the third layer uses a convolutional layer with a convolution kernel size of 3×3; the fourth layer has the same structure as the third layer; the fifth layer uses an upsampling maximum pooling layer and a convolutional layer with a convolution kernel size of 3×3; the sixth layer has the same structure as the fifth layer; finally, a sigmoid activation function is used to output the enhanced reflection map.

[0098] Furthermore, the loss value between the output image and the target image is calculated by the following loss function:

[0099] L equal_R =‖R′ low -Rhigh ‖2 (13)

[0100] L SSIM_R = SSIM(R′ low ,R high ) (14)

[0101]

[0102] Among them, formula (13) calculates the 2 norm between the enhanced reflection map and the real reflection map, and formula (14) uses the SSIM structural similarity loss function to calculate the SSIM index between the enhanced reflection map and the real reflection map as the loss. In order to make the enhanced reflection map consistent with the real reflection map in details, formula (15) is constructed to calculate the L1 norm between the gradient distribution of the enhanced reflection map and the gradient distribution of the real reflection map.

[0103] The training steps of the brightness adjustment subnetwork include:

[0104] S34g, constructing a brightness adjustment subnetwork, setting parameters for the brightness adjustment subnetwork, and constructing a training set and a test set, wherein the training set includes an underwater image illumination map obtained by the Retinex-Net decomposition subnetwork and an illumination map of GroundTruth;

[0105] S34h, input the input image of the training set into the brightness adjustment subnetwork for training to obtain a restored illumination map.

[0106] S34i. Calculate the loss value between the image output by the brightness adjustment subnetwork and the real target image, perform error back propagation according to the loss value, and update the weight of the brightness adjustment subnetwork.

[0107] S34j, determine whether the brightness adjustment sub-network has been trained. If so, obtain the trained brightness adjustment sub-network and execute S34k, otherwise return to S34h.

[0108] S34k, input the test set into the trained brightness adjustment network model for testing, and determine whether the trained network model meets the expected requirements based on the test results, if so, execute S341, otherwise return to S34h;

[0109] S341. Input the illumination map obtained by the Retinex-Net decomposition network into the brightness adjustment network after the test to obtain a restored illumination map.

[0110] Furthermore, the brightness adjustment network model includes a 5-layer network structure: the first layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the second layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the third layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the fourth layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the fifth layer is a sigmoid activation function, and finally the restored illumination map is output.

[0111] L equal_I =‖I′ low -I high ‖2 (16)

[0112]

[0113] The brightness adjustment part uses formula (16) and formula (17) as loss functions. Formula (16) calculates the L2 norm between the restored illumination map and the real illumination map as loss. In order to make the restored illumination map consistent with the real illumination map in details, formula (17) calculates the gradient distribution of the restored illumination map and the gradient distribution of the real illumination map respectively, and calculates their L1 norm as the loss function.

[0114] S4, obtaining underwater image data with a single spot to be processed, inputting it into the trained underwater image sharpening network model, and multiplying the restored reflection map and the restored illumination map output by the underwater image sharpening network model pixel by pixel, thereby outputting a clear underwater image. Specifically including:

[0115] S41, inputting underwater image data with a single-point artificial light source collected by UUV, and processing the underwater image data with the single-point artificial light source based on the spot estimation subnetwork, so as to obtain a low-quality underwater image with a halo layer and uniform illumination distribution.

[0116] Specifically, we first need to analyze the underwater image S with a single-point artificial light source. low Decompose the underwater image S with a single spot low Input the decomposition network model, which is the Judge-VNet network, and introduces the loss function of the radial gradient distribution symmetry characteristic. After being processed by the decomposition network model, the output is (1) a low-quality underwater image layer U′ with uniform illumination distribution low ; (2) Spot image V′ low .

[0117] S42, Retinex-Net decomposition sub-network decomposes the low-quality underwater image into a reflectance map and an illumination map.

[0118] Specifically, the Retinex theory is introduced and Retinex-Net is used to transform U′ low Decompose and obtain the intermediate result: (1) reflection map R′ low ; (2) Illuminance diagram I′ low The reflection map and illumination map are input into the reflection map restoration network and the restored brightness adjustment network, respectively. low and illumination diagram I′ low The two are input into the reflection map restoration network and the brightness adjustment network respectively, and finally the restored reflection map (1) R′ is obtained. high ; (2) Illuminance diagram I′ high .

[0119] S43, performing detail enhancement and color correction processing on the reflection image based on the reflection image restoration sub-network, so as to generate a restored reflection image; performing brightness adjustment processing on the illumination image based on the brightness adjustment sub-network, so as to generate a restored illumination image.

[0120] S44, multiplying the restored reflection map and the enhanced illumination map pixel by pixel to output a clear underwater image.

[0121] Specifically, according to the Retinex theory, the restored reflection image R′ (1) high ; (2) Illuminance diagram I′ hig Multiply pixel by pixel to obtain a clear underwater image Iout. According to formula (18), the restored reflection map and the restored illumination map are multiplied pixel by pixel to obtain the final clear underwater image for training.

[0122] I out =R h ' igh oI h ' igh (18)

[0123] The scheme and effect of the present invention are further described below through specific application examples.

[0124] like Figure 2 The method for clearing underwater images for UUV real operation lighting scenes disclosed in the present invention includes the following steps:

[0125] S100: Select a public underwater image dataset U low According to the proposed underwater imaging model, using the underwater image U low and spot image V low , synthesize underwater image S with light spots low ;

[0126] S200: image S lowInput the spot estimation subnetwork to obtain the estimated spot image V′ low and low-quality underwater image U′ with uniform illumination low , the steps are as follows:

[0127] First, calculate the underwater image U low and underwater image S with light spots low Radial gradient distribution:

[0128] Then, according to the halo radial gradient distribution characteristics, a loss function is proposed, and the low-quality underwater image U′ with uniform illumination is obtained through network learning. low and spot image V′ low .

[0129] S300: Retinex-Net is used to analyze the underwater image U′ low Decompose and get the corresponding illumination map R′ low and reflection diagram I′ low :

[0130] S400: detail enhancement and color restoration: input the reflection image obtained in S300 into the reflection image recovery sub-network to obtain a reflection image after detail enhancement and color correction; brightness adjustment: input the illumination image obtained in S300 into the brightness adjustment network to obtain an illumination image after brightness adjustment.

[0131] S500: Restore the reflection image R′ high and illumination diagram I′ high Multiply pixel by pixel to get the final clear underwater image.

[0132] like Figure 3 As shown in the figure, the underwater image sharpening network model implements functions based on the spot estimation subnetwork, Retinex-Net decomposition subnetwork, reflectance map recovery subnetwork and brightness adjustment subnetwork. The training and testing process includes the following steps:

[0133] S001. Construct the above network model based on the Tensorflow deep learning framework, set parameters for the network model, and construct a training set and a test set. 890 underwater images containing light spots are synthesized in the training set, and 800 of them are set as the training set, and the remaining 90 are set as the test set.

[0134] S002. Input the input image of the training set into the network model for training, calculate the loss value between the output image and the real target image, perform error back propagation according to the loss value, and update the weight of the network model.

[0135] S003. Determine whether the network model has been trained. If so, obtain the trained network model and execute S004. Otherwise, return to S002.

[0136] S004, input the test set into the trained network model for testing, and determine whether the trained network model meets the expected requirements based on the test results. If so, execute S005, otherwise return to S002;

[0137] S005. Input the image taken by the UUV with a single-point artificial light source into the trained network model to obtain a clear underwater image with uniform illumination distribution.

[0138] In this example, the spot estimation subnetwork model includes a six-layer network structure and two branches. The first to fifth network layers are all convolution layers with a convolution kernel size of 3×3 and a Relu layer. In the sixth network layer, the part of the underwater image with uniform illumination is estimated using the Sigmoid activation function to obtain the output, and the branch that estimates the spot image uses the TanH activation function as the last layer.

[0139] The Retinex-Net decomposition subnetwork includes estimating the reflection map R low and estimated illumination map I low There are two branches. The input part of the network is the output U of the above-mentioned spot estimation sub-network. l ' ow . Estimated reflection map branch: The first layer of the network uses a convolution layer with a convolution kernel of 3×3, a Relu layer, and a maximum pooling layer for downsampling. The second layer has the same structure as the first layer. The third layer of the network uses a convolution layer with a convolution kernel of 3×3, a Relu layer, and an upsampling layer. The fourth layer has the same structure as the third layer. The fifth layer outputs the estimated reflection map after a sigmoid function. Estimated illumination map branch: The first layer of the network uses a convolution layer with a convolution kernel size of 3×3 and a Relu layer. The second layer of the network also uses a convolution layer with a convolution kernel size of 3×3 and a Relu layer. The third layer uses the output of the second layer and the output of the fourth layer of the network of the estimated reflection map branch as input, and finally passes through the sigmoid activation function to obtain the estimated illumination map.

[0140] The reflection map restoration sub-network model includes a 5-layer network structure: the first layer uses two convolutional layers with a convolution kernel size of 3×3 and uses the Relu function for activation, followed by a maximum pooling layer; the second layer has the same structure as the first layer, and the third layer uses a convolutional layer with a convolution kernel size of 3×3; the fourth layer uses a maximum pooling layer and a convolutional layer with a convolution kernel size of 3×3; the fifth layer has the same structure as the fourth layer; finally, a sigmoid activation function is used to output the enhanced reflection map.

[0141] The illumination map brightness adjustment sub-network model includes a 5-layer network structure: the first layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the second layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the third layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the fourth layer is a convolution layer with a convolution kernel size of 3×3 and a Relu layer; the fifth layer is a sigmoid activation function, and finally the output is the illumination map after brightness adjustment.

[0142] The training parameters of the network model in the present invention are set as follows:

[0143] A synthetic underwater image dataset containing light spots is selected as the training set, and the Tensorflow framework is used to train the network. The optimization algorithm uses the Adam optimizer, calls sess.run() to calculate the final output value and the corresponding loss, then calls the backward algorithm to calculate the gradient of each layer, updates the parameters according to the Adam optimizer, and finally records and saves the weights of network training. The present invention sets the batch size to 16, the learning rate to 0.001, and runs 1000 epochs on the RTX2060 graphics card.

[0144] Beneficial effects of the network model in the present invention and problems solved:

[0145] (1) The radial gradient distribution of images with and without light spots is used to effectively separate the light spots introduced by artificial light sources, thus solving the problem of uneven illumination caused by artificial light sources.

[0146] (2) Based on the Retinex theory, the separated uniformly illuminated underwater image is decomposed into two parts: the illumination map and the reflectance map. Different network branches are used to enhance and restore the two parts, which not only effectively enhances the contrast and details of the image, but also significantly corrects the brightness and color of the image.

[0147] (3) By setting different learning rates in different epochs in the neural network, the problem of loss non-convergence and poor training results is solved.

[0148] The present invention discloses an underwater image sharpening method for UUV real operation lighting scenes, which has the following advantages. The present invention innovatively proposes an optical imaging model that describes the lighting scene by observing that the brightness of the light spot formed by the artificial light source in the foreground has radial gradient distribution characteristics. According to the difference in brightness distribution of different images, a radial gradient loss function is proposed to train the light spot estimation sub-network to accurately separate the light spot image and the uniformly illuminated underwater image; then, the separated underwater image is decomposed into a reflection map and an illumination map, and an enhancement network is used to adjust the contrast and color in the reflection map part, and a brightness adjustment network is used to enhance the brightness in the illumination map part. This method can effectively improve the quality of underwater images, overcome the serious impact of non-uniform illumination introduced by artificial light sources on subsequent computer vision tasks, and has excellent robustness, accuracy and effectiveness in complex marine operating environments.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for underwater image clarity in real UUV operation lighting scenes, characterized in that: The following steps are involved: Construct an underwater imaging model to describe the brightness distribution characteristics of local light spots caused by artificial light sources carried by UUVs; According to the underwater imaging model, an underwater image dataset with a single spot is synthesized on a real underwater image dataset; An underwater image sharpening network model is trained based on the underwater image data set with a single light spot, and the underwater image sharpening network model includes: The spot estimation subnetwork is used to process underwater images with single-point spots. A loss function with radial gradient distribution characteristics is introduced to decompose the halo layer and obtain low-quality underwater images with uniform illumination distribution. Retinex-Net decomposition sub-network, which is used to decompose the low-quality underwater image into a reflectance map and an illumination map, A reflection map restoration subnetwork is used to enhance the reflection map to obtain a restored reflection map. A brightness adjustment subnetwork, which is used to adjust the brightness of the illumination map to obtain a restored illumination map; The underwater image data with a single light spot to be processed is obtained, and the data is input into a trained underwater image sharpening network model, and the restored reflection map and the restored illumination map output by the underwater image sharpening network model are multiplied pixel by pixel to output a clear underwater image.

2. The underwater image sharpening method for UUV real operation lighting scene according to claim 1 is characterized in that: The training steps of training the underwater image sharpening network model based on the underwater image dataset with a single spot light spot include: The synthetic underwater image dataset with single-point light spots is divided into a training set and a test set; The training underwater image data in the training set is input into the spot estimation subnetwork for processing to generate a low-quality underwater image with a training halo layer and uniform illumination distribution; Based on the Retinex-Net decomposition sub-network, the low-quality underwater images for training are processed to generate reflection maps and illumination maps for training. Based on the reflectance map restoration sub-network, the training reflectance map is enhanced to obtain a restored reflectance map for training; Based on the brightness adjustment sub-network, the brightness of the training illumination map is adjusted to obtain a restored illumination map for training; Multiplying the training restored reflection map and the training restored illumination map pixel by pixel, thereby outputting a clear underwater image for training; The loss value between the output image of each sub-network and the true target image is calculated, and the error is back-propagated based on the loss value to update the weights of the spot estimation sub-network, Retinex-Net decomposition sub-network, reflectance map recovery sub-network and brightness adjustment sub-network.

3. The underwater image sharpening method for UUV real operation lighting scene according to claim 2 is characterized in that: The training steps of the spot estimation subnetwork include: Acquire training set data, wherein the training set data includes underwater images with light spots, light spot images corresponding to the underwater images with light spots, and underwater images with uniform illumination; The light spot estimation sub-network model is trained by using an underwater image with light spots as input data of the light spot estimation sub-network and using the light spot image and an underwater image with uniform illumination as output data; The radial gradient distribution loss value of the uniformly illuminated underwater image output by the calculation model and the uniformly illuminated underwater image in the training set is calculated, and error back propagation is performed according to the radial gradient distribution loss value to update the weight of the spot estimation subnetwork.

4. The underwater image sharpening method for UUV real operation lighting scene according to claim 3 is characterized in that: The light spot estimation subnetwork includes a six-layer network and two branches. The first to fifth network layers are all convolution layers with a convolution kernel size of 3×3 and a Relu layer. In the sixth network layer, the part of the underwater image that estimates uniform illumination uses a Sigmoid activation function to obtain an output, and the branch that estimates the light spot image uses a TanH activation function as the last layer.

5. The underwater image sharpening method for UUV real operation lighting scene according to claim 2 is characterized in that: The training steps of the Retinex-Net decomposition sub-network include: Acquire training set data, wherein the training set data includes an underwater image, and an illumination map and a reflectance map corresponding to the underwater image; The low-quality underwater image is used as input data of the Retinex-Net decomposition sub-network, and the illumination map and the reflectance map are used as output data to train the Retinex-Net decomposition sub-network; the Retinex-Net decomposition sub-network includes a reflectance map branch part and an illumination map branch part; The loss values ​​of the illumination map and reflectance map output by the calculation model and the illumination map and reflectance map of GroundTruth in the training set are compared, and the error is back-propagated according to the loss value to update the weight of the Retinex-Net decomposition sub-network.

6. The underwater image sharpening method for UUV real operation lighting scene according to claim 2 is characterized in that: The training steps of the reflection map recovery sub-network include: Acquire training set data, wherein the training set data includes a reflection map output by a Retinex-Net decomposition subnetwork and a reflection map of GroundTruth; The reflection map output by the Retinex-Net decomposition sub-network is used as input data of the reflection map recovery sub-network to train the reflection map recovery sub-network; The loss value between the restored reflection map output by the reflection map restoration subnetwork and the reflection map of GroundTruth in the training set is calculated, and error backpropagation is performed according to the loss value to update the weight of the reflection map restoration subnetwork.

7. The underwater image sharpening method for UUV real operation lighting scene according to claim 2 is characterized in that: The training steps of the brightness adjustment sub-network include: Acquire training set data, wherein the training set data includes an illumination map output by a Retinex-Net decomposition subnetwork and an illumination map of GroundTruth; The illumination map output by the Retinex-Net decomposition sub-network is used as input data of the brightness adjustment sub-network to train the brightness adjustment sub-network; The loss value between the restored illumination map output by the brightness adjustment subnetwork and the illumination map of GroundTruth in the training set is calculated, and error backpropagation is performed according to the loss value to update the weight of the brightness adjustment subnetwork.

Images