An underwater image restoration method with degree of attenuation perception

Abstract

The application discloses an underwater image recovery method with attenuation degree perception, comprising the following steps: generating a random number as a target attenuation degree of a synthetic underwater image; synthesizing training data with different attenuation degrees according to the generated random number; on the training side, taking a scene relative distance and the synthetic underwater image as inputs, calculating the structural similarity and mean square error of a recovery image output by a network and a real non-attenuation reference image, calculating the cross entropy of an attenuation degree predicted by the network and an actual attenuation degree, and taking the arithmetic sum of the three as a loss function to guide neural network training; on the inference side, distance estimation is performed on a real underwater image input, and the real underwater image is corrected; the real underwater image and the corrected underwater image distance estimation are input into the trained neural network to obtain a recovered underwater image. The method can simulate the hetero-attenuation generated by different types of water bodies, and can truly simulate the feature that the underwater image becomes darker and darker with the attenuation degree deepening.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a method for restoring images of water degradation caused by water absorption and scattering. Background Technology

[0002] Underwater imagery plays a crucial role in numerous engineering projects, including marine resource exploration, underwater archaeology, environmental monitoring, and underwater structure inspection. However, the absorption of light by water in nature often results in underwater images suffering from low brightness, color cast, and low contrast, severely reducing the efficiency of underwater work. Restoring underwater images and enhancing their quality can generate significant socio-economic value.

[0003] Similar to the attenuation of light propagating in the atmosphere due to scattering by atmospheric particles, light also experiences distance-dependent attenuation underwater due to scattering by water particles. However, unlike the former, underwater light is affected not only by scattering but also by absorption by water particles, resulting in distance-dependent attenuation. Currently, there are two main approaches to compensate for the image quality degradation caused by light attenuation during water propagation: image enhancement-based methods and image restoration-based methods.

[0004] Image enhancement methods do not consider the physical model of light propagation underwater, treating the enhancement of water-degraded images as a problem of enhancing low-contrast, low-saturation images. Representative methods include histogram equalization-based methods, algorithms based on biological visual mechanisms, and image fusion-based algorithms. These algorithms can enhance some low-contrast, low-saturation underwater images into high-contrast, high-saturation enhanced images. However, due to the lack of a physical basis, their stability is relatively poor, and they cannot guarantee that the enhancement results are consistent with the actual scene, often leading to over-enhancement.

[0005] Image restoration-based methods treat water-attenuated and degraded images as atmospheric scattering-degraded images. They introduce prior assumptions through natural image statistical analysis, eliminate some unknowns from the atmospheric scattering model, and inversely solve for the image quality degradation caused by light scattering to restore the image. Representative algorithms include dark channel prior algorithms and color line prior algorithms. However, in most water bodies, the attenuation of light due to absorption by water particles is far greater than the attenuation due to scattering by water particles. Therefore, methods that ignore the absorption of light by water and treat water-attenuated and degraded images as atmospheric scattering-degraded images have a fundamental flaw. These methods often only work properly in scenarios with limited imaging distances. As the imaging distance increases, the absorption of light by water increases exponentially, and the performance of these algorithms drops rapidly, making it difficult to achieve the desired enhancement effect. Summary of the Invention

[0006] The purpose of this invention is to overcome the aforementioned deficiencies of existing underwater image enhancement methods and provide an underwater image restoration method that uses distance estimation as a reference to compensate for the exponential increase in light absorption in water as the optical path length increases. This method restores image structural details while enhancing contrast and saturation, making it very suitable for underwater imaging scenarios with large depth of field.

[0007] The specific technical solution of this invention is: an underwater image restoration method with attenuation perception, comprising the following steps:

[0008] S1. Generate a random number as the target attenuation level for synthesizing the underwater image;

[0009] S2. Based on the random numbers generated in step S1, and taking into full account the absorption and scattering effect of water on light, training data with different attenuation levels are synthesized using the underwater image imaging formula.

[0010] S3. On the training side, using the relative distance of the scene and the underwater image synthesized in step S2 as input, calculate the structural similarity and mean square error between the restored image output by the network and the real unattenuated reference image, calculate the cross-entropy between the attenuation degree predicted by the network and the actual attenuation degree, and use the arithmetic sum of the three as the loss function to guide the training of the neural network.

[0011] S4. On the inference side, perform distance estimation on the input real underwater image;

[0012] S5. Correct the underwater image distance estimate obtained in step S4;

[0013] S6. Input the distance estimate between the real underwater image and the corrected underwater image obtained in step S5 into the neural network trained in step S3 to obtain the recovered underwater image.

[0014] The beneficial effects of this invention are as follows: The method of this invention uses an iterative approximation approach to generate underwater composite images with different attenuation levels. Compared with traditional underwater image synthesis methods, it fully considers the differences in optical propagation characteristics between water and atmospheric aerosol media, reflects the absorption effect of water on light propagation, and can realistically simulate the characteristic that underwater images become darker at infinity as the attenuation level increases by simulating heterogeneous attenuation of different types of water bodies. This is more in line with objective physical laws and has the characteristics of physical reliability, strong stability, and wide synthesis range. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the underwater image restoration method based on distance estimation according to the present invention;

[0016] Figure 2 This is a neural network structure diagram of an embodiment of the present invention. Detailed Implementation

[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The specific process is as follows (floating-point numbers are used in all the calculations below):

[0018] In this embodiment, images from the KITTI dataset are used as the data source to synthesize training data. The KITTI dataset training data images are denoted as J(x), and the reference distance information provided by this dataset is denoted as d(x).

[0019] The process of an underwater image restoration method with attenuation perception according to the present invention is as follows: Figure 1 As shown, the specific steps include the following:

[0020] S1. For each training data image J(x), denote its decay level N. d The value is 0, and a positive integer i between 1 and 10 is randomly generated as the target attenuation level for synthesizing the underwater image.

[0021] S2. Based on the random numbers generated in step S1, and taking into full account the absorption and scattering effects of water on light, use KITTI dataset images to synthesize training data according to the following formula.

[0022]

[0023] Where α is the absorption coefficient, β is the scattering coefficient, A is the atmospheric background light, and α is a floating-point number between 0 and 1, which was experimentally determined to be 0.7 in this embodiment.

[0024] To normalize the distance information, it is calculated using the following formula:

[0025]

[0026] Where MAX is the operation to find the maximum value of the elements in matrix d(x), MIN is the operation to find the minimum value of the elements in matrix d(x), and RANDOM(0,2) is the operation to find a random number between 0 and 2.

[0027] The scattering coefficient β is calculated by the following formula:

[0028]

[0029] β R =RANDOM(0,0.02)+0.15

[0030] The absorption coefficient α is calculated by the following formula:

[0031]

[0032] α B =RANDOM(0,0.07)+0.05

[0033] After completing one image synthesis calculation, the attenuation level N d Add 1, and use the degree of attenuation as a label, along with the generated synthetic image. Binding. It can be seen that for each synthesized image... Each image has a natural number Nd representing its degree of decay, indicating the extent of the image's decay.

[0034] For each positive integer i, if i is greater than 1, then subtract 1 from i and let i = 1. Repeat step S2 again; otherwise, proceed to step S3.

[0035] S3. Using normalized distance information The underwater image synthesized with step S2 Using the input as input, calculate the reconstructed image output by the network. The structural similarity loss and mean squared error loss between the network and the real unattenuated image J(x) are used to calculate the degree of attenuation predicted by the network. Compared with the actual attenuation level N d The cross-entropy is used, and the arithmetic sum of the three is used as the loss function to guide the training of the neural network.

[0036] Among them, the neural network structure model is as follows Figure 2 As shown, it is divided into three modules: feature extraction, feature fusion, and output prediction.

[0037] The feature extraction module uses tf-efficientnet-lite3 to extract multi-scale features from the attenuation image and distance image with a length of H, a width of W, and 3 channels. Finally, it obtains two feature maps with 24 channels in feature 1, two feature maps with 48 channels in feature 2, two feature maps with 96 channels in feature 3, and two feature maps with 192 channels in feature 4.

[0038] The feature fusion module is divided into an integrated feature module and an upsampling module. The integrated feature module adds the feature maps of the same scale from the feature extraction module to obtain a feature map that integrates distance and attenuation information. The integrated feature module takes features 1, 2, 3, and 4 output from the feature extraction module as input, integrates distance and attenuation information, and outputs integrated feature 1, integrated feature 2, integrated feature 3, and integrated feature 4.

[0039] The upsampling module consists of two processing paths, processing the decay features and distance features output from the feature extraction module, respectively. These two paths also receive the integrated features output from the integrated feature module. In path 1, input 1 of the first upsampling layer is the decay feature from feature 4 output from the feature extraction module, and input 2 is the integrated feature 4 output from the integrated feature module. Input 1 of the second upsampling layer is the output of the first upsampling layer, and input 2 is the integrated feature 3 from the integrated feature module. Input 1 of the third upsampling layer is the output of the second upsampling layer, and input 2 is the integrated feature 2 from the integrated feature module. Input 1 of the fourth upsampling layer is the output of the third upsampling layer, and input 2 is the integrated feature 1 from the integrated feature module. Each upsampling layer passes input 1 through a residual convolution unit, then adds it to input 2's feature map, passes it through another residual convolution unit, and finally performs upsampling as the output. One residual convolution unit contains two loops of ReLU activation function + convolution. In path 2, input 1 of the upsampling layer 1 is the distance feature in feature 4 output by the feature extraction module, and input 2 is the integrated feature module 4; the architecture of layers 2, 3, and 4 is the same as that of path 1.

[0040] The output prediction module is divided into a reconstruction layer and a prediction layer. The reconstruction layer reconstructs the two feature maps output from the fourth layer of the upsampling module into a size of length H, width W, and number of channels 3, and restricts their values ​​to 0-1. The reconstruction result of channel 1 is denoted as reconstructed feature map f1, and the reconstruction result of channel 2 is denoted as reconstructed feature map f2. The reconstruction layer consists of convolution-upsampling-convolution-ReLU activation function-convolution-Sigmoid activation function. In the prediction layer, the image is first restored according to the restoration formula based on the attenuated image, reconstructed feature maps f1 and f2, to obtain the restored image.

[0041] The recovery formula is:

[0042]

[0043] Where f1 is the feature map of reconstructed path 1, f2 is the feature map of reconstructed path 2, tanh is the hyperbolic tangent function, and e is the natural constant.

[0044] The information from the reconstructed feature maps is integrated by multiplying the reconstructed feature maps f1 and f2. Then, the feature maps are transformed into high-dimensional vectors through convolution-average pooling-convolution-average pooling-average pooling. Finally, the result of two linear layers plus a softmax activation function is used as the prediction result of the decay degree.

[0045] The loss function L is calculated as follows:

[0046]

[0047] Wherein, SSIM represents structural similarity calculation, MSE represents mean squared error calculation, and CE represents cross-entropy calculation.

[0048] Train the network, backpropagate the model loss function L, and stop training when the loss stops decreasing for four consecutive rounds.

[0049] S4. On the inference side, perform distance estimation on the input real underwater image I(x) to obtain a coarse estimate of the distance information.

[0050] Scene distance estimation can be accomplished using any distance estimation algorithm. For example, UDepth can be used to obtain a relative estimate of scene distance. UDepth, as a monocular distance estimation framework, has demonstrated its efficiency and accuracy in multiple vision tasks. UDepth first converts the image from the RGB domain to the RMI domain, uses MobileNetV2 as the backbone network to extract distance features from the RMI domain image, and generates a coarse distance estimation map. Subsequently, the distance is further refined by integrating the mVIT model, enabling the distance estimation map to capture more subtle distance changes and spatial relationships between objects in the image. Finally, a convolutional regressor generates a coarse estimate of the monocular scene distance information.

[0051] S5. Coarse distance estimation of the underwater image obtained in step S4. After making corrections, the corrected distance estimate is obtained.

[0052]

[0053] in, For a coarse distance estimate, MAX is used to extract the matrix. The maximum value operation is performed on the matrix element; MIN is used to extract the maximum value from the matrix. Minimum element operation.

[0054] S6. Estimate the distance between the real underwater image I(x) and the corrected underwater image obtained in step S5. The image is input into the neural network trained in step S3 to obtain the recovered underwater image.

[0055] In summary, the method of this invention utilizes distance information to guide underwater image synthesis, fully reflecting the absorption and scattering effects of water on light propagation. The synthesized underwater image, distance map, and attenuation degree, which vary with distance, are fed into the network as training data for learning. This enables the network to accurately estimate the low-distortion image that reflects real-world scene information and the attenuation degree of the input image. It can effectively recover the image attenuation caused by the absorption and scattering effects of the underwater medium, and has the advantages of high reliability and strong stability, which can provide convenience for subsequent computer vision tasks.

[0056] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for underwater image restoration with attenuation perception, comprising the following steps: S1. Generate a random number as the target attenuation level for synthesizing the underwater image; For each training data image J(x), denote its decay level N. d The value is 0, and a positive integer i between 1 and 10 is randomly generated as the target attenuation level for synthesizing the underwater image; S2. Based on the random numbers generated in step S1, and taking into full account the absorption and scattering effect of water on light, training data with different attenuation levels are synthesized using the underwater image imaging formula. Training data is synthesized according to the following formula. , ； Where α is the absorption coefficient, β is the scattering coefficient, and A is the atmospheric background light, all of which are floating-point numbers between 0 and 1; To normalize the distance information, it is calculated using the following formula: ； Where MAX is the operation to find the maximum value of the elements in matrix d(x), MIN is the operation to find the minimum value of the elements in matrix d(x), and RANDOM(0,2) is the operation to find a random number between 0 and 2; The scattering coefficient β is calculated by the following formula: ； The absorption coefficient α is calculated by the following formula: ； After completing one image synthesis calculation, the attenuation level N d Add 1, and use the degree of attenuation as a label, along with the generated synthetic image. Binding; For each positive integer i, if i is greater than 1, then subtract 1 from i and let i = 1. Repeat step S2 again; otherwise, proceed to step S3. S3. On the training side, using the relative distance of the scene and the underwater image synthesized in step S2 as input, calculate the structural similarity and mean square error between the restored image output by the network and the real unattenuated reference image, calculate the cross-entropy between the attenuation degree predicted by the network and the actual attenuation degree, and use the arithmetic sum of the three as the loss function to guide the training of the neural network. S4. On the inference side, perform distance estimation on the input real underwater image; S5. Correct the underwater image distance estimate obtained in step S4; S6. Input the distance estimate between the real underwater image and the corrected underwater image obtained in step S5 into the neural network trained in step S3 to obtain the recovered underwater image.

2. The underwater image restoration method with attenuation perception according to claim 1, characterized in that, The specific process of step S3 is as follows: using normalized distance information The underwater image synthesized with step S2 Using the input as input, calculate the reconstructed image output by the network. Compared to true non-attenuation images The structural similarity loss and mean squared error loss are used to calculate the degree of decay in the network prediction. Compared with the actual degree of attenuation The cross-entropy of the three factors is used as the loss function to guide the training of the neural network; The neural network structure model is divided into three modules: feature extraction, feature fusion, and output prediction. The feature extraction module uses tf-efficientnet-lite3 to extract multi-scale features from the attenuation image and distance image with a length of H, a width of W, and a number of channels of 3. Finally, it obtains two feature maps with a number of channels of 24 in feature 1, two feature maps with a number of channels of 48 in feature 2, two feature maps with a number of channels of 96 in feature 3, and two feature maps with a number of channels of 192 in feature 4. The attenuation image is the underwater image synthesized in step S2, and the distance image provides the relative distance of the scene. The feature fusion module is divided into an integrated feature module and an upsampling module. The integrated feature module adds the feature maps of the same scale in the feature extraction module to obtain a feature map that integrates distance information and attenuation information. The integrated feature module takes feature 1, feature 2, feature 3, and feature 4 output by the feature extraction module as input, integrates distance information and attenuation information, and outputs integrated feature 1, integrated feature 2, integrated feature 3, and integrated feature 4. The upsampling module is divided into two processing paths, which process the attenuation feature and distance feature output by the feature extraction module, respectively. These two processing paths also receive the integrated feature output by the integrated feature module. In path 1, input 1 of the first upsampling layer is the attenuation feature from feature 4 output by the feature extraction module, and input 2 is the integrated feature 4 output by the integrated feature module. Input 1 of the second upsampling layer is the output of the first upsampling layer, and input 2 is the integrated feature 3 from the integrated feature module. Input 1 of the third upsampling layer is the output of the second upsampling layer, and input 2 is the integrated feature 3 from the integrated feature module. Integrating Feature 2; Input 1 of the upsampling layer 4 is the output of layer 3 of the upsampling module, and input 2 is integrated feature 1 in the integrated feature module; Each upsampling layer passes input 1 through a residual convolutional unit, then adds the feature map of input 2 once, passes it through another residual convolutional unit, and finally performs upsampling as the output. One residual convolutional unit contains two loops of ReLU activation function + convolution; In Path 2, input 1 of the upsampling layer 1 is the distance feature in feature 4 output by the feature extraction module, and input 2 is integrated feature 4; The architecture of layers 2, 3, and 4 is the same as that of Path 1; The output prediction module is divided into a reconstruction layer and a prediction layer. The reconstruction layer reconstructs the two feature maps output by the fourth layer of the upsampling module into a size with length H, width W, and number of channels 3, and restricts their values ​​to 0~1. The reconstruction result of channel 1 is recorded as the reconstructed feature map. The reconstruction result of pathway 2 is denoted as the reconstructed feature map. The reconstruction layer consists of convolution-upsampling-convolution-ReLU activation function-convolution-Sigmoid activation function. In the prediction layer, the feature map is first reconstructed based on the decayed image. and The image is restored using the restoration formula to obtain the restored image. ; The recovery formula is: ； in, f1 represents the feature map of reconstructed path 1, f2 represents the feature map of reconstructed path 2, tanh is the hyperbolic tangent function, and e is the natural constant; By reconstructing the feature map and reconstructed feature maps The information from the reconstructed feature map is multiplied and integrated, then transformed into a high-dimensional vector through a series of convolutional-average pooling operations. Finally, the result of two linear layers plus a softmax activation function is used as the prediction result for the degree of decay. ; The loss function L is calculated as follows: ； Wherein, SSIM represents structural similarity calculation, MSE represents mean squared error calculation, and CE represents cross-entropy calculation; Train the network, backpropagate the model loss function L, and stop training when the loss stops decreasing for four consecutive rounds.

3. The underwater image restoration method with attenuation perception according to claim 2, characterized in that, The specific process of step S5 is as follows: coarsely estimate the distance of the underwater image obtained in step S4. After making corrections, the corrected distance estimate is obtained. : ； in, For a coarse distance estimate, MAX is used to extract the matrix. The maximum value operation is performed on the matrix element; MIN is used to extract the maximum value from the matrix. Minimum element operation.

Images