Underwater image restoration method based on attenuation degree perception

By combining distance estimation and neural network, considering the absorption and scattering of light by water bodies, the problem of existing underwater image recovery methods neglecting the absorption effect, and the effective recovery and enhancement of underwater images in large depth of field scenes is achieved. High stability and physical reliability.

CN119006313AActive Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA

Patent Information

Application Number
CN202411160695.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-22
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

When dealing with the attenuation of light propagation of water bodies, the existing underwater image recovery method ignores the absorption of light by water bodies, resulting in poor results when the imaging distance is long. The method based on physical model is poor, making it difficult to ensure the accuracy of the enhancement results. sex.

Method used

Using distance estimation as a reference, an underwater image recovery method is generated through a neural network to sense the degree of attenuation, taking into account the absorption and scattering effects of water on light, and using structural similarity, mean square error and cross-entropy loss functions for training to generate recovery images. Simulate the heterogeneous attenuation characteristics of different types of water bodies.

Benefits of technology

It realizes the contrast and saturation of the image in large depth of field underwater imaging scenes, with physical reliability and stability, and can truly simulate the characteristics of underwater images deepening with the degree of attenuation, which is suitable for a wide range of underwater image recovery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006313A_ABST
    Figure CN119006313A_ABST
Patent Text Reader

Abstract

The invention discloses an attenuation degree perceived underwater image recovery method, which comprises the following steps of: generating a random number as a target attenuation degree for synthesizing an underwater image; according to the generated random number, synthesizing training data with different attenuation degrees; at a training side, taking a scene relative distance and a synthesized underwater image as input, calculating a structural similarity and a mean square error between a recovered image output by the network and a real non-attenuation reference image, calculating a cross entropy between an attenuation degree predicted by the network and an actual attenuation degree, and taking an arithmetic sum of the three as a loss function to guide neural network training; on the reasoning side, distance estimation and correction are carried out on an input real underwater image; and inputting the distance estimation of the real underwater image and the corrected underwater image into the trained neural network to obtain a recovered underwater image. According to the method, heterogeneous attenuation generated by different types of water bodies is simulated, so that the characteristic that an underwater image becomes darker and darker at infinity as the attenuation degree is deepened can be truly simulated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to a method for restoring an underwater degraded image caused by water absorption and scattering. Background Art

[0002] Underwater images play an extremely important role in many projects such as marine resource exploration, underwater archaeology, environmental monitoring, and underwater structure inspection. However, due to the absorption of light propagation by natural water bodies, underwater images often have problems such as low brightness, color deviation, and low contrast, which seriously reduces the efficiency of underwater work. Restoring underwater images and enhancing their image quality can produce significant social and economic value.

[0003] Similar to the attenuation of light in the process of propagating in the atmosphere due to the scattering of atmospheric particles and its propagation distance, the attenuation of light in the process of propagating underwater due to the scattering of water particles and its propagation distance. However, the difference is that when light propagates underwater, in addition to being affected by the scattering of water particles and its propagation distance, it is also affected by the absorption of water particles and its propagation distance. Currently, there are two mainstream ways to compensate for the degradation of image quality caused by the attenuation of light propagation in water: image enhancement-based methods and image restoration-based methods.

[0004] The image enhancement-based methods do not consider the physical model of light propagation in the underwater process, and regard the enhancement of water attenuation-degraded images as a low-contrast, low-saturation image enhancement problem. Representative methods include histogram equalization-based methods, algorithms based on biological vision mechanisms, and algorithms based on image fusion. These algorithms can enhance some low-contrast, low-saturation underwater images into high-contrast, high-saturation enhanced images, but due to the lack of physical basis, the stability is relatively poor, and it is impossible to ensure that the enhancement results are consistent with the actual scene, which often leads to over-enhancement.

[0005] The image restoration-based method regards the enhancement of water body attenuation-degraded images as atmospheric scattering-degraded images. It introduces a priori assumptions through statistical analysis of natural images, eliminates some unknowns from the atmospheric scattering model, and reversely solves the image quality degradation process caused by light scattering to restore the image. Representative algorithms include dark channel prior algorithm and color line prior algorithm. However, in most waters, the attenuation of light caused by the absorption of water particles is much greater than the attenuation caused by the scattering of water particles. Therefore, the method that ignores the absorption of light by water and regards the enhancement of water body attenuation-degraded images as atmospheric scattering-degraded images has a fundamental defect. Such methods can often only work properly in scenes with limited imaging distance. When the imaging distance is extended, the absorption of light by water increases exponentially, and the performance of such algorithms drops rapidly, making it difficult to achieve the enhancement effect. Summary of the invention

[0006] The purpose of the present invention is to make up for the above-mentioned defects of the existing underwater image enhancement methods, and to provide an underwater image restoration method which uses distance estimation as a reference to compensate for the absorption effect of light in water that increases exponentially with the optical path length. It can enhance the contrast and saturation while restoring the image structure details, and is very suitable for underwater imaging scenes with a large depth of field.

[0007] The specific technical solution of the present invention is: an underwater image restoration method with attenuation degree perception, comprising the following steps:

[0008] S1. Generate a random number as the target attenuation degree of the synthetic underwater image;

[0009] S2. Based on the random number generated in step S1, taking into full account the absorption and scattering effect of water on light, the training data with different attenuation degrees are synthesized through the underwater image imaging formula;

[0010] S3. On the training side, the scene relative distance and the underwater image synthesized in step S2 are used as inputs, the structural similarity and mean square error between the restored image output by the network and the true non-attenuated reference image are calculated, and the cross entropy between the attenuation degree predicted by the network and the actual attenuation degree is calculated, and the arithmetic sum of the three is used as the loss function to guide the neural network training;

[0011] S4. On the inference side, performing distance estimation on the input real underwater image;

[0012] S5. Correcting the underwater image distance estimation obtained in step S4;

[0013] S6. Input the distance estimation of the real underwater image and the corrected underwater image obtained in step S5 into the neural network trained in step S3 to obtain a restored underwater image.

[0014] Beneficial effects of the present invention: The method of the present invention adopts an iterative approximation idea to generate underwater synthetic images with different attenuation degrees. Compared with the traditional underwater image synthesis method, it fully considers the difference in optical propagation characteristics between water media and atmospheric aerosol media, and reflects the absorption effect of water on light propagation. By simulating the heterogeneous attenuation caused by different types of water bodies, it can truly simulate the characteristic that underwater images become darker and darker at infinity as the attenuation degree deepens. It is more in line with objective physical laws, and has the characteristics of physical reliability, strong stability, and a wide synthesis range. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the flow of the underwater image restoration method based on distance estimation of the present invention;

[0016] Figure 2 4 is a diagram of a neural network structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. The specific process is as follows (floating point numbers are used for calculation in the following calculation process):

[0018] In this embodiment, the KITTI dataset image is used as the data source to synthesize the training data. The KITTI dataset training data image is denoted as J(x), and the reference distance information provided by the dataset is denoted as d(x).

[0019] The process of the attenuation degree-aware underwater image restoration method of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0020] S1. For each training data image J(x), record its attenuation degree N d is 0, and a positive integer i between 1 and 10 is randomly generated as the target attenuation degree of the synthetic underwater image.

[0021] S2. Based on the random number generated in step S1, taking into account the absorption and scattering effect of water on light, the KITTI dataset images are used to synthesize training data according to the following formula

[0022]

[0023] Wherein, α is the absorption coefficient, β is the scattering coefficient, and A is the atmospheric background light, which is a floating point number between 0 and 1 and is determined to be 0.7 through experiments in this embodiment.

[0024] To normalize the distance information, it is calculated by the following formula:

[0025]

[0026] Among them, MAX is the operation of taking the maximum value of the elements in the matrix d(x), MIN is the operation of taking the minimum value of the elements in the matrix d(x), and RANDOM(0,2) is the operation of taking a random number between 0 and 2.

[0027] The scattering coefficient β is calculated by the following formula:

[0028]

[0029] β R =RANDOM(0,0.02)+0.15

[0030] The absorption coefficient α is calculated by the following formula:

[0031]

[0032] α B =RANDOM(0,0.07)+0.05

[0033] After completing an image synthesis calculation, the attenuation degree N d Add 1 and use the attenuation degree as a label to generate a synthetic image Binding. It can be seen that for each composite image There is a natural number Nd representing its attenuation degree to identify the attenuation degree of the image.

[0034] Judge the positive integer i. If the positive integer i is greater than 1, then subtract 1 from the positive integer i. Repeat step S2 again; otherwise go to step S3.

[0035] S3. Normalized distance information The underwater image synthesized with step S2 As input, calculate the restored image output by the network The structural similarity loss and mean square error loss with the true unattenuated image J(x) are used to calculate the attenuation degree predicted by the network. The actual attenuation degree N d The cross entropy of the three is used as the loss function to guide the neural network training.

[0036] Among them, the neural network structure model is as follows Figure 2 As shown. It is divided into three modules: feature extraction, feature fusion, and output prediction.

[0037] The feature extraction module uses tf-efficientnet-lite3 to extract multi-scale features of the attenuation image and the range image with an input length of H, a width of W, and a channel number of 3. Finally, two feature maps with 24 channels in feature 1, two feature maps with 48 channels in feature 2, two feature maps with 96 channels in feature 3, and two feature maps with 192 channels in feature 4 are obtained.

[0038] Feature fusion module,The feature fusion module is divided into an integrated feature module and an upsampling module. The integrated feature module adds the feature maps of the same scale in the feature extraction module to obtain a feature map that integrates distance information and attenuation information. The integrated feature module takes feature 1, feature 2, feature 3, and feature 4 output by the feature extraction module as input, integrates distance information and attenuation information, and outputs integrated feature 1, integrated feature 2, integrated feature 3, and integrated feature 4.

[0039] The upsampling module is divided into two processing paths, which process the attenuation feature and distance feature output by the feature extraction module respectively. At the same time, these two processing paths also input the integrated feature output by the integrated feature module. In path 1, the input 1 of the first upsampling layer is the attenuation feature in feature 4 output by the feature extraction module, and the input 2 is the integrated feature 4 output by the integrated feature module. The input 1 of the second upsampling layer is the output of the first upsampling layer, and the input 2 is the integrated feature 3 in the integrated feature module. The input 1 of the third upsampling layer is the output of the second upsampling module, and the input 2 is the integrated feature 2 in the integrated feature module. The input 1 of the fourth upsampling layer is the output of the third upsampling module, and the input 2 is the integrated feature 1 in the integrated feature module. Each upsampling layer passes input 1 through the residual convolution unit, then adds the feature map with input 2 and passes through the residual convolution unit once, and finally performs upsampling as the output. A residual convolution unit contains two cycles of ReLU activation function + convolution. In path 2, the upsampling layer 1 input 1 is the distance feature in feature 4 output by the feature extraction module, and input 2 is the integrated feature module 4; the architecture of the 2nd, 3rd, and 4th layers is the same as path 1.

[0040] The output prediction module is divided into a reconstruction layer and a prediction layer. The reconstruction layer reconstructs the two feature maps output by the 4th layer of the upsampling module into a size of length H, width W, and number of channels 3, and limits their values ​​to 0-1. The reconstruction result of channel 1 is recorded as the reconstructed feature map f1, and the reconstruction result of channel 2 is recorded as the reconstructed feature map f2; the reconstruction layer is composed of convolution-upsampling-convolution-ReLU activation function-convolution-Sigmoid activation function. In the prediction layer, the image is first restored according to the restoration formula based on the attenuated image, the reconstructed feature maps f1 and f2 to obtain the restored image.

[0041] The recovery formula is:

[0042]

[0043] Among them, f1 is the reconstructed path 1 feature map, f2 is the reconstructed path 2 feature map, tanh is the hyperbolic tangent function, and e is a natural constant.

[0044] The information of the reconstructed feature map is integrated by multiplying the reconstructed feature map f1 and the reconstructed feature map f2, and then the feature map is converted into a high-dimensional vector through convolution-average pooling-convolution-average pooling-average pooling. Finally, the result of the two linear layers + Softmax activation function is used as the attenuation degree prediction result.

[0045] The calculation method of the loss function L is:

[0046]

[0047] Among them, SSIM represents structural similarity calculation, MSE represents mean square error calculation, and CE represents cross entropy calculation.

[0048] Train the network and backpropagate the model loss function L. Stop training when the loss stops decreasing for four consecutive rounds.

[0049] S4. On the inference side, perform distance estimation on the input real underwater image I(x) to obtain a rough estimate of the distance information

[0050] Scene distance estimation can be done using any distance estimation algorithm. For example, UDepth can be used to obtain a relative estimate of scene distance. As a monocular distance estimation framework, UDepth has demonstrated its efficiency and accuracy in multiple visual tasks. UDepth first converts the image from the RGB domain to the RMI domain, uses MobileNetV2 as the backbone network, extracts distance features from the RMI domain image, and generates a rough distance estimation rendering. Subsequently, the distance is further refined by integrating the mVIT model, so that the distance estimation map can capture more subtle distance changes in the image and the spatial relationship between objects. Finally, a coarse estimate of the distance information of the monocular scene distance is generated through a convolutional regressor.

[0051] S5. Roughly estimate the underwater image distance obtained in step S4 Make corrections to get the corrected distance estimate

[0052]

[0053] in, is a rough estimate of the distance, and MAX is the matrix The maximum value operation of the middle element, MIN is to take the matrix Minimum operation of elements.

[0054] S6. Estimating the distance between the real underwater image I(x) and the modified underwater image obtained in step S5 The trained neural network in step S3 is input to obtain the restored underwater image.

[0055] In summary, it can be seen that the method of the present invention uses distance information to guide underwater image synthesis, fully reflects the absorption and scattering effects of water on light propagation, and sends the synthetic underwater image, distance map, and attenuation degree that are distorted with distance as training data into the network for learning, so that the network can correctly estimate the low-distortion image that reflects the real scene information and the attenuation degree of the input image, and can effectively restore the image attenuation caused by the absorption and scattering effects of the underwater medium. It has the advantages of high reliability and strong stability, and can provide convenience for subsequent computer vision tasks.

[0056] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An attenuation-aware underwater image restoration method, The following steps are involved: S1. Generate a random number as the target attenuation degree of the synthetic underwater image; S2. Based on the random number generated in step S1, taking into full account the absorption and scattering effect of water on light, the training data with different attenuation degrees are synthesized through the underwater image imaging formula; S3. On the training side, the scene relative distance and the underwater image synthesized in step S2 are used as inputs, the structural similarity and mean square error between the restored image output by the network and the true non-attenuated reference image are calculated, and the cross entropy between the attenuation degree predicted by the network and the actual attenuation degree is calculated, and the arithmetic sum of the three is used as the loss function to guide the neural network training; S4. On the inference side, performing distance estimation on the input real underwater image; S5. Correcting the underwater image distance estimation obtained in step S4; S6. Input the distance estimation of the real underwater image and the corrected underwater image obtained in step S5 into the neural network trained in step S3 to obtain a restored underwater image.

2. According to the attenuation-aware underwater image restoration method of claim 1, It is characterized in that The specific process of step S1 is as follows: For each training data image J(x), record its attenuation degree N d is 0, and a positive integer i between 1 and 10 is randomly generated as the target attenuation degree of the synthetic underwater image.

3. The attenuation-aware underwater image restoration method according to claim 2, It is characterized in that The specific process of step S2 is as follows: Synthesize training data according to the following formula Among them, α is the absorption coefficient, β is the scattering coefficient, and A is the atmospheric background light, which is a floating point number between 0 and 1; To normalize the distance information, it is calculated by the following formula: Among them, MAX is the operation of taking the maximum value of the elements in the matrix d(x), MIN is the operation of taking the minimum value of the elements in the matrix d(x), and RANDOM(0,2) is the operation of taking a random number between 0 and 2. The scattering coefficient β is calculated by the following formula: β R =RANDOM(0,0.02)+0.15 The absorption coefficient α is calculated by the following formula: α B =RANDOM(0,0.07)+0.05 After completing an image synthesis calculation, the attenuation degree N d Add 1 and use the attenuation degree as a label to generate a synthetic image Binding; Judge the positive integer i. If the positive integer i is greater than 1, then subtract 1 from the positive integer i. Repeat step S2 again; otherwise go to step S3.

4. The attenuation-aware underwater image restoration method according to claim 3, It is characterized in that The specific process of step S3 is as follows: normalize the distance information The underwater image synthesized with step S2 As input, calculate the restored image output by the network The structural similarity loss and mean square error loss with the true unattenuated image J(x) are used to calculate the attenuation degree predicted by the network. The actual attenuation degree N d The cross entropy of the three is used as the loss function to guide the neural network training; The neural network structure model is divided into three modules: feature extraction, feature fusion, and output prediction; The feature extraction module uses tf-efficientnet-lite3 to extract multi-scale features of the attenuation image and the range image with an input length of H, a width of W, and a channel number of 3, and finally obtains two feature maps with a channel number of 24 in feature 1, two feature maps with a channel number of 48 in feature 2, two feature maps with a channel number of 96 in feature 3, and two feature maps with a channel number of 192 in feature 4; The feature fusion module is divided into an integrated feature module and an upsampling module. The integrated feature module adds the feature maps of the same scale in the feature extraction module to obtain a feature map integrating distance information and attenuation information. The integrated feature module takes the feature 1, feature 2, feature 3, and feature 4 output by the feature extraction module as input, integrates the distance information and attenuation information, and outputs integrated feature 1, integrated feature 2, integrated feature 3, and integrated feature 4. The upsampling module is divided into two processing paths, which process the attenuation feature and the distance feature output by the feature extraction module respectively, and the two processing paths also input the integrated feature output by the integrated feature module; in path 1, the upsampling first layer input 1 is the attenuation feature in the feature 4 output by the feature extraction module, and input 2 is the integrated feature 4 output by the integrated feature module; The input 1 of the upsampling layer 2 is the output of the upsampling layer 1, and the input 2 is the integrated feature 3 in the integrated feature module; Input 1 of the upsampling layer 3 is the output of the upsampling module layer 2, and input 2 is the integrated feature 2 in the integrated feature module; The input 1 of the 4th layer of upsampling is the output of the 3rd layer of the upsampling module, and the input 2 is the integrated feature 1 in the integrated feature module; each layer of upsampling passes the input 1 through the residual convolution unit, then adds the feature map with the input 2 and passes it through the residual convolution unit once, and finally performs an upsampling as the output. A residual convolution unit contains two ReLU activation functions + convolution cycles; in path 2, the input 1 of the 1st layer of upsampling is the distance feature in the feature 4 output by the feature extraction module, and the input 2 is the integrated feature module 4; the architecture of the 2nd, 3rd, and 4th layers is the same as path 1; The output prediction module is divided into a reconstruction layer and a prediction layer. The reconstruction layer reconstructs the two feature maps output by the 4th layer of the upsampling module into a size of length H, width W, and number of channels 3, and limits their values ​​to 0-1. The reconstruction result of channel 1 is recorded as the reconstructed feature map f1, and the reconstruction result of channel 2 is recorded as the reconstructed feature map f2; the reconstruction layer is composed of convolution-upsampling-convolution-ReLU activation function-convolution-Sigmoid activation function. In the prediction layer, the image is first restored according to the restoration formula based on the attenuated image, the reconstructed feature maps f1 and f2 to obtain the restored image The recovery formula is: Among them, f1 is the reconstructed path 1 feature map, f2 is the reconstructed path 2 feature map, tanh is the hyperbolic tangent function, and e is a natural constant. The information of the reconstructed feature map is integrated by multiplying the reconstructed feature map f1 and the reconstructed feature map f2, and then the feature map is converted into a high-dimensional vector through convolution-average pooling-convolution-average pooling-average pooling. Finally, the result of the two linear layers + Softmax activation function is used as the attenuation degree prediction result. The calculation method of the loss function L is: Among them, SSIM represents structural similarity calculation, MSE represents mean square error calculation, and CE represents cross entropy calculation; Train the network and backpropagate the model loss function L. Stop training when the loss stops decreasing for four consecutive rounds.

5. The attenuation-aware underwater image restoration method according to claim 4, It is characterized in that The specific process of step S5 is as follows: roughly estimate the distance of the underwater image obtained in step S4 Make corrections to get the corrected distance estimate in, is a rough estimate of the distance, and MAX is the matrix The maximum value operation of the middle element, MIN is to take the matrix Minimum operation of elements.

Citation Information

Patent Citations

  • Image defogging method based on depth estimation assistance

    CN111553856A

  • Underwater polarization image restoration method based on polarization and wavelength attenuation joint optimization

    CN112070683A

  • Image defogging method based on double-path residual convolutional neural network

    CN112365414A

  • Underwater image enhancement network and method based on physical and deep learning dual guidance

    CN115272115A

  • Underwater image restoration method based on water body absorption compensation

    CN117670739A

Cited By

  • Degenerated image restoration method based on perception guidance

    CN119624786A