Underwater image absorption and scattering dual recovery method based on implicit distance estimation
Through implicit distance estimation and neural network prediction, the quality degradation caused by water absorption and scattering in underwater images is solved, efficient image recovery is achieved, image quality and robustness are improved, and it is suitable for a variety of water environments.
Patent Information
- Application Number
- CN202411160698.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-22
AI Technical Summary
Existing underwater image recovery methods are difficult to effectively solve the problem of image quality degradation caused by water absorption and scattering. Especially in deep-sea areas, traditional methods lack physical model support, and methods based on external distance information are insufficient in accuracy.
The dual recovery method of underwater image absorption and scattering using implicit distance estimation is adopted. By synthesizing the absorption transmission map and scattering transmission map, using neural networks to perform feature extraction and fusion, predicting background light and transmission maps, and realizing image recovery.
This method can accurately simulate the absorption and scattering effect of water medium on light propagation without external information, improve the contrast and saturation of the image, restore image details and depth information, facilitate deployment, strong robustness, and suitable for multiple applications Attenuation of water bodies.
Smart Images

Figure CN119006314A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to a method for restoring an underwater degraded image caused by water absorption and scattering. Background Art
[0002] As human beings continue to explore and develop the underwater world, the demand for high-definition and high-fidelity images of underwater environments is growing. As an important means to improve underwater visual quality, underwater image enhancement technology has broad development prospects and is full of potential. However, using current underwater images as input for high-level visual tasks is very likely to fail to produce satisfactory output.
[0003] Unlike land environments, light in underwater environments is not only affected by the scattering of water particles, but also by the absorption of light by dissolved substances in the water (such as plankton, minerals, etc.). This dual effect often causes underwater images to have problems such as color distortion, reduced contrast, and blurred texture, which seriously affects the subsequent analysis and application of the images. Especially in deep sea areas, these problems become more serious due to the significant increase in the distance that light travels, greatly limiting the quality of underwater images.
[0004] Existing underwater image restoration methods can be roughly divided into two categories: one is based on traditional image processing technology, such as histogram equalization-based methods, biological vision mechanism-based algorithms, image fusion-based algorithms, etc. These methods treat underwater image enhancement as a traditional image enhancement problem, and enhance the image by improving image brightness, contrast, and eliminating image noise. Although these methods can improve image quality to a certain extent, due to the lack of support from physical models, it is difficult to fundamentally solve the problem of underwater image degradation. The other is based on physical models, such as underwater image restoration algorithms based on radiation transfer equations. This type of method compares the degradation process of underwater images to the image quality degradation caused by atmospheric scattering. On this basis, it cleverly uses these prior knowledge to reduce the unknown variables in the atmospheric scattering model, and then reversely deduce the negative effects of light scattering on image quality, and finally achieves effective image restoration and enhancement. However, in most water environments, there is no order of magnitude difference between the attenuation of light due to particle absorption in the water body and the attenuation caused by particle scattering. Therefore, if the absorption effect of water on light is ignored (i.e., t in the model is ignored), the image quality will be effectively restored and enhanced. a ), only treating the image quality degradation caused by water attenuation as a problem similar to atmospheric scattering, which is obviously insufficient on a theoretical basis. In practical applications, the effectiveness of such methods is often limited to a short imaging distance. As the imaging distance increases, the absorption of light by water will show an exponential growth trend, which directly leads to the rapid deterioration of the performance of such algorithms, making it difficult to effectively achieve the goal of image enhancement.
[0005] In recent years, the image quality degradation caused by water absorption attenuation has gradually received attention. Some methods have begun to use the distance estimation input by a third party as a guide, and combine the water absorption coefficient to estimate the quality degradation of the image caused by water absorption. In principle, this type of method is more in line with the physical process of underwater image degradation and has relatively good results. When there is accurate distance information external input, the best results can be obtained. However, due to its reliance on external input distance estimation, it is inconvenient in practical applications. When the external input distance estimation is inaccurate, the effect is seriously degraded. Summary of the invention
[0006] In order to solve the above problems existing in the prior art, the present invention proposes an underwater image absorption and scattering dual restoration method for implicit distance estimation.
[0007] The specific technical solution of the present invention is: an underwater image absorption and scattering dual recovery method for implicit distance estimation, comprising the following steps:
[0008] S1. Considering the absorption and scattering effect of water on light, the synthetic absorption and transmission graph t a (x) and the scattered transmission diagram t s (x) Synthesizing training data with different attenuation levels through underwater image imaging formula to obtain underwater attenuation images for training;
[0009] S2. extracting the feature map of the training underwater attenuation image obtained in step S1;
[0010] S3. Fusing the feature maps extracted in step S2;
[0011] S4. Predict the absorption transmission map of the absorbing water body based on the features fused in step S3 Water scattering transmission map and background light Get the trained network;
[0012] S5. Send the real input image to the trained network to get the background light Absorption transmission diagram Scattering transmission diagram Substitute it into the underwater image restoration formula to obtain the final output restored image.
[0013] Furthermore, the specific process of step S1 is as follows: randomly generate the scattering coefficient β and the absorption coefficient α, and the specific generation rule is calculated by the following formula:
[0014]
[0015] β R =0.1×RANDOM(0,1)+0.15
[0016]
[0017] α B =0.2×RANDOM(0,1)+0.1
[0018] Among them, RANDOM (0, 1) is an operation to obtain a random number between 0 and 1;
[0019] Then the background light A is randomly generated, and the specific generation rule is calculated by the following formula:
[0020] A=0.3×RANDOM(0,1)+0.4
[0021] The absorption transmission map and the scattering transmission map are generated according to the generated scattering coefficient β, absorption coefficient α and depth information d(x). The specific generation formula is:
[0022] t s (x) = e -βd(x)
[0023] t a (x) = e -αdd(x)
[0024] Combined with the absorption and transmission graph a (x) and the scattered transmission diagram t s (x) Synthesize underwater attenuated images for training The synthesis formula is
[0025]
[0026] Where J(x) is the reference image without attenuation.
[0027] Furthermore, step S4 obtains a trained network which is trained specifically under the guidance of the prediction loss function.
[0028] Beneficial effects of the present invention: Compared with the traditional underwater image synthesis technology, the method of the present invention comprehensively considers the differences in optical propagation characteristics between water medium and atmospheric aerosol medium, especially the absorption effect on light propagation, and restores the absorption transmission map and the scattering transmission map to make the details of the restoration result more accurate. Compared with the underwater image restoration method guided by external distance information, the method of the present invention does not require external information, is more convenient to deploy, and is more robust. It has the advantages of solid physical foundation, high stability, and wide synthetic application, and can accurately simulate the attenuation phenomenon of different properties caused by various types of water bodies. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 A schematic flow chart of the underwater image absorption and scattering dual restoration method for implicit distance estimation of the present invention;
[0030] Figure 2 4 is a diagram of a neural network structure according to an embodiment of the present invention. DETAILED DESCRIPTION
[0031] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments. The specific process is as follows (floating point numbers are used for calculation in the following calculation process):
[0032] In this embodiment, the KITTI dataset image is used as the data source to synthesize the training data. The KITTI dataset image is denoted as J(x), which is a reference image without attenuation, and the reference depth information provided by the dataset is denoted as d(x).
[0033] The process of the underwater image absorption and scattering dual restoration method for implicit distance estimation of the present invention is as follows: Figure 1 As shown, the specific steps include:
[0034] S1. Considering the absorption and scattering effect of water on light, the scattering coefficient β and absorption coefficient α are randomly generated first. The specific generation rules are calculated by the following formula:
[0035]
[0036] β R =0.1×RANDOM(0,1)+0.15
[0037]
[0038] α B =0.2×RANDOM(0,1)+0.1
[0039] Among them, RANDOM(0,1) is an operation of taking a random number between 0 and 1; then the background light A is randomly generated, and the specific generation rule is calculated by the following formula:
[0040] A=0.3×RANDOM(0,1)+0.4
[0041] The absorption transmission map and the scattering transmission map are generated according to the generated scattering coefficient β, absorption coefficient α and the reference depth information d(x) provided by the KITTI dataset. The specific generation formula is:
[0042] t s (x) = e -βd(x)
[0043] t a (x) = e -αd(x)
[0044] Combined with the absorption and transmission graph a (x) and the scattered transmission diagram t s(x) Synthesize underwater attenuated images for training The synthesis formula is
[0045]
[0046] Where J(x) is the reference image without attenuation, which comes from the KITTI dataset image.
[0047] S2. Using a feature extraction module, extract the feature map of the training underwater attenuation image obtained in step S1.
[0048] In this embodiment, tf_efficientnet_lite3 is specifically used as a feature extraction network, and tf_efficientnet_lite3 is divided into four layers. The first layer input is composed of conv_stem, BatchNorm, RELU6 and 0 and 1 layers of tf_efficientnet_lite3. The output channel number is 32, and the scale is a feature map of 1 / 4 of the input image; the second layer is composed of the 2nd layer block of tf_efficientnet_lite3, and the output channel number is 48, and the scale is 1 / 8 of the input image. The feature map; the third layer is composed of the 3rd and 4th layer blocks of tf_efficientnet_lite3, and the output channel number is 136, and the scale is 1 / 16 of the input image. The fourth layer is composed of the 5th to 9th layer blocks of tf_efficientnet_lite3, and the output channel number is 384, and the scale is a feature map of 1 / 32 of the input image.
[0049] S3. Use the feature fusion module to fuse the feature maps extracted in step S2.
[0050] In this embodiment, the feature fusion module is divided into an integration feature module and an upsampling module. The integration feature module performs convolution integration on the four feature maps of the feature extraction module, and finally obtains a feature map with 24 channels in feature 1, a feature map with 48 channels in feature 2, a feature map with 96 channels in feature 3, and a feature map with 192 channels in feature 4. The input of the first layer of the feature fusion module is feature 4 with 192 channels and a scale of 1 / 32 of the input image.
[0051] The first layer input of the upsampling module is feature 4, which is upsampled once after passing through a residual convolution unit as the output, with an output channel of 96 and a feature map with a scale of 1 / 16 of the input image. A residual convolution unit contains two cycles of ReLU activation function + convolution, and finally the result of the convolution is added to the input value. The second layer input is the result of the upsampling of the first layer and feature 3. The result of the upsampling of the first layer is added to the result of the third layer of the feature extraction module after passing through a residual convolution unit, and then upsampled once after passing through a residual convolution unit as the output, with an output channel of 48 and a feature map with a scale of 1 / 8 of the input image. The third layer input is the result of the upsampling of the second layer and feature 2. The result of the upsampling of the second layer is added to the result of the third layer of the feature extraction module after passing through a residual convolution unit, and then upsampled once after passing through a residual convolution unit as the output, with an output channel of 24 and a feature map with a scale of 1 / 4 of the input image. The fourth layer input is the result of the upsampling of the second layer and feature 1. The result of the second layer upsampling is added to the result of the third layer of the feature extraction module after passing through a residual convolution unit, and then upsampled again after passing through a residual convolution unit as the output. The output channel is 12, and the scale is 1 / 2 of the feature map of the input image.
[0052] S4. Use the transmission prediction module to predict the absorption transmission map of the absorbing water body according to the features fused in step S3 Water scattering transmission map and background light Get the trained network. The specific process is as follows:
[0053] After the fourth layer of the upsampling module, a transmittance mapping layer consisting of a convolution kernel of 3, a step size of 1, and an output channel of 3 is used to map the feature map to the product of the absorption transmission map and the scattering transmission map. s ×t a , the loss function of this mapping process adopts the L1 norm loss function.
[0054] The product of the absorption transmission graph and the scattering transmission graph t s ×t a After a convolution with a kernel size of 3, a step size of 1, and an output channel of 96, as well as normalization and prelu nonlinear activation, it is converted into high-dimensional information. The high-dimensional information is sent to the absorption transmission map and scattering transmission map mapping branches respectively to obtain the absorption transmission map and scatter transmission diagram Among them, the absorption transmission map mapping branch structure is: convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - tanh nonlinear activation function - convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - prelu nonlinear activation function; the scattering transmission map mapping branch structure is: convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - tanh nonlinear activation function - convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - prelu nonlinear activation function.
[0055] Since the impact of background light is global, we first perform a convolution with a kernel of 3, a step size of 2, and an output channel of 1 on the feature map with 384 output channels and a scale of 1 / 32 of the input image in the feature extraction module to integrate all color feature information, and then perform an average pooling to obtain the global background light.
[0056] The loss function for transmission prediction is:
[0057]
[0058] Where L1 is the L1 norm loss function and TV is the total variation loss function.
[0059] During the training process, in order to ensure the smooth completion of the training, L E As the loss function, observe that when the loss does not decrease for four consecutive rounds, the training ends and the trained network is obtained.
[0060] Figure 2 A neural network structure diagram of an embodiment of the present invention is given, wherein the yellow part is a feature extraction module, the gray part is a feature fusion module, and the red part is a transmission prediction module.
[0061] S5. Send the real input image to the trained network to get the background light Absorption transmission diagram Scattering transmission diagram Substitute it into the underwater image restoration formula to obtain the final output restored image
[0062]
[0063] From the above, it can be seen that the method of the present invention fully considers the dual attenuation effects of absorption and scattering of water on light propagation. During the training process, the network is guided to implicitly estimate the distance information by constraints on the transmission map, relying on the implicit estimation ability of the distance obtained during the training process; during the reasoning process, the network can correctly estimate the absorption attenuation and scattering attenuation without external input distance estimation, and then restore the image according to the inverse imaging formula. Compared with the method based on external distance information input, the present invention can implicitly extract distance information by itself, can effectively restore the image attenuation caused by the absorption and scattering of underwater media, is more convenient to deploy, has stronger robustness, and has better generalization performance. It can not only effectively improve the contrast and saturation of the image, but also restore the depth information and detail level of the image to a certain extent, providing more reliable data support for the subsequent analysis and application of underwater images.
[0064] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.
Claims
1. An underwater image absorption and scattering dual recovery method for implicit distance estimation, comprising the following steps: S1. Considering the absorption and scattering effect of water on light, the synthetic absorption and transmission graph t a (x) and the scattered transmission graph t s (x) Synthesizing training data with different attenuation levels through underwater image imaging formula to obtain underwater attenuation images for training; S2. extracting the feature map of the training underwater attenuation image obtained in step S1; S3. Fusing the feature maps extracted in step S2; S4. Predict the absorption transmission map of the absorbing water body based on the features fused in step S3 Water scattering transmission map and background light Get the trained network; S5. Send the real input image to the trained network to get the background light Absorption transmission diagram Scattering transmission diagram Substitute it into the underwater image restoration formula to obtain the final output restored image.
2. The underwater image absorption and scattering dual restoration method for implicit distance estimation according to claim 1 is characterized in that: The specific process of step S1 is as follows: randomly generate the scattering coefficient β and the absorption coefficient α, and the specific generation rule is calculated by the following formula: b R =0.1×RANDOM(0,1)+0.15 a B =0.2×RANDOM(0,1)+0.1 Among them, RANDOM(0,1) is an operation to obtain a random number between 0 and 1; Then the background light A is randomly generated, and the specific generation rule is calculated by the following formula: A=0.3×RANDOM(0,1)+0.4 The absorption transmission map and the scattering transmission map are generated according to the generated scattering coefficient β, absorption coefficient α and depth information d(x). The specific generation formula is: t s (x)=e -βd(x) t a (x)=e -αd(x) Combined with the absorption and transmission graph a (x) and the scattered transmission graph t s (x) Synthesize underwater attenuated images for training The synthesis formula is: Where J(x) is the reference image without attenuation.
3. The underwater image absorption and scattering dual restoration method for implicit distance estimation according to claim 1, characterized in that: Step S4 obtains the trained network, which is specifically trained under the guidance of the prediction loss function.
4. The underwater image absorption and scattering dual restoration method for implicit distance estimation according to claim 2 is characterized in that: The feature extraction module is used to extract the feature map of the input image. The specific process is as follows: tf_efficientnet_lite3 is used as the feature extraction network, and tf_efficientnet_lite3 is divided into four layers. The first layer input is composed of conv_stem, BatchNorm, RELU6 of tf_efficientnet_lite3 and the block of the 0th and 1st layers in tf_efficientnet_lite3. The output channel number is 32, and the scale is a feature map of 1 / 4 of the input image; the second layer is composed of the 2nd layer block of tf_efficientnet_lite3, and the output channel number is 48, and the scale is 1 / 8 of the input image; the third layer is composed of the 3rd and 4th layer blocks of tf_efficientnet_lite3, and the output channel number is 136, and the scale is 1 / 16 of the input image; the fourth layer is composed of the 5th to 9th layer blocks of tf_efficientnet_lite3, and the output channel number is 384, and the scale is 1 / 32 of the input image.
5. The underwater image absorption and scattering dual restoration method for implicit distance estimation according to claim 4 is characterized in that: The feature fusion module is used to fuse the feature maps extracted in step S2. The specific process is as follows: The feature fusion module is divided into an integration feature module and an upsampling module. The integration feature module performs convolution on the four feature maps of the feature extraction module to integrate the features. Finally, feature maps with 24 channels in feature 1, 48 channels in feature 2, 96 channels in feature 3, and 192 channels in feature 4 are obtained. The input of the first layer of the feature fusion module is feature 4 with 192 channels and a scale of 1 / 32 of the input image. The first layer input of the upsampling module is feature 4, which is upsampled once after passing through a residual convolution unit as the output, with an output channel of 96 and a feature map with a scale of 1 / 16 of the input image; a residual convolution unit contains two cycles of ReLU activation function + convolution, and finally the result of the convolution is added to the input value; the second layer input is the result of the upsampling of the first layer and feature 3; the result of the upsampling of the first layer is added to the result of the third layer of the feature extraction module after passing through a residual convolution unit, and then upsampled once after passing through a residual convolution unit as the output, with an output channel of 48 and a feature map with a scale of 1 / 8 of the input image ; The input of the third layer is the upsampling result of the second layer and feature 2. The upsampling result of the second layer passes through a residual convolution unit once, and is added to the result of the third layer of the feature extraction module. After passing through a residual convolution unit, it is upsampled once as the output. The output channel is 24, and the scale is 1 / 4 of the input image feature map; The input of the fourth layer is the upsampling result of the second layer and feature 1. The upsampling result of the second layer passes through a residual convolution unit once, and is added to the result of the third layer of the feature extraction module. After passing through a residual convolution unit, it is upsampled once as the output. The output channel is 12, and the scale is 1 / 2 of the input image feature map.
6. The underwater image absorption and scattering dual restoration method for implicit distance estimation according to claim 5, characterized in that: The specific process of step S4 is as follows: After the fourth layer of the upsampling module, a transmittance mapping layer consisting of a convolution kernel of 3, a step size of 1, and an output channel of 3 is used to map the feature map to the product of the absorption transmission map and the scattering transmission map. s ×t a , the loss function of this mapping process adopts the L1 norm loss function. The product of the absorption transmission graph and the scattering transmission graph t s ×t a After a convolution with a kernel size of 3, a step size of 1, and an output channel of 96, as well as normalization and prelu nonlinear activation, it is converted into high-dimensional information. The high-dimensional information is sent to the absorption transmission map and scattering transmission map mapping branches respectively to obtain the absorption transmission map and scatter transmission diagram Among them, the absorption transmission map mapping branch structure is: convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - tanh nonlinear activation function - convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - prelu nonlinear activation function; the scattering transmission map mapping branch structure is: convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - tanh nonlinear activation function - convolution (convolution kernel size is 3, step size is 1, output channel is 3) - normalization operation - prelu nonlinear activation function. First, the feature map with 384 output channels and 1 / 32 scale of the input image in step S2 is convolved with a kernel of 3, a step length of 2, and an output channel of 1 to integrate all color feature information, and then the global background light is obtained by average pooling. The loss function for transmission prediction is: Where L1 is the L1 norm loss function and TV is the total variation loss function. During the training process, in order to ensure the smooth completion of the training, L E As a loss function, observe that when the loss stops decreasing for four consecutive rounds, the training ends.
7. The underwater image absorption and scattering dual restoration method for implicit distance estimation according to claim 6, characterized in that: The restored image outputted in step S5 Among them, I(x) is the real input image.
Citation Information
Patent Citations
Underwater image restoration method based on a convolutional neural network
CN109584170A
Convolutional neural network underwater image restoration method suitable for artificial and natural light sources
CN110223251A
Underwater image restoration method based on convolutional neural network
CN110838092A
Underwater image restoration method based on water body absorption compensation
CN117670739A
Real underwater image restoration method and system
CN118037599A
