An underwater image enhancement method based on cross-knowledge embedding learning

Abstract

This invention discloses an underwater image enhancement method based on cross-knowledge embedding learning. It embeds three types of cross-knowledge—red channel knowledge, contrast knowledge, and gradient knowledge—of underwater images into a deep learning model. Guided by multiple knowledge bases, it improves the adaptability of the deep learning model to underwater optical scenes, comprehensively solves the problems of wavelength selective attenuation and forward and backward scattering in the underwater imaging process, and effectively improves the data quality of underwater images.

Description

Technical Field

[0001] This invention relates to an underwater image enhancement method based on cross-knowledge embedding learning, belonging to the field of image processing technology. Background Technology

[0002] In underwater imaging, light attenuation and scattering often occur during propagation, leading to image quality degradation. The attenuation characteristics of light in water differ from those on land. Compared to red, which is the best absorber, blue and green short-wavelengths have the smallest absorption attenuation coefficients, so underwater images often appear bluish-green. Simultaneously, the presence of large molecular particles in water reflects light, causing scattering during imaging. Therefore, underwater images generally exhibit color degradation, low contrast, and blurred details.

[0003] Clear underwater images are crucial for underwater operations. Initially, underwater image processing focused on the image itself. Image processing techniques adjusted individual pixels to eliminate noise and improve image quality; this method is known as non-physical model image enhancement. As research into underwater imaging deepened, considering the unique characteristics of underwater imaging processes, many studies based on physical models began to invert underwater degradation processes, establishing underwater imaging degradation models. By analyzing model parameters, images were restored to a non-degraded state. With the development of deep learning, deep learning-based underwater image enhancement methods have become mainstream. However, simply improving the model structure is insufficient; for underwater images, image enhancement methods struggle to handle the complex underwater environment. Summary of the Invention

[0004] Purpose of the invention: To address the problems and shortcomings of existing technologies, this invention combines deep learning with cross-physical knowledge. From network design to model training, it incorporates prior physical knowledge to guide the process, proposing a cross-knowledge embedding model. This method solves the problem that traditional underwater image enhancement methods struggle to handle complex underwater environments, thereby improving underwater image enhancement capabilities.

[0005] Technical solution: An underwater image enhancement method based on cross-knowledge embedding learning, comprising the following steps:

[0006] S1. Based on the underwater light propagation process, construct an underwater imaging model and analyze the light attenuation mechanism of the target during the underwater imaging process;

[0007] S2. Based on the light attenuation mechanism in the underwater imaging process, the underwater degradation model is inverted to obtain the red channel knowledge, contrast knowledge, and gradient knowledge of the current underwater image.

[0008] S3. Design an encoder-decoder deep learning network that embeds three types of cross-knowledge: red channel knowledge, contrast knowledge, and gradient knowledge. Propose a cross-knowledge guidance module to embed cross-knowledge into the model. Red channel knowledge is embedded into the encoder module, contrast knowledge is embedded into the decoder module, and gradient knowledge is used as the loss function to train the network model.

[0009] S4. Train the cross-knowledge embedding model designed in S3 using an underwater image database until it converges;

[0010] S5. Input the current original underwater image into the trained cross-knowledge embedding model, and output the enhanced underwater image result.

[0011] Preferably, step S1 constructs an underwater imaging model based on the underwater light propagation process, and the specific formula is as follows:

[0012] I(x)=J(x)t(x)+A(1-t(x)) (1)

[0013] Where I(x) is the light ray reaching the imaging plane, J(x) is the direct component of the target reflected light reaching the imaging plane, A is the background light, t(x) is the transmission rate, and x is the pixel.

[0014] According to Beer-Lambert law, the light beam decreases exponentially with respect to distance d(x) during transmission, where β is the attenuation coefficient. The specific formula for the transmission rate t(x) is:

[0015] t(x)=e -βd(x) (2). Preferably,

[0016] Step S2, based on the light attenuation mechanism that red waves attenuate faster with increasing distance during underwater imaging, inverts the degradation model of the underwater red channel. The specific formula is as follows:

[0017] 1-I R =t(1-J R )+(1-t)(1-A R (3)

[0018] Based on the red channel degradation model, we propose red channel knowledge for underwater images, namely, the red channel transmittance estimation of clear images, with the specific formula as follows:

[0019]

[0020] Among them, I R (y), IG(y), I B (y) represents the intensity of the red, green, and blue channels of the underwater image, respectively. CLet represent the intensity of each color channel of the underwater background light, C∈{R,G,B} represent the three color channels: red, green, and blue, y∈Ω(x) represent the pixel neighborhood of x, min() is the function to find the minimum value, and max() is the function to find the maximum value.

[0021] Preferably, in step S2, assuming that the scene depth is similar in each block after the underwater image is segmented, the optimal transmission rate is found in each block, and the underwater contrast degradation model is inverted. The specific formula is as follows:

[0022]

[0023] Where J(y) is the sharp image, t is the transmission rate, and A is the background light.

[0024] By adjusting the transmission rate of each small block, image contrast is improved at the pixel level, thus proposing contrast knowledge, namely, the contrast transmission rate estimate of a clear image, with the specific formula as follows:

[0025]

[0026] (6) Among them, I c (x)={I r (x),I g (x),I b (x)} is the original image, A C ={A R A G A B Let} be the background light, y∈Ω(x) be the pixel neighborhood of x, min() be the minimum value solution function, and max() be the maximum value solution function.

[0027] Preferably, step S2 involves retrieving a degradation model of the underwater gradient based on the fact that the direct components of underwater imaging generally have a lot of texture and significant image gradients; while the backscattering component layer information of underwater images is relatively smooth and the gradient approaches zero. The specific formula is as follows:

[0028] I = L p (λ)e -α(λ)l +E b (λ) (7)

[0029] Among them, L p (λ) represents the target reflected light ray on the imaging plane, e -α(λ)l E is the optical attenuation coefficient. b (λ) is the backscattering component of the background light.

[0030] The gradient distribution of underwater images is modeled using the following formula:

[0031]

[0032] in, Let J and B be the probability density functions of the gradient distributions of the direct component J and the scattering component B, respectively.

[0033] In a sharp image, the gradient of the direct component J is maximized, while the gradient of the scattering component B approaches zero, achieving optimal separation between the sharp and noisy layers. Gradient knowledge, i.e., the gradient optimization function, is proposed, with the following formula:

[0034]

[0035] Here, min() is the function for finding the minimum value.

[0036] Preferably, the cross-knowledge guidance module proposed in step S3 combines the input image with the prior image and the transmission image. The prior knowledge serves as a feature selector, which can label the features that need special attention in a weighted manner. Specifically, it includes the following steps: first, inverting the transmission image; multiplying the input feature image with the transmission image based on the prior knowledge pixel by pixel; and adding the result of the multiplication to the input feature image pixel by pixel as a weight.

[0037] Preferably, step S3 proposes a gradient knowledge loss function based on gradient knowledge.

[0038] Preferably, step S3, constructing a cross-knowledge embedding model, specifically includes the following steps:

[0039] Step S301: Based on the encoder-decoder network as the basic framework and the residual module as the basic module, the encoder extracts features through downsampling and the decoder recovers the original dimensions through upsampling; the input of the encoder-decoder network is an RGB image, a red channel transport map, and a contrast transport map.

[0040] In step S302, prior knowledge is added to the network through the cross-knowledge guidance module. Based on the characteristic that underwater red waves decay quickly, red channel knowledge is added when the encoder extracts features. In order to improve the contrast of the enhancement effect, contrast knowledge is added to the decoder.

[0041] In step S303, the encoder-decoder network incorporates a channel attention mechanism, which uses different weights to label features containing more information.

[0042] Step S304: Gradient knowledge is added as a loss function to the model training. As part of the model training loss function, it is linearly combined with the mean squared error loss function and the perceptual loss function.

[0043] Step S4 improves the accuracy of the cross-knowledge embedding network model designed in Step S3 by learning and training on data from the underwater image database.

[0044] Step S5: Input the original underwater image into the model trained in step S4 to obtain a clear enhanced underwater image, thus enhancing the original underwater image.

[0045] Beneficial effects: Through comparative experiments with existing methods, the cross-knowledge embedding learning method proposed in this invention can achieve better results in color correction and preservation of image details. Attached Figure Description

[0046] Figure 1 This is a flowchart of the underwater image enhancement method using cross-knowledge embedded learning, as described in an embodiment of the present invention.

[0047] Figure 2 This is a diagram of the cross-knowledge embedded network structure in an embodiment of the present invention;

[0048] Figure 3 This is a structural diagram of the cross-knowledge guidance module according to an embodiment of the present invention;

[0049] Figure 4 These are comparison images of underwater image enhancement effects in embodiments of the present invention. Detailed Implementation

[0050] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0051] The underwater image enhancement method based on cross-knowledge embedding learning first inverts the underwater imaging degradation process and models the underwater imaging process. Based on this, prior knowledge of underwater imaging is derived for different degradation problems.

[0052] Secondly, based on the combination of cross-knowledge and neural networks, multi-knowledge embedding is achieved, and a cross-knowledge guidance module and a prior knowledge loss function are designed.

[0053] Finally, based on the encoder-decoder framework, a cross-knowledge embedding model is constructed. This model combines prior knowledge with neural networks and uses prior knowledge to guide the learning and training of the network. The trained model is used to process underwater images and achieve underwater image enhancement.

[0054] like Figure 1 Specifically, the process includes the following steps:

[0055] S1. Based on the underwater light propagation process, construct an underwater imaging model, the specific content of which is as follows:

[0056] First, according to the McGlamery-Jaffe underwater imaging model, the light energy obtained by the underwater imaging system is divided into three parts: directly transmitted light energy, forward-scattered light energy, and back-scattered light energy; the specific formula is as follows:

[0057] I(x)=J(x)t(x)+A(1-t(x)) (1)

[0058] Where I(x) is the light ray reaching the imaging plane, J(x) is the direct component of the target reflected light reaching the imaging plane, A is the background light, t(x) is the transmission rate, and x is the pixel.

[0059] According to Beer-Lambert law, the light beam decreases exponentially with respect to distance d(x) during transmission, where β is the attenuation coefficient. The specific formula for the transmission rate t(x) is:

[0060] t(x)=e -βd(x) (2)

[0061] S2. Based on the attenuation problem during underwater imaging, the underwater degradation model is inverted to obtain prior knowledge of underwater imaging. The specific content is as follows:

[0062] (1) Constructing knowledge of red channels

[0063] In underwater imaging, red waves attenuate faster with increasing distance. To reflect this characteristic, the red channel prior of the underwater image is adjusted compared to the original model in the dark channel prior, while other channels remain unchanged. A method is proposed to recover a clear image using the red channel's transmission rate, with the specific formula as follows:

[0064] 1-I R =t(1-J R )+(1-t)(1-A R (3)

[0065]

[0066] Wherein, the original image and the sharpened image are respectively I = (I R ,I G ,I B ) and J = (J R J G J B The water-light component is A = (A R A G A B ), where Ω(x) is the pixel neighborhood of x, and t is the transmission rate. Let be the red channel transmittance of the clear image, and min() be the function to find the minimum value.

[0067] (2) Contrast knowledge construction

[0068] In underwater imaging, assuming that the scene depth is similar in each image block, the optimal transmission rate is found for each block. The block-based transmission rate is sufficient to improve image contrast. Since contrast is inversely proportional to the projection rate t, assuming the enhanced image is J(y), the transmission rate estimate based on contrast knowledge is: The specific formula is as follows:

[0069]

[0070] Where J(y) is the sharp image, t is the transmission rate, A is the background light, and I is the image sharpness. c (x)={I r (x),I g (x),I b (x)} is the original image, A C ={A R A G A B Let} be the background light, y∈Ω(x) be the pixel neighborhood of x, min() be the minimum value solution function, and max() be the maximum value solution function.

[0071] (3) Constructing gradient knowledge

[0072] Analyzing the underwater light scattering process, the original image I can be divided into target reflection components and background backscattering components. The direct component J of underwater imaging generally has more texture and significant image gradients. The backscattering component layer information of underwater imaging is relatively smooth, and the gradient approaches zero. Therefore, the gradient distribution density function ε of the underwater image is modeled. According to the model, when ε reaches its minimum value, the gradient of the sharp layer is the largest, and the gradient of the noise layer approaches zero, achieving optimal separation between the sharp layer and the noise layer. The specific formula is as follows:

[0073] I = L p (λ)e -α(λ)l +E b (λ) (7)

[0074]

[0075] Among them, L p (λ) represents the target reflected light ray on the imaging plane, e -α(λ)l E is the optical attenuation coefficient. b (λ) is the backscattering component of the background light. Let J and B be the probability density functions of the gradient distributions of the direct component J and the scattering component B, respectively, and min() be the function for finding the minimum value.

[0076] S3. Based on the method of combining cross-knowledge with neural networks, design a cross-knowledge guidance module, through... Figure 3The module shown combines the input image with prior knowledge; first, the transport map is inverted. Then, the input feature map is multiplied pixel-by-pixel with the transport map based on prior knowledge, and the result is added pixel-by-pixel as weights to the input feature map. The specific formula is:

[0077]

[0078] Where X represents the input feature map and T represents the transmission map. This indicates pixel-by-pixel multiplication. This indicates pixel-by-pixel addition. This represents a feature map that incorporates prior knowledge.

[0079] To achieve the feature of cross-knowledge embedding, a gradient loss function is designed. The original image is fed into the cross-knowledge-guided network for training. The result obtained in each round is passed through the gradient loss function to optimize the model. The specific formula is as follows:

[0080]

[0081] Among them, ||·|| F Let f be the Frobenius norm, f1 be the first-order horizontal gradient operator, f2 be the transpose of the first-order gradient operator, and f La λ is a second-order Laplace operator, and ρ are weighting coefficients.

[0082] Constructing a cross-knowledge embedding network includes the following sub-steps:

[0083] S301. This network is based on an encoder-decoder framework and consists of residual modules; the residual modules are as follows: Figure 2 As shown, each residual module contains two residual blocks, each residual block consists of three 3×3 convolutional kernels with ReLU activation functions, and a single 3×3 convolutional kernel; each residual block is connected by a residual structure.

[0084] S302. Add the cross-knowledge guidance module proposed in S3 to the network, such as... Figure 2 As shown, the network input consists of an RGB image, a red channel transport map, and a contrast transport map; each residual module is connected to the cross-knowledge guidance module via a concat operation; the red channel transport map and the RGB image are input into the encoder, where features are extracted by downsampling; in the decoder, the contrast transport map is combined with the RGB feature map and upsampled to restore the size.

[0085] Add a connection between S303, encoder, and decoder Figure 2 The SE attention mechanism in the model serves as a channel attention module. After the encoder feature map is weighted by the channel attention module, it is combined with the decoder feature map to obtain the relevant features of the underwater image.

[0086] S304. The gradient loss function is linearly combined with the mean squared error function and the perceptual loss function as the loss function for optimizing model training, achieving a balance between visual effects and quantitative metrics; mean squared error loss function. Calculate the mean of the squared differences between the predicted value f(x) and the true value y; perceptual loss function. The computational space is transformed into the feature space, defined by the ReLU activation layer of the pre-trained ImageNet VGG-19 network; during model training, the gradient optimization function... As part of the objective function, optimal separation of the clear layer and the noisy layer can be achieved. The specific formula is as follows:

[0087]

[0088] in, Represents pixel value x i The predicted value, y i The actual value; This represents a pre-trained VGG-19 network. Let C represent the j-th convolutional layer. j H j W j J represents the size of the feature map at layer j, ||·||2 is the 2-norm, and J f and B f These are the direct and scattered components of the predicted value f(x), respectively. F Let f be the Frobenius norm, f1 be the first-order horizontal gradient operator, f2 be the transpose of the first-order gradient operator, and f La For the second-order Laplace operator, λ and ρ are weighting coefficients; λ1 and λ2 are set to 0.01 and 0.05 respectively, representing the proportion of each function in the total objective function.

[0089] Step S4 improves the accuracy of the cross-knowledge embedding network model designed in Step S3 by learning and training on data from the underwater image database.

[0090] Step S5: Input the original underwater image into the model obtained in step S4 to obtain a clear enhanced underwater image, thus enhancing the original underwater image.

[0091] To verify the effectiveness of the underwater enhancement method proposed in this invention in terms of color correction and detail preservation, representative images were selected from six underwater datasets, including green underwater images, blue underwater images, and underwater images with rich detail. These images were then compared with six different underwater image processing methods, including… Figure 4 The second row describes an underwater image enhancement method based on prior knowledge of underwater scenes. Figure 4 The third row features underwater image enhancement based on gated fusion networks. Figure 4 The fourth row features underwater image enhancement based on conditional generative adversarial networks. Figure 4 The fifth line is based on underwater image enhancement using a multi-color space encoder-medium transmission guided decoder. Figure 4 The sixth line is underwater image restoration based on attenuation prior. Figure 4 The seventh line is underwater image restoration based on gradient priors. Figure 4 The results show a comparison between the method proposed in this invention and other methods, demonstrating that the present invention has good effects, with better color correction and detail restoration.

Claims

1. An underwater image enhancement method based on cross-knowledge embedding learning, characterized in that, Includes the following steps: S1. Based on the underwater light propagation process, construct an underwater imaging model and analyze the light attenuation mechanism of the target during the underwater imaging process; S2. Based on the light attenuation mechanism in the underwater imaging process, the underwater degradation model is inverted to obtain the red channel knowledge, contrast knowledge, and gradient knowledge of the current underwater image. S3. Design an encoder-decoder deep learning network that embeds three types of cross-knowledge: red channel knowledge, contrast knowledge, and gradient knowledge. Propose a cross-knowledge guidance module to embed cross-knowledge into the model. Red channel knowledge is embedded into the encoder module, contrast knowledge is embedded into the decoder module, and gradient knowledge is used as the loss function to train the network model. S4. Train the encoder-decoder deep learning network designed in S3 using an underwater image database until it converges. S5. Input the current original underwater image and output the enhanced underwater image result.

2. The underwater image enhancement method based on cross-knowledge embedding learning according to claim 1, characterized in that, Step S1 involves constructing an underwater imaging model based on the underwater light propagation process. The specific formula is as follows: I(x)=J(x)t(x)+A(1-t(x)) (1) Where I(x) is the light ray reaching the imaging plane, J(x) is the direct component of the target reflected light reaching the imaging plane, A is the background light, t(x) is the transmission rate, and x is the pixel.

3. The underwater image enhancement method based on cross-knowledge embedding learning according to claim 1, characterized in that, Step S2, based on the light attenuation mechanism that red waves attenuate faster with increasing distance during underwater imaging, inverts the degradation model of the underwater red channel. The specific formula is as follows: 1-I R =t(1-J R )+(1-t)(1-A R (3) Based on the red channel degradation model, the red channel knowledge of underwater images is proposed, namely, the red channel transmittance estimation of clear images, and the specific formula is as follows: Among them, I R (y), I G (y), I B (y) represents the intensity of the red, green, and blue channels of the underwater image, respectively. C Let represent the intensity of each color channel of the underwater background light, C∈{R,G,B} represent the three color channels: red, green, and blue, y∈Ω(x) represent the pixel neighborhood of x, min() is the function to find the minimum value, and max() is the function to find the maximum value.

4. The underwater image enhancement method based on cross-knowledge embedding learning according to claim 1, characterized in that, In step S2, assuming that the scene depth is similar in each block after the underwater image is segmented, the optimal transmission rate is found in each block, and the underwater contrast degradation model is inverted. The specific formula is as follows: Where J(y) is the sharp image, t is the transmission rate, and A is the background light; By adjusting the transmission rate of each block, image contrast is improved at the pixel level, thus deriving contrast knowledge, namely, the contrast transmission rate estimate of a clear image, with the specific formula as follows: Among them, I c (x)={I r (x),I g (x),I b (x)} is the original image, A C ={A R A G A B Let} be the background light, y∈Ω(x) be the pixel neighborhood of x, min() be the minimum value solution function, and max() be the maximum value solution function.

5. The underwater image enhancement method based on cross-knowledge embedding learning according to claim 1, characterized in that, The degradation model of the underwater gradient is derived by inversion, and the specific formula is as follows: I=L p (l)e -α(λ)l +E b (l) (7) Among them, L p (λ) represents the target reflected light ray on the imaging plane, e -α(λ)l E is the optical attenuation coefficient. b (λ) is the backscattering component of the background light; The gradient distribution of underwater images is modeled using the following formula: in, Let J and B be the probability density functions of the gradient distributions of the direct component J and the scattering component B, respectively. In a sharp image, the gradient of the direct component J is maximized, while the gradient of the scattering component B approaches zero, achieving optimal separation between the sharp and noisy layers. Gradient knowledge, namely the gradient optimization function, is proposed, with the following formula: Here, min() is the function for finding the minimum value.

6. The underwater image enhancement method based on cross-knowledge embedding learning according to claim 1, characterized in that, The cross-knowledge guidance module proposed in step S3 combines the input image with the prior image and the transmission image. The prior knowledge is used as a feature selector to label the features that need special attention in a weighted manner. Specifically, it includes the following steps: first, invert the transmission image; multiply the input feature image with the transmission image based on the prior knowledge pixel by pixel; and add the result of the multiplication to the input feature image pixel by pixel as a weight.

7. The underwater image enhancement method based on cross-knowledge embedding learning according to claim 1, characterized in that, Step S3 proposes a gradient knowledge loss function based on gradient knowledge.

8. The underwater image enhancement method based on cross-knowledge embedding learning according to claim 1, characterized in that, Step S3, constructing a cross-knowledge embedding model, specifically includes the following steps: Step S301: Based on the encoder-decoder network as the basic framework and the residual module as the basic module, the encoder extracts features through downsampling and the decoder recovers the original dimensions through upsampling; the input of the encoder-decoder network is an RGB image, a red channel transport map, and a contrast transport map. In step S302, prior knowledge is added to the network through the cross-knowledge guidance module. Based on the characteristic that underwater red waves decay quickly, red channel knowledge is added when the encoder extracts features. In order to improve the contrast of the enhancement effect, contrast knowledge is added to the decoder. In step S303, the encoder-decoder network incorporates a channel attention mechanism, which uses different weights to label features containing more information. Step S304: Gradient knowledge is added as a loss function to the model training. As part of the model training loss function, it is linearly combined with the mean squared error loss function and the perceptual loss function.

Images