Underwater image enhancement method and system based on hierarchical decomposition and fusion

Abstract

The invention discloses an underwater image enhancement method and system based on hierarchical decomposition and fusion. The method comprises the following steps: constructing an HDNet model; the HDNet model is trained; performing underwater image enhancement by adopting the trained HDNet model; according to an image processing process, the HDNet model sequentially comprises a first point convolution layer used for expanding a channel of an input image; the first sub-channel color correction module is used for splitting channels of the input image and independently adjusting color distribution of each channel; the first feature fusion module is used for fusing the color of each channel; the sub-pixel detail adjustment module is used for representing pixels and modulating signals by adopting various wave functions; the second feature fusion module is used for carrying out fusion again; the second sub-channel color correction module is used for adjusting color distribution of each channel to enable the color distribution to be in normal distribution; and the second point convolution is used for compressing the channel to output an enhanced image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an underwater image enhancement method and system based on hierarchical decomposition and fusion. Background Art

[0002] The ocean is a resource-rich ecological treasure house and is of great significance to sustainable development. In recent years, advances in marine technology are ushering in a new era of ocean exploration. However, the uniqueness of the underwater environment, especially light attenuation and light scattering, seriously reduces the quality of underwater images, resulting in images that generally appear blue-green, blurred, and lack contrast. Underwater images are a key source of information for ocean exploration missions, and their quality directly affects the effectiveness and efficiency of these activities. Therefore, underwater image enhancement technology is critical to ensuring the efficiency and effectiveness of ocean exploration activities.

[0003] Underwater image degradation is a unique nonlinear distortion phenomenon caused by multiple factors. Its complexity and irreversibility constitute a major challenge in the field of image processing. Although computer vision technology has made some progress in underwater image enhancement, existing methods still need to be further optimized and innovated in dealing with underwater image degradation problems. Traditional methods rely on the statistical characteristics and physical assumptions of images. Although they can correct color casts, they are only effective in specific underwater scenes. In recent years, deep learning technology has become increasingly popular in the field of image processing. However, in the field of underwater image enhancement, scholars often encounter obstacles in obtaining real underwater scenes and their ideal reference images. The initial research methods are to generate datasets using generative adversarial networks (GANs) or perform unpaired learning paradigms, such as WaterGAN and UWGAN. Although the model can achieve significant color enhancement effects on synthetic underwater datasets, empirical evidence shows that there are distribution differences in synthetic and real underwater datasets, which limits the generalization effect of the model on real datasets. To this end, Li et al. established a real-world underwater image enhancement dataset (UIEB), whose reference images are generated by more than a dozen excellent algorithms and selected by volunteers. The authors experimentally confirmed that the model trained on the proposed dataset has good generalization ability. This effort provides a reliable benchmark for the subsequent deep learning-based underwater image enhancement methods, and many excellent algorithms have been developed. However, these algorithms currently treat the degraded image as a whole and focus on using deep learning models to fit the dataset without fully considering the main factors of underwater image degradation: light attenuation and light scattering. This limits the further improvement of the performance of deep learning networks. For light attenuation, the attenuation degree of light waves varies depending on the color, resulting in abnormal color distribution of the RGB channels of the image, and the image often appears blue-green. The light reflected by the object is scattered by suspended particles in the water during its propagation to the camera. This scattering effect causes the intensity and direction of the light received by the camera that captures the image by collecting light signals to change, thereby reducing the contrast and clarity of the captured image. In addition, underwater real data can better reflect the underwater imaging mode and has good generalization ability, but hand-picked reference images inevitably introduce some defects in quality. Although there are semi-supervised frameworks in the prior art to achieve image enhancement, the semi-supervised framework aims to utilize labeled and unlabeled data so that the deep learning model does not rely entirely on paired datasets. Nevertheless, the model still relies on some supervised learning, and low-quality reference images may hinder its learning of high-quality images. Therefore, the underwater image enhancement network based on deep learning still needs to be further improved and innovated according to degradation factors, and the adverse effects of suboptimal reference images on the model learning process need to be alleviated. Summary of the invention

[0004] Purpose of the invention: In order to solve the shortcomings of existing methods in image enhancement and to reduce the negative impact of suboptimal reference images on the model learning process, the present invention proposes an underwater image enhancement method and system based on hierarchical decomposition and fusion.

[0005] Technical solution: An underwater image enhancement method based on hierarchical decomposition and fusion, including:

[0006] Build the HDNet model;

[0007] Train the HDNet model;

[0008] Use the trained HDNet model for underwater image enhancement;

[0009] Among them, the constructed HDNet model includes the following steps according to the image processing process:

[0010] The first point convolution layer is used to expand the channels of the input image;

[0011] The first channel-by-channel color correction module is used to split the channels of the input image and adjust the color distribution of each channel separately;

[0012] The first feature fusion module is used to fuse the colors of each channel;

[0013] A pixel detail adjustment module for expressing pixels and modulating signals using various wave functions;

[0014] The second feature fusion module is used for further fusion;

[0015] The second sub-channel color correction module is used to adjust the color distribution of each channel to make it normally distributed;

[0016] The second point convolution is used to compress the channels to output an enhanced image.

[0017] Furthermore, the first sub-channel color correction module and the second sub-channel color correction module are both constructed according to the following steps:

[0018] Split the feature map to obtain 3 color channel information F R 、F G 、F B and image content information F CAM_base ;

[0019] The three color channel information F R 、F G 、F B Expand to a higher dimensional space through 1×1 convolution, and use InstanceNorm normalization to adjust the color distribution.

[0020] Then, the three color channel information is compressed to the original spatial dimension through 1×1 convolution to obtain the enhanced three color channel information;

[0021] The enhanced three color channel information and image content information F CAM_base Add them together as enhanced image content information;

[0022] Finally, the enhanced three color channel information and the enhanced image content information are merged to form new features.

[0023] Furthermore, the first sub-channel color correction module is expressed as:

[0024] F′ color =Conv 1×1 (InstanceNorm(Conv 1×1 (F color ))), color∈{R,GB}

[0025] F C ' AM_base =F R '+F G '+F b '+F CAM_base

[0026] F CAM_out =concat([F R ',F G ',F b ',F′ CAM_base ])

[0027] The second sub-channel color correction module is similar to the first sub-channel color correction module.

[0028] Furthermore, in the pixel detail adjustment module, various wave functions are used to represent pixels and modulate signals, including:

[0029] Based on the optical principle of camera image acquisition, the pixels of underwater images are regarded as light signals affected by light scattering, and the pixels are adjusted by imitating the principle of Fourier decomposition of signals through sine and cosine, including:

[0030] The optical signal is decomposed into a combination of different signal wave characteristics, and the signal wave characteristics are expressed as:

[0031] F wave =Conv 1×1 (F PAM_in )·wave(Conv 1×1 (F PAM_in ))wave∈{sin,cos,tanh}

[0032] In the formula, F wave Indicates the signal wave characteristics, F PAM_in Represents the features input to the sub-pixel detail adjustment module;

[0033] Use convolution operations of different scales on the signal wave features to obtain contextual information;

[0034] Each signal wave feature and the feature F input to the sub-pixel detail adjustment module PAM_in Fusion, get the fused wave characteristics It is expressed as:

[0035] F PAM_base =Conv 1×1 (F PAM_in )

[0036] F w ' ave =Conv k×k (F wave )

[0037]

[0038] Where k is the size of the convolution kernel;

[0039] The feature representation of the output of the pixel detail adjustment module is:

[0040]

[0041] Furthermore, both the first feature fusion module and the second feature fusion module adopt a dense residual connection structure.

[0042] Furthermore, the first feature fusion module is constructed according to the following steps:

[0043] The fusion process is started by a 3×3 convolution, followed by a PReLU activation function;

[0044] Subsequently, a cascade of three 3×3 convolutions is used, in which the results of each convolution layer are combined with the results of all previous layers, and the combined features are then input to the next convolution stage; the output f of each layer in the cascade convolution is l It is expressed as:

[0045] f0=PReLU(Conv 3×3 (F Fusion_in ))

[0046] f l =PReLU(Conv 3×3 ([f l-1 ,f l-2,...,f0]))

[0047] In the formula, F Fusion_in Represents the features input to the first feature fusion module;

[0048] Establish a residual connection between the input and output of the cascaded convolution;

[0049] The output of the first feature fusion module is expressed as:

[0050] Fusion(f)=f3(f2(f1(f0)))+f0.

[0051] The construction of the second feature fusion module is similar to that of the first feature fusion module.

[0052] Furthermore, the HDNet model is trained by combining the multi-dimensional combined loss function with the auxiliary gradient guidance strategy based on random CLAHE;

[0053] The multi-dimensional combined loss function includes: a pixel content loss function, a structural similarity loss function and a perceptual loss function.

[0054] Furthermore, the multi-dimensional combined loss function, combined with the auxiliary gradient guidance strategy based on random CLAHE, is expressed as:

[0055]

[0056] Where, L pcont (·) represents the pixel content loss function, L ssim (·) represents the structural similarity loss function, L per (·) represents the perceptual loss function, I gt represents the reference image, Represents the image output by the HDNet model, I m represents the image output by the pixel detail adjustment module; α, β, γ and η are the corresponding weights, represents the reference image enhanced by CLAHE, represents the original given reference image, p represents the probability of the reference image enhanced using CLAHE, and r represents a random floating point number between 0 and 1 randomly generated during the training process.

[0057] The present invention proposes an underwater image enhancement system based on hierarchical decomposition and fusion, including a trained HDNet model, and uses the trained HDNet model to perform underwater image enhancement;

[0058] The HDNet model includes the following steps according to the image processing process:

[0059] The first point convolution layer is used to expand the channels of the input image;

[0060] The first channel-by-channel color correction module is used to split the channels of the input image and adjust the color distribution of each channel separately;

[0061] The first feature fusion module is used to fuse the colors of each channel;

[0062] A pixel detail adjustment module for expressing pixels and modulating signals using various wave functions;

[0063] The second feature fusion module is used for further fusion;

[0064] The second sub-channel color correction module is used to adjust the color distribution of each channel to make it normally distributed;

[0065] The second point convolution is used to compress the channels to output an enhanced image.

[0066] Furthermore, the HDNet model is trained through a multi-dimensional combined loss function combined with an auxiliary gradient guidance strategy based on random CLAHE. The multi-dimensional combined loss function includes: pixel content loss function, structural similarity loss function and perceptual loss function.

[0067] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0068] (1) In view of the inadequate underwater image enhancement capabilities of existing methods, this paper proposes an underwater image enhancement network HDNet based on hierarchical decomposition and fusion. It corrects color deviation through macro channel decomposition and improves detail degradation through micro pixel decomposition, and fully and meticulously considers two key factors of image degradation: light attenuation and scattering.

[0069] (2) In terms of pixel decomposition, the present invention innovatively regards the pixels of the degraded underwater image as light signals affected by scattering; by utilizing signal representation and modulation, the method can effectively adjust the local features of the pixels and enhance the image details;

[0070] (3) When constructing the combined loss function, the present invention enables the model to learn the enhanced effect of the reference image in multiple dimensions. In addition, it also combines an auxiliary gradient guidance strategy based on random CLAHE. This method can enhance the gradient information of the model about white balance and contrast during the training process, and to a certain extent can alleviate the adverse effects of low-quality data, so that the model can learn better. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic diagram of training the HDNet model of the present invention;

[0072] Figure 2 This is a schematic diagram of the structure of each module in the HDNet model proposed in the present invention; wherein, Figure 2 (a) is a schematic diagram of the structure of the channel color correction module. Figure 2 (b) is a schematic diagram of the sub-pixel detail adjustment module structure. Figure 2 (c) is a schematic diagram of the feature fusion module structure;

[0073] Figure 3 The following is a comparison chart of the enhancement effects of each model on the V90, U45 and UCCS datasets. DETAILED DESCRIPTION

[0074] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.

[0075] like Figure 1 As shown, this embodiment proposes an underwater image enhancement method based on hierarchical decomposition and fusion, which mainly includes the following steps:

[0076] Step 1: Obtain a real underwater image dataset, including a training set and a test set: The real underwater image datasets used in this embodiment include the UIEB dataset, the U45 dataset, and the UCCS dataset. The UIEB dataset consists of 890 pairs of original real underwater images and their corresponding high-quality reference images. The U45 dataset contains 45 original underwater images showing various types of degradation, such as color deviation, lack of contrast, and blur. The UCCS dataset contains 300 underwater images showing various color deviations. 800 image pairs are randomly selected from the UIEB dataset as a training set (T800), and the remaining 90 image pairs are used as a test set with reference images (V90). The U45 dataset and the UCCS dataset are used as test sets without reference images.

[0077] Step 2: Data preprocessing: The training set is first randomly scaled, cropped, and flipped, and then resized to 256×256.

[0078] Step 3: Construct an underwater image enhancement model based on hierarchical decomposition and fusion, namely the HDNet model: The HDNet model designed in this embodiment enhances images at different levels, and each level uses a decomposition method to solve the specific problem of underwater image degradation. It follows the processing flow from macro to micro and then to macro, combining modules into a systematic enhancement model. At the macro level, the HDNet model decomposes channels to correct color deviations. At the micro level, the HDNet model regards pixels as light signals, decomposes them and modulates the signals to reduce the degradation of image details. The two feature fusion operations in the middle are used to fuse the decomposed features and improve the enhancement effect of the model. The specific design operations include:

[0079] Model structure design: For degraded underwater images D∈R H×W×3 , the first layer of the HDNet model uses 1×1 point convolution to expand the channels of the input image. Then, the second layer uses the macro-level channel-by-channel color adjustment module (CAM), which splits the channels and adjusts the color distribution of each channel separately. The third layer uses a fusion module (Fusion) for fusion. The image then enters the micro-level pixel detail adjustment module (PAM) of the fourth layer, which represents pixels and modulates signals with various wave functions. After that, the fifth layer uses a fusion module (Fusion) for further fusion. Subsequently, the image enters the macro-level pixel detail adjustment module (PAM) of the sixth layer. Branch The color adjustment module (CAM) is used to ensure that the enhanced colors are normally distributed, that is, they are still in a normal distribution state. Finally, after the seventh layer of 1x1 convolution compression channel, the enhanced image is obtained. In addition, in order to speed up the convergence of model parameters, the feature map F is extracted from the fourth layer, i.e. the middle layer m ∈R H×W×C , and reduce it to I by 1×1 point convolution. m ∈R H×W×3 , to calculate the loss for the reference image. Figure 1 The left half of the figure shows the overall structure of the HDNet model.

[0080] The whole process can be summarized as follows:

[0081] F m =PAM(Fusion(CAM(Conv 1×1 (D))))

[0082] I m =Conv 1×1 (F m )

[0083]

[0084] The construction idea of ​​the channel-by-channel color adjustment module (CAM) is: from a macroscopic level, underwater images usually appear blue-green due to the different attenuation degrees of light waves of different colors in water. The module can effectively correct the color cast of the image by splitting the RGB channels of the image and adjusting the color distribution of each channel separately. Specifically: In an underwater environment, the attenuation characteristics of light vary with wavelength. Red light has the longest wavelength and decays the fastest, while blue and green light have shorter wavelengths and decay relatively slowly, which leads to abnormal distribution of red, green and blue. The channel-by-channel color adjustment module (CAM) can adjust the distribution of the three channels of blue, red and green in the image to normal. First, the expanded feature map F in_expand Split into 4 parts, FR 、F G 、F B Corresponding to 3 color channels, F CAM_base is the image content information, which serves as the basis for color adjustment. R 、F G 、F B The 1×1 convolution is used to expand the space to a higher dimension, and the InstanceNorm normalization method is used to adjust the color distribution. R 、F G 、F B Compress to the original spatial dimension. Then combine the enhanced information of the three channels with the image content F CAM_base The last 4 parts are channel-merged to form a new feature. Figure 2 Part (a) shows the specific structure of the channel color adjustment module (CAM). It can be expressed as follows:

[0085] F′ color =Conv 1×1 (InstanceNorm(Conv 1×1 (F color ))), color∈{R,GB}

[0086] F′ CAM_base =F R '+F G '+F b '+F CAM_base

[0087] F CAM_out =concat([F R ',F G ',F b ',F C ' AM_base ])

[0088] The construction idea of ​​the pixel detail adjustment module (PAM) is as follows: from a microscopic level, according to the principle that the camera collects light signals to form image pixels, the pixels of the underwater image can be regarded as light signals affected by light scattering. The module can effectively optimize the image details by characterizing the signals of the degraded pixels, decomposing them into linear combinations of different signal waves and modulating the signals. The specific operation is as follows: pixels are the basic units that constitute the image details, and they are obtained by sampling, quantizing and encoding the captured light signals by the camera. In an underwater environment, this process is affected by light scattering, resulting in reduced image contrast and clarity. Based on this, this embodiment represents the pixels of the underwater image as light signals disturbed by scattering, and improves the image details through pixel signal characterization and signal modulation. Imitating the principle of Fourier decomposition of signals through sine and cosine, the pixel detail adjustment module (PAM) of this embodiment decomposes the signal into a combination of multiple waveforms, including sine (sin), cosine (cos) and hyperbolic tangent (tanh). The first two are periodic functions, while the added hyperbolic tangent (tanh) is a non-periodic function that can enrich the expression of pixels. Wave function F wave The representation of requires phase and amplitude, which can be obtained through a series of convolution operations. It can be expressed mathematically as:

[0089] F wave =Conv 1×1 (F PAM_in )·wave(Conv 1×1 (F PAM_in ))

[0090] wave∈{sin,cos,tanh}

[0091] In underwater image processing, the adjustment of pixels usually needs to consider the context information of the area where the pixel is located. This method helps the module to optimize the details of the image more accurately, thereby improving the clarity and visual quality of the underwater image. Therefore, this embodiment uses convolution operations of different scales on the wave function to obtain context information. In order to preserve the structure and texture of the image, each wave feature F wave It is also necessary to fuse the information of the input features F PAM_in The wave characteristics after fusion The mathematical formula can be expressed as:

[0092] F PAM_base =Conv 1×1 (F PAM_in )

[0093] F w ' ave =Conv k×k (F wave )

[0094]

[0095] Among them, for F wave , including F cos , F sin , F tanh ; k is the size of the convolution kernel. In order to obtain multi-scale context information, k is set to 5, 7 and 9 in this embodiment. Finally, the signal representation of the image pixel can be expressed by a mathematical formula:

[0096]

[0097] Figure 2 Part (b) shows the specific structure of PAM.

[0098] Fusion module design: Although fusion operations are included in the channel color adjustment module (CAM) and the pixel detail adjustment module (PAM), the fusion granularity is still relatively coarse. Therefore, this embodiment uses a dense residual connection structure to fuse the decomposed features layer by layer and more finely to better fuse and enhance the effects of these two decomposition enhancements. Its dense connection can combine the features of each previous layer layer by layer to fuse the features more finely. The Fusion module starts the fusion process with a 3×3 convolution, followed by a PReLU activation function. Subsequently, a cascade of three 3×3 convolutions is used. In this cascade, the results of each convolution layer are combined with the results of all previous layers, and then the combined features are input into the next convolution stage. The output f of each layer in the cascaded convolution l It can be expressed by the following equation:

[0099] f0=PReLU(Conv 3×3 (F Fusion_in ))

[0100] f l =PReLU(Conv 3×3 ([f l-1 ,f l-2 ,...,f0]))

[0101] In the formula, F Fusion_in Represents the features input to the first feature fusion module;

[0102] In order to strengthen the learning process, a residual connection is established between the input and output of the cascade convolution. The fusion module can finally be expressed as:

[0103] Fusion(f)=f3(f2(f1(f0)))+f0

[0104] Figure 2Part (c) shows the specific structure of Fusion.

[0105] Step 4: Loss function design: In order to enhance the multi-dimensional learning reference image of the deep learning model, the present invention uses the pixel content loss function L pcont , that is, pixel-based L1 loss, structural similarity loss function L ssim , that is, SSIM loss, and perceptual loss function L per , which is the loss calculated based on the feature map of the classification model. These three losses can train the model to comprehensively learn the pixel details, structural features and overall content of the reference image. In order to speed up the convergence of the parameters in the first half of the model, this embodiment calculates these losses at the end of the model, and also calculates the perceptual loss function in the middle layer of the model. In this embodiment, the perceptual loss function uses the "relu1_2" and "relu2_2" layer features in VGG16 to calculate the mean square error loss.

[0106] In order to mitigate the negative impact of suboptimal reference images on the model learning process, this embodiment uses the three training losses and also adds a designed auxiliary gradient guidance strategy based on random CLAHE. The auxiliary gradient guidance strategy based on random CLAHE randomly applies the contrast-limited adaptive histogram equalization (CLAHE) algorithm to the reference image to enhance its contrast and balance the lighting. This strategy can enhance the gradient of the model in these aspects, thereby mitigating the adverse effects of suboptimal reference images. The complete loss function calculation formula is:

[0107]

[0108] Among them, α, β, γ and η are the weights of each loss; I gt is the reference image, is the model prediction image, I m It is the image output by the fourth layer of the model, that is, the middle layer. The perceptual loss between the middle layer and the reference image is calculated to speed up the convergence of the parameters of the first half of the model. is the reference image enhanced using CLAHE (Contrast Limited Histogram Equalization), is the original given reference image, p is the probability of using CLAHE to enhance the reference image, and r is a random floating point number between 0 and 1 randomly generated during the model training process. Figure 1 The right half of the figure shows the use of the auxiliary gradient guidance strategy based on random CLAHE during the model training process.

[0109] Step 5: Selection of evaluation indicators: To evaluate the enhancement effect of the model, the present invention uses PSNR, SSIM and UCIQE as evaluation indicators. PSNR is a pixel-based evaluation indicator with a reference image, with a value between 0 and +∞. SSIM can comprehensively evaluate the similarity of the image's brightness, structure and contrast with the reference image, with a value between 0 and 1. UCIQE is specifically for underwater images, and quantitatively evaluates image quality through a linear combination of chroma, saturation and contrast, with a value between 0 and 1. The higher the value, the better the enhancement effect.

[0110] Step 6: Model training: Use the training set T800 with uniform image size to train the HDNet model, and select appropriate hyperparameters based on factors such as hardware configuration, data scale, and data characteristics. Specifically:

[0111] Experimental environment: The model experiment in this embodiment is completed on a device with Ubuntu 20.04 system. The device is configured with Intel i5-12400F CPU, 12GB memory and RTX3060 GPU. The software environment is PyTorch 1.12.1, Python3.7 and CUDA11.3.

[0112] Parameter setting: Adam is selected as the optimizer, the initial learning rate is 1e-3, and the beta parameters are 0.9 and 0.999 respectively. The batch size is set to 16, the training is 400 cycles, and the loss weights α, β, γ and η are 1.0, 0.2, 0.5 and 0.2 respectively. The probability p in the auxiliary gradient guidance strategy based on random CLAHE is 0.4.

[0113] Dataset import and model training: Import the training set T800 processed in step 2 into the HDNet model and train the model to fit the provided reference image. When the model parameters converge, save the model parameters with the best enhancement effect based on the PSNR index of the model on the test set V90.

[0114] Step 7: Model effect evaluation: Use the trained HDNet model to enhance and visualize the original images of the V90 test set, U45 and UCCS test sets. For the V90 test set with reference images, use PSNR, SSIM, and UCIQE indicators for evaluation, and for the U45 and UCCS test sets without reference images, use UCIQE indicators for evaluation.

[0115] The underwater image enhancement model designed in this embodiment is based on hierarchical decomposition and fusion. It performs channel splitting and color distribution adjustment and performs detailed fusion at the macro level, and performs pixel signal decomposition and modulation and performs detailed fusion at the micro level, which respectively alleviate the color distortion problem caused by light attenuation and the detail blurring problem caused by light scattering. The combination function is used to allow the model to learn the enhancement effect of the reference image in multiple dimensions. In addition, the designed auxiliary gradient guidance strategy based on random CLAHE provides additional auxiliary gradients for model training, which can reduce the adverse effects of suboptimal reference images on the model learning process to a certain extent. This embodiment uses the UIEB, U45 and UCCS datasets to train and test the designed model. The quantitative comparison results with other methods are shown in Table 1, and the visual comparison results are shown in Table 1. Figure 3 As shown in Table 1 and Figure 3 It can be seen that the enhancement effect of the method of this embodiment is better than that of the comparison method.

[0116] Table 1

[0117] Network Model V90 / PSNR V90 / SSIM U45 / UCIQE UCCS / UCIQE Shallow-UWnet 18.062 0.796 0.518 0.483 UHD-SFNet 23.258 0.868 0.603 0.536 UIEC^2-Net 20.497 0.898 0.597 0.543 HDNet 24.158 0.923 0.610 0.564

Claims

1. An underwater image enhancement method based on hierarchical decomposition and fusion, characterized by: include: Build the HDNet model; Train the HDNet model; Use the trained HDNet model for underwater image enhancement; Among them, the constructed HDNet model includes the following steps according to the image processing process: The first point convolution layer is used to expand the channels of the input image; The first channel-by-channel color correction module is used to split the channels of the input image and adjust the color distribution of each channel separately; The first feature fusion module is used to fuse the colors of each channel; A pixel detail adjustment module for expressing pixels and modulating signals using various wave functions; The second feature fusion module is used for further fusion; The second sub-channel color correction module is used to adjust the color distribution of each channel to make it normally distributed; The second point convolution is used to compress the channels to output an enhanced image.

2. The underwater image enhancement method based on hierarchical decomposition and fusion according to claim 1, characterized in that: The first sub-channel color correction module and the second sub-channel color correction module are both constructed according to the following steps: Split the feature map to obtain 3 color channel information F R 、F G 、F B and image content information F CAM_base ; The three color channel information F R 、F G 、F B Expand to a higher dimensional space through 1×1 convolution, and use InstanceNorm normalization to adjust the color distribution. Then, the three color channel information is compressed to the original spatial dimension through 1×1 convolution to obtain the enhanced three color channel information; The enhanced three color channel information and image content information F CAM_base Add them together as enhanced image content information; Finally, the enhanced three color channel information and the enhanced image content information are merged to form new features.

3. The underwater image enhancement method based on hierarchical decomposition and fusion according to claim 2, characterized in that: The first sub-channel color correction module is expressed as: F′ color =Conv 1×1 (InstanceNorm(Conv 1×1 (F color ))),color∈{R,G.B} F′ CAM_base =F′ R +F′ G '+F b '+F CAM_base F CAM_out =concat([F′ R ,F′ G ,F′ b ,F′ CAM_base ]) The second sub-channel color correction module is similar to the first sub-channel color correction module.

4. The underwater image enhancement method based on hierarchical decomposition and fusion according to claim 1, characterized in that: In the sub-pixel detail adjustment module, various wave functions are used to represent pixels and modulate signals, including: Based on the optical principle of camera image acquisition, the pixels of underwater images are regarded as light signals affected by light scattering, and the pixels are adjusted by imitating the principle of Fourier decomposition of signals through sine and cosine, including: The optical signal is decomposed into a combination of different signal wave characteristics, and the signal wave characteristics are expressed as: F wave =Conv 1×1 (F PAM_in )·wave(Conv 1×1 (F PAM_in ))wave∈{sin,cos,tanh} In the formula, F wave Indicates the signal wave characteristics, F PAM_in Represents the features input to the sub-pixel detail adjustment module; Use convolution operations of different scales on the signal wave features to obtain contextual information; Each signal wave feature and the feature F input to the sub-pixel detail adjustment module PAM_in Fusion, get the fused wave characteristics It is expressed as: F PAM_base =Conv 1×1 (F PAM_in ) F w ' ave =Conv k×k (F wave ) F fusewave =F PAM_base +F w ' ave Where k is the size of the convolution kernel; The feature representation of the output of the pixel detail adjustment module is:

5. The underwater image enhancement method based on hierarchical decomposition and fusion according to claim 1, characterized in that: The first feature fusion module and the second feature fusion module both adopt a dense residual connection structure.

6. The underwater image enhancement method based on hierarchical decomposition and fusion according to claim 1, characterized in that: The first feature fusion module is constructed according to the following steps: The fusion process is started by a 3×3 convolution, followed by a PReLU activation function; Subsequently, a cascade of three 3×3 convolutions is used, in which the results of each convolution layer are combined with the results of all previous layers, and the combined features are then input to the next convolution stage; the output f of each layer in the cascade convolution is l It is expressed as: f0=PReLU(Conv 3×3 (F Fusion_in )) f l =PReLU(Conv 3×3 ([f l-1 ,f l-2 ,...,f0])) In the formula, F Fusion_in Represents the features input to the first feature fusion module; Establish a residual connection between the input and output of the cascaded convolution; The output of the first feature fusion module is expressed as: Fusion(f)=f3(f2(f1(f0)))+f0 The construction of the second feature fusion module is similar to that of the first feature fusion module.

7. The underwater image enhancement method based on hierarchical decomposition and fusion according to claim 1, characterized in that: The HDNet model is trained by combining a multi-dimensional combined loss function with an auxiliary gradient guidance strategy based on random CLAHE. The multi-dimensional combined loss function includes: a pixel content loss function, a structural similarity loss function and a perceptual loss function.

8. The underwater image enhancement method based on hierarchical decomposition and fusion according to claim 7, characterized in that: The multi-dimensional combined loss function, combined with the auxiliary gradient guidance strategy based on random CLAHE, is expressed as: Where, L pcont (·) represents the pixel content loss function, L ssim (·) represents the structural similarity loss function, L per (·) represents the perceptual loss function, I gt represents the reference image, Represents the image output by the HDNet model, I m represents the image output by the pixel detail adjustment module; α, β, γ and η are the corresponding weights, represents the reference image enhanced by CLAHE, represents the original given reference image, p represents the probability of the reference image enhanced using CLAHE, and r represents a random floating point number between 0 and 1 randomly generated during the training process.

9. An underwater image enhancement system based on hierarchical decomposition and fusion, characterized by: Including the trained HDNet model, using the trained HDNet model for underwater image enhancement; The HDNet model includes the following steps according to the image processing process: The first point convolution layer is used to expand the channels of the input image; The first channel-by-channel color correction module is used to split the channels of the input image and adjust the color distribution of each channel separately; The first feature fusion module is used to fuse the colors of each channel; A pixel detail adjustment module for expressing pixels and modulating signals using various wave functions; The second feature fusion module is used for further fusion; The second sub-channel color correction module is used to adjust the color distribution of each channel to make it normally distributed; The second point convolution is used to compress the channels to output an enhanced image.

10. The underwater image enhancement system based on hierarchical decomposition and fusion according to claim 9, characterized in that: The HDNet model is trained by a multi-dimensional combined loss function combined with an auxiliary gradient guidance strategy based on random CLAHE. The multi-dimensional combined loss function includes: pixel content loss function, structural similarity loss function and perceptual loss function.

Images