Joint supervision underwater image enhancement algorithm based on detail enhancement and multi-color space learning
Through a joint supervision algorithm of detail enhancement and multi-color space learning, combined with supervision and unsupervised learning, the color shift and atomization problems of underwater image enhancement are solved, the contrast and visual quality of the image are improved, and the underwater image is realized. Effectively enhance.
Patent Information
- Application Number
- CN202411013142.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-19
AI Technical Summary
Among the existing underwater image enhancement methods, the prior-based method has problems of overexposure and color imbalance, while deep learning-driven methods are difficult to effectively learn the texture detail characteristics of images, resulting in poor visual quality.
Using a joint supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning, underwater imaging is used through the structure and detail separation module, the T component extraction module, and the multi-color feature extraction and fusion module, combined with supervised and unsupervised learning methods. The model performs image enhancement to reduce color shift and atomization.
It effectively improves the contrast and visual quality of the image, reduces color shift and atomization, and improves the clarity of the image and the fidelity of the detail feature.
Smart Images

Figure CN118982468A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a joint supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning. Background Art
[0002] Underwater images refer to the images of oceans, lakes and other water bodies captured by underwater photography or submersibles. These images usually show clear and transparent water, as well as various forms of aquatic plants and animals, rocks and coral reefs. Underwater images can be used in scientific research, environmental monitoring, tourism promotion and other fields, and can also be displayed as works of art.
[0003] However, due to the special underwater imaging environment, underwater images often have serious color cast and fogging, and underwater image enhancement can effectively improve the visibility and contrast of underwater images, thereby better showing the beautiful scenery of the underwater world. Therefore, underwater image enhancement is an important visual task, which is of great significance to underwater operations, underwater detection and other tasks.
[0004] At present, underwater image enhancement can be divided into three methods: prior-based methods, physical model-based methods, and deep learning-driven methods. Among them, the prior-based method directly processes the pixel values of the input image, or uses the histograms of different color spaces to improve the contrast of the image; the physical model-based method can decompose the multiple components of the input underwater image according to the underwater optical imaging model, and then solve the clear enhancement result. Finally, the deep learning-driven method enhances the underwater image through end-to-end learning mapping from underwater degraded images to underwater clear images.
[0005] However, methods based on priors and physical models often have problems with overexposure or color imbalance in some areas, while methods based on deep learning are often unable to learn detailed features such as image texture well. Summary of the invention
[0006] To solve the above problems, the present invention provides a joint supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning, which can effectively improve the contrast of the image, reduce color cast and fogging, and thus improve the visual quality of the image.
[0007] According to one aspect of the present invention, a joint supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning is provided, comprising the following steps:
[0008] 1) Through the structure and detail separation module, the detail component and background light component are extracted from the underwater degraded image;
[0009] 2) Extracting the perspective component from the underwater degraded image through the T component extraction module;
[0010] 3) Input the underwater degraded image into three color spaces respectively, use a multi-color feature extraction module to extract features in each color space, and fuse the extracted features through a multi-color feature fusion module;
[0011] 4) After the detail components are enhanced, the pixels are added together with the features fused by the multi-color space fusion module to obtain a clear underwater image;
[0012] 5) The features fused by the color space fusion module are input into the background light component, and are input into an underwater imaging model together with the perspective component and the underwater clear image to generate the underwater degraded image again.
[0013] In some embodiments, in step 1), in the structure and detail separation module, Gaussian blur is first performed on the image under water degradation to obtain most of the structural information of the image under water degradation, and then the background light component is decomposed from the structural information. The benefit is that a specific method for obtaining the background light component is further described.
[0014] In some embodiments, in step 1), the background light components include:
[0015] A λ (x) = B(I λ (x)),λ∈{R,G,B}
[0016] Among them, I λ (x) is the input underwater degraded image, B(.) is the Gaussian blur operation, A λ (x) is the background light component obtained by decomposition. The benefit thereof is that it describes the calculation formula of the background light component.
[0017] In some implementations, in step 2), the specific structure of the perspective component extraction module is as follows:
[0018]
[0019] Among them, I λ (x) is the input underwater degraded image, T λ (x) is the transmission diagram, is the network for estimating the T component, and i is the layers of the network. It is useful to describe the calculation formula of the perspective component.
[0020] In some embodiments, in step 3), each color space is HSV, LAB, and RGB. The benefit is that the type of each color space is described.
[0021] In some implementations, in step 3), the formula for feature extraction in each color space is as follows:
[0022] f HSV (x) = F HSV (I HSV (x))
[0023] f Lab (x) = F Lab (I Lab (x))
[0024] f RGB (x) = F RGB (I(x))
[0025] Among them, f HSV (x), f Lab (x) and f Lab (x) is the extracted color feature, I HSV (x) and I Lab (x) are the degraded underwater images in HSV and LAB color spaces, respectively. HSV , F Lab and F RGB They are feature extraction networks for three color spaces: HSV, LAB, and RGB. The benefit of this is that it describes the formulas for feature extraction in each color space.
[0026] In some embodiments, in step 3), the type of feature extraction network is ResNet. This is beneficial in that it describes the type of the selected feature extraction network.
[0027] In some embodiments, in step 1), the underwater degraded image is:
[0028] I λ (x) = J λ (x)T λ (x)+A λ (x)(1-T λ (x)),λ∈{R,G,B}
[0029] Among them, I λ (x) is the input underwater degraded image, T λ (x) is the transmission diagram, A λ (x) is the background light, J λ (x) Clear underwater image. The usefulness of this is that it describes the formula for generating degraded underwater images
[0030] In some embodiments, a combined supervised learning method that combines supervised learning and unsupervised learning is used to design an underwater image enhancement algorithm in conjunction with an underwater imaging model. The benefit of this is that the learning method used is described.
[0031] In some embodiments, L1 loss and VGG loss are used as supervised losses, and color loss and reconstruction loss are used as unsupervised losses. It is beneficial to describe the losses used in the learning method. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 This is an overall framework diagram of a jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings.
[0034] The unsupervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning (UUIE-DEMCSL) in the present invention is used to enhance the underwater degraded image. According to the underwater optical imaging model, the underwater degraded image can be expressed as:
[0035] I λ (x) = J λ (x)T λ (x)+A λ (x)(1-T λ (x)),λ∈{R,G,B}
[0036] Among them, I λ (x) is the input underwater degraded image, T λ (x) is the transmission diagram, A λ (x) represents background light, J λ (x) is the clear underwater image that can be obtained by solving.
[0037] like Figure 1 As shown, the algorithm includes several main steps, which are described below.
[0038] In the first step, the detail component and background light component (called A component) are extracted from the underwater degraded image through the structure and detail separation module.
[0039] In the structure and detail separation module, we first perform the underwater degraded image I λ (x) Gaussian blur can obtain most of the structural information of the underwater degraded image, and then the background light can be decomposed from the structural information.
[0040] A λ (x) = B(I λ (x)),λ∈{R,G,B}
[0041] Among them, I λ (x) is the input underwater degraded image, B(.) represents the Gaussian blur operation, A λ (x) is the background light component obtained by decomposition.
[0042] In the second step, the perspective component (called T component) is extracted from the underwater degraded image through the T component extraction module.
[0043] The specific structure of the T component extraction module is as follows:
[0044]
[0045] Among them, I λ (x) is the input underwater degraded image, T λ (x) is the transmission diagram, represents the network T-Net for estimating the T component, and i represents each layer of T-Net.
[0046] Taking the first layer as an example, the specific operations are as follows:
[0047]
[0048] Among them, σ is a linear activation function and C is a convolution with a convolution kernel size of 3×3.
[0049] In the third step, the underwater degraded images are input into multiple color spaces respectively, and a multi-color feature extraction module is used to extract features in each color space, and the extracted features are fused through a multi-color feature fusion module.
[0050] Multiple color spaces mainly include HSV, LAB, RGB, etc., and the formulas for feature extraction in each color space are as follows:
[0051] f HSV (x) = F HSV (I HSV (x)
[0052] f Lab (x) = F Lab (I Lab (x)
[0053] f RGB (x) = F RGB (I(x))
[0054] Among them, f HSV (x), f Lab (x) and f Lab (x) is the extracted color feature, I HSV (x) and I Lab (x) are the degraded underwater images in HSV and LAB color spaces, respectively. HSV , F Lab and F RGB They are feature extraction networks for three color spaces: HSV, LAB, and RGB, and the present invention selects ResNet as the feature extraction network.
[0055] In order to fully utilize the advantages of the three color spaces, the present invention also designs a multi-color space fusion module, which adds the features from the three color spaces through channels and then inputs them into the multi-color space fusion module. The multi-color space fusion module can adaptively learn the excellent features of each of the three color spaces and fuse these features.
[0056] In the fourth step, after the detail component is enhanced, it is pixel-added with the features fused by the multi-color space fusion module to obtain a clear underwater image, namely the J component.
[0057] In the fifth step, the underwater degraded image can be generated again as needed.
[0058] In this step, the features fused by the multi-color space fusion module are input into the background light component (A component), and are input into an underwater imaging model together with the perspective component (T component) and the underwater clear image (J component), so as to generate an underwater degraded image, namely, the I component.
[0059] In addition, the existing deep learning-based underwater image enhancement algorithm learning methods can be roughly divided into supervised learning and unsupervised learning. Among them, the supervised learning method can learn the mapping from low-quality images to clear images end-to-end, while the unsupervised learning method restores the image based on the existing image prior and underwater optical imaging model.
[0060] In order to combine the advantages of the two learning methods, this paper proposes a joint supervised learning method, which combines the supervised learning method with the unsupervised learning method, and jointly designs the underwater image enhancement algorithm in the present invention in conjunction with the underwater imaging model.
[0061] Among them, L1 loss and VGG loss are used as supervised losses. L1 loss can reduce the difference between pixel values, while VGG loss can shorten the distance between the generated image and the true image in the high-dimensional feature space. At the same time, color loss and reconstruction loss are also used as unsupervised losses.
[0062] The UUIE-DEMCSL algorithm in the present invention is constructed using the Pytorch library, using AdamW as the optimizer, and completing the network fitting through 180 batches. The initial learning rate is set to 0.0001, and it decreases by 50% every 30 batches. During the network learning process, the batch size is set to 4. All images in training are cropped to a size of 256*256 and normalized to [0, 1].
[0063] In addition, the UUIE-DEMCSL algorithm in the present invention uses the UIEB dataset and the SQUID dataset for experiments. Among them, since the number of UIEB datasets is small and does not meet the training requirements, the present invention completes the data augmentation of the UIEB dataset by horizontally and vertically flipping the images in the dataset and rotating them 90 degrees, 180 degrees and 270 degrees, using the augmented 12680 pairs of images as the training set, and selecting 80 pairs of images from the remaining images as the test set. For the SQUID dataset, 80 pairs of images are also selected as the cross test set.
[0064] For the full reference image quality evaluation, the present invention uses two indicators, PSNR and SSIM. Among them, PSNR represents the ratio of the image information to the noise. The higher the PSNR, the less image noise and the closer it is to the true image. SSIM represents the structural similarity between the generated image and the true image. The higher its value, the clearer the image and the better the color and texture.
[0065] As for the no-reference image quality evaluation, the present invention uses two commonly used underwater image quality evaluation indicators, UIQM and UCIQE, for evaluation. Among them, the UCIQE indicator is measured by the CIELab color space, which can evaluate the degree of degradation such as color distortion and atomization of underwater images; while the UIQM indicator is evaluated by the HSV color model and takes into account the three attributes of color, sharpness and contrast. The higher the UCIQE and UIQM values, the better the visual quality of the underwater image.
[0066] In order to verify the effectiveness of the proposed algorithm, this paper selects 16 underwater image enhancement algorithms for comparison, including 6 traditional algorithms: GDCP, HLRP, NUDCP, ACDC, ERH and Fusion, and 10 deep learning-based methods: UWCNN, URSCT, L2UWE, FUnIE-GAN, LCNet, CLUIE-Net, TOPAL, Water-Net, UICoE-Net and Ucolor.
[0067] By comparison, it can be seen that the traditional underwater image enhancement algorithm can improve color degradation, and the enhanced results have no obvious green or blue color cast. However, these processed images often have serious noise and are relatively single in color, and the tones are black and white compared to the undegraded images taken on land.
[0068] Among them, the GDCP and NUDCP algorithms have some improvements in brightness, but they tend to produce uneven fogging and color distortion; the ACDC algorithm produces grayish-white images with poor visual effects; the HLRP algorithm tends to overexpose the image, while the ERH and Fusion algorithms produce darker results.
[0069] In terms of learning-based methods, most methods show better colors and well-processed details. However, URSCT and L2UWE have difficulty handling severe color distortion; the enhanced images of FUnIE-GAN and LCNet introduce red artifacts, and most of the processed results of LCNet show severe red color shift; CLUIE-Net, TOPAL, and UICoE-Net still show some color degradation in some results.
[0070] In contrast, the UUIE-DEMCSL algorithm proposed in the present invention has achieved good results on the UIEB-80 and SQUID-80 test sets, can effectively solve problems such as color distortion and atomization, and the enhanced image has excellent visual effects.
[0071] Specifically, as shown in Tables 1 and 2, in Table 1, the four indicators of PSNR, SSIM, UICQE and UIQM are used to evaluate the enhancement results of the above algorithms in the UIEB-80 dataset; and in Table 2, the two indicators of UICQE and UIQM are used to evaluate the enhancement results of some algorithms with excellent performance in the SQUID-80 dataset.
[0072] Table 1 Comparison of PNSR, SSIM, UCIQE and UIQM indicators in UIEB-80 dataset
[0073]
[0074]
[0075] Table 2 Comparison of UCIQE and UIQM indicators in the SQUID-80 dataset
[0076] method UCIQE UIQM Input Image 0.5267 1.0342 FUnIE-GAN +0.0092 +0.0206 LCNet -0.0013 -0.0344 CLUIE-Net -0.0168 -0.0689 TOPAL -0.0136 -0.1032 Water-Net +0.0354 -0.0631 UICoE-Net +0.1245 -0.0364 Ucolor -0.0232 -0.1404 This algorithm +0.0578 +0.0937
[0077] The above are only some embodiments of the present invention. For those skilled in the art, several modifications and improvements can be made without departing from the creative concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. Jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning, Features: The following steps are included 1) Through the structure and detail separation module, the detail component and background light component are extracted from the underwater degraded image; 2) Extracting the perspective component from the underwater degraded image through the T component extraction module; 3) Input the underwater degraded image into three color spaces respectively, use a multi-color feature extraction module to extract features in each color space, and fuse the extracted features through a multi-color feature fusion module; 4) After the detail components are enhanced, the pixels are added together with the features fused by the multi-color space fusion module to obtain a clear underwater image; 5) The features fused by the color space fusion module are input into the background light component, and are input into an underwater imaging model together with the perspective component and the underwater clear image to generate the underwater degraded image again.
2. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 1, Features: In step 1), in the structure and detail separation module, Gaussian blur is performed on the image under water degradation to obtain most of the structural information of the image under water degradation, and then the background light component is decomposed from the structural information.
3. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 2, Features: In step 1), the background light component is A λ (x)=B(I λ (x)),λ∈{R,G,B} Among them, I λ (x) is the input underwater degraded image, B(.) is the Gaussian blur operation, A λ (x) is the background light component obtained by decomposition.
4. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 1, Features: In step 2), the specific structure of the perspective component extraction module is as follows: Among them, I λ (x) is the input underwater degraded image, T λ (x) is the transmission diagram, is the network for estimating the T component, and i is the layers of the network.
5. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 1, Features: In step 3), the color spaces are HSV, LAB, and RGB.
6. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 5, Features: In step 3), the formula for feature extraction in each color space is as follows: f HSV (x)=F HSV (I HSV (x)) f Lab (x)=F Lab (I Lab (x)) f RGB (x)=F RGB (I(x)) Among them, f HSV (x), f Lab (x) and f Lab (x) is the extracted color feature, I HSV (x) and I Lab (x) are the degraded underwater images in HSV and LAB color spaces, respectively. HSV , F Lab and F RGB They are feature extraction networks for three color spaces: HSV, LAB, and RGB.
7. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 6, Features: In step 3), the type of feature extraction network is ResNet.
8. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 1, Features: In step 5), the underwater degraded image is I λ (x)=J λ (x)T λ (x)+A λ (x)(1-T λ (x)),λ∈{R,G,B} Among them, I λ (x) is the input underwater degraded image, T λ (x) is the transmission diagram, A λ (x) is the background light, J λ (x) Clear underwater images.
9. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 1, Features: A joint supervised learning method that combines supervised learning and unsupervised learning is adopted, and an underwater image enhancement algorithm is jointly designed in conjunction with an underwater imaging model.
10. The jointly supervised underwater image enhancement algorithm based on detail enhancement and multi-color space learning according to claim 1, Features: L1 loss and VGG loss are adopted as supervised losses, and color loss and reconstruction loss are adopted as unsupervised losses.
Citation Information
Patent Citations
Multi-scale fusion underwater image enhancement method based on HSV color space
CN114612354A
Underwater image enhancement network and method based on physical and deep learning dual guidance
CN115272115A
Lightweight island multi-temporal remote sensing image change detection method based on multi-scale characteristic distillation
CN116645553A
Underwater image progressive generative adversarial enhancement module based on channel-space attention
CN117788330A
Three-color space underwater image enhancement method based on deep learning
CN117952853A
Cited By
Optical remote sensing image authenticity identification method and system based on multi-color space fusion
CN120913040A
