Underwater image enhancement method based on multi-color space fusion network

Abstract

The application provides an underwater image enhancement method based on a multi-color space fusion network. An RGB underwater image and a corresponding XYZ color space image are first subjected to an encoder to generate initial features; then, the feature maps are input into multiple MFFB blocks for multi-round iteration to enhance colors and details; and finally, an enhanced image is generated through a decoder.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of computer vision, underwater image enhancement, etc., and particularly relates to an underwater image enhancement method based on a multi-color space fusion network. BACKGROUND

[0002] The existing data-driven multi-color space guided underwater image enhancement method (UIE) combines the visual features of multi-color space for underwater image enhancement. However, such method adopts computationally expensive standard convolution to integrate these features, which leads to poor computational efficiency. At the same time, the existing full-reference UIE dataset contains a large number of low-quality reference images, which introduces color bias and structural blur, reducing the mapping accuracy of the data distribution between the original image and the reference image. SUMMARY

[0003] In order to overcome the defects and deficiencies existing in the prior art, the present application proposes an underwater image enhancement method based on a multi-color space fusion network, mainly involving a lightweight multi-color space fusion network (MCSF-Net). Specifically, the present application designs a multi-dimensional feature fusion block (MFFB) and a weighted feature fusion scheme to effectively integrate complementary features from RGB and XYZ color spaces. Grouped convolution is used to reduce the computational cost. At the same time, the present application develops a new texture color loss function to further improve image details and colors. In addition, the present application establishes a large-scale hybrid UIE dataset (LSMU), which filters out low-quality reference images of eight UIE datasets using nine no-reference indicators, thereby promoting more effective network learning.

[0004] Among them, the lightweight multi-color space fusion network (MCSF-Net) is composed of multiple multi-dimensional feature fusion blocks (MFFB) and a weighted feature fusion scheme to integrate complementary features and effectively enhance colors and details.

[0005] The present application specifically adopts the following technical solutions:

[0006] An underwater image enhancement method based on a multi-color space fusion network first generates initial features from RGB underwater images and corresponding XYZ color space images through an encoder; then, the feature maps are input into multiple MFFB blocks for multi-round iteration to enhance colors and details; finally, the enhanced images are generated through a decoder.

[0007] Further, the encoder is composed of three convolutional layers for constructing initial feature maps from RGB and XYZ images; the MFFB block has seven in total; and the decoder is composed of three convolutional layers for restoring the enhanced feature maps to underwater images.

[0008] Further, the first MFFB block and the seventh MFFB block constitute a skip connection; the second MFFB block and the sixth MFFB block constitute a skip connection; and the third MFFB block and the fifth MFFB block constitute a skip connection.

[0009] Further, in the MFFB block, first, a multi-dimensional calibration weight W mdcw is obtained for constraining the mean and variance by a MDCW module

[0010] First, the mean μ and variance σ of the space s, channel c and batch b are calculated 2 :

[0011]

[0012] where H and W are the height and width of the feature map;

[0013] Then, three multi-dimensional weight factors δ k are established, each of which is composed of three trainable parameters; for balancing the importance of different dimensions, as follows:

[0014]

[0015] where λ s , λ c , λ b are three trainable parameters;

[0016] Subsequently, the multi-dimensional constraint MDC(·) is used to constrain the mean μ and variance σ 2 range in each feature map:

[0017]

[0018] where α, β, ε are trainable scaling factor, trainable bias factor and a constant respectively; after the multi-dimensional constraint MDC(·), the initial multi-dimensional calibration weight W is obtained: mdcw The final multi-dimensional calibration weight W is obtained by fusing with the global scale factor {w t}:

[0019]

[0020] where t and z represent the index of all feature maps M; α t represents the trainable scaling factor in MDC(·); the global scale factor {w t} is used to adjust the global structure of .

[0021] Then, W is adjusted according to the threshold T. mdcw The image is divided into W1, which emphasizes global structure, and W2, which focuses on rich weighted colors. After the initial feature map is fused with W1 and W2, it is divided into four groups of feature maps greater than or equal to the threshold T (1_1, 1_2, 1_3, 1_4) and four groups of feature maps less than the threshold T (2_1, 2_2, 2_3, 2_4). Channel blending is used to obtain the image after color feature fusion, so as to simultaneously fuse colors and enhance details.

[0022] Furthermore, the MFFB block uses channel blending to obtain an image with fused color features, and then further enhances the details using a multi-scale feature pyramid construction method:

[0023] First, four multi-scale feature maps (3_1, 3_2, 3_3, and 3_4) are obtained using four grouped convolutions with different receptive fields; these feature maps are used to construct the initial multi-scale feature pyramid X. ini Then, a PWE module is used to obtain the multi-scale weights for multi-scale feature fusion:

[0024]

[0025] Among them, Adaptive Average Pooling Downsampling (AAP(·)) allows specifying the size of the output feature map; fully connected layers Used for global feature association; Bic(·) is bilateral upsampling, P is the initial multi-scale weights; then passed through a fully connected layer The combination with the Softmax(·) function yields the final multi-scale weights W. msw :

[0026]

[0027] The final multi-scale feature pyramid M msfp Through the initial multi-scale feature pyramid X ini With the final multi-scale weight W msw Harmony yields:

[0028]

[0029] in, This represents pixel-level multiplication.

[0030] Furthermore, using the mean square error loss function L mse Evaluation of the enhanced underwater image x and the reference underwater image y:

[0031]

[0032] Where H and W represent the height and width of the underwater image;

[0033] The texture-color loss function L tc The difference between the enhanced underwater image and the reference underwater image is measured with high-frequency and low-frequency maps and a grayscale map; for detail enhancement, high-frequency M h (t) and low-frequency map M l (t) are obtained by the following method:

[0034] M l (t) = AvgPool(t), M h (t) = (t - M l (t)), (9)

[0035] Where AvgPool(·) represents the average filter convolution operation; t is the input image;

[0036] For color restoration, the grayscale map M g (t) is obtained by the operation L tc of averaging the values of the three color channels:

[0037]

[0038] Where x represents the enhanced underwater image; y is the reference underwater image; the final loss function L f is a linear combination of L tc and L mse .

[0039] Further, the underwater image dataset is obtained by the following method:

[0040] Underwater images are collected from several full-reference datasets, and then high-quality underwater reference images are selected;

[0041] The selection of high-quality underwater reference images includes the following steps:

[0042] Integrate all reference underwater images from the full-reference dataset into a set;

[0043] Obtain several no-reference index scores for each reference underwater image;

[0044] Normalize the scores of the no-reference index of each reference underwater image respectively;

[0045] Calculate the average score of the no-reference index of each reference underwater image;

[0046] Keep the top 30% of the average ranked reference underwater images and the corresponding original underwater images.

[0047] And a multi-color space fusion network-based underwater image enhancement system, comprising an MCSF-Net model, the MCSF-Net model comprising an encoder, a plurality of MFFB blocks and a decoder;

[0048] The encoder is configured to generate initial features from an RGB underwater image and corresponding XYZ color space images;

[0049] The MFFB block is configured to iterate feature maps to enhance colors and details;

[0050] The decoder is configured to generate enhanced images.

[0051] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the multi-color space fusion network-based underwater image enhancement method as described above when executing the program.

[0052] A non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multi-color space fusion network-based underwater image enhancement method as described above.

[0053] Compared with the prior art, the present application and its preferred schemes propose a lightweight multi-color space fusion network for underwater image enhancement. Firstly, a multi-dimensional feature fusion block (MFFB) and a weighted feature fusion scheme are designed to effectively integrate complementary features from RGB and XYZ color spaces. Grouped convolution is used to reduce computational cost. At the same time, a new texture color loss function is proposed to further improve image details and colors. In addition, the present application establishes a large-scale mixed UIE dataset (LSMU), which filters out low-quality reference images from eight UIE datasets using nine no-reference indicators, thereby promoting more effective network learning. Extensive experiments on mainstream datasets show that the proposed method significantly outperforms existing methods in color restoration and detail enhancement. BRIEF DESCRIPTION OF DRAWINGS

[0054] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0055] Figure 1 The number of images between the original dataset and the dataset proposed by the embodiment of the present application is compared.

[0056] Figure 2 (a) is an example of an image in LSMU, and (b) is a data statistic of LSMU, UIEB and LSUI.

[0057] Figure 3 The quantitative comparison chart on the UIEB dataset is shown.

[0058] Figure 4 The overall framework diagram of the embodiment MCSF-Net of the present application.

[0059] Figure 5 The visualization results of W mdcw , W1 and W2. The higher the brightness, the greater the weight, and vice versa. The range of brightness value is [0, 1].

[0060] Figure 6 The visualization results of features Figure 1 _1 to 1_4 and 2_1 to 2_4. The higher the brightness, the greater the weight, and vice versa. The range of brightness value is [0, 1].

[0061] Figure 7 The comparison chart of color feature fusion before, without channel fusion and with channel fusion. The top row respectively represents the average and variance of the red, green and blue channels.

[0062] Figure 8 The visualization results of features Figure 3 _1 to 3_4, X ini , P, W msw and M msfp . The higher the brightness, the greater the weight, and vice versa. The range of brightness value is [0, 1].

[0063] Figure 9 The visualization comparison chart of different MFFB iteration numbers.

[0064] Figure 10 The ablation study result chart of underwater images with different color degradation.

[0065] Figure 11 The visual comparison chart of blue-biased underwater images in the test set.

[0066] Figure 12 The visual comparison chart of green-biased underwater images in the test set.

[0067] Figure 13 The visual comparison chart of low-visibility underwater images in the test set.

[0068] Figure 14 The visual comparison chart of underwater images with other color biases in the test set.

[0069] Figure 15 The application test chart of underwater depth estimation (first row), underwater edge detection (second row), underwater key point detection (third row), underwater saliency detection (fourth row), and underwater image segmentation (fifth row). DETAILED DESCRIPTION

[0070] In order to make the features and advantages of the present patent more apparent, the following detailed examples are described in conjunction with the accompanying drawings, in which:

[0071] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0072] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, they indicate the presence of a feature, step, operation, device, component and / or combination thereof.

[0073] The embodiment scheme of the present application filters out low-quality reference images from the existing eight UIE data sets through nine no-reference metrics, thereby establishing a large-scale mixed UIE data set (LSMU). Test results show that LSMU effectively promotes network learning.

[0074] And the designed MCSF-Net achieves excellent performance in color restoration, detail enhancement and computational efficiency. In addition, MCSF-Net still maintains excellent performance in some downstream underwater applications, including underwater depth estimation, edge detection, key point detection, saliency detection and target segmentation.

[0075] The scheme of the embodiment of the present application is specifically introduced as follows:

[0076] 1. Data collection of data set

[0077] The image acquisition of the UIE data set has the following three principles:

[0078] 1) Rich underwater scenes, different degrees of distortion and rich image content;

[0079] 2) The number of underwater images should be large;

[0080] 3) Provide corresponding high-quality underwater reference images to enable fair evaluation of image quality and end-to-end learning.

[0081] To achieve the first two goals, the embodiments of the present application collect underwater images from 8 full-reference datasets (UGAN, UIEB, NYU-U, EUVP, UFO-120, UVE38K, SUID and LSUI), which amount to 78,269 underwater images and 64,058 corresponding reference images. Since some of the reference images correspond to multiple underwater images in the collected data due to the inclusion of some synthetic underwater image datasets, the following screening scheme is proposed.

[0082] 2Reference image screening of the proposed dataset

[0083] The selection of high-quality underwater reference images includes the following steps:

[0084] Integrate all reference underwater images from the 8 full-reference datasets into one set;

[0085] Obtain 9 no-reference index scores for each reference underwater image; wherein the 9 no-reference indexes are from the following literatures respectively: ([1] M. Yang and A. Sowmya, “An underwater color image quality evaluation metric,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 6062-6071, 2015.

[0086] [2] K. Panetta, C. Gao, and S. Agaian, “Human-visual-system-inspired underwater image quality measures,” IEEE Journal of Oceanic Engineering, vol. 41, no. 3, pp. 541-551, 2016.

[0087] [3] N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, “Blind image quality evaluation using perception based features,” in Proceedings of 21st National Conference on Communications (NCC), pp. 1-6, 2015.

[0088] [4] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Transactions on Image Processing, vol. 21, no. 12, pp. 4695-4708, 2012.

[0089] [5] J. Zhou, Q. Liu, Q. Jiang, W. Ren, K.-M. Lam, and W. Zhang, “Underwater camera: Improving visual perception via adaptive dark pixel prior and color correction,” International Journal of Computer Vision, vol. 12, pp. 1-21, 082023.

[0090] [6] Y. Wang, N. Li, Z. Li, Z. Gu, H. Zheng, B. Zheng, and M. Sun, “An imaging-inspired no-reference underwater color image quality assessment metric,” Computers & Electrical Engineering, vol. 70, pp. 904-913, 2018.

[0091] [7] N. Yang, Q. Zhong, K. Li, R. Cong, Y. Zhao, and S. Kwong, “A reference-free underwater image quality assessment metric in frequency domain,” Signal Processing: Image Communication, vol. 94, p. 116218, 2021.

[0092] [8] K. Gu, W. Lin, G. Zhai, X. Yang, W. Zhang, and C. W. Chen, “No-reference quality metric of contrast-distorted images based on information maximization,” IEEE Transactions on Cybernetics, vol. 47, no. 12, pp. 4559-4565, 2017.

[0093] [9] L. K. Choi, J. You, and A. C. Bovik, “Referenceless prediction of perceptual fog density and perceptual image defogging,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3888-3901, 2015.

[0094] The scores of the 9 no-reference indicators of each reference underwater image are normalized;

[0095] The average scores of the 9 no-reference indicators of each reference underwater image are calculated;

[0096] The reference underwater images with the top 30% average scores and the corresponding original underwater images are reserved.

[0097] After using the above steps, the number of images between the proposed dataset and the original dataset changes as Figure 1 shown.

[0098] Although a large number of low-quality image pairs are deleted, the proposed LSMU dataset still covers a variety of underwater scenes, as Figure 2 shown, LSMU contains more underwater scene / object categories, water types and lighting conditions than the original UIEB and LSUI.

[0099] As Figure 3 shown, a series of comparative experiments are also carried out to prove the advantages of the LSMU dataset. First, take the PUIE-Net and MCSF-Net methods as examples. Then, the PUIE-Net and our MCSF-Net methods are trained on the 16 datasets in Figure 1 respectively. After 50 epochs of training, the PUIE-Net and MCSF-Net with the best PSNR and SSIM scores are reserved. As Figure 3The performance of each PUIE-Net and MCSF-Net is quantified by PSNR and SSIM on the UIEB dataset. As shown in FIG. 3, the PUIE-Net and MCSF-Net methods trained using the LSMU dataset produce higher PSNR and SSIM scores, which indicates that the enhanced images are closer to the reference images. In summary, the underwater image pairs of LSMU are less, which is conducive to improving the mapping accuracy of the data distribution between the original image and the reference image in the training stage.

[0100] 3. Overview of MCSF-Net

[0101] Figure 4 The overall framework of MCSF-Net. An underwater image (RGB image) and the corresponding XYZ color space image (XYZ image) are first generated by an encoder to generate initial features. Then, these feature maps are input into the MFFB block to enhance the color and details. Finally, the enhanced image is generated by the decoder.

[0102] The overall framework of MCSF-Net mainly consists of 1 encoder, 7 MFFB blocks and 1 decoder. The encoder consists of three convolutional layers for constructing initial feature maps from RGB and XYZ images. MFFB is used to fuse color features and enhance details. Iterative MFFB can achieve better performance, but it increases the difficulty of network training, therefore, the network introduces a skip connection: the first MFFB block and the seventh MFFB block form a skip connection; the second MFFB block and the sixth MFFB block form a skip connection; the third MFFB block and the fifth MFFB block form a skip connection. The decoder consists of three convolutional layers for restoring the enhanced feature map to an underwater image.

[0103] In the MCSF-Net framework, the MFFB block is the main design to achieve the effect of the present application, therefore, the structure and mechanism thereof are described in detail below.

[0104] 4. Color feature fusion

[0105] Different color spaces have different visual properties. For example, the RGB space provides rich color information, including three color channels of red, green and blue, and the intensity of each channel is [0, 255]. The XYZ space provides clearer underwater image details, including three channels of red-green (X), light-dark (Y) and blue-yellow (Z), and the value range of each channel is [0, 1]. The three channels of XYZ space are constructed based on human visual perception, and after converting the RGB space to the XYZ space, the distribution of color and brightness is more balanced, which is more conducive to producing clear image details.

[0106] To combine their characteristics, the MFFB block first uses an MDCW module to obtain the multidimensional calibration weights W for constraining the mean and variance. mdcw Then, their color features are blended through channel-level blending.

[0107] First, calculate the mean μ and variance σ of the multidimensional array (spatial s, channel c, batch b). 2 :

[0108]

[0109] Where H and W are the height and width of the feature map.

[0110] Then, this invention establishes three multidimensional weighting factors δ k Each parameter consists of three trainable parameters. These parameters are used to balance the importance of different dimensions, as shown below:

[0111]

[0112] Where, λ s , λ c , λ b These are three trainable parameters. Subsequently, this invention proposes a multidimensional constraint MDC(·) to constrain the mean μ and variance σ in each feature map. 2 scope:

[0113]

[0114] Here, α, β, and ε represent the trainable scaling factor, the trainable bias factor, and a constant close to zero, respectively. After performing multidimensional constrained MDC(·), the initial multidimensional calibration weights are obtained. The final multidimensional calibration weight W mdcw Depend on With global scaling factor {w t The result of fusion is:

[0115]

[0116] Where t and z represent the indices of all feature maps M. α t This represents the trainable scaling factor in MDC(·). Global scaling factor {w} t} is used for adjustment The global structure. For example... Figure 5 As shown, compared to W mdcw It has a clearer structure and richer weighted color features, but its weighted color features are still insufficient.

[0117] Therefore, further, the present invention will adjust W based on the threshold T.mdcw The initial feature maps are divided into W1 and W2, with the former emphasizing global structure and the latter focusing on rich weighted colors. After merging the initial feature maps with W1 and W2, these feature maps are divided into four groups of feature maps greater than or equal to the threshold T (1_1, 1_2, 1_3, 1_4) and four groups of feature maps less than the threshold T (2_1, 2_2, 2_3, 2_4). Figure 6 As shown, features Figure 1 _1 to 1_4 and 2_1 to 2_4 respectively contain different levels of structural information and weighted colors.

[0118] Finally, this invention employs channel blending to obtain an image with fused color features, thereby simultaneously fusing colors and enhancing details. For example... Figure 7 As shown, (a) exhibits obvious red artifacts and noise, and its mean and variance values ​​in the red channel are higher than those in other channels. (This is achieved through W...) mdcw After constraining the mean and variance, the enhanced image showed improved color, such as... Figure 7 As shown in (b). In contrast, Figure 7 (c) has a relatively reasonable color scheme, and the mean and variance values ​​of each channel are more uniform and closer. It should be noted that... Figure 7 (c) is the image of the result of one MFFB iteration. Because Figure 7 (c) The image lacks detail. This solution further enhances the detail by constructing a multi-scale feature pyramid.

[0119] 5 Enhanced Details

[0120] Following the previous processing step, to further improve the details of the underwater image, in the MFFB block, this invention first utilizes four CCLs composed of grouped convolutions with different receptive fields to obtain four multi-scale feature maps (3_1, 3_2, 3_3, 3_4). These feature maps are used to construct the initial multi-scale feature pyramid. For example... Figure 8 As shown, features Figure 3 _1 and 3_2 contain a fine global structure, while 3_3 and 3_4 emphasize the local details of the shipwreck.

[0121] from Figure 8 (e) It can be seen that, due to the lack of multi-scale feature fusion, X ini The overall structure and local details are rather blurry. Therefore, a PWE module is further introduced to obtain multi-scale weights for multi-scale feature fusion:

[0122]

[0123] Among them, Adaptive Average Pooling Downsampling (AAP(·)) allows specifying the size of the output feature map. Fully Connected Layer The features can be globally correlated. Bic(·) is used to obtain smoother image structure and finer details. P is the initial multi-scale weight. Through the combination of full connection layer and Softmax(·) function, we get the final multi-scale weight W msw :

[0124]

[0125] Where Softmax(·) is used to obtain the weight of the global importance of each feature. As Figure 8 (f) and Figure 8 (g) show, W msw provides more rich weighted details than P. The final multi-scale feature pyramid M msfp is obtained by fusing the initial multi-scale feature pyramid X ini with the final multi-scale weight W msw :

[0126]

[0127] Where, represents pixel-level multiplication. As Figure 8 (h) shows, W msw obtains better global structure and finer local details than X ini .

[0128] Based on the above operation, one iteration of MFFB is completed. Figure 9 Visual comparisons of MFFB under different iteration numbers are shown. As shown in the figure, Figure 9 the color restoration and detail enhancement effect of (a) is better than that of (b), which confirms that constructing a multi-scale feature pyramid can significantly improve the details and color restoration of underwater images. It is worth noting that as the preferred, seven iterations are enough to produce satisfactory underwater color and image details. Figure 7

[0129] 6Loss function

[0130] The present application uses the mean square error loss function L mse to evaluate the enhanced underwater image x and the reference underwater image y:

[0131]

[0132] Where H and W represent the height and width of the underwater image. Since the traditional L mse is not enough to produce fine details and rich colors, the embodiment designs a simple and effective texture-color loss function L tc ​The difference between the enhanced underwater image and the reference underwater image is measured with high-frequency and low-frequency maps and grayscale maps. Specifically, for detail enhancement, the high-frequency map M h (t) and the low-frequency map M l (t) are obtained by the following method:

[0133] M l (t) = AvgPool, M h (t) = (t - M l (t)), (9)

[0134] where AvgPool(·) represents the average filter convolution operation. t is the input image. For color restoration, the grayscale map M g (t) is obtained by averaging the values of the three color channels. Therefore, L tc can be represented as:

[0135]

[0136] where x represents the enhanced underwater image. y is the reference underwater image. The final loss function L f is a linear combination of L tc and L mse .

[0137] The following further demonstrates the effects of the embodiments of the present application through specific verification examples:

[0138] 1. Experimental data and evaluation method

[0139] The LSMU dataset designed by the present application is randomly divided into a training set and a test set using a ratio of 4:1. The training set contains 16214 pairs of image pairs. The test set has 4054 pairs of image pairs. The present application uses the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) as evaluation indicators for the test set.

[0140] 2. Ablation study

[0141] Table 1 shows the quantitative evaluation results of the ablation experiments with and without LEBG and PF-SubNet

[0142] Methods PSNR SSIM Our-settingI 20.6901 0.6145 Our-settingII 21.4139 0.6381 Our-settingIII 24.8150 0.7917 Our 26.8142 0.8914

[0143] To demonstrate the effectiveness of each component in MCSF-Net, an ablation study is conducted on four underwater images with blue, green, low visibility, and yellow degradation, including the following experiments: (1) MCSF-Net without XYZ space, MFFB, and TC-Loss (Our-setting I), (2) MCSF-Net without MFFB and TC-Loss (Our-setting II), (3) MCSF-Net without TC-Loss (Our-setting III), (4) the complete method of the present solution (Our). The visual comparison results are shown in Figure 10 The complete method obtains more realistic colors and richer image details, and suppresses the undesirable results of high local brightness. In addition, quantitative comparisons are made in Table 1. The complete method obtains the best PSNR and SSIM results, which indicates that the complete MCSF-Net has superior performance in restoring underwater content and structure.

[0144] 3 Qualitative and quantitative comparisons

[0145] Table 2 gives the quantitative evaluation results on the test set. In the test set, the present solution algorithm achieves the best PSNR and SSIM indicators compared to other methods. Figures 11 to 14 The visual comparison results of different methods are shown respectively. The present solution algorithm achieves the best visual quality.

[0146] Table 2 Quantitative evaluation results of the test set

[0147]

[0148]

[0149] 4 Application test

[0150] To test the practicality of the present solution, non-local priors are used for underwater depth estimation, Canny operators are used for underwater edge detection, SIFT key point detection is used for underwater key point detection, BASNet is used for underwater saliency detection, and a superpixel-based clustering algorithm is used for underwater image segmentation. The results are shown in Figure 15 Compared with other methods, the enhanced results of the present solution obtain more accurate depth maps, which indicates that the present solution algorithm has superiority in restoring more reliable depth. In addition, the present solution algorithm produces more edge numbers and key point numbers, and restores richer local details and clearer structure. Compared with other methods, the segmentation results of the present solution algorithm are more consistent and accurate, and the saliency detection of the present solution algorithm contains more salient objects and better boundaries. These results show that the present solution algorithm can more effectively improve the performance of underwater image segmentation and underwater saliency detection.

[0151] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0152] The present application is described in reference to the flowchart and / or block diagram illustrations of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks.

[0153] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure One one or more functions specified in the flowchart and / or block diagram block or blocks. Figure One Figure One one or more functions specified in the flowchart and / or block diagram block or blocks.

[0155] The patent is not limited to the above best mode, and anyone can derive other various forms of underwater image enhancement methods based on multi-color space fusion networks under the inspiration of the patent. Any equivalent changes and modifications made within the scope of the patent application should be covered by the patent.

Claims

1. An underwater image enhancement method based on a multi-color space fusion network, characterized in that: RGB underwater images and corresponding XYZ color space images are first generated initial features by an encoder; then, the feature maps are input into multiple MFFB blocks for multiple rounds of iteration to enhance colors and details; finally, the enhanced images are generated by a decoder; the MFFB block is a multi-dimensional feature fusion block; The encoder is composed of three convolutional layers for constructing initial feature maps from RGB and XYZ images; The MFFB block has a total of seven; The decoder is composed of three convolutional layers for restoring enhanced feature maps to underwater images; In the MFFB block, first, a multi-dimensional calibration weight W for constraining the mean and variance is obtained by an MDCW module mdcw Then, the color features are fused by channel-level blending: First, the mean μ and variance σ of the space s, channel c, and batch b are computed 2 : Wherein, H and W are the height and width of the feature map; Then, three multi-dimensional weight factors δ k are established, each consisting of three trainable parameters; for balancing the importance of different dimensions, as follows: where λ s , λ c , and λ b are three trainable parameters; Subsequently, the multi-dimensional constraint MDC(·) is used to constrain the mean μ and variance σ in each feature map 2 Range: Wherein, a, β, ε are trainable scaling factor, trainable bias factor and a constant respectively; after multi-dimensional constraint MDC(·), the initial multi-dimensional calibration weight is obtained Final multi-dimensional calibration weight W mdcw By And global scale factor {w t} fusion: where t and z represent the indices of all feature maps M; a t represents the trainable scaling factor in MDC(·); the global scale factor {w t} is used to adjust the global structure of Then, W is adjusted according to the threshold T. mdcw The initial feature map is divided into W1, which emphasizes global structure, and W2, which focuses on rich weighted colors. After the initial feature map is fused with W1 and W2, it is divided into four groups of feature maps that are greater than or equal to the threshold T and four groups of feature maps that are less than the threshold T. The color feature fused image is obtained by channel mixing fusion to simultaneously fuse colors and enhance details.

2. The multi-color space fusion network-based underwater image enhancement method according to claim 1, characterized in that: The first MFFB block and the seventh MFFB block constitute a skip connection; the second MFFB block and the sixth MFFB block constitute a skip connection; the third MFFB block and the fifth MFFB block constitute a skip connection.

3. The underwater image enhancement method based on the multi-color space fusion network according to claim 1, characterized in that: After the MFFB block obtains the color feature fused image by channel mixing fusion, the details are further strengthened in the form of a multi-scale feature pyramid structure: Firstly, four multi-scale feature maps are obtained by four CCLs composed of grouping convolutions with different receptive fields; these feature maps are used to construct an initial multi-scale feature pyramid X ini Then, multi-scale weights for multi-scale feature fusion are obtained by a PWE module: where the adaptive average pooling down-sampling AAP(·) allows to specify the size of the output feature map; the fully connected layer for globally associated features; Bic(·) is a bilinear up-sampling, P is the initial multi-scale weight; and the final multi-scale weight W is obtained by the combination of the fully connected layer and the Softmax(·) function msw : Final multi-scale feature pyramid M msfp Through initial multi-scale feature pyramid X ini With final multi-scale weights W msw Blending obtains: wherein, represents a pixel-wise multiplication.

4. The underwater image enhancement method based on the multi-color space fusion network according to claim 1, characterized in that: Using the mean squared error loss function L mse Evaluate the enhanced underwater image x and the reference underwater image y: Wherein, H and W represent the height and width of the underwater image; The difference between the enhanced underwater image and the reference underwater image is measured with a texture-color loss function L tc with high and low frequency maps and a gray map; for the details enhancement, the high frequency map M h (t) and the low frequency map M l (t) are obtained by the following methods: M l (t) = AvgPool(t), M h (t) = (t - M l (t)), (9) Wherein, AvgPool(·) represents an average filter convolution operation; t is the input image; For color restoration, a gray map M g (t) an operation L by averaging the values of the three color channels tc is represented as: where x denotes the enhanced underwater image; y is the reference underwater image; the final loss function L f is a linear combination of L tc and L mse .

5. The underwater image enhancement method based on the multi-color space fusion network according to claim 1, characterized in that: The underwater image dataset is obtained by the following method: Underwater images are collected from several full-reference datasets, and then high-quality underwater reference images are selected; The selection of high-quality underwater reference images includes the following steps: Integrate all reference underwater images from the full-reference dataset into a set; Obtain several no-reference index scores for each reference underwater image; Normalize the scores of the no-reference index of each reference underwater image respectively; Calculate the average score of the no-reference index of each reference underwater image; Keep the top 30% of the average ranked reference underwater images and the corresponding original underwater images.

6. An underwater image enhancement system based on multi-color space fusion network for implementing the method of claim 1, characterized in that: The MCSF-Net model includes an encoder, a plurality of MFFB blocks and a decoder; the MCSF-Net is a lightweight multi-color space fusion network; The encoder is used to generate initial features from RGB underwater images and corresponding XYZ color space images; The MFFB block is used to iterate the feature map to enhance colors and details; The decoder is used to generate enhanced images.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the underwater image enhancement method based on the multi-color space fusion network according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the underwater image enhancement method based on the multi-color space fusion network according to any one of claims 1-5.

Images