Underwater image enhancement method based on multi-color space fusion network

Abstract

The invention provides an underwater image enhancement method based on a multi-color space fusion network. The method comprises the following steps: firstly, generating initial features of an RGB underwater image and a corresponding XYZ color space image through an encoder; then, inputting the feature map into a plurality of MFFB blocks to carry out multi-round iteration so as to enhance colors and details; and finally, generating an enhanced image through a decoder.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, underwater image enhancement and the like, and in particular to an underwater image enhancement method based on a multi-color space fusion network. Background Art

[0002] Existing data-driven multi-color space guided underwater image enhancement methods (UIE) combine visual features from multiple color spaces for underwater image enhancement. However, such methods use computationally expensive standard convolutions to integrate these features, resulting in poor computational efficiency. At the same time, the existing full-reference UIE dataset contains a large number of low-quality reference images, which introduce color casts and structural blurs, reducing the mapping accuracy of data distribution between original images and reference images. Summary of the invention

[0003] In order to overcome the defects and shortcomings of the prior art, the present invention proposes an underwater image enhancement method based on a multi-color space fusion network, which mainly involves a lightweight multi-color space fusion network (MCSF-Net). Specifically, the present invention designs a multidimensional feature fusion block (MFFB) and a weighted feature fusion scheme to effectively integrate complementary features from RGB and XYZ color spaces. Grouped convolution is used to reduce computational costs. At the same time, the present invention develops a new texture color loss function to further improve image details and colors. In addition, the present invention establishes a large-scale mixed UIE dataset (LSMU), and uses 9 reference-free indicators to filter out low-quality reference images of 8 UIE datasets, thereby promoting more effective network learning.

[0004] Among them, a lightweight multi-color space fusion network (MCSF-Net) is composed of multiple multi-dimensional feature fusion blocks (MFFBs) and a weighted feature fusion scheme to integrate complementary features and effectively enhance color and details.

[0005] The present invention specifically adopts the following technical solutions:

[0006] An underwater image enhancement method based on a multi-color space fusion network is proposed. The RGB underwater image and the corresponding XYZ color space image are firstly generated with an encoder to generate initial features. Then, the feature maps are input into multiple MFFB blocks for multiple rounds of iterations to enhance the color and details. Finally, the enhanced image is generated with a decoder.

[0007] Furthermore, the encoder consists of three convolutional layers for constructing an initial feature map from RGB and XYZ images; the MFFB blocks have a total of seven; and the decoder consists of three convolutional layers for restoring the enhanced feature map to an underwater image.

[0008] Furthermore, the first MFFB block and the seventh MFFB block form a skip connection; the second MFFB block and the sixth MFFB block form a skip connection; and the third MFFB block and the fifth MFFB block form a skip connection.

[0009] Furthermore, in the MFFB block, a multidimensional calibration weight W for constraining the mean and variance is first obtained through an MDCW module. mdcw , and then fuse the color features through channel-level mixing fusion:

[0010] First, calculate the mean μ and variance σ of space s, channel c, and batch b 2 :

[0011]

[0012] Among them, H and W are the height and width of the feature map;

[0013] Then, three multidimensional weight factors δ are established k , each of which consists of three trainable parameters; used to balance the importance of different dimensions, as follows:

[0014]

[0015] Among them, λ s , c , b There are 3 trainable parameters;

[0016] Then, the multidimensional constraint MDC(·) is used to constrain the mean μ and variance σ in each feature map. 2 scope:

[0017]

[0018] Among them, α, β, and ε are respectively a trainable scaling factor, a trainable bias factor, and a constant; after performing multidimensional constraint MDC(·), the initial multidimensional calibration weights are obtained. The final multidimensional calibration weight W mdcw Depend on With the global scale factor {w t}Fuse to get:

[0019]

[0020] Among them, t and z represent the indexes of all feature maps M; α t represents the trainable scaling factor in MDC(·); the global scaling factor {w t} is used to adjust The global structure.

[0021] Then, W is set according to the threshold T. mdcw Divided into W emphasizing the global structure 1 and W, which focuses on rich weighted colors 2 ; Initial feature map and W 1 and W 2 After fusion, it is divided into four groups of feature maps (1_1, 1_2, 1_3, 1_4) greater than or equal to the threshold T and four groups of feature maps (2_1, 2_2, 2_3, 2_4) less than the threshold T; channel mixing fusion is used to obtain the image after color feature fusion to simultaneously fuse colors and enhance details.

[0022] Furthermore, the MFFB block uses channel mixing and fusion to obtain a color feature fused image and then further enhances the details by constructing a multi-scale feature pyramid:

[0023] First, four CCLs consisting of grouped convolutions with different receptive fields are used to obtain four multi-scale feature maps (3_1, 3_2, 3_3, 3_4); these feature maps are used to construct the initial multi-scale feature pyramid X ini , and then a PWE module is used to obtain the multi-scale weights of multi-scale feature fusion:

[0024]

[0025] Among them, adaptive average pooling downsampling AAP(·) allows to specify the size of the output feature map; the fully connected layer Used for global correlation features; Bic(·) is bilateral upsampling, P is the initial multi-scale weight; then through the fully connected layer The combination of the Softmax(·) function gives the final multi-scale weight W msw :

[0026]

[0027] The final multi-scale feature pyramid M msfp Through the initial multi-scale feature pyramid X ini and the final multi-scale weight W msw Fusion obtains:

[0028]

[0029] in, Represents pixel-level multiplication.

[0030] Furthermore, using the mean square error loss function L mse Evaluate the enhanced underwater image x and the reference underwater image y:

[0031]

[0032] Where H and W represent the height and width of the underwater image;

[0033] Through the texture-color loss function L tc , the difference between the enhanced underwater image and the reference underwater image is measured with high-frequency and low-frequency maps and grayscale maps; for detail enhancement, the high-frequency M h (t) and low frequency graph M l (t) obtained by:

[0034] M l (t)=AvgPool(t),M h (t)=(tM l (t)), (9)

[0035] Where AvgPool(·) represents the average filter convolution operation; t is the input image;

[0036] For color restoration, the grayscale map M g (t) By averaging the values ​​of the three color channels L tc It is expressed as:

[0037]

[0038] Where x represents the enhanced underwater image; y is the reference underwater image; the final loss function L f YesL tc and L mse A linear combination of .

[0039] Furthermore, the underwater image dataset is obtained by the following method:

[0040] Collect underwater images from several full reference datasets and then select high-quality underwater reference images;

[0041] The selection of the high-quality underwater reference image comprises the following steps:

[0042] Integrate all reference underwater images from the full reference dataset into one set;

[0043] Obtain several no-reference indicator scores for each reference underwater image;

[0044] Normalize the scores of the no-reference metrics for each reference underwater image separately;

[0045] Calculate the average score of the no-reference metric for each reference underwater image;

[0046] The top 30% of the average ranked reference underwater images and the corresponding original underwater images are retained.

[0047] And, an underwater image enhancement system based on a multi-color space fusion network, comprising: a MCSF-Net model, the MCSF-Net model comprising an encoder, a plurality of MFFB blocks and a decoder;

[0048] The encoder is used to generate initial features from the RGB underwater image and the corresponding XYZ color space image;

[0049] The MFFB block is used to iterate the feature map to enhance color and details;

[0050] The decoder is used to generate an enhanced image.

[0051] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the underwater image enhancement method based on a multi-color space fusion network as described above are implemented.

[0052] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the underwater image enhancement method based on a multi-color space fusion network as described above.

[0053] Compared with the prior art, the present invention and its preferred embodiment propose a lightweight multi-color space fusion network for underwater image enhancement. Firstly, a multidimensional feature fusion block (MFFB) and a weighted feature fusion scheme are designed to effectively integrate complementary features from RGB and XYZ color spaces. Grouped convolution is used to reduce the computational cost. At the same time, a new texture color loss function is proposed to further improve image details and colors. In addition, the present invention establishes a large-scale mixed UIE dataset (LSMU) and uses 9 reference-free indicators to filter out low-quality reference images of 8 UIE datasets, thereby promoting more effective network learning. Extensive experiments on mainstream datasets show that the proposed method significantly outperforms existing methods in both color restoration and detail enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0055] Figure 1 Comparison of the number of images between the original dataset and the dataset proposed in the embodiment of the present invention.

[0056] Figure 2 (a) is an image example in LSMU, and (b) is the data statistics of LSMU, UIEB and LSUI.

[0057] Figure 3 A quantitative comparison chart on the UIEB dataset.

[0058] Figure 4 It is the overall framework diagram of MCSF-Net according to an embodiment of the present invention.

[0059] Figure 5 W mdcw , W 1 and W 2 Visualization result. The higher the brightness, the greater the weight, and vice versa. The range of brightness value is [0,1].

[0060] Figure 6 Features Figure 1 Visualization results of _1 to 1_4 and 2_1 to 2_4. The higher the brightness, the greater the weight, and vice versa. The range of brightness value is [0,1].

[0061] Figure 7 Comparison diagram of color feature before fusion, without channel fusion, and with channel fusion. The top row shows the mean and variance of the red, green, and blue channels, respectively.

[0062] Figure 8 Features Figure 3 _1 to 3_4, X ini , P, W msw and M msfp The visualization result of . The higher the brightness, the greater the weight, and vice versa. The range of brightness value is [0,1].

[0063] Fig. 9 A visual comparison chart of different MFFB iteration numbers.

[0064] Fig.10 Figure 2 shows the ablation study results for underwater images with different color degradations.

[0065] Fig.11 Visual comparison of the bluish underwater images in the test set.

[0066] Fig.12 Visual comparison of the greenish underwater images in the test set.

[0067] Fig.13 Visual comparison of low-visibility underwater images in the test set.

[0068] Fig.14 Visual comparison of underwater images with other color casts in the test set.

[0069] Fig.15 These are application test images for underwater depth estimation (first row), underwater edge detection (second row), underwater key point detection (third row), underwater saliency detection (fourth row), and underwater image segmentation (fifth row). DETAILED DESCRIPTION

[0070] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail with the accompanying drawings as follows:

[0071] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0072] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0073] The embodiment of the present invention filters out low-quality reference images from the existing eight UIE datasets through nine non-reference metrics, thereby establishing a large-scale mixed UIE dataset (LSMU). Test results show that LSMU effectively promotes network learning.

[0074] The designed MCSF-Net has achieved excellent performance in color restoration, detail enhancement and computational efficiency. In addition, MCSF-Net still maintains excellent performance in some downstream underwater applications, including underwater depth estimation, edge detection, key point detection, saliency detection and object segmentation.

[0075] The following is a detailed introduction to the solution of the embodiment of the present invention:

[0076] 1. Data Collection of Datasets

[0077] The image collection of UIE dataset has the following three principles:

[0078] 1) Rich underwater scenes, varying degrees of distortion, and rich image content;

[0079] 2) The number of underwater images should be large;

[0080] 3) Corresponding high-quality underwater reference images should be provided to enable fair evaluation of image quality and end-to-end learning.

[0081] To achieve the first two goals, the embodiment of the present invention collects underwater images from 8 full reference datasets (UGAN, UIEB, NYU-U, EUVP, UFO-120, UVE38K, SUID and LSUI), totaling 78,269 underwater images and 64,058 corresponding reference images. Since some synthetic underwater image datasets are included, some reference images correspond to multiple underwater images in the collected data. In order to meet the requirements of high-quality underwater reference images, the following screening scheme is proposed.

[0082] 2 Reference image screening of the proposed dataset

[0083] The selection of high-quality underwater reference images involves the following steps:

[0084] Integrate all reference underwater images from 8 full-reference datasets into one set;

[0085] Obtain 9 no-reference metric scores for each reference underwater image; the 9 no-reference metrics are from the following literature: ([1] M. Yang and A. Sowmya, "An underwater color image quality evaluation metric," IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 6062–6071, 2015.

[0086] [2]K.Panetta, C.Gao, and S.Agaian, "Human-visual-system-inspiredunderwater image quality measures," IEEE Journal ofOceanic Engineering, vol.41, no.3, pp.541–551, 2016.

[0087] [3] N. Venkatanath, D. Praneeth, MCBh, SS Channappayya, and SS Medasani, "Blind image quality evaluation using perception based features," in Proceedings of 21st National Conference on Communications (NCC), pp. 1–6, 2015.

[0088] [4]A.Mittal,A.K.Moorthy,and A.C.Bovik,“No-reference image qualityassessment in the spatial domain,”IEEE Transactions on Image Processing,vol.21,no.12,pp.4695–4708,2012.

[0089] [5]J.Zhou,Q.Liu,Q.Jiang,W.Ren,K.-M.Lam,and W.Zhang,“Underwatercamera:Improving visual perception via adaptive dark pixel prior and colorcorrection,”International Journal ofComputerVision,vol.12,pp.1–21,082023.

[0090] [6]Y.Wang,N.Li,Z.Li,Z.Gu,H.Zheng,B.Zheng,and M.Sun,“An imaging-inspired no-reference underwater color image quality assessment metric,”Computers&Electrical Engineering,vol.70,pp.904–913,2018.

[0091] [7]N.Yang,Q.Zhong,K.Li,R.Cong,Y.Zhao,and S.Kwong,“Areference-freeunderwater image quality assessment metric in frequency domain,”SignalProcessing:Image Communication,vol.94,p.116218,2021.

[0092] [8] K.Gu, W.Lin, G.Zhai,

[0093] [9] LK Choi, J. You, and A. C. Bovik, “Referenceless prediction of perceptual fog density and perceptual image defogging,” IEEE Transactions on Image Processing, vol. 24, no. 11, pp. 3888–3901, 2015.)

[0094] Normalize the scores of the nine no-reference metrics for each reference underwater image;

[0095] Calculate the average score of the 9 no-reference metrics for each reference underwater image;

[0096] The top 30% of the average ranked reference underwater images and the corresponding original underwater images are retained.

[0097] After using the above steps, the number of images between the proposed dataset and the original dataset changes as follows: Figure 1 shown.

[0098] Although a large number of low-quality image pairs are removed, the proposed LSMU dataset still covers a wide variety of underwater scenes, such as Figure 2 As shown, LSMU contains more underwater scene / object categories, water body types, and lighting conditions than the original UIEB and LSUI.

[0099] like Figure 3 As shown in Figure 2, the present invention also conducts a series of comparative experiments to prove the advantages of the LSMU dataset. First, the PUIE-Net and MCSF-Net methods are taken as examples. Then, the LSMU dataset is used to compare the PUIE-Net and MCSF-Net methods. Figure 1 PUIE-Net and our MCSF-Net method are trained on 16 datasets in . After 50 epochs of training, the PUIE-Net and MCSF-Net with the best PSNR and SSIM scores are retained. Figure 3As shown in Figure 3, the performance of each PUIE-Net and MCSF-Net is quantified by PSNR and SSIM on the UIEB dataset. As shown in Figure 3, the PUIE-Net and MCSF-Net methods trained with the LSMU dataset produce higher PSNR and SSIM scores, which indicates that the enhanced images are closer to the reference images. In summary, the small number of underwater image pairs in LSMU is conducive to improving the mapping accuracy of data distribution between original images and reference images in the training stage.

[0100] 3. MCSF-Net Overview

[0101] Figure 4 The overall framework of MCSF-Net. An underwater image (RGB image) and the corresponding XYZ color space image (XYZ image) are first generated through the encoder to generate initial features. Then, these feature maps are input into the MFFB block to enhance the color and details. Finally, the enhanced image is generated through the decoder.

[0102] The overall framework of MCSF-Net mainly consists of 1 encoder, 7 MFFB blocks and 1 decoder. The encoder consists of three convolutional layers and is used to construct the initial feature map from RGB and XYZ images. MFFB is used to fuse color features and enhance details. Iterative MFFB can achieve better performance, but it increases the difficulty of network training. Therefore, jump connections are introduced into the network: the first MFFB block forms a jump connection with the seventh MFFB block; the second MFFB block forms a jump connection with the sixth MFFB block; the third MFFB block forms a jump connection with the fifth MFFB block. The decoder consists of three convolutional layers and is used to restore the enhanced feature map to the underwater image.

[0103] In the MCSF-Net framework, the MFFB block is the main design for achieving the effects of the present invention, so its structure and mechanism are introduced in detail below.

[0104] 4Color feature fusion

[0105] Different color spaces have different visual properties. For example, the RGB space provides rich color information, including three color channels of red, green, and blue, and the intensity of each channel is [0,255]. The XYZ space provides clearer underwater image details, including three channels of red and green (X), light and dark (Y), and blue and yellow (Z), and the value range of each channel is [0,1]. The three channels of the XYZ space are constructed based on human visual perception. After converting the RGB space to the XYZ space, the distribution of color and brightness is more balanced, which is more conducive to producing clear image details.

[0106] In order to combine their characteristics, in the MFFB block, a multidimensional calibration weight W for constraining the mean and variance is first obtained through an MDCW module. mdcw , and then their color features are fused through channel-level blending.

[0107] First, calculate the mean μ and variance σ of multiple dimensions (space s, channel c, batch b) 2 :

[0108]

[0109] Among them, H and W are the height and width of the feature map.

[0110] Then, the present invention establishes three multidimensional weight factors δ k , each of which consists of three trainable parameters. These parameters are used to balance the importance of different dimensions as follows:

[0111]

[0112] Among them, λ s , c , b are three trainable parameters. Subsequently, the present invention proposes a multidimensional constraint MDC(·) to constrain the mean μ and variance σ in each feature map. 2 scope:

[0113]

[0114] Among them, α, β, and ε are respectively a trainable scaling factor, a trainable bias factor, and a constant close to 0. After performing multidimensional constraint MDC(·), the initial multidimensional calibration weights are obtained. The final multidimensional calibration weight W mdcw Depend on With the global scale factor {w t}Fuse to get:

[0115]

[0116] Among them, t and z represent the indexes of all feature maps M. t represents the trainable scaling factor in MDC(·). The global scaling factor {w t} is used to adjust The global structure of Figure 5 As shown, compared to W mdcw It has clearer structure and richer weighted color features, but its weighted color features are not enough.

[0117] Therefore, further, the present invention sets W according to the threshold value Tmdcw Divided into W 1 and W 2 , the former emphasizes the global structure, and the latter focuses on rich weighted colors. The initial feature map and W 1 and W 2 After fusion, these feature maps are divided into four groups of feature maps (1_1, 1_2, 1_3, 1_4) greater than or equal to the threshold T and four groups of feature maps (2_1, 2_2, 2_3, 2_4) less than the threshold T. Figure 6 As shown, the features Figure 1 _1 to 1_4 and 2_1 to 2_4 contain different degrees of structural information and weighted colors, respectively.

[0118] Finally, the present invention uses channel blending to obtain an image after color feature fusion, so as to fuse colors and enhance details at the same time. Figure 7 As shown in Figure 2, (a) shows obvious red artifacts and noise, and its mean and variance values ​​in the red channel are higher than those in other channels. mdcw After constraining the mean and variance, the enhanced image has improved color, such as Figure 7 (b). In contrast, Figure 7 The color of (c) is relatively reasonable, and the mean and variance values ​​of each channel are more uniform and closer. It should be noted that Figure 7 (c) is the result image of one MFFB iteration. Figure 7 The image details of (c) are insufficient, and this scheme further enhances the details by constructing a multi-scale feature pyramid.

[0119] 5. Detail Enhancement

[0120] After the processing in the previous link, in order to further improve the details of the underwater image, in the MFFB block, the present invention first uses 4 CCLs composed of grouped convolutions with different receptive fields to obtain 4 multi-scale feature maps (3_1, 3_2, 3_3, 3_4). These feature maps are used to construct the initial multi-scale feature pyramid. Figure 8 As shown, the features Figure 3 _1 and 3_2 contain a fine global structure, while 3_3 and 3_4 emphasize local details of the wreck.

[0121] from Figure 8 (e) It can be seen that due to the lack of multi-scale feature fusion, X ini The overall structure and local details of are relatively vague. Therefore, a PWE module is further introduced to obtain the multi-scale weights of multi-scale feature fusion:

[0122]

[0123] Among them, adaptive average pooling downsampling AAP(·) allows to specify the size of the output feature map. Fully connected layer It can globally correlate features. Bilateral upsampling Bic(·) is used to obtain smoother image structures and finer details. P is the initial multi-scale weight. Through the fully connected layer By combining the Softmax(·) function, we get the final multi-scale weight W msw :

[0124]

[0125] Among them, Softmax(·) is used to obtain the weight of the global importance of each feature. Figure 8 (f) and Figure 8 As shown in (g), W msw Provides richer weighted details than P. The final multi-scale feature pyramid M msfp Through the initial multi-scale feature pyramid X ini and the final multi-scale weight W msw Blending:

[0126]

[0127] in, Represents pixel-level multiplication. Figure 8 As shown in (h), W msw X ini Better global structures and finer local details are obtained.

[0128] Based on the above operations, one iteration of MFFB is completed. Fig. 9 A visual comparison of MFFB at different iteration numbers is shown. As shown in the figure, Fig. 9 (a) The color restoration and detail enhancement effects are better than Figure 7 (b) It is confirmed that constructing a multi-scale feature pyramid can significantly improve underwater image detail and color restoration. It is worth noting that, as a preference, seven iterations are sufficient to produce satisfactory underwater color and image details.

[0129] 6 Loss Function

[0130] The present invention uses the mean square error loss function L mse Evaluate the enhanced underwater image x and the reference underwater image y:

[0131]

[0132] Where H and W represent the height and width of the underwater image. mseIt is not enough to produce fine details and rich colors. In this embodiment, a simple and effective texture-color loss function L is designed. tc , the difference between the enhanced underwater image and the reference underwater image is measured with high-frequency and low-frequency maps and grayscale maps. Specifically, for detail enhancement, the high-frequency M h (t) and low frequency graph M l (t) obtained by:

[0133] M l (t)=AvgPool,M h (t)=(tM l (t)), (9)

[0134] Where AvgPool(·) represents the average filter convolution operation. t is the input image. For color restoration, the grayscale map M g (t) is obtained by averaging the values ​​of the three color channels. Therefore, L tc It can be expressed as:

[0135]

[0136] Where x represents the enhanced underwater image and y represents the reference underwater image. The final loss function L f YesL tc and L mse A linear combination of .

[0137] The effects of the embodiments of the present invention are further demonstrated below through specific verification examples:

[0138] 1 Experimental data and evaluation methods

[0139] The LSMU dataset designed by the present invention is randomly divided into a training set and a test set using a ratio of 4:1. The training set contains 16214 pairs of images. The test set has 4054 pairs of images. The present invention uses peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) as evaluation indicators of the test set.

[0140] 2 Ablation studies

[0141] Table 1 Quantitative evaluation results of ablation experiments with / without LEBG and PF-SubNet

[0142] Methods PSNR SSIM Our-settingI 20.6901 0.6145 Our-settingII 21.4139 0.6381 Our-settingIII 24.8150 0.7917 Our algorithm 26.8142 0.8914

[0143] To demonstrate the effect of each component in MCSF-Net, an ablation study is conducted on four underwater images degraded in blue, green, low visibility, and yellow, including the following experiments: (1) MCSF-Net without XYZ space, MFFB, and TC-Loss (Our-settingI), (2) MCSF-Net without MFFB and TC-Loss (Our-settingII), (3) MCSF-Net without TC-Loss (Our-settingIII), and (4) the complete method of our scheme (Our). The visual comparison results are shown in Figure 2. Fig.10 As shown in Figure 1, the complete method obtains more realistic colors and richer image details, and suppresses the undesirable results with high local brightness. In addition, a quantitative comparison is made in Table 1. The complete method obtains the best PSNR and SSIM results, which shows that the complete MCSF-Net has superior performance in recovering underwater content and structure.

[0144] 3 Qualitative and quantitative comparison

[0145] Table 2 gives the quantitative evaluation results on the test set. In the test set, compared with other methods, the proposed algorithm achieves the best PSNR and SSIM indicators. Figures 11 to 14 The visual comparison results of different methods are shown respectively. The algorithm of the present invention achieves the best visual quality.

[0146] Table 2 Quantitative evaluation results of the test set

[0147]

[0148]

[0149] 4 Application Testing

[0150] In order to test the practicality of the present invention, non-local priors are used for underwater depth estimation, Canny operator is used for underwater edge detection, SIFT key point detection is used for underwater key point detection, BASNet is used for underwater saliency detection, and superpixel-based clustering algorithm is used for underwater image segmentation. The results are shown in Fig.15 As shown. Compared with other methods, the enhanced results of the present invention obtain more accurate depth maps, which shows that the algorithm of the present invention is superior in recovering more reliable depth. In addition, the algorithm of the present invention produces more edges and key points, and recovers richer local details and clearer structures. Compared with other methods, the segmentation results of the algorithm of the present invention are more consistent and accurate, and the saliency detection of the algorithm of the present invention contains more salient objects and better boundaries. These results show that the algorithm of the present invention can more effectively improve the performance of underwater image segmentation and underwater saliency detection.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other various forms of underwater image enhancement methods based on multi-color space fusion networks under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. An underwater image enhancement method based on a multi-color space fusion network, characterized by: The RGB underwater image and the corresponding XYZ color space image are first passed through an encoder to generate initial features; then, the feature maps are input into multiple MFFB blocks for multiple rounds of iterations to enhance color and details; finally, the enhanced image is generated through a decoder.

2. The underwater image enhancement method based on multi-color space fusion network according to claim 1 is characterized in that: The encoder consists of three convolutional layers to construct initial feature maps from RGB and XYZ images; There are seven MFFB blocks in total; The decoder consists of three convolutional layers to restore the enhanced feature maps to underwater images.

3. The underwater image enhancement method based on multi-color space fusion network according to claim 2 is characterized in that: The first MFFB block and the seventh MFFB block form a skip connection; the second MFFB block and the sixth MFFB block form a skip connection; the third MFFB block and the fifth MFFB block form a skip connection.

4. The underwater image enhancement method based on multi-color space fusion network according to claim 1 is characterized in that: In the MFFB block, a multidimensional calibration weight W for constraining the mean and variance is first obtained through an MDCW module. mdcw , and then fuse the color features through channel-level mixing fusion: First, calculate the mean μ and variance σ of space s, channel c, and batch b 2 : Among them, H and W are the height and width of the feature map; Then, three multidimensional weight factors δ are established k , each of which consists of three trainable parameters; used to balance the importance of different dimensions, as follows: Among them, λ s , c , b There are 3 trainable parameters; Then, the multidimensional constraint MDC(·) is used to constrain the mean μ and variance σ in each feature map. 2 scope: Among them, α, β, and ε are respectively a trainable scaling factor, a trainable bias factor, and a constant; after performing multidimensional constraint MDC(·), the initial multidimensional calibration weights are obtained. The final multidimensional calibration weight W mdcw Depend on With the global scale factor {w t }Fuse to get: Among them, t and z represent the indexes of all feature maps M; α t represents the trainable scaling factor in MDC(·); the global scaling factor {w t } is used to adjust The global structure. Then, W is set according to the threshold T. mdcw It is divided into W1 that emphasizes the global structure and W2 that focuses on rich weighted colors; after the initial feature map is fused with W1 and W2, it is divided into four groups of feature maps (1_1, 1_2, 1_3, 1_4) greater than or equal to the threshold T and four groups of feature maps (2_1, 2_2, 2_3, 2_4) less than the threshold T; channel mixing fusion is used to obtain the image after color feature fusion to simultaneously fuse colors and enhance details.

5. The underwater image enhancement method based on multi-color space fusion network according to claim 4 is characterized in that: The MFFB block uses channel mixing and fusion to obtain the color feature fused image, and then further enhances the details by constructing a multi-scale feature pyramid: First, four CCLs consisting of grouped convolutions with different receptive fields are used to obtain four multi-scale feature maps (3_1, 3_2, 3_3, 3_4); these feature maps are used to construct the initial multi-scale feature pyramid X ini , and then a PWE module is used to obtain the multi-scale weights of multi-scale feature fusion: Among them, adaptive average pooling downsampling AAP(·) allows to specify the size of the output feature map; the fully connected layer Used for global correlation features; Bic(·) is bilateral upsampling, P is the initial multi-scale weight; then through the fully connected layer The combination of the Softmax(·) function gives the final multi-scale weight W msw : The final multi-scale feature pyramid M msfp Through the initial multi-scale feature pyramid X ini and the final multi-scale weight W msw Fusion obtains: in, Represents pixel-level multiplication.

6. The underwater image enhancement method based on multi-color space fusion network according to claim 1, characterized in that: Using the mean square error loss function L mse Evaluate the enhanced underwater image x and the reference underwater image y: Where H and W represent the height and width of the underwater image; Through the texture-color loss function L tc , the difference between the enhanced underwater image and the reference underwater image is measured with high-frequency and low-frequency maps and grayscale maps; for detail enhancement, the high-frequency M h (t) and low frequency graph M l (t) obtained by: M l (t)=AvgPool(t),M h (t)=(t-M l (t)), (9) Where AvgPool(·) represents the average filter convolution operation; t is the input image; For color restoration, the grayscale map M g (t) By averaging the values ​​of the three color channels L tc It is expressed as: Where x represents the enhanced underwater image; y is the reference underwater image; the final loss function L f YesL tc and L mse A linear combination of .

7. The underwater image enhancement method based on multi-color space fusion network according to claim 1 is characterized in that: The underwater image dataset is obtained by the following method: Collect underwater images from several full reference datasets and then select high-quality underwater reference images; The selection of the high-quality underwater reference image comprises the following steps: Integrate all reference underwater images from the full reference dataset into one set; Obtain several no-reference indicator scores for each reference underwater image; Normalize the scores of the no-reference metrics for each reference underwater image separately; Calculate the average score of the no-reference metric for each reference underwater image; The top 30% of the average ranked reference underwater images and the corresponding original underwater images are retained.

8. An underwater image enhancement system based on a multi-color space fusion network, characterized by: Comprising a MCSF-Net model, wherein the MCSF-Net model comprises an encoder, a plurality of MFFB blocks and a decoder; The encoder is used to generate initial features from the RGB underwater image and the corresponding XYZ color space image; The MFFB block is used to iterate the feature map to enhance color and details; The decoder is used to generate an enhanced image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the underwater image enhancement method based on a multi-color space fusion network as described in any one of claims 1 to 8 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the underwater image enhancement method based on a multi-color space fusion network as described in any one of claims 1 to 8 are implemented.

Images