Full-reference underwater enhanced image quality assessment method based on paired samples

Abstract

The application discloses a kind of full reference underwater enhanced image quality evaluation methods based on paired samples, comprising: constructing paired underwater image dataset;The paired underwater image is made of reference image and distorted underwater image;Four kinds of feature maps of reference image and enhanced image are obtained;The enhanced image is generated by underwater image enhancement algorithm;The color distortion existing in the enhanced image is evaluated using attenuation map and saturation map, and the semantic similarity index is obtained;The artifact or blur existing in the enhanced image is evaluated using dark channel prior map and MSCN coefficient map, and the structural difference index is obtained;Fusion semantic similarity index and structural difference index, and the quality score of enhanced image is obtained;The application can effectively evaluate the similarity between underwater enhanced image and ground true reference image in semantics and structure, so as to evaluate different underwater image enhancement algorithms, and the method has high correlation with human subjective evaluation.

Description

Technical Field

[0001] This invention relates to the field of underwater image quality assessment, and more particularly to a method for assessing the quality of fully referenced underwater enhanced images based on paired samples. Background Technology

[0002] To improve the quality of underwater images, underwater image enhancement techniques have been extensively studied. The different focuses of various underwater image enhancement algorithms result in varying effects, making the objective evaluation of their performance a crucial issue driving the development of this technology. Existing image quality assessment methods are divided into subjective and objective methods. Subjective methods are time-consuming, labor-intensive, and cannot be automated. Objective methods are generally categorized into no-reference and full-reference methods; however, no-reference methods have low accuracy and do not accurately reflect human subjective perception. In recent years, datasets containing paired underwater image data have been released, leading to the rapid development of learning-based enhancement algorithms using such paired data as training data, and making full-reference image quality assessment applicable to underwater images possible. Summary of the Invention

[0003] To address the technical problems in existing image quality assessment technologies, such as the time-consuming and labor-intensive nature of subjective methods and the inability to automate them, and the low accuracy of objective methods due to the lack of reference indicators and their inconsistency with human subjective perception,

[0004] This invention provides a method for evaluating the quality of full-reference underwater enhanced images based on paired samples, comprising:

[0005] Construct a dataset of paired underwater images; the paired underwater images consist of a reference image and a distorted underwater image.

[0006] Four types of feature maps are obtained from the reference image and the enhanced image; the enhanced image is generated by an underwater image enhancement algorithm.

[0007] The color distortion in the enhanced image is evaluated using attenuation and saturation maps to obtain a semantic similarity index;

[0008] The artifacts or blurring present in the enhanced image are evaluated using the dark channel prior map and the MSCN coefficient map to obtain the structural difference index;

[0009] By fusing semantic similarity index and structural difference index, the quality score of the enhanced image is obtained.

[0010] Further: attenuation map, saturation map, dark channel prior map, and graph MSCN coefficient map;

[0011] Construct attenuation maps for the reference and enhanced images:

[0012]

[0013] Among them I c (x,y) refers to the pixel values ​​of the R, G, and B channels, C refers to the R, G, and B color channels, x and y refer to the pixel coordinates; γ = 1.2 refers to the parameters of the correction operation;

[0014] Construct saturation maps for the reference image and the enhanced image:

[0015]

[0016] Where m(x,y) refers to the minimum value of the pixel values ​​of the three channels R, G, and B, and M(x,y) refers to the average value of the pixel values ​​of the three channels R, G, and B.

[0017] Constructing dark channel prior maps for the reference and enhanced images:

[0018]

[0019] I c (x) refers to the pixel values ​​of the R, G, and B color channels of pixel x, min means taking the minimum value, and Ω(x) refers to a 3×3 window area centered at point x.

[0020] Constructing the dark channel prior map MSCN coefficient map of the reference image and the enhanced image:

[0021]

[0022] Where: x, y refer to the coordinates of the pixel, I gray (x,y) refers to the gray value of point (x,y), μ(x,y) refers to the average gray value of the 7×7 square window centered at point (x,y), σ(x,y) refers to the standard deviation of the gray values ​​of the 7×7 square window centered at point (x,y), and C0 is a constant to prevent the denominator from being 0.

[0023] Furthermore, the process of evaluating the color distortion in the enhanced image using attenuation and saturation maps to obtain the semantic similarity index is as follows:

[0024] Calculate the semantic similarity of the decay maps:

[0025]

[0026] Calculate the semantic similarity of saturation maps:

[0027] SSI SM =λMSE(I satura (I e )-I satura(I r ));

[0028] Combining the semantic similarity of the decay map and the semantic similarity of the saturation map yields the semantic similarity index:

[0029]

[0030] in: This refers to the attenuation map of the R, G, and B channels of an enhanced image. Refers to the attenuation map of the R, G, and B channels of the reference image, I satura (I e (I) refers to the saturation map of an enhanced image. satura (I r ) refers to the saturation map of the reference image, and λ = 10 is a constant that adjusts the range of values ​​for SSI.

[0031] Furthermore: the process of evaluating artifacts or blurring in the enhanced image using the dark channel prior map and the MSCN coefficient map to obtain the structural difference index includes:

[0032] Calculate the SSIM difference of the local window dark channel prior map for each pixel:

[0033]

[0034] Calculate the SSIM difference of the local window MSCN coefficients for each pixel:

[0035]

[0036] By combining the SSIM difference of the dark channel prior map and the SSIM difference of the MSCN coefficient, the structural difference index is obtained:

[0037]

[0038] Among them, I dark (I e (I) refers to the dark channel prior map of the enhanced image. dark (I r (I) refers to the dark channel prior map of the reference image. MSCN (I e (I) refers to the MSCN coefficient map of the enhanced image. MSCN (I r ) refers to the MSCN coefficient map of the reference image, W refers to the width of the enhanced image, and H refers to the height of the enhanced image.

[0039] Furthermore: the SSIM difference expression is as follows:

[0040]

[0041] Where p and q refer to the two objects being compared, μ refers to the mean, σ refers to the variance, and C1 and C2 are constants to avoid equaling 0.

[0042] Furthermore: the fusion of the semantic similarity index and the structural difference index yields the following improved image quality score:

[0043] S = SSI 3 +SDI 3

[0044] Limit the range of FUIEQA scores:

[0045] Where: S refers to the difference between the enhanced image and the reference image.

[0046] A full-reference image quality evaluation device for evaluating underwater image enhancement algorithms includes: a construction module for constructing a pair of underwater image datasets; wherein the pair of underwater images consists of a reference image and a distorted underwater image;

[0047] Acquisition module: used to acquire four types of feature maps of the reference image and the enhanced image; the enhanced image is generated by an underwater image enhancement algorithm;

[0048] Semantic similarity evaluation module: The color distortion present in the enhanced image is evaluated using attenuation map and saturation map to obtain a semantic similarity index;

[0049] Structural difference evaluation module: used to evaluate the artifacts or blurring present in the enhanced image using the dark channel prior image and the MSCN coefficient image, and obtain the structural difference index;

[0050] Fusion module: Used to fuse semantic similarity index and structural difference index to obtain an improved image quality score.

[0051] A computer-readable storage medium storing a computer program, wherein when the computer program is executed, it performs the full-reference image quality evaluation method for evaluating underwater image enhancement algorithms as described in one claim.

[0052] This invention provides a full-reference underwater augmented image quality assessment method based on paired samples, which can effectively evaluate the semantic and structural similarity between underwater augmented images and ground-based real reference images, thereby evaluating different underwater image augmentation algorithms. Furthermore, this method has a high correlation with human subjective evaluation. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart of a full-reference underwater enhanced image quality assessment method based on paired samples according to the present invention;

[0055] Figure 2 (a) is the original image of one pair of images in the underwater image pair dataset; (b) is the reference image of one pair of images in the underwater image pair dataset.

[0056] Figure 3 (a) is a scatter plot of the MSE assessment method score and the human mean opinion; (b) is a scatter plot of the PSNR assessment method score and the human mean opinion; (c) is a scatter plot of the SSIM assessment method score and the human mean opinion; (d) is a scatter plot of the FRFSIM assessment method score and the human mean opinion; (e) is a scatter plot of the VI assessment method score and the human mean opinion; (f) is a scatter plot of the RI assessment method score and the human mean opinion; (g) is a scatter plot of the VSI assessment method score and the human mean opinion; (h) is a scatter plot of the FSIM assessment method score and the human mean opinion; (i) is a scatter plot of the FSIMC assessment method score and the human mean opinion. (j) is the fitted scatter plot of the human mean opinion; (k) is the fitted scatter plot of the GMSD evaluation method score and the human mean opinion; (l) is the fitted scatter plot of the BRISQUE evaluation method score and the human mean opinion; (m) is the fitted scatter plot of the FADE evaluation method score and the human mean opinion; (n) is the fitted scatter plot of the UIQM evaluation method score and the human mean opinion; (o) is the fitted scatter plot of the UCIQE evaluation method score and the human mean opinion; (p) is the fitted scatter plot of the NUIQ evaluation method score and the human mean opinion; (q) is the fitted scatter plot of the FUIEQA evaluation method score and the human mean opinion.

[0057] Figure 4 This is a ranking chart of the best results from seven image quality assessment methods. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] Figure 1 This is a flowchart of a method for evaluating the quality of fully referenced underwater enhanced images based on paired samples, as described in this invention. Figure 1 As shown, the method in this embodiment may include:

[0060] The present invention provides a flowchart of a full-reference image quality evaluation method for evaluating underwater image enhancement algorithms, comprising:

[0061] S1: Construct a paired underwater image dataset;

[0062] S2: Construct four types of feature maps for the reference image and the enhanced image;

[0063] S3: Use semantic similarity index to quantify color distortion produced by underwater image enhancement algorithms;

[0064] S4: Use the structural difference index to detect artifacts or blurs generated by underwater image enhancement algorithms;

[0065] S5: The final image quality score is obtained by fusing semantic similarity and structural differences.

[0066] Steps S1 / S2 / S3 / S4 / S5 are executed sequentially;

[0067] Furthermore, the construction of the underwater image dataset includes:

[0068] Sixty pairs of images were collected from the EUVP dataset;

[0069] Forty pairs of images were collected from the UIEB dataset;

[0070] One hundred pairs of images were collected from a synthetic dataset of ten types of ocean images.

[0071] like Figure 2 Figure (a) shows the original image of one pair of images in the underwater image pair dataset, and Figure (b) shows the reference image of one pair of images in the underwater image pair dataset.

[0072] Furthermore, the four types of feature maps obtained from the reference image and the enhanced image include: attenuation map, saturation map, dark channel prior map, and dark channel prior map MSCN coefficient map;

[0073] Construct attenuation maps for the reference and enhanced images:

[0074]

[0075] Among them I c (x,y) refers to the pixel values ​​of the R, G, and B channels, C refers to the R, G, and B color channels, x and y refer to the pixel coordinates; γ = 1.2 refers to the parameters of the correction operation;

[0076] Construct saturation maps for the reference image and the enhanced image:

[0077]

[0078] Where m(x,y) refers to the minimum value of the R, G, and B channels of the pixel point with pixel coordinates (x,y), and M(x,y) refers to the average value of the R, G, and B channels of the pixel point with pixel coordinates (x,y).

[0079] Constructing dark channel prior maps for the reference and enhanced images:

[0080]

[0081] I c (x) refers to the pixel values ​​of the R, G, and B color channels of pixel x, min means taking the minimum value, and Ω(x) refers to a 3×3 window area centered on point x.

[0082] Specifically, firstly, the minimum value of the R, G, and B channels of pixel x is calculated as the pixel value of the current pixel. Then, minimum value filtering is used to calculate the minimum value of the pixel in a 3×3 window area centered on the current pixel as the pixel value of the current pixel.

[0083] Constructing the MSCN coefficient map of the reference image and the enhanced image:

[0084]

[0085] Where: x, y refer to the coordinates of the pixel, I gray (x,y) refers to the gray value of point (x,y), μ(x,y) refers to the average gray value of the 7×7 square window centered at point (x,y), σ(x,y) refers to the standard deviation of the gray values ​​of the 7×7 square window centered at point (x,y), and C0 is a constant to prevent the denominator from being 0.

[0086] Specifically, the RGB color image is first converted into a grayscale image, and then the average grayscale value of the current window is calculated within a 7×7 square window for each pixel. and gray value variance Last use Calculate the MSCN coefficient for this pixel.

[0087] Furthermore, the process of evaluating the color distortion in the enhanced image using attenuation and saturation maps to obtain the semantic similarity index is as follows:

[0088] Calculate the semantic similarity of the decay maps:

[0089]

[0090] Calculate the semantic similarity of saturation maps:

[0091] SSI SM =λMSE(I satura (I e )-I satura (I r (6)

[0092] Combining the semantic similarity of the decay map and the semantic similarity of the saturation map yields the final semantic similarity index:

[0093]

[0094] in This refers to the attenuation map of the R, G, and B channels of an enhanced image. Refers to the attenuation map of the R, G, and B channels of the reference image, I satura (I e (I) refers to the saturation map of an enhanced image. satura (I r ) refers to the saturation map of the reference image, and λ = 10 is a constant that adjusts the range of values ​​for SSI.

[0095] Specifically, the mean square error of the attenuation maps of the enhanced image and the reference image is first calculated. Then, substituting λ=10, we calculate the semantic similarity of the decay map. Then, the mean square error of the saturation maps of the enhanced image and the reference image is calculated. Then, substituting λ=10, we calculate the semantic similarity (SSI) of the decay map. SM =λMSE(I satura (I e )-I satura (I r Finally, the semantic similarity of the decay map and the semantic similarity of the saturation map are combined to obtain the final semantic similarity index:

[0096]

[0097] Furthermore, the process of evaluating artifacts or blurring / details in the enhanced image using the dark channel prior map and the dark channel prior map MSCN coefficient map to obtain the structural difference index includes:

[0098] Calculate the SSIM difference of the local window dark channel prior map for each pixel:

[0099]

[0100] Calculate the SSIM difference of the local window MSCN coefficients for each pixel:

[0101]

[0102] By combining the SSIM difference of the dark channel prior map and the SSIM difference of the MSCN coefficients, the final structural difference index is obtained:

[0103]

[0104] Among them, I dark (I e (I) refers to the dark channel prior map of the enhanced image. dark (I r (I) refers to the dark channel prior map of the reference image. MSCN (I e (I) refers to the MSCN coefficient map of the enhanced image. MSCN (I r ) refers to the MSCN coefficient map of the reference image, W refers to the width of the enhanced image, and H refers to the height of the enhanced image.

[0105] Furthermore, the expression for the SSIM difference is as follows:

[0106]

[0107] Where p and q refer to the two objects being compared, μ refers to the mean, and σ refers to the variance. C1 and C2 are constants to avoid them being equal to 0.

[0108] Specifically, the average of two compared objects within a local 11×11 square window for each pixel is calculated. and Calculate the variance of two compared objects and Calculate the covariance of two compared objects Calculate the SSIM difference between two comparison objects

[0109] Furthermore, the final image quality score is obtained by fusing semantic similarity and structural difference, including:

[0110] Calculate the final difference score:

[0111] S = SSI 3 +SDI 3 (12)

[0112] Limit the range of FUIEQA scores:

[0113]

[0114] Pearson linear correlation coefficient (PLCC), Spearman rank correlation coefficient (SROCC), Kendall rank correlation coefficient (KROCC), and root mean square error (RMSE) are several commonly used coefficients for evaluating image quality assessment methods. PLCC scores the monotonicity between the predicted score and the mean square error (MOS), SROCC and KROCC measure the monotonicity of the IQA model with respect to MOS, and RMSE quantifies the prediction accuracy of each IQA model. Higher PLCC, SROCC, and KROCC scores, and lower MSE scores, indicate better predictive performance of the image quality assessment metrics. As shown in Table 1, our proposed method demonstrates the best performance compared to 16 other image quality assessment methods.

[0115] Table 1

[0116]

[0117]

[0118] Figure 3(a) is a scatter plot of the MSE assessment method score and the human mean opinion; (b) is a scatter plot of the PSNR assessment method score and the human mean opinion; (c) is a scatter plot of the SSIM assessment method score and the human mean opinion; (d) is a scatter plot of the FRFSIM assessment method score and the human mean opinion; (e) is a scatter plot of the VI assessment method score and the human mean opinion; (f) is a scatter plot of the RI assessment method score and the human mean opinion; (g) is a scatter plot of the VSI assessment method score and the human mean opinion; (h) is a scatter plot of the FSIM assessment method score and the human mean opinion; (i) is a scatter plot of the FSIMC assessment method score and the human mean opinion. (j) is the fitted scatter plot of the human mean opinion; (k) is the fitted scatter plot of the GMSD evaluation method score and the human mean opinion; (l) is the fitted scatter plot of the BRISQUE evaluation method score and the human mean opinion; (m) is the fitted scatter plot of the FADE evaluation method score and the human mean opinion; (n) is the fitted scatter plot of the UIQM evaluation method score and the human mean opinion; (o) is the fitted scatter plot of the UCIQE evaluation method score and the human mean opinion; (p) is the fitted scatter plot of the NUIQ evaluation method score and the human mean opinion; (q) is the fitted scatter plot of the FUIEQA evaluation method score and the human mean opinion.

[0119] like Figure 3 The figure shows a scatter plot of the fitted scores of 17 image quality assessment methods and the average human opinion. It can be seen that, compared with the other 16 image quality assessment methods, the scores of our proposed method have a strong correlation and good consistency with the subjective perception scores.

[0120] The most basic application of underwater image quality assessment metrics is to accurately and automatically select the best result from a set of enhancement results generated from the same distorted image using different underwater image enhancement methods. For example... Figure 4 The image shows a ranking of the best results from seven image quality assessment methods. It can be seen that our proposed method is the only one that accurately selects the best enhancement result, demonstrating the effectiveness of our method in selecting the best result from all candidates.

[0121] This invention can effectively evaluate the semantic and structural similarity between underwater enhanced images and ground-based real reference images, thereby evaluating different underwater image enhancement algorithms. Furthermore, this method has a high correlation with human subjective evaluation.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A full-reference underwater enhanced image quality assessment method based on paired samples, characterized in that, The method comprises the following steps: a paired underwater image dataset is constructed; the paired underwater images are composed of a reference image and a distorted underwater image; four types of feature maps of the reference image and the enhanced image are obtained; the enhanced image is generated by an underwater image enhancement algorithm; the four types of feature maps of the reference image and the enhanced image include an attenuation map, a saturation map, a dark channel prior map and an MSCN coefficient map; an attenuation map of the reference image and the enhanced image is constructed; wherein wherein R, G, B refer to the pixel values of the three channels, C refers to the three color channels R, G, B, and x, y refer to the pixel point coordinates; =1.2 refers to the parameters of the correction operation; a saturation map of the reference image and the enhanced image is constructed; wherein denotes the minimum value of the R, G, B three channel pixel values, denotes the average value of the R, G, B three channel pixel values; a dark channel prior map of the reference image and the enhanced image is constructed; ； Refers to the pixel value of the R, G, B three color channels of the pixel point x, min refers to taking the minimum value, Ω(x) Refers to a 3x3 window area centered on the x point; an MSCN coefficient map of the reference image and the enhanced image is constructed; wherein: x, y refer to the coordinates of the pixel point, refers to the gray value of the (x, y) point, μ (x, y) refers to the average value of the gray values of the 7x7 square window centered on the (x, y) point, σ (x, y) refers to the standard deviation of the gray values of the 7x7 square window centered on the (x, y) point, is a constant to prevent the denominator from being 0; color distortion existing in the enhanced image is evaluated by using the attenuation map and the saturation map to obtain a semantic similarity index; artifacts or blurs existing in the enhanced image are evaluated by using the dark channel prior map and the MSCN coefficient map to obtain a structural difference index; the semantic similarity index and the structural difference index are fused to obtain a quality score of the enhanced image.

2. The method of claim 1, wherein, the process of evaluating the color distortion existing in the enhanced image by using the attenuation map and the saturation map to obtain the semantic similarity index is as follows: the semantic similarity of the attenuation map is calculated: ； the semantic similarity of the saturation map is calculated: ； the semantic similarity index is obtained by combining the semantic similarity of the attenuation map and the semantic similarity of the saturation map: ； wherein: denotes the attenuation map of the enhanced image R, G, B, denotes the attenuation map of the reference image R, G, B, denotes the saturation map of the enhanced image, denotes the saturation map of the reference image, = 10 is a constant that adjusts the value range of the SSI.

3. The method of claim 1, wherein, the process of evaluating the artifacts or blurs existing in the enhanced image by using the dark channel prior map and the MSCN coefficient map to obtain the structural difference index comprises: the SSIM difference of the local window dark channel prior map of each pixel is calculated: the SSIM difference of the local window MSCN coefficient of each pixel is calculated: the structural difference index is obtained by combining the SSIM difference of the dark channel prior map and the SSIM difference of the MSCN coefficient: wherein, denotes a dark channel prior map of the enhanced image, denotes a dark channel prior map of the reference image, denotes a MSCN coefficient map of the enhanced image, denotes a MSCN coefficient map of the reference image, W denotes a width of the enhanced image, and H denotes a height of the enhanced image.

4. The method of claim 3, wherein, the SSIM difference expression is as follows: where: p and q refer to the two objects being compared, refers to the mean value, refers to the variance, and is to avoid a constant equal to 0.

5. The method of claim 1, wherein, the quality score of the enhanced image is obtained by fusing the semantic similarity index and the structural difference index as follows: Range limit the FUIEQA score: , wherein S refers to the difference size of the enhanced image and the reference image.

6. A full-reference image quality evaluation device for evaluating an underwater image enhancement algorithm, characterized in that, a construction module: used for constructing a paired underwater image dataset; the paired underwater images are composed of a reference image and a distorted underwater image; an acquisition module: used for obtaining four types of feature maps of the reference image and the enhanced image; the enhanced image is generated by an underwater image enhancement algorithm; the four types of feature maps of the reference image and the enhanced image include an attenuation map, a saturation map, a dark channel prior map and an MSCN coefficient map; an attenuation map of the reference image and the enhanced image is constructed; wherein wherein R, G, B are the pixel values of the three channels, C is the three color channels of R, G, B, and x, y are the pixel point coordinates; =1.2 is a parameter of the correction operation; a saturation map of the reference image and the enhanced image is constructed; wherein denotes the minimum value of the R, G, B channel pixel values, denotes the average value of the R, G, B channel pixel values; a dark channel prior map of the reference image and the enhanced image is constructed; ； Refers to the pixel value of the R, G, B three color channels of the pixel point x, min refers to taking the minimum value, Ω(x) Refers to the 3x3 window area centered on the x point; an MSCN coefficient map of the reference image and the enhanced image is constructed; wherein: x, y refer to the coordinates of the pixel point, refers to the gray value of the (x, y) point, μ (x, y) refers to the average value of the gray values of the 7x7 square window centered on the (x, y) point, σ (x, y) refers to the standard deviation of the gray values of the 7x7 square window centered on the (x, y) point, is a constant to prevent the denominator from being 0; a semantic similarity evaluation module: used for evaluating color distortion existing in the enhanced image by using the attenuation map and the saturation map to obtain a semantic similarity index; a structural difference evaluation module: used for evaluating artifacts or blurs existing in the enhanced image by using the dark channel prior map and the MSCN coefficient map to obtain a structural difference index; Fusion module: used for fusing semantic similarity index and structure difference index to obtain an increased image quality score.

7. A computer-readable storage medium having stored therein a computer program comprising, The computer program runs, and executes the full reference underwater enhanced image quality evaluation method based on paired samples in any one of claims 1-5.

Images