Multi-prior guidance underwater image quality evaluation method

By adopting a multi-priori guidance method of Transformer structure in underwater image quality evaluation, using serial and parallel feature extraction modules and twin networks, the accuracy and stability of underwater image quality evaluation are solved, and the underwater image quality evaluation is achieved Accurate evaluation of the performance of image processing methods and effective sorting of enhanced results.

CN118918086APending Publication Date: 2024-11-08DALIAN MARITIME UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410974581.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing underwater image quality evaluation methods are difficult to accurately evaluate the quality of the enhancement results in complex underwater environments, especially the stability of the unreferenced image quality evaluation method and the subjective evaluation deviation of humans are large, making it difficult to promote the underwater image enhancement method. improve.

Method used

Using a multi-priori guidance method based on Transformer structure, a network model is constructed through the serial feature extraction module and the parallel feature interaction module, multi-scale features are extracted and fusion is performed, and a twin network with shared weights and marginal ranking loss function is used for training to generate water The quality scores of the images are sorted differently.

Benefits of technology

Accurate evaluation of the performance of various underwater image processing methods is achieved, the accuracy and stability of underwater image quality evaluation is improved, and the deviation from human subjective evaluation is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918086A_ABST
    Figure CN118918086A_ABST
Patent Text Reader

Abstract

The invention provides a multi-prior guidance underwater image quality evaluation method. The multi-prior guidance underwater image quality evaluation method comprises the steps of dividing an underwater image data set into a training set and a test set; in a network model, a serial feature extraction module designs color prior, contrast prior and detail prior to guide learning of key features of an underwater image and is used for extracting multi-scale features of the underwater image, and a parallel feature interaction module is used for fusing the multi-scale features and extracting the multi-scale features of the underwater image. The linear mapping layer is used for calculating a quality score according to the fused multi-scale features; selecting any image group from the test set, and inputting all underwater images of the selected image group into the trained network model to obtain a score corresponding to each underwater image in the group; and according to the score corresponding to each underwater image in the group, all the underwater images in the group are sorted according to a score increasing or score decreasing sequence, and the performance of various underwater image processing methods can be accurately evaluated while underwater image quality evaluation is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image quality assessment, and in particular to a multi-priori guided underwater image quality assessment method. Background Art

[0002] In complex underwater environments, due to the absorption and scattering of light by suspended particles and soluble substances in the water, underwater images usually suffer from problems such as color attenuation, poor contrast, and blurred details, which limits the application of underwater images in practical scenarios. To solve this problem, scholars have developed a variety of underwater image enhancement methods to improve the quality of underwater images. These methods have shown excellent performance and achieved considerable results in improving the visual quality of underwater images. However, the enhancement results generated by different enhancement methods show different characteristics and have different visual quality. How to correctly evaluate the quality of the enhancement results and fairly compare the performance of different algorithms has become a key issue in promoting the development and progress of underwater image enhancement methods.

[0003] Existing image quality assessment methods can be divided into subjective methods and objective methods. Subjective methods evaluate the visual quality of images through human subjective perception. This method requires a lot of time and manpower and has great disadvantages in practical applications. Objective methods can be divided into full reference image quality assessment methods, partial reference image quality assessment methods and no reference image quality assessment methods according to whether reference images are required. Because it is almost impossible to obtain high-quality reference images corresponding to distorted underwater images, the research on underwater image quality assessment methods usually focuses on no reference image quality assessment methods. The main idea is to design the corresponding physical function based on a certain characteristic of the image, such as color and contrast, and then use the multivariate linear regression method to solve the parameters in the physical function so that it is close to human subjective visual evaluation. The stability of this type of method is easily affected by various underwater scenes. The evaluation results of underwater images with complex scenes and various degradation factors have a large deviation from human subjective evaluation and little reference significance, which makes it difficult to promote the improvement and development of underwater image enhancement methods.

[0004] In view of this, the present invention provides an underwater image quality assessment method guided by multiple priors. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned existing underwater image quality evaluation methods, the present invention provides an underwater image quality evaluation method guided by multiple priors. The present invention is mainly based on the Transformer structure, consisting of a serial feature extraction module and a parallel feature interaction module, and can more accurately evaluate the performance of various underwater image processing methods.

[0006] The technical means adopted by the present invention are as follows:

[0007] The present invention provides a method for underwater image quality assessment guided by multiple priors, comprising:

[0008] Constructing an underwater image dataset, wherein the underwater image dataset includes a plurality of image groups, wherein the image groups include a plurality of underwater images, and the underwater images in the same image group have the same scene but different qualities;

[0009] Dividing the underwater image dataset into a training set and a test set;

[0010] Constructing a network model, the network model comprising a serial feature extraction module, a parallel feature interaction module and a linear mapping layer, the serial feature extraction module is used to extract multi-scale features of the underwater image, the parallel feature interaction module is used to fuse the multi-scale features, and the linear mapping layer is used to calculate the quality score of the underwater image according to the fused multi-scale features;

[0011] Using the training set to train the network model to obtain the trained network model;

[0012] Select any one of the image groups from the test set, input all the underwater images of the selected image group into the trained network model respectively, and obtain a quality score corresponding to each underwater image of the group;

[0013] According to the quality score corresponding to each underwater image in the group, all the underwater images in the group are sorted in the order of increasing or decreasing quality scores.

[0014] Preferably, constructing the underwater image dataset includes:

[0015] Acquire a UIEB underwater dataset, where the UIEB underwater dataset includes a plurality of original underwater images;

[0016] Determine M underwater image enhancement methods, where M is an integer and M≥2;

[0017] The original underwater image is processed by using M underwater image enhancement methods respectively to obtain M enhanced underwater images corresponding to the original underwater image, and the original underwater image and its corresponding M enhanced underwater images are scored for quality to obtain the image group;

[0018] The image groups corresponding to all the original underwater images constitute the underwater image dataset.

[0019] Preferably, constructing the network model includes:

[0020] A first serial feature extraction module, a second serial feature extraction module, a third serial feature extraction module and a fourth serial feature extraction module connected in sequence;

[0021] The first parallel feature interaction module, the second parallel feature interaction module, the third parallel feature interaction module, the fourth parallel feature interaction module, the fifth parallel feature interaction module, the sixth feature interaction module and the linear mapping layer connected in sequence constitute a branch, the branch includes a first branch, a second branch and a third branch, the first parallel feature interaction module of the first branch is connected to the second serial feature extraction module, the first parallel feature interaction module of the second branch is connected to the third serial feature extraction module, and the first parallel feature interaction module of the third branch is connected to the fourth serial feature extraction module; the first parallel feature interaction module of the second branch is respectively connected to the first parallel feature interaction modules of the first branch and the third branch The second parallel feature interaction module of the second branch is connected with the second parallel feature interaction modules of the first branch and the third branch respectively, the third parallel feature interaction module of the second branch is connected with the third parallel feature interaction modules of the first branch and the third branch respectively, the fourth parallel feature interaction module of the second branch is connected with the fourth parallel feature interaction modules of the first branch and the third branch respectively, the fifth parallel feature interaction module of the second branch is connected with the fifth parallel feature interaction modules of the first branch and the third branch respectively, and the sixth parallel feature interaction module of the second branch is connected with the sixth parallel feature interaction modules of the first branch and the third branch respectively.

[0022] Preferably, the serial feature extraction module comprises:

[0023] A color prior module, used to generate a dark channel prior map of the underwater image;

[0024] A contrast prior module, used for generating a MSCN coefficient histogram of the underwater image;

[0025] A detail prior module, used for generating a gradient amplitude distribution histogram of the underwater image;

[0026] The multi-scale features of the underwater image are extracted according to the guidance of the dark channel prior map, the MSCN coefficient histogram and the gradient amplitude distribution histogram.

[0027] Preferably, the color prior module is calculated in the following manner:

[0028]

[0029] Among them, I c(p) is the pixel value of the underwater image at the pixel point p, Ω(x) is the window area centered on the pixel point x, min is the minimum function, and c is the r channel, g channel and b channel.

[0030] Preferably, the contrast prior module is calculated in the following manner:

[0031]

[0032] Among them, (x, y) is the coordinate of the pixel point, μ(x, y) is the grayscale average of the window centered on the pixel point (x, y), σ(x, y) is the grayscale standard deviation of the window centered on the pixel point (x, y), C0 is the first constant, I gray (x, y) is the grayscale value of the underwater image at the pixel point (x, y).

[0033] Preferably, the detail prior module is calculated in the following manner:

[0034]

[0035] Among them, (x, y) is the coordinate of the pixel point, I GM (x,y) is the gradient amplitude of the pixel point (x,y), is the linear convolution operator, h x is the Gaussian filter in the horizontal direction, h y is a Gaussian filter in the vertical direction.

[0036] Preferably, the parallel feature interaction module includes:

[0037] The convolutional attention module, feature addition module and forward feedback module connected in sequence constitute a branch, which includes a first branch, a second branch and a third branch. The convolutional attention module of the first branch is connected to the feature addition module of the second branch and the feature addition module of the third branch, the convolutional attention module of the second branch is connected to the feature addition module of the first branch and the feature addition module of the third branch, and the convolutional attention module of the third branch is connected to the feature addition module of the first branch and the feature addition module of the second branch.

[0038] Preferably, using the training set to train the network model includes:

[0039] S1: providing two network models, wherein the two network models share weights;

[0040] S2: selecting two underwater images from the training set and inputting them into two network models respectively, wherein the two underwater images belong to the same image group;

[0041] S3: The outputs of the two network models are used as inputs of a loss module, which calculates the loss and updates the parameters of the network model by back propagation according to the loss;

[0042] Repeat steps S2 to S3, and when the termination condition is met, the training of the network model is completed.

[0043] Preferably, the loss module calculates the loss in the following manner:

[0044] L(I h ,I l ;θ)=max(0,f(I h ;θ)-f(I l ;θ)+ε)

[0045] Among them, L(Ih,Il;θ) is the marginal ranking loss, I h is the underwater image with the highest quality score among the two selected underwater images, I l is the underwater image with the lowest quality score among the two selected underwater images, f(I h ;θ) is I h The corresponding score, f(I l ;θ) is I l The corresponding score, ε is the second constant, and θ is the parameter of the network model.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] The present invention provides a method for evaluating underwater image quality guided by multiple priors. The serial feature extraction module can fully extract the multi-scale features of underwater images, and the parallel feature interaction module can realize the interaction of multi-scale features. Convolutional attention modules are used in both the serial and parallel parts to extract image features. In view of the color attenuation, poor contrast, and blurred details caused by the complex and diverse underwater environment on underwater imaging, the present invention specially designs three physical priors to guide the network to learn the features of underwater distorted images. In the training stage, the present invention uses a twin network with shared weights to take enhanced images of different visual qualities generated from the same underwater distorted image as input to obtain quality estimation scores of different enhanced images, and then constrains the network parameters through the marginal ranking loss function to iteratively update the network parameters to obtain a network model that can sort the visual quality of different enhancement results of the same underwater distorted image. Through experimental verification, the underwater image quality evaluation method proposed by the present invention can more accurately evaluate the performance of various underwater image processing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 A structural diagram of the network model of the underwater image quality assessment method guided by multiple priors.

[0050] Figure 2 A schematic diagram of the proposed image group construction.

[0051] Figure 3 A structural diagram of a serial feature extraction module.

[0052] Figure 4 A structural diagram of the convolutional attention module.

[0053] Figure 5 A structural diagram of a parallel feature interaction module.

[0054] Figure 6 A flowchart for training a network model for underwater image quality assessment method guided by multiple priors. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0056] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0057] Reference Figure 1 , Figure 1 A structural schematic diagram of a network model of a method for evaluating underwater image quality under multi-prior guidance is provided to illustrate a specific embodiment of a method for evaluating underwater image quality under multi-prior guidance provided by the present invention, including:

[0058] Construct an underwater image dataset, which includes multiple image groups. Each image group includes multiple underwater images. Each group of underwater images is subjectively sorted according to visual quality. The underwater images in the same image group have the same scene but different quality.

[0059] Divide the underwater image dataset into training set and test set;

[0060] Constructing a network model, the network model includes a serial feature extraction module, a parallel feature interaction module and a linear mapping layer, the serial feature extraction module is used to extract multi-scale features of underwater images, the parallel feature interaction module is used to fuse multi-scale features, and the linear mapping layer is used to calculate the quality score of underwater images according to the fused multi-scale features;

[0061] The network model is trained using the training set to obtain a trained network model;

[0062] Select any image group from the test set, input all underwater images of the selected image group into the trained network model respectively, and obtain the quality score corresponding to each underwater image in the group;

[0063] According to the quality score corresponding to each underwater image in the group, all underwater images in the group are sorted in the order of increasing or decreasing quality scores.

[0064] It should be noted that, refer to Figure 1 , Serlal Block represents the serial feature extraction module, Parallel Block represents the parallel feature interaction module, and Linear represents the linear mapping layer.

[0065] It can be understood that the serial feature extraction module can fully extract the multi-scale features of underwater images, and the parallel feature interaction module can realize the interaction of multi-scale features. Convolutional attention modules are used in both the serial and parallel parts to extract image features. In view of the color attenuation, poor contrast, and blurred details caused by the complex and diverse underwater environment to underwater imaging, the present invention specially designs three physical priors to guide the network to learn the features of underwater distorted images. In the training stage, the present invention uses a twin network with shared weights to take enhanced images of different visual qualities generated from the same underwater distorted image as input to obtain quality estimation scores of different enhanced images, and then constrains the marginal ranking loss function to iteratively update the network parameters to obtain a network model that can visually rank different enhancement results of the same underwater distorted image. Through experimental verification, the underwater image quality evaluation method guided by multiple priors proposed in the present invention can accurately evaluate the performance of various underwater image processing methods while obtaining underwater image quality evaluation.

[0066] In some optional embodiments, referring to Figure 2 , Figure 2 A schematic diagram of the proposed image group construction is shown below. The underwater image dataset is constructed, including:

[0067] Obtain the UIEB underwater dataset, which includes multiple original underwater images;

[0068] Determine M underwater image enhancement methods, where M is an integer and M≥2;

[0069] The original underwater image is processed by M underwater image enhancement methods respectively to obtain M enhanced underwater images corresponding to the original underwater image, and the original underwater image and its corresponding M enhanced underwater images are subjectively scored to obtain an image group;

[0070] The image group corresponding to all original underwater images constitutes the underwater image dataset.

[0071] It should be noted that, in this embodiment, M is 15, and the 15 underwater image enhancement methods are all existing technologies. The 15 underwater image enhancement methods are UDCP, GDCP, ColorBalance, UNTV, TUDA, Retinex, Bayesian, WaterNet, Ushape, TwoStep, MMLE, IBLA, FUnIE, TEBCF, and Ucolor. Of course, they are not limited to these and can be set according to actual needs.

[0072] First, we selected the UIEB underwater dataset containing 890 real distorted underwater images, and then selected 15 currently superior underwater image enhancement methods with obvious differences in effect to enhance each of the underwater images, obtaining 13,350 enhanced underwater images, which, together with the original 890 underwater images, constituted a dataset containing 890 groups of 16 underwater images each. Secondly, we found 10 professional scholars in the field of underwater image processing, who conducted subjective evaluations of each group of 16 underwater images in terms of color, contrast, blur, etc., and gave a quality score for each underwater image. Finally, we obtained the ranking of each underwater image in its group, and obtained a ranking-based image group. Figure 2 is a schematic diagram of the image group, and Table 1 is the average opinion of 10 professional scholars. Figure 2 The visual quality ranking of the image groups shown in Table 1 is as follows:

[0073] Table 1. Figure 2 Image quality ranking table of the image groups shown

[0074] method Original Image UDCP GDCP ColorBalance UNTV TUDA Retinex Bayesian Sorting 9 7 15 11 8 12 13 14 method WaterNet Ushape TwoStep MMLE IBLA FUnIE TEBCF Ucolor Sorting 4 5 1 16 2 3 10 6

[0075] Based on the constructed underwater image dataset, 800 image groups can be selected as training sets, denoted as trainRank_800, and the other 90 image groups can be used as test sets, denoted as testRank_90.

[0076] In some optional embodiments, continue to refer to Figure 1 , build a network model, including:

[0077] A first serial feature extraction module, a second serial feature extraction module, a third serial feature extraction module and a fourth serial feature extraction module connected in sequence;

[0078] The first parallel feature interaction module, the second parallel feature interaction module, the third parallel feature interaction module, the fourth parallel feature interaction module, the fifth parallel feature interaction module, the sixth feature interaction module and the linear mapping layer connected in sequence constitute a branch, and the branch includes a first branch, a second branch and a third branch, the first parallel feature interaction module of the first branch is connected to the second serial feature extraction module, the first parallel feature interaction module of the second branch is connected to the third serial feature extraction module, and the first parallel feature interaction module of the third branch is connected to the fourth serial feature extraction module; the first parallel feature interaction module of the second branch is connected to the first branch, the third branch and the fourth serial feature extraction module respectively. The first parallel feature interaction module is connected to the first branch, the second parallel feature interaction module of the second branch is connected to the second parallel feature interaction module of the first branch and the third branch respectively, the third parallel feature interaction module of the second branch is connected to the third parallel feature interaction module of the first branch and the third branch respectively, the fourth parallel feature interaction module of the second branch is connected to the fourth parallel feature interaction module of the first branch and the third branch respectively, the fifth parallel feature interaction module of the second branch is connected to the fifth parallel feature interaction module of the first branch and the third branch respectively, and the sixth parallel feature interaction module of the second branch is connected to the sixth parallel feature interaction module of the first branch and the third branch respectively. It can be understood that the network structure of this embodiment is composed of a serial part and a parallel part. The serial part includes 4 groups of 8 serial feature extraction modules, 4 groups of 4 color prior modules, 4 groups of 4 contrast prior modules, and 4 groups of 4 detail prior modules. The parallel part includes 6 groups of 18 parallel feature interaction modules and 1 group of 3 linear mapping layers. The serial feature extraction module is used to extract the multi-scale features of the underwater enhanced image; the three physical prior modules integrate the prior information extracted from the underwater image into the network model to further guide the network to learn the key features of the input image; the parallel feature interaction module is used to fuse the extracted multi-scale features to promote the extraction of favorable complementary features while suppressing noise interference. The final multi-scale image features are averaged after being mapped by the linear layer to obtain the final image quality score.

[0079] In some optional embodiments, referring to Figure 3 and Figure 4 , Figure 3 A structural diagram of a serial feature extraction module. Figure 4 A structural diagram of the convolutional attention module. The serial feature extraction module includes:

[0080] Color prior module, used to generate dark channel prior maps of underwater images;

[0081] Contrast prior module, used to generate MSCN coefficient histogram of underwater images;

[0082] A detail prior module is used to generate a gradient amplitude distribution histogram of underwater images;

[0083] Multi-scale features of underwater images are extracted under the guidance of dark channel prior map, MSCN coefficient histogram and gradient amplitude distribution histogram.

[0084] It should be noted that, refer to Figure 3 , Patch Embed represents the feature embedding module, Conv-Attention represents the convolutional attention module, Feed-Forword represents the forward feedback module, HPM represents the color prior module, CPM represents the contrast prior module, and DPM represents the detail prior module. The input underwater image is first downsampled by the feature embedding module to generate image feature blocks. At the same time, the underwater image is injected into the network through the prior information generated by the three prior modules to guide the network to learn the image features.

[0085] Reference Figure 4 , the convolutional attention module first inputs the input image feature block into the convolutional position encoding module. In the convolutional position encoding module, the original image feature block is residually connected with the image feature after the depth-separable convolution, and then the Q, K, and V vectors are generated, which pass through the factored attention module and the convolutional relative position encoding module respectively. In the factored attention module, the K vector is matrix multiplied with the V vector, and the result is matrix multiplied with the Q vector. In the convolutional relative position encoding module, the Q vector is Hadamard-producted with the V vector after the depth-separable convolution. The results generated by the two modules are matrix-added to generate the output image feature block as the input of the subsequent forward feedback module.

[0086] It can be understood that after obtaining the three physical prior information, the four serial feature extraction modules respectively perform multi-scale feature extraction on the input image under their guidance to obtain complementary features under different receptive fields.

[0087] In some optional embodiments, the color prior module is calculated as follows:

[0088]

[0089] Among them, I c (p) is the pixel value of the underwater image at pixel point p, Ω(x) is the window area centered on pixel point x, min is the minimum function, and c is the r channel, g channel, and b channel.

[0090] It can be understood that the color prior module is used to generate a dark channel prior map of the underwater image, guide the network to learn the color characteristics of the image, favor color-balanced images, and despise images with severe color deviation. Specifically, Ω(x) can be a square window area with a size of 3×3 centered on the pixel point x, but it is not limited to this.

[0091] In some optional embodiments, the contrast prior module is calculated as follows:

[0092]

[0093] Among them, (x, y) is the coordinate of the pixel point, μ(x, y) is the grayscale average of the window centered on the pixel point (x, y), σ(x, y) is the grayscale standard deviation of the window centered on the pixel point (x, y), C0 is the first constant, I gray (x, y) is the grayscale value of the underwater image at the pixel point (x, y).

[0094] It can be understood that the contrast prior module is used to generate the MSCN coefficient histogram of the underwater image, guide the network to learn the contrast characteristics of the image, biased towards the image with strong contrast, and despised the image with weak contrast. Specifically, μ(x, y) is the grayscale average of the window with a size of 7×7 centered on the pixel point (x, y), and σ(x, y) is the grayscale standard deviation of the window with a size of 7×7 centered on the pixel point (x, y). Of course, it is not limited to this. The first constant C0 is set to prevent the denominator from being zero.

[0095] In some optional embodiments, the detail prior module is calculated as follows:

[0096]

[0097] Among them, (x, y) is the coordinate of the pixel point, I GM (x,y) is the gradient amplitude of the pixel point (x,y), is the linear convolution operator, h x is the Gaussian filter in the horizontal direction, h y is a Gaussian filter in the vertical direction.

[0098] It can be understood that the detail prior module is used to generate the gradient amplitude distribution histogram of the underwater image, guiding the network to learn the detail features of the image, favoring images with clear details and despising images with blurred details.

[0099] In some optional embodiments, referring to Figure 1 , Figure 4 and Figure 5 , Figure 5 FIG. 1 is a structural diagram of a parallel feature interaction module, which includes:

[0100] The convolutional attention module, feature addition module and forward feedback module connected in sequence constitute a branch, which includes a first branch, a second branch and a third branch. The convolutional attention module of the first branch is connected to the feature addition module of the second branch and the feature addition module of the third branch, the convolutional attention module of the second branch is connected to the feature addition module of the first branch and the feature addition module of the third branch, and the convolutional attention module of the third branch is connected to the feature addition module of the first branch and the feature addition module of the second branch.

[0101] It should be noted that, refer to Figure 5 , Conv-Attention represents the convolutional attention module, and Feed-Forword represents the forward feedback module. The parallel feature interaction module is used to adaptively fuse the multi-scale features obtained by the serial feature extraction module. The features of the three scales are adaptively fused through the convolutional attention module to extract complementary feature information and suppress noise interference.

[0102] It can be understood that the adaptive feature fusion operation is calculated as follows:

[0103] F=α0F0+α1F1+α2F2

[0104] Among them, F is the fused feature, α0, α1 and α2 are learnable weight parameters, and F0, F1 and F2 are multi-scale image features from the serial feature extraction module.

[0105] Specifically, six groups of parallel feature interaction modules jointly fuse multi-scale features. The fused features are respectively passed through linear mapping layers to generate underwater image quality scores. Finally, the average score of the three quality scores is taken as the final score of the underwater image.

[0106] In some optional embodiments, referring to Figure 6 , Figure 6 A flowchart of a network model training method for underwater image quality assessment guided by multiple priors is shown. The network model is trained using a training set, including:

[0107] S1: Provides two network models, and the two network models share weights;

[0108] S2: Select two underwater images from the training set and input them into two network models respectively. The two underwater images belong to the same image group;

[0109] S3: The outputs of the two network models are used as inputs to the loss module, which calculates the loss and updates the parameters of the network model based on the loss back propagation;

[0110] Repeat steps S2 to S3 until the termination condition is met and the training of the network model is completed.

[0111] It can be understood that the termination condition may be the number of iterations of steps S2 to S3, but is not limited thereto.

[0112] In some optional embodiments, the loss module calculates the loss in the following manner:

[0113] L(I h ,I l ;θ)=max(0,f(I h ;θ)-f(I l ;θ)+ε)

[0114] Among them, L(Ih,Il;θ) is the marginal ranking loss, I h is the underwater image with the highest quality score among the two selected underwater images, I l is the underwater image with the lowest quality score among the two selected underwater images, f(I h ;θ) is I h The corresponding score, f(I l ;θ) is I l The corresponding score, ε is the second constant, and θ is the parameter of the network model.

[0115] It can be understood that in order to constrain the learning process of the proposed network model and enable the invented method to correctly sort underwater enhanced images of different visual qualities, the marginal ranking loss function is used in this paper to supervise the model during model training. The second constant ε is set to prevent the scores of underwater enhanced images of different visual qualities from being too close to each other. In this embodiment, the second constant ε is set to 0.5, but it is not limited to this. Under the action of the marginal ranking loss function, the method of the present invention can effectively sort underwater enhanced images of different visual qualities.

[0116] Another specific embodiment of a multi-priori guided underwater image quality assessment method provided by the present invention includes:

[0117] Construct training and testing datasets, including:

[0118] The training set is trainRank_800, and the test set is testRank_90. Among them, one sample of the training set and the test set used includes underwater real distorted images and underwater enhanced images obtained using 15 different underwater image processing methods, and their image quality is manually ranked one by one. In addition, the training set and test set used are randomly selected, and there is no intersection between the training set and the test set.

[0119] Set up the experimental environment and parameters, including:

[0120] The proposed network model was built using the Pytorch deep learning framework; the model was trained using a GTX 3090 graphics processing unit on a Linux workstation; the Adam algorithm was used to optimize the training process; the learning efficiency was fixed at 0.0001; the input image size was 256×192, and the pixel values ​​were normalized to [0,1]; the BatchSize size was 1; and the number of training iterations was 100.

[0121] The network model is constructed according to the method, and the constructed network model is trained on a training data set to obtain trained weight parameters.

[0122] Test the test set testRank_90, including:

[0123] In this example, the Spearman rank correlation coefficient (SROCC) and the Kendall rank correlation coefficient (KROCC) are selected to demonstrate the effectiveness of the method of the present invention.

[0124] The Spearman rank correlation coefficient (SROCC) is calculated as follows:

[0125]

[0126] Among them, v i is the true sorting position of the i-th underwater image, p i is the predicted sorting position of the i-th underwater image, and N is the number of underwater images contained in the image group.

[0127] It can be understood that the true sorting refers to the order in which the underwater images of the image group are arranged in ascending or descending order according to the quality scores, and the predicted sorting refers to the order in which the underwater images of the image group are arranged in ascending or descending order according to the scores output by the network model.

[0128] Kendall rank correlation coefficient (KROCC) is calculated as follows:

[0129]

[0130] Among them, v h and v l are the true ranking positions of higher quality images and lower quality images, respectively, h and p l are the predicted ranking positions of higher quality images and lower quality images respectively, N is the number of underwater images contained in the image group, is the number of image pairs where the true ranking position of higher quality images and lower quality images is consistent with the predicted ranking position, is the number of image pairs whose actual ranking positions of higher-quality images and lower-quality images are inconsistent with the predicted ranking positions, and 1 / 2N(N-1) is the number of teams that can be obtained by pairing N underwater image samples.

[0131] The experiment of this embodiment on the test set can prove that the method of the present invention can rank the visual quality of different enhancement results of the same underwater distorted image, and has a high correlation with the results of human subjective evaluation.

[0132] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0133] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for underwater image quality assessment guided by multiple priors, characterized in that: include: Constructing an underwater image dataset, wherein the underwater image dataset includes a plurality of image groups, wherein the image groups include a plurality of underwater images, and the underwater images in the same image group have the same scene but different qualities; Dividing the underwater image dataset into a training set and a test set; Constructing a network model, the network model comprising a serial feature extraction module, a parallel feature interaction module and a linear mapping layer, the serial feature extraction module is used to extract multi-scale features of the underwater image, the parallel feature interaction module is used to fuse the multi-scale features, and the linear mapping layer is used to calculate the quality score of the underwater image according to the fused multi-scale features; Using the training set to train the network model to obtain the trained network model; Select any one of the image groups from the test set, input all the underwater images of the selected image group into the trained network model respectively, and obtain a quality score corresponding to each underwater image of the group; According to the quality score corresponding to each underwater image in the group, all the underwater images in the group are sorted in the order of increasing or decreasing quality scores.

2. The underwater image quality assessment method guided by multiple priors according to claim 1, characterized in that: Constructing the underwater image dataset includes: Acquire a UIEB underwater dataset, where the UIEB underwater dataset includes a plurality of original underwater images; Determine M underwater image enhancement methods, where M is an integer and M≥2; The original underwater image is processed by using M underwater image enhancement methods respectively to obtain M enhanced underwater images corresponding to the original underwater image, and the original underwater image and its corresponding M enhanced underwater images are scored for quality to obtain the image group; The image groups corresponding to all the original underwater images constitute the underwater image dataset.

3. The underwater image quality assessment method guided by multiple priors according to claim 1 or 2, characterized in that: Constructing the network model includes: A first serial feature extraction module, a second serial feature extraction module, a third serial feature extraction module and a fourth serial feature extraction module connected in sequence; The first parallel feature interaction module, the second parallel feature interaction module, the third parallel feature interaction module, the fourth parallel feature interaction module, the fifth parallel feature interaction module, the sixth feature interaction module and the linear mapping layer connected in sequence constitute a branch, the branch includes a first branch, a second branch and a third branch, the first parallel feature interaction module of the first branch is connected to the second serial feature extraction module, the first parallel feature interaction module of the second branch is connected to the third serial feature extraction module, and the first parallel feature interaction module of the third branch is connected to the fourth serial feature extraction module; the first parallel feature interaction module of the second branch is respectively connected to the first parallel feature interaction modules of the first branch and the third branch The second parallel feature interaction module of the second branch is connected with the second parallel feature interaction modules of the first branch and the third branch respectively, the third parallel feature interaction module of the second branch is connected with the third parallel feature interaction modules of the first branch and the third branch respectively, the fourth parallel feature interaction module of the second branch is connected with the fourth parallel feature interaction modules of the first branch and the third branch respectively, the fifth parallel feature interaction module of the second branch is connected with the fifth parallel feature interaction modules of the first branch and the third branch respectively, and the sixth parallel feature interaction module of the second branch is connected with the sixth parallel feature interaction modules of the first branch and the third branch respectively.

4. The underwater image quality assessment method guided by multiple priors according to claim 1, characterized in that: The serial feature extraction module comprises: A color prior module, used to generate a dark channel prior map of the underwater image; A contrast prior module, used for generating a MSCN coefficient histogram of the underwater image; A detail prior module, used for generating a gradient amplitude distribution histogram of the underwater image; The multi-scale features of the underwater image are extracted according to the guidance of the dark channel prior map, the MSCN coefficient histogram and the gradient amplitude distribution histogram.

5. The underwater image quality assessment method guided by multiple priors according to claim 4, characterized in that: The color prior module is calculated as follows: Among them, I c (p) is the pixel value of the underwater image at the pixel point p, Ω(x) is the window area centered on the pixel point x, min is the minimum function, and c is the r channel, g channel and b channel.

6. The underwater image quality assessment method guided by multiple priors according to claim 4, characterized in that: The contrast prior module is calculated as follows: Among them, (x, y) is the coordinate of the pixel point, μ(x, y) is the grayscale average of the window centered on the pixel point (x, y), σ(x, y) is the grayscale standard deviation of the window centered on the pixel point (x, y), C0 is the first constant, I gray (x, y) is the grayscale value of the underwater image at the pixel point (x, y).

7. The underwater image quality assessment method guided by multiple priors according to claim 4, characterized in that: The detail prior module is calculated as follows: Among them, (x, y) is the coordinate of the pixel point, I GM (x,y) is the gradient amplitude of the pixel point (x,y), is the linear convolution operator, h x is the Gaussian filter in the horizontal direction, h y is a Gaussian filter in the vertical direction.

8. The underwater image quality assessment method guided by multiple priors according to claim 1, characterized in that: The parallel feature interaction module includes: The convolutional attention module, feature addition module and forward feedback module connected in sequence constitute a branch, which includes a first branch, a second branch and a third branch. The convolutional attention module of the first branch is connected to the feature addition module of the second branch and the feature addition module of the third branch, the convolutional attention module of the second branch is connected to the feature addition module of the first branch and the feature addition module of the third branch, and the convolutional attention module of the third branch is connected to the feature addition module of the first branch and the feature addition module of the second branch.

9. The underwater image quality assessment method guided by multiple priors according to claim 1, characterized in that: The network model is trained using the training set, including: S1: providing two network models, wherein the two network models share weights; S2: selecting two underwater images from the training set and inputting them into two network models respectively, wherein the two underwater images belong to the same image group; S3: The outputs of the two network models are used as inputs of a loss module, which calculates the loss and updates the parameters of the network model by back propagation according to the loss; Repeat steps S2 to S3, and when the termination condition is met, the training of the network model is completed.

10. The underwater image quality assessment method guided by multiple priors according to claim 9, characterized in that: The loss module calculates the loss in the following way: L(I h ,I l ;θ)=max(0,f(I h ;θ)-f(I l ;i)+e) Among them, L(I h ,I l ; θ) is the marginal ranking loss, I h is the underwater image with the highest quality score among the two selected underwater images, I l is the underwater image with the lowest quality score among the two selected underwater images, f(I h ; θ) is I h The corresponding score, f(I l ;θ) is I l The corresponding score, ε is the second constant, and θ is the parameter of the network model.