Data-efficient blind super-resolution image quality assessment method and system based on model ranking

Abstract

A model ranking-based data-efficient blind super-resolution image quality evaluation learning method, characterized by: generating a super-resolution image set from an original low-resolution image set using multiple super-resolution methods, selecting a subset that maximizes the difference using the maximum difference competition method, conducting subjective testing on the super-resolution images corresponding to the selected subset to obtain a global ranking of the multiple super-resolution methods, transferring the global ranking to instance-level pseudo-labels of unlabeled super-resolution images to obtain a pseudo-label data set, and training a blind super-resolution image quality evaluation model in a semi-supervised learning manner using a pair-wise ranking learning method combined with the pseudo-label data set and an existing image quality evaluation data set. The method can effectively alleviate the problem of requiring a large amount of manually labeled data for supervised training in blind image quality evaluation methods and effectively improve the performance of existing blind image quality evaluation models, resulting in better prediction results consistent with subjective evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image quality evaluation, in particular, to a blind super-resolution image quality evaluation method and system. BACKGROUND

[0002] Image quality evaluation is an important research field in computer vision, aiming to quantify the perceptual quality of images, so as to provide important performance indicators for other visual tasks (such as image super-resolution). The research in this field mainly revolves around two evaluation methods: subjective quality evaluation and objective quality evaluation. Subjective quality evaluation can best reflect human perception of image quality, but it is limited by high labor and time costs, and is not real-time, making it difficult to be applied on a large scale. Objective quality evaluation, as a substitute for subjective quality evaluation, aims to automatically evaluate the quality of images using a computational model, which is efficient and has good scalability.

[0003] According to whether it depends on the reference image, objective quality evaluation can be further divided into full-reference quality evaluation and no-reference quality evaluation, the latter is also called blind image quality evaluation. Full-reference quality evaluation quantifies the quality of images by directly comparing the differences between the reference image and the test image. In recent years, related research has made significant progress. However, in practical applications, especially in image super-resolution tasks, the original reference image is often not available, so the application scenario of full-reference quality evaluation is actually very limited. In contrast, blind image quality evaluation does not depend on the reference image, but only evaluates the quality of the image itself based on the information of the distorted image. In recent years, with the rapid development of deep learning, blind image quality evaluation methods based on deep learning have gradually become a research hotspot. Such methods usually need a large number of image-quality score pairs annotated by humans for supervised training, and high-quality human-annotated data requires high cost, and it is often unrealistic to obtain a large amount of high-quality human-annotated data.

[0004] Therefore, how to reduce the dependence on human-annotated data in model training while designing a data-efficient blind super-resolution image quality evaluation learning method has become a key problem to be solved at present. SUMMARY

[0005] In view of the above deficiencies in the prior art, the purpose of the present application is to provide a data-efficient blind super-resolution image quality evaluation method and system based on model ranking.

[0006] According to one aspect of the present application, a data-efficient blind super-resolution image quality evaluation learning method based on model ranking is provided, comprising:

[0007] Generating a super-resolution image set from an original low-resolution image set using a plurality of super-resolution methods;

[0008] selecting, from the original low-resolution image set, a subset that maximizes the difference between each pair of super-resolution methods using a max-difference competition method;

[0009] performing a subjective test on the super-resolution images corresponding to the subset selected for each pair of super-resolution methods to obtain a global ranking of the plurality of super-resolution methods;

[0010] transferring the global ranking of the plurality of super-resolution methods to instance-level pseudo-labels of unannotated super-resolution images to obtain a pseudo-label dataset;

[0011] training a blind super-resolution image quality evaluation model in a semi-supervised learning manner using a pairwise ranking learning method and combining the pseudo-label dataset with an existing image quality evaluation dataset.

[0012] Preferably, the super-resolution images are generated from an original low-resolution image set using a plurality of super-resolution methods, wherein:

[0013] The original low-resolution image set is an unannotated and unreference low-resolution image set from the real world.

[0014] The plurality of super-resolution methods can be selected from mainstream image super-resolution methods.

[0015] Preferably, the subset that maximizes the difference between each pair of super-resolution methods is selected from the original low-resolution image set using a max-difference competition method, comprising:

[0016] For each pair of different super-resolution methods, each sample in the original low-resolution image set is traversed to iteratively select K samples that maximize the difference between the two methods.

[0017] Preferably, the super-resolution images corresponding to the subset selected for each pair of super-resolution methods are subjected to a subjective test to obtain a global ranking of the plurality of super-resolution methods, comprising:

[0018] In each round of the subjective test, a pair of super-resolution images generated from the same original low-resolution image is displayed, and a two-alternative forced choice (2AFC) method is used.

[0019] A count matrix is constructed according to the human preference data obtained from the subjective test, and a maximum likelihood estimation is used under a Thurstone model to infer the global ranking of the plurality of super-resolution methods.

[0020] Preferably, the global ranking of the plurality of super-resolution methods is transferred to instance-level pseudo-labels of unannotated super-resolution images to obtain a pseudo-label dataset, wherein:

[0021] The unlabeled images include all the remaining images in the set of super-resolution images generated by the plurality of super-resolution methods that are not selected by the maximum difference competition method.

[0022] The pseudo label is the relative position of the super-resolution method that generates the current unlabeled image in the global ranking vector.

[0023] Preferably, a blind super-resolution image quality evaluation model is trained in a semi-supervised learning manner by adopting a pairwise ranking learning manner and combining the pseudo label data set with an existing image quality evaluation data set, including:

[0024] For each two samples x and y in the same batch of training data, a 0-1 binary relative quality label is calculated, and the probability that the predicted value of x is greater than the predicted value of y is calculated under the Thurstone model.

[0025] Based on the 0-1 binary relative quality label and the probability, the fidelity loss of each two samples is calculated.

[0026] The fidelity losses of the pseudo label data set and the existing image quality evaluation data set are calculated respectively, added after setting a weight factor to obtain a total loss, and the model parameters are updated by back propagation.

[0027] Compared with the prior art, the present application has the following beneficial effects:

[0028] The present application can effectively alleviate the problem that a large amount of artificial labeled data is required for supervised training of a blind image quality evaluation method.

[0029] The present application can effectively improve the performance of an existing blind image quality evaluation model and obtain a prediction result with better consistency with subjective evaluation. BRIEF DESCRIPTION OF DRAWINGS

[0030] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0031] Figure 1 Flow chart of the data-efficient blind super-resolution image quality evaluation learning method based on model ranking of an embodiment of the present application; DETAILED DESCRIPTION

[0032] The embodiments of the present application will be described in detail below: The present embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation manners and specific operation processes are given. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0033] As Figure 1 shown, a model ranking-based data-efficient blind super-resolution image quality evaluation learning method in an embodiment of the present application includes:

[0034] S1: Generate a super-resolution image set from the original low-resolution image set using multiple super-resolution methods to improve the resolution of the image and thus improve the details and clarity of the image. Specifically, it processes an unlabeled and reference-free low-resolution image set from the real world. The super-resolution method can be selected from mainstream image super-resolution methods.

[0035] In this embodiment, 12 super-resolution methods are selected, namely USRGAN, Real-ESRGAN, BSRGAN, DASR, FeMaSR, LDL, DiffBIR, RGT, ResShift, SinSR, StableSR and SeeSR.

[0036] For a given set of original low-resolution images, the above 12 super-resolution methods are used for processing. Each method will extract features, map features and reconstruct high-resolution images according to its own algorithm and model, including forward propagation calculation, i.e. mapping low-resolution images to high-resolution image space through deep learning model. Finally, each method will generate a corresponding super-resolution image set. These image sets not only have improved resolution, but also may contain more detailed and clearer edges.

[0037] S2: Use the maximum difference competition method to select a subset from the original low-resolution image set that maximizes the difference between each pair of super-resolution methods in step S1; that is, use the maximum difference competition method to select a subset from the original low-resolution image set that maximizes the difference between two super-resolution methods. Specifically, for each pair of different super-resolution methods f1 and f2, iterate through each sample x in the low-resolution image set X to solve the following expression to select the kth sample that maximizes the difference between the two methods:

[0038]

[0039] where S is the set of k-1 selected samples according to the max difference competition method, D1 is a metric for estimating the perceptual distance between two super-resolution methods f1 and f2, D2 is a metric for estimating the semantic distance between a certain image sample x and the selected sample set S, λ is a balancing factor between the two distances D1 and D2, usually ranging from 0 to 1, for adjusting the importance of perceptual difference and semantic diversity, where the perceptual distance usually reflects the subjective feeling of human visual system on image difference, which can be measured by structural similarity (SSIM), peak signal-to-noise ratio (PSNR) or more advanced perceptual quality evaluation indicators. The semantic distance measures the similarity in image content.

[0040] S3: Subjective test is performed on the super-resolution images corresponding to the subset selected by each pair of super-resolution methods in step S2 to obtain a global ranking of the plurality of super-resolution methods in S1;

[0041] Specifically, according to the human preference data obtained by the subjective test, an N×N count matrix C is constructed, where N is the number of selected super-resolution methods, C ij is the number of times f i is selected in the comparison with f j . Thereafter, maximum likelihood estimation is used under the Thurstone model to infer the global ranking of the selected plurality of super-resolution methods. Specifically, a global ranking vector μ is solved to maximize the log-likelihood function of the count matrix C as follows:

[0042]

[0043] where μ = [μ1, μ2,..., μ N ] is the global ranking vector of the plurality of super-resolution methods, Ф is the standard normal distribution function, and satisfies Σ i μ i = 0.

[0044] S4: The global ranking of the plurality of super-resolution methods obtained in S3 is transferred to instance-level pseudo-labels of unannotated super-resolution images to obtain a pseudo-label dataset;

[0045] Specifically, the instance-level pseudo-label is the relative position of the super-resolution method generating the current unannotated image in the global ranking, and the value range is {1, 2,..., N}. Each pseudo-label data sample is an image-pseudo-label data pair.

[0046] S5: A blind super-resolution image quality evaluation model is trained in a semi-supervised learning manner by using the pair-wise ranking learning method and combining the pseudo-label dataset obtained in S4 with the existing image quality evaluation dataset.

[0047] Specifically, for each pair of samples x, y in the same batch of training data, a relative quality label is computed:

[0048]

[0049] where μ x ,μ y are the mean opinion scores of x, y.

[0050] For a blind image quality assessment model q w (·) with a predefined computational structure, under the Thurstone model, the probability that the predicted value of x is higher than that of y is computed as:

[0051]

[0052] where Φ(·) is the probability cumulative function of the standard normal distribution.

[0053] As a preferred embodiment, the blind image quality assessment model q w (·) with a predefined computational structure is chosen to be LIQE.

[0054] The model is trained with fidelity loss, specifically:

[0055]

[0056] For the pseudo-label dataset, the instance-level pseudo-labels are only available between the super-resolution images generated from the same original low-resolution image. Specifically, the data loader loads all the super-resolution images generated from the same original low-resolution image as a training batch each time, and the relative quality label is computed by the global ranking of the corresponding super-resolution methods.

[0057] The pseudo-label dataset is combined with existing image quality assessment datasets for joint training. Specifically, for the image quality assessment datasets with real labels, an additional fidelity loss term is computed, resulting in a semi-supervised learning loss function:

[0058] l = l t + αl p

[0059] where l t is the fidelity loss of the image quality assessment datasets with real labels, l p is the fidelity loss of the pseudo-label dataset, and α is the weight factor.

[0060] As a preferred embodiment, the existing image quality assessment datasets are chosen to be KonIQ-10k and PIPAL, and the weight factor α is chosen to be 0.1.

[0061] Thereafter, error back propagation is performed and the model parameters are updated.

[0062] Implementation effects:

[0063] In order to verify the effectiveness of the model ranking-based data-efficient blind super-resolution image quality evaluation learning method provided in the above embodiments of the application, the average 2AFC score of the model on all test images can be calculated based on the results of the subjective test. The 2AFC score is defined as:

[0064] 2AFCscore=pq+(1-p)(1-q)

[0065] Where p is the vote rate, and q∈{0,1} is the model preference.

[0066] The performance test results are shown in Table 1. Several baseline models are also tested for comparison. It can be seen that the embodiments of the application are significantly better than the baseline models in performance.

[0067] Table 1

[0068]

[0069] In order to verify the generality of the model ranking-based data-efficient blind super-resolution image quality evaluation learning method provided in the above embodiments of the application, tests can be performed on multiple image quality evaluation data sets, including: QADS, Ma17, PIPAL, KonIQ-10k, KADID-10k, and SPAQ. The model performance is mainly measured by two classical image quality evaluation indicators, including PLCC and SRCC. The performance test results are shown in Table 2. Several baseline models are also tested for comparison. It can be seen that the embodiments of the application are better than most of the baseline models in performance.

[0070] Table 2

[0071]

[0072] The model ranking-based data-efficient blind super-resolution image quality evaluation learning method provided in the above embodiments of the application can effectively alleviate the problem that blind image quality evaluation methods need a large amount of manually labeled data for supervised training, effectively improve the performance of existing blind image quality evaluation models, and obtain better prediction results consistent with subjective evaluation.

[0073] It should be noted that the steps in the method provided by the present application can be realized by the corresponding modules, devices, units, etc. in the system, and those skilled in the art can refer to the technical solutions of the system to realize the step flow of the method, that is, the embodiments in the system can be understood as preferred examples of realizing the method, and will not be described here.

[0074] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device thereof in the form of pure computer readable program code, the same functions can also be realized by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. Therefore, the system provided by the present application and each device thereof can be considered as a hardware component, and the devices contained therein for realizing various functions can also be considered as structures within the hardware component; the devices for realizing various functions can also be considered as both software modules for realizing methods and structures within the hardware component.

[0075] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The above embodiments and features in the embodiments can be combined with each other without conflict.

Claims

1. A model-based ranking data-efficient blind super-resolution image quality assessment learning method, characterized in that, The method comprises the following steps: S1. Obtain a set of original low-resolution images, and generate a corresponding set of super-resolution images using a plurality of super-resolution methods; S2. Use a max-difference competition method to select a subset from the set of original low-resolution images that maximizes the difference between each pair of super-resolution methods; S3. Perform a subjective test on the super-resolution images corresponding to the selected subset, and determine a global ranking of the plurality of super-resolution methods; S4. Transfer the global ranking of the plurality of super-resolution methods to instance-level pseudo-labels of unlabeled super-resolution images to construct a pseudo-label dataset; S5. Combine the pseudo-label dataset with an existing image quality evaluation dataset, and use a pair-wise ranking learning and semi-supervised learning method to train a blind super-resolution image quality evaluation model. The S2 specifically comprises the following steps: S2.1 Obtain the set of original low-resolution images X in step S1, wherein the set of original low-resolution images X contains a plurality of low-resolution image samples; S2.2 Select at least two different super-resolution methods f1 and f2, wherein each super-resolution method can convert a low-resolution image into a high-resolution image; S2.3 Define a perceptual distance D1 for estimating the perceptual distance between the reconstruction results of the same low-resolution image sample by the two super-resolution methods f1 and f2; S2.4 Define a semantic distance D2 for estimating the semantic distance between the low-resolution image sample and the selected sample set S; S2.5 Set a balance factor λ, wherein 0≤λ≤1, for adjusting the weight of the perceptual distance and the semantic distance in the calculation of the comprehensive score; S2.6 Initialize an empty set S as the set of selected samples; S2.7 For each pair of different super-resolution methods f1 and f2, traverse each sample x in the set of low-resolution images X, and perform the following operations: Calculate the perceptual distance D1(f1(x), f2(x)) between the images reconstructed by the different super-resolution methods f1 and f2 from the sample x; Calculate the semantic distance D2(x, S) between the sample x and all samples in the set S; According to the perceived distance D1, the semantic distance D2 and the balance factor λ, a comprehensive score is calculated, and the sample that makes the comprehensive score maximum is selected The kth sample is added to the set S, and the formula is as follows: S2.8 Repeat step S2.7 until a predetermined number of samples is reached; S2.9 Output the set S as the subset that maximizes the difference between the super-resolution methods.

2. The model-based ranking data-efficient blind super-resolution image quality assessment learning method according to claim 1, characterized in that, The set of original low-resolution images in step S1 is a set of unlabelled and unreference low-resolution images from the real world; the plurality of super-resolution methods are selected from mainstream image super-resolution methods.

3. The model-based ranking data-efficient blind super-resolution image quality assessment learning method according to claim 1, characterized in that, The step S3 specifically comprises the following steps: S3.1 Obtain the image subsets generated by the different super-resolution methods in step S2, wherein each subset corresponds to each pair of super-resolution methods selected in step S2; S3.2 Perform a subjective test on each subset, and let human observers compare the images generated by different super-resolution methods and select the images that they think are better in quality; S3.3 Construct an N x N count matrix C according to the results of the subjective test, where N is the number of super-resolution methods selected, C ij is the number of times super-resolution method fi is selected in comparison with super-resolution method f2. S3.4 Under the Thurstone model, use a maximum likelihood estimation method to solve a global ranking vector that maximizes the log-likelihood function based on the count matrix C, wherein the global ranking vector represents the relative ranking of the plurality of super-resolution methods. S3.5 outputting the global ranking vector as the global ranking of the selected super-resolution method.

4. The model-based ranking data-efficient blind super-resolution image quality assessment learning method according to claim 1, characterized in that, The step S4 specifically comprises: S4.1 obtaining the global ranking of the plurality of super-resolution methods obtained in step S3; S4.2 providing a set of unlabeled super-resolution images; S4.3 for each unlabeled image, selecting a super-resolution method to process it and generating a high-resolution version of the image; S4.4 assigning an instance-level pseudo-label to the generated high-resolution image according to the position of the selected super-resolution method in the global ranking, the pseudo-label representing the relative position of the super-resolution method generating the current image in the global ranking, and the value range of the pseudo-label being {1, 2,..., N}, wherein N is the number of super-resolution methods in the global ranking; S4.5 combining each unlabeled image and its corresponding pseudo-label into a data pair to form a pseudo-label dataset.

5. The model-based ranking data-efficient blind super-resolution image quality assessment learning method according to claim 1, characterized in that, The step S5 specifically comprises: S5.1 for each two samples x, y in the same batch of training data, calculating a 0-1 binary relative quality label and calculating the probability that the predicted value of sample x is greater than the predicted value of sample y under the Thurstone model; S5.2 calculating the fidelity loss of each two samples based on the 0-1 binary relative quality label and the probability; S5.3 calculating the fidelity loss of the pseudo-label dataset and the existing image quality evaluation dataset respectively, adding the total loss after setting the weight factor, and updating the model parameters through back propagation.

6. A model-based ranking data-efficient blind super-resolution image quality assessment terminal, characterized by The computer program product comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor executes the program to execute the method of any one of claims 1-5. The computer program product comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor executes the program to execute the method of any one of claims 1-5.

Images