Data-anonymized sampling and differentiable related loss function

Abstract

A processing mechanism (1000) for evaluating a plurality of data units (201, 202) is described, the processing mechanism comprising one or more processors (1001) for processing the data units by means of a machine learning algorithm and thereby forming for each data unit an output value representing (i) a rating of the unit and (ii) a ranking of the unit relative to other data units of the plurality of data units. This can support easily comparing characteristics of the data units, e.g. their quality, using the output values.

Description

Technical Field

[0001] This invention relates to a mapping from data units (e.g., images) to quantitative values. These values ​​can be used to rank data units or otherwise compare them. Background Technology

[0002] Successfully mapping from a data unit (e.g., an image) to a quantitative (scalar) value of an arbitrary dimension (axis) involves generating a function that takes the data unit as input and outputs a scalar value representing some property of the input data unit.

[0003] An example of such a function involves perceptual image quality assessment and quantitative image scoring based on image visual quality. Once a single scalar is assigned to the images in the set, the list of images can be sorted along any dimension using the corresponding score, thus ranking the images in an order. In this application, sorting the list using a scalar will produce a list of images sorted by visual quality.

[0004] Using subjective visual quality as an example of image-score mapping, the process of automatically assigning scalars to images based on qualitative appearance is often called image quality assessment (IQA).

[0005] For this IQA application, a series of mapping techniques have been developed to automatically assess the perceived visual quality of images. An acceptable automated IQA estimate should generally be highly correlated with the quality assessments performed by a large number of human evaluators (often referred to as the mean opinion score (MOS)). IQA has been widely applied to downstream computational photography and computer vision problems, where image quality constitutes a fundamental aspect, such as image restoration, image super-resolution, and image retrieval. Research on IQA has focused on FR-IQA (full-reference) and NR-IQA (no-reference) settings, the latter being the case where image quality must be established for images with potential quality defects when no reference (baseline quality) images are available.

[0006] Both trainable (learnable) models and deterministic systems can be used to map images to quantitative evaluation scores.

[0007] Trainable models (such as neural networks) use labeled data (images, quantitative score tuples) in conjunction with supervised learning techniques to learn a regression function that maps images to corresponding evaluation scores. In this case, the evaluation ratings can constitute any image attribute.

[0008] For example, in IQA, image "perceived quality" can be used as an exemplary property. However, subjective image "perceived quality" is not a well-defined term in a mathematical sense, leading to examples of naturally ill-posed mapping problems.

[0009] The success and widespread popularity of convolutional neural networks (CNNs) have encouraged researchers to explore their potential applications in IQA problems and many other computer vision tasks. This research on a learnable, data-driven approach represents a significant improvement over previous hand-crafted methods.

[0010] A significant component of these learnable models is their requirement for large training data. Large, publicly available datasets for IQA problems are particularly scarce due to the relatively high costs associated with collecting such datasets (i.e., the significant labor and time required for MOS recording and processing of each image). Therefore, a key issue that any work considering a data-driven approach to solving IQA problems must address is the lack of large IQA datasets available for model training (function fitting).

[0011] Neural network methods are particularly sensitive to this, as the number of internal model parameters increases dramatically as the model becomes deeper and wider (increasing its expressiveness and flexibility), requiring significantly more data to avoid the well-known problem of "model overfitting," which often impairs generalization when considering unseen new test data (i.e., as the number of parameters increases, the model becomes more data-dependent, potentially harming predictive performance). In summary, a drawback of previous learnable IQA methods is the inherent conflict between the need for large amounts of data-intensive trainable CNN models and the high cost and expense of manually labeling and collecting IQA datasets.

[0012] Due to the aforementioned issue of dataset size, model training requires increasingly larger annotated datasets. However, as explained, the annotation process for IQA image datasets requires multiple manual annotations for each image, making collection and organization extremely labor-intensive and costly. Consequently, most available IQA datasets are too small to efficiently train CNNs.

[0013] A method needs to be developed to overcome these problems. Summary of the Invention

[0014] According to a first aspect, a processing mechanism for evaluating a plurality of data units is provided, the processing mechanism comprising one or more processors configured to process the data units via a machine learning algorithm and thus generate an output value for each data unit, the output value representing (i) a rating of the unit and (ii) a ranking of the unit relative to other data units among the plurality of data units. This allows for easy comparison of the characteristics of the data units, such as their quality, using the output values.

[0015] The data units can represent semantically distinct content, and the one or more processors are configured, as part of the algorithm, to select a group of data units for comparison processing, at least some of which include data units representing semantically distinct content. Compared to typical previous image semantic content-constrained comparisons, this method can support image content-agnostic sampling, and therefore can support downstream image-agnostic quality comparisons.

[0016] The machine learning algorithm can be operated on multiple batches, each batch including a partial set of the data units.

[0017] The machine learning algorithm can operate based on a training objective using a first differentiable correlation coefficient, which ensures a monotonic relationship between the output value of each member in the batch and the corresponding rating of each member within the batch. This allows for the integration of the differentiable correlation coefficient into the loss function. This enables the model to have a more global understanding of the ranking of the image set measured by the correlation coefficient.

[0018] The monotonic relationship can be a linear relationship. This can be a convenient way to implement it.

[0019] The first differentiable correlation coefficient can be the Pearson correlation coefficient. This makes the formula differentiable.

[0020] The machine learning algorithm can operate based on a training objective using a second differentiable correlation coefficient, which is used to establish a rank-dependent correlation between the output value of each member in the batch and the corresponding rating of each member within the batch. This allows for the integration of the differentiable correlation coefficient into the loss function. This enables the model to have a more global understanding of the ranking of the image set measured by the correlation coefficient.

[0021] The second differentiable correlation coefficient can be the Spearman correlation coefficient. Since the Spearman correlation coefficient calculates its rank based on the indicator function, it makes the formula non-differentiable.

[0022] In the preferred implementation, the first differentiable correlation coefficient and the second differentiable correlation coefficient are derived using the first derivative.

[0023] The data unit can be, for example, an image file, video file, text file, or audio file. The data unit can also be a perceptual data unit. Therefore, this mechanism can be used to sort and compare various types of data units.

[0024] The data unit can be an image file, and the rating can represent image quality. This allows the processing mechanism to be used for image quality assessment.

[0025] The image quality can be represented by one or more of the following: sharpness, contrast, color balance, blur, noise, haze, software-induced artifacts, and objects. This allows various aspects of the image to be considered when evaluating image quality.

[0026] According to a second aspect, a computer-implemented method for evaluating a plurality of data units is provided, the method comprising forming an output value for each data unit using a machine learning algorithm, the output value representing (i) a rating of the unit and (ii) a ranking of the unit relative to other data units among the plurality of data units. This can enable easy comparison of the characteristics of the data units, such as their quality, using the output values.

[0027] According to another aspect, a processing mechanism for evaluating a plurality of data units is provided, the processing mechanism comprising one or more processors, the processors being configured to: receive the plurality of data units, the data units representing semantically distinct content; form a plurality of groups of the data units for comparison processing, at least some of the groups including data units representing semantically distinct content; and for each group, perform comparison processing on the members of the group using a machine learning algorithm to form a rating for each member of the group relative to the other members of the group.

[0028] According to another aspect, a computer program is provided that, when executed by a computer, causes the computer to perform the above-described method. The computer program can be implemented on a non-transitory computer-readable storage medium. Attached Figure Description

[0029] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0030] Figure 1 Three images with a resolution of 64×64 are shown. x1 and x2 are two examples of distortion from the same reference image. y1 and y2 are the ground truth MOS scores, available during model training.

[0031] Figure 2A schematic diagram is provided illustrating how an exemplary perceptual quality algorithm uses a network f to derive a scalar rating d1 for an image x1.

[0032] Figure 3 A comparison is shown between images (ref^n, x^n_i) and (ref^n, x^n_j), where x^n_i and x^n_j originate from the same ref^n image (currently the standard IQA method).

[0033] Figure 4 The comparison between the image (ref^n, x^n_i) and (ref^m, x^m_j) is shown, where n and m, as well as i and j, can be completely different.

[0034] Figure 5 A schematic diagram is shown using the same pair for a fixed batch.

[0035] Figure 6 A schematic diagram is shown using arbitrary pairs of fixed batches.

[0036] Figure 7 A schematic diagram of batch all with arbitrary pairs is shown.

[0037] Figure 8 A schematic diagram of all batches with the same reference is shown.

[0038] Figure 9 An overview of an example of the steps performed as part of a processing mechanism for evaluating multiple data units is shown.

[0039] Figure 10 An example of an apparatus for implementing the method described herein is shown. Detailed Implementation

[0040] This invention provides a learning-based method for assigning quantitative ratings to data units (e.g., images). In the specific case of images, the rankings can be well aligned and correlated in both FR-IQA and NR-IQA settings.

[0041] To illustrate the approach, we take the IQA problem as an example use case. Specifically, it involves automatically mapping images to scalar quantitative values ​​that represent the visual (perceptual) quality of the images. The method described in this paper improves the performance of the outlined task compared to previous approaches to this problem.

[0042] Here, the data unit is a perceptual data unit in the form of an image. The image can be, for example, an RGB image or a RAW image.

[0043] This system is capable of learning a function mapping from input images to (quantitative, scalar) output scores. The learning process extends the well-known supervised learning paradigm. In the exemplary case of IQA, data including supervised learning training samples ([image, rating] pairs) is obtained.

[0044] As mentioned earlier, such training data is typically expensive to collect and requires manual labeling. Therefore, training datasets are usually small in terms of the number of samples.

[0045] The method described in this paper uses an image sampling strategy that effectively utilizes the training data and extracts as much information as possible from the data in order to obtain all available mileages from a small training data sample tuple.

[0046] Training involves using pairwise comparisons between image samples, so that during testing (deployment), the learned scoring function requires a (single) distorted image and a reference for its FR-IQA setting as input. For the NR-IQA setting, the learned scoring function only requires a (single) distorted image as input, without any reference. Training is performed in pairs, and valid sources of distorted images are defined.

[0047] Embodiments of the present invention include a training image sampling strategy, referred to herein as batch full sampling. The key to this method is relaxing the training image sampling constraints that typically constitute "training image pairs".

[0048] Compared to typical previous image semantic content-constrained comparisons, this method supports image content-agnostic sampling, thus enabling downstream image-agnostic quality comparisons (see "PieAPP: Perceptual Image-Error Assessment through Pairwise Preference" by Ekta Prashnani, Hong Cai, Yasamin Mostofi, and Pradeep Sen, Conference on Computer Vision and Pattern Recognition, 2018).

[0049] As will be described in more detail below, the result is that, theoretically, there are "N*N" pairs in the training "batch" of the model, not N pairs. In reality, due to bidirectional redundancy, there may be (N*N–N) / 2 pairs.

[0050] This effectively increases the training signals the model can obtain from the same number of original training images (original data size), thereby improving the "mileage" of available training data.

[0051] Improving (adding) the model training signal typically leads to better downstream quality of learned mappings for new, previously unseen images. To further increase the number of available comparisons, the "batch all" sampling strategy also computes scores for the entire available training mini-batch, enabling the model to gain a more comprehensive understanding of image content comparisons (compared to previous content-constrained comparisons). Integrated comparisons without any content constraints more closely mimic the fundamental IQA objective of comparing images, regardless of their semantic content.

[0052] For pairwise image comparisons, this method involves learning a scoring function that, in FR-IQA applications, takes the distorted image and its reference as input. Training is performed in pairs, with both distorted images derived from the same reference.

[0053] Figure 1 Three images with a resolution of 64×64 are shown. x1 and x2 are two distortion examples from the same reference image (ref). y1 and y2 are the ground truth MOS scores, available during model training.

[0054] In this example, the task is to learn a scoring function f that measures the quality of image x relative to its reference image ref. The output of the scoring function corresponds to a quality score.

[0055] For training, pairwise comparisons are performed between s1 = f(ref, x1) and s2 = f(ref, x2). In the exemplary case above, since the ground truth MOS value y1 > y2, s1 should be higher than s2.

[0056] During system deployment (i.e., during test-time inference and after model learning), pairwise comparisons are not required, and s = f(ref, x) can be simply computed.

[0057] This particular formula assumes access to a reference image during both the training and inference phases. However, the method can also be implemented without a reference image (i.e., in NR-IQA).

[0058] Any learning-based data-driven strategy that aims to fit a function that maps data units such as images to scalar values ​​(representing perceived quality or otherwise) typically requires a loss function. This is a surrogate objective that signals the model based on its current performance during learning, enabling it to update its internal parameters to improve iterative performance.

[0059] This process is often understood as "supervised learning." An example of a suitable loss function that can be used to learn a perceptual quality mapping function will now be described.

[0060] In marginal ranking loss, the goal is to learn which distorted image x1 or x2 is best (e.g., in terms of image quality). Learning involves minimizing the following losses:

[0061] l(s1,s2,y)=max(0,–y*(s1–s2)+m), where m is the boundary, if y1>y2, then y=1, otherwise y=–1.

[0062] This loss function was originally introduced in "Rank-IQA" by Liu, Xialei, Joost Van De Weijer and Andrew D. Bagdanov (IEEE International Conference on Computer Vision, 2017).

[0063] In temperature-based Bradley-Terry sigmoid, the goal is to minimize the difference between the predicted probability of x1 becoming better and the ground truth probability of x1 becoming better.

[0064] The predicted probability of x1 “getting better” (i.e., of better quality) corresponds to the Bradley-Terry sigmoid model:

[0065]

[0066] Where T is temperature. Then, learning involves minimizing the following loss:

[0067]

[0068] Elo ranking (see “PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual image Restoration”, Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, and ChaoDong, European Conference on Computer Vision, 2020) can also be used to consider MOS and for calculation:

[0069]

[0070] Where M = 400, as defined in the Elo ranking.

[0071] The loss function was originally introduced in PieAPP (see “PieAPP: Perceptual Image-Error Assessment through Pairwise Preference”, Ekta Prashnani, Hong Cai, Yasamin Mostofi, Pradeep Sen, Conference on Computer Vision and Pattern Recognition, 2018) and has been slightly adapted for the method described in this paper (i.e., by adding temperature and using Elo to calculate the ground truth).

[0072] In some embodiments of this method, a regularizer is derived based on the Pearson and Spearman correlation coefficients.

[0073] One component of this method involves modifying the previous loss function used for model training. Additional loss terms (relevance regularizers) can be used to improve model accuracy.

[0074] Taking the IQA model as an example, in this specific case, model evaluation is typically performed by measuring the Pearson and Spearman correlation coefficients. In the previous model training formula, only pairwise comparisons were performed. This ignores a "global" understanding of the ranking of the image set based on the correlation coefficient measurement.

[0075] In a preferred embodiment of the invention, the Pearson and Spearman correlation coefficients are configured to be differentiable so that they can be integrated into the loss function.

[0076] Obtain a dataset of n pairs:

[0077] {x i y i ), ..., (x n y n )}

[0078] This corresponds to the predicted score x and the ground truth score y. Here, x is the model output, and y is the MOS target.

[0079] The Pearson correlation coefficient r is defined as the product of the covariance of x and y divided by their standard deviations:

[0080]

[0081] in, and Let x and y be the sample means, respectively. From this definition, we can derive a regularizer at the batch level (i.e., calculate the Pearson correlation for mini-batches):

[0082] R P =||1-p|| p,

[0083] Among them, ||·|| p It is a p-norm.

[0084] The Spearman correlation coefficient is similar to the Pearson correlation coefficient, but it considers the ranks of x and y instead of the original values:

[0085]

[0086] in:

[0087] d i =rg(x i )-rg(y i ).

[0088] Formally, rank is defined as:

[0089] rg(x) = 1 + ∑ i＝j I{(x i -x j )<0},

[0090] Where I{.} is the indicator function. Since the Spearman correlation coefficient is ranked according to the indicator function, it makes the formula non-differentiable. Following the method in Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman's "Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval" (European International Conference on Computer Vision, 2020), the indicator function can be approximated by a temperature-based sigmoid:

[0091]

[0092] Here, T is typically set to 0.01. This approximation can then be replaced with rg(x) to make the Spearman correlation differentiable.

[0093] Based on this definition, a regularizer can be derived at the batch level (i.e., calculating the Spearman correlation for mini-batches):

[0094] R s =||1-s|| p ,

[0095] Among them, ||·|| p It is a p-norm.

[0096] Then, the final loss function becomes:

[0097]

[0098] In summary, the Pearson correlation and Spearman correlation, used as exemplary instances, are differentiable to allow them to be integrated into the model training loss function. Intuitively, the modified loss provides the model with additional signals about global image ranking information, which is not visible in the previous case of pairwise image loss only.

[0099] The above description illustrates how the model compares its predicted scores with ground truth scores (which can be obtained from the labeled training data).

[0100] The following outlines the model architecture and how to generate model prediction scores.

[0101] Consider an image tuple (ref, x^i), where x^i is the i-th distortion of the reference image ref. In other words, ref and x^i display the same semantic image content with different image qualities.

[0102] The goal of this task is to learn a function f to compute a quality score s_i = f(ref, x^i). In an exemplary implementation, f corresponds to a neural network learnable model (specifically, for example, the LPIPS network proposed in "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" by Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang, Computer Vision and Pattern Recognition, 2018).

[0103] Figure 2 A schematic diagram is provided illustrating how the perceptual quality algorithm uses network f to derive a scalar rating d1 for image x1.

[0104] Given a reference image ref 201 and a distorted version x1 202, a scalar score d1 203 is obtained by subtracting image feature representations at different levels. Intuitively, this measures the difference between image versions ref and x1 in different spaces (latent image representations).

[0105] More specifically, in this example, the algorithm includes the following steps:

[0106] • Extract image features of ref and x1 at five different levels (other levels are also possible). For images 201 and 202, this is typically shown at 204 and 205, respectively.

[0107] • The extracted image features were L2 normalized.

[0108] • Calculate the squared differences between features at all levels. For example, (F^L(ref)–F^L(x1))^2 represents the features at level L.

[0109] ● Apply the learned linear transformation to the features to map them into a single channel.

[0110] • Apply spatial averaging to the output scalar values ​​across all levels.

[0111] • Apply summation to finally obtain the scalar value d1 at 203.

[0112] The previous section described an overview of how to obtain scalar scores from sample data unit inputs using learnable models, such as neural networks.

[0113] The following describes how to select data unit pairs for the model during network training.

[0114] The example below describes an IQA application that uses multiple images as training input. However, this method is also applicable to non-image data units.

[0115] To learn the function, a pairwise learning scheme is used. Consider a pair of tuples (ref^n, x^n_i) and (ref^m, x^m_j), where no constraints are imposed on m and n, and i and j. In other words, the pair can include different reference images (i.e., m ≠ n) and different distortions (i.e., i ≠ j). For simplicity, s1 and s2 are referred to as pair 1 and pair 2, respectively. Then, we determine which tuple has better image quality.

[0116] This formula differs from those used in previous works, where operations typically impose constraints on (m and n) or (i and j) (e.g., PieAPP or RankIQA).

[0117] Therefore, in this method, instead of comparing two distorted images from a single reference image, we compare two distorted images from two different reference images. This allows for the creation of more pairs during model training than previous sampling strategies. This, in turn, provides the model with more "training signals," thereby improving model performance.

[0118] In the previous simple setup, the sampling strategy for image pairs was fixed. Specifically, the samples in the training mini-batch consisted of triples (ref^i, x^i_1, x^i_2), where the i-th reference image, x^i_1, and x^i_2 are two different distorted images derived from ref^i and randomly sampled.

[0119] Then, compare during training:

[0120] s1 = f(ref^i, x^i_1) and s2 = f(ref^i, x^i_2)

[0121] This sampling strategy is limited because only a small number of pairs are seen in each training iteration. This can particularly harm the pairwise ranking loss because when a pair is positive, it is actually useless as it does not provide any gradient (i.e., provide a training signal to the neural network model to iteratively improve the model).

[0122] In a preferred embodiment of the present invention, by comparing two distorted images derived from two different reference images, more flexibility can be introduced in the setup rather than artificially constraining by comparing two distorted images derived from the same reference image as previously discussed. This is referred to herein as "batch all" sampling.

[0123] During training, compare:

[0124] s1 = f(ref^i, x^i) and s2 = f(ref^j, x^j), where i ≠ j

[0125] Relaxing the constraint allows the use of this "batch all" sampling. This involves constructing all possible pairs in a mini - batch.

[0126] In theory, there can now be N * N pairs in a mini - batch, not N pairs. In practice, due to some redundancy in the comparisons, there may be (N * N – N) / 2 pairs.

[0127] When performing pairwise comparisons, it is not necessary to score both: s1 is better (s1 > s2) and s2 is worse (s2 < s1). Scoring one of the two is sufficient as the other is redundant information.

[0128] For example, consider a training batch size of 16. Feed 16 distorted images and 16 reference images into the network. Based on this input, 120 pairs can be constructed. By comparison, using the traditional simple sampling strategy (used in previous works) would result in 2 x 16 distorted images and 16 reference images, giving only 16 pairs in total.

[0129] Now some illustrative examples will be provided to show how the sampling strategy used in embodiments of the present invention differs from previous methods.

[0130] Figure 3 Shows a comparison between an image (ref^n, x^n_i) and (ref^n, x^n_j), where x^n_i and x^n_j are derived from the same ref^n image (currently a standard IQA method). This is referred to as same - pair comparison.

[0131] Figure 4The comparison between images (ref^n, x^n_i) and (ref^m, x^m_j) is shown, where n and m, and i and j, can be completely different. This is called an arbitrary pair comparison.

[0132] Once image pairs are defined, there are several ways to organize them into mini-batches for model training.

[0133] In a preferred implementation, the pairs in the mini-batch (including pairs 1 to N) are fixed. This is typically used for identical image pairs. In this context, triples are input into the model:

[0134]

[0135] Where i≠j represents the distortion type and N represents the batch size.

[0136] Figure 5 A schematic diagram is shown using identical pairs to fix the batch size. Given a small batch of size N, N triples, as shown in 501 to 503, are input into a model providing N pairwise comparisons. Each triple contains a reference image and two distorted versions of that reference image. For triple 501, the reference image ref... 1 As shown at 504, reference image x 1 The distorted versions are shown at 505 and 506. For each of the N triples, comparisons are made between the distorted images, shown at 507 for images 505 and 506.

[0137] Alternatively, any pair can be used in a fixed batch. In this context, the quadruple is input into the network:

[0138]

[0139] Figure 6 A schematic diagram is shown using arbitrarily fixed batches. Given a mini-batch of size N, N quadruplets as shown in 601 to 603 are input into a model providing N pairwise comparisons. Each quadruplet contains two pairs. Each pair includes a reference image and a distorted version of it. For quadruplet 601, the reference image ref... 1 As shown at 604, its distorted version x 1 Shown at 605. Images 604 and 605 comprise an image pair. Reference image ref N+1 As shown at 606, its distorted version x N+1 Shown at 607. Images 606 and 607 comprise a second image pair. For each of the N quadruples, comparisons are made between image pairs; for quadruple 601, shown at 608.

[0140] In the full batch implementation, pairs from the mini-batch are constructed for calculating the loss function. This implementation considers all possible available pairs within the mini-batch. In this context, tuples are input into the network:

[0141]

[0142] Pairs are then constructed by performing pairwise comparisons.

[0143] Figure 7 A schematic diagram of a batch-wide sampling method with arbitrary pairs is shown. Given a mini-batch of size N, the N pairs indicated at positions 701 to 703 are inputs providing N*N comparisons (theoretically, but this can be inefficient as some comparisons are redundant). Each pair includes a reference image and a distorted version of the reference image. For pair 701, the reference image ref 1 As shown at 704, its distorted version x 1 Shown at 705. For 702, refer to image ref. 2 As shown at 706, its distorted version x 2 Shown at 707. For 703, refer to image ref. N As shown at 708, its distorted version x N Shown at 709. At least some images may differ. Preferably, all images are different. An N*N comparison is performed. A comparison between pair 701 and pair 702 is shown at 710, a comparison between pair 701 and pair 702 is shown at 711, and a comparison between pair 701 and pair 703 is shown at 712. All other pairs among the N pairs are also compared.

[0144] It should be noted that some implementations may involve more constraint variations. For example, RankIQA initially proposed fixing the ref image and providing multiple distorted image inputs with the same ref image:

[0145]

[0146] Figure 8 A schematic diagram of a batch-wide sampling method with the same reference image is shown. Given a mini-batch of size N, the N pairs indicated at positions 801 to 803 are the inputs providing N*N comparisons (theoretically, but this can be inefficient because some comparisons are redundant). Within the mini-batch, for each pair, there is only one reference image. 1 (Shown at 804) and its N distorted versions x 1(For pairs 801 to 803, comparisons are shown at 805 to 807 respectively). At least some of the N distortion versions may differ. Preferably, all N distortion versions are different. A comparison between pair 801 and pair 802 is shown at 808, a comparison between pair 802 and pair 803 is shown at 809, and a comparison between pair 801 and pair 803 is shown at 810. Comparisons are also made for all other pairs among the N pairs.

[0147] The constraints are relaxed by having the training sample different reference images and different distortions in each iteration.

[0148] Following this terminology, the framework of this embodiment of the invention is based on sampling all batches having any pairs.

[0149] Therefore, data units operated by the network represent semantically different content, and groups of data units are selected for comparison processing, with at least some of the groups including data units representing semantically different content.

[0150] Machine learning algorithms operate on multiple batches, each batch consisting of a partial set of data units.

[0151] As described above, the machine learning algorithm can operate based on a training objective using a first differentiable correlation coefficient, which is used to ensure a monotonic relationship between the output value of each member in the batch and the corresponding rating of each member among the members in that batch. The monotonic relationship is preferably a linear relationship. As mentioned above, the first differentiable correlation coefficient can be a Pearson correlation coefficient.

[0152] Machine learning algorithms can operate based on a training objective using a second differentiable correlation coefficient, which is used to establish a rank-dependent correlation between the output value of each member in the batch and the corresponding rating of each member within that batch. As mentioned above, the second differentiable correlation coefficient can be the Spearman correlation coefficient.

[0153] The method described in this paper can facilitate the provision of better training signals to learnable models, which in turn yield more accurate image quality assessments downstream (with a higher correlation to the ground truth rating of the mean opinion score (MOS) of the image).

[0154] Using this method, images can be mapped to quantitative scalar scores across any dimension. One example involves quantitative image scoring based on subjective visual quality. Once scalars are assigned to images, the list of images can be sorted using the corresponding scores, allowing for sequential ranking of the images.

[0155] Therefore, the processing mechanism described in this paper typically processes multiple data units using machine learning algorithms, generating an output value for each data unit that represents the rating of the data unit and its ranking relative to other data units.

[0156] Using the above method, a system implementing this method can automatically (i.e., it can learn) quantitatively score data units, such as images in a given dimension (one example being image-score pairs, which can be seen during model training), such that the resulting image proxy rankings can faithfully reproduce the ground truth list ordering even when the original image scores are not allowed to be viewed or are not visible.

[0157] A trainable scoring system is able to assign quantitative scalar values ​​to images (or other types of data units) based on the appearance of an input image, where the values ​​evaluate the image (e.g., image visual quality) in previously specified and learned dimensions.

[0158] When the data unit is an image, image "quality" can represent, but is not limited to, one or more of the following: image sharpness, contrast, color balance, blur, noise, haze, and objects.

[0159] Image quality can also represent the presence of distortions from machine learning models. For example, machine learning models can be used with super-resolution images, which may create distortions in those images. Such software-induced artifacts can be included in the image quality definition for a particular image.

[0160] While the evaluation of images was described in the example above, the multiple data units can be in the form of video files, text files, audio files, or any other data units. For these data units, it is desirable to form an output value representing a rating of that data unit and its ranking relative to other data units. The data units are preferably perceptual data units that can be interpreted by one or more human senses.

[0161] Figure 9 Examples of steps performed by a processing mechanism for evaluating multiple data units are summarized. In step 901, the mechanism includes receiving multiple data units representing semantically distinct content. In step 902, the method includes forming groups of the multiple data units for comparison processing, at least some of the groups including data units representing semantically distinct content. In step 903, the method includes, for each group, performing comparison processing on the members of the group using a machine learning algorithm to form a rating for each member of the group relative to the other members of the group.

[0162] The rating can be based on the scalar output of each data unit in the model. As mentioned above, the scalar output of a data unit can represent the quality of that data unit, such as image quality, or the quality of an audio, video, or text file.

[0163] Figure 10 A schematic diagram of a device 1000 and its associated components for implementing the above-described mechanism is shown. The device may include a processor 1001 and a non-volatile memory 1002. The device may include more than one processor and more than one memory. The memory may store processor-executable data. The processor may be used to operate according to a computer program stored in a machine-readable storage medium in a non-transitory form. The computer program may store instructions that cause the processor to perform its methods in the manner described herein. These components may be implemented in physical hardware or deployed on various edge devices or cloud devices.

[0164] The applicant hereby discloses individually each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, regardless of whether such features or combinations of features solve any problem disclosed herein; and without limiting the scope of the claims. The applicant notes that aspects of the invention may include any such individual feature or combination of features. In view of the foregoing description, those skilled in the art will appreciate that various modifications can be made within the scope of the invention.

Claims

1. A processing mechanism for evaluating multiple data units, characterized in that, The processing mechanism includes one or more processors (1001) for processing the data units via a machine learning algorithm and thus forming an output value for each data unit, the output value representing (i) the rating of the data unit and (ii) the ranking of the data unit relative to other data units among the plurality of data units, the machine learning algorithm operating on multiple batches, each batch comprising a partial set of the data units, the machine learning algorithm operating according to a training objective using a first differentiable correlation coefficient, the first differentiable correlation coefficient being used to correlate the output value of each member of the batch with the corresponding rating of each member among the members of the batch. The data unit has a monotonic relationship with the reference image, and the machine learning algorithm operates based on a training objective using a second differentiable correlation coefficient. The second differentiable correlation coefficient is used to make the output value of each member of the batch and the corresponding rating of each member in the batch have a rank correlation over the entire batch. The method for determining the rating of the data unit includes: obtaining a scalar score of the data unit based on the squared difference between the reference image and the data unit at different levels of image features; outputting the rating of the data unit based on the scalar score of the data unit; the data unit includes an image file, the data unit is a distorted image of the reference image, and the data unit also includes a video file, a text file, or an audio file.

2. The processing mechanism according to claim 1, characterized in that, The data units represent semantically distinct content, and the one or more processors are configured, as part of the algorithm, to select a group of the data units for comparison processing, at least some of the group including data units representing semantically distinct content.

3. The processing mechanism according to claim 1, characterized in that, The monotonic relationship is a linear relationship.

4. The processing mechanism according to claim 1 or 3, characterized in that, The first differentiable correlation coefficient is the Pearson correlation coefficient.

5. The processing mechanism according to claim 1, characterized in that, The second differentiable correlation coefficient is the Spearman correlation coefficient.

6. The processing mechanism according to claim 1, characterized in that, The rating indicates image quality.

7. The processing mechanism according to claim 6, characterized in that, The image quality refers to one or more of the following: sharpness, contrast, color balance, blur, noise, haze, software-induced artifacts, and objects.

8. A method for evaluating a computer implementation of multiple data units, characterized in that, The method includes generating an output value for each data unit using a machine learning algorithm, the output value representing (i) the rating of the data unit and (ii) the ranking of the data unit relative to other data units of the plurality of data units, the machine learning algorithm operating on multiple batches, each batch including a partial set of the data units, the machine learning algorithm operating according to a training objective using a first differentiable correlation coefficient, the first differentiable correlation coefficient being used to make the output value of each member of the batch have a monotonic relationship with the corresponding rating of each member among the members of the batch throughout the batch, the machine learning algorithm operating according to a training objective using a second differentiable correlation coefficient, the second differentiable correlation coefficient being used to make the output value of each member of the batch have a rank correlation with the corresponding rating of each member among the members of the batch throughout the batch, the method for determining the rating of the data unit including: obtaining a scalar score of the data unit based on the squared difference between image features of a reference image and the data unit at different levels, outputting the rating of the data unit based on the scalar score of the data unit, the data unit including an image file, the data unit being a distorted image of the reference image, the data unit also including a video file, a text file, or an audio file.

9. A processing mechanism for evaluating multiple data units, characterized in that, The processing mechanism includes one or more processors (1001), which are used for: Receive the plurality of data units, wherein the data units represent semantically different content; Multiple groups of the data units are formed for comparison processing, at least some of the groups including data units representing semantically different content; For each group, members of the group are compared using a machine learning algorithm (903) to form a rating for each member of the group relative to the other members of the group. The machine learning algorithm operates on multiple batches, each batch including a partial set of the data units. The machine learning algorithm operates according to a training objective using a first differentiable correlation coefficient, which is used to make the output value of each member of the batch monotonically related to the corresponding rating of each member in the batch. The machine learning algorithm operates according to a training objective using a second differentiable correlation coefficient, which is used to make the output value of each member of the batch rank-correlated with the corresponding rating of each member in the batch. The method for determining the rating of the data unit includes: obtaining a scalar score of the data unit based on the squared difference between image features at different levels of a reference image and the data unit; outputting the rating of the data unit based on the scalar score of the data unit; the data unit includes an image file, the data unit being a distorted image of the reference image, and the data unit also including a video file, a text file, or an audio file.

Images