Target data set analysis

Abstract

A system and method for performing an analysis relating to objects in a target data set using a computational model are described, wherein the computational model is obtained by applying machine learning to training data including training objects associated with training labels that classify the training objects. The method includes determining a prior indicative of an assumed label distribution in the target data set, using the computational model to classify a number of objects in the target data set and thereby determine target labels for the number of objects, calculating a predicted label distribution for the target data set based on the prior and the target labels, and using the predicted label distribution to perform an analysis relating to the objects. Other embodiments are also described and claimed.

Description

TARGET DATA SET ANALYSIS Background of the Invention

[0001] The present invention relates to a system and method for performing an analysis relating to objects in a target data set, and specifically to a system and method for performing analysis using a predicted label distribution for the target data set. Description of the Prior Art

[0002] The reference in this specification to any prior publication (or information derived from it), or to any matter which is known, is not, and should not be taken as an acknowledgement or admission or any form of suggestion that the prior publication (or information derived from it) or known matter forms part of the common general knowledge in the field of endeavour to which this specification relates.

[0003] Label Shift Estimation is the problem of estimating target label distributions based on source domain data and unlabelled target domain samples. Specifically, in supervised learning tasks, the performance of a classifier, such as a neural network classifier, can decrease considerably under distribution shift between source and target domains. As a type of distribution shift, label shift occurs when label distributions ℙ(Y = ·) are different in the source and target domains, while the data distribution conditioned on the label ℙ(X = x | Y = ·) is preserved.

[0004] Under label shift, an optimal classifier on the source domain may no longer be optimal on the target domain. Class imbalance problems can be modelled as label shift problems. One extreme case is Long-Tailed classification, where ℙ(X = x | Y = ·) is preserved while the training set has a Long-Tailed label distribution and the test set has an unknown label distribution. The classifier trained on the source domain has to be adjusted for optimal performance on the target domain. Label shift studies the general case of arbitrary source and target label distributions, including the Long-Tailed classification case.

[0005] Three important problems arise due to label shift, namely detection, estimation and correction. Detection involves detecting if label shift has occurred, estimation estimates the target label distribution and correction aligns the classifier to the target domain.

[0006] Typical approaches to the label shift problem require explicit modelling of the conditional probability ℙ(X = x | Y = y), which is not feasible for high-dimensional data such as images.

[0007] Jiaxian Guo, Mingming Gong, Tongliang Liu, Kun Zhang, and Dacheng Tao, "Ltf: A label transformation framework for correcting label shift", in International Conference on Machine Learning, pages 3843–3853. PMLR, 2020 propose to construct a marginal distribution of data ℙ(X = x), the target label distribution is estimated by matching the constructed distribution with a ground truth target domain distribution estimated by the unlabelled data.

[0008] For high dimensional datasets, Marco Saerens, Patrice Latinne, and Christine Decaestecker "Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure.", Neural computation, 14(1):21–41, 2002, proposes an expectation maximisation (EM) algorithm to obtain Maximum Likelihood Estimates (MLE) of the target label distribution, referred as Maximum Likelihood Label Shift (MLLS). MLLS can be effective on CIFAR10 and MNIST datasets.

[0009] Estimation can be based on labelled data from the source domain, using a blackbox classifier and unlabelled target domain data, as described for example in Zachary Lipton, Yu- Xiang Wang, and Alexander Smola "Detecting and correcting for label shift with black box predictors" In International conference on machine learning, pages 3122–3130. PMLR, 2018. BlackBox Shift Estimation (BBSE) is used to first model the correlation between predicted labels from a blackbox classifier and the ground truth labels. The target label distribution is then predicted with the correlation and the unlabelled target samples. Based on BBSE, non- negative constraints can be added to the optimization objective of BBSE as described in Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon, "Domain adaptation with conditional distribution matching and generalized label shift", Advances in Neural Information Processing Systems, 33:19276–19289, 2020. Similarly, Kamyar Azizzadenesheli,Anqi Liu, Fanny Yang, and Animashree Anandkumar in "Regularized learning for domain adaptation under label shifts", in International Conference on Learning Representations, 2018, develop Regularized Learning under Label Shift (RLLS) as a constrained BBSE model. BBSE was extended to a continuous learning setting with a target label distribution evolving with time in Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger, "Online adaptation to label distribution shift", Advances in Neural Information Processing Systems, 34:11340–11351, 2021.

[0010] Two shortcomings hinder application of existing label shift estimation approaches to real world problems. Firstly, existing models are typically usually tested using source domains with uniform label distribution, whereas the more realistic settings of long-tailed distributions are rarely analyzed. Secondly, the effectiveness of existing models on large-scale datasets with many classes is rarely studied, and instead most evaluate on small scale datasets like CIFAR10 or MNIST, see for example Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey J Gordon "Domain adaptation with conditional distribution matching and generalized label shift", Advances in Neural Information Processing Systems, 33:19276–19289, 2020. As a result, existing label shift estimation models often do not perform well on large-scale datasets with large numbers of classes or highly imbalanced label distributions.

[0011] If the target domain label distribution is given, label shift correction models can help align an existing classifier to a target domain. The approach can be performed either online during training or offline without retraining. Saerens et al. propose an offline label shift correction (LSC) method to adjust the decision boundary of the classifier and correct for label shift avoiding the need for retraining. On the other hand, BBSE and related methods also adopt an importance-weighted Empirical Risk Minimization (ERM) approach to retrain a new classifier for the target domain.

[0012] Class imbalance can lead to a decrease in classification performance if the source domain has an imbalanced label distribution. Recent works on class-imbalance usually aim to correct label shift for a Neural Network classifier with an imbalanced source label distribution and uniform target label distribution.

[0013] Re-Weighting and Re-Sampling can be used to re-weight the training loss or up-sample rare classes to create a class-balanced train set. The re-weighting approach is similar to importance-weighted ERM proposed by BBSE. However, these methods have been shown to overfit rare classes on highly imbalanced train sets.

[0014] Recently, several works introduce LSC to correct the classifier for the target domain. For example, to combine the original classifier and an LSC corrected classifier for class imbalance.

[0015] Other recent works on the class-imbalance problem propose more complicated mechanisms to obtain better performance. However, these models usually have less flexibility to adjust for different target label distributions. For example, Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma in "Learning imbalanced datasets with label- distribution-aware margin loss", Advances in neural information processing systems, 32, 2019, proposes a loss that aims to minimize the generalization error bound on a uniform test set. Hanyu Peng, Mingming Sun, and Ping Li, "Optimal transport for long-tailed recognition with learnable cost matrix", in International Conference on Learning Representations, 2022OT

[0028] proposes an optimal transport algorithm to optimize a classifier for a uniform test set. These models require retraining or algorithmic adjustment for a different target label distribution. Summary of the Present Invention

[0016] In one broad form, an aspect of the present invention seeks to provide a method of performing an analysis relating to objects in a target data set using a computational model, wherein the computational model is obtained by applying machine learning to training data including training objects associated with training labels that classify the training objects, the method including, in one or more processing devices: determining a prior indicative of an assumed label distribution in the target data set; using the computational model to classify a number of objects in the target data set and thereby determine target labels for the number of objects; calculating a predicted label distribution for the target data set based on the prior and the target labels; and, using the predicted label distribution to perform an analysis relating to the objects.

[0017] In one embodiment the method includes, in the one or more processing devices, determining the prior based on at least one of: a training label distribution for the training data; knowledge of an expected label distribution in the target data set; a user estimate of a label distribution; a provisional label distribution obtained by provisionally classifying one or more objects in the target data set; and, a previously calculated predicted label distribution.

[0018] In one embodiment the method includes, in the one or more processing devices, using a sampling model to select the number of objects.

[0019] In one embodiment the method includes, in the one or more processing devices, calculating a predicted label distribution by: classifying a number of objects in the target data set using the predicted label distribution to thereby determine target labels for the number of objects; and, calculating an updated predicted label distribution using the determined target labels.

[0020] In one embodiment the method includes, in the one or more processing devices, iteratively calculating a predicted label distribution.

[0021] In one embodiment the method includes, in the one or more processing devices, classifying the objects in the target data set using the computational model and the predicted target data set label distribution.

[0022] In one embodiment the method includes, in the one or more processing devices: using the predicted target data set label distribution to adjust classification parameters associated with the computational model; and, classifying the objects in the target data set using the adjusted classification parameters.

[0023] In one embodiment the method includes, in the one or more processing devices, generating a label shift correction model to align the computational model to the target data set.

[0024] In one embodiment the method includes, in the one or more processing devices, at least one of: using label shift correction to adjust a decision boundary associated with the computational model; and, retraining the computational model for the target domain.

[0025] In one embodiment the method includes, in the one or more processing devices, calculating the predicted label distribution using a Bayesian approach.

[0026] In one embodiment the method includes, in the one or more processing devices, determining the prior using at least one of: an Adaptive Prior Learning (APL) model; and, a Dirichlet distribution.

[0027] In one embodiment the method includes, in the one or more processing devices, using an expectation-maximization (EM) algorithm to calculate the predicted label distribution.

[0028] In one embodiment the method includes, in the one or more processing devices, using the expectation-maximization (EM) algorithm to obtain a Maximum A Posteriori (MAP) estimate of label distribution in the target data set.

[0029] In one embodiment the method includes, in the one or more processing devices: constructing a posterior estimate of the label distribution in the target data set; and using a posterior sampling model to select samples from the posterior.

[0030] In one embodiment the method includes, in the one or more processing devices, implementing the posterior sampling model using at least one of: a Markov Chain Monte Carlo (MCMC) method; and, Hamiltonian Monte-Carlo (HMC) method.

[0031] In one embodiment the computational model is a neural network classifier.

[0032] In one broad form, an aspect of the present invention seeks to provide a system for performing an analysis relating to objects in a target data set using a computational model, wherein the computational model is obtained by applying machine learning to training data including training objects associated with training labels that classify the training objects, the system including one or more processing devices configured to: determine a prior indicative of an assumed label distribution in the target data set; use the computational model to classify a number of objects in the target data set and thereby determine target labels for the number of objects; calculate a predicted label distribution for the target data set based on the prior and the target labels; and, use the predicted label distribution to perform an analysis relating to the objects.

[0033] It will be appreciated that the broad forms of the invention and their respective features can be used in conjunction and / or independently, and reference to separate broad forms is not intended to be limiting. Furthermore, it will be appreciated that features of the method can be performed using the system or apparatus and that features of the system or apparatus can be implemented using the method. Brief Description of the Drawings

[0034] Various examples and embodiments of the present invention will now be described with reference to the accompanying drawings, in which: -

[0035] Figure 1 is a flow chart of an example of a process for analysing a target data set;

[0036] Figure 2 is a schematic diagram of an example of a processing system;

[0037] Figure 3A is a flow chart of a first specific example of a process for analysing a target data set;

[0038] Figure 3B is a flow chart of a second specific example of a process for analysing a target data set;

[0039] Figure 4A is a schematic diagram illustrating the label shift problem;

[0040] Figure 4B is a schematic diagram of an example of a Bayesian approach to the label shift problem;

[0041] Figure 5A is a graph of an example of label shift estimation error analysis showing relative model misspecification errors;

[0042] Figure 5B is a graph of an example of label shift estimation error analysis showing relative sampling errors;

[0043] Figure 6 is a schematic diagram of an example structure of an Adaptive Prior Learning model;

[0044] Figure 7 is a density histogram showing example label distributions for different models; and,

[0045] Figure 8 is a graph showing example results of an ablation study. Detailed Description of the Preferred Embodiments

[0046] An example of a data analysis process using predictive label shift will now be described with reference to Figure 1.

[0047] For the purpose of illustration, it is assumed that the process is performed at least in part using one or more electronic processing devices forming part of one or more processing systems, such as computer systems, or the like. Whilst the system can use multiple processing devices, with processing performed by one or more of the devices, for the purpose of ease of illustration, the following examples will refer to a single device, but it will be appreciated that reference to a singular processing device should be understood to encompass multiple processing devices and vice versa, with processing being distributed between the devices as appropriate.

[0048] In this example, at step 100 a computational model is obtained by applying machine learning to training data including training objects associated with training labels that classify the training objects. The manner in which this is achieved will vary depending on the nature of the computational model, and the preferred implementation. As the training process is not part of the present invention, assuming a trained model is available, then this will not be further described for the purpose of the current example.

[0049] At step 110, the processing device determines a prior indicative of an assumed label distribution in the target data set. The prior can be determined using a number of different mechanisms, depending on the preferred implementation and the information available. For example, the prior could be estimated by a user based on some knowledge of the target data set, in which case this information would typically be received by the processing device, based on user inputs. Alternatively, the prior could be calculated, for example, based on sampling of the target data set, could be retrieved from stored information regarding the target data set, or could initially be assumed to be similar to the training data. Further examples will be describedin more detail below, but in any event it will be understood that the term "determining" encompasses receiving, calculating, retrieving, or otherwise obtaining.

[0050] At step 120, the processing device uses the computational model to classify a number of objects in the target data set and thereby determine target labels for the number of objects. The number of objects are typically a subset of randomly selected objects used to represent a sample of the entire target data set. This information can then be used to at step 130 to calculate a predicted label distribution for the target data set. Thus, classifying some objects in the target data set, reveals information which can be used to more accurately understand the likely distribution of labels, hence allowing the predicted label distribution to be calculated.

[0051] As will be described in more detail below, as an optional step, the predicted label distribution can be used to help refine the prior, allowing the process of steps 110 to 130 to be repeated with an improved prior to help further improve the prediction.

[0052] Once the predicted label distribution has been determined, this can be used to perform an analysis relating to the objects at step 140, for example allowing the computational model to be refined so that this is more effective when classifying objects in the target data set. Alternatively, this could be used for other forms of analysis, depending on the intended usage.

[0053] Accordingly, the above described process provides a mechanism for estimating the label shift between training and target data sets, which can in turn help inform and improve analysis of the target data set. This can be used, for example, to generate or modify the computational model, making the computational model more effective at classifying, or otherwise analysing objects in a target data set which has a different label distribution to the training data set. This enhances the ability of a computational model to be used with a wider variety of data sets, particularly when training can only be performed on a limited training data set, or a data set with a significantly different distribution to the target data set.

[0054] A number of further features will now be described.

[0055] In one example, the processing device determines the prior based on one or more of a training label distribution for the training data, knowledge of an expected label distribution in the target data set, a user estimate of a label distribution, a provisional label distributionobtained by provisionally classifying one or more objects in the target data set, a previously calculated predicted label distribution, or the like. In one particular example, which will be described in more detail below, the prior is a Dirichlet distribution, with the processing device calculating the prior using an Adaptive Prior Learning (APL) model. It will be appreciated that a range of different techniques for determining the prior can be used.

[0056] In one example, the processing device uses a sampling model to select the number of objects from the target data set, which are then classified. This can be used to ensure a representative sample of objects is selected, thereby minimising bias in the predicted calculation of the label distribution in the target data set.

[0057] As previously mentioned, the process can be repeated to refine the predicted label distribution. In this regard, when a predicted label distribution is calculated, this can be used to improve classification of the target objects, which in turn can affect the predicted label distribution. In one example, this can be repeated a single time, for example using the predicted label distribution as an improved prior. Thus, in one example, this is achieved by having the processing device classify a number of objects in the target data set using a current predicted label distribution to thereby determine target labels for the number of objects and then calculate an updated predicted label distribution using the determined target labels. As an alternative, this process could be repeated iteratively using the latest predicted label distribution on each iteration, with this being performed as required, for example, until a number of iterations exceeds a threshold, or until successive label prediction estimates begin to converge, for example when a difference between a predicted label distribution and an updated predicted label distribution is smaller than a threshold.

[0058] As previously mentioned, the processing device can classify objects in the target data set using the computational model and the predicted target data set label distribution. To achieve this, the processing device typically uses the predicted target data set label distribution to adjust classification parameters associated with the computational model and then classify the objects in the target data set using the adjusted classification parameters. The nature in which the model is adjusted will vary depending on the preferred implementation. For example, this could involve generating a label shift correction model to align the computational model to the target data set, with this in turn involving using label shift correction to adjust adecision boundary associated with the computational model and / or retraining the computational model for the target domain.

[0059] In one example, the process involves calculating the predicted label distribution using a Bayesian approach. In one particular example, an expectation-maximization (EM) algorithm is used to calculate the predicted label distribution, with this optionally being performed to obtain a Maximum A Posteriori (MAP) estimate of label distribution in the target data set. As part of this process, the processing device can construct a posterior estimate of the label distribution in the target data set and then use a posterior sampling model, such as a Markov Chain Monte Carlo (MCMC) method, or Hamiltonian Monte-Carlo (HMC) method, to select samples from the posterior.

[0060] In the above examples, the computational model is typically a neural network classifier. However, whilst these techniques are particularly suited for neural network classifiers, they could also be applied to other computational models, such as one or more decision trees, a random forest including multiple decision trees, a Bayes classifier, such as a Gaussian naïve Bayes classifier, a multi-layer perceptron neural network, or the like. It will also be appreciated that the techniques could be applied to other machine learning approaches and subsequent resulting computational models, and the above list is therefore intended to be illustrative rather than limiting.

[0061] The process is typically implemented using a processing system, and an example of a suitable processing system 200 is shown in Figure 2. In this example, the processing system 200 includes at least one microprocessor 201, a memory 202, an optional input / output device 203, such as a keyboard and / or display, and an external interface 204, interconnected via a bus 205 as shown. In this example the external interface 204 can be utilised for connecting the processing system 200 to peripheral devices, such as communications networks, databases 211, other storage devices, or the like. Although a single external interface 204 is shown, this is for the purpose of example only, and in practice multiple interfaces using various methods (eg. Ethernet, serial, USB, wireless or the like) may be provided.

[0062] In use, the microprocessor 201 executes instructions in the form of applications software stored in the memory 202 to allow the required processes to be performed. Theapplications software may include one or more software modules, and may be executed in a suitable execution environment, such as an operating system environment, or the like.

[0063] Accordingly, it will be appreciated that the processing system 200 may be formed from any suitable processing system, such as a suitably programmed client device, PC, web server, network server, or the like. In one particular example, the processing system 200 is a standard processing system such as an Intel Architecture based processing system, which executes software applications stored on non-volatile (e.g., hard disk) storage, although this is not essential. However, it will also be understood that the processing system could be any electronic processing device such as a microprocessor, microchip processor, logic gate configuration, firmware optionally associated with implementing logic such as an FPGA (Field Programmable Gate Array), or any other electronic device, system or arrangement.

[0064] A first specific example of the process will now be described with reference to Figure 3A.

[0065] In this example, at step 300 a computational model is trained using the training data. At step 305, the processing system 200 determines a prior indicative of an assumed label distribution in the target data set, with this typically being performed based on an initial prior, for example by basing this on the label distribution in the training data set.

[0066] At step 310, the processing system 200 classifies a number of items, before constructing a Bayesian posterior of the target label distribution at step 315, using the prior and the target data set. This is then sampled by the processing system 200 at step 320, for example using a posterior sampling model, such as a Markov Chain Monte Carlo (MCMC) method to draw independent and identically distributed (i.i.d) samples from the posterior. The processing system 200 then uses an expectation maximisation (EM) algorithm to calculate a predicted target label distribution at step 325.

[0067] Steps 305 to 325 can optionally be repeated using an improved prior based on the predicted label distribution, which can then be used to further refine the predicted label distribution.

[0068] Once the predicted label distribution has been determined, this can be used to update the computational model, allowing this to be refined so that this is more effective when classifying objects in the target data set, for example by modifying a decision boundary associated with the model, or by model retraining. Additionally and / or alternatively, the predicted label distribution could be utilised in performing other target data set analysis, as desired.

[0069] A second specific example of the process will now be described with reference to Figure 3B.

[0070] In this example, at step 350 a computational model is trained using the training data. At step 355, the processing system 200 determines a prior indicative of an assumed label distribution in the target data set, with this typically being performed based on an initial prior, for example by basing this on the label distribution in the training data set.

[0071] At step 360, the processing system 200 classifies a number of items, before implementing an expectation maximisation algorithm. Specifically, at step 365, the processing system 200 initialises a predicted label distribution, before adjusting a classification result at step 370. The processing device then updates the predicted label distribution at step 375 using the updated classification result and the prior, before determining if the iterative process has been completed at step 380. In this regard, this is typically performed for either a set number of iterations, or until the predicted target label distribution converges over successive iterations.

[0072] Once the predicted label distribution has been determined, this can be used to generate the predicted label distribution at step 385.

[0073] Steps 355 to 385 can optionally be repeated using an improved prior based on the predicted label distribution, which can then be used to further refine the predicted label distribution.

[0074] Once the predicted label distribution has been determined, this can be used to update the computational model, allowing this to be refined so that this is more effective when classifying objects in the target data set, for example by modifying a decision boundary associated with the model, or by model retraining. Additionally and / or alternatively, thepredicted label distribution could be utilised in performing other target data set analysis, as desired.

[0075] A specific example, including experimental results will now be described. Introduction

[0076] To tackle the label shift problem, the approach described herein performs label shift estimation model using a Bayesian framework. Specifically, in one example, this involves constructing a Bayesian posterior of the target label distribution parameters given data and a prior. To achieve this, an expectation-maximization (EM) algorithm can be used to obtain a Maximum A Posteriori (MAP) estimate of the target label distribution. An Adaptive Prior Learning (APL) model can be used to adaptively choose the prior parameters given data; and a posterior sampling model that uses a Markov Chain Monte Carlo (MCMC) method to draw independent and identically distributed (i.i.d) samples from the posterior.

[0077] The label shift estimation problem is represented schematically in Figure 4A, whilst the proposed Bayesian approach is shown in Figure 4B. Specifically, the approach constructs the analytical Bayesian posterior of target label distribution given data and a prior. Then based on a proposed APL model that adaptively learns prior parameters given data, a MAPLS algorithm is derived to obtain a MAP estimate of π and propose a posterior sampling model that uses MCMC to obtain samples from the posterior.

[0078] Experiments were conducted to train and test set pairs with different label distributions. In contrast to previous methods that mainly focus on MNIST and CIFAR10, these use the CIFAR100, ImageNet, Places datasets and Long-Tailed versions of each dataset.

[0079] For target label distributions, as well as evaluating under previous label shift estimation settings with Dirichlet shift, Long-Tailed benchmark test set shifts are used, as per Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang, "Disentangling label distribution for long-tailed visual recognition." In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 6626–6636, 2021.

[0080] Experimental results show that the approach consistently outperforms existing approaches, particularly, obtaining better accuracy when the train set is highly imbalanced. These results demonstrate the applicability of the model to real world label shift tasks.

[0081] Thus, the approach can use a label shift model under a Bayesian framework to estimate and correct label shift without retraining the classifier, which in one example involves constructing a posterior of the target label distribution given data and a prior.

[0082] An EM algorithm can be used that computes the maxima of the posterior (MAP estimate) by minimising a strictly convex objective. An Adaptive Prior Learning (APL) model can also be used to determine the parameters of the prior adaptively given data. A posterior sampling model can be used to estimate and correct label shift based on i.i.d samples drawn from the posterior via MCMC.

[0083] Experiments show that the approach consistently outperforms previous label shift estimation models in a variety of label shift settings on CIFAR100, ImageNet, Places and the Long-Tailed version of each dataset. Problem Setup

[0084] Denoting the input image space aswhere H, W, C are the height, width and channels of the image, and the corresponding label space as = {1, 2, ...K}, where K is the number of classes. The random variable of image and label pairs on source and target domains are denoted as (Xs, Ys) ∼ Ps and (Xt, Yt) ∼ Pt respectively.

[0085] Under the label shift setting, it is assumed that the source and target domain have adifferent label distribution ℙ(^^^^ = ^^) ≠ ℙ(^^^^ = ^^). The conditional probability of an imagegiven its label is identical is given by: ℙ(^^^^ = ^^|^^^^ = ^^) = ℙ(^^^^ = ^^|^^^^ = ^^) (1)

[0086] Three main problems are usually discussed in label shift, namely detection, estimation and correction. In the following the focus is on label shift estimation. To tackle label shiftestimation for classification, a blackbox classifier f: → ∆^^−1 is usually assumed to beavailable, which is sometimes required to perform well on the source domain. Hereis the space of a K dimensional probability simplex.

[0087] Note that the source label Ys~ Cat(K, c) and target label Yt~ Cat(K, π) each follow a categorical distribution over K classes. c, π are parameters of the two distributions respectively, with c, π ∈ ∆K−1. Estimating the target label distribution ℙ(Yt = ·) = π is equivalent to estimating the parameters π of the categorical distribution Cat(K, π).

[0088] Table 1 below shows the label shift problem setup. The available data is: labelled samples from source domain Ps, a classifier f, and unlabelled target domain data ^^. Label shift detection tests if label shift occurs, estimation estimates the target label distribution and correction adopt the classifier for the target domain. Table 1MLLS Label Shift Estimation

[0089] Saerens et al. derive MLLS by assuming the classifier reflects the true conditionalprobability ^^(^^)^^ = ℙ(^^^^ = ^^|^^^^ = ^^), ^^ = 1, 2, … ^^. The source domain label distributionℙ(^^^^ = ^^) = ^^ can be estimated using labelled data. With unlabelled target domain data ^^,MLLS estimates the target label distribution ℙ(^^^^ = ^^) = ^^ by maximizing the log likelihoodusing the EM algorithm:

[0090] In the algorithm, the parameter π is first initialized as π(0). The EM algorithm then proceeds by repeatedly applying two alternating steps, the E-Step and the M-Step.

[0091] The E-Step evaluates the conditional probability g(xi; π(t))j := ℙ(Yt= j|Xt= xi, π(t)) under label shift with:

[0092] Equality has been proved by Saeren et al. under label shift along with the assumptions of MLLS.

[0093] The M-Step calculates for the next iteration via:

[0094] Given π(0), the iterative procedure of the EM algorithm is repeated until numericalconvergence to obtain the ML of the target label distribution ℙ(^^^^ = ^^) = ^^iOffline Label Shift Correction

[0095] For classifier, f, trained on the source domain Ps, Saerens et al. proposed to construct anew target domain classifier g : → ∆^^−1 to correct for label shift:

[0096] where cj, πj, j = 1, 2...K are parameters of source and target label distributions respectively. The advantage of this model is that adjustment does not require retraining of f.Proposed Method

[0097] A Bayesian approach is used for the label shift estimation problem. By employing a prior distribution over target label distribution ℙ(Yt= ·), the posterior of π is obtained given available data ^^. Based on the posterior, an EM algorithm is derived to obtain the Maximum A Posteriori (MAP) estimate of π. To utilize the information of the entire posterior, a Hamiltonian Monte-Carlo (HMC) method can be used to obtain i.i.d samples from the posterior.

[0098] The categorical distribution Yt ∼ Cat(K, π) requires that the prior distribution over π is supported on ∆K-1. K dimensional Dirichlet distributions satisfy this constraint, and are often used as a prior over parameters of categorical distributions.

[0099] Therefore, a Dirichlet prior can be employed over the parameters π ∼ Dir(K, α) of thetarget label distribution Cat(K,where ^^ ∈ ℝ>^^1. With the Dirichlet prior as ℙ (π|α) and unlabelled target domain samples ^^, the posterior of π given X and α can be written as:

[0100] where ^^ = ∫(ℙ (^^|^^′)ℙ(^^′|^^))^^^^′ is a constant w.r.t π.

[0101] The marginal distribution ℙ(^^^^ = ^^|π) can be rewritten as a combination of knownexpressions. Given the source domain labelled data, the source domain label distribution ℙ(Ys= j) = cjin Yt∼ Cat(K, π) which is also a categorical distribution can be estimated.ℙ(^^^^ = ^^|^^^^ = ^^^^)| on the source domain can be modelled by the blackbox classifier f, and thetarget label distribution is ℙ(^^^^ = ^^|π).

[0102] Formally:

[0103] where ci > 0, i = 1, 2...K because each class has non-zero sample frequency on the source domain.

[0104] In equation (7), it is assumed the classifier f is well-specified to model ℙ(Ys= ·| Xs= x). It is discussed further below when this may not be the case in practice. With equation (7) available, utilizing Bayes rule, this the posterior equation (6) can be rewritten as:

[0105] Note that ℙ(Xt = xi) and Z are constants w.r.t π and ℙ (π|α) is the Dirichlet prior. Therefore the analytical expression for the un-normalized posterior ℙ(π| ^^, α) can be obtained from equation (8). MAP estimate

[0106] An EM algorithm is derived to obtain MAP estimate of π. By definition, any MAP estimate π∗minimizes the negative log posterior:

[0107] The optimization problem in equation (9) is strictly convex in π and an EM algorithm can be used to find π∗, referred to as: Maximum a Posteriori Label Shift (MAPLS).

[0108] Proposition 1 Under label shift defined in equation (1), suppose equation (7) holds forall (x, i) ∈ × . Let π ∼ Dir(K, α) with ^^ ∈ ℝ>^^1. Then in equation (9), the objective is strictly convex in π, π∗is unique and EM Algorithm 1 below converges to π∗.

[0109] Algorithm1 can be seen as a generalization of MLLS. In the M-Step, equation (11) can be rewritten as:

[0110] As the algorithm degenerates to MLLS. As λ→0+, the MAP estimate will converge to the Dirichlet prior Dir(K, α). In this manner, λ can be seen as confidence in the label distribution estimation.

[0111] The choice of α and corresponding λ affect the MAP estimate π∗. In practice, it is important to determine an appropriate α and A to give a good MAP estimate π∗for the target label distribution ℙ(Yt = ·).

[0112] After obtaining π∗, equation (5) can be used to correct the source domain classifier f to the target domain under label shift.

[0113] The Dirichlet prior possesses K parameters in α = [α1, ..., αK ] ∈ ℝ>^^1. When no information about the target domain label distribution is available, set αj= α0. This has the advantage of reducing the number of parameters to be chosen, at the cost of limiting expressivity.

[0114] The Dirichlet prior satisfies π ∼ Dir(K, α01). Then the M-Step of the MAPLS algorithm in the form of equation (12) can be further simplified as:

[0115] where λ = N / (N + K(α0−1)) also has a simpler form.

[0116] The MAPLS algorithm with λ→0+will converge to a uniform categorical distribution with π = 1 / K in Yt∼ Cat(K, π).

[0117] Note that now α = α01 is fully determined by λ, parameter α0 can be determined in the prior by selecting a value for λ. In this case, 1-λ represents the strength of regularization in the MAP estimation procedure. Adaptive Prior Learning Model

[0118] In the MAPLS algorithm1, the prior parameter α should be determined before the estimation of π. In this work, based on the analysis of the possible estimation error, an Adaptive Prior Learning (APL) model is provided to adaptively learn α given available data. The model is inspired by an empirical Bayesian approach.

[0119] Intuitively, two factors can induce estimation error in the posterior. Firstly, a classifier f(x) is used to model ground truth ℙ(Ys= ·|Xs= x) in equation (7), when the classifier fails to represent the ground truth, the model is subject to misspecification error. Secondly, even if equation (7) is satisfied, the MAPLS model will have an associated sampling error due to using a finite number of samples, like other models.

[0120] In the APL model, a heuristic is used to loosely evaluate the magnitude of model misspecification error. The sampling error of the label shift estimation model can be magnified with large label shift between the target and source domains. Therefore, the APL model can include a heuristic to mitigate sampling error.

[0121] Practically, MAPLS is first run with λ = 1 to obtain an initial MLE of the target label distribution πMLE. Then the APL model quantifies the two estimation errors based on the three KL-divergences below:

[0122] where 1 / K denotes a uniform label distribution and c is the parameter of source label distribution. S, T, U represents source, target and uniform label distribution respectively.

[0123] Example label shift estimation error analysis is shown in Figures 5A and 5B. In this example, the Mean Square Error increases when: (1) the model is miss-specified, i.e. ℙ(Ys= ·|Xs= x) = f(x) is not satisfied, as shown in Figure 5A; (2) the sampling error gets magnified when source and target domains have large label shift, as shown in Figure 5B. The MAPLS with fixed prior (λ = 0.9 in equation (14)) can reduce both errors compared with MLLS. Our MAPLS-APL model with prior parameters learned given data can further reduce MSE and outperform BBSE under large label shift, as shown in Figure 5B.

[0124] An example structure of an Adaptive Prior Learning model is shown in Figure 6, with parameters in the prior being adaptively determined by the available data with a heuristic based on dTU , dTS , dSU as defined in equation (15).

[0125] A Neural Network classifier f trained on a source domain usually has poor performance when the source domain has a highly imbalanced label distribution (dSU ≫ 0). In this case, the classifier is more likely to be subject to model misspecification error when estimating label shift. Hence prior contribution in equation (14) is increased with a higher dSU.

[0126] Two approaches can be used to mitigate the problem that sampling error tends to increase given large label shift. Firstly, using dTS to approximate the amount of shift between target and source label distribution. A higher dTSimplies larger label shift, which will lead to more severe sampling error. Thus, the APL model should increase the contribution of the prior in equation (14) with a higher dTS.

[0127] Secondly, whenis close to a uniform label distribution 1 / K, the prior contribution can be increased so that equation (14) can push the estimate more towards 1 / K.

[0128] By defining a normalization function F (x) = x / (1 + x), the APL model determines λ via:

[0129] where γ = 1-F(b·dSU) takes into account the model misspecification error and dTU , dTS evaluates the sampling error. Here a ∈ [0, 1] represents the trade-off between the two approaches to reduce sampling error and b ∈ [0, 1] represents the strength of misspecification error. Sampling from the Bayesian posterior

[0130] Apart from the point estimate π∗, Bayesian analysis is also used to utilize the entire posterior ℙ(π|^^, α) as the estimated target label distribution. In this work, a Markov-Chain Monte-Carlo (MCMC) method is used to obtain i.i.d samples of the posterior. The samples can then be used for downstream label shift correction tasks.

[0131] Based on equation (8), ℙ(π|^^, α) can be written as:

[0132] Where Z contains ∫(ℙ (^^|^^′)ℙ(^^′|^^))^^^^′^^ and ℙ (Xt = xi), which are constant w.r.t πand are usually intractable.

[0133] To avoid evaluation of Z, the MCMC method is used to obtain samples of the posterior. Because the Hamiltonian Monte-Carlo (HMC) sampler can be more efficient than otherMCMC methods in high dimensional space, the HMC is used to obtain i.i.d samples of the posterior:

[0134] where α is determined by the APL model.

[0135] After collecting Π, each πiis used as a point of π for the down stream task. Forexample, for the label shift correction problem, every ∈ Π is used to correct the source domain classifier f(x) to the target domain gi(x) under label shift based on equation (5). The target domain average SoftMax classifier can then be constructed as:

[0136] With samples of the posterior, the uncertainty of the estimated π given data can also be analyzed. Comparing with Algorithm1, this approach utilizes the entire posterior at the cost of computation resources.

[0137] In this regard, MCMC can be computationally expensive in high dimensional space, because sufficient warm up steps are required if the Markov chain is initialized randomly in value space. Fortunately, since the posterior in the model is strictly log concave (Proposition1) with known maximal point π∗ obtained by MAPLS, the Markov chain can be initialised at π∗and the HMC sampler can then collect i.i.d samples more efficiently without warm up steps. Estimation of Source Label Distribution 138] Given source domain data ^^^^^^^[0 {^^^, ^^^^^^}^^=1 and blackbox classifier f, there are two known methods to estimate the source domain label distribution ℙ(Ys = ·) = c. MLE is a standard method to estimate c with source domain ground truth labels ^^^^^^. On the other hand, when classifier f is calibrated on the source domain, it is also proposed to estimate c with source domain images

[0139] In this example, both approaches are adopted to estimate c. The MLE approach is referred to as the “hard” method and the approach of equation (20) as the “soft” method. Overall Method

[0140] Estimation and correction of label shift can be achieved as follows:

[0141] The model that uses the MAP estimate is referred to as MAPLS-APL and the model that uses posterior sampling PSLS-APL, where the “APL” indicates that parameter α in the prior distribution is learned with the APL model. Experiments Datasets

[0142] The model was evaluated on the CIFAR100, ImageNet 2012 and Places2 datasets. Following common use in Long-Tailed research, LongTail versions of ImageNet, Places and CIFAR100 were also used. The models were tested on test sets with Dirichlet shift proposed by previous label shift estimation models.

[0143] Dirichlet Shift generates a random test set label distribution from a K dimensional Dirichlet distribution. The ordered Long-Tailed shifted test set used in LADE was also adopted, which has the same or inverse order of the Long-Tailed distributed train set. This was further extended to a shuffled Long-Tailed test set, where the test set still has a LongTailed label distribution but with random class order.

[0144] Table 2 below shows label shift experiment settings. R is referred to as the imbalance ratio — the ratio of maximum and minimum sample number per class respectively in test set. α is the parameter of the Dirichlet distribution. Table 2Model Setup

[0145] Both the MAPLS / MAPLS-APL algorithm and previous MLLS algorithm are initialized with π(0)= c and run for 100 epochs to ensure convergence. Because π∗is unique as proved in Proposition1, the MAPLS is guaranteed to converge to a single MAP estimate. In the APL model, empirically values were set at a = 0.9, b = 0.5 in (16) for all the label shift settings in all datasets. For the PSLS model, a HMC sampler called a No-U-Turn Sampler, described in Matthew D Hoffman, Andrew Gelman, et al, "The no-u- turn sampler: adaptively setting path lengths in hamiltonian monte carlo", J. Mach. Learn. Res., 15(1):1593–1623, 2014 was used to collect 5000 samples from the posterior.

[0146] Neural Network classifiers were implemented using PyTorch, described in Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito,Zeming Lin, Al- ban Desmaison, Luca Antiga, and Adam Lerer, "Automatic differentiation in pytorch". 2017. The ResNet32 classifier, described in Yerlan Idelbayev, "Proper ResNet implementation for CIFAR10 / CIFAR100 in PyTorch", was used for CIFAR100 and every CIFAR100-LT dataset. Pretrained ResNet50 and pre-trained Resnet152 were used for ImageNet and Places datasets respectively. A ResNet50 and ResNet152 were trained for ImageNet-LT and Places-LT datasets respectively. Evaluation Metrics

[0147] Known methods were used to evaluate label shift estimation(^^ − ^̂^)2 / ^^, where wi = ℙ(Yt = i) / ℙ(Ys = i), i = 1, 2...K is the target over source labeldistribution ratio. w is the ground truth ratio estimated by the source and target labels. ^̂^ is the predicted ratio with ℙ(Yt = · ) estimated by each model. Top1 accuracy is provided for different label shift estimation models with LSC (5) on all datasets. State-of-the-art (SOTA) Comparison

[0148] The performance of the current approach and several state-of-the-art (SOTA) label shift estimation methods, including MLLS, BBSE and RLLS, are compared. In BBSE and RLLS, there are also “soft” and “hard” versions of each model. Performance of these models was evaluated with previously available implementations.

[0149] In the setting of large-scale datasets, methods that require retraining the classifier on the source domain will suffer from high computational cost and so these are not considered.

[0150] The SOTA comparison of the MAPLS-APL model in terms of (^^− are shownin Tables 3 and 4. It should be noted that unlike SOTA models that obtainpoint estimate of π, the PSLS-APL model obtains samples of π from the posterior instead. Thus only Top1 Accuracy is compared for the PSLS-APL model instead of both metrics.

[0151] Table 3 shows a SOTA comparison summary of estimation error. The percentage of settings that the MAPLS-APL model outperforms SOTA models (MLLS, BBSE, RLLS) interms of (^^ − ^̂^)2 / ^^.Table 3

[0152] Table 4 shows a SOTA comparison summary of Top1 Accuracy. The percentage of settings that the MAPLS-APL model outperforms SOTA models and the baseline classifier in terms of accuracy. Table 4

[0153] Table 5 shows performance of (^^ − ^̂^)2 / ^^ (↓) on ImageNet and ImageNet-LT datasets,with shuffled Long-Tailed test set that have an imbalance ratio {50, 10, 5, 2} and Dirichlet test set that have α = {1, 10} and total test sample number {12500, 25000} in each setting. Best performances are in bold face and second best are in blue. The PSLS-APL model is only suitable for Top1 Accuracy comparison. Table 5

[0154] Table 6 shows the performance of Top1 Accuracy (↑) on ImageNet-LT and Place-LT dataset, with Ordered Long-Tailed test set that have imbalance ratio R = {25, 10, 5, 2}. Best performances are in bold face and second best are in blue. Table 6

[0155] As shown in Table 3, the MAPLS-APL model outperforms SOTA models in at least 50% of the label shift and dataset settings.

[0156] As an example on ImageNet in Table 5, the model outperforms other models by a large margin for the highly imbalanced train set ImageNet-LT.

[0157] As shown in Table 4, in terms of Top1 Accuracy, the MAPLS-APL model outperforms SOTA models and baseline in at least 50% of the settings. As an example in Table 6, the MAPLS-APL and PSLS-APL model have similar performance and outperform SOTA models in most settings.

[0158] By analyzing the performance in Tables 5 and 6, one obvious advantage of the model is its robustness to the source label distribution. When source domains have highly imbalanced label distributions (e.g. ImageNet-LT, Places-LT), the label shift estimation performance of the model stays relatively stable while previous models degrade significantly. Ablation Study

[0159] Density histogram of posterior samples Π collected by the PSLS-APL model are shown in Figure 7, with single value of π estimated by other models as well. Specifically, Figure 7 illustrates the label shift estimation result (π) on the Long-Tailed CIFAR100 dataset with Ordered Long-Tailed test set. The PSLS-APL model uses HMC to obtain samples of theposterior ℙ(π| ^^, α) (posterior sample density histogram plot as a bar heatmap), which fit nicely with the ground truth.

[0160] The posterior ℙ(π| ^^, α) fits well with the ground truth and is able to provide a sense of uncertainty of our estimation. The estimation stability of the model was also analysed during the training of classifier f on the source domain. Specifically, the performance of each label shift estimation model during the training of a Neural Network classifier was monitored on the Long-Tailed CIFAR100 dataset. The test sets have Ordered Long-Tailed label distribution.

[0161] Figure 8 shows results of an ablation study on stability of the MAPLS-APL model. On the Long-Tailed CIFAR100 dataset with Ordered LT test set, our model is stable during the training of the classifier and performs better than SOTA methods. In particular, the performance of BBSE, MLLS and the model improves during the training of the classifier. This observation suggests that label shift estimation performance of these models could be further improved with a better classifier. The MAPLS (λ = 0.9) and MAPLS-APL model performs better and stable in the last 50 epochs. Discussion and Conclusion

[0162] In this work, label shift estimation methods MAPLS-APL and PSLS-APL are developed under a Bayesian framework that are applicable to large-scale datasets and robust to highly imbalanced source label distributions. In the MAPLS model, an EM algorithm is derived to obtain the MAP estimate of the target label distribution and an Adaptive Prior Learning model is used to adaptively adjust the prior parameter.

[0163] In a PSLS model, HMC is used to sample from the strictly log-concave posterior ℙ(π| ^^, α).

[0164] Unlike previous benchmark evaluations, the experimental settings additionally cover a variety of large-scale datasets (ImageNet, Places) with highly imbalanced label distributions, which provide a more realistic evaluation of SOTA methods. Experiments on these datasets have demonstrated the effectiveness of our model and its potential to be applied in real world label shift problems.

[0165] Throughout this specification and claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” or “comprising”, will be understood to imply the inclusion of a stated integer or group of integers or steps but not the exclusion of any other integer or group of integers. As used herein and unless otherwise stated, the term "approximately" means ±20%.

[0166] Persons skilled in the art will appreciate that numerous variations and modifications will become apparent. All such variations and modifications which become apparent to persons skilled in the art, should be considered to fall within the spirit and scope that the invention broadly appearing before described.

Claims

CLAIMS 1) A method of performing an analysis relating to objects in a target data set using a computational model, wherein the computational model is obtained by applying machine learning to training data including training objects associated with training labels that classify the training objects, the method including, in one or more processing devices: a) determining a prior indicative of an assumed label distribution in the target data set; b) using the computational model to classify a number of objects in the target data set and thereby determine target labels for the number of objects; c) calculating a predicted label distribution for the target data set based on the prior and the target labels; and, d) using the predicted label distribution to perform an analysis relating to the objects. 2) A method according to claim 1, wherein the method includes, in the one or more processing devices, determining the prior based on at least one of: a) a training label distribution for the training data; b) knowledge of an expected label distribution in the target data set; c) a user estimate of a label distribution; d) a provisional label distribution obtained by provisionally classifying one or more objects in the target data set; and, e) a previously calculated predicted label distribution. 3) A method according to claim 1 or claim 2, wherein the method includes, in the one or more processing devices, using a sampling model to select the number of objects. 4) A method according to any one of the claims 1 to 3, wherein the method includes, in the one or more processing devices, calculating a predicted label distribution by: a) classifying a number of objects in the target data set using the predicted label distribution to thereby determine target labels for the number of objects; and, b) calculating an updated predicted label distribution using the determined target labels. 5) A method according to claim 4, wherein the method includes, in the one or more processing devices, iteratively calculating a predicted label distribution.6) A method according to any one of the claims 1 to 5, wherein the method includes, in the one or more processing devices, classifying the objects in the target data set using the computational model and the predicted target data set label distribution. 7) A method according to claim 6, wherein the method includes, in the one or more processing devices: a) using the predicted target data set label distribution to adjust classification parameters associated with the computational model; and, b) classifying the objects in the target data set using the adjusted classification parameters. 8) A method according to claim 7, wherein the method includes, in the one or more processing devices, generating a label shift correction model to align the computational model to the target data set. 9) A method according to any one of the claims 6 to 8, wherein the method includes, in the one or more processing devices, at least one of: a) using label shift correction to adjust a decision boundary associated with the computational model; and, b) retraining the computational model for the target domain. 10) A method according to any one of the claims 1 to 9, wherein the method includes, in the one or more processing devices, calculating the predicted label distribution using a Bayesian approach. 11) A method according to any one of the claims 1 to 10, wherein the method includes, in the one or more processing devices, determining the prior using at least one of: a) an Adaptive Prior Learning (APL) model; and, b) a Dirichlet distribution. 12) A method according to any one of the claims 1 to 11, wherein the method includes, in the one or more processing devices, using an expectation-maximization (EM) algorithm to calculate the predicted label distribution. 13) A method according to claim 12, wherein the method includes, in the one or more processing devices, using the expectation-maximization (EM) algorithm to obtain a Maximum A Posteriori (MAP) estimate of label distribution in the target data set. 14) A method according to claim 12 or claim 13, wherein the method includes, in the one or more processing devices:a) constructing a posterior estimate of the label distribution in the target data set; and b) using a posterior sampling model to select samples from the posterior. 15) A method according to claim 14, wherein the method includes, in the one or more processing devices, implementing the posterior sampling model using at least one of: a) a Markov Chain Monte Carlo (MCMC) method; and, b) Hamiltonian Monte-Carlo (HMC) method. 16) A method according to any one of the claims 1 to 15, wherein the computational model is a neural network classifier. 17) A system for performing an analysis relating to objects in a target data set using a computational model, wherein the computational model is obtained by applying machine learning to training data including training objects associated with training labels that classify the training objects, the system including one or more processing devices configured to perform the method of any one of claims 1 to 16. 18) A non-transitory computer-readable medium comprising computer-executable instructions, that when executed perform the method of any one of claims 1 to 16.

Images