Methods and apparatuses for training and utilising one or more of a plurality of cluster ML models

Abstract

Embodiments described herein relate to methods and apparatuses for training a plurality of cluster machine learning models and utilizing one or more of a plurality of cluster machine learning, ML, models to perform a task. A computer implemented method of training a plurality of cluster machine learning models to perform a task comprises applying a clustering process to an initial set of training samples to generate a plurality of final clusters of training samples; for each final cluster of training samples, training, utilizing the final cluster of training samples, a cluster ML model to perform the task; and initiating use of at least one of the plurality of cluster ML models to perform the task in response to an input query from a client entity.

Description

[0001] METHODS AND APPARATUSES FOR TRAINING AND UTILISING ONE OR MORE OF A PLURALITY OF CLUSTER ML MODELS

[0002] TECHNICAL FIELD

[0003] Embodiments described herein relate to methods and apparatuses for training and utilizing one or more of a plurality of cluster ML models.

[0004] BACKGROUND

[0005] Machine learning (ML) and Artificial Intelligence (Al) are increasingly being deployed to enable automation of all kinds of processes in a wide range of applications. For many deployments the ML / AI -model is accessible via a query-response Application Processing Interface (API). It will be appreciated that models accessible via a query-response may be subject to a threat called model extraction.

[0006] Model extraction refers to a process in which an adversary tries to replicate a ML / AI -model by learning how the model responds to different queries. The known types of attacks may be divided into two strategies (see for example, Matthew Jagielski et al. (2020) High Accuracy and High Fidelity Extraction of Neural Networks. In 29th {USENIX} Security Symposium (pp. 1345-1362)):

[0007] 1) Direct extraction - In which the adversary tries to directly extract model parameters by posing queries that reveal model boundaries (see for example, Tram.r, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016). Stealing machine learning models via prediction apis. In 25th {USENIX} Security Symposium ({USENIX} Security 16) (pp. 601 -618). This method of model extraction may produce a copy of the model with high fidelity to the victim model .

[0008] 2) Learning based - in which the adversary builds a query-response set that can overlap the training distribution enough to be able to build an equivalent model. This model may not always agree with the victim model but can achieve similar performance on the task addressed by the victim model. This attack may succeed even though no real training samples are used, only that the attack distribution is similar enough (see, for example, Orekondy, Tribhuvanesh, Bernt Schiele, and Mario Fritz. "Knockoff nets: Stealing functionality of black-box models." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019). Existing defenses against model extraction focus on detecting abnormal queries (see, for example, Juuti, Mika, et al. "PRADA: protecting against DNN model stealing attacks." 2019 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2019.), estimating extraction progress by building a surrogate model (see for example, Kesarwani, Manish, et al. "Model extraction warning in mlaas paradigm." Proceedings of the 34th Annual Computer Security Applications Conference. 2018.), limiting the leak of model information by reducing the input features (see for example, Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. Enhancing Robustness of Machine Learning Systems via Data Transformations.arXiv preprint 1704.02654, 2017) or truncating the model response (see for example, Tram.r, F., Zhang, F., Juels, A., Reiter, M. K., & Ristenpart, T. (2016). Stealing machine learning models via prediction apis. In 25th {USENIX} Security Symposium ({USENIX} Security 16) (pp. 601-618)).

[0009] There are several limitations to existing defenses against model extraction, for example:

[0010] 1 . Estimating extraction progress by building new models for several users can be costly; and

[0011] 2. Exclusively detecting out-of-distribution samples will not work if adversary has access to unlabeled data from the same distribution (collection settings) as the training data.

[0012] SUMMARY

[0013] According to some embodiments there is provided a computer implemented method for utilizing one or more of a plurality of cluster machine learning, ML, models to perform a task, wherein each of the plurality of cluster ML models is configured to perform the task. The method comprises receiving an input query from a client entity; inputting the input query into a classifying ML model to determine a subset of cluster ML models from the plurality of cluster ML models that the input query is associated with; and inputting the input query into at least one of the subset of cluster ML models to generate at least one first output. The method also comprises providing an output response to the client entity, wherein the output response is derived from the at least one first output.

[0014] According to some embodiments there is provided a computer implemented method for training a plurality of cluster machine learning models to perform a task. The method comprises applying a clustering process to an initial set of training samples to generate a plurality of final clusters of training samples; for each final cluster of training samples, training, utilizing the final cluster of training samples, a cluster ML model to perform the task; and initiating use of at least one of the plurality of cluster ML models to perform the task in response to an input query from a client entity. According to some embodiments there is provided a computer program, comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out any of the methods as described above.

[0015] According to some embodiments there is provided a carrier containing the computer program as described above, wherein the carrier comprises one of an electronic signal, optical signal, radio signal or computer readable storage medium.

[0016] According to some embodiments there is provided a computer program product comprising non transitory computer readable media having stored thereon a computer program as described above.

[0017] According to some embodiments there is provided a computer-readable medium comprising instructions that, when executed on at least one processor, cause the at least one processor to perform any of the methods described above.

[0018] According to some embodiments there is provided an apparatus for utilizing one or more of a plurality of cluster machine learning, ML, models to perform a task. Each of the plurality of cluster ML models is configured to perform the task. The apparatus comprises processing circuitry and a memory, the memory containing instructions executable by the processing circuitry whereby the apparatus is operable to: receive an input query from a client entity; input the input query into a classifying ML model to determine a subset of cluster ML models from the plurality of cluster ML models that the input query is associated with; input the input query into at least one of the subset of cluster ML models to generate at least one first output; and provide an output response to the client entity, wherein the output response is derived from the at least one first output.

[0019] According to some embodiments there is provided an apparatus for training a plurality of cluster machine learning models to perform a task. The apparatus comprises processing circuitry and a memory, the memory containing instructions executable by the processing circuitry whereby the apparatus is operable to: apply a clustering process to an initial set of training samples to generate a plurality of final clusters of training samples; for each final cluster of training samples, training, utilizing the final cluster of training samples, a cluster ML model to perform the task; and initiating use of at least one of the plurality of cluster ML models to perform the task in response to an input query from a client entity.

[0020] Embodiments described herein provide a method to that may automatically obtain a coarse granularity for model extraction progress by registering the cluster ML models used to respond to queries. By building multiple cluster ML models capable of performing the task instead of relying on one model performing the task, embodiments described herein provide a mitigation strategy to limit the possibility of a model extraction attempt by, for example, restricting a client entity to only be able to receive responses from a subset of the cluster ML models - thereby limiting the client entities access to the input space. This mitigation strategy will still provide the same accuracy for parts of input space that is similar to input query whilst preventing leaking more information about model performance on other parts of the input space.

[0021] Although it would be possible to detect where in input space the input queries originate (up to the cluster granularity), without separate cluster ML models, it would be difficult to know how to respond upon receiving a query from a part of input space that has been labelled as forbidden - for example if too many queries have been received already in that part of the input space. If there is only access to one ML model for performing the task, the only way to protect that part of the input space from extraction would be to not respond to the query, whereas with the proposed plurality of cluster ML models approach, one of the closest cluster ML models can respond to the query. This will probably provide an acceptable response in many cases, but will not leak any information about the protected part of the input space (since it has not been trained on any samples belonging to that part of the input space).

[0022] As seen in the experimental results provided below, the embodiments described herein may be provided without loss in performance on the original task, in fact, for a majority of the tested OpenML tabular datasets the embodiments described herein even provided an increase in accuracy for a xgboost model, while only a small drop in accuracy is seen for the RandomForestClassifier.

[0023] For the purposes of the present disclosure, the term "ML model” encompasses within its scope the following concepts:

[0024] Machine Learning algorithms, comprising processes or instructions through which data may be used in a training process to generate a model artefact for performing a given task, or for representing a real world process or system; the model artefact that is created by such a training process, and which comprises the computational architecture that performs the task; and the process performed by the model artefact in order to complete the task.

[0025] References to "ML model”, "model”, model parameters”, "model information”, etc., may thus be understood as relating to any one or more of the above concepts encompassed within the scope of "ML model”.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] For a better understanding of the embodiments of the present disclosure, and to show how it may be put into effect, reference will now be made, by way of example only, to the accompanying drawings, in which: Figure 1 illustrates a system comprising a ML model execution environment;

[0028] Figure 2 illustrates a computer implemented method for training a plurality of cluster machine learning models to perform a task;

[0029] Figure 3 illustrates an example implementation of initialising the clustering process;

[0030] Figure 4 illustrates an example implementation of the clustering process of step 201 of Figure 2;

[0031] Figure 5 illustrates an example of the labelling process for a first cluster;

[0032] Figure 6 illustrates a computer implemented method for utilizing one or more of a plurality of cluster machine learning, ML, models to perform a task, wherein each of the plurality of cluster ML models is configured to perform the task;

[0033] Figure 7 illustrates an apparatus comprising processing circuitry (or logic);

[0034] Figure 8 is a block diagram illustrating an apparatus according to some embodiments; and

[0035] Figure 9 is a block diagram illustrating an apparatus according to some embodiments;

[0036] Figure 10 illustrates an exmaple of a cluster tree.

[0037] DETAILED DESCRIPTION

[0038] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0039] The following sets forth specific details, such as particular embodiments or examples for purposes of explanation and not limitation. It will be appreciated by one skilled in the art that other examples may be employed apart from these specific details. In some instances, detailed descriptions of well-known methods, nodes, interfaces, circuits, and devices are omitted so as not obscure the description with unnecessary detail. Those skilled in the art will appreciate that the functions described may be implemented in one or more nodes using hardware circuitry (e.g., analog and / or discrete logic gates interconnected to perform a specialized function, ASICs, PLAs, etc.) and / or using software programs and data in conjunction with one or more digital microprocessors or general purpose computers. Nodes that communicate using the air interface also have suitable radio communications circuitry. Moreover, where appropriate the technology can additionally be considered to be embodied entirely within any form of computer-readable memory, such as (ROM, EEPROM, Flash memory, a memory disc, RAM etc.) solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein.

[0040] Hardware implementation may include or encompass, without limitation, digital signal processor (DSP) hardware, a reduced instruction set processor, hardware (e.g., digital or analogue) circuitry including but not limited to application specific integrated circuit(s) (ASIC) and / or field programmable gate array(s) (FPGA(s)), and (where appropriate) state machines capable of performing such functions.

[0041] Certain aspects of the present disclosure and their embodiments may provide solutions to these or other challenges. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step.

[0042] Particular embodiments are described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0043] In embodiments described herein an initial set of training samples is clustered into disjunct final clusters. The aim of the clustering is to successfully split the training input space into regions that are chosen / derived to maximize or improve the chance that a classifying ML model will choose the correct region of input space for an input query. There may be constraints on number of samples and classes in each final cluster.

[0044] By clustering the input space in this way, it is possible to keep track of how large part of input space has been covered by the input query in terms of the granularity of the clusters. When the clustering process has been completed, one cluster ML model is built per cluster using the samples in the training set being classified as that cluster by the clustering process. The term "cluster ML model” is used to define an ML model that has been trained based on a "cluster” of training samples. A cluster ML model itself may not perform any clustering of data.

[0045] During inference an input query is first assigned to one (or several) clusters by the classifying ML model and then only the response from a cluster ML model associated with that cluster (or a response derived from the output of the cluster models) is returned.

[0046] When exposing the inference API of the cluster ML models, it is possible to keep track of the cluster ML models used by the various client entities and in this way keep track of how large part of input space has been potentially exposed to a particular client entity. Note that this can be used in conjunction with existing extraction detection methods to detect the extraction level of each cluster ML model by a particular client entity. Since there is a separate cluster ML model for each cluster, each cluster ML model will produce accurate inference responses to input queries classified as belonging to its cluster of input space while the performance is expected to be lower for input queries belonging to other clusters of input space, since training of each cluster ML model has only been conducted on samples belonging its respective cluster of training samples. Upon detection of a possible extraction attack the embodiments described herein may therefore enable limiting the leakage of model knowledge to certain parts of input space by restricting the cluster ML models that can respond to a given input query. In this way accurate responses may still be obtained for input queries corresponding to parts of input space covered by cluster ML models, whilst the leakage of information from the cluster ML models that are vulnerable to an extraction attack may be limited as queries corresponding to those restricted parts of input space would not be responded to by cluster ML models trained by that input space.

[0047] Figure 1 illustrates a system 100 comprising a ML model execution environment 102. The ML model execution environment comprises a cluster selection module 104 according to some embodiments. The system may be implemented in a cloud environment, as illustrated in Figure 1.

[0048] One or more client entities (e.g. users or malicious adversaries) 106 may be transmitting input queries over a query API interface to the ML model execution environment to request that a task is performed. The ML model execution environment 102 may have access to a plurality of cluster ML models 108 that are capable of performing the task.

[0049] As will be described in more detail with reference to Figures 2 to 5, each of the plurality of cluster ML models 108 has been trained utilising respective clusters of training samples from an initial training data set.

[0050] A User account control block 110 controls the access of the client entities 106 to the ML model execution environment 102. The client entities 106 may, to the UAC 110, provide user information identifying the individual client entity, for example, along with a password or key in order to gain access to the ML model execution environment 102.

[0051] When the ML model execution environment 102 receives an input query, the input query may be passed to the cluster selection module 104. User information identifying the client entity 106 within the ML model execution environment, e.g. the user information provided to the UAC 1 10, may also be passed to the cluster selection module 104. The cluster selection module 104 may apply any user restrictions and may select the best permissible cluster ML model or cluster ML models for responding to the input query. A response may then be provided the client entity 106 over the API using the selected cluster ML model(s). The progress towards extraction (e.g. the number of input queries from the user responded to by a particular cluster ML model) of each of the cluster ML models made by that particular client entity may then be updated.

[0052] Embodiments described herein may therefore protect the cluster ML models serving the query API from extraction attacks by malicious users by providing means to track the extraction progress of each cluster ML model as well as the capability to reduce model information leakage as different cluster ML models utilize different input space, and different cluster ML models may respond to a user at each query .

[0053] Figures 2 to 5 describe an example implemented of a training phase for the cluster ML models.

[0054] In the training phase with an initial set of training samples X, y (y being the original class), the goal may be to produce final clusters for which a classifying ML model is capable of producing high accuracy whilst also managing to classify the required share of training samples and classes into each cluster. Two parameters constraining the size of the final clusters may be utilised: Nmin and Nmax. Nmin and Nmax may be defined as absolute values or may be defined as maximum shares of an initial set of training samples. These parameters may determine the granularity of the clustering process. Nmaxmay be the maximum number of samples allowed in a single cluster, whereas Nmin may be utilised during a clustering process to avoid some of the final clusters becoming too small.

[0055] Figure 2 illustrates a computer implemented method for training a plurality of cluster machine learning models to perform a task.

[0056] The method 100 may be performed by an apparatus, for example, a network node, which may comprise a physical or virtual node, and may be implemented in a computing device or server apparatus and / or in a virtualized environment, for example in a cloud, edge cloud or fog deployment. The method of Figure 2 may be offline or online. For example, a data centre or server may perform the method of Figure 2, and the resulting plurality of cluster ML models may be transmitted or uploaded into the ML model execution environment 102 of Figure 1.

[0057] In step 201 , the method comprises applying a clustering process to an initial set of training samples to generate a plurality of final clusters of training samples. The clustering process of step 201 will be described in more detail with reference to Figure 4. It will be appreciated that the clustering process may generate cluster labels for each of the initial set of training samples. In some examples, one or more training samples may belong to two or more final clusters. For instance there may be a set of "centrally located" training samples that are ambiguous and bordering to several final clusters. These centrally located training samples may therefore belong to a plurality of final clusters. However, these centrally located training samples would then not be separable and would not be "protected" as well as other training sample belonging to only one final cluster.

[0058] In order to provide final clusters that are capable of being used to adequately train respective cluster ML models, the clustering process may be subject to constraints on the sizes of the final clusters. The clustering process described with reference to Figure 3 provides one method for applying maximum and minimum size constraints to the final clusters. However, it will be appreciated that other methods may be available such as utilizing constrained K-means clustering to control the cluster size [K.P. Bennett, P.S. Bradley, A. Demiriz, "Constrained K-Means Clustering”, Microsoft Research Technical Report MSR-TR-2000-65, https: / / www.microsoft.com / en-us / research / publication / constrained-k-means-clustering / , May 2000],

[0059] In step 202, the method comprises, for each final cluster of training samples, training, utilizing only the final cluster of training samples, a cluster ML model to perform the task. It will be appreciated that the method of training and the type of ML model that the cluster ML model comprises may be dependent on the task to be performed by the cluster ML model. For example, high dimensional data may not be suitable for clustering with Euclidean distance metrics (due to the high dimensionality). For such cases, the data may first be converted to a suitable problem-specific feature encoding and clustering process. Such encodings are recently being learnt by deep networks (for instance word / token-embeddings for GPT and similarly pretrained feature encodings for images).

[0060] In step 203 the method comprises initiating use of at least one of the plurality of cluster ML models to perform the task in response to an input query from a client entity. For example, step 203 may comprise transferring (e.g. transmitting or uploading) the plurality of cluster models to a ML model execution environment such as that illustrates in Figure 1. In order to perform the clustering process of step 201 , the clustering process may be initialized and performed as illustrated in Figures 3, 4 and 5.

[0061] In step 301 , a cluster tree root node is initialized with the set of all training samples. An example of a cluster tree is illustrated in Figure 10.

[0062] In step 302, the method of initializing the clustering process comprises performing an initial clustering on the input set of training samples to generate K, where K is an integer value, initial clusters of training samples. The number K of initial clusters may be chosen, for example, K may be equal to or lower than the output number of cluster ML models. Typically, K may be determined as the number of samples in the initial set of training samples divided by Nmax. The initial clustering may comprise for example utilizing one of: centroidbased clustering (e.g. a K-means clustering process), agglomerative hierarchical clustering (e.g. using Ward linkage), and a partitioning process (e.g. using K medoids). . For low dimensional data step 302 may comprise utilizing ordinary k-Means clustering process, whereas for high dimensionaldata (such as image data) step 302 may require a suitable feature extractor (for image data this could be the first layers of a pretrained Neural network for instance).

[0063] In step 303 the method of initializing the clustering process may further comprise training a classifying ML model to classify the current set of training samples into K initial clusters. The classifying ML model may be the same type of ML model as the cluster ML models. For example, if the cluster ML models comprise a random forest (RF) classifier, then an RF model may also be used as classifying ML model. It will be appreciated that the classifying ML model may be trained on the entire (or majority of) set of training samples in a given cluster node that is not final (with the cluster labels) whereas the cluster ML models are trained on training samples only in final cluster nodes (with the original classes). In addition to RF, other types of classifying ML models may be: XGB (xtreme gradient boosted trees), neural networks (with architecture depending on the data - transformers, MLP, convolutional etc), k-NN (Nearest Neighbor) etc .

[0064] Figure 4 illustrates an example implementation of the CLUSTER() process of step 303 of Figure 3.

[0065] In step 401 the clustering process comprises for each training sample in the set of training samples in a clusternode, utilizing the classifying ML model to associate the training sample with respective confidence values S for the K initial clusters. It should therefore be understood that S is a matrix where each row corresponds to the confidence values for each cluster label of one sample. Thus each column in S corresponds to the confidence values of one cluster label for all the samples. In step 402 the clustering process comprises, for each sample in the clusternode to determine one of the the K cluster labels through a LABEL() process. The labelling process may be performed for each cluster label in turn. It will be appreciated that there may be a number of different ways in which to perform the labelling process. The labelling process may be utilised to ensure that each final cluster is of an appropriate size and / or is diverse enough in the samples it comprises. In other words, the labelling process may employ constraints on one or more of the size of the clusters and the content of the clusters. The labelling process may iteratively change the cluster labels of training samples according to current classifications by the classifying ML model and or possible constraints on a number of samples per cluster, and a number of different classes per cluster. For example, the labelling process may ensure that each cluster comprises training samples having at least two different original classes in the original problem to ensure that each cluster contains at least one decision boundary.

[0066] Figure 5 illustrates an example of the LABEL() process for a first cluster. It will be appreciated that the process of Figure 5 may be performed for each cluster C_i„ 1=1 , ..., K.

[0067] Herein Nmin and Nmax are implemented as absolute values, however, it will be appreciated that the labelling process and / or clustering process may be adjusted to implement these values as maximum shares of the initial set of training samples.

[0068] In step 501 , the labelling process comprises selecting N training samples, where N is an integer, from within the first cluster (e.g. N samples that the classifying ML model classifies with the first cluster label C_1), wherein the N training samples are associated with the N highest respective confidence values for the first cluster, ie the top scores in SJ such that argmax(Sj) == C_1 , for 1=1 , ...,N, where Si implies the ithrow in S. N may be less than or equal to a minimum number, Nmin, of training samples to be in each of the final clusters.

[0069] In step 502 the labelling process comprises setting a cluster label for each of the selected N training samples as a first cluster label associated with the first cluster.

[0070] In step 503 the labelling process comprises adding the N training samples to a set of labelled training samples for the first cluster.

[0071] In other words, the steps 501 to 503 effectively set labels in the first cluster indexed by C_1 for at most Nmin samples, where argmax(S) == C_1 , and sorted by largest margin max(S)-max(SA), where SAis S, where the scores for C_1 have been set to 0. This implies selecting the at most Nmin samples with the largest separating margin to the closest surrounding cluster label. In step 504, the labelling process comprises determining whether N is less than the minimum number, Nmin, of training samples.

[0072] If in step 504 it is determined that N is less than Nmin the labelling process passes to step 505 which comprises selecting, from outside of the first cluster (e.g. training samples for which the classifying ML model has classified as having a different cluster label to the first cluster label, eg argmax(S) C_1), M training samples, where M is an integer, associated with the M highest respective confidence values associated with the first cluster, wherein M is the difference between the minimum number and N. In other words, if the initial cluster was not large enough, the labeling process selects training samples from outside the first cluster that are associated with the M highest respective confidence values for the first cluster, in order to get a cluster of at least the minimum size Nmin.

[0073] In some examples step 505 comprises steps 505a and 505b. Step 505a comprises determining whether at least a first threshold number, Ti, of training samples in the N training samples is associated with a first original class. It will be appreciated that the intention of step 505a is to determine how many training samples in the N training samples have the same original class. The first original class may therefore be any original class.

[0074] The original class of a training sample comprises the class of the training sample for training each cluster ML model. In other other words, the original class of the training sample is the class of the training sample with respect to the task being performed by the cluster ML models.

[0075] If in step 505a it is determined that there are less than or equal to the first threshold number of training samples (e.g. none of the original classes are over represented in the N training samples), the method passes to step 505c in which the M training samples are selected regardless of their original class.

[0076] If in step 505a it is determined that there are greater than the first threshold number of training having the first original class (e.g. the first original class is over represented in the N training samples), selecting the M training samples from training samples associated with a different original class to the first original class. In other words, steps 505a and 505b ensure that when selecting the M training samples, the labelling process considers the diversity of the training samples that it is selecting. In some examples, the first threshold number is N / 2. In other words, in some examples, if more than half of the samples with L == C_1 have the same y (original class) - then the M training samples may be selected such that y is both different sorted by high score, for the column for label C_1 in S. In step 506 the labelling process comprises setting a cluster label for each of the selected M training samples as the first cluster label associated with the first cluster.

[0077] In step 507 the labelling process comprises adding the M training samples to the set of labelled training samples for the first cluster.

[0078] In other words, steps 505 to 507 effectively, if number of labels L == C_1 < Nmin, set label C_1 to the currently unlabeled samples chosen according to samples with highest scores for cluster label C_1 in S (eg by largest value in corresponding column of S) so that #{L==C_1 } == Nmin

[0079] After step 507 the method passes to step 508. Also, if in step 504 it is determined than N equal to Nmin, the method passes straight to step 508.

[0080] In step 508 the labelling process comprises determining whether at least second threshold number, T2, of training samples in the set of labelled training samples is associated with a second original class. The second threshold number may be % the minimum number, 3Nmin / 4. It will be appreciated that the intention of step 508 is to determine how many training samples in the set of labelled training samples have the same original class. The second original class may therefore be any original class.

[0081] If in step 508 it is determined that less than the second threshold number of training samples in the set of labelled training same is associated with the second original class (e.g. if no original class is over represented in the set of labelled training samples), the method passes to step 510 in which the labelling process ends.

[0082] If in step 508 it is determined that at least T2 training samples in the set of labelled training samples are associated with a second original class (e.g. if the second original class is over represented in the set of labelled training samples), the labelling process passes to step 509 which comprises selecting Z training samples from outside of the set of labelled training samples (e.g. from training samples that have not yet been assigned a new cluster label). The Z training samples may be associated with a different original class to the second original class and may be selected based on their respective confidence values associated with the first cluster (eg the values in the corresponding column of S). In other words, in step 509 training samples that have a different original class to the second original class and have the highest confidence values associated with the first cluster. This ensures the diversity of the training samples in the set of labelled training samples. Z may be Nmin / 2. In other words, in some examples, if more than3 / 4*Nmin of the samples where L==C_1 have the same y (original class) the labelling process comprises add Nmin / 2 samples with both different y and sorted by high score in the column in S corresponding to C_1.

[0083] The method then passes to step 510 in which the labelling process for the first cluster ends.

[0084] It will be appreciated that the LABEL() process of Figure 5 may be performed for each cluster in the initial set of clusters. Once the labelling process has been performed for each initial cluster, there may be some training samples in the initial set of training samples that have not yet been assigned a cluster label.

[0085] Returning to Figure 4, the method then passes to step 403 in which it is determined whether all the initial set of training samples have been provided cluster labels in step 402.

[0086] If in step 403 it is determined that one or more of the training samples have not had a cluster label set, the method passes to step 404 which comprises utilizing the classifying ML model to set cluster labels for the one or more of the training samples. In other words, the cluster labels of any training samples not labelled in step 402 are set to the output of the classifying ML model for that training sample. The method then passes to step 405.

[0087] If in step 403 is determined that all of the initial set of training samples have had a cluster label set, the method passes to step 405.

[0088] In step 405, the classifying ML model is retrained on the cluster labels obtained in step 402 and optionally step 404. Subsequently the method comprises updating the K initial clusters utilising the cluster labels output by the updated classifying ML model. In other words, the clusters are at this stage reorganised based on the cluster labels that have been determined at this point in the clustering process.

[0089] In step 406 the method comprises determining whether a stopping condition has been met. The stopping condition may comprise a condition that no cluster labels are changed in steps 402 or 404 from a previous iteration of the clustering process. Alternatively, or additionally, the stopping condition may comprise a threshold number of iterations of the clustering process. Note that between each round of the clustering process it is possible to use cross-validation and change the selection of training samples for each round of the clustering process for robustness. For example, when the training samples within the initial set of training samples is limited you can use cross-validation to help avoid over-training. In other words, during each round of training the classifying ML model, a held-out set of training samples may be randomly selected that will not be part of any training - only testing. If the partitioning is changed in every iteration of training, the risk for over-training on a specific train split can be greatly reduced. If in step 406, the stopping condition is not met, the method passes to step 408 in which the cluster labels are removed from the initial set of training samples. The method then passes back to step 401 and the clustering process is repeated with the retrained classifying ML model (retrained in step 405).

[0090] If in step 406 it is determined that the stopping condition is met, the method passes to step 407 which comprises setting resulting clusters based on the last cluster labels generated by the clustering process. In some examples, the resulting clusters are set as the final clusters to be used in step 202 of Figure 2.

[0091] Returning to Figure 3, in step 304 the resulting classifying ML model (which outputs cluster labels for the current node) is set to current node and a link to a new child node in the cluster tree is created for each resulting set of samples with the same labels from the CLUSTER() process in step 303.

[0092] The method may then passes to step 305 in which it is determined whether any of the resulting clusternodes comprise greater than a third threshold number of training samples. In other words, the third threshold number may be Nmax. The maximum allowed size may be set such that the input space covered by a single cluster ML model is not too large.

[0093] For example, in step 305 the number of samples in each childnode may be compared against the maximum size Nmax . If the number of samples in the childnode is equal to or smaller than Nmaxthen the clusternode is set to be a final cluster node in step 307. If the number of samples is larger than Nmaxthen in step 306 the child node will be initiated as a split node and the samples will be further subclustered by returning to step 302.

[0094] Figure 10 illustrates an example of a cluster tree in which a root node (e.g. the initial set of training samples) is used to generate final cluster nodes (e..g the leaf nodes) via one or more split nodes.

[0095] If in step 305 it is determined that a resulting cluster comprises greater than the third threshold number of training samples the method passes to step 306 in which the resulting cluster is set to be a split clusternode and passed through the clustering process independently. -. After returning to step 302 with the subcluster, for the initialisation of the clustering process r, the value of K may be chosen such that the expected number of training samples in each sub-cluster will be smaller than Nmaxbut larger than Nmin.

[0096] In other words, the clustering process may be continued to divide a resulting clusternode into two or more smaller clusters. The classifying ML model is associated with the root and split clusternodes are stored in this case for use in the inference phase. If in step 305 it is determined that no resulting cluster comprise greater than the third threshold number of training samples the method passes to step 307 in which the resulting clusters may be set as the final clusters. Final clusters imply endnodes (also called leaves) in the cluster tree (e.g. as illustrated in Figure 10).

[0097] Figure 6 describes an inference phase for one or more of a plurality of cluster ML models. It will be appreciated that the method of Figure 6 may be performed by the ML model execution environment 102 illustrated in Figure 1.

[0098] Figure 6 illustrates a computer implemented method for utilising one or more of a plurality of cluster machine learning, ML, models to perform a task, wherein each of the plurality of cluster ML models is configured to perform the task. It will be appreciated that the plurality of cluster ML models may be trained according to the method as described with reference to any one of Figures 2 to 5. For example, the plurality of cluster ML models are trained utilising respective clusters of training samples from an initial training data set.

[0099] The method of Figure 6 may be performed by the ML model execution environment 102 of Figure 1.

[0100] In step 601 the method comprises receiving an input query from a client entity.

[0101] In step 602, the method comprises inputting the input query into a classifying ML model cluster tree to determine a subset of cluster ML models from the plurality of cluster ML models that the input query is associated with. It will be appreciated that the classifying ML model cluster tree may be, trained as described with reference to any of Figures 2 to 5. For example, the method of claim 2 wherein the classifying ML model is trained utilizing the clusters of training samples used to train the cluster ML models. Step 602 may be performed by the cluster selection module 104 of Figure 1. It will be appreciated that the clusters may be arranged in a tree structure as described with reference to any of Figures 2 to 5. In some examples step 602 comprises steps 602a, 602b and 602c. In step 602a, the input query is first input to the root of the cluster tree and evaluated by the associated classifying ML model to output confidence scores of the cluster nodes that are children to the root node. The confidence score of each cluster node is associated with the link between the root node and the child cluster node. Any child node that is a split node also has an associated classifying ML model and the input query may be evaluated by each such model to obtain the confidence scores of the links to nodes further down the tree. In step 602b, the confidence score of each cluster endnode E, that contains a cluster ML model, may be determined as the product of the confidence scores of the links on the path from the root node to E. In step 602c, a subset of the most probable cluster nodes can be selected according to the accumulated confidence scores determined in step 602b. It will be appreciated that there are a number of effective ways of determining a subset of most probable paths without evaluating all paths in the tree by, at each level in the tree, only continuing to evaluate the classifying ML model in a split node if the confidence is higher than the Nth highest confidence score to a cluster end node so far.

[0102] It will be appreciated that in performing step 602 the classifying ML model may be configured to receive the input query and classify the input query as belonging to at least one of the clusters, wherein the input query is associated with the subset of cluster ML models that were trained utilising the at least one of the clusters. The input query may be determined to belong to more than one of the clusters.

[0103] The classifying ML model may be configured to associate the input query with cluster models in the subset of clusters ML models according to respective confidence values. For example, there may be a threshold confidence level, wherein if the confidence level for the input query belonging to a particular cluster is greater than the threshold confidence level, the classifying ML model may associate the input query with the cluster ML model trained by that particular cluster.

[0104] In step 603 the method comprises inputting the input query into at least one of the subset of cluster ML models to generate at least one first output.

[0105] In some examples, step 603 comprises steps 603a and 603b. Step 603a comprises filtering the subset of cluster ML models to generate a selected set of cluster models, and step 603b comprises inputting the input query into at least one of the selected set of cluster ML models to generate the at least one first output. For example, step 603a may comprise removing cluster ML models for which the client entity has already inputted more than a threshold number of input queries. Step 603a may therefore protect an individual cluster ML model from being queried too many times by a particular client entity (e.g. enough times for the client entity to extract the cluster ML model with reasonable accuracy). It will be appreciated that the threshold number of input queries may be a design choice dependent on the level of security desired by the designer of the particular system.

[0106] In some examples, step 603a comprises limiting the number of cluster ML models in the selected set of cluster ML models to a maximum number. The selected set of cluster ML models may be the cluster models associated with the highest confidence values for the input query. Step 603a may be performed by the cluster selection module 104 of Figure 1 , In other words, the cluster selection module 104 may be provided with the functionality to apply possible restrictions on which cluster ML models should be active or useable for a client entity. The cluster selection module 104 may also keep track of the cluster ML models that are responding or have responded to queries for each client entity. For instance, there could be one allowed extraction level per cluster ML model (before blocking new responses from a particular cluster, and / or an allowed extraction level in the form of the number of cluster ML models that are allowed to respond to a particular client entity, or to a particular input query).

[0107] It will be appreciated that step 603a may comprise any filtering to ensure that the selected set of cluster ML models are not at risk of extraction by the client entity. The filtering may therefore be based on a determined possible extraction progress that the client entity may have made.

[0108] Depending on the type of model / learner of the cluster ML models, some existing methods for estimating the possible model extraction progress may be used per individual cluster ML model. For instance, tree-based estimators [Error! Reference source not found.] might be used for ML models trained using small clusters either where tree-models are used for the cluster ML models, or where they can be an acceptable approximation. In other cases, such as complex decision boundaries (as can arise with Neural Network models), or for cluster ML models trained with large clusters where estimator retraining time is a problem, those existing methods are unlikely to be a good fit. For such cases, a type of upper bound may be estimated - e.g., by tracking which classes in a cluster have been reported to the client entity. Until all classes have been reported, the adversary cannot know the decision boundaries, so the cluster ML model resulting from the cluster may be considered "safe”. Additionally or alternatively, the fraction of reported classes in a cluster can be tracked as a "loose upper bound” on extraction per cluster ML model.

[0109] In step 604 the method comprises providing an output response to the client entity, wherein the output response is derived from the at least one first output. In some examples, step 603 or 603b comprises inputting the input query into only one cluster model and generating only one first output, in step 604 the output response comprises the first output.

[0110] In some examples, step 604 may comprise steps 604a and 604b. Step 604a comprises inputting the input query into a plurality of the subset of cluster ML models to generate a respective plurality of first outputs; and step 604b comprises aggregating the plurality of first outputs to generate the output response. In some examples, step 604b is performed based on the respective confidence values associated with the plurality of the subset of cluster ML models. For example, the first outputs may be weighted according to the respective confidence values. In some examples, step 604 comprises inputting the input query into all of the selected set of cluster ML models to generate respective first outputs for all of the selected set of cluster ML models.

[0111] In some examples, the embodiments described herein may be utilized in the 5G core network for example implemented by a NWDAF. In these cases, the training samples and the input query may take the form of IP (Internet Protocol) traffic data, for example, type of traffic, size, number of packets etc. The task performed by the cluster ML models may be to classify user and / or user equipment (UE) behavior (in order to optimize experience). The original classes may be different categories of users / devices / application usage. Example original classes may be :voice, web browsing, video streaming, etc. In this example, the user plane function may supply the traffic data to the Network Data Analytics Function (NWDAF) which may then perform analytics utilizing one or more cluster ML models and may respond to the consuming NF (e.g., Policy Control Function PCF). The NWDAF may be standalone or co-located with other 5G core functions. In this example, the method of Figure 6 may be performed by the NWDAF. The methods of Figures 2 to 5 may be performed at the NWDAF, or may be performed in a computationally capable cloud somewhere prior to deploying the plurality of cluster ML models in the NWDAF.

[0112] In some examples, the embodiments described herein may be utilized in a Business Support System, for example, to monitor customer-churn. In this example, the training samples and / or input queries may comprise numerical and categorical values related to subscriber statistics and usage history, e.g., subscription information (services, date established, etc.), Call Detail Records (CDRs). In this example, the task may comprise predicting customer-churn, for example, whether or not subscriber will move to another provider. In this example, the original class may comprise possible customer choices (for example just yes, customer will move / no, customer will not move, or which specific competitor a subscriber will move to). In this example, the method of Figure 6 may be performed by a node in a BSS system. The methods of Figures 2 to 5 may be performed at a node in a BSS system, or may be performed in a computationally capable cloud somewhere prior to deploying the plurality of cluster ML models in the BSS system.

[0113] In some examples, the embodiments described herein may be utilized in an Open Radio Access Network (O-RAN) for example, to perform extended application (xApp) handover optimization. In this example, the training samples and / or input queries may comprise information relating to signal strength of a cell, quality of surrounding cells and possibly UE location . In this example, the task may comprise predicting whether or not a handover should be performed now and, if so, to which cell. In these examples, the original classes may comprise: no handover, handover to neighboring cell 1, handover to neighboring cell 2, ...etc. In this example, the method of Figure 6 may be performed by an xApp that receives an input query from an O-RAN Central Unit-Control Plane O-CU-CP layer in a base station gNB, performs handover optimization analysis (utilizing one or more cluster ML models) and responds to the O-CU-CP layer to initiate or prevent a handover.

[0114] The methods of Figures 2 to 5 may be performed at a node in the xApp, or may be performed in a computationally capable cloud somewhere prior to deploying the plurality of cluster ML models in the xApp. Typically, the training may be performed offline in a lab environment based on collected measurement data.

[0115] EXPERIMENTAL RESULTS

[0116] Experiments have been conducted utilizing embodiments described herein to train plurality of cluster ML models and utilize them for inference as described herein. In these experiments it is illustrated that the resulting performance does not drop significantly. In fact, the embodiments described herein produced better results than the original model for several of the OpenML-sets tested, when utilizing the proposed embodiments and returning results only from the best cluster xgboost model compared to only providing results from a single xgboost model. For the tests using a random forest classifier, only a minor drop in accuracy was seen when the embodiments described herein were utilized. The experiments use different tabular datasets.

[0117] Note that all experiments were conducted without any hyperparameter optimization and that it is possible that even better results could have been obtained by tuning cluster sizes and or the clustering process for each dataset.

[0118] Now standard settings were used and the same clustering process, labelling processing and classifying ML model used as for the final output (the classification model of the original task in each cluster).

[0119] Nmin for the experiments in Table 1 was set to 4% of the size of the initial set of training samples and Nmax 12%. It will be appreciated that, according to some embodiments, the Nmin and Nmax parameters means that a minimal cluster size may still be smaller than Nmin while the maximum cluster size may not exceed Nmax.

[0120] There is only one dataset where the accuracy dropped significantly and that is cnae-9 which had fewer samples than features making distance-based clustering work less well. This could probably be mitigated with other clustering strategies or with access to a larger dataset.

[0121] Table 1 illustrates the performance of cluster models vs an original model (trained on the entire initial set of training samples) for an original task (e.g. XGBoost and RandomForest).

[0122] Table 1 also indicates the number of created clusters along with the accuracy of the model when the best cluster (from the cluster classifier) is blocked.

[0123] As one way to get an indication of how successful the proposed cluster strategy is at preventing leakage of model information, that could be used to build a replicated model, Table 1 shows the accuracy obtained for the test set when the cluster model with the best match according to the cluster classification model is blocked. This simulates the situation on average (over the different blocked clusters) when at least one cluster would be blocked because of many queries to being asked in different clusters by some user, indicating a possible model extraction attack. This is an average way of showing that some leakage is prevented. In a real case only a few specific clusters would be blocked resulting in unchanged accuracy for the other clusters but this average drop in accuracy for the queries that would have been handled by the blocked clusters.

[0124] Furthermore, for each individual cluster model it is possible to apply other model extraction protection mechanisms. The proposed model clustering strategy could for instance be used in combination with the deception techniques for out-of-distribution queries such as the one presented in [7] to "poison” the extracted dataset for all queries classified to belong to the blocked cluster.

[0125] The blocked cluster accuracy may be seen as proof that the clusters are indeed separate input spaces and that complete extraction of overall model would not be possible without response pairs from all clusters. Furthermore, it will be appreciated that the reported drop in accuracy when a cluster is blocked is not to show the effect on legitimate use but to show that there is an expected drop in accuracy for input queries that would have been handled by one cluster when it is blocked (i.e. that there is leakage protection by prohibiting answers to one part of the input space). Samples from any of the unblocked clusters would have unchanged accuracy. In other words, the embodiment described herein may be considered complementary to other extraction progress estimation techniques in that they may automatically provide a countermeasure that, at a coarser level (i.e number of clusters granularity), prohibits complete model extraction whilst maintaining high accuracy for legitimate users on most parts of input space (or even all parts relevant for that particular user). And in particular responses with more than 0 (i.e. not blocked) accuracy can even be achieved by a close cluster model when one model is blocked due to potential extraction risk.

[0126] It will be appreciated that there are no results illustrated from blocking multiple clusters at the same time as the main idea of the blocking the "best” cluster is to show that there is a protection against complete model leakage. This is not the accuracy for a legitimate user when a single cluster is dropped, as only (n training samples in blocked cluster) / (total number training samples) requests would in average reach the blocked cluster.

[0127] The effect on energy consumption may be task and ML-process dependent. At a first glance it may seem that running multiple cluster ML models - first determining the cluster model(s) to respond and secondly running the chosen cluster model - would implicate higher energy consumption. However, since the first task of determining a cluster would typically be an easier task than determining the true desired response from the model (in fact the proposed process seeks to find as separable clustering as possible) this would most often require fewer calculations (a simpler / shallower model) than the original model. Furthermore, the chosen cluster ML model may also be less complex since it will only handle a part of the complete input space. In some cases (as with the improved accuracy in our experiments) there could even be reduced number of calculations with this approach due to the smaller number of calculations in each sequential step (instead of solving a much more complex classification in one ML model).

[0128] Figure 7 illustrates an apparatus 700 comprising processing circuitry (or logic) 701 . The processing circuitry 701 controls the operation of the apparatus 700 and can implement the method described herein in relation to an apparatus 700. The processing circuitry 701 can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the apparatus 700 in the manner described herein. In particular implementations, the processing circuitry 701 can comprise a plurality of software and / or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein in relation to the apparatus 700. It will be appreciated that the apparatus 700 may comprise one or more virtual machines running different software and / or processes. The apparatus 700 may therefore comprise, or be implemented in or as one or more servers, switches and / or storage devices and / or may comprise cloud computing infrastructure that runs the software and / or processes. Optionally, the apparatus 700 may comprise a memory 703. In some embodiments, the memory 703 of the apparatus 700 can be configured to store instructions (e.g. program code) executable by the processing circuitry 701 of the apparatus 700 whereby the apparatus is operable to: perform the method as described with refernece to any of Figures 2 to 6.

[0129] Alternatively or in addition, the memory 703 of the apparatus 700, can be configured to store any requests, resources, information, data, signals, or similar that are described herein. The processing circuitry 701 of the apparatus 700 may be configured to control the memory 703 of the apparatus 700 to store any requests, resources, information, data, signals, or similar that are described herein

[0130] In some embodiments, the apparatus 700 may optionally comprise a communications interface 702. The communications interface 702 of the apparatus 700 can be for use in communicating with other nodes, such as other virtual nodes. For example, the communications interface 702 of the apparatus 700 can be configured to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The processing circuitry 701 of apparatus 700 may be configured to control the communications interface 702 of the apparatus 700 to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The communications interface 702 can use any suitable communication technology.

[0131] The apparatus 700 may be configured operate in the manner described herein in respect of an apparatus.

[0132] Figure 8 is a block diagram illustrating an apparatus 800 according to some embodiments. The apparatus 800 can utilize one or more of a plurality of cluster machine learning, ML, models to perform a task, wherein each of the plurality of cluster ML models is configured to perform the task. The apparatus 800 comprises a receiving module 802 configured to receive an input query from a client entity. The appartus 800 comprises a first inputting module 804 configured to input the input query into a classifying ML model to determine a subset of cluster ML models from the plurality of cluster ML models that the input query is associated with. The apparatus 800 further comprises a second inputting module 806 configured to input the input query into at least one of the subset of cluster ML models to generate at least one first output. The apparatus 800 further comprises a providing module 808 configured to provide an output response to the client entity, wherein the output response is derived from the at least one first output.

[0133] The apparatus 800 may operate in the manner described herein in respect of an ML model execution environment. For example, the apparatus 800 may be configured to perform the method as described with reference to Figure 6. Figure 9 is a block diagram illustrating an apparatus 900 according to some embodiments. The apparatus 800 can train a plurality of cluster machine learning models to perform a task. The apparatus 900 comprises an applying module 902 configured to apply a clustering process to an initial set of training samples to generate a plurality of final clusters of training samples. The apparatus 900 further comprises a training module 904 configured to for each final cluster of training samples, train, utilizing the final cluster of training samples, a cluster ML model to perform the task. The appartus 900 further comprises an initiating module 906 configured to initiate use of at least one of the plurality of cluster ML models to perform the task in response to an input query from a client entity.

[0134] The apparatus 900 may operate in the manner described herein in respect of an apparatus. For example, the apparatus 900 may be configured to perform the method as described with reference to Figures 2 to 5.

[0135] There is also provided a computer program comprising instructions which, when executed on a least one processor (such as the processor 701 of the apparatus 700 described earlier), cause the processor to carry out at least part of the method(s) described herein. According to some embodiments there is provided a carrier containing the computer program. In some embodiments, the carrier can be any one of an electronic signal, an optical signal, an electromagnetic signal, an electrical signal, a radio signal, a microwave signal, or a computer-readable medium. There is also provided a (for example, tangible and / or non-transient) computer-readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform at least part of the method(s) described herein.

[0136] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word "comprising” does not exclude the presence of elements or steps other than those listed in a claim, "a” or "an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims. Any reference signs in the claims shall not be construed so as to limit their scope.

Claims

CLAIMS1 . A computer implemented method for utilizing one or more of a plurality of cluster machine learning, ML, models to perform a task, wherein each of the plurality of cluster ML models is configured to perform the task, the method comprising: receiving (601) an input query from a client entity; inputting (602) the input query into a classifying ML model to determine a subset of cluster ML models from the plurality of cluster ML models that the input query is associated with; inputting (603) the input query into at least one of the subset of cluster ML models to generate at least one first output; and providing (604) an output response to the client entity, wherein the output response is derived from the at least one first output.

2. The method of claim 1 wherein the plurality of cluster ML models are trained utilizing respective clusters of training samples from an initial training data set.

3. The method of claim 2 wherein the classifying ML model is trained utilizing the clusters of training samples.

4. The method of claim 3 wherein the classifying ML model is configured to receive the input query and classify the input query as belonging to at least one of the clusters, wherein the input query is associated with the subset of cluster ML models that were trained utilising the at least one of the clusters.

5. The method of claim 4 wherein the classifying ML model is configured to associate the input query with cluster models in the subset of clusters ML models according to respective confidence values.

6. The method of any one of claims 1 to 5 further comprising: inputting (604a) the input query into a plurality of the subset of cluster ML models to generate a respective plurality of first outputs; and aggregating (604b) the plurality of first outputs to generate the output response.

7. The method as claimed in claim 6 when dependent on claim 5 wherein the aggregating is performed based on the respective confidence values associated with the plurality of the subset of cluster ML models.

8. The method of any one of claims 1 to 7 further comprising: filtering (603a) the subset of cluster ML models to generate a selected set of cluster models; and inputting (603b) the input query into at least one of the selected set of cluster ML models to generate the at least one first output.

9. The method as claimed in claim 8 wherein the inputting comprises: inputting the input query into all of the selected set of cluster ML models to generate respective first outputs for all of the selected set of cluster ML models.

10. The method as claimed in claim 8 or 9 wherein the filtering comprises: removing cluster ML models for which the client entity has already inputted more than a threshold number of input queries.

11. The method as claimed in any one of claims 8 to 10 wherein the filtering comprises: limiting the number of cluster ML models in the selected set of cluster ML models to a maximum number.

12. The method as claimed in claim 11 when dependent on claim 5 wherein the limiting comprises: selecting the maximum number of cluster ML models in the selected set of cluster ML models associated with highest confidence values.

13. The method as claimed in any preceding claim wherein the input query comprises internet protocol traffic data.

14. The method as claimed in claim 13 wherein the task comprises classifying user equipment behavior.

15. The method as claimed in any one of claims 1 to 14 wherein the method is performed by a Network Data Analytics Function.

16. The method as claimed in any one of claims 1 to 15 wherein the plurality of cluster ML models are trained according to any one of claims 17 to 4217. A computer implemented method for training a plurality of cluster machine learning models to perform a task, the method comprising:applying (201) a clustering process to an initial set of training samples to generate a plurality of final clusters of training samples; for each final cluster of training samples, training (202), utilizing the final cluster of training samples, a cluster ML model to perform the task; and initiating (203) use of at least one of the plurality of cluster ML models to perform the task in response to an input query from a client entity.

18. The method of claim 17 wherein the method further comprises performing (302) an initial clustering on the initial set of training samples to generate K, where K is an integer value, initial clusters of training samples.

19. The method of claim 18 wherein the initial clustering comprises utilizing one of: centroid-based clustering, agglomerative hierarchical clustering, and a partitioning process.

20. The method of any one of claims 17 to 19 wherein applying the clustering process is subject to constraints on sizes of the final clusters.21 . The method of any one of claims 17 to 20 wherein applying the clustering process generates cluster labels for each of the initial set of training samples.

22. The method of claim 21 further comprising: training (303) a classifying ML model to classify the initial set of training samples into the K initial clusters.

23. The method of claim 22 wherein applying the clustering process comprises: for each training sample in the initial set of training samples, utilizing (401) the classifying ML model to associate the training sample with respective confidence values for the K initial clusters.

24. The method of claim 23 wherein applying the clustering process further comprises: for a first cluster in the K initial clusters, performing a labelling process (403), wherein the labelling process comprises: selecting (501) N training samples, where N is an integer, from within the first cluster, wherein the N training samples are associated with the N highest respective confidence values for the first cluster; and setting (502) a cluster label for each of the selected N training samples as a first cluster label associated with the first cluster; andadding (503) the N training samples to a set of labelled training samples for the first cluster.

25. The method of claim 24 wherein N is less than or equal to a minimum number of training samples to be in each of the final clusters.

26. The method of claim 25 wherein the labelling process comprises: after setting the cluster label for each of the selected N training samples as the first cluster label associated with the first cluster, and responsive to N being less than the minimum number, selecting (505), from outside of the first cluster, M training samples, where M is an integer, associated with the M highest respective confidence values associated with the first cluster, wherein M is the difference between the minimum number and N; and setting (506) a cluster label for each of the selected M training samples as the first cluster label associated with the first cluster; and adding (507) the M training samples to the set of labelled training samples for the first cluster.

27. The method as claimed in claim 26 wherein, selecting, from outside of the first cluster, M training samples associated with the M highest respective confidence values associated with the first cluster comprises: responsive to at least a first threshold number of training samples in the N training samples being associated with a first original class, selecting (505b) the M training samples from training samples associated with a different original class to the first original class.

28. The method as claimed in claim 27 wherein the first threshold number is N / 2.

29. The method as claimed in any one of claims 24 to 28, wherein the labelling process further comprises, responsive to at least second threshold number of training samples in the set of labelled training samples being associated with a second original class, selecting (509) Z training samples from outside of the set of labelled training samples, wherein the Z training samples are: associated with a different original class to the second original class and are selected based on their respective confidence values associated with the first cluster.

30. The method as claimed in claim 29 when dependent on claim 25, wherein the second threshold number is % the minimum number.31 . The method as claimed in claim 29 or 30 when dependent on claim 25, wherein Z is half the minimum number.

32. The method as claimed in any one of claims 24 to 31 further comprising performing the labelling process for each cluster in the K initial clusters.

33. The method as claimed in claim 32 wherein the clustering process further comprises, after performing the labelling process for each cluster in the K clusters, responsive to determining that one or more of the initial set of training samples have not had a cluster label set, utilizing (404) the classifying ML model to set cluster labels for the one or more of the initial set of training samples.

34. The method as claimed in any one of claims 17 to 33 when dependent on claim 18 wherein the clustering process further comprises updating the K initial clusters utilising the cluster labels determined by the clustering process.

35. The method as claimed in claim 17 to 34 wherein the clustering process further comprising retraining (405) the classifying ML model with the cluster labels determined by the clustering process.

36. The method as claimed in claim 35 further comprising repeating iterations of the clustering process until a stopping condition is met, and after the stopping condition is met, setting (407) resulting clusters based on the last cluster labels generated by the clustering process.

37. The method as claimed in claim 36 wherein the resulting clusters are the final clusters.

38. The method as claimed in claim 36 or 37 wherein repeating iterations of the clustering process comprises after retraining the classifying ML model, removing (408) the cluster labels from the initial set of training samples.

39. The method as claimed in claim 36, wherein the clustering process further comprises, responsive to a first resulting cluster in the resulting clusters comprising more than a third threshold number of training samples, performing (306) the clustering process on training samples within the first resulting cluster.

40. The method as claimed in any one of claims 17 to 39 wherein the initiating use comprises: receiving (601) an input query from a client entity; inputting (603) the input query into at least one of the plurality of cluster ML models to generate at least one first output; and providing (604) an output response to the client entity, wherein the output response is derived from the at least one first output.

41. The method as claimed in any preceding claim wherein the training samples comprise internet protocol traffic data.

42. The method as claimed in claim 41 wherein the task comprises classifying user equipment behavior.

43. An apparatus (700) for utilizing one or more of a plurality of cluster machine learning, ML, models to perform a task, wherein each of the plurality of cluster ML models is configured to perform the task, the apparatus comprising processing circuitry (701) and a memory, the memory containing instructions executable by the processing circuitry whereby the apparatus is operable to: receive (601) an input query from a client entity; input (602) the input query into a classifying ML model to determine a subset of cluster ML models from the plurality of cluster ML models that the input query is associated with; input (603) the input query into at least one of the subset of cluster ML models to generate at least one first output; and provide (604) an output response to the client entity, wherein the output response is derived from the at least one first output.

44. The apparatus as claimed in claim 43 wherein the memory further contains instructions executable by the processing circuitry whereby the apparatus is operable to perform the method as claimed in any one of claims 2 to 16.

45. An apparatus (700) for training a plurality of cluster machine learning models to perform a task, the apparatus comprising processing circuitry and a memory, the memory containing instructions executable by the processing circuitry (701) whereby the apparatus is operable to: apply (201) a clustering process to an initial set of training samples to generate a plurality of final clusters of training samples; for each final cluster of training samples, training (202), utilizing the final cluster of training samples, a cluster ML model to perform the task; andinitiating (203) use of at least one of the plurality of cluster ML models to perform the task in response to an input query from a client entity.

46. The apparatus as claimed in claim 45 wherein the memory further contains instructions executable by the processing circuitry whereby the apparatus is operable to perform the method as claimed in any one of claims 18 to 42.

47. A computer program, comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out a method according to any of claims 1 to 42.

48. A carrier containing the computer program according to claim 47, wherein the carrier comprises one of an electronic signal, optical signal, radio signal or computer readable storage medium.

49. A computer-readable medium comprising instructions that, when executed on at least one processor, cause the at least one processor to perform the method according to any of claims 1 to 4250. A computer program product comprising non transitory computer readable media having stored thereon a computer program according to claim 47.

Images