Methods and apparatuses for performing distributed learning of a task machine learning model utilising an evaluator machine learning model

Abstract

Embodiments described herein relate to methods and apparatuses for enabling performance of a distributed learning, DL process. The DL process is conducted between a server node and a plurality of distributed nodes comprising the distributed node, each distributed node having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model. A method in a management node comprises for individual distributed nodes of at least a subset of the plurality of distributed nodes, receiving a representation of a local version of an evaluator ML model, wherein the evaluator ML model is configured to predict performance of the task ML model, and wherein the local version of the evaluator ML model has been trained using data from the local distributed node data set to which the distributed node has access; aggregating the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model; using the global version of the evaluator ML model to predict performance of a version of the task ML model; and initiating adjustment of at least one parameter of the DL process according to the predicted performance.

Description

[0001] METHODS AND APPARATUSES FOR PERFORMING DISTRIBUTED LEARNING OF A TASK MACHINE

[0002] LEARNING MODEL UTILISING AN EVALUATOR MACHINE LEARNING MODEL

[0003] TECHNICAL FIELD

[0004] Embodiments described herein provide methods and apparatuses for enabling performing of a distributed learning process. In particular, methods and embodiments described herein provide for generation of a evaluator ML model trained using distributed learning for evaluating performance of a version of the task ML model.

[0005] BACKGROUND

[0006] Federated learning, FL, is a privacy-preserving distributed learning technique which enables the training machine learning models collaboratively utilizing remote agents (e.g. edge devices, core network nodes etc), herein generally referred to as client nodes or distributed nodes, without sharing raw data. FL may begin with a server, herein generally referred to as a server node, initializing a global model with a predefined architecture and sending the global model to the client nodes. Local models may then be trained in respective client nodes (or distributed nodes) using the local data at the respective client node for some epochs. The model parameters of the local models may then be sent to the server and aggerated (e.g. by averaging in FedAvg) to generate a new global model. The new global model may then be communicated back to the client nodes, which can then resume training. This process may be repeated for a given number of global communication rounds.

[0007] FL leads to a shared global trained model that combines the knowledge from the client nodes without sharing raw data. The local models may be combined, typically by weighted averaging of parameters from the local models (which all share the same architectures). As the learning process is controlled by the server, it would be very beneficial to evaluate the global model from server during training.

[0008] In order to efficiently combine local models, voting mechanisms (see K. Yue, R. Jin, C. -W. Wong and H. Dai, "Federated Learning via Plurality Vote," in IEEE Transactions on Neural Networks and Learning Systems, doi: 10.1109 / TNNLS.2022.3225715) have been proposed to aggregate binary neural networks for federated learning. These voting mechanisms allow aggregation by setting each weight in the global model as the majority choice weighted by a credibility-score from the local models contribution to the federation.

[0009] It is possible to predict the performance of a Deep Neural Network (DNN) using only its weights (or simple statistics thereof) as inputs (see, for example, Thomas Unterthiner and Daniel Keysers and Sylvain Geliy and Olivier Bousquet and Ilya Tolstikhin, "Predicting Neural Network Accuracy from Weights, arXiv:2002.11448, 202f0). Unterthiner et al. investigated different image datasets and model architectures and shows that a model can be trained using CNN model weights to predict the accuracy of the model. The evaluation results in Unterthiner et al. illustrate that using only 7 numbers based on the statistics of weights per model layers (i.e., the mean, the variance, and 5 percentile values) as input features leads to most accurate evaluator model compared to using all model weights as input features.

[0010] Additionally, transfers to new architectures and datasets have been studied and it has been shown that these predictions can still rank networks trained on unobserved natural image datasets / with different large architectures (This approach is also disclosed in a patent application US20210256422A1 ). In the FL setting, the data exists on the client nodes and does not leave clients. However, it is challenging to evaluate the global model in the server. Sending sample data from clients to server completely breaks the privacy assumptions. Some existing methods communicate the client nodes' individual performances of some validation set so that the global model can be evaluated centrally. However, even sending model performance from client nodes to the server may be sensitive, for instance in the case of cross silo FL with multiple operators participating in the federation - sharing how well their models are doing may not lie in their interests.

[0011] A second approach for evaluation of the FL is based on the loss of the individual client nodes (agents). This approach requires client nodes to send their loss values at each round to the server. The server then terminates the learning once the average of the losses (or some other statistics of the losses) from all client nodes is converged. If the loss from a client node fluctuates around a saddle point, these fluctuations could slow down or disturb the convergence evaluation of the aggregated loss. Another problem with this approach arises where agents have access to limited data, or the data are not representative of the underlying distribution. In these cases, the loss from the local agents should not be used for the assessment of the model convergence. Given that FL is most relevant when agents have access to limited data, such an approach for evaluation of the model convergence is not applicable in real-world problems.

[0012] A third approach for evaluation of the FL is via monitoring the changes in the global weights (learnable parameters) at the server across rounds of learning. Once the change at several successive rounds is smaller than a threshold, the learning is terminated. Similar problems exist with this approach as in the case of the aggregated loss. If the model parameters (weights) from a client node fluctuates around a saddle point, these fluctuations could slow down or disturb the convergence evaluation of the aggregated global model. Another problem with this approach arises where client nodes have access to limited data, or the data is not representative of the underlying distribution. In such scenarios, client nodes' models may be easily overfitted to their local data. The resulting global aggregated model from all the local models is then naturally subject to inherited problems that arise from overfitted models. As the result, evaluation of the convergence of the global model based on the changes in the state of the model across rounds becomes unreliable.

[0013] One could also do global model selection based on voting mechanisms as described above. However, the problem with this approach is simply that it is only a method for aggregation and cannot be used for general evaluation - thus it cannot be used for hyperparameter tuning, early stopping or monitoring of the global model's performance. Furthermore, the voting mechanism for aggregation does not attempt to deal with the non-linearities of the loss function present in FL.

[0014] SUMMARY

[0015] According to some embodiments there is provided a method for managing a Distributed Learning, DL, process. The DL process is conducted between a server node and a plurality of distributed nodes, each distributed node having access to a local distributed node data set. The DL process is for developing a task Machine Learning, ML, model. The method, performed by a management node, comprises: for individual distributed nodes of at least a subset of the plurality of distributed nodes, receiving a representation of a local version of an evaluator ML model, wherein the evaluator ML model is configured to predict performance of the task ML model, and wherein the local version of the evaluator ML model has been trained using data from the local distributed node data set to which the distributed node has access; aggregating the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model; using the global version of the evaluator ML model to predict performance of a version of the task ML model; and initiating adjustment of at least one parameter of the DL process according to the predicted performance.

[0016] According to some embodiments there is provided a method for facilitating management of a Distributed Learning, DL, process. The DL process is conducted between a server node and a plurality of distributed nodes, each distributed note having access to a local distributed node data set. The DL process is for developing a task Machine Learning, ML, model. The method, performed by the server node, comprises: performing the DL process with the plurality of distributed nodes; providing a version of the task ML model to a management node; receiving from the management node information relating to a predicted performance of the version of the task ML model; and implementing an adjustment of at least one parameter of the DL process according to the received information.

[0017] According to some embodiments there is provided a method for facilitating management of a Distributed Learning, DL, process. The DL process is conducted between a server node and a plurality of distributed nodes, each distributed node having access to a local distributed node data set. The DL process is for developing a task Machine Learning, ML, model. The method, performed by a distributed node, comprises: performing the DL process with the server node; training a local version of an evaluator ML model using data from the local distributed node data set to which the distributed node has access, wherein the evaluator ML model is configured to predict performance of the task ML model; and providing a representation of the local version of the evaluator ML model to a management node.

[0018] According to some embodiments there is provided a computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein. The computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform any of the methods described above.

[0019] According to some embodiments there is provided a carrier containing the computer program as described above, wherein the carrier comprises one of an electronic signal, optical signal, radio signal or computer readable storage medium.

[0020] According to some embodiments there is provided a computer program product comprising non transitory computer readable media having stored thereon a computer program as described above.

[0021] According to some embodiments there is provided a computer-readable medium comprising instructions that, when executed on at least one processor, cause the at least one processor to perform any of the methods described above.

[0022] According to some embodiments there is provided a management node for or managing a Distributed Learning, DL, process. The DL process is conducted between a server node and a plurality of distributed nodes, each distributed node having access to a local distributed node data set. The DL process is for developing a task Machine Learning, ML, model. The management node comprises processing circuitry and a memory, the memory containing instructions executable by the processing circuitry whereby the management node is configured to: for individual distributed nodes of at least a subset of the plurality of distributed nodes, receive a representation of a local version of an evaluator ML model, wherein the evaluator ML model is configured to predict performance of the task ML model, and wherein the local version of the evaluator ML model has been trained using data from the local distributed node data set to which the distributed node has access; aggregate the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model; use the global version of the evaluator ML model to predict performance of a version of the task ML model; and initiate adjustment of at least one parameter of the DL process according to the predicted performance.

[0023] According to some embodiments there is provided a server node for facilitating management of a Distributed Learning, DL, process. The DL process is conducted between the server node and a plurality of distributed nodes, each distributed note having access to a local distributed node data set. The DL process is for developing a task Machine Learning, ML, model. The server node comprises processing circuitry and a memory, the memory containing instructions executable by the processing circuitry whereby the server node is configured to: perform the DL process with the plurality of distributed nodes; provide a version of the task ML model to a management node; receive from the management node information relating to a predicted performance of the version of the task ML model; and implement an adjustment of at least one parameter of the DL process according to the received information.

[0024] According to some embodiments there is provided a distributed node for facilitating management of a Distributed Learning, DL, process. The DL process is conducted between a server node and a plurality of distributed nodes comprising the distributed node, each distributed node having access to a local distributed node data set. The DL process is for developing a task Machine Learning, ML, model. The distributed node comprises processing circuitry and a memory, the memory containing instructions executable by the processing circuitry whereby the distributed node is configured to: perform the DL process with the server node; train a local version of an evaluator ML model using data from the local distributed node data set to which the distributed node has access, wherein the evaluator ML model is configured to predict performance of the task ML model; and provide a representation of the local version of the evaluator ML model to a management node.

[0025] Aspects and examples of the present disclosure thus provide methods and apparatuses that allow for training of an evaluator ML model in one or more distributed nodes in parallel to training a task ML model. The local versions of the evaluator ML model may then be communicated along with the representations of the local versions of the task ML model, and a global version of the evaluator ML model may be queried for decision making in the training of the task ML model, such as: weighing the contributing worker nodes to the task ML model in the aggregations, early stopping in the training of the task ML model, and reducing frequency of training rounds of the task ML model.

[0026] The embodiments described herein therefore allow for evaluation of the global task ML model in a data driven way. There is no need for sending data or reports about individual local versions of the task ML model performances to the server node. If the local versions belong to competing operators, they may not want to reveal how well they are doing - but sharing an evaluator ML model does not disclose the current performance in the client for use cases like early stopping and general evaluation of the global model, if combined with secure aggregation. This also helps with making better decisions in terms of finding the optimal aggregation method e.g., using weighted averaging or client selection at the server node, or early stopping of the task ML model training. It will be appreciated that, in many telecoms cases, privacy may be less of an issue anyway, and aggregation of the local task ML models to generate the global task ML model may be adjusted - which results in better performance of the global task ML model faster. However, even if you cannot adjust aggregation of the local task ML models to generate the global task ML model, improvements in time and performance may still be made from being able to evaluate the global task ML model.

[0027] Another advantage embodiments described herein provide is that several objectives can be evaluated during training. As an example, one evaluator ML model may be responsible for evaluation of the convergence, while another evaluator ML model is responsible for evaluation of the security risks. For example, when the parameter distribution of one client is significantly different from others, there might be a risk of security threats, and that client may be left out of the local ML model aggregation for the global task ML model (as it may be suspect for malicious activities e.g., to negatively impact the performance predictions of other clients etc.).

[0028] The evaluator ML model learned from one use case can also be transformed to another use case. The evaluator ML model itself is a model with learnable parameters. These learnable parameters from one use case can be transformed to another use case.

[0029] The evaluator ML model may be sent and received using the same message structures that are used for communicating task ML model weights, therefore there may be no need for defining additional communication messages between the distributed nodes and the server node with respect to the weights. However, differentiation between evaluator and task ML model weights is needed and can be achieved by amending existing messages with an indicator flag, not yet specified in e.g. 3GPP.

[0030] BRIEF DESCRIPTION OF THE DRAWINGS

[0031] For a better understanding of the embodiments of the present disclosure, and to show how it may be put into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0032] Figure 1 illustrates a system comprising a sever node, a management node and a plurality of distributed nodes;

[0033] Figure 2 illustrates a method of for managing a distributed Learning, DL, process, wherein the DL process is conducted between a server node and a plurality of distributed nodes;

[0034] Figure 3 illustrates a method of predicting performance of the task ML model utilizing the global version of the evaluator ML model;

[0035] Figure 4 illustrates a method for facilitating management of a DL process;

[0036] Figure 5 illustrates a method for facilitating management of a DL process;

[0037] Figure 6 is a signalling diagram illustrating an example implementation of one or more steps from the methods of Figures 2 to 5; Figure 7 is a signalling diagram illustrating an example implementation of one or more steps from the methods of Figures 2 to 5;

[0038] Figure 8 is a signalling diagram illustrating an example implementation of one or more steps from the methods of Figures 2 to 5 in which early stopping of the DL process is initiated;

[0039] Figure 9 is a signalling diagram illustrating an example implementation of one or more steps from the methods of Figures 2 to 5;

[0040] Figure 10 illustrates a management node comprising processor (or logic);

[0041] Figure 11 is a block diagram illustrating a management node;

[0042] Figure 12 illustrates a server node comprising processing circuitry (or logic);

[0043] Figure 13 is a block diagram illustrating a server node;

[0044] Figure 14 illustrates a distributed node comprising processing circuitry (or logic); and

[0045] Figure 15 is a block diagram illustrating a distributed node.

[0046] DETAILED DESCRIPTION

[0047] For the purposes of the present disclosure, the term "ML model” encompasses within its scope the following concepts:

[0048] Machine Learning algorithms, comprising processes or instructions through which data may be used in a training process to generate a model artefact for performing a given task, or for representing a real world process or system; the model artefact that is created by such a training process, and which comprises the computational architecture that performs the task; and the process performed by the model artefact in order to complete the task.

[0049] References to "ML model”, "model”, model parameters”, "model information”, etc., may thus be understood as relating to any one or more of the above concepts encompassed within the scope of "ML model”.

[0050] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0051] The following sets forth specific details, such as particular embodiments or examples for purposes of explanation and not limitation. It will be appreciated by one skilled in the art that other examples may be employed apart from these specific details. In some instances, detailed descriptions of well-known methods, nodes, interfaces, circuits, and devices are omitted so as not obscure the description with unnecessary detail. Those skilled in the art will appreciate that the functions described may be implemented in one or more nodes using hardware circuitry (e.g., analog and / or discrete logic gates interconnected to perform a specialized function, ASICs, PLAs, etc.) and / or using software programs and data in conjunction with one or more digital microprocessors or general purpose computers. Nodes that communicate using the air interface also have suitable radio communications circuitry. Moreover, where appropriate the technology can additionally be considered to be embodied entirely within any form of computer-readable memory, such as (ROM, EEPROM, Flash memory, a memory disc, RAM etc.) solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein.

[0052] Hardware implementation may include or encompass, without limitation, digital signal processor (DSP) hardware, a reduced instruction set processor, hardware (e.g., digital or analogue) circuitry including but not limited to application specific integrated circuit(s) (ASIC) and / or field programmable gate array(s) (FPGA(s)), and (where appropriate) state machines capable of performing such functions.

[0053] Certain aspects of the present disclosure and their embodiments may provide solutions to these or other challenges. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step.

[0054] Particular embodiments are described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject matter disclosed herein. The disclosed subject matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0055] An evaluator ML model may be considered to comprise an ML model that takes as an input the weights and / or statistics another ML model (e.g. a task ML model), and aims to predict something regarding the other ML model, typically, model performance in terms of some metric (the area under the receiver operating characteristic curve (AUG), the coefficient of determination (R2), mean squared error (MSE), etc), but, can be anything that is the function of the weights of the other ML model being evaluated.

[0056] A task ML model comprises an ML model for performing a specific task (other than model evaluation), for instance predicting a Key Performance Indicator (KPI) of a service. It will be appreciated that an evaluator ML model may be utilised to evaluate the model performance or another metric (e.g. security) of a task ML model.

[0057] Embodiments described herein provide methods and apparatuses for training an evaluator ML model in distributed nodes. The evaluator ML model may be able to predict performance of a task ML model. The evaluator ML model may be trained in a distributed learning, DL, manner in parallel to the task ML model. A representation of the evaluator ML model may then be communicated to a server node along with a representation of the task ML model, and in the server node, the evaluator ML model may be queried to assist in decision making in the training process. For example, the predicted performance of a version of the task ML model may be used to assist in decisions regarding one or more of: weighting the contributing distributed nodes in the aggregations of the task ML model (e.g. by adjusting their respective credibility scores if a voting mechanism is used or adjusting their weighting if weighted averaging is utilized), early stopping of the training, and reducing a frequency of rounds of training.

[0058] In some embodiments, methods and apparatuses for generating an evaluator ML model are provided.

[0059] In particular, the evaluator ML model may be utilized to evaluate the performance of a version of a task ML model where the task ML model is being developed using an DL process.

[0060] The generation of the evaluator ML model may utilize another DL process, for example, in parallel to the DL process being used to develop the task ML model. The output of the evaluator ML model may then be utilized to initiate an adjustment to the DL process being used to develop the task ML model.

[0061] Figure 1 illustrates a system 100 comprising a sever node 101 a management node 102 and a plurality of distributed nodes 103a to 103c. In this example system federated learning, FL, is being used as an example of a computer-implemented method of performing distributed learning. It will however be appreciated that other forms of distributed learning, DL, may be performed by a system implementing the embodiments described herein. In particular, it will be appreciated that the levels of data privacy or security implied by FL may not be required by the DL process of the embodiments described herein.

[0062] In this example system 100, the server node 101 and the plurality of distributed nodes 103a to 103c are performing a first DL process to generate a global version of the task ML model 104. Each distributed node 103a to 103c trains its own respective local version of the task ML model 105a to 105c and representations of these are transmitted to the server node 101 , which aggregates the local versions of the task ML models 105a to 105c to generate the global version of the task ML model 104.

[0063] The management node 102 is performing a second DL process to generate a global version of an evaluator ML model 106. Each distributed node 103a to 103c trains its own respective local version of the evaluator ML model 107a to 107c and representations of these are transmitted to the server node 101. In this example, all of the plurality of distributed nodes 103a to 103c are performing the second DL process, however, it will be appreciated that a subset of the plurality of distributed nodes 103a to 103c may be used.

[0064] A local version of the evaluator ML model 107a to 107c may, for example, take as an input parameters from the local version of the task ML model 105a to 105c and may predict the performance of the local version of the task ML model, e, where 0 comprises the parameters of the local version of the task ML model (such as weight matrices and bias vectors), <p is the parameters of the local version of the evaluator ML model, and e is the estimated performance (e.g. KPI) in the unit of the chosen metric. During the DL process, the distributed nodes 103a to 103c may transmit <p together with 0 to the management node.

[0065] It will be appreciated that a representation of a ML model (either the task ML model or evaluator ML model) may comprise the parameters of the ML model, the ML model itself, or any other suitable representation of the ML model that may be utilised by the node receiving the representation appropriately. A representation of an ML model may for example include details of the architecture of the model and values for its trainable parameters, for example number of hidden layers, number of neurons per layer, activation function of neurons and / or weights of neurons.

[0066] In this example system the management node 102 and the server node 101 are illustrated as separate nodes. However, it will be appreciated that these nodes may be co-located or may form part of the same logical entity.

[0067] It will also be appreciated that the effect of the output of the global evaluator ML model is not illustrated in Figure 1. The output of the global version of the evaluator ML model may be used for many things, for example, general monitoring of the health of the global version of the task ML model, selection of the plurality of distributed nodes, adjustment of the weighting applied to the local versions of the task ML model generated by the different distributed nodes (this has great potential but can potentially remove some of the privacy benefits if not implemented carefully), early stopping of the training of the task ML model, hyperparameter selection etc.

[0068] In some examples, in addition to the global version of the task ML model 104, the global version of the evaluator ML model 106 may also be stored at the server node 101, and local version of the evaluator ML model may be communicated between the distributed nodes and the server node 101.

[0069] It will be appreciated that the local versions of the evaluator ML models may not need to be communicated in every round of the federation learning or by all the participating distributed nodes in the DL process generating the task ML model.

[0070] The embodiments described herein comprise transmitting one extra model from the distributed nodes to the management node or the server node. This increases communication cost proportionally to the size of the local versions of the evaluator ML model. The evaluator ML model does not have to be very large, as it does not need to take all weights of the task ML model as an input. In some examples, the evaluator ML model may use statistics of the task ML model weights, or the weights as such, as an input.

[0071] However, it will also be appreciated that as embodiments described herein have the potential to speed up the training of the task ML model, they may reduce rounds of training (thereby reducing the level of communication cost). Since the evaluator ML model is transferable, a pre-trained evaluator ML model may be used for a new task and only fine-tuned using the proposed embodiments.

[0072] In some embodiments, the evaluator ML model is not updated or trained in every round of training the task ML model. It will be appreciated that the evaluator ML model may typically be much smaller than the task ML model. This is because it only requires a few weight statistics per task ML model layers as input features, the model itself does not need to be deep as learning the predicted performance from weight statistics may be a simple task. If the evaluator ML model is used for selection of distributed nodes for involvement in the first DL process, it will naturally save communication cost by de-selecting some of the participating distributing nodes in the aggregation of the task ML model. Of course, the embodiments described herein may be combined with any existing methods for compressing communication, truncating weights and so on.

[0073] In some examples, the server node and the management node may be comprised within a centralised Network Data Analytics Function (NWDAF). Each distributed node may comprise an NWDAF for a different geographical area.

[0074] In some examples, in a Open Radio Access Network (O-RAN) scenario, the server node and the management node may be comprised within a Service Management and Orchestration (SMO) unit which can be seen as a centralized server. Each distributed node may then comprise an extended Application (xApp).

[0075] Figure 2 illustrates a method of for managing a distributed Learning, DL, process, wherein the DL process is conducted between a server node (e.g. server node 101 of Figure 1) and a plurality of distributed nodes (e.g. nodes 103a to 103c of Figure 1). Each of the plurality of distributed nodes has access to a local distributed node data set. The DL process is for developing a task Machine Learning, ML, model.

[0076] It will be appreciated that the DL process may comprise any suitable DL process, for example Federated Learning. In particular, the DL process may comprise a ML process in which local task ML models are generated in distributed nodes, and knowledge from those local task ML models is transferred to other nodes for use in developing the task ML model.

[0077] The method of Figure 1 may be performed by a management node (e.g. management node 102 of Figure 1). The management node may comprise a physical or virtual node, and may be implemented in a computing device or server apparatus and / or in a virtualized environment, for example in a cloud, edge cloud or fog deployment. It will be appreciated that in some examples the management node comprises, is comprised within, or is co-located with, the server node 101.

[0078] In some examples, the server node is implemented in a core network node or Radio Access Network (RAN) or Open RAN (O-RAN) network node of a telecommunications network. For example, the server may be implemented in a Network Data Analytics Function (NWDAF).

[0079] In some examples, individual distributed nodes of the plurality of distributed nodes may be implemented in at least one of: a core network node of a telecommunications network; a RAN or O-RAN network node of a telecommunications network; a wireless device; a vehicle; an industrial or commercial device; an eXtended / Augmented / Virtual Reality (XR / AR / VR) device; and a virtual node.

[0080] In some examples, the method comprises step 205 which comprises selecting one or more distributed nodes from the plurality of distributed nodes for inclusion in a subset. The one or more distributed nodes may be selected such that performance of the global version of the task ML model is maximized.

[0081] It will be appreciated that the at least a subset of the plurality of distributed nodes may comprise anything from only one of the distributed nodes involved in the DL process developing the task ML model to all of the distributed nodes involved in the DL process developing the task ML model. In some examples, the management node may not need to make such a selection, but may, for example, only receive local versions of the evaluator ML model (as will be described with reference to step 210) from those distributed nodes that are within the subset.

[0082] In step 210, for individual distributed nodes of at least a subset of the plurality of distributed nodes, the method comprises receiving a representation of a local version of an evaluator ML model. It will be appreciated that each representation of a local version of the evaluator ML model may be received directly from the respective individual distributed node, or may be received via some intermediate node. The evaluator ML model is configured to predict performance of the task ML model (for example as will be described in more detail with reference to Figure 3), and the local version of the evaluator ML model has been trained using data from the local distributed node data set to which the distributed node has access. It will also be appreciated that the local versions of the evaluator ML model may be received at the management node via a server node, some other intermediate node, or they may be received directly from the respective digstributed nodes.

[0083] In step 220, the method comprises aggregating the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model. The local versions of the evaluator ML model may be transferred or signaled, for example using existing model formats such as Open Neural Network Exchange (ONNX), or formats of commonly used toolboxes such as Keras or Pytorch. In general, an ML model could be signaled using a high-level model description, potentially together with more detailed information. High-level model description (model parameter vector) may for example comprise parameters defining the structure and characteristics of the model, such as for example number of layers, activation function of respective layer, nature of connections between nodes of respective layer, weights, loss function, just to mention a few, of a neural network.

[0084] Alternatively, a ML model can be signaled using container-based signalling, for example using a containerized image.

[0085] Step 220 may comprise aggregating the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model comprises using an aggregation function to generate the global version of the evaluator ML model. The aggregation function may comprise a FedAvg, FedProx or any other suitable aggregation function.

[0086] In step 230, the method comprises using the global version of the evaluator ML model to predict performance of a version of the task ML model. It will be appreciated that step 230 may comprise using the global version of the evaluator ML model to predict performance of a global version of the task ML model or one or more local versions of the task ML model, or both.

[0087] In step 240, the method comprises initiating adjustment of at least one parameter of the FL process according to the predicted performance. In performing step 240 the management node may directly adjust or instruct adjustment of the at least one parameter. For example, the management node may be capable of determining an appropriate adjustment to the at least one parameter given the predicted performance and may then be able to cause the at least one parameter to change, for example by transmitting instructions to the plurality of distributed nodes and / or the server node. In other examples, step 240 comprises the management node transmitting the predicted performance to the server node. The server node may then be capable of determining and causing adjustment to the at least one parameter (e.g. by implementing the adjustment at the server node or instructing one or more distributed nodes to do so). It will be appreciated, that by transmitting the predicted performance to the server node, the management node effectively initiates the adjustment to the at least one parameter.

[0088] For example, step 240 may comprise initiating an adjustment of an aggregation parameter for generating a global version of the task ML model. For example, in a weighted aggregation of a federation, instead of weighing all local versions of the task ML model equally, weights may be selected for aggregating the task ML model so that the resulting global version of the task ML model maximises or improves the performance predicted by the global model evaluator. Step 240 may comprise initiating adjustment of a number of federation rounds for the DL process. For example, if the task ML model is performing well or better than expected given the number of federation rounds that have already been performed, the number of federation rounds may be reduced (or vice versa).

[0089] Step 240 may comprise initiating adjustment of a number of local training rounds performed in each federation round of the FL process. For example, if the task ML model is performing well or better than expected given the number of federation rounds that have already been performed, the number of training rounds performed in each federation round may be reduced (or vice versa).

[0090] Step 240 may comprise initiating adjustment of the plurality of distributed nodes included in the DL process (e.g. the selection of the plurality of distributed nodes to be included in the DL process may be altered). For example, if the evaluator ML model is being utilized to predict performance of local versions of the task ML model, the results of the evaluation may be utilized to exclude distributed nodes from the DL process if the local version of the task ML model they are produces is performing poorly.

[0091] Utilizing the evaluator ML model to select the plurality of distributed nodes for inclusion in the DL process, could in some embodiments, lead to the ML evaluator model being able to simply predict the model performance of each participating Task ML model. This method is not very strict with regards to privacy since performance of a single local version of the task ML model can be evaluated and will return something that is close to the global task ML model performance, but it is still more private than actually sending the raw numbers. This method also has the benefit, compared to sending the raw model validation performances, that it takes into account the aggregated model and not only the sum of all contributing models.

[0092] However, if more network overhead is permitted, and secure aggregation can be utilized, it may be possible to utilise the evaluator ML model to select the plurality of distributed nodes in an anonymized fashion. In secure aggregation noise is added to each local model, but the sum of all the local models (e.g. the global model) cancels out the noise. To utilise secure aggregation in embodiments described herein therefore, it may be required to send masked local versions of the task ML models for each way of combining of weights to be evaluated.

[0093] Step 240 may comprise initiating adjustment of a trigger for retraining of the task ML model. For example, the training of the task ML model may be re-started from scratch if the predicted performance is too low (maybe due to concept drift).

[0094] Step 240 may comprise initiating adjustment of an amount of training to do in a local training round. For example, the amount of training in a round could vary from one update with a single training sample to any number of epochs. Typically, the amount of training may range from only a few batches to many epochs.

[0095] The method of Figure 2 enables data driven evaluation of an DL process, which supports faster convergence and improved performance of the global version of the task ML model. In the performance of the method of Figure 1 , these advantages are achieved without compromising privacy, and without the need to transfer data or reports about individual model performance. The improvements to the management of the DL process may be realized in different ways, depending on what version(s) of the task model are being evaluated in step 230, but improved model performance and faster convergence can be achieved with or without privacy protection, which saves resources for both the transfer of data and for computation. In some examples, the method of Figure 2 further comprises in step 250 sending the global version of the evaluator ML model to one or more distributed nodes in the plurality of distributed nodes.

[0096] It will be appreciated that the one or more distributed nodes may then continue to train and / or improve the evaluator model locally prior to a further round of the second DL process training the evaluator ML model. It will also be appreciated that the management node may send the global version of the evaluator ML model to, for example, the subset of distributed nodes or to all of the plurality of distributed nodes. The one or more distributed nodes may update the global version of the evaluator ML model by retraining it using their local data set.

[0097] In some examples, the local version of the evaluator ML model has also been trained using a local version of the task ML model. For example, local training of the local version of the evaluator ML model may be performed by comparing performance predicted by the evaluator ML model to an actual performance of the local version task ML model when used in the distributed node.

[0098] It will be appreciated that the method of Figure 2 may be repeated (for example, substantially in parallel) for multiple different evaluator ML models, where each evaluator ML model may be evaluating a different aspect (e.g. different model performance factor) of the task ML model. The different aspects may for example comprise one or more of: a rate of convergence, a security risk, or an amount of model poisoning by a malicious agent. The different evaluator ML models may receive inputs from each other.

[0099] It will be appreciated that the method of Figure 2 may be repeated until satisfaction of a termination condition. The termination condition may comprise an early stopping condition, a condition that all versions of the task ML model that are relevant have been evaluated in step 230, a condition that no more resources available, a condition that the predicted task ML model performance satisfies a criterion.

[0100] Figure 3 illustrates a method of predicting performance of the task ML model utilizing the global version of the evaluator ML model. The method of Figure 3 may be performed by the management node performing the method of Figure 2. It will be appreciated that the method of Figure 3 may be considered to comprise an example implementation of step 230 of Figure 2.

[0101] In some examples the method of Figure 3 may commence with step 305 which comprises selecting aversion of the task ML model for performance evaluation according to a criterion. In other examples, the version of the task ML model may be indicated by or received from the server node (in other words, the server node may perform step 305 and may indicate the version of the task ML model to the management node). It will be appreciated that the management node may request the version of the task ML model to evaluate from the server node.

[0102] The criterion may comprise at least one of: an availability of local versions of the task ML model; one or more previous evaluations of versions of the task ML model; an indication of substandard performance of at least one version of the task ML model; a particularity of a local data set to which a distributed node has access.

[0103] If the local versions of the task ML model are available, for example, if there are no specific privacy concerns in the network that prevent making the local version fully available to a centralized function such as the server node or the management node, then the management node may select (e.g. sequentially) all or some of the local versions of the task ML model.

[0104] If not all local versions of the task ML model are to be evaluated, then there may be a focus on those for which there is some doubt as to their performance of relative to others (i.e., if the local data set is skewed somewhat compared to others). The global version of the task ML model may also, or alternatively, be selected for evaluation, for example if local versions of the task ML model are not available, or to support early stopping of DL for the task ML model, or to evaluate the general health of the global version of the task ML model.

[0105] It will be appreciated that the distributed nodes do not have to disclose the local ML model performance of their local Ml models to the server node / management node. However, if the server node has access to both raw models (in their original format) and high quality model evaluators, the server node will be able to estimate the performance of the distributed node's local versions of the task ML model. However, there are ways to perform federated learning in a privacy preserving manner such as utilising secure aggregation, which masks the individual model weights so that it is not possible to deduct how an individual distributed node's model looks, and only the sum of all the local models is preserved.

[0106] By using techniques like secure aggregation privacy of the performances of individual distributed node's local versions of the task ML model may be preserved while still using the proposed method for use cases such as general evaluation of the global model, early stopping and similar.

[0107] In step 310 the method comprises receiving as input a representation of the version of the task ML model. The input representation of the version of the task ML model may comprise, for example, values of trainable parameters of the version of the task ML model. As described above it will be appreciated that the version of the task ML model referred to in step 310 may be a local version of the task ML model or the global version of the task ML model.

[0108] In step 320, the method comprises processing the input representation according to the global version of the evaluator ML model.

[0109] In step 330, the method comprises outputting a predicted value of a performance parameter for the task ML model.

[0110] In some examples, the management node performing the methods of Figures 2 and 3 comprises at least some of the functionality of the server node. In these examples, the method of Figure 2 may further comprise one or more of: conducting the DL process for developing the task ML model with the plurality of distributed nodes, and sending the global task ML model to the plurality of distributed nodes.

[0111] It will be appreciated that different subunits in the same logical entity (e.g. the management node) may be conducting the DL process, model evaluation, and DL of the evaluator ML model.

[0112] Figure 4 illustrates a method for facilitating management of a DL process, wherein the DL process is conducted between a server node and a plurality of distributed nodes, each distributed note having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model. The method of Figure 4 is performed by the server node. In these embodiments, the DL process is performed by the server node and the methods of Figures 2 and 3 are performed by a separate management node. The server node (e.g. server node 101 of Figure 1) may comprise a physical or virtual node, and may be implemented in a computing device or server apparatus and / or in a virtualized environment, for example in a cloud, edge cloud or fog deployment.

[0113] In step 410 the method comprises performing the DL process with the plurality of distributed nodes. It will be appreciated that the DL process may comprise an FL process. Performing the DL process may comprise receiving local versions of the task ML model from the distributed nodes, aggregating the local versions of the task ML model to generate a global version of the task ML model, and providing the global version of the task ML model to the distributed nodes. The FL process may be repeated for a number of federation rounds or until some other stopping condition is met.

[0114] In step 420 the method comprises providing a version of the task ML model to a management node. As described above it will be appreciated that the version of the task ML model referred to in step 420 may be a local version of the task ML model or the global version of the task ML model.

[0115] In step 430 the method comprises receiving from the management node information relating to a predicted performance of the version of the task ML model. It will be appreciated that step 403 may comprise receiving the information directly from the management node, or via some intermediate node. Step 430 may be considered an example implemented of step 240 of Figure 2. As described above with reference to step 240, the received information may comprise the predicted performance itself, or an instruction generated by the management node on the basis of the predicted performance.

[0116] In step 440 the method comprises implementing an adjustment of at least one parameter of the DL process according to the received information.

[0117] Figure 5 illustrates a method for facilitating management of a Federated Learning, DL, process, wherein the DL process is conducted between a server node and a plurality of distributed nodes, each distributed note having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model. The method of Figure 5 is performed by one of the distributed nodes. The distributed node may comprise any one of the distributed nodes 103a to 103c of Figure 1 and may comprise a physical or virtual node, and may be implemented in a computing device or server apparatus and / or in a virtualized environment, for example in a cloud, edge cloud or fog deployment.

[0118] Step 510 comprises performing the DL process with the server node. It will be appreciated that the DL process may comprise an FL process. Performing the DL process may comprise transmitting a local versions of the task ML model to the server node, and receiving a global version of the task ML model from the server node. The DL process may be repeated for a number of federation rounds or until some other stopping condition is met.

[0119] Step 520 comprises training a local version of an evaluator ML model using data from the local distributed node data set to which the distributed node has access, wherein the evaluator ML model is configured to predict performance of the task ML model.

[0120] Step 520 may comprise training the local version of the evaluator ML model using the local version of the task ML model. For example, the local training of the local version of the evaluator ML model may comprise comparing the performance predicted by the evaluator ML model to an actual performance of a local version task ML model as used in the distributed node.

[0121] Step 530 comprises providing, for example by means of transmitting, a representation of the local version of the evaluator ML model to a management node.

[0122] In some examples, the method of Figure 5 may further comprise step 540 which comprises receiving a representation of a global version of the evaluator ML model from the management node.

[0123] It will be appreciated that steps 530 and 540 may be performed either directly to / from the management node or via the server node.

[0124] Figure 6 is a signalling diagram illustrating an example implementation of one or more steps from the methods of Figures 2 to 5. The signalling in this example occurs between a first distributed node 103a, a second distributed node 103b, a server node 101 and a management node 102.

[0125] In step 601 , a first distributed node 103a transmits a first representation of a first local version of the evaluator ML model to the management node 102. It will be appreciated that the first local version of the evaluator ML model will have been trained utilizing local data to the first distributed node 103a. Step 601 comprises an example implementation of step 210 of Figure 2 and step 530 of Figure 5.

[0126] In step 602, the second distributed node 103b transmits a second representation of a second local version of the evaluator ML model to the management node 102. It will be appreciated that the second local version of the evaluator ML model will have been trained utilizing local data to the second distributed node 130b. Step 602 comprises an example implementation of step 210 of Figure 2 and step 530 of Figure 5.

[0127] It will be appreciated that steps 601 and 602 may be performed for every round of training of the task ML model, or independently of the task ML model training, such as only shared every N rounds, or the frequency of steps 601 and 602 may be fully independent of the frequency of trai ning of the task ML model.

[0128] The datasets used for training the evaluator ML model would be at least a subset of the task ML model weights (or some statistics derived from at least a subset of the task ML model weights) and corresponding performance on a validation dataset in each distributed node. Note that as the evaluator ML model typically does not use all weights of a model as input, there may be some meta-data regarding what parameters are used and how. This meta-data does not have to be included in every round, but may be initialized in the beginning, and if something changes within the meta-data the update may be indicated to the management node.

[0129] In step 603 the management node aggregates the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model. Step 603 comprises an example implementation of step 220 of Figure 2. Step 603 may comprise utilising FedAvg, FedProx, or any other suitable aggregation process.

[0130] In step 604 the server node 101 selects a version of the task ML model to evaluate. Step 604 comprises an example implementation of step 305 of Figure 3. In step 605, the server node 101 indicates the selected version of the task ML model to the management node 102. Step 605 comprises an example implementation of step 310 of Figure 3 or step 420 of Figure 4. As described above it will be appreciated that the version of the task ML model referred to in step 604 and 605 may be a local version of the task ML model or the global version of the task ML model.

[0131] In step 606, the management node evaluates the version of the task ML model using the global version of the evaluator ML model generated in step 603. Step 606 comprises an example implementation of step 230 of Figure 2 or step 320 of Figure 3. The resulting predicted performance of the version of the task ML model is stored or communicated to the server node 101 in step 607.

[0132] In step 608, the server node determines whether any other versions of the task ML model should be evaluated. This determination depends on what criteria are used in step 604 to select the version of the task ML model for evaluation.

[0133] For example, if the aim is to do a grid search, more versions of the task ML model should be evaluated until all evaluated configurations (for instance weights in FedAvg) are evaluated. If more models should be evaluated, go back to step 604 and change some of the configurations, for instance the weights in the FedAvg.

[0134] If it determined in step 608 that there is at least one further version of the task ML model is to be evaluated, the method returns to step 604 another version of the task ML model is selected. It will be appreciated that 604 to 607 may be repeated for any number of versions of the task ML model that are determi ned to be evaluated. It will be appreciated that step 607 comprises an example implementation of step 240 of Figure 2. In other words, by transmitting the predicted performances to the server node 101 , the management node 102 is initiating adjustment of the last one parameter of the DL process.

[0135] If in step 608 it is determined that there are no further versions of the task ML model to be evaluated the method passes to step 609.

[0136] In step 609 the server node implements an adjustment of at least one parameter of the DL process according to the received predicted performance (e.g. received in step 607). Step 609 comprises an example implementation of step 440 of Figure 4.

[0137] Step 609 may for instance comprise selecting the weights of the FedAvg for the task ML model that maximizes the evaluated performance of the global version of the task ML model, but could also be a decision to stop the training of the federation early or ask a particular distributed node to stop contributing to the federation due to poor performance of the local version of the task ML model that it produces.

[0138] In step 610 the management node transmits the global version of the evaluator ML model to the first distributed node and the second distributed node. Step 610 comprises an example implementation of step 250 of Figure 2. It will be appreciated that step 610 may be performed at any time after the performance of step 603. It will be appreciated that the process of Figure 6 may be repeated for every round of training of the evaluator ML model. It will also be appreciated that the information sent between the distributed nodes and the server node / management node is the local versions of the evaluator ML model along with the local versions of the task ML model, compared to only sending the local versions of the task ML model as in traditional federated learning.

[0139] It will also be appreciated that the process of Figure 6 may be repeated for every round until training of the task ML model is completed, something that may potentially be determined by the global version of the evaluator ML model.

[0140] Figure 7 is a signalling diagram illustrating an example implementation of the methods of one or more of the steps from the methods of Figures 2 to 5. The signalling in this example occurs between a first distributed node 103a, a second distributed node 103b, a server node 101 and a management node 102.

[0141] Steps 701 to 703 correspond to steps 601 to 603 of Figure 5.

[0142] In step 704 the server node 101 transmits one or more versions of the task ML model for evaluation. Step 704 comprises an example implementation of step 420 of Figure 4 and / or step 310 of Figure 3. As described above it will be appreciated that one or more versions of the task ML model referred to in step 704 may be one or more local versions of the task ML model or the global version of the task ML model.

[0143] In step 705, the management node selects a version of the task ML model to evaluate. Step 705 comprises an example implementation of step 305 of Figure 3.

[0144] In step 706, the management node evaluates the version of the task ML model using the global version of the evaluator ML model generated in step 703. Step 706 comprises an example implementation of step 230 of Figure 2 or step 320 of Figure 3. The resulting predicted performance of the version of the task ML model is stored for later use (e.g. for use in in step 708).

[0145] In step 707, the management node determines whether any other versions of the task ML model should be evaluated. This determination depends on what criteria are used in step 707 to select the version of the task ML model for evaluation.

[0146] If in step 707 it is determined that there is at least one further version of the task ML model is to be evaluated, the method returns to step 705 and another version of the task ML model is selected for evaluation. It will be appreciated that the steps 705 and 706 may be repeated for any number of versions of the task ML model that are determined to be evaluated.

[0147] If in step 707 it is determined that there are no further versions of the task ML model to be evaluated, the method passes to step 708.

[0148] In step 708 the management node determines an adjustment of at least one parameter of the DL process according to the predicted performances (e.g. received in step 706).

[0149] Step 708 may for instance comprise selecting the weights of the FedAvg for the task ML model that maximize the evaluated performance of the global version of the task ML model, but could also be a decision to stop the training of the federation early or ask a particular distributed node to stop contributing to the federation due to poor performance of the local version of the task ML model that it produces. In step 709 the management node then initiates the adjustment of the at least one parameter by transmitting an indication of the determination made in step 708 to the server node 101 . Step 709 comprises an example implementation of step 240 of Figure 2.

[0150] In step 710 the management node transmits the global version of the evaluator ML model to the first distributed node and the second distributed node. Step 710 comprises an example implementation of step 250 of Figure 2. It will be appreciated that step 710 may be performed at any time after the performance of step 703.

[0151] It will be appreciated that the process of Figure 7 may be repeated for every round of training of the evaluator ML model. It will also be appreciated that the information sent between the distributed nodes and the server node / management node is the local versions of the evaluator ML model along with the local versions of the task ML model, compared to only sending the local versions of the task ML model as in traditional federated learning.

[0152] It will also be appreciated that the process of Figure 7 may be repeated for every round until training of the task ML model is completed, something that may potentially be determined by utilising the global version of the evaluator ML model as described with reference to Figure 8 below.

[0153] Figure 8 illustrates an example implementation of one or more steps from the methods of Figures 2 to 5 in which early stopping of the DL process is initiated. The signalling in this example occurs between a first distributed node 103a, a second distributed node 103b, a server node 101 and a management node 102.

[0154] The training of the task ML model is commenced at round i in step 802, where i is initialized at 1 in step 801 .

[0155] Steps 803 to 805 correspond to steps 601 to 603 of Figure 6. Step 806 corresponds to step 610 of Figure 6. It will be appreciated that the steps 803 to 805 may occur in every round of training, i, or may occur more or less frequently.

[0156] In step 807 of Figure 8, the server node transmits a global version of the task ML model generated at training round i to the management node for evaluation. Step 807 comprises an example implementation of step 605 of Figure 6.

[0157] Steps 808 and 810 comprise an example implementation of steps 606 and 607 of Figure 6. In each round of training, step 808 may utilise the most recently generated global version of the evaluator ML model.

[0158] In step 811 , the server node determines, when the training round i has been completed, whether the predicted performance received in step 810 indicates that the performance of the global version of the task ML model is improved from the previous round of training.

[0159] If in step 811 , the server node determined that the predicted performance has improved from the previous round of training the method passes to step 809 in which the value of i, is incremented to i+1. The method then passes back to step 802 and a new round of training is commenced.

[0160] If in step 811 , the server node determines that the predicted performance has not improved from the previous round of training the method passes to step 812. As the performance of the global version of the task ML model has not improved from the previous round of training, the server node determines that the DL process should be stopped early in step 813.

[0161] Figure 9 illustrates an example implementation of one or more steps from the methods of Figures 2 to 5. In this example, the evaluator ML model is utilised to adjust a weight configuration utilised in the aggregation of the local versions of the task ML model.

[0162] Steps 901 to 903 correspond to steps 601 to 603 of Figure 6.

[0163] In step 904, the server node generates the global version of the task ML model utilising a weight configuration, i, where i = [1 , ...., Z] where there are Z weight configurations to be evaluated for aggregating the local versions of the task ML model to generate the global version of the task ML model. In step 905, the server node transmits the global version of the task ML model generated in step 904 to the management node.

[0164] In step 906, the management node evaluates the global version of the task model received in step 905 utilising the global version of the evaluator ML model generated in step 903. Step 906 determines a predicted performance of the global version of the task ML model received in step 905.

[0165] In step 907, the management node transmits the predicted performance determined in step 906 to the server node.

[0166] In step 908, the server node determines whether there are one or more weight configurations that have not yet been evaluated. If in step 908 it is determined that there are one or more weight configurations that have not yet been evaluated, the method passes to step 909 in which i is incremented such that i = i+1. The method then passes back to step 904.

[0167] If in step 908 it is determined that there are no more weight configurations for the aggregation of the task ML model to evaluate, the method passes to step 910 in which it is determined to utilise the weight configuration that resulted in a best predicted performance to generate the global version of the task ML model to transmit to the distributed nodes in step 911 .

[0168] It will be appreciated that the embodiments described herein may be utilised in any scenario in which distributed learning (e.g. federated learning) is implemented. For example, it will be appreciated that DL may be used for a number of different tasks in a telecommunications network (e.g. power consumption prediction, blocked call rate prediction as described below). The embodiments described herein may therefore be implemented generally in a telecommunications network where the distributed nodes comprise for example UEs and / or base stations and / or distributed units of a base station.

[0169] DL may be used to provide a task ML model that predicts a blocked call rate based on performance management (PM) counters in a telecommunications network. In this example, the server node may comprise or be comprised in a third party node that manages the network for various operators, the management node may comprise or be comprised in the third party node; and the plurality of distributed nodes may comprise of be comprised in user equipment, UEs, such as smart phones. DL may be used to provide a task ML model that predicts service level Key Performance Indicators (KPIs) for example, from cloud infrastructure statistics (for example, read time of distributed Key-value-store (KvS), framerate, audio buffer rate, Mean opinion score of Video on Demand (VoD) service, etc). This can be done in many ways, as typically the labels and the input features are not generated in the same place. This can for example be facilitated by split learning. The distributed nodes in this example may comprise, for example, a data centre, and the server node and / or management node may be another data centre. Assuming that the labels are sent to wherever the service is being served from, for example, an "edge data centre” that exists close to a user for the purpose of serving this (and possibly other) application. The edge datacentre may be considered a distributed node and a (e.g. larger) central data centre may then comprise the server node and / or management node.

[0170] DL may be used to provide a task ML model that predicts power consumption in base stations based on historical data. In this example, the server node may comprise or be comprised in a data centre, the management node may comprise or be comprised in a data centre; and the plurality of distributed nodes may comprise or be comprised in a base station.

[0171] DL may be used to provide a task ML model that detects malicious activities based on input data. In this example, the server node may comprise or be comprised in a data centre, the management node may comprise or be comprised in a data centre; and the plurality of distributed nodes may comprise or be comprised in, for example, a phone, a base station, an XR device or any other user or edge device.

[0172] DL may be used to provide a task ML model that tracks eye movements based on information received from sensors. In this example, the server node may comprise or be comprised in data centre, the management node may also comprise or be comprised in a data centre; and the plurality of distributed nodes may comprise or be comprised in user devices such as Augmented Reality or Virtual Reality (AR / VR) glasses.

[0173] DL may be used to provide a task ML model that detects objects for example in an extended reality application based on, for example based on input data from a camera. For example, such a task ML model may be required to change how an object looks in XR in a safe manner (e.g. to make sure it does not make a dangerous object look safe). In this example, the server node may comprise or be comprised in a data centre, the management node may also comprise or be comprised in a data centre; and the plurality of distributed nodes may comprise or be comprised in a device comprising a camera.

[0174] DL may also be used to provide a task ML model that detects whether one or more devices are out of sync based on one or more of: data regarding the state of the one or more devices and sensor input data and movement of the user and other user behavior. For example, such a task ML model may detect that a VR device such as glasses is not working, resulting in nausea at the user, and may initiate the device preemptively stopping in response to such a detection. In this example, the server node may comprise or be comprised in a data centre, the management node may comprise or be comprised in a data centre; and the plurality of distributed nodes may comprise or be comprised in a VR device.

[0175] Figure 10 illustrates a management node 1000 comprising processing circuitry (or logic) 1001. The processing circuitry 1001 controls the operation of the management node 1000 and can implement the method described herein in relation to a management node 1000. The processing circuitry 1001 can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the management node 1000 in the manner described herein. In particular implementations, the processing circuitry 1001 can comprise a plurality of software and / or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein in relation to the management node 1000. It will be appreciated that the management node 1000 may comprise one or more virtual machines running different software and / or processes. The management node 1000 may therefore comprise, or be implemented in or as one or more servers, switches and / or storage devices and / or may comprise cloud computing infrastructure that runs the software and / or processes.

[0176] Optionally, the management node 1000 may comprise a memory 1003. In some embodiments, the memory 1003 of the management node 1000 can be configured to store instructions (e.g. program code) executable by the processing circuitry 1001 of the management node 1000 whereby the management node is configured to perform the method as described herein with reference to a management node.

[0177] Alternatively or in addition, the memory 1003 of the management node 1000, can be configured to store any requests, resources, information, data, signals, or similar that are described herein. The processing circuitry 1001 of the management node 1000 may be configured to control the memory 1003 of the management node 1000 to store any requests, resources, information, data, signals, or similar that are described herein.

[0178] In some embodiments, the management node 1000 may optionally comprise a communications interface 1002. The communications interface 1002 of the management node 1000 can be for use in communicating with other nodes, such as other virtual nodes. For example, the communications interface 1002 of the management node 1000 can be configured to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The processing circuitry 1001 of management node 1000 may be configured to control the communications interface 1002 of the management node 1000 to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The communications interface 1002 can use any suitable communication technology.

[0179] The management node 1000 may be configured operate in the manner described herein in respect of a management node.

[0180] Figure 11 is a block diagram illustrating a management node 1100 according to some embodiments. The management node can manage a Distributed Learning, DL, process, wherein the DL process is conducted between a server node and a plurality of distributed nodes, each distributed node having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model. The management node 1100 comprises a receiving module 1102 configured to for individual distributed nodes of at least a subset of the plurality of distributed nodes, receive a representation of a local version of an evaluator ML model, wherein the evaluator ML model is configured to predict performance of the task ML model, and wherein the local version of the evaluator ML model has been trained using data from the local distributed node data set to which the distributed node has access. The management node 1100 further comprises an aggregating module 1104 configured to aggregate the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model. The management node 1106 further comprises a using module 1106 configured to use the global version of the evaluator ML model to predict performance of a version of the task ML model. The management node 1100 further comprises an initiating module 1108 configured to initiate adjustment of at least one parameter of the DL process according to the predicted performance.

[0181] The management node 1100 may operate in the manner described herein in respect of a management node. Figure 12 illustrates a server node 1200 comprising processing circuitry (or logic) 1201. The processing circuitry 1201 controls the operation of the server node 1200 and can implement the method described herein in relation to a server node 1200. The processing circuitry 1201 can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the server node 1200 in the manner described herein. In particular implementations, the processing circuitry 1201 can comprise a plurality of software and / or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein in relation to the server node 1200. It will be appreciated that the server node 1200 may comprise one or more virtual machines running different software and / or processes. The server node 1200 may therefore comprise, or be implemented in or as one or more servers, switches and / or storage devices and / or may comprise cloud computing infrastructure that runs the software and / or processes.

[0182] Optionally, the server node 1200 may comprise a memory 1203. In some embodiments, the memory 1203 of the server node 1200 can be configured to store instructions (e.g. program code) executable by the processing circuitry 1201 of the server node 1200 whereby the server node is configured to perform the method as described herein with reference to a server node.

[0183] Alternatively or in addition, the memory 1203 of the server node 1200, can be configured to store any requests, resources, information, data, signals, or similar that are described herein. The processing circuitry 1201 of the server node 1200 may be configured to control the memory 1203 of the server node 1200 to store any requests, resources, information, data, signals, or similar that are described herein.

[0184] In some embodiments, the server node 1200 may optionally comprise a communications interface 1202. The communications interface 1202 of the server node 1200 can be for use in communicating with other nodes, such as other virtual nodes. For example, the communications interface 1202 of the server node 1200 can be configured to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The processing circuitry 1201 of server node 1200 may be configured to control the communications interface 1202 of the server node 1200 to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The communications interface 1202 can use any suitable communication technology.

[0185] The server node 1200 may be configured operate in the manner described herein in respect of a server node.

[0186] Figure 13 is a block diagram illustrating a server node 1300 according to some embodiments. The sever node can facilitate management of a Distributed Learning, DL, process, wherein the DL process is conducted between the server node and a plurality of distributed nodes, each distributed note having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model. The server node 1300 comprises a performing module 1302 perform the DL process with the plurality of distributed nodes. The server node 1300 further comprises a providing module 1304 configured to provide a version of the task ML model to a management node. The server node 1300 further comprises a receiving module 1306 configured to receive from the management node information relating to a predicted performance of the version of the task ML model. The server node 1300 further comprises an implementing module 1308 configured to implement an adjustment of at least one parameter of the DL process according to the received information. The server node 1300 may be configured operate in the manner described herein in respect of a server node.

[0187] Figure 14 illustrates a distributed node 1400 comprising processing circuitry (or logic) 1401 . The processing circuitry 1401 controls the operation of the distributed node 1400 and can implement the method described herein in relation to an distributed node 1400. The processing circuitry 1401 can comprise one or more processors, processing units, multi-core processors or modules that are configured or programmed to control the distributed node 1400 in the manner described herein. In particular implementations, the processing circuitry 1401 can comprise a plurality of software and / or hardware modules that are each configured to perform, or are for performing, individual or multiple steps of the method described herein in relation to the distributed node 1400. It will be appreciated that the distributed node 1400 may comprise one or more virtual machines running different software and / or processes. The distributed node 1400 may therefore comprise, or be implemented in or as one or more servers, switches and / or storage devices and / or may comprise cloud computing infrastructure that runs the software and / or processes.

[0188] Optionally, the distributed node 1400 may comprise a memory 1403. In some embodiments, the memory 1403 of the distributed node 1400 can be configured to store instructions (e.g. program code) executable by the processing circuitry 1401 of the distributed node 1400 whereby the distributed node is configured to perform the method as described herein with reference to a distributed node.

[0189] Alternatively, or in addition, the memory 1403 of the distributed node 1400, can be configured to store any requests, resources, information, data, signals, or similar that are described herein. The processing circuitry 1401 of the distributed node 1400 may be configured to control the memory 1403 of the distributed node 1400 to store any requests, resources, information, data, signals, or similar that are described herein.

[0190] In some embodiments, the distributed node 1400 may optionally comprise a communications interface 1402. The communications interface 1402 of the distributed node 1400 can be for use in communicating with other nodes, such as other virtual nodes. For example, the communications interface 1402 of the distributed node 1400 can be configured to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The processing circuitry 1401 of distributed node 1400 may be configured to control the communications interface 1402 of the distributed node 1400 to transmit to and / or receive from other nodes requests, resources, information, data, signals, or similar. The communications interface 1402 can use any suitable communication technology.

[0191] The distributed node 1400 may be configured operate in the manner described herein in respect of a distributed node.

[0192] Figure 15 is a block diagram illustrating a distributed node 1500 according to some embodiments. The distributed node can facilitate management of a Distributed Learning, DL, process, wherein the DL process is conducted between a server node and a plurality of distributed nodes comprising the distributed node, each distributed node having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model. The distributed node 1500 comprises a performing module 1502 configured to perform the DL process with the server node. The distributed node 1500 further comprises a training node 1504 configured to train a local version of an evaluator ML model using data from the local distributed node data set to which the distributed node has access, wherein the evaluator ML model is configured to predict performance of the task ML model. The distributed node 1506 further comprises a providing module 1506 configured to provide a representation of the local version of the evaluator ML model to a management node.

[0193] The distributed node 1500 may be configured operate in the manner described herein in respect of a distributed node.

[0194] There is also provided a computer program comprising instructions which, when executed on a least one processing circuitry (such as the processing circuitry 1001 of the App1 1000 described earlier), cause the processing circuitry to carry out at least part of the method(s) described herein. According to some embodiments there is provided a carrier containing the computer program. In some embodiments, the carrier can be any one of an electronic signal, an optical signal, an electromagnetic signal, an electrical signal, a radio signal, a microwave signal, or a computer-readable medium. There is also provided a (for example, tangible and / or non-transient) computer-readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform at least part of the method (s) described herein.

[0195] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word "comprising” does not exclude the presence of elements or steps other than those listed in a claim, "a” or "an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims. Any reference signs in the claims shall not be construed so as to limit their scope.

Claims

CLAIMS1 . A method for managing a Distributed Learning, DL, process, wherein the DL process is conducted between a server node (101) and a plurality of distributed nodes (103a, 103b), each distributed node having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model, the method, performed by a management node (102), comprising: for individual distributed nodes of at least a subset of the plurality of distributed nodes, receiving (210) a representation of a local version of an evaluator ML model, wherein the evaluator ML model is configured to predict performance of the task ML model, and wherein the local version of the evaluator ML model has been trained using data from the local distributed node data set to which the distributed node has access; aggregating (220) the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model; using (230) the global version of the evaluator ML model to predict performance of a version of the task ML model; and initiating (240) adjustment of at least one parameter of the DL process according to the predicted performance.

2. A method as claimed in claim 1, further comprising selecting (205) distributed nodes from the plurality of distributed nodes for inclusion in the subset.

3. A method as claimed in claim 1 or 2, further comprising: sending (250) the global version of the evaluator ML model to distributed nodes in the plurality of distributed nodes.

4. A method as claimed in any one of the preceding claims, wherein the local version of the evaluator ML model has also been trained using a local version of the task ML model.

5. A method as claimed in any one of the preceding claims, wherein the evaluator ML model is configured to predict performance of the task ML model by: receiving (310) as input a representation of a version of the task ML model; processing (320) the input representation according to trained parameters of the evaluator ML model; and outputting (330) a predicted value of a performance parameter for the task ML model.

6. A method as claimed in claim 5, wherein the input representation of a version of the task ML model comprises values of trainable parameters of the version of the task ML model.

7. A method as claimed in any one of the preceding claims, wherein aggregating the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model comprises using an aggregation function to generate the global version of the evaluator ML model.

8. A method as claimed in any one of the preceding claims, further comprising selecting (305) a version of the task ML model for performance evaluation according to a criterion.

9. A method as claimed in claim 8, wherein the criterion comprises at least one of: availability of local versions of the task ML model; previous evaluation of versions of the task ML model; indication of substandard performance of at least one version of the task ML model; and a particularity of a local data set to which a distributed node has access.

10. A method as claimed in any one of the preceding claims, wherein using the global version of the evaluator ML model to predict performance of a version of the task ML model comprises: using the global version of the evaluator ML model to predict performance of at least one of: a global version of the task ML model; or a local version of the task ML model.

11. A method as claimed in any one of the preceding claims, wherein initiating adjustment of at least one parameter of the DL process according to the predicted performance comprises initiating at least one of: adjustment of an aggregation parameter for generating a global version of the task ML model; adjustment of learning rounds for the DL process; adjustment of the plurality of distributed nodes included in the DL process; adjustment of a trigger for retraining of the task ML model; and adjustment of an amount of training to do in a local training round.

12. A method as claimed in any one of the preceding claims, further comprising repeating the method of any of the preceding claims until satisfaction of a termination condition.

13. A method as claimed in any one of the preceding claims, wherein the management node comprises the functionality of the server node, and wherein the method further comprises: conducting the DL process for developing the task ML model with the plurality of distributed nodes.

14. The method as claimed in claim 13 further comprising: sending the global task ML model to the plurality of distributed nodes.

15. A method as claimed in any one of the preceding claims, wherein the server node is implemented in a core or Radio Access Network (RAN) network node of a telecommunications network.

16. A method as claimed in any one of the preceding claims, wherein individual distributed nodes of the plurality are implemented in at least one of: a core network node of a telecommunications network; a radio access network (RAN) node of a telecommunications network; a wireless device; a vehicle; an industrial or commercial device; and a virtual node.17: A method as claimed in any one of the preceding claims wherein the task ML model is configured to predict power consumption in a base station based on historical data.

18. The method as claimed in claim 17 wherein the server node comprises or is comprised in a first data centre, the management node comprises or is comprised in a second data centre; and each of the plurality of distributed nodes comprise or are comprised in a base station.

19. A method for facilitating management of a Distributed Learning, DL, process, wherein the DL process is conducted between a server node (101) and a plurality of distributed nodes (103a, 103b), each distributed note having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model, the method, performed by the server node (101), comprising: performing (410) the DL process with the plurality of distributed nodes; providing (420) a version of the task ML model to a management node; receiving (430) from the management node information relating to a predicted performance of the version of the task ML model; and implementing (440) an adjustment of at least one parameter of the DL process according to the received information.

20. A method for facilitating management of a Distributed Learning, DL, process, wherein the DL process is conducted between a server node and a plurality of distributed nodes, each distributed node having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model, the method, performed by a distributed node, comprising: performing (510 the DL process with the server node; training (520) a local version of an evaluator ML model using data from the local distributed node data set to which the distributed node has access, wherein the evaluator ML model is configured to predict performance of the task ML model; and providing (530) a representation of the local version of the evaluator ML model to a management node.21 . A method as claimed in claim 20, wherein training the local version of an evaluator ML model further comprises training the local version of the evaluator ML model using the local version of the task ML model.

22. A method as claimed in claim 20 or 21 , further comprising: receiving (540) a representation of a global version of the evaluator ML model from the management node.

23. A computer program product comprising a computer readable medium, the computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform a method as claimed in any one of claims 1 to 22.

24. A carrier containing the computer program according to claim 23, wherein the carrier comprises one of an electronic signal, optical signal, radio signal or computer readable storage medium.

25. A computer-readable medium comprising instructions that, when executed on at least one processor, cause the at least one processor to perform the method according to any of claims 1 to 22.

26. A computer program product comprising non transitory computer readable media having stored thereon a computer program according to claim 23.

27. A management node (1000, 102) for or managing a Distributed Learning, DL, process, wherein the DL process is conducted between a server node (101) and a plurality of distributed nodes (130a, 103b), each distributed node having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model, the management node comprising processing circuitry (1001) and a memory (1003), the memory containing instructions executable by the processing circuitry whereby the management node is configured to: for individual distributed nodes of at least a subset of the plurality of distributed nodes, receive (210) a representation of a local version of an evaluator ML model, wherein the evaluator ML model is configured to predict performance of the task ML model, and wherein the local version of the evaluator ML model has been trained using data from the local distributed node data set to which the distributed node has access; aggregate (220) the received representations of local versions of the evaluator ML model to generate a global version of the evaluator ML model; use (230) the global version of the evaluator ML model to predict performance of a version of the task ML model; and initiate (240) adjustment of at least one parameter of the DL process according to the predicted performance.

28. The management node as claimed in claim 27 wherein the memory further contains instructions executable by the processing circuitry whereby the management node is configured to perform the method as claimed in any one of claims 2 to 18.

29. A server node (1200, 101) for facilitating management of a Distributed Learning, DL, process, wherein the DL process is conducted between the server node and a plurality of distributed nodes (103a, 103b), each distributed note having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model, the server node comprising processing circuitry (1201) and a memory (1203), the memory containing instructions executable by the processing circuitry whereby the server node is configured to: perform (410) the DL process with the plurality of distributed nodes; provide (420) a version of the task ML model to a management node (102); receive (430) from the management node information relating to a predicted performance of the version of the task ML model; and implement (440) an adjustment of at least one parameter of the DL process according to the received information.

30. A distributed node (1400, 103a) for facilitating management of a Distributed Learning, DL, process, wherein the DL process is conducted between a server node (101) and a plurality of distributed nodes comprising the distributed node, each distributed node having access to a local distributed node data set, and wherein the DL process is for developing a task Machine Learning, ML, model, the distributed node comprising processing circuitry (1401) and a memory (1403), the memory containing instructions executable by the processing circuitry whereby the distributed node is configured to: perform (510) the DL process with the server node; train (520) a local version of an evaluator ML model using data from the local distributed node data set to which the distributed node has access, wherein the evaluator ML model is configured to predict performance of the task ML model; and provide (530) a representation of the local version of the evaluator ML model to a management node(102).

31. The distributed node as claimed in claim 30 wherein the memory further contains instructions executable by the processing circuitry whereby the distributed node is configured to perform the method as claimed in any one of claims 21 to 22.

Images