A system, device, and method for scatter-gather procedures

Abstract

A method for selecting worker devices for executing a computing task out of a total number of worker devices connected to a network. The total number of worker devices is greater than the selection. The method comprises selecting by a scheduler a set of worker devices from the total worker devices in the network, issuing to the network a request to scatter a packet to a group address, transmitting by the scheduler to the network the selected set, distributing the packet within the network according to the group address and selected set such that the packet is delivered to each worker device of the set, and providing by each worker device of the set a reply to the coordinator of the network. A system comprising a coordinator, a network, a plurality of worker devices, and a scheduler and configured to implement the method. A scheduler configured to implement the method.

Description

[0001] A SYSTEM, DEVICE, AND METHOD FOR SCATTER-GATHER PROCEDURES

[0002] FIELD OF THE INVENTION

[0003] This invention relates to federated learning communications across multiple distributed devices.

[0004] BACKGROUND

[0005] Federated learning, FL, (also known as collaborative learning) is a sub-field of machine learning focusing on settings in which multiple entities (often referred to as clients) collaboratively train a model while ensuring that their data remains decentralized. That is, federated learning aims at training a machine learning algorithm, for instance deep neural networks, on multiple local datasets contained in local nodes without explicitly exchanging data samples. The general principle consists in training local models on local data samples and exchanging parameters (e.g. the weights and biases of a deep neural network) between these local nodes at some frequency to generate a global model shared by all nodes.

[0006] Federated learning is generally concerned with and motivated by issues such as data privacy, data minimization, and data access rights. One of the primary defining characteristics of federated learning is data heterogeneity. Due to the decentralized nature of the clients' data, there is no guarantee that data samples held by each client are independently and identically distributed.

[0007] Distributed learning originally aims at parallelizing computing power whereas federated learning originally aims at training on heterogeneous datasets. While distributed learning also aims at training a single model on multiple servers, a common underlying assumption is that the local datasets are independent and identically distributed and roughly have the same size. However, for federated learning, the datasets are typically heterogeneous and their sizes may span several orders of magnitude.

[0008] An additional factor to consider is that the clients involved in federated learning may be unreliable as they are subject to more failures or drop out. This is because they commonly rely on less powerful communication media (i.e. Wi-Fi) and battery- powered systems (i.e. smartphones and loT devices) compared to distributed learning; where nodes are typically datacenters that have powerful computational capabilities and are connected to one another with fast networks.

[0009] Specifically, federated learning assumes a partitioned data set distributed across and stored locally at 1 , . . . ,n. worker clients or workers. These workers perform localized training over their own data set. A central coordinator may then gather training model input from the workers and scatter an updated model to the workers.

[0010] Training may continue until an accuracy threshold is met. A similar technique is performed in a single data centre network (DCN), partitioning the joint data set into smaller sets across server clusters for tractability of the computationally intensive training task.

[0011] However, training workers may also be distributed across network (or data centre) locations, where FL would be applied. Data sets may also start out as homogenous, but diverge based on localized input (e.g. inferences specific to the location and the data provided there). Data sets may also be heterogenous to start with and can thus not be shared to a central DCN. For example, due to confidentiality of the relevant data sources.

[0012] The term worker will be used herein to indicate the intention that the receiving endpoint ‘work’ over the training data, which is a common notion for such nodes in an Al scenario. Other notions may use terms like ‘server’ or ‘receiving endpoint’, and they may alternatively be used here. Communication in federated learning follows a typical scatter-gather semantic. That is, model parameters or training data are scattered to all workers, while results are gathered back at the coordinator to update the model.

[0013] Additionally, there is a difficulty in these types of networks to consider, which is that the set of N workers may be continuously formed from a larger set M(>=N) through a worker-specific constraint. For example, the constraint may be a costs or fairness of resource usage (e.g. compute, communication, or energy), against a system-level objective. That is, there is a threshold that each constraint must not exceed in order to fulfil the system objective.

[0014] Energy-aware scatter-gather semantic based distributed training has been proposed in an existing approach. This approach uses, for example, energy metrics, to select N out of M workers, forming a set of receivers to which unicast messages are dispatched. However, the frontend replication has poor efficiency since it has to maintain N unicast relations.

[0015] In another existing approach there is proposed compute-aware traffic steering. The approach uses compute metrics to select one out of M possible service instances and typical load balancing across the M possible service instances. The selection may change per client request, depending on the changing compute metrics. The anycast used is limited to single destination selection, encapsulated in an ingress-egress tunnel to that destination. However, it is not possible in this configuration to select N out of M possible workers.

[0016] It is desirable to provide a method for dynamically scheduling the selection of a plurality of workers in an efficient way which also complies with existing constraints on federated learning.

[0017] SUMMARY OF THE INVENTION

[0018] According to one aspect there is provided a method for selecting worker devices for executing a computing task out of a total number of worker devices connected to a network, where the total number of worker devices is greater than the selection, the method comprising: selecting by a scheduler component of the network a set of worker devices from the total worker devices in the network; issuing by a coordinator of the network to the network a request to scatter a packet to a group address; transmitting by the scheduler to the network the selected set; distributing the packet within the network according to the group address and selected set such that the packet is delivered to each worker device of the set; and providing by each worker device of the set a reply to the coordinator of the network.

[0019] In an embodiment, the group address may be an anycast address. In an embodiment, the anycast address may be formulated to cause the packet to be scattered to all of the worker devices in the set as selected by the scheduler. This may help in efficient distribution of the packet.

[0020] In an embodiment, the method may comprise receiving at the scheduler status information from each of the total number of worker devices in the network and selecting by the scheduler the set of devices from the total devices based on the status information. This may help to efficiently designate the set of devices.

[0021] In an embodiment, the method may comprise receiving status information from at least one of the total number of worker devices repeatedly at predetermined time intervals. This may allow that status to be used as an input for transmissions to worker devices. In an embodiment, the method may comprise receiving status information from an individual worker device of the total number of worker devices when its status changes. The method may comprise a worker device determining that tis status has changed, and in response transmitting information indicative of its status. This may allow status changes to be efficiently conveyed.

[0022] In an embodiment, the scheduler may transmit the selected set of devices to the network in response to receiving a request from the network to identify the set. This may allow entities in the network to adapt their activity in dependence on the identities of the devices in the selected set.

[0023] In an embodiment, the method may comprise providing a unique worker device identifier alongside each reply to the coordinator so as to identify the source. This may allow the source to be identified.

[0024] In an embodiment, distributing the packet within the network may comprise: receiving the packet at an ingress tunnel endpoint of the network; encapsulating the packet; forwarding the encapsulated packet to one or more egress tunnel endpoints of the network identified as serving one or more worker devices of the set; decapsulating the packet; and forwarding the packet to the one or more locally served worker devices of the one or more egress tunnel endpoints. This may allow the packet to be efficiently distributed.

[0025] In an embodiment, the method may comprise adding the set size alongside the packet and encapsulating, forwarding, and decapsulating the set size with the packet. This may allow the set size to be efficiently distributed.

[0026] In an embodiment, the scatter packet may comprise a computer model to be implemented or trained or both at each worker device. In this way, the scatter packet may define the model and enable it to be implemented or trained in a distributed manner.

[0027] In an embodiment, the reply from each worker device to the coordinator may comprise an updated version of a computer model based on locally available data. The updated model may be adopted as an enhanced model for future use.

[0028] According to another aspect there is provided a system for selecting worker devices for executing a computing task out of a total number of worker devices connected to a network, where the total number of worker devices is greater than the selection, the system comprising: a coordinator, a network, a plurality of worker devices, and a scheduler; the scheduler configured to select a set of worker devices for performing a task and provide the set to the network to formulate an anycast address for only the set. This can allow the worker devices to be selected in an effective way.

[0029] In an embodiment, the scheduler may be configured to be located within the network and the network comprises an ingress tunnel endpoint where the coordinator connects to the network as well as a set of egress tunnel endpoints where the plurality of worker devices connects to the network. This can allow communications with the scheduler to be effectively directed.

[0030] In an embodiment, the scheduler may be configured to receive status information from each of the worker devices connected to the network. This may allow that status to be used as an input for transmissions to worker devices.

[0031] In an embodiment, the scheduler is configured to receive a status update from one or more of the worker devices at predetermined intervals. This may give the scheduler timely updates on the status of the workers.

[0032] In an embodiment, the scheduler may be configured to receive a status update from an individual worker device of the total number of worker devices upon its status changing. This may be an efficient way of worker devices communicating their status. In an embodiment, the scheduler may be configured to determine which worker devices should be included in the set based on the current status of each worker device. This can make effective user of the worker devices.

[0033] In an embodiment, the ingress tunnel may be configured to forward the packet according to the anycast address and the set of worker devices to one or more egress tunnels serving the worker devices in the set. This can allow the packet to reach those worker devices effectively.

[0034] In an embodiment, the scheduler and the ingress tunnel endpoint may be collocated within the network. This can provide for efficient communications between them.

[0035] In an embodiment, any one of the worker devices of the system may be configured to operate as the coordinator of another system. Thus, a worker device may participate in two systems.

[0036] According to another aspect there is provided a scheduler configured to be embedded into an ingress-egress framework of a network, the scheduler configured to determine a set of worker devices for executing a computing task out of a total number of worker devices connected to the network, where the total number of worker devices is greater than the set, and the set is based on a status of each worker device, the scheduler configured to provide the set to an ingress tunnel network element for forwarding a scatter packet according to a group address.

[0037] In an embodiment, the group address may be an anycast address. In an embodiment, the scheduler may be configured to receive status information from at least one of the total number of worker devices at predetermined time intervals. These features can have analogous effects to those identified for them in the discussion above.

[0038] In an embodiment, the scheduler may be configured to receive status information from at least one of the total number of worker devices upon a change in the status of the at least one worker device. This can allow the status of a worker device to be communicated effectively.

[0039] BRIEF DESCRIPTION OF THE FIGURES

[0040] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0041] Figure 1 shows an example network, connected to which is a coordinator and a plurality of workers.

[0042] Figure 2 shows example messaging timings and order for carrying out the proposed scheme.

[0043] Figure 3 shows an example embodiment where the proposed approach is implemented in a 3GPP Network Function.

[0044] Figure 4 shows an example embodiment of the proposed approach with a hierarchical architecture.

[0045] DETAILED DESCRIPTION OF THE INVENTION

[0046] The problem with the above-described existing approaches is that it is not possible to adaptively select which workers to use without requiring multiple single destination links to be defined. This can be costly in energy and complicated to organise repeatedly for each instance of a link. Thus, there is proposed herein a collective communication mechanism that allows for the dynamic selection of workers per update operation within defined constraints. More specifically, the proposed approach dynamically schedules the selection of N(t) workers from a total of M workers. Packets are then sent according to a scatter-gather semantic. This is achieved in part by moving the selection and scheduling point into the network. This may provide increased efficiency, while also allowing for the selection of N out M instead of 1 out of M destinations.

[0047] The proposed approach may be achieved by extending the existing by restricted anycast semantic of ‘any one’ out of multiple destinations to a semantic that comprises ‘any ones’ out of multiple destinations. This allows for all other scatter-gather semantics to be maintained while allowing the number of destinations in the scatter-gather process to be extended to multiples rather than a single destination or all destinations.

[0048] The proposed approach is embedded into an ingress-egress tunnelling framework where the tunnelled message is an anycast addressed message. This is the mode of transport of the scatter-gather messages over the network.

[0049] The ingress tunnel endpoints are therefore extended to make set-based destination decisions, rather than a single destinationbased decision, where the selection of the set is based on a metric derived from individual input from up to M workers.

[0050] By making this change in this way it is possible to maintain the standard gathering of replies as relied on in traditional unicast addressing.

[0051] As mentioned above, the proposed approach comprises instead interpreting an anycast address as an N out of M destination address. Thus, it is possible to extend the anycast semantic in IP, i.e., selecting ANY ONE out of AT possible destinations to selecting ANY ONES out of AT possible destinations for a given anycast address W.

[0052] Accordingly, the result of an anycast forwarding decision extends from an address W of one out of the set (wi, ...., WM) = Wi, to an address W consisting of the set (wi, ...., WN) = {w;} with i=l, out of the possible (wi, ...., WM) addresses.

[0053] The set {w;} may be determined through a selection function provided via a new function which herein is referred to as ‘scheduler ’, where N(t) is time variant with updates of N(t) driven by the updates of metrics received at the coordinator.

[0054] The high-level steps of the proposed approach will now be explained in reference to Figure 1. Figure 1 shows a network 102, connected to which is a coordinator 104 and a plurality of workers 106. There may be one or more worker devices 106 available to the network 102 at any one time for the purposes of executing tasks thereon. The scheduler 104 and the workers 106 operate at the same level, i.e. at the application level. Therefore, it is possible to think of the coordinator and workers as part of the network or not part of the network depending on at which level the boundary of the network is considered to reside. However, whichever boundary is considered, the network coordinator and the worker client devices are located at the same level as each other. That is, the system may be configured such that the coordinator 104 and the workers 106 are considered to be part of the network 102. Alternatively, the system may be configured such that the coordinator 104 and the workers 106 are not considered to be part of the network 102.

[0055] Once a subset of the set of workers 106 has been determined at the scheduler 108, the coordinator 104 may issue a scatter request for an anycast address comprising the selected subset of workers 106. The ingress tunnel endpoint 110 may then encapsulate one or more packets with the destination of one or more egress tunnel endpoints 112 which serve the workers 106 of the subset. Input into the selection of the subset is information received 114 from the workers 106 to the scheduler 108 for the purposes of determining the subset of workers 106 Ni to form the modified anycast address. Thus, there is provided a system for selecting worker devices for executing a computing task out of a total number of worker devices connected to a network, where the total number of worker devices is greater than the selection, the system comprising a coordinator, a network, a plurality of worker devices, and a scheduler. The scheduler may be configured to select a set of worker devices for performing a task and provide the set to the network to formulate an anycast address for only the set.

[0056] The system comprises a scheduler which may be configured to be located within the network and the network may comprise an ingress tunnel endpoint where the coordinator connects to the network as well as a set of egress tunnel endpoints where the plurality of worker devices connect to the network. The scheduler may be configured to receive status information from each of the worker devices connected to the network. The scheduler of the system may be configured to receive a status update from one or more of the worker devices at predetermined intervals. The scheduler of the system may be configured to receive a status update from an individual worker device of the total number of worker devices upon its status changing. The scheduler of the system may be configured to determine which worker devices should be included in the set based on the current status of each worker device.

[0057] The ingress tunnel of the system may be configured to forward the packet according to the anycast address and the set of worker devices to one or more egress tunnels serving the worker devices in the set. The scheduler and the ingress tunnel endpoint of the system may be collocated within the network.

[0058] The ingress tunnel endpoint needs to encapsulate based on scheduler input. This input can either be retrieved by communicating with the scheduler explicitly, in which case the scheduler is not collocated with the ingress tunnel endpoint; or by collocating the scheduler into the ingress tunnel endpoint, in which case said communication is ‘internal’ to the ingress tunnel endpoint. The egress tunnel endpoints simply receive encapsulated messages, decapsulate and forward them to locally attached workers.

[0059] Any one of the worker devices of the system may be configured to operate as the coordinator of another system.

[0060] There is also provided a scheduler configured to be embedded into an ingress-egress framework of a network. The scheduler may be configured to determine a set of worker devices for executing a computing task out of a total number of worker devices connected to the network, where the total number of worker devices is greater or equal than the set, and the set is based on a status of each worker device. The scheduler may be configured to provide the set to an ingress tunnel network element for forwarding a scatter packet according to a group address. The group address may be an anycast address.

[0061] The scheduler may be configured to receive status information from at least one of the total number of worker devices at predetermined time intervals. The scheduler may be configured to receive status information from at least one of the total number of worker devices upon a change in the status of the at least one worker device.

[0062] Figure 2 shows the messaging timing and order for carrying out the proposed scheme. The messages are transmitted in order from the top down, with arrows dictating the direction of each message or data stream from its source to its destination.

[0063] There is provided as part of the proposed approach, a scheduler 108 configured to control the selection of V out of AT workers 106. The selection may be based on dynamic metric updates received from the AT workers 106. The proposed approach may be described as starting with the receipt of metric updates 202 from one or more of the workers 106. Having received the metric updates, the scheduler 108 may determine 204 the set of workers to be used.

[0064] Each worker 106 may continuously update 114 the scheduler 108 with renewed metric mt, where i uniquely denotes the worker Upon receiving metric updates, the scheduler 108 determines 204 the set N(t) of workers 106. The scheduler 108 may update the set after the elapsing of a pre-determined time period since the previous update. The set may be updated in response to a network function or a change in the status of a network function. The set may be updated in response to an indication that one or more workers 106 have become available or are no longer available. The scheduler 108 may then select a set NY of workers out of a total of AT available workers 106.

[0065] The coordinator 104 may then proceed to issue a scatter request 206 to an anycast address W, representing the set N(t) determined at scheduler 108. The address for the set may be represented by an anycast address where ANY is interpreted not just as ‘any one’ out of M but ‘any ones’ out of M, allowing for multiple destinations to be specified.

[0066] The ingress tunnel endpoint 110 requests 208 the current set of workers from the scheduler 108. The scheduler 108 may be collocated within the network at an ingress tunnel endpoint 110. The ingress tunnel endpoint 110 may be configured to maintain suitable tunnelling relations with egress tunnel endpoints 112, which in turn maintain connectivity to the M workers 106.

[0067] Upon receiving the selected set N(t) from the scheduler 108, the ingress tunnel endpoint may then scatter 208 the received packet to the multiple destinations in the modified format anycast address. For this, upon arrival of the scatter request at the ingress tunnel endpoint 110, said tunnel endpoint retrieves the current set N(t) from the scheduler 108 and tunnels the incoming packet to ALL of the tunnel endpoints at which the N(t) workers are connected. The retrieval comprises requesting the set from the scheduler and receiving the set as a reply message from the scheduler 108. The ingress tunnel endpoint 110 may then encapsulate the packet. It may also then add the group size M=|N(t)| into the tunnelled message as additional information. The encapsulated packet may then be forwarded from the ingress tunnel endpoint 110 to the one or more suitable egress tunnel endpoints 112 according to the anycast address W.

[0068] The selected egress tunnel endpoints 112 may then decapsulate the tunnelled message to retrieve the original packet P and the set size N(t) to be forwarded 210 to the appropriate locally served worker(s) 106.

[0069] The workers 106 may reply 212 to the coordinator 104 in a separate unicast message, where the anycast address may represent the reply’s source address. That is, the worker may include additional unique information i into the return packet to allow unique identification of the source at the coordinator 104. The reply may also include the size of the set of workers 106 to which the original scatter packet was forwarded.

[0070] Thus, there is provided a method for selecting worker devices for executing a computing task out of a total number of worker devices connected to a network, where the total number of worker devices is greater than the selection. The method comprises selecting, by a scheduler component of the network, a set of worker devices from the total worker devices in the network. Then issuing, by a coordinator of the network to the network, a request to scatter a packet to a group address. Then transmitting, by the scheduler to the network, the selected set. Then distributing the packet within the network according to the group address and selected set such that the packet is delivered to each worker device of the set. Then providing, by each worker device of the set, a reply to the coordinator of the network.

[0071] It should be understood that the coordinator and the workers may not be considered to be part of the infrastructure of the network as the coordinator is at the same ‘level’ as the worker devices, i.e., they are part of the application level, and not the network per se. That is, it is the network’s role to disseminate a packet, addressed to group address w. Here, it is the scheduler being part of the network and working with the ingress point to perform that function which is key. However, there may be an embodiment where the “Al application” is, in fact, part of the network, doing training for, e.g., improving a network function. Here then, the coordinator AND the workers will be part of the network. Although, they may be considered an application that happens to be executed by a network entity. In another embodiment the coordinator may be a worker of another system. The group address may be an anycast address. The anycast address may be formulated to cause the packet to be scattered to all of the worker devices in the set as selected by the scheduler.

[0072] The proposed method may comprise receiving at the scheduler status information from each of the total number of worker devices in the network and selecting by the scheduler the set of devices from the total devices based on the status information. The method may comprise receiving status information from at least one of the total number of worker devices repeatedly at predetermined time intervals. The proposed method may comprise receiving status information from an individual worker device of the total number of worker devices when its status changes.

[0073] The scheduler may transmit the selected set of devices to the network in response to receiving a request from the network to identify the set. The method may comprise providing a unique worker device identifier alongside each reply to the coordinator so as to identify the source.

[0074] More specifically, distributing the packet within the network may comprise receiving the packet at an ingress tunnel endpoint of the network, encapsulating the packet, forwarding the encapsulated packet to one or more egress tunnel endpoints of the network identified as serving one or more worker devices of the set, decapsulating the packet, and forwarding the packet to the one or more locally served worker devices of the one or more egress tunnel endpoints.

[0075] The proposed method may comprise adding the set size alongside the packet and encapsulating, forwarding, and decapsulating the set size with the packet. The scatter packet may comprise a computer model to be implemented or trained or both at each worker device. The reply from each worker device to the coordinator may comprise an updated version of a computer model based on locally available data.

[0076] As mentioned above, a metric may be used for the selection of workers 106 to form the set N(t) to receive the scatter packet(s). Any one or more of a plurality of different selection metrics may be used.

[0077] Energy-aware selection - Each worker 106 may report compute, network, and / or energy information as a metric to the scheduler 108. The scheduler 108, in turn, may then perform an objective function to select all those workers 106 who remain under a desired threshold for combined energy consumption.

[0078] Delay-based selection - Each worker 106 may report latencies experiences in previous interactions to the scheduler 108. The scheduler 108, in turn, may then use this to select those workers 106 remaining under a given latency threshold.

[0079] Training Performance-based selection - Each worker 106 may report metrics representing the accuracy of the training model over the local data to the scheduler 108. The scheduler 108, in turn, may use this information to select those workers 106 below a given accuracy threshold. For example, selecting only those workers 106 that still require improvement from a training perspective.

[0080] Computational capability - Each worker 106 may report computational capabilities, such as current load or possibly averaged over time, to the scheduler 108. The scheduler 108, in turn, may select those workers 106 remaining under a given load threshold.

[0081] Various existing tunneling methods may be used with the proposed approach. Additionally, various traffic steering processes may be combined used in IETF. An existing compute-aware traffic steering mechanism provides an ingress / egress tunnelling framework which could be used for the tunnelling endpoints as described herein. The anycast address w described herein would accordingly be encoded as an IP anycast address and resolved via a DNS request, where the ingress tunnel ingress point would encapsulate based on a set of IP unicast addresses (of workers 106) received from the scheduler.

[0082] In a further existing mechanism, a ROSA-based overlay may be used, providing a service routing overlay, where Service Address Routers (SARs) perform the forwarding of service requests to endpoints. In this mechanism the ingress tunnel endpoint is the ingress SAR, while the egress tunnel endpoint is the destination (there is no intermediary tunnelling in ROSA). The anycast address w described herein could be encoded, e.g., using RFC8609, and inserted into an IP extension header.

[0083] Both of these existing steering approaches utilize any one of the following tunnelling methods; IP-in-IP encapsulation (RFC1823), VPN (L2 or L3) (RC2763), or IPv6 extension headers (see RFC2460).

[0084] In an embodiment, as described above, the ingress tunnel endpoint may include both the group address W (for example, through the above-mentioned tunnelling methods), and the group size N(t) as additional fields in the tunnelled message.

[0085] The egress tunnel endpoint 112 may decapsulate, using any of the embodiments above, and forward the original client message and the additional group size N(t) to the worker 106.

[0086] The specific details as to where in the encapsulated packet the additional information will be added will likely be subject to standardizations in place, e.g., in the IETF. An example implementation may be to add the information as new fields after the existing tunnelling header as an extension to specific steering methods (e.g., ROSA or CATS). Another option would be to extend the tunnelling standard itself.

[0087] As described above, the identification of the source worker 106 may be included in replies to the coordinator 104. This may be implemented in various ways as described below.

[0088] At the application level, the worker 106 may add a unique worker identification w, into the application message sent back to the originating coordinator 104. The coordinator 104 may in turn use this identification to associate the reply to the specific worker 106. This requires app-level collaboration at the client, which is achievable as there is application knowledge of the scatter-gather semantic.

[0089] At the network level, the worker 106 may perform IP-in-IP encapsulation, where the anycast message is encapsulated in a unicast packet and with the workers’ IP address being used there. This embodiment may therefore require IP stack level collaboration at the client.

[0090] As described above, the scheduler 108 may also indicate the group size M to the ingress tunnel endpoint for inclusion in the tunnelled message, as provided by the worker in its reply to the coordinator 104. This may aid possible loss detection at the coordinator 104.

[0091] The components of the described approach may map onto CATS components, consisting of service metric agents (C-SMA) that receive endpoint metric information as well as a path selector (C-PS) to make decisions on the unicast address of the endpoint chosen. For example, the scheduler 108 may implement a C-SMA, CATS service metric agent, when receiving the dynamic updates from the workers 106 for selection. The scheduler 108 may also implement a C-PS (CATS path selector), in determining the path(s) to the CATS egress forwarders, where there may be up to M paths due to the M dynamically chosen workers 106. The tunnel ingress and egress endpoints may implement the CATS forwarder. The coordinator 104 and workers 106 are then endpoints in the CATS framework.

[0092] Therefore, the proposed approach may impact the interfaces between a C-PS and a CATS forwarder in that a collection of paths, up to N possible CATS forwarders, would be communicated to the CATS forwarder acting as the ingress tunnel endpoint.

[0093] The possible inclusion of the dynamic group size N into the tunnel message may also impact the encapsulation procedure at the ingress tunnel endpoint.

[0094] An embodiment where the proposed approach is implemented in a 3GPP Network Function is shown in figure 3. The ingress and egress tunnel endpoints may represent two UPF (user plane function) instances 302 and 304 respectively in a mobile communications subsystem 300. Instead of collocating the scheduler 108 with the originating UPF 302 (ingress tunnel endpoint), the scheduler 108 may also be realized as a part of an NRF (name resolution function) 306. The originating UPF 302 may then use service-based architecture (SBA) concepts, i.e., invocation via HTTP / 2, to retrieve the suitable worker set N(t) from the scheduler 306. This is implemented in figure 3 as step 4.

[0095] The proposed approach is described step by step in figure 3 in relation to a 3GPP System.

[0096] Figure 4 shows a hierarchical application of the proposed approach. A worker 106 of a first coordinator 104 may in return act as a second coordinator 404 to another set of workers 406. This allows for building hierarchical, federated training on a larger scale.

[0097] There are many different scenarios in which the described approach may be used.

[0098] Alibaba is known to use CATS and focusses on distributing inferencing over localized replicas of the training model. However, inferencing requests may lead to a need for re-training the localized model since the initial data sets at each replicated site diverge over time. That is, with different inference requests sent to each site the models generated eventually no longer move towards the same model. Collecting all new training data to a central site for re-training may be inefficient but it is also prohibitive for confidentiality reasons. That is, if new data originates from separate governance regimes, e.g. countries, it cannot be centralised.

[0099] The proposed approach may also be used in cases where heterogenous data sets are used and over which federated workers train a common model. Examples of such organisations and operations are listed below. For example, cross-hospital health data mining, to extract patterns for diseases over locally confidential patient data. Cross-operator network management, to extract network management features over confidential operator traffic data. Cross-factory industrial networking, to extract optimizations of supply chain over confidential manufacturing data of individual factories and across several federated companies. Cross-car / OEM vehicular network, to extract features related to traffic management or incidents over otherwise confidential driver data such as exact location paths etc.. Cross-device mobile applications (for example, different keyboard usage across devices), extracting common writing patterns over otherwise confidential users’ input on their mobile devices. The proposed approach may also have uses across community interests. For instance, SDOs like 3GPP have established work items on federated Al, while IETF work in the CATS WG already include inferencing in relation to distributed DC training, where training itself lies equally in scope of the CATS community work. 1. SDOs like 3GPP, have established work items on federated Al directed at use cases 2, 3, and 4 in relation to the

[0100] NWDAF (TS23.288), while IETF CATS work could be extended to include training.

[0101] 2. The National Science Foundation (NSF) has established large-scale programme (https: / / icicle.osu.edu / ) in this space.

[0102] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A method for selecting worker devices for executing a computing task out of a total number of worker devices connected to a network, where the total number of worker devices is greater than the selection, the method comprising: selecting by a scheduler component of the network a set of worker devices from the total worker devices in the network; issuing by a coordinator of the network to the network a request to scatter a packet to a group address; transmitting by the scheduler to the network the selected set; distributing the packet within the network according to the group address and selected set such that the packet is delivered to each worker device of the set; and providing by each worker device of the set a reply to the coordinator of the network.

2. The method of claim 1 , wherein the group address is an anycast address.

3. The method of claim 2, wherein the anycast address is formulated to cause the packet to be scattered to all of the worker devices in the set as selected by the scheduler.

4. The method of any preceding claim, wherein the method comprises receiving at the scheduler status information from each of the total number of worker devices in the network and selecting by the scheduler the set of devices from the total devices based on the status information.

5. The method of any preceding claim, wherein the method comprises receiving status information from at least one of the total number of worker devices repeatedly at predetermined time intervals.

6. The method of any preceding claim, wherein the method comprises receiving status information from an individual worker device of the total number of worker devices when its status changes.

7. The method of any preceding claim, wherein the scheduler transmits the selected set of devices to the network in response to receiving a request from the network to identify the set.

8. The method of any preceding claim, wherein the method comprises providing a unique worker device identifier alongside each reply to the coordinator so as to identify the source.

9. The method of any preceding claim, wherein distributing the packet within the network comprises: receiving the packet at an ingress tunnel endpoint of the network; encapsulating the packet; forwarding the encapsulated packet to one or more egress tunnel endpoints of the network identified as serving one or more worker devices of the set; decapsulating the packet; and forwarding the packet to the one or more locally served worker devices of the one or more egress tunnel endpoints.

10. The method of claim 9, wherein the method comprises adding the set size alongside the packet and encapsulating, forwarding, and decapsulating the set size with the packet.

11. The method of any preceding claim, wherein the scatter packet comprises a computer model to be implemented or trained or both at each worker device.

12. The method of any preceding claim, wherein the reply from each worker device to the coordinator comprises an updated version of a computer model based on locally available data.

13. A system for selecting worker devices for executing a computing task out of a total number of worker devices connected to a network, where the total number of worker devices is greater than the selection, the system comprising: a coordinator, a network, a plurality of worker devices, and a scheduler; the scheduler configured to select a set of worker devices for performing a task and provide the set to the network to formulate an anycast address for only the set.

14. The system of claim 13, wherein the scheduler is configured to be located within the network and the network comprises an ingress tunnel endpoint where the coordinator connects to the network as well as a set of egress tunnel endpoints where the plurality of worker devices connect to the network.

15. The system of claim 13 or 14, wherein the scheduler is configured to receive status information from each of the worker devices connected to the network.

16. The system of any of claims 13 to 15, wherein the scheduler is configured to receive a status update from one or more of the worker devices at predetermined intervals.

17. The system of any of claims 13 to 16, wherein the scheduler is configured to receive a status update from an individual worker device of the total number of worker devices upon its status changing.

18. The system of any of claims 13 to 17, wherein the scheduler is configured to determine which worker devices should be included in the set based on the current status of each worker device.

19. The system of any of claims 13 to 18, wherein the ingress tunnel is configured to forward the packet according to the anycast address and the set of worker devices to one or more egress tunnels serving the worker devices in the set.

20. The system of any of claims 13 to 19, wherein the scheduler and the ingress tunnel endpoint are collocated within the network.

21. The system of any of claims 13 to 20, wherein any one of the worker devices of the system is configured to operate as the coordinator of another system.

22. A scheduler configured to be embedded into an ingress-egress framework of a network, the scheduler configured to determine a set of worker devices for executing a computing task out of a total number of worker devices connected to the network, where the total number of worker devices is greater than the set, and the set is based on a status of each worker device, the scheduler configured to provide the set to an ingress tunnel network element for forwarding a scatter packet according to a group address.

23. The scheduler of claim 22, wherein the group address is an anycast address.

24. The scheduler of claim 22 or 23, wherein the scheduler is configured to receive status information from at least one of the total number of worker devices at predetermined time intervals.

25. The scheduler of any of claims 22 to 24, wherein the scheduler is configured to receive status information from at least one of the total number of worker devices upon a change in the status of the at least one worker device.

Images