Model pool for multi-modal distributed learning

Abstract

A method performed by a central server node is provided. The method includes receiving, from a local client node, local model weights and a corresponding key; and updating a model pool having a plurality of central models and a corresponding key associated with each of the central models. Updating the model pool is based on the local model weights and one or more of: the key corresponding to the local client node, and the key common to each of the central models. Updating the model pool includes updating at least two of the plurality of central models included in the model pool.

Description

Technical Field

[0001] Implementations related to distributed learning, and specifically to model pools for multimodal distributed learning, are disclosed. Background Technology

[0002] Distributed learning involving distributed data is a machine learning technique that addresses various data-related constraints (such as privacy and bandwidth) that prohibit or otherwise limit the transfer of local training data from local client nodes to a central node for centralized processing. In such a scenario, learning from the data occurs locally, and updates are sent from local client nodes to a central server node, allowing different local client nodes to influence the global model without sacrificing the security or privacy of their local training data.

[0003] As an example, consider the problem of predicting words from prefixes typed on a mobile device. Local devices equipped with a learning process can model the user's typing behavior to suggest or recommend suffixes to complete partially typed words. Given many users sharing a common language, the resulting models from each local device can be combined (e.g., averaged) to produce a global model that still represents the problem. Joint learning (a type of distributed learning) is a technique suitable for such problems, where multiple local models can be averaged to create an accurate global model.

[0004] Joint learning is conducted in rounds, each round including:

[0005] • At the central server node, select a subset of participating nodes.

[0006] • At the central server node, send the current global model to the selected node.

[0007] • At each selected node, train the global model locally on the local dataset.

[0008] • At each selected node, updates reflecting local training are sent back to the central server node.

[0009] • At the central server node, updates are aggregated and applied to the global model.

[0010] Continuing with the previous word prediction example, consider multiple users typing in different languages. In such an example, the distribution of training data might vary greatly from one user to another. Applying traditional joint learning methods in this example would undoubtedly lead to an undesirable drop in accuracy; that is, updates learned from one language might contradict updates from another. Of course, in this example, maintaining a different global model for each language might be sufficient. However, in more general real-world scenarios, such prior separations might be difficult or impossible to identify. Summary of the Invention

[0011] Problems arise if the training data distributions differ among local client nodes. For example, typical federated learning techniques achieve model synthesis through averaging, which is not robust to scenarios where the local data distribution is far from independent and identically distributed (IID). Typical federated learning techniques only support a central model that improves upon the local models using updates from them, which poses a challenge for non-overlapping, non-IID data distributions. Given a target scenario or target data distribution, typical federated learning techniques can only provide a central model that is unlikely to be a good starting point if the local target distribution differs significantly from that of other local client nodes.

[0012] Some federated learning techniques apply clustering to group local client nodes into different groups. Such techniques are limited and do not fully address the aforementioned problems, as well as other issues they have. For example, such clustering methods require specifying the number of clusters used to classify local client nodes; if the data distribution of local client nodes changes over time, it must be updated to reflect the current situation; and it is also limited by how such methods cluster local client nodes, thus relying on local training data rather than other local client resources. Furthermore, such clustering techniques are all-or-nothing methods, where updates from one local client node will not be propagated to the other if two local client nodes have similar data distributions but belong to different clusters.

[0013] Therefore, improved learning techniques are needed. The embodiments disclosed herein provide improved machine learning in a distributed learning environment when local training data exhibit significant differences in their distributions. For example, such significant differences in the distribution of local training data may arise due to the inherent local characteristics (e.g., hardware, software, geographic location, system type) of different local client nodes. Regardless of the cause of the different distributions, if such differences are significant, they may affect the ability to maintain a centralized model that represents each local client node well, for example, because updates from local client nodes with different distributions may lead to performance degradation or even prevent global model convergence. In this case, if the local client nodes have significantly different training data distributions, the global data distribution can be considered multimodal. The embodiments of this paper provide improved learning by maintaining a model pool of a central model to cover multiple local data distributions.

[0014] The embodiments employ and manage a model pool for distributed learning environments (e.g., federated learning). The model pool can be used to store, update, and retrieve a set of central models. The model pool defines read and write operations for manipulating the stored central models. In some embodiments, each central model in the model pool can be associated with a key (e.g., a carefully crafted key) that identifies the training scenario and the data distribution for which the central model was trained. Then, given an input key (e.g., from a local client node) and the distance of that input key to the keys of the central models in the model pool, a composite model can be aggregated from the central models of the model pool as a weighted average, wherein the weighting is biased based on the similarity between the input key and the keys of the central models in the model pool. In the embodiments, the keys are designed to have a locality preservation property, meaning that similar training scenarios and data distributions produce similar keys.

[0015] In this embodiment, read and write operations are defined for the model pool, and these operations can be implemented in a distributed manner. That is, a key can be used to read from and / or write to multiple central models in the model pool. The impact of each central model on the operation result depends on how similar the corresponding key of the central model is to the input key. Because in this embodiment, the key carries information about the training scenario and data distribution, central models from conflicting contexts (with different training scenarios or data distributions) have weaker mutual influence, allowing coverage of a wide range of scenarios and data distributions.

[0016] The embodiments advantageously allow determining a central model (e.g., read from a model pool) to match a specific target deployment scenario and / or target data distribution. The embodiments advantageously allow updating the central model (e.g., writing to the model pool) using a full key (both the local scenario and the data distribution) or a partial key (one of the local scenario and the data distribution). The embodiments also advantageously allow adjusting the size of the model pool (e.g., increasing and / or decreasing the number of central models in the model pool).

[0017] The implementation also advantageously defines a uniform and efficient way to specify keys to identify training data and resources for local client nodes using locality-preserving hashes. The implementation is effective, for example, because it eliminates the need for clustering based on the keys of local client nodes and for evaluating clustering consistency across learning epochs.

[0018] According to a first aspect, a method is provided performed by a central server node. The method includes: receiving local model weights and corresponding keys from a local client node; and updating a model pool having a plurality of central models and corresponding keys associated with each of the central models. Updating the model pool is based on the local model weights and one or more of the following: keys corresponding to the local client nodes, and keys commonly associated with each of the central models. Updating the model pool includes: updating at least two of the plurality of central models included in the model pool.

[0019] In some embodiments, updating the model pool further includes: calculating a similarity score, the similarity score being compared with the key corresponding to the local client node and the key jointly corresponding to each of the plurality of central models, wherein updating at least two of the plurality of central models is based on the similarity score. In some embodiments, the method further includes: selecting the at least two central models from the plurality of central models from the model pool based on the similarity score. In some embodiments, the at least two central models from the plurality of central models in the model pool include all central models in the plurality of central models. In some embodiments, updating the at least two central models from the plurality of central models includes: for each of the at least two central models from the plurality of central models, combining the local model weights with the current version of that central model, such that the local model weights are given greater weight for central models with higher corresponding similarity scores compared to central models with lower corresponding similarity scores.

[0020] In some embodiments, updating at least two of the plurality of central models includes: calculating new model weights for each i in a set of indices corresponding to the at least two of the plurality of central models. in:

[0021] This refers to the new model weights used for the central model corresponding to index i.

[0022] This refers to the old model weights used for the central model corresponding to index i.

[0023]

[0024] W = (w1, ..., w M ) is the similarity score, such that w i The similarity between the key corresponding to the local client node and the key corresponding to the i-th central model among the plurality of central models.

[0025] M is the number of the plurality of central models in the model pool, and

[0026] This refers to the weights of the local model.

[0027] In some embodiments, the similarity score W is normalized such that ∑w i =1. In some embodiments, one or more keys of the key corresponding to the local client node and the key of the model pool that collectively corresponds to each of the plurality of central models each include a data distribution portion and a deployment portion, wherein the data distribution portion of each of the one or more keys includes information describing the data distribution corresponding to the corresponding local client node or central model, and wherein the deployment portion of each of the one or more keys includes information describing the deployment environment corresponding to the corresponding local client node or central model.

[0028] In some embodiments, similarity scores are calculated based on a data distribution portion. and deployment part The first key k1 and the data distribution part and deployment part The similarity function d(k1,k2) between the second keys k2 is such that... In some embodiments, as a result of receiving local model weights and corresponding keys, the model pool is updated without waiting for additional local model weights and corresponding keys from additional local client nodes. In some embodiments, the method further includes modifying the model pool such that the distribution of the keys corresponding to the central model of the model pool is modified based on the corresponding keys from the local client nodes.

[0029] According to a second aspect, a method executed by a central server node is provided. The method includes: receiving a request for a model from a local client node, wherein the request includes a key corresponding to the local client node; in response to receiving the request, constructing a model from a model pool having a plurality of central models and corresponding keys associated with each of the central models; and sending the constructed model to the local client node. Constructing the model from the model pool is based on the key corresponding to the local client node and a key commonly associated with each of the central models. Constructing the model from the model pool includes: aggregating at least two central models from the plurality of central models included in the model pool.

[0030] In some embodiments, constructing the model from the model pool further includes: calculating a similarity score, the similarity score being compared with the key corresponding to the local client node and the key commonly corresponding to each of the plurality of central models, wherein aggregating at least two central models from the plurality of central models is based on the similarity score. In some embodiments, the method further includes: selecting the at least two central models from the plurality of central models from the model pool based on the similarity score. In some embodiments, the at least two central models from the plurality of central models in the model pool include all central models in the plurality of central models. In some embodiments, aggregating at least two central models from the plurality of central models is performed such that central models with higher corresponding similarity scores are weighted more than models with lower corresponding similarity scores among the at least two central models in the plurality of central models.

[0031] In some embodiments, aggregating at least two of the plurality of central models includes: calculating the aggregation model weights. in:

[0032] This refers to the aggregated model weights used in the constructed model.

[0033] M is the number of the plurality of central models in the model pool.

[0034] θi This refers to the model weights used in the central model corresponding to index i, and

[0035] W = (w1, ..., w M ) is the similarity score, such that w i The similarity between the key corresponding to the local client node and the key corresponding to the i-th central model among the plurality of central models.

[0036] In some embodiments, the similarity score W is normalized such that ∑w i =1. In some embodiments, one or more keys of the key corresponding to the local client node and the key of the model pool that collectively corresponds to each of the plurality of central models each include a data distribution portion and a deployment portion, wherein the data distribution portion of each of the one or more keys includes information describing the data distribution corresponding to the corresponding local client node or central model, and wherein the deployment portion of each of the one or more keys includes information describing the deployment environment corresponding to the corresponding local client node or central model.

[0037] In some embodiments, similarity scores are calculated based on a data distribution portion. and deployment part The first key k1 and the data distribution part and deployment part The similarity function d(k1,k2) between the second keys k2 is such that... In some embodiments, the method further includes: modifying the model pool such that the distribution of the keys corresponding to the central model of the model pool is modified based on the corresponding keys from the local client nodes.

[0038] According to a third aspect, a central server node is provided, the central server node including processing circuitry and a memory storing instructions executable by the processing circuitry. The processing circuitry is operable to: receive local model weights and corresponding keys from local client nodes; and update a model pool having a plurality of central models and corresponding keys associated with each of the central models. Updating the model pool is based on the local model weights and one or more of the following: keys corresponding to the local client nodes, and keys collectively corresponding to each of the central models. Updating the model pool includes: updating at least two of the plurality of central models included in the model pool. In some embodiments, the central server node is used to implement accurate distributed machine learning.

[0039] According to a fourth aspect, a central server node is provided, the central server node including processing circuitry and a memory storing instructions executable by the processing circuitry. The processing circuitry is operable to: receive a request for a model from a local client node, wherein the request includes a key corresponding to the local client node; in response to receiving the request, construct a model from a model pool having a plurality of central models and corresponding keys associated with each of the central models; and send the constructed model to the local client node. Constructing the model from the model pool is based on the key corresponding to the local client node and a key commonly associated with each of the central models. Constructing the model from the model pool includes: aggregating at least two central models from the plurality of central models included in the model pool. In some embodiments, the central server node is used to implement accurate distributed machine learning.

[0040] According to a fifth aspect, a computer program including instructions is provided, which, when executed by a processing circuit, cause the processing circuit to perform a method according to any embodiment of the first or second aspect.

[0041] According to a sixth aspect, a carrier is provided that includes the computer program of the fifth aspect, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0042] According to a seventh aspect, a network is provided comprising a central server node and local client nodes communicating with each other, the central server node including processing circuitry and a memory storing instructions executable by the processing circuitry. The processing circuitry is operable to: receive local model weights and corresponding keys from the local client nodes; and update a model pool having a plurality of central models and corresponding keys associated with each of the central models, wherein updating the model pool is based on the local model weights and one or more of the following: keys corresponding to the local client nodes and keys commonly associated with each of the central models, and wherein updating the model pool includes: updating at least two of the plurality of central models included in the model pool.

[0043] According to an eighth aspect, a network is provided comprising a central server node and local client nodes communicating with each other, the central server node including processing circuitry and a memory storing instructions executable by the processing circuitry. The processing circuitry is operable to: receive a request for a model from the local client nodes, wherein the request includes a key corresponding to the local client node; in response to receiving the request, construct a model from a model pool having a plurality of central models and corresponding keys associated with each of the central models; and send the constructed model to the local client nodes, wherein constructing the model from the model pool is based on the key corresponding to the local client node and keys commonly corresponding to each of the central models, and wherein constructing the model from the model pool includes: aggregating at least two central models from the plurality of central models included in the model pool. Attached Figure Description

[0044] Various embodiments are illustrated in the accompanying drawings, which are incorporated herein and form part of the specification.

[0045] Figure 1 A machine learning system according to an embodiment is shown;

[0046] Figure 2 A model pool according to an embodiment is shown;

[0047] Figure 2A and 2B The weight distribution according to an embodiment is shown;

[0048] Figure 3A and 3B The key distribution according to an embodiment is shown;

[0049] Figure 4 A message diagram according to an embodiment is shown;

[0050] Figure 5 A message diagram according to an embodiment is shown;

[0051] Figure 6 This is a flowchart based on an embodiment;

[0052] Figure 7 This is a flowchart based on an embodiment;

[0053] Figure 8 This is a block diagram of the apparatus according to an embodiment;

[0054] Figure 9 This is a block diagram of an apparatus according to an embodiment. Detailed Implementation

[0055] Figure 1A machine learning system 100 according to an embodiment is illustrated. As shown, a central server node or computing device 102 communicates with one or more local client nodes or computing devices 104. Optionally, the local client nodes or computing devices 104 can communicate with each other using any of a variety of network topologies and / or network communication systems. For example, local client nodes 104 include user computing devices such as smartphones, tablets, laptops, and personal computers, and can also be communicatively coupled via public networks such as the Internet (e.g., via WiFi) or communication networks (e.g., LTE or 5G). The central server node 104 may include computing devices such as servers, base stations, mainframes, and cloud computing resources. Although a central server node or computing device 102 is shown, the functionality of the central server node 102 can be distributed across multiple nodes and can be shared among one or more local client nodes 104.

[0056] As shown in the figure, each local client node 104 can transmit model updates to the central server node 102, while the central server node 102 can send the updated central model back to the local client node 104. The links between local client nodes 104 are shown as bidirectional between these entities (e.g., having a bidirectional link, or via different communication channels).

[0057] The joint learning described in this embodiment may involve one or more rounds, in which a central model is trained iteratively in each round. Local client nodes 104 may register with central server node 102 to indicate their willingness to participate in the joint learning of the central model, and this may be done continuously or in a rolling fashion. Upon registration (and possibly at any time thereafter), central server node 102 sends training parameters to local client nodes 104. Central server node 102 may send an initial model to local client nodes 104. For example, central server node 102 may send a central model (e.g., a newly initialized model or one partially trained through previous rounds of joint learning) to local client nodes 104. Local client nodes 104 may train their individual models locally using their own data. The results of this local training can then be fed back to central server node 102, which can aggregate the results and update the global model. This process may be repeated iteratively. Furthermore, in each round of training the central model, central server node 102 may select a subset (e.g., a random subset) of all registered local client nodes 104 to participate in that training round.

[0058] Central server node 102 is responsible for managing the model pool. As used in this description, a "training node" refers to a local client node 104 participating in learning; while an "aggregation node" refers to the central server node 102 where the model pool is active. In the following discussion, it is assumed that the training node implements a regression or classification task using a neural network, where labels are only available locally. Other types of machine learning models may also be applicable in this embodiment.

[0059] In some embodiments, the central server node 102 can be used to implement accurate distributed machine learning.

[0060] For the purposes of discussion, the following description assumes multiple training nodes i = 1, ..., N (a subset of participating local client nodes 104) and one aggregation node i = 0 (central server node 102). Each training node i collects data from an unknown distribution x ~ pi, where the label y is provided locally by the system of interest, and x ∈ R. n And y∈R m , n, m ≥ 1. The problem in node i then includes learning the parameterized function f from the example (x, y). i : Where the function parameter Θ i This represents the weights of the neural network.

[0061] The model pool can be identified by (1) the key K and (2) the central model Θ. Therefore, the model pool can be represented in this paper as (K,Θ), where bold indicates that the referenced elements are vectors. The central models are easy to define because they are the weights from the neural network architecture, equal to the weights in the training nodes, and are denoted here as F.

[0062] exist Figure 1 In this context, node 0 (central server node 102) represents the aggregation node, where the model pool (K, Θ) is active and manages updates from training nodes i = 1, ..., N. Each training node is defined by its key k. i and the local model Θ i To identify.

[0063] Model pool operations

[0064] Treat the key as a vector Where l > 0, and the weights of the corresponding neural network are considered as vector Θ. j ∈R d Where d >> 0. Then, the model pool will have M > 0 key-model pairs (k j ,Θ j The data is stored in a matrix P of dimension (M, l+d). Although P is referred to as a matrix in this discussion, any suitable data structure or storage format for the model pool is within the scope of the disclosed embodiments.

[0065] From matrix P, each read or write operation depends on calculating the similarity between the input key k stored in P and all keys K. The calculated similarity W = (w1,…,w M Then it is normalized to a sum equal to 1. For example, normalization can be performed using the softmax function, for example:

[0066]

[0067] It was observed that increasing the cardinality b > 0 to a larger value produces a more concentrated probability distribution around the maximum similarity, while keeping the cardinality b small produces a more broadly concentrated probability distribution (comparison). Figure 2A and 2B In the limit, as b increases, only the most similar rows will have a sufficiently large value to affect read or write operations. Conversely (when b is close to 0), most rows will affect read or write operations. Through this preprocessing of similarity and a specific b value, W can be redefined as:

[0068] W=(σ(W,b,1),σ(W,b,2),…,σ(W,b,M))

[0069] It should be understood that other methods exist to normalize the calculated similarity in W.

[0070] Read

[0071] The read operator receives the preprocessed (i.e., normalized) similarity vector W, and, for example, the aggregate model Θ generated by a weighted sum of the central models stored in the pool. READ An "operator" is typically a mapping that acts on elements of one space to produce elements of another space (which may be the same space). The similarity vector W and model Θ are read from the model pool to produce the aggregated model Θ. READ This action is called a read operation.

[0072] The read operation is performed in response to a model request from local client node 104. The model request from local client node 104 may include a key corresponding to local client node 104. Based on the key corresponding to local client node 104 and the common key K stored in the model pool corresponding to each central model in the central model, an aggregate model Θ is constructed. READ An aggregate model Θ is constructed by using at least two models from the aggregate model pool. READFor example, the key corresponding to local client node 104 can be used to generate a similarity vector W, which stores the similarity between that key and all keys K stored in the model pool. At least two models in the aggregated model pool can be further selected based on the similarity vector W. At least two models in the model pool can be selected based on the similarity vector W, or may include all models in the model pool.

[0073] This aggregation can be performed such that models containing corresponding keys with higher similarity scores (i.e., larger corresponding components of the similarity vector W) are weighted more than models containing corresponding keys with lower similarity scores. For example, in some embodiments, the read operation produces an aggregated model Θ by a weighted sum of the central models stored in the pool. READ As shown below:

[0074]

[0075] The read operation generates the aggregation model Θ READ Then, the aggregation model Θ READ It can be sent to the local client node 104 that sent the model request.

[0076] Figure 2 The model pool 200 is shown in matrix P. As shown in the figure, there are M key-model pairs (k...) in matrix P of dimension (M, l+d). j ,Θ j ). Figure 2A and 2B Two different normalized similarity vectors are shown, one using a small cardinality ( Figure 2A And another uses a large cardinality ( Figure 2B The similarity vector identifies which matrix rows (the central models stored in the model pool) should have a stronger influence on the resulting aggregate model. The higher the similarity score, the greater the influence; the lower the similarity score, the smaller the influence. For example, for w... i A similarity score of ≈0 should have minimal impact. It was observed that, in the extreme case where the cardinality b is very large, only the row whose key is closest to the key received as input significantly affects the creation of the resulting model.

[0077] In some embodiments, each model Θ j Possible impact on aggregation model Θ READ Even if the impact is small. In other embodiments, the model Θ is selected based on the similarity vector W. j A subset such that only the selected subset affects the aggregation model Θ. READFor example, in some embodiments, only models with similarity scores above a threshold (e.g., 0.05) may be selected, and in some embodiments, only the top n models (e.g., the top 20) or the top n% (e.g., the top 20%) based on similarity scores may be selected.

[0078] Write

[0079] The write operator receives the preprocessed similarity vector W and the model update generated from local training. As input, an "operator" is typically a mapping that operates on elements of one space to produce elements of another space (which may be the same space). The write operator operates on the similarity vector W and the received model update. This generates an updated model pool. This action is called a write operation.

[0080] The write operation is in response to receiving local model weights (i.e., local updates) from local client node 104. This is executed when the local client node 104 provides local model weights. The local client node 104 may include a key corresponding to the local client node 104. The model pool may be updated based on the local model weights and one or more of the following: the key corresponding to the local client node 104, and a common key K stored in the model pool corresponding to each central model in the central model pool. Updating the model pool includes updating at least two models among the models stored in the model pool. For example, the key corresponding to the local client node 104 may be used to generate a similarity vector W, which stores the similarity between that key and all keys K stored in the model pool. The similarity vector W may be used when updating the model pool. The at least two models being updated may be selected based on the similarity vector W, or may include all models in the model pool.

[0081] By updating The model pool is updated in combination with the current version of each model in the model pool being updated, such that for a given model with a higher corresponding similarity score (i.e., a larger corresponding component of the similarity vector W) compared to a given model with a lower corresponding similarity score, the update is performed. The similarity vector W is weighted more than the current version of the given model. For example, in some embodiments, the similarity vector W can be used to scale model updates as follows. First, a weighted average can be calculated for each entry i in the model pool:

[0082]

[0083] Next, the new update and stored value Θ iIt can be averaged and stored back into the model pool (note that during the calculation...). (Weightedness has already been applied):

[0084]

[0085] Instead of operating on each entry i in the model pool, some embodiments can operate on a subset of entries i in the model pool.

[0086] Because each model update It can be processed independently, so it is not necessary to wait for each training node to provide its update before updating the model pool. In other words, the model pool can be updated as model updates arrive from each training node.

[0087] In some embodiments, each model Θ j May be affected by model updates Even if the impact is small. In other embodiments, the model Θ is selected based on the similarity vector W. j A subset such that only the selected subset is updated by the model. The impact of similarity scores. For example, in some embodiments, only models with similarity scores above a threshold (e.g., 0.05) may be selected, and in other embodiments, only the top n models or the top n% of models based on similarity scores may be selected.

[0088] Key properties

[0089] Read and write operations depend on keys and are used to determine the similarity between models. Therefore, it is important that the keys associated with a model carry information about two or more aspects of that model: (1) the target data distribution and (2) the target scenario. The target data distribution refers to the distribution of the training data for the model, while the target scenario refers to the characteristics of the deployment environment related to the non-training data. For example, the target scenario for a mobile phone may include manufacturer information, network standards (e.g., 4G, 5G), and device resource information (e.g., memory, processing power, etc.). In some embodiments, the target scenario information may depend on the problem domain.

[0090] Each training node can generate its own keys locally. The local dataset and scene description of each training node are referred to as D and S. The process of generating keys k involves mapping the high-dimensional inputs D and S to the low-dimensional outputs h(D) and h(S).

[0091] To ensure that keys enable read and write operations for appropriate entries in matrix P, the function h should be defined as locality-preserving. This property effectively guarantees that the two output keys h(D1) and h(D2) should be similar if and only if the local datasets D1 and D2 are similar. Techniques for locality-sensitive hashing typically employ functions with this property. In some embodiments, a specific locality-preserving hash function can be chosen based on the domain of the dataset. The key property used in this discussion is that the locality-preserving hash function produces a vector with a lower dimension than the original data. Therefore, given a local pair of datasets and a scene description (D, S) and a corresponding partial key k... D =h(D) and k S =h(S), which can generate a complex bond K=(k D ,k S If other local information is relevant to describing the training node, then that other local information can also be part of the composite key.

[0092] The key used for training nodes identifies the current state of the training node at a given time. In some embodiments, such as periodically or at specific events (e.g., each time the local model is sent to the aggregation node for a write operation), the key can be recalculated (from the local training data and the scenario description). Typically, the cost of recalculating the key in each round of learning is negligible due to two facts: (1) the local training dataset in joint learning is usually not very large; and (2) local retention hashing techniques are generally not complex. As an example, the well-known MinHash technique has a worst-case complexity of O(nk), meaning it is no worse than scaling linearly with the size of the dataset n and a constant k, which allows for improved accuracy.

[0093] Calculate similarity

[0094] Read and write operations depend on the similarity between the input key k (corresponding to the training node) and the reference key K stored in the model pool (which together correspond to each central model in the central models stored in the model pool). As defined in the previous section, each partial key identifies a characteristic of the training node (e.g., data distribution, scenario), while the composite key concatenates all such identifiers.

[0095] In an embodiment, the similarity metric used to calculate similarity should be able to take into account partial keys, i.e., partial knowledge of the training nodes. For example, consider the following scenario:

[0096] • (Case A) Deploy the model to a new edge node whose characteristics and settings (scene) are known. Because it is a new deployment, local data is not available. The example allows the node to read the model from the model pool to match the target scene (partial key).

[0097] • (Scenario B) Deploy the model to an edge device where its characteristics and settings are unknown, but local data samples are already available. This example allows nodes to read models from a model pool to match the target data distribution.

[0098] In case A, no data is available. Therefore, there is no partial key k associated with the target data distribution. D That is, K = (-, k S Conversely, in case B, there is no description of the target deployment scenario. Therefore, there is no associated partial key k. S That is, K = (k D ,-).

[0099] The similarity function can take any form. In some embodiments, the similarity function can be the sum of the similarities between partial keys:

[0100]

[0101] To support partial key unavailability, the similarity between the unavailable partial key and the partial key (e.g.) The value of ) can be defined as zero. In this case, unavailable keys do not affect similarity, meaning that all central models in the pool fit equally well for such keys. Therefore, only the remainder of the key affects similarity.

[0102] A well-defined similarity metric can support both cases A and B, as illustrated in the following example:

[0103] • (Scenario A) Hash the scenario description S to generate a partial key k S =h(S). The model pool manager calculates the composite key K = (-,k S The similarity between the key k and the stored reference key. Because part of the key k D Unavailable, so only k S This affects similarity. Due to the local preservation of keys, read operations return an aggregate model that is biased towards the central model whose scene is closest to the target scene.

[0104] • (Scenario B) Hash the sample local data D to generate a partial key k D =h(D). The model pool manager calculates the composite key K = (k D The similarity between the key k and the stored reference key. Because some keys k... S Unavailable, so only k D This affects similarity. Due to the local preservation of keys, read operations return a model that is biased towards the central model whose training data distribution is closest to the target distribution.

[0105] • If both partial keys are available (which is the most common case), the read operation returns a model biased towards its training data distribution and the central model of the training data distribution of the scene closest to the training node. The same applies to the write operation.

[0106] It is worth noting that, given a locally preserved hash function, similarity metrics can be quite simple and are still useful for identifying the appropriate central model in a pool of models.

[0107] Maintaining reference keys

[0108] The read and write operations described in this paper allow the central model to converge to similar locally trained models. Because similarity is a result of comparing keys, it eliminates the need to directly compare the weight vectors of large neural networks, improving the scalability of the method. In addition to properly defined read and write operations, it may also help improve accuracy to ensure the representativeness of the stored central model.

[0109] To better explain what representativeness means in this context, consider Figure 3A and 3B The solid black circles represent the keys of the three stored central models, while the hollow circles represent the keys of the previously read or written local training models. Figure 3A This illustrates cases where the reference keys (and corresponding central models) do not adequately cover (or represent) many models being read or written because their keys are too far away. Figure 3B The diagram shows a better scenario where the reference keys (and corresponding central models) are better distributed. Read and write operations can be further improved by ensuring that the keys of the central model well represent the locally trained model.

[0110] At the start of the learning process from distributed data, the central model is not trained. Therefore, it's natural to use the initially written model to populate the model pool, using the corresponding key as the reference key within the pool. However, this approach may lead to... Figure 3A The situation depicted exhibits poor representativeness. In an embodiment, the central models stored in the model pool and their corresponding keys can be updated periodically or otherwise (e.g., after each read or write operation) to provide better representativeness, for example, by moving keys to areas where more operations are being performed. That is, in an embodiment, the key adaptation can cover the overall distribution of keys read or written during read and write operations.

[0111] To update the representativeness of the model pool, two basic methods can be used: one is to use a static model pool size, and the other is to use a dynamic model pool size. Exemplary embodiments of the two methods are described below.

[0112] Static model pool size

[0113] If the size of the model pool P (number of rows M) is fixed, i.e., it is not allowed to grow or shrink, then reference keys can be maintained by employing an online k-means clustering strategy. Such a strategy treats the data stream (of pairs (k, Θ) in the context of this disclosure) as input and updates the positions of the k centroids whenever a pair (k, Θ) is received. (Note that k in k-means refers to the number of clusters, which in this context is converted to the number of rows M in P. Italic letters are used when referring to keys, while non-italicized k is used as a descriptor for the traditional k-means process.) Given sufficient read and / or write operations, the online k-means clustering process adapts the keys stored in P to achieve good coverage of the overall distribution of the keys, resulting in... Figure 3B The situation is shown.

[0114] In short, the process can be performed as follows: The first M key-model pairs written to the model pool are stored as initial references. Each subsequent read or write operation triggers a slight adaptation of the nearest reference key toward the key provided to the read or write operator. This can be viewed as a warm-up phase, where the model pool and its key-model pairs adapt to the local model's data distribution.

[0115] Note that changing a reference key in the model pool necessarily requires changing the corresponding central model. For example, if a new reference key k is determined... new This allows us to modify the key k before the model pool is modified. new Perform a read operation to determine the corresponding model for the key. Then, the model pool can be modified by replacing the key-model pair with the newly computed key-model pair.

[0116] While simpler to implement, static model pool size has its limitations. It's impossible to know in advance how many reference keys will be used for a particular use case. If M is less than necessary, the model pool reference keys may not accurately represent the multi-modal local data distribution. If M is too large, many reference keys become useless for most read and write operations. Therefore, in some implementations, a dynamic approach may be convenient.

[0117] Dynamic model pool size

[0118] If the size of the model pool P (number of rows M) is dynamic, i.e., allowed to grow or shrink, reference keys can be maintained by employing alternative strategies (e.g., growing neural gas techniques). Such methods solve the same problem as online k-means clustering techniques, but they additionally provide a mechanism for detecting the need for inserting or deleting reference keys, which would allow the model pool to grow or shrink.

[0119] In short, the process can be performed as follows: The first two key-model pairs written to the model pool are stored as initial references. Each subsequent read or write operation triggers a slight adaptation of some reference keys toward the keys provided to the read or write operator. Shrinking occurs by removing reference keys that are rarely close to the keys provided to the read or write operator. Growth occasionally occurs by inserting new reference keys to cover the areas where read or write operations have occurred. The shrinking and growing phases create a dynamic model pool.

[0120] Note that changing a reference key in the model pool necessarily requires changing the corresponding central model. For example, if a new reference key k is determined... new This allows us to modify the key k before the model pool is modified. new Perform a read operation to determine the corresponding model for the key. The model pool can then be modified either by replacing the key-model pair with a newly computed key-model pair or by inserting a new key-model pair into the model pool without replacing the existing one.

[0121] Figure 4 A message flow diagram according to an embodiment is shown. As shown, two local client nodes 104 communicate with a central server node 102 that maintains the model pool 200.

[0122] In the illustrated "Federated Session Client 1" loop, local client node 1 requests a model from central server node 102 via a key (at 410). Central server node 102 performs a read operation on model pool 200 using the key from local client node 1 to obtain a model via the provided key (at 412, 414). Central server node 102 then sends the model to local client node 1 that requested the model (at 416).

[0123] A similar sequence occurs in the illustrated "Federated Session Client 2" loop. Local client node 2 requests a model from central server node 102 via a key (at 418). Central server node 102 performs a read operation on model pool 200 using the key from local client node 1 to obtain a model via the provided key (at 420, 422). Central server node 102 then sends the model to local client node 1 that requested the model (at 424).

[0124] As shown in the figure, when local client node 104 (local client node 1 or 2) requests the model, local client node 104 can include the key in its request.

[0125] Figure 5A message flow diagram according to an embodiment is shown. As shown, local client node 104 communicates with central server node 102 that maintains model pool 200. Local client node 104 may include one or more modules, including data source 502, local database 504, model trainer 506, and key manager 508. These modules are shown as separate entities of local client node 104; however, it should be understood that their functionality may be implemented in more or fewer modules than shown herein, and these modules may be located remotely from local client node 104.

[0126] In the illustrated "data collection" loop, data collected by data source 502 is stored in local database 504 (at 510). For example, in an embodiment where the model prediction part of the model pool contains the suffixes of words typed in the model pool, the data may include the character sequence typed by the user on local client node 104. Typically, the collected data will be application-specific.

[0127] In the illustrated "Obtain Model" loop, model trainer 506 requests a local key from key manager 508, which then provides the key (at 512, 514). Model trainer 506 can then use the local key to request a model from central server node 102 (at 516). Central server node 102 uses this key to obtain the model (see, for example, [link to relevant documentation]). Figure 4 Then the model is sent to the model trainer 506 (at 518) that requested the model.

[0128] In the illustrated “training” loop, model trainer 506 requests data samples from local database 504, which then provides the data samples (at 520, 522). This could, for example, include data collected during the “data collection” loop. Model trainer 506 then uses this data to train the model (at 524).

[0129] The trained model can then be sent to the central server node 102. The local client node 102 can also provide the local key to the central server node 102 so that the central server node 102 can perform write operations on the model pool 200.

[0130] Figure 6 A flowchart according to an embodiment is shown. Process 600 is a method executed by a central server computing device 102. Process 600 may begin at step s602.

[0131] Step s602 includes: receiving local model weights and corresponding keys from the local client node.

[0132] Step s604 includes: updating the model pool, which has multiple central models and corresponding keys associated with each central model in the central models.

[0133] Step s608 includes: wherein updating the model pool is based on local model weights and one or more of the following: keys corresponding to local client nodes, and keys that collectively correspond to each central model in the central model.

[0134] Step s610 includes: wherein updating the model pool includes: updating at least two central models among a plurality of central models contained in the model pool.

[0135] In some embodiments, updating the model pool further includes: calculating a similarity score, which compares the key corresponding to the local client node and the key commonly corresponding to each of a plurality of central models, wherein updating at least two of the plurality of central models is based on the similarity score. In some embodiments, the method further includes: selecting at least two central models from the plurality of central models from the model pool based on the similarity score. In some embodiments, the at least two central models from the plurality of central models in the model pool include all central models among the plurality of central models. In some embodiments, updating at least two of the plurality of central models includes: for each of the at least two central models among the plurality of central models, combining local model weights with the current version of that central model, such that local model weights are given greater weight for central models with higher corresponding similarity scores compared to central models with lower corresponding similarity scores.

[0136] In some embodiments, updating at least two of the plurality of central models includes: calculating new model weights for each i in a set of indices corresponding to at least two of the plurality of central models. in:

[0137] This refers to the new model weights used for the central model corresponding to index i.

[0138] This refers to the old model weights used in the central model corresponding to index i.

[0139]

[0140] W = (w1, ..., w M ) is the similarity score, such that w i This corresponds to the similarity between the key corresponding to the local client node and the key corresponding to the i-th central model among multiple central models.

[0141] M is the number of central models in the model pool, and

[0142] This refers to the weights of the local model.

[0143] In some embodiments, the similarity score W is normalized such that ∑w i =1. In some embodiments, one or more keys, one or more of the keys corresponding to the local client node and the keys of the model pool corresponding to each of the plurality of central models, each include a data distribution portion and a deployment portion, wherein the data distribution portion of each of the one or more keys includes information describing the data distribution corresponding to the corresponding local client node or central model, and wherein the deployment portion of each of the one or more keys includes information describing the deployment environment corresponding to the corresponding local client node or central model. In some embodiments, the similarity score is calculated based on the data distribution portion. and deployment part The first key k1 and the data distribution part and deployment part The similarity function d(k1,k2) between the second keys k2 is such that...

[0144] In some embodiments, as a result of receiving local model weights and corresponding keys, the model pool is updated without waiting for additional local model weights (e.g., from so-called stragglers, i.e., local nodes that have not provided their local updates in a timely manner) and corresponding keys from additional local client nodes. In some embodiments, the method further includes modifying the model pool such that the distribution of keys corresponding to the central model of the model pool is modified based on the corresponding keys from the local client nodes. For example, the static or dynamic model pool size method described above can be used to update the model pool such that the distribution of keys corresponding to the central model of the model pool is modified based on the corresponding keys from the local client nodes.

[0145] Figure 7 A flowchart according to an embodiment is shown. Process 700 is a method executed by a central server computing device 102. Process 700 may begin at step s702.

[0146] Step s702 includes: receiving a request for the model from a local client node, wherein the request includes a key corresponding to the local client node.

[0147] Step s704 includes: in response to receiving the request, constructing a model from a model pool having multiple central models and corresponding keys associated with each central model.

[0148] Step s706 includes: sending the constructed model to the local client node.

[0149] Step s708 includes: wherein, constructing a model from the model pool is based on a key corresponding to the local client node and a key that commonly corresponds to each central model in the central model.

[0150] Step s710 includes: wherein constructing a model from the model pool includes: aggregating at least two central models from a plurality of central models contained in the model pool.

[0151] In some embodiments, constructing a model from a model pool further includes: calculating a similarity score, which compares the similarity score with the key corresponding to the local client node and the key commonly corresponding to each of a plurality of central models, wherein aggregating at least two central models from the plurality of central models is based on the similarity score. In some embodiments, the method further includes: selecting at least two central models from the plurality of central models from the model pool based on the similarity score. In some embodiments, the at least two central models from the plurality of central models include all central models among the plurality of central models. In some embodiments, aggregating at least two central models from the plurality of central models is performed such that central models with higher corresponding similarity scores are weighted more heavily than models with lower corresponding similarity scores.

[0152] In some embodiments, aggregating at least two of a plurality of central models includes: calculating the aggregation model weights. in:

[0153] This refers to the aggregated model weights used in the constructed model.

[0154] M is the number of central models in the model pool.

[0155] θ i This refers to the model weights used in the central model corresponding to index i, and

[0156] W = (w1, ..., w M ) is the similarity score, such that w i This corresponds to the similarity between the key corresponding to the local client node and the key corresponding to the i-th central model among multiple central models.

[0157] In some embodiments, the similarity score W is normalized such that ∑w i =1. In some embodiments, one or more keys, one or more of the keys corresponding to the local client node and the keys of the model pool that collectively correspond to each of the plurality of central models, each include a data distribution portion and a deployment portion, wherein the data distribution portion of each of the one or more keys includes information describing the data distribution corresponding to the corresponding local client node or central model, and wherein the deployment portion of each of the one or more keys includes information describing the deployment environment corresponding to the corresponding local client node or central model.

[0158] In some embodiments, similarity scores are calculated based on a data distribution portion. and deployment part The first key k1 and the data distribution part and deployment part The similarity function d(k1,k2) between the second keys k2 is such that... In some embodiments, the method further includes: modifying the model pool such that the distribution of keys corresponding to the central model of the model pool is modified based on the corresponding keys from local client nodes. For example, the static or dynamic model pool size method described above can be used to update the model pool so that the distribution of keys corresponding to the central model of the model pool is modified based on the corresponding keys from local client nodes.

[0159] Figure 8 This is a block diagram of an apparatus 800 (e.g., a local client node 104 and / or a central server node 102) according to some embodiments. Figure 8As shown, the device may include: a processing circuitry (PC) 802, which may include one or more processors (P) 855 (e.g., a general-purpose microprocessor and / or one or more other processors, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.); a network interface 848, which includes a transmitter (Tx) 845 and a receiver (Rx) 847, for enabling the device to send data to and receive data from other nodes connected to a network 810 (e.g., an Internet Protocol (IP) network) to which the network interface 848 is connected; and a local storage unit (also referred to as a “data storage system”) 808, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where the PC 802 includes a programmable processor, a computer program product (CPP) 841 may be provided. The CPP 841 includes a computer-readable medium (CRM) 842 storing a computer program (CP) 843, the CP 843 including computer-readable instructions (CRI) 844. CRM 842 may be a non-transitory computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a storage device (e.g., random access memory, flash memory), etc. In some embodiments, CRI 844 of computer program 843 is configured such that when executed by PC 802, CRI causes the device to perform the steps described herein (e.g., the steps described herein with reference to the flowcharts). In other embodiments, the device may be configured to perform the steps described herein without requiring code. That is, for example, PC 802 may comprise only one or more ASICs. Therefore, the features of the embodiments described herein may be implemented in hardware and / or software.

[0160] Figure 9 This is a schematic block diagram of an apparatus 800 according to some other embodiments. The apparatus 800 includes one or more modules 900, each implemented in software. Modules 900 provide the functionality of the apparatus 800 described herein (e.g., the steps described herein, for example, for...). Figure 4-7 ).

[0161] Although various embodiments of this disclosure have been described herein, it should be understood that they are provided by way of example only and not as limiting. Therefore, the breadth and scope of this disclosure should not be limited to any of the exemplary embodiments described above. Furthermore, unless otherwise indicated herein or otherwise clearly contradicted by the context, this disclosure includes any combination of all possible variations of the foregoing elements.

[0162] Additionally, although the process described above and shown in the accompanying drawings is presented as a series of steps, this is for illustrative purposes only. Therefore, it is conceivable that some steps may be added, some steps may be omitted, the order of steps may be rearranged, and some steps may be performed in parallel.

Claims

1. A method executed by a central server node (102), the method comprising: Receive a request for the model from the local client node (104), wherein the request includes a key corresponding to the local client node (104); In response to receiving the request, a model is constructed from a model pool (200), the model pool (200) having multiple central models and corresponding keys associated with each of the central models; and The constructed model is sent to the local client node (104). The model is constructed from the model pool (200) based on the key corresponding to the local client node (104) and the key that commonly corresponds to each central model in the central model. The process of constructing the model from the model pool (200) includes: aggregating at least two central models from the plurality of central models contained in the model pool (200); The construction of the model from the model pool (200) further includes: calculating a similarity score, which is compared with the key corresponding to the local client node (104) and the key corresponding to each of the plurality of central models, and wherein the aggregation of at least two of the plurality of central models is based on the similarity score; Among them, one or more keys, including the key corresponding to the local client node (104) and the key of the model pool that collectively corresponds to each of the plurality of central models, each include a data distribution portion and a deployment portion. The data distribution portion of each of the one or more keys includes information describing the data distribution corresponding to the corresponding local client node (104) or central model, and The deployment portion of each of the one or more keys includes information describing the deployment environment corresponding to the corresponding local client node (104) or central model, the deployment environment information including mobile phone manufacturer information, network standards, memory, and processing power.

2. The method according to claim 1, further comprising: Based on the similarity score, at least two central models are selected from the plurality of central models in the model pool (200).

3. The method according to claim 1, wherein, The at least two central models from the plurality of central models in the model pool (200) include all the central models in the plurality of central models.

4. The method according to any one of claims 1-3, wherein, At least two central models from the plurality of central models are aggregated such that the central model with a higher corresponding similarity score is weighted more than the model with a lower corresponding similarity score.

5. The method according to claim 4, wherein, Aggregating at least two of the multiple central models includes: calculating the aggregated model weights. ,in: This refers to the aggregated model weights used in the constructed model. M is the number of the plurality of central models in the model pool (200). Refers to the use of indexes i The corresponding model weights of the central model, and W = (w1, …, w M ) is the similarity score, such that w i The key corresponding to the local client node (104) and the key in the plurality of central models i The similarity between the keys corresponding to the central model.

6. The method according to claim 5, wherein, The similarity score W is normalized so that .

7. The method according to claim 1, wherein, The similarity score is calculated based on the data distribution portion. and deployment part First key With data distribution part and deployment part The second key Similarity function between So that .

8. The method according to any one of claims 1-3, further comprising: The model pool (200) is modified such that the distribution of the keys corresponding to the central model of the model pool (200) is modified based on the corresponding keys from the local client node (104).

9. A central server node (102) comprising processing circuitry (802) and a memory storing instructions (844) executable by the processing circuitry (802), thereby enabling the processing circuitry (802) to: Receive a request for the model from the local client node (104), where, The request includes a key corresponding to the local client node (104); In response to receiving the request, a model is constructed from a model pool (200), the model pool (200) having multiple central models and corresponding keys associated with each of the central models; and The constructed model is sent to the local client node (104). The model is constructed from the model pool (200) based on the key corresponding to the local client node (104) and the key that commonly corresponds to each central model in the central model. The process of constructing the model from the model pool (200) includes: aggregating at least two central models from the plurality of central models contained in the model pool (200); The construction of the model from the model pool (200) further includes: calculating a similarity score, which is compared with the key corresponding to the local client node (104) and the key corresponding to each of the plurality of central models, and wherein the aggregation of at least two of the plurality of central models is based on the similarity score; Among them, one or more keys, including the key corresponding to the local client node (104) and the key of the model pool that collectively corresponds to each of the plurality of central models, each include a data distribution portion and a deployment portion. The data distribution portion of each of the one or more keys includes information describing the data distribution corresponding to the corresponding local client node (104) or central model, and The deployment portion of each of the one or more keys includes information describing the deployment environment corresponding to the corresponding local client node (104) or central model, the deployment environment information including mobile phone manufacturer information, network standards, memory, and processing power.

Images