Layered aggregation method to optimize communication overhead for federated learning

Abstract

First client (10, 10'a, 10'b, 10'c) of a network with at least the first client (10, 10'a, 10'b, 10'c) and a second client (11a, 11b, 11c), having the following features: a receiver configured to receive a first (L1) of at least two layers of model data of a second client (11a, 11b, 11c) and a second of the at least two layers of second model data of the second client (11a, 11b, 11c); a processor configured to train and / or update a local or global model (M), wherein the processor updates the local or global model (M) based on the received first (L1) of the at least two layers in a first step; wherein the processor is designed to use and train an initially obtained global model (M) as a local model (M) in order to update the first (L1) and / or second layer (L2) of the local model (M); and wherein the updating of the local model (M) in the first (L1) and / or second layer is carried out on the basis of a received first (L1) and / or second layer (L2) of the second model data of the second client (11a, 11b, 11c), and / or on the basis of an internally determined first (L1) and / or second layer (L2) of first model data of the first client (10, 10'a, 10'b, 10'c) during training in operation in order to obtain an updated local model (M); wherein the first client (10, 10'a, 10'b, 10'c) further comprises a transmitter configured to forward an updated first (L1) and / or second layer (L2) of the local model (M) to another decentralized client, the second client (11a, 11b, 11c) or third client; where the forwarding is repeated for a layer that was not forwarded in the previous step(s).

Description

[0001] Embodiments relate to a first client of an artificial neural network, in particular to a decentralized or centralized client with an aggregation function. Further embodiments relate to a second client of an artificial neural network (with and without an aggregation function). Further embodiments relate to a corresponding system with clients, corresponding operating methods for operating the first and / or second client, and the execution of computer programs. Application examples relate to federated learning, in particular to the aggregation method in a hierarchical or non-hierarchical communication structure. In general, embodiments of the present invention are based on a concept for federated machine learning with layered aggregation.

[0002] Federated machine learning is already partially disclosed in the prior art. For example, the publication titled "From Distributed Machine Learning to Federated Learning: a Survey" (Liu, Ji [et al.]) discloses a functional architecture of a federated learning system and a classification of the associated technology. Furthermore, US 2022 / 114475 A1 discloses methods and systems for decentralized federated learning. The publications "Federated Learning with Matched Averaging" (Wang, Hongyi [et al.]) and "Federated Optimization in Heterogeneous Networks" (Li, Tian [et al.]) are further prior art documents.

[0003] Given predictions that there will be one trillion connected devices by 2035, machine learning (ML) is a key technology for decision-making in the big data era. Traditional ML algorithms typically deploy ML models in a centralized system, collecting data generated by end devices and consolidating it onto a single server for training. This data often comes from raw data generated by Internet of Things (IoT) devices (wearables, telemetry devices, smartphones, etc.). However, data from IoT devices is often private or sensitive and thus subject to the General Data Protection Regulation (GDPR). Furthermore, the data volumes can be too large to transmit across networks, making training a centralized model impossible. To overcome these limitations, Google developed federated learning (FL) in 2017.

[0004] FL is a decentralized ML method in which training is carried out on the devices on which the data (e.g. sensor data) is collected. Only the trained ML models are then sent and subsequently summarized or aggregated at a central location or at multiple decentralized locations. After aggregation, a new ML model (global model) is created, which is then sent back to the participating devices, for example. This allows the devices to learn from each other without having to exchange raw data. This differs fundamentally from traditional centralized ML, in which the collected raw data is sent, for example, to a cloud where training takes place. However, it has already been demonstrated that the global model can be used to calculate pixel-precisely back to the original image in an image classification task.

[0005] In FL, it is essential that the weights or gradients are exchanged after training on a device (local model), either with a cloud, a local unit, or multiple units. All received local models are then aggregated, creating a new ML model, which is called a global model. Sending the local and global models results in a high data transfer. Data transfer is particularly problematic for low-performance devices or for devices with limited bandwidth to the aggregating unit. Leading ML models have several million or even billions of parameters (the SwinV2-G model has three billion parameters), which must be transferred to multiple devices in FL.

[0006] There are a variety of solutions (currently 852 papers 3, which were generally developed and are suitable for specific use cases. Currently existing methods offer solutions, but these approaches are usually based on the use of one or more high-performance systems. A list of various aggregation algorithms is shown in Table 1. The most widely used methods are Federated Averaging (FedAvg) and FedProx, which are explained in more detail below. The asynchronous model update strategy is also presented, as this method also sends individual layers for aggregation. Aggregation algorithm Berelch (destinations) Data distribution Communication architecture Notes FedAvg Basic algorithms Horizontal Central SGD-based FedProx FedNova QNA Vertical Quasi-Newton method FedMA Communication and / or computing efficient Horizontal Central Layer-wise customized aggregation FedBoost Pre-trained base predictors FetchSGD Compression of the model by sketches FedPAQ Periodic averaging and quantization AsynchronousModelUpdate Asynchronous model update strategy and weighted aggregation QuantizationRobust FL onHeterogeneousdevices Quantization with heterogeneous devices PruneFL Adapts the model size during FL VAFL Vertical Asynchronous FL SCAFFOLD Model Quality Improvement Horizontal Central Correct control variables around client drift OFMTL Online multi-task learning algorithm SMA Selective aggregation approach in the vehicle edge computing context Joint Learning and Communications Framework for FL overWireless Networks Improving communication Horizontal Central Wireless communication BrainTorrent Decentralized network structure Horizontal Decentralized Peer-to-peer network structure BFCL Blockchain-based data exchange Decentralized FL with unreliable communications Decentralized SGD approach called Soft-DSGD for unreliable devices

[0007] The use of FL on embedded systems is primarily limited by memory space, computing power, and communication costs. This creates a need for a more efficient approach that can be deployed on resource-constrained systems or devices.

[0008] The object of the present invention is to create a concept for federated machine learning that creates an improved compromise between learning efficiency and resource efficiency.

[0009] The problem is solved by the subject matter of the independent patent claims.

[0010] Embodiments of the present invention provide a first client of a network having at least the first client and a second client. The first client has a receiver and a processor. The receiver is configured to receive a first of at least two layers of model data from a second client and a second of the at least two layers of model data from the second client. The processor is configured to train and / or update a local or global model, wherein the processor updates the local or global model based on the received first of the at least two layers in a first step.

[0011] The processor has an artificial neural network (ANN) or an AI algorithm (artificial intelligence) that is trained during training.

[0012] According to embodiments, the processor is further configured to update the local and global models based on the received second at least two layers in a second step. In this respect, a layer-by-layer aggregation of model data occurs.

[0013] According to embodiments, this first client can either be a central client with aggregation functionality (aggregation of Layer 1 (Layer 1)) and Layer 2 (Layer 2) for updating the local or global model, or a decentralized client. In the central client, the global model is maintained centrally in the first client and updated layer by layer, in order to then distribute it to the other clients at a later time, according to further embodiments. The maintenance / updating is performed based on information received about layers of the model from other clients, e.g., a second client and / or a third client. According to embodiments, the first client can also be a decentralized client that interacts with another decentralized client.Here, information regarding a layer is always exchanged between the decentralized clients so that layer-by-layer aggregation can be carried out.

[0014] Embodiments of the present invention are based on the creation of an aggregation method for federated learning (FL), in which a local model or a global model adopted as a local model is updated in decentralized clients, and the trained model is passed on layer by layer, either to a central client that maintains the global model or to other decentralized clients. Layer-by-layer exchange and layer-by-layer aggregation enable an efficient aggregation process to be implemented in limited systems or devices. This reduces communication overhead and storage space by computational overhead (computation time) while simultaneously increasing privacy.

[0015] According to embodiments, a second client of the network is created. This comprises a receiver, a processor, and a transmitter. The receiver is configured to receive a global model with at least two layers from a first client. The processor is configured to use the global model as a local model and to train it, e.g., using a processor or an artificial intelligence (ANN), in order to update the first and / or second layer of the local model. The transmitter is configured to transmit the (trained) first of the two layers to the first client or a third client in a first step. In a further step, the other trained layer(s) can then be transmitted (layer by layer).

[0016] The main advantages are the reduction of communication costs, which results in the following benefits: - Reduction of calculation time or calculation duration - Greater privacy - Reduction of energy costs - No loss of old knowledge or previous knowledge.

[0017] In the above and the following embodiments, transmission is always referred to in the sense of sending and receiving layers. This means that information regarding a layer of a model, i.e., a part of the model, is exchanged from a second client to a first client, or generally between two clients. After a layer has been exchanged, the received layers can then be aggregated into an updated model in the receiving client. For example, the updated (local or global) model is revised so that the received layer is aggregated with the existing layer or with other received layers. The purpose of this is to revise the model based on the layers trained in the second client.

[0018] According to embodiments, the processor of the first client is configured to update the local or global model based on the received second of the at least two layers in a second step. This results in the layer-by-layer aggregation of model data.

[0019] According to a further embodiment, the receiver is configured to receive a first of at least two layers of third model data from a third client and a second of the at least two layers of the third model data. This results in the layer-by-layer aggregation of model data from at least two clients. Consequently, the processor trains (or further develops) the local or global model based on the first and / or second layer of the third model data. Alternatively, the further development / training of the local and / or global model takes place based on the first and / or second layer of the third model data, taking into account the first and / or second layer of the second model data. Furthermore, the first and second layers of first model data can also be taken into account. In other words, after aggregation, the global model becomes the local model with which the training takes place.The first model data can represent either the initially existing local or global model. Alternatively, it is also conceivable that the first model data has already been or is being updated based on a training run in the first client. This is especially the case if the first client is a decentralized client.

[0020] According to embodiments, the first / central client can perform training in the same way, e.g. using its own data or training of a local model (derived from the global one) and / or using the gradually aggregated data.

[0021] Advantages of the main embodiments and the overall concept are explained below. Regarding the layers, it should be noted that the information according to the embodiments essentially relates to weight and bias. Gradients are transferred and / or used additionally or alternatively. This means that the first and / or second layer of the (first), second (third) model data has a weight and / or a bias.

[0022] According to embodiments, the updating comprises aggregating the first layers of the first and second or the first, second and third model data. Additionally or alternatively, the updating can comprise aggregating the second layers of the first and second model data or the first, second and third model data. By using more than two model data, an update is not only carried out on the basis of the training on one client, but also on the basis of the training on several clients. For this purpose, a weighting can be used according to further embodiments. In this respect, the aggregation comprises a weighting of the layers to be aggregated and / or an averaging of the layers to be aggregated (weighting with equal components). According to embodiments, the updated or to be updated local or global model w global determined based on the following formula: w global = w l0+ ... + w ln , where w l0 to w ln the weighted averages for the different layers l0 to l n represent.

[0023] Additionally or alternatively, updating the global / local model based on weighted averages can be l0 to w ln For this purpose, the formulas ←∑k=1kl0nklnl0wl0k wln←∑k=1klnnklnnlnwlnk, used. Here, I0 to I n the individual layers of the ML model, where w l0 w ln about the number of clients k l0 , k ln is calculated using the training data n.

[0024] The principles explained above can be used for both a decentralized and a centralized approach. Specific extensions for the centralized approach are explained below. The processor is configured to update the global model to obtain an updated global model. This global model is updated based on the layers transmitted by a second or further client. To initially transmit the global model to the clients, the first client may have a sender configured to initially transmit the global model to the clients, e.g., the second or third client. To then transmit the updated model to the clients, e.g., the second or third client, the sender may be used, according to further embodiments, to transmit the updated global model (after updating / updating several layers).

[0025] In the centralized case, for example, the models are transmitted to the clients that had previously sent their local models to the central location. In this case, according to the exemplary embodiments, the first client is a central unit configured to perform the aggregation and distribute the updated model. Furthermore, the first client can also train the local model on its local data, which is also used during the aggregation.

[0026] According to further embodiments, the first client can also represent a decentralized unit. In this case, the processor is then configured to use an initially obtained global model as a local model and to train this in order to update the first and / or second layer of the local model. According to embodiments, the first and second layers can be updated based on a received first and / or second layer of the second model data from a second client. Additionally or alternatively, the updating can be based on an internally determined first and / or second layer of so-called first model data from the first client itself, namely by training during operation. The result in both cases is the receipt of an updated local model.

[0027] According to embodiments, the client is configured to share layers of the updated model, i.e., an updated first and / or second layer of the local model. For this purpose, according to embodiments, it has a transmitter configured to forward the updated first and / or the updated second layer of the local model to another decentralized client, such as the second client or a third client. According to embodiments, this third client can also be a decentralized client, i.e., one that is essentially identical in structure to the first client.

[0028] According to embodiments, the selection of the first and / or second layer (= level of the model) for sharing is based on the random principle. This means that, according to embodiments, the first layer is randomly selected first or the second layer is randomly selected then. According to embodiments, the model can of course also have a third, fourth, etc. layer. According to embodiments, these are of course then also updated and shared in a similar way to the first and second layer. According to embodiments, the layer that was not forwarded in the previous step is forwarded. This means that the forwarding step is repeated accordingly. Each layer can be updated and forwarded multiple times. As an alternative to random transmission, a rule-based method (e.g. Kullback Leibler divergence or L1 loss) can also be used.

[0029] Another embodiment creates a second client of a network. This client has a receiver, a processor, and a transmitter and thus represents the client that interacts either with the first client in the decentralized configuration or with the first client in the centralized configuration. The second client is configured to transmit the first of the at least two layers to a first client and / or the third client in a first step. As above, the selection is based on the random principle. According to embodiments, this step can also be repeated.

[0030] Further embodiments create a system comprising a first client (as the central client) and a second client. According to embodiments, the system can also include a third or additional client. According to a further embodiment, a system comprising two (three, four, etc. / at least two, three, etc.) first (decentralized) clients is created.

[0031] Another embodiment relates to a method with the following steps: Receiving a first of at least two layers of model data from a second client and a second of the at least two layers of second model data from the second client; modeling and / or updating a local or global model, wherein the local or global model is updated in a first step based on the received first of the at least two layers.

[0032] According to embodiments, the method may include the optional step of training the global (initially received) or local model to update the first and / or second layer. According to embodiments, the updated first and / or second layer of the local model may be forwarded to another decentralized client, the second client, or the third client.

[0033] Another method provides a method comprising the following steps: receiving a global model having at least two layers from a first client; using and training the global model as a local model to update the first and / or second layer of the local model; transmitting the first of the two layers to the first client or a third client in a first step.

[0034] The process can of course also be computer-implemented.

[0035] Embodiments of the present invention are explained with reference to the accompanying drawings, in which: Fig. 1 a schematic representation of a client according to a basic embodiment; Fig. 2 a schematic representation of different layers of a deep neural network (including shallow layer and deep layer) to illustrate layers divided in embodiments; Fig. 3a,b representations of synchronous model update strategy (3a) and asynchronous model update strategy (3b) according to conventional approaches to illustrate the differences to the approach according to embodiments; Fig. 4 a schematic diagram illustrating a layer-wise aggregation with a central topology according to embodiments; and Fig. 5 a schematic representation of the layer-wise aggregation without a central unit according to further embodiments.

[0036] Before the following embodiments of the present invention and the accompanying drawings are explained, it should be noted that similar elements and structures are provided with the same reference numerals so that the description is applicable to one another.

[0037] Fig.Figure 1 shows a client 10, such as a first client belonging to a network. Federated learning (FL) machine learning approaches can be used in the network. For this purpose, the first client 10, or possibly all clients, has an ANN or AI algorithm. Data regarding the model M to be trained is exchanged. Each client can be a client in any network environment or cloud system, such as an industrial plant or medical technology. An example of an ML model for training would be the modeling of a drilling process in an industrial plant, for example, with the objective of learning an efficient rotation speed (see below). An alternative application example is from medical technology, particularly in analysis, e.g., motion data, sleep analysis, or the examination of tumors in CT or MRI images (see below).These examples are only exemplary. The main difference lies in the model or rather the model content, not in the structure of the models.

[0038] Typically, ML models are multi-layered, meaning they have at least two levels (hereinafter referred to as layers) or even several. A model with six levels, namely shallow layers and deep layers, is Fig. 2 shown.

[0039] According to embodiments, the client that optimizes the model M comprises at least one receiver 12, a processor 14 for processing the model M and an optional transmitter 16.

[0040] Processor 14 trains and / or aggregates (generally maintains) the (initial) local or global model M and updates it. In detail: To update the global model M of client 10, the model M is trained externally on another client (not shown), i.e., externally, based on its own, e.g., locally generated data, and parts of the model are sent back to client 10, namely layer by layer. This can be done on one or more external clients. Furthermore, a local model can / is also trained locally on client 10 (this is done in both the centralized and decentralized topologies).

[0041] The receiver 12 of the client 10 receives the externally trained first and / or second layers (e.g., the first layer in a first step and the second layer in a different second step). Using these layer-by-layer received (externally trained) layers, the model M is updated and aggregated by the processor 14 to obtain a current model M'. The updated model M' is thus updated layer by layer. Using the optional transmitter 16, this model M' after being updated, or even the initial model M, can be distributed to the external clients so that they can then train or further train the model M or M'. The model is further developed (maintained) so that M'', M''', ..., M'n is subsequently created.

[0042] According to embodiments, the first client 10 can be a central client that receives layer-by-layer information regarding the trained model from one or more second clients (not shown), or the trained model is reflected back layer-by-layer, in order to centrally update the model M to obtain the model M'. According to a further embodiment, the client 10 can be a decentralized client that interacts with other similar decentralized clients 10 and thus exchanges layer-by-layer information regarding the model M after training and then locally updates the model M in each case to obtain a locally updated model M'.The advantage of layer-by-layer updating, i.e. that only individual layers of the model are replaced in individual steps, lies in the reduced resource requirements for the communication between the individual trained or updated layers.

[0043] Fig.Figure 2 illustrates a model M with six layers L1-L6. Layers L1, L2, and L3 belong to the so-called shallow layers, while layers L4, L5, and L6 are deep layers. This model M can, for example, be distributed by client 10 to other clients (second clients), who then apply and train model M as a local model, namely based on their own data. A wide variety of approaches can be used for training, which differ from use case to use case. Typically, one or more layers L1-L6 are trained and then updated. These layers L1-L6 can then be reflected back in updated form to the central unit 10 or, in a decentralized case, to another unit.

[0044] On the left side, the shallow layers L1-L3 are shown, representing the layers closest to the input layer. The deep layers L4-L6 are shown on the right side, representing the layers closer to the output layer.

[0045] The properties of shallow L1-L3 and deep layers L4-L6 indicate that the parameters of shallow layers L1-L3 are more crucial for model performance. Accordingly, in the algorithm proposed by Chen et al., the parameters of shallow layers L1-L3 are updated / communicated more frequently than those of deep layers L4-L6. Schematically, with five clients, the comparison of a typical synchronous update strategy and the proposed asynchronous update strategy, in which shallow layers L1-L3 are updated more frequently, is shown in Fig. 3. The advantage is that all layers (L1-L6) are sent and aggregated in all n communication rounds.

[0046] Based on Fig. 3a and Fig. 3b, state-of-the-art methods for updating are now presented, namely a synchronous model (cf. Fig. 3a) and an asynchronous model (cf. Fig. 3b), specifically with the background that the update process is examined in more detail. The update process essentially comprises three steps: initializing an ML model, training the model, and aggregating the trained model, e.g., centrally. These terms, which are also used in embodiments of the invention, are now explained below using conventional approaches.

[0047] Federated Averaging: The FL approach introduced by Google in 2017 includes the Federated Averaging Algorithm (FedAvg) and is one of the most important aggregation algorithms in FL. It is frequently used as a reference in publications, and the basic principle developed at that time still forms the basis for many newly developed methods. The aggregation step is mathematically determined by calculating the weighted average of the ML weights of the new round w t+1 described as follows: wt+1←∑k=1Knknwt+1k

[0048] The weights are determined based on the amount of locally available data n k in relation to the total amount of data n, whereby participants with a large amount of data have a greater influence on the results.

[0049] The presented approach encompasses the entire FL process. The proposed architecture provides a central FL topology. The process can be described in the following steps according to the algorithm: 1. Initialization: The server initializes the ML model 2. Participant selection: From the total pool of potential participants, a fraction C is selected to train with its local data in this round. The larger the C chosen, the more computationally intensive the subsequent aggregation step is. The current global model is shared with the selected participants. 3. Local training: Each participating client trains the model using its local data. Any number of local epochs E is possible. This specifies how often the entire dataset is processed in the respective training round. Furthermore, training usually takes place in batches of size B, where the size of a batch specifies the proportion of the entire dataset used for training in each training round. An epoch is complete when all data in the dataset has been used. Additional hyperparameters arise from the respective optimization method. For example, the learning rate n, which specifies the step size during optimization, is a typical hyperparameter when using SGD. 4. Aggregation: After local training is complete, the local models are sent to the central unit. In this step, the new global model is determined, which results from the weighted average of the local models. The weighting factor is the number of locally available training examples n. k divided by the total number of training examples n. The process then begins again with the newly calculated global model.

[0050] This approach was shown to achieve high model quality for various network architectures with a small number of communication rounds.

[0051] FedProx: The aggregation method FedProx was inspired by FedAvg and uses the same approach. A comparison between the two algorithms can be seen in the table below. It is noticeable that the algorithms are similar except for one step. For example, in FedProx, the server also randomly selects devices as clients to participate in the FL process and sends them the global model. Training then takes place on each client using the locally available data. With FedAvg, training typically occurs with stochastic gradient descent (SGD) – the classic backpropagation – and a previously specified loss function F. k (w) (loss function), e.g., the cross-entropy. FedProx also uses SGD and an F k(w) is used to determine new trainable parameters (weights and bias). In FedProx, an additional factor is added to the value of the loss function after each forward pass. This additional factor is a static value µ, which is divided by two and multiplied by the residual sum of squares. The residual sum of squares is calculated from the weights of the model of the current epoch w t and determined with the weights of the global model w. loss=Fk(w)+μ2‖w−wt‖2

[0052] A comparison between FedAvg and FedProx. The FedAvg algorithm is shown on the left, and the FedProx algorithm is shown on the right. It's noticeable that the algorithms are identical except for the generation of the local weights. FedProx uses an extension of the loss function.

[0053] After training is complete, the local model is sent back to the server. The server aggregates the global model in the same way as the FedAvg algorithm.

[0054] In Fig. 3a illustrates the synchronous update strategy, while an asynchronous update strategy is shown in Fig. 3b. With the synchronous update strategy, the shallow and deep layers are communicated between the central unit and the clients in all rounds, meaning that the entire model is shared. With the asynchronous update strategy, the shallow layers are communicated between the central unit and all other clients in all rounds. Every n rounds, in this case, two out of six rounds, the entire model is communicated. This allows the communication costs between server and client to be reduced by two-thirds of the parameters of the deep learning layer.

[0055] FedAvg achieves high model accuracies for many network architectures. However, many parameters must be transmitted as long as a low number of epochs is used during training. Increasing the number of epochs may result in less powerful devices requiring longer training and thus preventing them from continuously participating in the FL process. The FedProx algorithm overcomes these limitations by influencing the training. While communication costs have been reduced, the computational effort is significantly increased compared to FedAvg. This is due, among other things, to the calculation of the residual sum of squares, which determines the difference between all weights of the local model and the global model.The asynchronous model update strategy reduces communication costs by splitting the ML model into two layers (shallow layer and deep layer), with the shallow layer being transmitted every FL round and the entire ML model only every n rounds. While this minimizes communication costs, the entire ML model is still transmitted every n rounds, which poses a privacy risk and means that communication costs are not reduced in these FL rounds. In addition to the algorithms presented here, there are numerous other aggregation algorithms that have been developed to increase privacy, model accuracy, or reduce communication costs. For example, there are algorithms that calculate the difference between the weights or gradients of the local and global model and only transmit those parameters whose difference is greater than a threshold.However, there is still no aggregation method designed for low-performance devices.

[0056] Therefore, an aggregation method is needed that both minimizes communication costs and increases privacy through privacy-by-design. The concept according to embodiments of the present invention goes further than the asynchronous approach by addressing the need for aggregation methods for low-performance devices. It is based on the idea of ​​layered aggregation methods, called "layered aggregation."

[0057] In contrast to FedAvg, in which the entire ML model is communicated from each client, one layer, consisting of the weights and biases, is communicated from each client. This method has parallels to the asynchronous model update strategy, but differs in that neither more than one layer nor the entire ML model is sent. The new concept does not change the general process of FL. First, the global model is initialized and shared with the individual participants. Also in line with the classic FL approach, training takes place using the locally available data on each participating client, which calculates a local model. The choice of local epochs, batch size, and optimization method are flexible. After the local models have been trained, they are not communicated in full to the central unit, unlike the classic approach.Instead, each client only shares the weights and biases of one layer. The layers shared by each client are random and change in each communication round.

[0058] A concrete example of a network is described with reference to Fig. 4 explained.

[0059] Fig.Figure 4 shows a layer-by-layer aggregation method with a central topology. Here, the central unit 10 is provided, which exchanges model data, i.e., at least individual model layers, with three clients 11a, 11b, and 11c. The central unit 10 initially communicates the global model M to the clients 11a, 11b, and 11c. These clients train the global model M as a local model to then obtain an updated local model M'A, M'B, and M'C. The training is based on local data. In detail: The model M, or the locally trained models M'A, M'B, and M'C, can have three hidden layers (shown only as an example and can certainly contain multiple layers). The ANN shown here as an example can comprise one, two, three, or even more, such as ten clients.

[0060] The global model is shared by the central unit with the participating clients. The clients train a new model based on the global model and the locally available data. Each client then shares a layer with the central unit. The central unit aggregates the local models into a new global model, which is then shared with the participants. Note that the central unit also trains a local model, which is included in the aggregation process.

[0061] Here, the first client 11a could communicate the weights and biases of the third layer L3, and the second client 11b those of the second layer L2. The other clients would also communicate individual layers. It may be advantageous to ensure that all layers are sent by at least one client (this would be beneficial for the overall process and for performance). However, not all layers need to be transmitted. If, for example, a layer is missing, the layer of the local model M of the central unit is used for aggregation). In the central unit, the individual layers of the local models are aggregated into a new global model, which is then shared with the clients again in the following step and initializes another round.Since the number of clients in most cases exceeds the number of layers in the network, and therefore the same layers are communicated by multiple clients, the FedAvg algorithm calculates the average of the weights and biases of the individual layers. The procedure is shown schematically with three devices and a central unit in . Fig. 4. Mathematically, the procedure for the weights is described as follows. wl0←∑k=1Kl0nklnl0wl0k wln←∑k=1Klnnklnnlnwlnk wglobal=wl0+⋯wln

[0062] The new global model w global consists of the weighted averages of the weights and biases of the individual layers I0 to I n The weighted averages of the individual layers are calculated using the FedAvg algorithm. However, this algorithm is only used if more than one client has sent the same layer. For example, for w l0the weighted sum over the number of clients K l0 The weighting factor is determined from the number of training data of the client n kl0 and the total number of all training data n l0 of all K l0 The weighting factor determines the weight of each client wl0k multiplied, resulting in the weighted mean w l0 This is done analogously for all other layers of the model.

[0063] In the above example from Fig.4, it is assumed that there is a central unit 10 and clients 11a, 11b, 11c, etc. The central unit 10 is the unit that aggregates the received layers L1, L2, and L3. For example, either layer L1 from client 11c is combined, i.e., aggregated, with layer 2 from client 11a and layer 3 from client 11b. Alternatively, several identical layers L1 received from different clients, e.g., 11a or 11c, can also be aggregated together. This is done, as explained, via averaging and weighting. Similarly, averaging and weighting can take into account the fact that information about layers L1, L2, and L3 already exists in advance, which is then modified according to the new layers L1, L2, and L3.Here, there is a division between clients 11a, 11b, and 11c, which train the local model and feed the trained layers back to the central client 10. Central client 10 receives the trained layers L1-L3 and updates the global model based on them. The global model M can then be distributed back to clients 11a-11c as an updated model M'. Averaging is one of many possible examples and can be implemented as a FedAvg algorithm. Other possible alternatives include FedMA, FedProx, Fed-Nova, FedAdapt, etc., as well as simple statistical algorithms such as the median.

[0064] At this point, it should be noted that, according to the simplest embodiment, the central client 10 has at least one receiver for receiving the layers L1-L3 and a processor for aggregation. This processor also has a transmitter so that it can initially share the model M or distribute the updated model to the clients accordingly.

[0065] The second clients 11a, 11b, and 11c have at least one processor 14 to update the local model M' (corresponding to the initially received model M during initial commissioning or the previously stored model M' during operation) based on the data D. Furthermore, each client 11a, 11b, and 11c has a transmitter to share the individual layers L1, L2, and L3 layer by layer, namely with the central unit 10. According to embodiments, each client 11a, 11b, and 11c can also have a receiver to receive, for example, the initial model M or the updated model M' (updated / aggregated in the central unit 10).

[0066] According to a further embodiment, it would also be conceivable that there is not one central client 10, but rather decentralized clients 10' or 10'a, 10'b, and 10'c, which, based on a global model, have a local model and train this model locally. The trained layers are then exchanged with the other decentralized clients. This embodiment will be described below with reference to Fig. 5 explained.

[0067] Fig.Figure 5 shows three decentralized clients 10'a, 10'b, and 10'c. Each includes a processor 14 and communication means (not shown) for sending and receiving. The local model M' is maintained by the processor 14 based on the local data D. The individual layers L1, L2, and L3 are exchanged between the decentralized clients 10'a, 10'b, and 10'c using the communication means. For example, client 10'a sends layer L1 to 10'b and 10'c, while 10'c sends L2 to 10'b and 10'a. 10'b sends L3 to 10'a and 10'b. The assignment of which client trains and shares which layer is purely random or rule-based. The assignment can also vary from step to step, meaning that each client can maintain and share different layers at different times. Each layer 10'a, 10'b and 10'c respectively.Its processor 14 is configured to train the local model M' based on the data D and to update it based on the received layers L1, L2, and L3 by means of aggregation. The functionality of the system is described below. Fig. 5 explained.

[0068] The clients train a model using the locally available data, which is then shared with the other participating clients. Each participant shares a layer of the model with the other participants. From the individual local models M', each participant aggregates a new "global" model M, which is then trained again.

[0069] In contrast to the centralized approach, the aggregation step is not handled by a central unit, but takes place on each of the end devices. Furthermore, the step of communicating the global model M after aggregation is omitted, since each client 10'a, 10'b, 10'c receives the local models M'. The initialization of the models also changes. Initially, the global model M is initialized locally on each client 10'a, 10'b, 10'c. If a client joins the process later, it requests and receives the current model from one of the clients already present in the network.

[0070] Although the concept is initially designed for feed-forward or fully connected neural networks, it can also be applied to other ML methods such as convolutional neural networks (CNNs). Furthermore, several optimizations of the method are planned. For example, not all weights of a layer will be transferred, but only those that have undergone significant changes during training, resulting in a difference that exceeds a threshold.

[0071] Regarding the above-mentioned embodiments, in particular the detailed connection with Fig. 4 as well as the detailed embodiment described in connection with Fig. 5 described embodiment, it should be noted that the following main areas of application are possible.

[0072] The main application area is the distribution of data in the context of FL. Since the method is designed for resource-constrained systems, but not limited, it can be used in all possible FL areas, from microcontrollers to cloud systems. Furthermore, FL is typically used where the GDPR applies or where data needs to be protected for other reasons, such as protecting a company's intellectual property. For example, the method can be used in the field of industrial plants or medical technology.

[0073] In industrial plants, for example, the process of drilling tools can be learned from ML models. The use of ML ensures that an optimal and efficient rotation speed is selected, allowing the drill to drill with minimal damage to the surface and material, thus ensuring high-quality drilling. Furthermore, a dynamic rotation speed ensures that the drill does not break during the drilling process. Furthermore, the pressure or torque required to drill the hole without exposing the drill to excessive wear can be learned. FL can be used to expand the process to multiple drills. FL can help ensure that different drills (e.g., different threads or pitches) and different materials (e.g., wood, aluminum, iron) can be learned across multiple machines.In addition, FL and the presented method ensure that no sensitive data from the drilling process can be viewed by third parties and that the process is as energy-efficient as possible.

[0074] Medical technology products and applications are inherently subject to high data protection requirements, which is why it is often difficult to train a generic ML model when developing ML applications. Therefore, many FL research projects and FL applications are located in the medical technology sector. These developments range from the analysis of motion sensor data (Human Activity Recognition, HAR), sleep analysis, and air quality analysis in hospitals to the training of image segmentation algorithms for segmenting tumors in CT or MRI images. In all developments, FL can be used within a hospital, allowing the measurement instruments used on a floor to learn together and thus develop a generic ML model.

[0075] These main areas of application are of course also limited to the general design example Fig.1 is applicable. Of course, other applications would also be conceivable based on the exemplary embodiments. All exemplary embodiments have in common that only individual layers are exchanged per communication process. A second client 11 or a decentralized client sends only a single layer per communication process, while a central client 10 or a decentralized client 10 receives only individual layers.

[0076] As already explained above, a layer can contain weights and biases, and according to the implementation examples, other contents would of course also be conceivable, such as gradients or other parameters of the ML models with and without weighting.

[0077] In addition, all weights of a layer are currently sent, so a workaround would be to send only a subset of the weights of a layer.

[0078] Based on this, it is stated that, according to the exemplary embodiments, only subsets of the weights can be contained in the individual layers.

[0079] Furthermore, it could be used with another ML method that has not yet been investigated. These could be, for example, recurrent neural networks (RNN), long-short-term memories (LSTM), graph neural networks, etc. Furthermore, the method could be transferred and used with another ML method. This could, for example, be data parallelism during training on GPUs.

[0080] This means that, depending on the embodiments, other ML models or ML methods can also be applied to the above embodiments.

[0081] Although some aspects have been described in the context of a device, it should be understood that these aspects also represent a description of the corresponding method, so that a block or component of a device can also be understood as a corresponding method step or as a feature of a method step. Analogously, aspects described in the context of or as a method step also represent a description of a corresponding block, detail, or feature of a corresponding device. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, some or more of the key method steps may be performed by such an apparatus.

[0082] Depending on specific implementation requirements, embodiments of the invention may be implemented in hardware or software. The implementation may be performed using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray Disc, a CD, a ROM, a PROM, an EPROM, an EEPROM, or a FLASH memory, a hard disk, or other magnetic or optical storage device storing electronically readable control signals that can interact or cooperate with a programmable computer system to perform the respective method. Therefore, the digital storage medium may be computer-readable.

[0083] Some embodiments according to the invention thus comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system such that one of the methods described herein is carried out.

[0084] In general, embodiments of the present invention may be implemented as a computer program product having a program code, wherein the program code is effective to perform one of the methods when the computer program product is run on a computer.

[0085] The program code can, for example, also be stored on a machine-readable medium.

[0086] Other embodiments include the computer program for performing one of the methods described herein, wherein the computer program is stored on a machine-readable medium. In other words, one embodiment of the method according to the invention is thus a computer program that has program code for performing one of the methods described herein when the computer program is executed on a computer.

[0087] A further embodiment of the method according to the invention is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded.

[0088] A further embodiment of the method according to the invention is thus a data stream or a sequence of signals that represents the computer program for carrying out one of the methods described herein. The data stream or the sequence of signals can be configured, for example, to be transferred via a data communication connection, for example, via the Internet.

[0089] A further embodiment comprises a processing device, for example a computer or a programmable logic device, which is configured or adapted to carry out one of the methods described herein.

[0090] A further embodiment comprises a computer on which the computer program for performing one of the methods described herein is installed.

[0091] A further embodiment according to the invention comprises a device or system designed to transmit a computer program for performing at least one of the methods described herein to a recipient. The transmission can be electronic or optical, for example. The recipient can be, for example, a computer, a mobile device, a storage device, or a similar device. The device or system can, for example, comprise a file server for transmitting the computer program to the recipient.

[0092] In some embodiments, a programmable logic device (e.g., a field-programmable gate array, an FPGA) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field-programmable gate array may interact with a microprocessor to perform any of the methods described herein. In general, in some embodiments, the methods are performed by any hardware device. This may be general-purpose hardware such as a computer processor (CPU) or method-specific hardware such as an ASIC.

[0093] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein.

Claims

[] Decentralized / Centralized Client with Aggregation Function [1] First client (10, 10'a, 10'b, 10'c) of a network with at least the first client (10, 10'a, 10'b, 10'c) and a second client (11a, 11b, 11c), having the following features: a receiver configured to receive a first (L1) of at least two layers of model data of a second client (11a, 11b, 11c) and a second of the at least two layers of second model data of the second client (11a, 11b, 11c); a processor configured to train and / or update a local or global model (M), wherein the processor updates the local or global model (M) based on the received first (L1) of the at least two layers in a first step; wherein the processor is designed to use and train an initially obtained global model (M) as a local model (M) in order to update the first (L1) and / or second layer (L2) of the local model (M); and wherein the updating of the local model (M) in the first (L1) and / or second layer is carried out on the basis of a received first (L1) and / or second layer (L2) of the second model data of the second client (11a, 11b, 11c), and / or on the basis of an internally determined first (L1) and / or second layer (L2) of first model data of the first client (10, 10'a, 10'b, 10'c) during training in operation in order to obtain an updated local model (M); wherein the first client (10, 10'a, 10'b, 10'c) further comprises a transmitter configured to forward an updated first (L1) and / or second layer (L2) of the local model (M) to another decentralized client, the second client (11a, 11b, 11c) or third client; where the forwarding is repeated for a layer that was not forwarded in the previous step(s). [2] First client (10, 10'a, 10'b, 10'c) according to claim 1, wherein the processor is configured to update the local or global model (M) based on the received second (L2) of the at least two layers in a second step (layer-by-layer aggregation of model data). [3] First client (10, 10'a, 10'b, 10'c) according to one of the preceding claims, wherein the receiver is configured to receive a first (L1) of at least two layers of third model data of a third client and / or a second (L2) of the at least two layers of the third model data (layer-by-layer aggregation of model data of at least two clients). [4] First client (10, 10'a, 10'b, 10'c) according to claim 3, wherein the updating of the local or global model (M) is carried out on the basis of the first (L1) and / or second layer (L2) of the third model data, or wherein the updating of the local or global model (M) is carried out on the basis of the first and / or second layer (L2) of the third model data taking into account the first (L1) and / or second layer (L2) of the second model data. [5] First client (10, 10'a, 10'b, 10'c) according to one of the preceding claims, wherein the first (L1) and / or the second layer (L2) of the second model data has a weight and / or a bias. [6] The first client (10, 10'a, 10'b, 10'c) according to any one of the preceding claims, wherein the updating comprises aggregating the first layers (L1) of the first and second or the first, second and third model data; and / or wherein the updating comprises aggregating the second layers (L2) of the first and second model data or the first, second and third model data. [7] First client (10, 10'a, 10'b, 10'c) according to claim 6, wherein the aggregating comprises a weighting of the layers to be aggregated and / or an averaging of the layers to be aggregated. [8] First client (10, 10'a, 10'b, 10'c) according to one of the preceding claims, wherein the updated local or global model (M) w global is determined based on the formula: wglobal=wl0+⋯wln, where w l0 to w ln represent the weighted averages for the different layers l0 to ln; and / or where the update of the local or global model (M) is based on weighted averages w l0 to w ln which can be calculated using the following formulas: wl0←∑k=1kl0nklnl0wl0k wln←∑k=1klnnklnnlnwlnk, where I0 to In represent the individual layers and w l0 to w ln about the number of clients k l0 , k ln is calculated using the training data n. Central [9] First client (10, 10'a, 10'b, 10'c) according to one of the preceding claims, wherein the processor is configured to update the global model (M) to obtain an updated global model (M) and wherein the first client (10, 10'a, 10'b, 10'c) comprises a transmitter configured to transmit the updated global model (M) to the second and / or a third client. [10] First client (10, 10'a, 10'b, 10'c) according to one of the preceding claims, wherein the first client (10, 10'a, 10'b, 10'c) has a transmitter configured to initialize the global model (M) and to transmit it to the second client (11a, 11b, 11c) or a third client. Decentralized [11] First client (10, 10'a, 10'b, 10'c) according to one of the preceding claims, wherein the selection of the first layer (L1) or the second layer (L2) for sharing is based on randomness or rules. Second / Third Client [12] Second client (11a, 11b, 11c) of a network with a first client (10, 10'a, 10'b, 10'c) and the second client (11a, 11b, 11c), having the following features: a receiver configured to receive a global model (M) having at least two layers from a first client (10, 10'a, 10'b, 10'c); a processor configured to use and train the global model (M) as a local model (M) to update the first (L1) and / or second layer (L2) of the local model (M); a transmitter configured to transmit the first (L1) of the two layers to the first client or a third client in a first step; wherein the processor is designed to use and train an initially obtained global model (M) as a local model (M) in order to update the first (L1) and / or second layer (L2) of the local model (M); and wherein the updating of the local model (M) in the first (L1) and / or second layer is carried out on the basis of a received first (L1) and / or second layer (L2) of the second model data of the second client (11a, 11b, 11c), and / or on the basis of an internally determined first (L1) and / or second layer (L2) of first model data of the first client (10, 10'a, 10'b, 10'c) during training in operation in order to obtain an updated local model (M); wherein the first client (10, 10'a, 10'b, 10'c) further comprises a transmitter configured to forward an updated first (L1) and / or second layer (L2) of the local model (M) to another decentralized client, the second client (11a, 11b, 11c) or third client; where the forwarding is repeated for a layer that was not forwarded in the previous step(s). [13] Second client (11a, 11b, 11c) according to claim 12, wherein the transmitter is configured to transmit the second of the two layers (L2) to the first (10, 10'a, 10'b, 10'c) or the third client in a second step. [14] Second client (11a, 11b, 11c) according to claim 12 or 13, wherein the selection of the first layer (L1) or the second layer (L2) for sharing is based on the random principle. [15] A second client (11a, 11b, 11c) according to claim 12, 13 or 14, wherein the forwarding is repeated for a layer that was not forwarded in the previous step(s). [16] System comprising a first client (10, 10'a, 10'b, 10'c) according to one of claims 1 to 12 and a second client (11a, 11b, 11c) according to claim 13, 14 or 15. [17] System comprising a first client (10, 10'a, 10'b, 10'c) according to one of claims 1 to 10 or according to claims 11 to 14, and a further first client (10, 10'a, 10'b, 10'c) according to one of claims 1 to 10. Proceedings [18] A method for operating a first client (10, 10'a, 10'b, 10'c) of a network comprising the first client (10, 10'a, 10'b, 10'c) and a second client (11a, 11b, 11c), comprising the following steps: Receiving a first (L1) of at least two layers of model data of a second client (11a, 11b, 11c) and a second (L2) of the at least two layers of second model data of the second client (11a, 11b, 11c); Modeling and / or updating a local or global model (M), wherein the local or global model (M) is updated in a first step based on the received first (L1) of the at least two layers; Using and training an initially obtained global model (M) as a local model (M) to update the first (L1) and / or second layer (L2) of the local model (M); and wherein the updating of the local model (M) in the first (L1) and / or second layer is carried out on the basis of a received first (L1) and / or second layer (L2) of the second model data of the second client (11a, 11b, 11c), and / or on the basis of an internally determined first (L1) and / or second layer (L2) of first model data of the first client (10, 10'a, 10'b, 10'c) during training in operation in order to obtain an updated local model (M); Forwarding an updated first (L1) and / or second layer (L2) of the local model (M) to another decentralized client, the second client (11a, 11b, 11c) or third client; where the forwarding is repeated for a layer that was not forwarded in the previous step(s). [19] The method of claim 18, further comprising the step of training the global or local model (M) to update the first (L1) and / or second layer (L2). [20] Method according to claim 19, further comprising the step of forwarding an updated first (L1) and / or second layer (L2) of the local model (M) to another decentralized client, the second client (11a, 11b, 11c) or third client. [21] Method for operating a second client (11a, 11b, 11c) of a neural network with a first client (10, 10'a, 10'b, 10'c) and the second client (11a, 11b, 11c), comprising the following steps: Receiving a global model (M) with at least two layers from a first client (10, 10'a, 10'b, 10'c); Using and training the global model (M) as a local model (M) to update the first (L1) and / or second layer (L2) of the local model (M); Transmitting the first (L1) of the two layers to the first client or a third client in a first step; Using and training an initially obtained global model (M) as a local model (M) to update the first (L1) and / or second layer (L2) of the local model (M); and wherein the updating of the local model (M) in the first (L1) and / or second layer is carried out on the basis of a received first (L1) and / or second layer (L2) of the second model data of the second client (11a, 11b, 11c), and / or on the basis of an internally determined first (L1) and / or second layer (L2) of first model data of the first client (10, 10'a, 10'b, 10'c) during training in operation in order to obtain an updated local model (M); Forwarding an updated first (L1) and / or second layer (L2) of the local model (M) to another decentralized client, the second client (11a, 11b, 11c) or third client; where the forwarding is repeated for a layer that was not forwarded in the previous step(s). [22] A computer program for carrying out a method according to any one of claims 18, 19, 20 or 21, when the method runs on a processor.

Images