Federal learning acceleration method, device and federal learning system based on dynamic local update

Abstract

The application discloses a federated learning acceleration method and device based on dynamic local update and a federated learning system. In one aspect, the method uses node-specific data distribution information to set a local update frequency in a local node based on a federated learning scenario, thereby accelerating model convergence, wherein the local update frequency of a client is calculated according to the relationship between a global gradient update direction and a local gradient update direction of the client. The application alleviates data heterogeneity by setting different local update frequencies, and realizes faster model convergence in federated learning without increasing communication costs.

Description

Technical Field

[0001] This invention belongs to the field of federated learning, and specifically relates to a method, apparatus and system for accelerating federated learning based on dynamic local updates. Background Technology

[0002] Federated learning, as an emerging machine learning paradigm, has garnered significant attention in recent years for addressing the challenges of distributed data and privacy protection. Traditional machine learning methods typically require centralized collection and integration of data, followed by model training on a central server. However, with the proliferation of mobile devices, sensor networks, and edge computing, there is growing awareness of the potential data privacy, security, and communication overhead issues associated with centralized approaches. In this context, federated learning offers a decentralized solution, allowing distributed model training across devices or nodes without transmitting raw data to a central server. Each participant (device or node) shares updates to its local model parameters, enabling collaborative learning while protecting data privacy. This decentralized learning approach not only helps reduce communication overhead but also adapts better to distributed environments, including mobile devices, edge computing nodes, and IoT devices.

[0003] Data heterogeneity is a key challenge in federated learning because real-world data often varies in distribution, type, and scale. Addressing this issue is crucial because effectively handling heterogeneity can improve model generalization performance, enabling it to adapt to diverse scenarios. Simultaneously, heterogeneity directly impacts collaborative training among participants, requiring effective updates and fusion of data with different features and distributions. Furthermore, considering data heterogeneity is essential for protecting privacy and security, ensuring that federated learning protects the privacy of the original data while sharing model parameters. Therefore, progress in researching and solving the data heterogeneity problem will make federated learning more reliable and feasible in practical applications. Previous work has addressed data heterogeneity in federated learning through local updates and global variance reduction techniques. Local updates allow each participant to train the model locally, sharing only model parameter updates, thus handling heterogeneous data. Global variance reduction utilizes global information to reduce differences between participant models and mitigate the impact of node drift. By effectively integrating information contributed by all parties, this method improves model consistency and stability, ensuring better handling of heterogeneous data in federated learning and achieving more powerful collaborative learning effects. However, using the same local update frequency across different nodes is not conducive to mitigating the negative impact of data heterogeneity from a data distribution perspective. Summary of the Invention

[0004] To address the problems in the background art, the purpose of this invention is to provide a federated learning acceleration method based on dynamic local updates. This method reduces the negative impact of data heterogeneity by setting different update frequencies for local nodes, thereby accelerating model convergence and overcoming the data heterogeneity challenge in communication-sensitive federated learning.

[0005] The present invention also provides corresponding client devices, server devices, and federated learning systems.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] Firstly, a method for accelerating federated learning based on dynamic local updates is provided, executed on a server, including the following steps:

[0008] The model parameters and local update frequency are sent to the client, which then performs local model updates. In the first iteration, all client nodes have the same model parameters and local update frequency. In subsequent iterations, the client's local update frequency is calculated based on the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0009]

[0010] Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0011] Receive the updated model parameters from the client, aggregate the model parameters from all clients, determine the new global model, and return to the previous step to distribute it.

[0012] Secondly, a computing device is provided for use as a federated learning server, comprising:

[0013] The model and parameter delivery module is used to deliver model parameters and local update frequencies to the client, which then performs local model updates. In the first iteration, the model parameters and local update frequencies are the same for all client nodes. In subsequent iterations, the client's local update frequency is calculated based on the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0014]

[0015] Where k jLet k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0016] The model aggregation and update module is used to receive the updated model parameters from the client, aggregate the model parameters from all clients, determine the new global model, and return to the previous step to distribute it.

[0017] Thirdly, a computer device is provided, comprising:

[0018] One or more processors;

[0019] Memory; and

[0020] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the federated learning acceleration method based on dynamic local updates as described in the first aspect.

[0021] Fourthly, a method for accelerating federated learning based on dynamic local updates is provided, executed on the client side, including the following steps:

[0022] The system receives the local update frequency and model parameters from the server. Based on the model parameters, it updates the local model using the local dataset. The number of update operations is determined by the local update frequency, which is calculated by the server considering the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0023]

[0024] Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0025] After the local model is updated, the obtained local model parameters are sent back to the server.

[0026] Fifthly, a computing device is provided for use as a federated learning client, comprising:

[0027] The local model update module receives the local update frequency and model parameters from the server, and updates the local model using the local dataset based on the model parameters. The number of update operations is determined by the local update frequency, which is calculated by the server considering the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0028]

[0029] Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0030] The parameter feedback module is used to send the obtained local model parameters back to the server after the local model update is completed.

[0031] Sixthly, a computer device is provided, comprising:

[0032] One or more processors;

[0033] Memory; and

[0034] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the federated learning acceleration method based on dynamic local updates as described in the fourth aspect.

[0035] In a seventh aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the federated learning acceleration method based on dynamic local updates as described in the first or fourth aspect.

[0036] Eighthly, a federated learning system is provided, characterized in that it includes multiple clients and a server, the server being configured to execute the federated learning acceleration method based on dynamic local updates as described in the first aspect, the clients being configured to execute the federated learning acceleration method based on dynamic local updates as described in the fourth aspect, or the server employing the computing device described in the second aspect, and the clients employing the computing device described in the fifth aspect.

[0037] Beneficial Effects: This invention investigates how to improve model training speed without increasing communication costs within the classic federated learning architecture. Specifically, it designs a federated learning acceleration method based on dynamic local updates under heterogeneous data conditions. By setting different local update frequencies for clients, it alleviates the data heterogeneity problem, enabling faster model convergence without increasing communication costs, thus overcoming the challenge of data heterogeneity. This makes federated learning more practical and greatly improves its broad applicability. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of a federated learning system architecture based on dynamic local updates under heterogeneous data.

[0039] Figure 2 This is a schematic diagram of data flow in a federated learning acceleration method based on dynamic local updates under heterogeneous data conditions. Detailed Implementation

[0040] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0041] Reference Figures 1-2 This embodiment provides a method for accelerating federated learning based on dynamic local updates under heterogeneous data conditions, and details it using a classic federated learning architecture as an example. Figure 1 As shown, the current federated learning system has one central server and three local computing nodes. The central server is also called the central server, or simply the server; the local computing nodes are also called client nodes, user nodes, local nodes, or simply nodes. Each node has different data characteristics and different computing power. Furthermore, the central server is interconnected with each node via a network, making communication sensitive. The federated learning system will utilize the data possessed by the local nodes to train a model with all data characteristics under the control, scheduling, and aggregation of the central server.

[0042] Each local node's tasks include local model updates, gradient calculation, and uploading the model update. Specifically, after receiving the global model parameters, the local node performs a model update using its local data. This may involve performing model inference, calculating the loss function, and calculating gradients on the local device. The local node calculates the gradients of the model parameters based on its local data; these gradients are typically calculated using backpropagation or other optimization algorithms. Afterward, the local node uploads the local model update to the central server. This process usually requires ensuring the security and effectiveness of communication while protecting data privacy.

[0043] The central server's tasks include model initialization, model distribution, and model aggregation. Specifically, the central server is responsible for initializing the global model, which typically involves selecting appropriate initial parameters based on the application scenario and model type. Once initialization is complete, the central server distributes the initial model parameters to all participating devices. After the local nodes have finished their work, the central server is responsible for collecting model updates from all participants. This may include performing operations such as denoising, aggregation, or filtering on the received model updates to ensure the stability and accuracy of the global model. The connection between the central server and the participating nodes is usually achieved through a network, which can be the Internet, a local area network, or other communication networks.

[0044] The method of this invention requires determining the system-wide information in a federated learning environment. This system-wide information includes the learning rate γ, the basic local update frequency k0, the initial model parameters ω, and the number of iterations N. (Refer to...) Figure 2 The specific implementation steps of the method are as follows:

[0045] (1) Based on the characteristics of federated learning and global system information, the central server designs and adjusts the initial model parameters ω and the basic local update frequency k0, and sends them to each local node. Typically, the initial model parameters are the same; for example, the weights and biases in the neural network are randomly initialized to small random values. The basic local update frequency is usually set to 5–10. This is done to ensure that all participating nodes start from the same point in the initial stage of federated learning, thus ensuring consistency in the early stages of model training. Since federated learning often faces challenges such as heterogeneity of participating nodes and uneven data distribution, the central server needs to formulate suitable initial model parameters and update frequencies based on global system information and considering the characteristics and conditions of each participating node. This may involve balancing factors such as the amount of data, computing power, and network connection quality of the participating nodes. Furthermore, since the basic local update frequency is usually a scalar, the communication cost incurred during communication is almost negligible.

[0046] In the first iteration, all local nodes have the same model and update frequency, which helps ensure the initial model convergence and stability. As model training progresses, local nodes will receive update frequencies from the server and perform the required number of local updates, thus achieving the goals of personalized model updates and model adaptation in federated learning. This phased adjustment and optimization process allows the federated learning system to fully utilize the local data of each participating node while taking into account the consistency of the global model, achieving continuous optimization and improvement of the overall model.

[0047] (2) Once a local node receives the local update frequency and model parameters from the central server, it begins its local model update process. Common update algorithms in federated learning include gradient descent and stochastic gradient descent (SGD). These algorithms optimize the model by calculating the gradient or update value of the model parameters and updating the local model parameters according to their direction and magnitude.

[0048] This update process is performed on the node's local dataset, which typically reflects the characteristics and changes in the node's environment. For example, in Figure 1 In the federated learning environment shown, different nodes may possess or generate different data features. Each node performs a certain number of local update operations based on the frequency of received local updates to ensure that the local model fully reflects the characteristics and trends of the node's local data.

[0049] The number of local updates is typically related to the frequency of received local updates. If a node receives updates frequently, it may perform more local update operations to synchronize the local model with the global model more quickly. Conversely, if the update frequency is low, the node may adopt a more cautious strategy to avoid over-updating and wasting computational resources. This flexible local update mechanism enables federated learning systems to operate under different network conditions and computational environments, maintaining model stability and performance in various scenarios.

[0050] After completing the local update, the node sends the obtained local model parameters back to the central server. This process typically involves data transmission and communication, so the node needs to ensure data integrity and security.

[0051] (3) After the server receives the updated model parameters from all nodes, it performs an average operation on all model parameters to obtain a new global model.

[0052] After determining the new global model, the server also needs to consider the relationship between the global update direction and the local update direction. This relationship is usually inversely proportional; that is, if the Euclidean distance between a node's local update and the global update direction is small, the local update frequency of that node in the next round may increase accordingly to more quickly align the local model with the global model. Conversely, if the Euclidean distance between a node's local update and the global update direction is large, the local update frequency of that node in the next round may decrease accordingly to avoid over-updating and model instability.

[0053] The specific formula for calculating the local update frequency is as follows:

[0054]

[0055] Where kj Let kj represent the local update frequency at node j, k0 represent the basic local update frequency, P represent the total number of local nodes, and ΔF represent the local update frequency at node j. j ΔF represents the gradient at node j after model training, and ΔF represents the global gradient after one round of server aggregation. Figure 1 Taking the architecture shown as an example: Assume that the Euclidean distance between node 1 and the global update direction is 1; the Euclidean distance between node 2 and the global update direction is 2; the Euclidean distance between node 3 and the global update direction is 3, and the basic local update frequency k0 is 3. Then, according to the formula for calculating the local update frequency, in the next iteration, the local update frequency of node 1 is calculated to be 6; the local update frequency of node 2 is calculated to be 3; and the local update frequency of node 3 is calculated to be 2.

[0056] This strategy of adjusting the update frequency based on the update direction helps maintain the stability and convergence of the global model. It enables the federated learning system to maintain balance across different nodes and dynamically adjust the model's update rate based on the characteristics and changes of local data. In this way, each node can effectively participate in the optimization and improvement of the global model based on its own contributions and characteristics, thereby achieving an overall performance improvement for the federated learning system.

[0057] Repeat steps (1) to (3) until the model converges.

[0058] Formally, the entire algorithm can be represented as a distributed optimization problem with P federated learning nodes:

[0059]

[0060] Here, P is defined as a group of P clients. F j F(x) represents the local expected loss function of client j. The entire training process of federated learning is a process of minimizing the objective function F(x). To establish an upper bound on the convergence rate, it is usually necessary to introduce some assumptions to provide a basis for problem analysis. These assumptions are widely used in machine learning research and are also common in the theoretical analysis of federated learning. These assumptions include:

[0061] Bounded expectation gradient variance assumption: For a given parameter w, the stochastic gradient... The difference between the squares of the expected value and the squares of the expected value is bounded, with its upper bound controlled by a constant M. This means that the variance of the stochastic gradient does not grow indefinitely but is limited, thus ensuring the stability and controllability of the gradient.

[0062] The Lipschitz continuity assumption states that the loss function is Lipschitz continuous, meaning there exists a constant L such that for any two points w1 and w2, the change in the loss function between these two points does not exceed L times the distance between w1 and w2.

[0063] Bounded variable assumption: It is assumed that the model parameters vary within a bounded set.

[0064] The unbiased estimation assumption of the stochastic gradient: The unbiased estimation assumption of the stochastic gradient states that, given a fixed parameter w, the stochastic gradient... It is an unbiased estimate of the true gradient of the parameter.

[0065] w α Let α be the global update in the algorithm, and Let be the t-th local update at node j. The (α+1)-th global average can be written as:

[0066]

[0067] Based on assumption 2 and the above expression, the following inequality can be derived:

[0068]

[0069] By analyzing the upper bounds of the two parts on the right side of the inequality separately, we can conclude that:

[0070]

[0071] In other words, the algorithm can theoretically guarantee convergence through a finite number of iterations.

[0072] Based on the above description of the federated learning process, in a first aspect, the present invention provides a method for accelerating federated learning based on dynamic local updates, executed on a server, comprising the following steps:

[0073] The model parameters and local update frequency are sent to the client, which then performs local model updates. In the first iteration, all client nodes have the same model parameters and local update frequency. In subsequent iterations, the client's local update frequency is calculated based on the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0074]

[0075] Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. jΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0076] Receive the updated model parameters from the client, aggregate the model parameters from all clients, determine the new global model, and return to the previous step to distribute it.

[0077] In a second aspect, a computing device is provided for use as a federated learning server, comprising:

[0078] The model and parameter delivery module is used to deliver model parameters and local update frequencies to the client, which then performs local model updates. In the first iteration, the model parameters and local update frequencies are the same for all client nodes. In subsequent iterations, the client's local update frequency is calculated based on the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0079]

[0080] Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0081] The model aggregation and update module is used to receive the updated model parameters from the client, aggregate the model parameters from all clients, determine the new global model, and return to the previous step to distribute it.

[0082] In a third aspect, a computer device is provided, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the federated learning acceleration method based on dynamic local updates as described in the first aspect.

[0083] In the fourth aspect, a federated learning acceleration method based on dynamic local updates is provided, which is executed on the client side and includes the following steps:

[0084] The system receives the local update frequency and model parameters from the server. Based on the model parameters, it updates the local model using the local dataset. The number of update operations is determined by the local update frequency, which is calculated by the server considering the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0085]

[0086] Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0087] After the local model is updated, the obtained local model parameters are sent back to the server.

[0088] In a fifth aspect, a computing device is provided for use as a federated learning client, comprising:

[0089] The local model update module receives the local update frequency and model parameters from the server, and updates the local model using the local dataset based on the model parameters. The number of update operations is determined by the local update frequency, which is calculated by the server considering the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows:

[0090]

[0091] Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration.

[0092] The parameter feedback module is used to send the obtained local model parameters back to the server after the local model update is completed.

[0093] In a sixth aspect, a computer device is provided, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the federated learning acceleration method based on dynamic local updates as described in the fourth aspect.

[0094] In a seventh aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the federated learning acceleration method based on dynamic local updates as described in the first or fourth aspect.

[0095] In an eighth aspect, a federated learning system is provided, characterized in that it includes multiple clients and a server, the server being configured to execute the federated learning acceleration method based on dynamic local updates as described in the first aspect, the clients being configured to execute the federated learning acceleration method based on dynamic local updates as described in the fourth aspect, or the server employing the computing device described in the second aspect, and the clients employing the computing device described in the fifth aspect.

[0096] This invention investigates how to improve model training speed without increasing communication costs within the classic federated learning architecture. Specifically, it designs a federated learning acceleration method based on dynamic local updates for heterogeneous data. By setting different local update frequencies, it alleviates the data heterogeneity problem, enabling faster model convergence without increasing communication costs, thus overcoming the challenge of data heterogeneity. This makes federated learning more practical and greatly improves its broad applicability.

Claims

1. A federated learning acceleration method based on dynamic local updates, characterized in that, The method is executed on a server and includes the following steps: The model parameters and local update frequency are sent to the client, which then performs local model updates. In the first iteration, all client nodes have the same model parameters and local update frequency. In subsequent iterations, the client's local update frequency is calculated based on the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows: Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration. Receive the updated model parameters from the client, aggregate the model parameters from all clients, determine the new global model, and return to the previous step to distribute it.

2. A computing device, characterized in that, The device serves as a federated learning server and includes: The model and parameter delivery module is used to deliver model parameters and local update frequencies to the client, which then performs local model updates. In the first iteration, the model parameters and local update frequencies are the same for all client nodes. In subsequent iterations, the client's local update frequency is calculated based on the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows: Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration. The model aggregation and update module receives the updated model parameters from the client, aggregates the model parameters from all clients, determines the new global model, and returns to the previous step to distribute it.

3. A computer device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein when the programs are executed by the processors, they implement the steps of the federated learning acceleration method based on dynamic local updates as described in claim 1.

4. A federated learning acceleration method based on dynamic local updates, characterized in that, The method is executed on the client side and includes the following steps: The system receives the local update frequency and model parameters from the server. Based on the model parameters, it updates the local model using the local dataset. The number of update operations is determined by the local update frequency, which is calculated by the server considering the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows: Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration. After the local model is updated, the obtained local model parameters are sent back to the server.

5. A computing device, characterized in that, The device is used as a federated learning client and includes: The local model update module receives the local update frequency and model parameters from the server, and updates the local model using the local dataset based on the model parameters. The number of update operations is determined by the local update frequency, which is calculated by the server considering the relationship between the global gradient update direction and the client's local gradient update direction. The calculation formula is as follows: Where k j Let k0 represent the local update frequency on client node j, k0 represent the basic local update frequency, P represent the total number of client nodes, and ΔF represent the local update frequency on client node j. j ΔF represents the gradient of the model after training on client node j, and ΔF represents the global gradient after one iteration. The parameter feedback module is used to send the obtained local model parameters back to the server after the local model update is completed.

6. A computer device, characterized in that, include: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the programs, when executed by the processors, implement the steps of the federated learning acceleration method based on dynamic local updates as described in claim 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the federated learning acceleration method based on dynamic local updates as described in claim 1 or claim 4.

8. A federated learning system, characterized in that, The system includes multiple clients and a server, wherein the server is configured to execute the federated learning acceleration method based on dynamic local updates as described in claim 1, the clients are configured to execute the federated learning acceleration method based on dynamic local updates as described in claim 4, or the server uses the computing device described in claim 2, and the clients use the computing device described in claim 5.

Images