A method, device, server, and medium for accelerating federated learning

Abstract

This invention discloses a federated learning acceleration method, apparatus, server, and medium. The method includes sending a global model to be trained to a client with an established communication connection. The client trains the global model based on a local dataset to obtain a trained local model. The method receives parameters of the local model uploaded by the client and compares the parameters to be optimized in the global model with the parameters of the local model to determine gradient information for model optimization. Based on the gradient information, an optimization path is determined, and the global model is optimized according to at least two optimization paths and their corresponding weight information. The currently optimized global model is used as a new global model to be trained, and the method returns to send the new global model to be trained to the client with the established communication connection until the number of communications with the client reaches a set threshold. By adopting the above technical solution, the training efficiency of the model in the federated learning process is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of machine learning technology, and more specifically, to a federated learning acceleration method, apparatus, server, and medium. Background Technology

[0002] Federated Learning (FL) is a communication-constrained distributed learning paradigm designed to collaboratively train widely applicable models among multiple participants without sharing the original data. Optimization of FL includes not only achieving optimal model performance at convergence (e.g., test accuracy for image classification, recall for text prediction), but also achieving model performance with minimal communication time to reduce overall system communication overhead.

[0003] In related technologies, gradient descent is commonly used to optimize the model during federated learning. However, gradient descent is highly sensitive to feature scaling; if the features in the dataset have different scales, it can converge very slowly, especially in real-world scenarios where neural networks are widely used, such as optimizing non-convex neural networks. Furthermore, gradient descent requires multiple iterations to update model parameters, increasing communication overhead and latency. Therefore, a more widely applicable acceleration method is urgently needed to speed up the federated learning process. Summary of the Invention

[0004] This invention provides a federated learning acceleration method, apparatus, server, and medium to improve the acceleration effect of federated learning.

[0005] In a first aspect, this invention provides a federated learning acceleration method applied to a server, the method comprising:

[0006] The global model to be trained is sent to the client with the established communication connection, wherein the client is used to train the global model based on the local dataset to obtain the trained local model;

[0007] The system receives parameters of a local model uploaded by the client and compares the parameters to be optimized in the global model with the parameters of the local model to determine the gradient information for model optimization.

[0008] The optimization path is determined based on the gradient information, and the global model is optimized according to at least two optimization paths and their corresponding weight information.

[0009] The currently optimized global model is used as the new global model to be trained, and the operation of sending the new global model to be trained to the client with the established communication connection is executed until the number of communications with the client reaches the set threshold.

[0010] Optionally, determining the optimization path based on the gradient information and optimizing the global model according to at least two optimization paths and their corresponding weight information includes:

[0011] Construct two paths to be optimized with the same gradient information but different step sizes, where each optimization path satisfies the gradient descent method;

[0012] Based on the asynchronous update accelerated Nesterov method, the target optimization path between the two optimization paths is determined according to the two optimization paths and their respective weight information.

[0013] The global model is optimized according to the target optimization path.

[0014] Optionally, the global model is optimized according to the target optimization path, including:

[0015] The global model is optimized according to the following formula:

[0016] z t+1 =τ1μo t+1 +τ1η -1 y t+1

[0017] in, The first normalization coefficient is represented by μ, which is a hyperparameter, and τ is another hyperparameter representing the global learning rate. t This represents the constructed optimized path, whose update formula is o. t+1 =o t +(1 / μ)u t , t represents time t, t+1 represents the next time after time t, and Δt represents two adjacent time points. y t+1 Let z represent the optimized path at time t+1. t Let z represent the global model obtained by optimization at time t. t+1 Let m represent the global model obtained by optimization at time t+1. t s represents the approximate value of the first-order gradient at time t. t This represents adaptive momentum, and ∈ is a subterm added to enhance numerical stability.

[0018] Optionally, an optimization path is determined based on the gradient information, and the global model is optimized according to at least two optimization paths and their corresponding weight information, including:

[0019] The first optimization path is determined based on the gradient information, and the parameters of the local model sent by the client are used as the second optimization path, wherein the first optimization path satisfies the gradient descent method.

[0020] The global model is optimized based on the asynchronous update accelerated Nesterov method, according to the first optimization path, the second optimization path and their corresponding weight information.

[0021] Optionally, the asynchronous update-based accelerated Nesterov method optimizes the global model according to the first optimization path, the second optimization path, and their corresponding weight information, including:

[0022] The global model is optimized according to the following formula:

[0023]

[0024] Among them, z t+1 This represents the global model obtained by optimization at time t+1. Let μ represent the local model optimized by the i-th client at time t, S represent the total number of clients, and μ represent the local model optimized by the i-th client at time t. i This represents the hyperparameters used when the i-th client updates the local model, and is used to adjust... and z t The degree of approximation This represents the second normalization coefficient. Let z represent the first optimal path at time t+1. t Let m represent the global model obtained by optimization at time t. t s represents the approximate value of the first-order gradient at time t. t This represents adaptive momentum.

[0025] Optionally, after the number of communications with the client reaches a set threshold, the method provided in this embodiment of the invention further includes:

[0026] A convergence analysis is performed on the trained global model. If the prediction result of the trained global model converges to a set value, the training ends; if the prediction result of the trained global model does not converge to the set value, the trained global model is retrained.

[0027] Secondly, embodiments of the present invention also provide a federated learning acceleration device, comprising:

[0028] The global model sending module is configured to send the global model to be trained to a client with an established communication connection, wherein the client is used to train the global model based on a local dataset to obtain a trained local model;

[0029] The gradient determination module is configured to receive the parameters of the local model uploaded by the client, and compare the parameters to be optimized in the global model with the parameters of the local model to determine the gradient information for model optimization.

[0030] The global model optimization module is configured to determine an optimization path based on the gradient information, and optimize the global model according to at least two optimization paths and their corresponding weight information.

[0031] The loop module is configured to use the currently optimized global model as the new global model to be trained, and return to execute the operation of sending the new global model to be trained to the client with the established communication connection, until the number of communications with the client reaches a set threshold.

[0032] Optional, global model optimization module, including:

[0033] The path construction unit is configured to construct two paths to be optimized with the same gradient information but different step sizes, wherein each optimized path satisfies the gradient descent method;

[0034] The target optimization path determination unit is configured to determine the target optimization path between the two optimization paths based on the asynchronous update accelerated Nesterov method, and to determine the target optimization path between the two optimization paths according to the two optimization paths and their respective weight information.

[0035] The first global model optimization unit is configured to optimize the global model according to the target optimization path.

[0036] Optionally, the first global model optimization unit is configured as follows:

[0037] The global model is optimized according to the following formula:

[0038] z t+1 =τ1μo t+1 +τ1η -1 y t+1

[0039] in, The first normalization coefficient is represented by μ, which is a hyperparameter, and η is another hyperparameter representing the global learning rate. t This represents the constructed optimized path, whose update formula is o. t+1 =o t +(1 / μ)u t , t represents time t, t+1 represents the next time after time t, and Δt represents two adjacent time points. y t+1 Let z represent the optimized path at time t+1. tLet z represent the global model obtained by optimization at time t. t+1 Let m represent the global model obtained by optimization at time t+1. t s represents the approximate value of the first-order gradient at time t. t This represents adaptive momentum, and ∈ is a subterm added to enhance numerical stability.

[0040] Optional, global model optimization module, including:

[0041] An asynchronous path determination unit is configured to determine a first optimized path based on the gradient information, and to use the parameters of the local model sent by the client as a second optimized path, wherein the first optimized path satisfies the gradient descent method.

[0042] The second global model optimization unit is configured as follows:

[0043] The global model is optimized based on the asynchronous update accelerated Nesterov method, according to the first optimization path, the second optimization path and their corresponding weight information.

[0044] Optionally, the second global model optimization unit is specifically configured as follows:

[0045] The global model is optimized according to the following formula:

[0046]

[0047] Among them, z t+1 This represents the global model obtained through optimization at time t+1. Let μ represent the local model optimized by the i-th client at time t, S represent the total number of clients, and μ represent the local model optimized by the i-th client at time t. i This represents the hyperparameters used when the i-th client updates the local model, and is used to adjust... and z t The degree of approximation This represents the second normalization coefficient. y t+1 Let z represent the first optimal path at time t+1. t Let m represent the global model obtained by optimization at time t. t s represents the approximate value of the first-order gradient at time t. t This represents adaptive momentum.

[0048] Optionally, the apparatus provided in this embodiment of the invention further includes:

[0049] The convergence analysis module is configured to perform convergence analysis on the trained global model. If the prediction result of the trained global model converges to a set value, the training ends; if the prediction result of the trained global model does not converge to the set value, the trained global model is retrained.

[0050] Thirdly, embodiments of the present invention also provide a server, comprising:

[0051] Memory containing executable program code;

[0052] A processor coupled to the memory;

[0053] The processor calls the executable program code stored in the memory to execute the federated learning acceleration method provided in any embodiment of the present invention.

[0054] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the federated learning acceleration method provided in any embodiment of the present invention.

[0055] The technical solution provided in this invention involves the server sending the global model to be trained to a client with an established communication connection during the federated learning model optimization process. The client trains the global model based on its local dataset and sends the trained local model back to the server. The server receives the parameters of the local model uploaded by the client and compares the parameters to be optimized in the global model with the parameters of the local model to determine the gradient information for model optimization. This is achieved by applying the results of a federated learning adaptive optimization method with the same gradient information to another optimization path (including a constructed path satisfying gradient descent). t Or parameters uploaded by the client By performing weight summation, the same gradient information as the adaptive optimization method can be used to optimize the global model. This allows the adaptive optimization method to be combined with the Nesterov-type method, thereby improving the efficiency of model optimization. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart of the federated learning acceleration method provided in Embodiment 1 of the present invention;

[0058] Figure 2 This is a structural block diagram of the federated learning acceleration device provided in Embodiment 2 of the present invention;

[0059] Figure 3 This is a schematic diagram of the structure of a server provided in Embodiment 3 of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0062] This invention discloses a federated learning acceleration method, apparatus, server, and medium. The implementation principle of this invention will be briefly introduced below.

[0063] First, regarding federated learning, the objective function for optimization in federated learning is:

[0064]

[0065] There are M clients participating in the optimization of the global model parameter z, where F i Let D represent the optimization objective for the i-th client. Typically, the optimization objective includes using data from the client distribution D. i The data h is used to calculate the loss f i .

[0066] On the server side, the basic optimization steps of applying adaptive optimization methods, such as FedOPT (Federated Optimization, an optimization framework based on federated learning), are as follows: At time t, the server attempts to connect to active clients, whose set is S. t Then the global model z t Send the data to the client for independent training, with the client using a step size η. l Update K rounds. It's worth noting that the models between clients are based on their respective datasets D. iThe updates are received from client i, meaning the update trajectories between clients are different, i.e., "asynchronous". The server will utilize the updated model received from client i. Constructing pseudo gradient Use pseudo gradients to update the global model z t ω i These are the weights used during aggregation; for example, they could be n. i / N t n i It is the number of samples on client i. Then, the global model is updated using an adaptive method: y t+1 =z t +ηu t Where η is the step size on the server side, u t =Ψ(Δ t Ψ represents existing adaptive optimization methods, such as Adam (Adaptive Moment Estimation, a widely used deep learning optimization algorithm), Adagrad (a gradient descent optimization algorithm with an adaptive learning rate), and Yogi (an improved deep learning optimization algorithm).

[0067] Specifically, Adam's update process is as follows:

[0068] m t =β1m t-1 -(1-β1)Δ t ,

[0069]

[0070] Where m and v represent the first-order momentum (or the unbiased estimate of the gradient) and second-order momentum, respectively. β1 and β2 are coefficients used to calculate the pseudo-gradient mean and pseudo-gradient square. Adaptive update is... Here, ∈ is a subterm added to enhance numerical stability.

[0071] Secondly, regarding asynchronous update acceleration (Nesterov type), specifically, there are two trajectories a k+1 and b k+1 Their learning rates are different, meaning their update trajectories are asynchronous. Let's consider a certain trajectory c between them. k+1 The gradient is used as the update direction, c k+1 This is a global update of the trajectory. The specific update format is as follows:

[0072]

[0073]

[0074] c k+1=ρa k+1 +(1-ρ)c k+1

[0075] In this embodiment of the invention, a Nesterov-type acceleration method is applied to the adaptive optimization process on the server side of federated learning to accelerate the training process of deep learning models. Since adaptive optimization methods typically only use first-order gradient information, the objective function of the above-mentioned federated learning can be approximated as a first-order term through Taylor expansion. Then, two regularization terms are added to the approximation expression to obtain a quadratic function of the global model z to be optimized. Intuitively, Nesterov-type updates always have two asynchronous trajectories. In this application, the result of the above-mentioned adaptive optimization method can be used as one trajectory, and the other asynchronous trajectory can be the parameters received from the client, or a trajectory can be constructed. The two added regularization terms are used to control the distance (also called the approximation degree) between the global model parameters z and the sum of the two asynchronous trajectories. By solving for the minimum value of this quadratic function, the optimal solution for the global model parameters z at the current time t can be obtained. The following details the construction of the asynchronous trajectories and the application of the constructed Nesterov-type asynchronous trajectories to the adaptive optimization process of federated learning.

[0076] Example 1

[0077] Figure 1 This is a flowchart of a federated learning acceleration method provided in Embodiment 1 of the present invention, which is applied on the server side. The method provided in this embodiment can be executed by a federated learning acceleration device, which can be implemented in software and / or hardware. Figure 1 As shown, the method provided in this embodiment specifically includes:

[0078] S110. Send the global model to be trained to the client with the established communication connection.

[0079] The global model to be trained can be any neural network model that needs to be trained. The client can be a mobile device, such as a computer or mobile phone. The client stores the sample data needed to train the model. The number of clients can be one or more, and this embodiment does not specifically limit this.

[0080] In this embodiment, the server needs to establish a communication connection with the client before sending the global model to be trained. If multiple clients have successfully established a communication connection with the current server, the server can randomly select a set number of clients as target clients and send the global model to be trained to the target clients. After receiving the global model to be trained from the server, each target client will independently train the global model based on its own stored dataset. The number of training iterations can be set according to actual needs. After training is completed, a local model is obtained. Each client then uploads its trained local model to the server.

[0081] S120. Receive the parameters of the local model uploaded by the client, and compare the parameters to be optimized in the global model with the parameters of the local model to determine the gradient information for model optimization.

[0082] In this embodiment, the server can determine the gradient information for model optimization based on the ratio of the parameters of the local model uploaded by each client to the parameters to be optimized in the global model. This can be achieved through the following formula:

[0083]

[0084] Where, Δ t Let S represent the gradient information at time t, S represent the total number of clients, i represent the i-th client, and n represent the gradient information at time t. i This represents the number of samples on the i-th client. This represents the total number of samples across all clients. Let z represent the parameters of the local model uploaded by the i-th client at time t. t This represents the parameters to be optimized in the global model.

[0085] S130. Determine the optimization path based on the gradient information, and optimize the global model according to at least two optimization paths and their corresponding weight information. Use the currently optimized global model as the new global model to be trained, and return to execute the operation of sending the new global model to be trained to the client with the established communication connection until the number of communications with the client reaches the set threshold.

[0086] Those skilled in the art will understand that the native Nesterov model has two trajectories with the same gradient but different learning rates, meaning the update process is asynchronous. In this embodiment, to combine the federated adaptive optimization process with the Nesterov model to improve the model's update speed, two paths to be optimized can be constructed with the same gradient information but different step sizes. One optimization path can be the result of the aforementioned adaptive optimization method, which satisfies gradient descent. The other path can be an artificially constructed path that satisfies gradient descent but has a different learning efficiency than the adaptive optimization method. After constructing the two asynchronous optimization paths, the target optimization path between the two optimization paths can be determined based on the Nesterov model method, according to the two optimization paths and their corresponding weight information. That is, the federated learning adaptive optimization process is converted into the aforementioned Nesterov model form, and the global model is optimized according to the target optimization path. Specifically, regularization can be used when constructing the aforementioned native Nesterov model form.

[0087] As an alternative implementation, in combining the objective function of federated learning with Nesterov-type acceleration, the objective function of federated learning can first be approximated as a first-order term through Taylor expansion, and then the two regularization terms, i.e., the weighted L2 norm, can be applied. These are added to the first-order approximation expression to obtain a quadratic function of z. These two regularization terms are used to control the relationship between the global model parameter z and the two asynchronous trajectories o. t and y t The distance between them (also known as the degree of approximation), where the trajectory o t It is the constructed trajectory that satisfies the gradient descent method, the trajectory y t This is the result of the federated learning adaptive optimization method. Solving for the minimum value of this quadratic function yields the optimal solution for the parameters z to be optimized in the global model at the current time t. The process of solving for the minimum value of this quadratic function is the process of optimizing the global model according to the target optimization path in this embodiment, which can be specifically represented by the following formula:

[0088]

[0089] Among them, o t The constructed optimized path is represented by the update formula: Its weighting coefficient is τ1μ. This represents the first normalization coefficient, and μ is a hyperparameter used to control z. t With o t+1 The degree of approximation. The traditional adaptive optimization method is used. In this embodiment, an alternative optimization path is constructed, with a weight coefficient of τ1η. -1η is another hyperparameter, representing the global learning rate. t represents time t, t+1 represents the next time after time t, Δt represents two adjacent time points, and z t z represents the parameters to be optimized in the global model obtained at time t. t+1 m represents the parameters to be optimized in the global model obtained at time t+1. t m represents the approximate value of the first-order gradient at time t. t =β1m t-1 -(1-β1)Δ t s t This represents adaptive momentum. Let represent the second-order momentum at time t, β1 and β2 are coefficients used to calculate the pseudo-gradient mean and pseudo-gradient square, and ∈ is a subterm added to enhance numerical stability.

[0090] As an alternative implementation, since the server's extraction of the parameters of the local model updated by client i at time t is asynchronous with the parameters of the local model updated by other clients, in order to construct two Nesterov-type trajectories, the parameters of the local model sent by the client can be used as one optimization path, and another optimization path can be determined based on gradient information. That is, the adaptive optimization method in the federated learning process (such as the result of the Adam method, or the result of other adaptive optimization methods) can be used as the other path. Then, based on the Nesterov-type adaptive optimization method, the global model is optimized according to the first optimization path, the second optimization path, and their corresponding weight information. This setup, compared to the above construction of asynchronous trajectories, provides a more efficient approach. t The implementation method takes into account the actual update situation of asynchronous clients, and each update involves searching for the local model parameters obtained by the client. and adaptive optimization result y t The approximation between the two values ​​allows the updated model to combine the performance of both, thereby improving the model's accuracy.

[0091] Specifically, to combine the objective function of federated learning with Nesterov-type acceleration, the objective function of federated learning can first be approximated as a first-order term through Taylor expansion, and then two regularization terms, i.e., the weighted L2 norm, can be applied. These are added to the approximation expression to obtain a quadratic function of the parameters z to be optimized in the global model. These two regularization terms are used to control the relationship between the global model parameters z and the two asynchronous trajectories x. t and y t The distance between them (also known as the approximation level). Solving for the minimum of this quadratic function yields the optimal solution for the parameters z to be optimized in the global model at the current time t. The process of finding the minimum of this quadratic function is the process of optimizing the global model in this embodiment, which can be specifically represented by the following formula:

[0092]

[0093]

[0094] Among them, z t+1 This represents the global model obtained through optimization at time t+1. Let μ represent the local model optimized by the i-th client at time t, S represent the total number of clients, and μ represent the local model optimized by the i-th client at time t. i This represents the hyperparameters used when the i-th client updates the local model, and is used to adjust... and z t The degree of approximation, of which, This represents the second normalization coefficient. y t+1 Let z represent the first optimal path at time t+1. t Let m represent the global model obtained by optimization at time t. t s represents the approximate value of the first-order gradient at time t. t Denotes adaptive momentum, m t and s t For details, please refer to the calculation method provided above; it will not be repeated here.

[0095] In this embodiment, the results of the federated learning adaptive optimization method and another optimization path (o) are compared. t or By performing weight summation, the same gradient information as in adaptive optimization methods can be used to improve the efficiency of model optimization. Hyperparameter μ or μ i Used to control global model parameters z and y t The degree of similarity to other step trajectories. When μ = 1 / η or μ i When μ = 1 / η, all trajectories have the same weight. Specifically, the hyperparameter can be adjusted to have the same weight as the anchor point (α), for example, μ = α / η or μ i =α / η, at this point only α needs to be adjusted, avoiding the need to set μ individually for each client. i .

[0096] Furthermore, after the number of communications with the client reaches a set threshold, the method provided in this embodiment also includes:

[0097] Perform convergence analysis on the trained global model. If the prediction result of the trained global model converges to a set value, such as 0, or other values, then the training ends. If the prediction result of the trained global model does not converge to the set value, then the trained global model is retrained.

[0098] Specifically, when using the first implementation method in step S130, the convergence result can be calculated according to the following convergence calculation formula. When the convergence result is converged to a set value, it indicates that the training result of the global model has met the requirements. If the converged value is greater than the set threshold, it indicates that the training result of the global model has not met the requirements and needs to be retrained.

[0099] The first convergence criterion formula is:

[0100]

[0101] Where t represents time, T represents the total number of communications between the server and the client, and z t Let represent the global model obtained by optimization at time t. Indicate z t The gradient of , β2 represents the second momentum, G represents the upper bound of the gradient norm, ∈ represents the minimum value of the Adam denominator, and is a small term added to enhance numerical stability. K represents the number of local training iterations on the client side, and Φ represents the bound controlled by the initial value. var1 Indicates the first other noise boundary information, Φ var2 This indicates the second other noise boundary information. η represents the upper bound related to G and K. l This represents the client-side local learning rate, where μ is a hyperparameter used to control the global model parameters z and y. t The degree of similarity to other step trajectories. L represents the Lipschitz coefficient, τ1 represents the first normalization coefficient, M represents the total number of all clients, and η represents the step size on the server side.

[0102] Specifically, when using the second implementation method in step S130, the convergence result can be calculated according to the following convergence calculation formula two. When the convergence result is converged to a set value, it indicates that the training result of the global model has met the requirements. If the converged value is greater than the set threshold, it indicates that the training result of the global model has not met the requirements and needs to be retrained.

[0103] The second convergence criterion formula is:

[0104]

[0105]

[0106] Where, Φ var2 This represents the second other noise boundary information, τ2 represents the second normalization coefficient, and μ i This represents the hyperparameters when the i-th client updates the local model. In the two formulas above, The upper bound of local gradient bias is σ l The boundary value representing the upper bound of the local gradient bias, x i,k F represents the parameters of the local model trained by the i-th client for the k-th time. i (x i,k Let ) represent the loss function for the i-th client. Let g represent the first gradient of the loss function for the i-th client. i,k Let represent the gradient estimate for the i-th client. The upper bounds for local and global gradient bias are... σ g This represents the upper bound of the local and global gradient bias, where x is an arbitrary real-valued vector representing the model's parameters. This represents the gradient of the server-side vector with respect to any real number vector x. Let x represent the gradient of the loss function for the i-th client with respect to any real vector x.

[0107] The above two can be derived through the following reasoning:

[0108] First, we set two assumptions. Assumption 1: Hypothesis function F i It is L-smooth for all i∈[M], that is, for any real vector x,y∈R d (x, y are d-dimensional vectors defined in the real number field), satisfying L represents the Lipschitz smoothing coefficient, which is the bound controlling the change in the gradient norm. Assumption 2: Client-side gradient estimate g k It is unbiased, that is, for any i, k, we have Furthermore, its magnitude and variance are bounded, i.e., ∥g i,k ∥≤G and Among them, g i,k Let x represent the gradient estimate for the i-th client, G represent the maximum norm of all gradients, and x represent the gradient estimate for the i-th client. i,k This represents the parameters of the local model trained on the i-th client for the k-th iteration. The loss function F... i (x) has an upper bound on the global variance. M represents the total number of clients. In the case of independent and identically distributed clients, σ g =0.

[0109] Under the two assumptions mentioned above, let Furthermore, and Φ=τ1μ((f(z0)-f(z) * )) / (ηη l T). The iterative optimization of the global model in the above implementation method one satisfies the following convergence calculation formula one:

[0110]

[0111] The convergence speed of the above iterative formula can be divided into two parts. The first part is directly affected by T and gradually converges to zero as T increases. The second part is not directly affected by T and is mainly affected by the number of local training iterations and the degree of distribution offset. Increasing the number of clients will accelerate the first part of the convergence speed to some extent.

[0112] Under the following conditions: Assume The computational complexity of the client's local learning rate is... The value is related to, And set The complexity of representing the first normalization coefficient is... The calculation results are related. If T is large enough, then the convergence calculation formula corresponding to the global model optimization method one will satisfy:

[0113]

[0114] in, This represents the computational complexity, which depends on the value in parentheses. The formula above represents the parameters z of the global model at time t. t gradient The expected value of the norm The computational complexity and related.

[0115] When using the first implementation method in step S130, the convergence result can be calculated according to the above formula one. When the convergence result is converged to the set value, it means that the training result of the global model has met the requirements. If the converged value is greater than the set threshold, it means that the training result of the global model has not met the requirements and needs to be retrained.

[0116] For the second convergence calculation formula mentioned above, under the two assumptions above, let... and, And Φ=(f(z0)-f(z) * )) / (τ2η l T). The iterative formula of the second implementation method for global model optimization described above satisfies the following convergence criterion formula 2:

[0117]

[0118] The above convergence calculation formula 2 is also divided into two parts: one part is directly affected by T, and the other part is indirectly affected by the gradient limit and the degree of non-independent and identically distributed distribution.

[0119] Under the following conditions, assume The computational complexity of the client's local learning rate is... The value is related to, And set The complexity of representing the second normalization coefficient is similar to that of MK. 3 The calculation results are related. In the above convergence calculation formula two, when all μ i When they are equal, If T is large enough, then convergence calculation formula two satisfies:

[0120]

[0121] The above formula represents the parameters z of the global model at time t. t gradient expectation of the norm The computational complexity and The calculation results are related.

[0122] When using the second implementation method in step S130, the convergence result can be calculated according to the above formula two. When the convergence result is converged to the set value, it means that the training result of the global model has met the requirements. If the converged value is greater than the set threshold, it means that the training result of the global model has not met the requirements and needs to be retrained.

[0123] This embodiment provides two high-precision convergence analysis formulas, which can be used to perform convergence analysis on the trained global model. Analysis shows that both global model optimization methods one and two meet the convergence requirements in the non-convex case. Furthermore, experimental results demonstrate that they have faster convergence speeds and higher performance metrics.

[0124] In this embodiment, during the federated learning model optimization process, the server sends the global model to be trained to the client with the established communication connection. The client trains the global model based on its local dataset and sends the trained local model back to the server. The server receives the parameters of the local model uploaded by the client and compares the parameters to be optimized in the global model with the parameters of the local model to determine the gradient information for model optimization. The results of the federated learning adaptive optimization method with the same gradient information are then compared with another optimization path (including a constructed path that satisfies the gradient descent method). t Or parameters uploaded by the client By performing weight summation, the same gradient information as the adaptive optimization method can be used to optimize the global model. This allows the adaptive optimization method to be combined with the Nesterov-type method, thereby improving the efficiency of model optimization.

[0125] Example 2

[0126] Figure 2 This is a structural block diagram of a federated learning acceleration device provided in Embodiment 2 of the present invention, as shown below. Figure 2 As shown, the device includes: a global model sending module 210, a gradient determination module 220, a global model optimization module 230, and a loop module 240, wherein,

[0127] The global model sending module 210 is configured to send a global model to be trained to a client with an established communication connection, wherein the client is used to train the global model based on a local dataset to obtain a trained local model.

[0128] The gradient determination module 220 is configured to receive the parameters of the local model uploaded by the client, and compare the parameters to be optimized of the global model with the parameters of the local model to determine the gradient information for model optimization.

[0129] The global model optimization module 230 is configured to determine an optimization path based on the gradient information, and optimize the global model according to at least two optimization paths and their respective weight information.

[0130] The loop module 240 is configured to use the currently optimized global model as the new global model to be trained, and return to execute the operation of sending the new global model to be trained to the client with the established communication connection, until the number of communications with the client reaches a set threshold.

[0131] Optional, the global model optimization module 230 includes:

[0132] The path construction unit is configured to construct two paths to be optimized with the same gradient information but different step sizes, wherein each optimized path satisfies the gradient descent method;

[0133] The target optimization path determination unit is configured to determine the target optimization path between the two optimization paths based on the asynchronous update accelerated Nesterov method, and to determine the target optimization path between the two optimization paths according to the two optimization paths and their respective weight information.

[0134] The first global model optimization unit is configured to optimize the global model according to the target optimization path.

[0135] Optionally, the first global model optimization unit is configured as follows:

[0136] The global model is optimized according to the following formula:

[0137] z t+1 =τ1ηo t+1 +τ1η -1 y t+1

[0138] in, The first normalization coefficient is represented by μ, which is a hyperparameter, and η is another hyperparameter representing the global learning rate. t This represents the constructed optimized path, whose update formula is o. t+1 =o t +(1 / μ)u t , t represents time t, t+1 represents the next time after time t, and Δt represents two adjacent time points. Let z represent the optimized path at time t+1. t Let z represent the global model obtained by optimization at time t. t+1 Let m represent the global model obtained by optimization at time t+1. t s represents the approximate value of the first-order gradient at time t. t This represents adaptive momentum.

[0139] Optional, the global model optimization module 230 includes:

[0140] An asynchronous path determination unit is configured to determine a first optimized path based on the gradient information, and to use the parameters of the local model sent by the client as a second optimized path, wherein the first optimized path satisfies the gradient descent method.

[0141] The second global model optimization unit is configured as follows:

[0142] The global model is optimized based on the asynchronous update accelerated Nesterov method, according to the first optimization path, the second optimization path and their corresponding weight information.

[0143] Optionally, the second global model optimization unit is specifically configured as follows:

[0144] The global model is optimized according to the following formula:

[0145]

[0146] Among them, z t+1 This represents the global model obtained through optimization at time t+1. Let μ represent the local model optimized by the i-th client at time t, S represent the total number of clients, and μ represent the local model optimized by the i-th client at time t. iThis represents the hyperparameters used when the i-th client updates the local model, and is used to adjust... and z t The degree of approximation This represents the second normalization coefficient. Let z represent the first optimal path at time t+1. t Let m represent the global model obtained by optimization at time t. t s represents the approximate value of the first-order gradient at time t. t This represents adaptive momentum.

[0147] Optionally, the apparatus provided in this embodiment of the invention further includes:

[0148] The convergence analysis module is configured to perform convergence analysis on the trained global model. If the prediction result of the trained global model converges to a set value, the training ends; if the prediction result of the trained global model does not converge to the set value, the trained global model is retrained.

[0149] Example 3

[0150] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a server provided in Embodiment 3 of the present invention. Figure 3 As shown, the server may include:

[0151] Memory 701 storing executable program code;

[0152] Processor 702 coupled to memory 701;

[0153] The processor 702 calls the executable program code stored in the memory 701 to execute the federated learning acceleration method provided in any embodiment of the present invention.

[0154] This invention discloses a computer-readable storage medium storing a computer program that causes a computer to execute the federated learning acceleration method provided in any embodiment of this invention.

[0155] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0156] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0157] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0158] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0159] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0160] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0161] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated learning acceleration method, applied to a server, characterized in that, include: The global model to be trained is sent to the client with the established communication connection, wherein the client is used to train the global model based on the local dataset to obtain the trained local model; The system receives parameters of a local model uploaded by the client and compares the parameters to be optimized in the global model with the parameters of the local model to determine the gradient information for model optimization. The optimization path is determined based on the gradient information, and the global model is optimized according to at least two optimization paths and their corresponding weight information, wherein the gradient information of each optimization path is the same. The currently optimized global model is used as the new global model to be trained, and the operation of sending the new global model to be trained to the client with the established communication connection is executed until the number of communications with the client reaches the set threshold. The step of determining an optimization path based on the gradient information and optimizing the global model according to at least two optimization paths and their corresponding weight information includes: Construct two paths to be optimized with the same gradient information but different step sizes, where each optimization path satisfies the gradient descent method; Based on the asynchronous update accelerated Nesterov method, the target optimization path between the two optimization paths is determined according to the two optimization paths and their respective weight information. The global model is optimized according to the stated target optimization path; Alternatively, the step of determining the optimization path based on the gradient information and optimizing the global model according to at least two optimization paths and their corresponding weight information includes: The first optimization path is determined based on the gradient information, and the parameters of the local model sent by the client are used as the second optimization path, wherein the first optimization path satisfies the gradient descent method. The global model is optimized based on the asynchronous update accelerated Nesterov method, according to the first optimization path, the second optimization path and their corresponding weight information.

2. The method according to claim 1, characterized in that, The optimization of the global model according to the target optimization path includes: The global model is optimized according to the following formula: ； in, This represents the first normalization coefficient, where μ is a hyperparameter. It is another hyperparameter, representing the global learning rate. The constructed optimized path is represented by the update formula: , t represents time t, and t+1 represents the next time after time t. Indicates two adjacent moments. , This represents the optimized path at time t+1. Let represent the global model obtained by optimization at time t. This represents the global model obtained through optimization at time t+1. This represents an approximate value of the first-order gradient at time t. This represents adaptive momentum. It is a sub-item added to enhance numerical stability.

3. The method according to claim 1, characterized in that, The asynchronous update-based accelerated Nesterov method optimizes the global model according to the first optimization path, the second optimization path, and their corresponding weight information, including: The global model is optimized according to the following formula: ； in, This represents the global model obtained through optimization at time t+1. It's a hyperparameter. Let S represent the local model optimized by the i-th client at time t, and let S represent the total number of clients. This represents the hyperparameters used when the i-th client updates the local model, and is used to adjust... and The degree of approximation This represents the second normalization coefficient. This represents the first optimal path at time t+1. Let represent the global model obtained by optimization at time t. This represents an approximate value of the first-order gradient at time t. This represents adaptive momentum.

4. The method according to any one of claims 1-3, characterized in that, After the number of communications with the client reaches a set threshold, the method further includes: A convergence analysis is performed on the trained global model. If the prediction result of the trained global model converges to a set value, the training ends; if the prediction result of the trained global model does not converge to the set value, the trained global model is retrained.

5. A federated learning acceleration device, characterized in that, include: The global model sending module is configured to send the global model to be trained to a client with an established communication connection, wherein the client is used to train the global model based on a local dataset to obtain a trained local model; The gradient determination module is configured to receive the parameters of the local model uploaded by the client, and compare the parameters to be optimized in the global model with the parameters of the local model to determine the gradient information for model optimization. The global model optimization module is configured to determine an optimization path based on the gradient information, and optimize the global model according to at least two optimization paths and their corresponding weight information. The loop module is configured to use the currently optimized global model as the new global model to be trained, and return to execute the operation of sending the new global model to be trained to the client with the established communication connection, until the number of communications with the client reaches a set threshold. The global model optimization module includes: The path construction unit is configured to construct two paths to be optimized with the same gradient information but different step sizes, wherein each optimized path satisfies the gradient descent method; The target optimization path determination unit is configured to determine the target optimization path between the two optimization paths based on the asynchronous update accelerated Nesterov method, and to determine the target optimization path between the two optimization paths according to the two optimization paths and their respective weight information. The first global model optimization unit is configured to optimize the global model according to the target optimization path; or, The global model optimization module includes: An asynchronous path determination unit is configured to determine a first optimized path based on the gradient information, and to use the parameters of the local model sent by the client as a second optimized path, wherein the first optimized path satisfies the gradient descent method. The second global model optimization unit is configured as follows: The global model is optimized based on the asynchronous update accelerated Nesterov method, according to the first optimization path, the second optimization path and their corresponding weight information.

6. The apparatus according to claim 5, characterized in that, The device further includes: The convergence analysis module is configured to perform convergence analysis on the trained global model after the number of communications with the client reaches a set threshold. If the prediction result of the trained global model converges to a set value, the training ends; if the prediction result of the trained global model does not converge to the set value, the trained global model is retrained.

7. A server, characterized in that, The server includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the federated learning acceleration method as described in any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the federated learning acceleration method as described in any one of claims 1-4.

Images