A federated learning adaptive synchronization period control method

Abstract

The application provides a federal learning adaptive synchronization period control method (Adaptive Period Control, AdaPC) and a system for implementing the method, and belongs to the field of federal learning. The method sets the initial total number of iterations to 3 times, and from the fourth round, before the parameter server aggregates the iteration data of each client, the number of local updates to be performed by the client is adaptively adjusted in a recursive manner according to the previous iteration information. The system is composed of n devices, one of which serves as a parameter server for collecting and processing the information uploaded by the clients, and the remaining devices serve as clients for performing local calculations and uploading the calculation data to the parameter server. The application achieves the best balance between convergence rate and communication efficiency, effectively reduces the communication overhead caused by frequent exchange of model parameters between the client server and the parameter server, and achieves faster model convergence while maintaining lower training loss and minimal communication overhead.

Description

Technical Field

[0001] This invention belongs to the field of federated learning. During the training process of a federated learning model, before each parameter server aggregation, the number of local updates to be performed by the client can be adaptively adjusted using this invention. Background Technology

[0002] Federated Learning (FL) is a distributed machine learning (DML) technique. Its core idea is to train a model across multiple data sources (parameter servers) that possess local data. Without exchanging local individual or sample data (client-server data), it constructs a global model based on virtual fused data by exchanging only model parameters or intermediate results, thus achieving a balance between data privacy protection and shared computation. In recent years, this field has received considerable attention from both research and application perspectives.

[0003] The framework of federated learning can be summarized as having a global model whose initial state can be represented by x0, which can be a randomly initialized vector or a pre-trained model. For each iteration k, all devices start from the same initial point x. k Initially, each device updates its local model by performing τ stochastic gradient descent (SGD) steps using a small batch of data randomly selected from its local dataset. These updates are then sent back to the parameter server, where they are averaged and used to update the global model. The scalability of the stochastic gradient descent (SGD) machine learning framework is inherently hindered by the communication overhead caused by the frequent exchange of model parameters between the client and the parameter server.

[0004] In existing technologies, there are three main methods to minimize communication overhead: gradient and model compression, asynchronous SGD, and periodic averaging. Gradient and model compression involves compressing the weight vectors exchanged in the network using techniques such as quantization and sparsity during distributed learning. This method can significantly reduce accuracy due to information loss. Asynchronous SGD allows synchronizing the global model without waiting for all clients, but it suffers from gradient instability, i.e., the synchronization problem between the global model and outdated gradients from slower clients. Periodic averaging can set a fixed or adaptive synchronization period. Fixed synchronization periods are simple, converge quickly during the initial iterations, and effectively minimize communication costs. However, considering communication latency and global model convergence issues, the short period results in high communication costs, and over a long period, the fixed-period averaging method converges to a non-optimal stationary solution. Research on adaptively adjusted periodic averaging methods is currently scarce. The latest research on this problem is the AdaComm strategy proposed by J. Wang and G. Joshi. The AdaComm strategy divides the training into sub-intervals and attempts to find the optimal period for each sub-interval. The AdaComm strategy has three main problems. First, it doesn't offer a solution for choosing the optimal initial period, only suggesting that the initial period must be a large value. Subsequent research has shown that starting training with a large period may lead to instability. Second, AdaComm tends to decay during model convergence, increasing communication overhead and slowing down training. Third, when the training loss reaches a plateau and can no longer be reduced, the AdaComm strategy cannot adapt to this period, and the optimal period will always return a constant value. Summary of the Invention

[0005] The purpose of this invention is to address the problems in existing technologies, such as instability caused by starting training with a large period, slow training speed due to high communication overhead, and the inability to further reduce training loss once it reaches a plateau. This invention provides an adaptive synchronization period control method for federated learning. Applying this invention to federated learning model training can achieve an optimal balance between convergence rate and communication overhead.

[0006] The objective of this invention is achieved through the following technical solution.

[0007] A federated learning adaptive cycle control method includes the following steps: Let the initial total number of iterations be 3. The synchronization cycle of the first iteration is equal to the synchronization cycle of the second iteration, while simultaneously satisfying that the synchronization cycle of the second iteration is less than the synchronization cycle of the third iteration. This is done because sufficient initial iteration information data is required, and the synchronization cycles of the first three iterations must be clearly defined. Setting the synchronization cycle of the first iteration equal to the synchronization cycle of the second iteration, while simultaneously satisfying that the synchronization cycle of the second iteration is less than the synchronization cycle of the third iteration, is for computational convenience. In this way, only the synchronization cycle τ2 of the second iteration and the synchronization cycle τ3 of the third iteration need to be defined to begin training. From the fourth iteration onwards, using the formula... Calculate the synchronization period τ for each round. k , where F(x) i ) represents the global loss corresponding to the i-th iteration, T represents the total number of training iterations, k represents the current iteration round, and τ i Let τ represent the synchronization period of the i-th iteration, and satisfy the following relationship: T ≠τ2 and Next, the calculated synchronization period τ k Normalization is performed to ensure the synchronization period τ k All are greater than or equal to 1. If the synchronization period of the calculated k-th iteration is not equal to the synchronization period of the second iteration, and the global loss corresponding to the weight parameters of the (k-1)-th iteration is less than or equal to the global loss corresponding to the weight parameters of the (T-1)-th iteration, then the total iteration count T is updated to the current iteration count k. Next, the parameter server will set the synchronization period τ of the k-th iteration. k and average weight parameter x k The broadcast is sent to all clients, and each client i receives the synchronization period τ. k , to perform τ k The local update calculates the local stochastic gradient descent weights for each client i during the j-th local iteration update in the k-th round of model training iteration. and local losses Weight and loss The data is uploaded to the parameter server, which averages the model weights of all clients to obtain the global model parameter weights. The global loss is then calculated based on these global parameter weights. Finally, the parameter server broadcasts the updated global model parameters and the synchronization period for the next iteration to the clients. The clients then use the new model parameters and synchronization period for the next calculation, and this process continues until the number of iterations exceeds the total global aggregation.

[0008] Beneficial effects

[0009] The federated learning adaptive cycle control method of this invention can achieve faster convergence of the trained model while maintaining low training loss and communication overhead. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating the federated learning adaptive cycle control method.

[0011] Figure 2 A schematic diagram of the system framework for the federated learning adaptive periodic control method;

[0012] Figure 3 To determine the learning curves for choosing between two strategies: short-cycle and long-cycle. Detailed Implementation

[0013] To make the advantages and features of the present invention more readily understood by those skilled in the art, the present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments are merely illustrative of the present invention and are not intended to limit the invention.

[0014] Please see Figure 1 , Figure 2 , Figure 3 .

[0015] In this embodiment of the invention, Tiny ImageNet, a miniature version of the ImageNet dataset, is used to conduct experiments on image classification tasks. Tiny ImageNet contains 110,000 images and 200 classes. Each class has 500 64*64 red, green, blue (RGB) images for training and 50 images for testing. The inventors used this dataset to train the MobileNetV2 network from scratch.

[0016] The invention method is implemented in Python using the PyTorch deep learning framework, on a local cluster with 10 clients, and communication between the clients is performed using the Open MPI library.

[0017] To simulate bandwidth and high latency similar to those in the real world and distributed learning networks, the NETEM API2 mini-batch size ξ was fixed at 64, and the learning rate η was 0.001. Local updates were performed using the Adam optimizer. For preprocessing, the network input was converted to grayscale and normalized with a mean and standard deviation of 0.5. To improve training accuracy while significantly increasing the complexity of the method, only two simple data augmentation techniques were applied: horizontal flipping and vertical flipping.

[0018] This invention embodiment considers both Independent Identically Distributed (IID) and Non-Independent Identically Distributed (Non-IID) scenarios for data between client machines. In the IID scenario, a subset of the dataset containing 10 classes is used, evenly distributed across 10 clients, with each client seeing 500 images. In the Non-IID scenario, a subset of the dataset containing 10 classes is used, with each client seeing only 3 classes of images, and each client |D i The number of instances of | is between 400 and 600.

[0019] See the instruction manual appendix Figure 1 The method of this invention is implemented through the following steps: Let the initial iteration count be 3. The synchronization period of the first iteration is equal to the synchronization period of the second iteration, i.e., T = 3, τ1 = τ2. Simultaneously, the synchronization period of the second iteration is less than the synchronization period of the third iteration, i.e., τ2 < τ3. Define τ2 and τ3 as (01, 02). Use the synchronization periods of the second and third iterations (01, 02) as input, and x0 as pre-trained or randomly initialized weights. Initialize the total number of training iterations T to 3. Then perform K iterations of training. The specific steps for each iteration are as follows:

[0020] First, based on the number of iterations k in this round, determine the synchronization period τ for the k-th iteration. k Perform the assignment. If the current iteration number k is less than or equal to 2, then set the synchronization period τ of the current iteration to... k Let τ2 be the synchronization period for the second iteration; if the current iteration number k equals 3, then let τ2 be the synchronization period for the second iteration. k Assign the value τ3; if the iteration number k is greater than 3, then set τ... k Assign the following value to expression (2):

[0021]

[0022] Where F(x) i ) represents the global loss corresponding to the weights in the i-th iteration, calculated using the formula: F i (x) represents the local loss of a client, and its calculation formula is: T represents the total number of training iterations, k represents the current iteration round, and τ i D represents the synchronization period of the i-th iteration, n represents the number of client devices, and D i It is its local private data slice. D i,j It is D iThe j-th data point in the dataset is f(.,.), which is the training loss function used by the client.

[0023] τ k Assigning the value to equation (2) can achieve the goal of keeping the model convergent. Since the choice of synchronization period has a significant impact on the convergence speed of distributed optimization algorithms, when the period is fixed at a small value throughout the training process, the frequency of model parameter aggregation is much higher, so the model converges very slowly and the error is small. Its main advantages are small period, small training error, and good convergence. The disadvantage is that it requires frequent aggregation, which increases communication overhead and leads to a slower convergence speed. When the period is fixed at a large value, the aggregation frequency of model parameters is lower. Compared with the previous case, the model converges very quickly and the error is large. The main advantage of a large period is that it reduces communication overhead and has a fast convergence speed. The disadvantage is that the model will converge to a suboptimal solution and can no longer reduce the training error. The learning curves of the two strategies are as follows: Figure 3 As shown. From this, we can deduce that the optimal period is not a constant and must be adaptively adjusted, starting with a large period to achieve rapid convergence, and then changing to a small period at their intersection to achieve a lower training error.

[0024] The specific derivation process of formula (2) is as follows:

[0025] Assumption 1: (Smoothness) F(x) is differentiable and L-Lipschitz is continuous, then:

[0026]

[0027] Assumption 2: (Bounded Gradient Variance) The stochastic gradient variance of each mini-batch is bounded:

[0028]

[0029] The standard measure of convergence of a distributed stochastic gradient descent framework is the average of the squared gradient norm (SGN), which is defined as:

[0030]

[0031] Where K is the total number of training iterations. When equation (3) decreases during iteration, the model is convergent, and the smaller equation (3) is, the better the convergence.

[0032] Under assumption (1,2), if the learning rate satisfies Since all clients are initialized at the same starting point x1, the range of the squared gradient norm after the Kth iteration is:

[0033]

[0034] Where L is the Lipschitz constant defined in the assumptions, and σ² is the variance bound defined in the assumptions. F(x) is the global loss function, F inf For the optimal global loss, k is the node index, n is the number of clients, K is the total number of global aggregations, which can be regarded as communication overhead; η is the learning rate; τ is the number of local updates, i.e., the synchronization period; Let F be the gradient.

[0035] Our goal is to minimize the average SGN on the left side of inequality (4) to ensure convergence. One way to achieve this is to minimize the upper bound on the right side of inequality (4). Therefore, if we can set the value of τ on the right side to minimize the upper bound, the average SGN on the left side will also be minimized, thus ensuring convergence. By minimizing the value of τ on the right side of the above equation... To minimize the sum of elements, the method to achieve this goal is to adjust the synchronization period.

[0036] because Therefore, the optimal adaptive periodic strategy should satisfy:

[0037]

[0038] Therefore, it can be inferred that the optimal synchronization period τ K The choice should satisfy the following equation:

[0039]

[0040] The calculation of τ can be derived from equation (6). K The formula is as follows. This equation still contains many unknown components, which need to be processed using information from previous iterations.

[0041] To remove the constant from equation (6) In the second iteration of training, equation (6) is redefined when K = 2.

[0042]

[0043] Subtracting equation (7) from equation (6) yields:

[0044]

[0045] Next, the constants in equation (8) are handled in the same way. When the range of T is 2 < T < K:

[0046]

[0047] Dividing formula (8) by formula (9) yields the following easily calculated synchronization period formula:

[0048]

[0049] Equation (10) gives the formula for calculating the synchronization period, τ K It is calculated based on the information from the iterations of K=2,T,K-1.

[0050] Next, regarding τ k Normalization is performed to ensure the synchronization period τ k All are greater than or equal to 1. That is, if the result calculated in equation (2) is less than 1, then let τ k Equal to 1, otherwise τ k The result is equal to the calculation result of equation (2). Then update the total number of iterations T. The update rules are shown in equations (11) and (12):

[0051] τ k ≠τ2 (11)

[0052] Where τ i This represents the synchronization period of the i-th iteration. That is, equation (11) indicates that the synchronization period of the k-th iteration cannot be equal to the synchronization period of the second iteration.

[0053] F(x k-1 )≤F(x T-1 (12)

[0054] Where F(x) i ) represents the global loss corresponding to the weight in the i-th iteration. That is, equation (12) means that the global loss corresponding to the weight in the (k-1)-th iteration is less than or equal to the global loss corresponding to the weight in the (T-1)-th iteration.

[0055] If both equations (11) and (12) are satisfied, then the total number of iterations T is equal to the number of iterations k in this round. The recursion in equation (2) occurs in τ. T The process terminates when τ = 2. In this case, τ K =τ2 is always a constant. To avoid this situation, T needs to be updated to the latest iteration number, where τ must satisfy... T ≠τ2. The second condition selects the best iteration with the minimum global loss to date. Although the update rule increases the local computational cost of the client, it reduces communication overhead and accelerates the convergence rate. Moreover, when the loss reaches a plateau and can no longer be minimized, the local computational cost will gradually decrease. Satisfying these two conditions allows T to be assigned to the best training with the minimum global loss, ensuring that the solution of this invention can reduce communication overhead.

[0056] Obtain the synchronization period τ k Afterwards, the parameter server will synchronize the period τ. k And the model parameters x in the kth roundk It is broadcast to the clients, and each client i performs τ. k The local update calculates the local SGD weights for each client i during the j-th local iteration update in the k-th round of model training iteration. And upload it to the parameter server. Equation (13) is the calculation formula for each iteration update:

[0057]

[0058] Where k is the number of training iterations, i is the client, η is the learning rate, and j is the client's local update of the jth time; Let be the local SGD weights of client i after performing j-1 local updates. Let η be the learning rate. According to A small batch ξ selected randomly i,j Calculate the stochastic gradient.

[0059] The parameter server calculates the average of the model parameter weights of all clients using formula (14) to obtain the global model parameters for the (k+1)th iteration.

[0060]

[0061] Where D i The local dataset representing client i, |D i |Is D i Size. Let be the local SGD weights of client i after performing j local updates in the k-th iteration. η is the learning rate, x... k+1 The average weight is used to train the (k+1)th iteration, where n is the number of clients.

[0062] The parameter server calculates the global model loss for the kth iteration using formula (15).

[0063]

[0064] in For client i in executing τ k The local SGD weights after the next local update, where n is the number of clients, F(x) k ) represents the model weights x k The overall loss.

[0065] Finally, output the global model parameters x obtained after K iterations. K .

[0066] The system model of this invention is composed of Figure 2As shown, the system consists of n devices, which act as clients for local training. Each device then transmits its model weights and loss to a parameter server. The parameter server averages the information uploaded by all clients to obtain the synchronization period for the current iteration, which is then broadcast to the clients. Each client performs local updates according to the obtained synchronization period, uploading its local loss and model weights to the parameter server, iteratively executing the above steps.

[0067] Beneficial effects are described in the instruction manual. Figure 3 In the figure, the black dotted lines represent the experimental results. As can be seen from the figure, although a large synchronization period can reduce the training loss relatively quickly over time, it can no longer reduce the training loss after a certain point. In contrast, the training loss of a small synchronization period decreases very slowly over time, requiring a long time to reduce the training loss to the same level as that of this invention. Therefore, this invention achieves a good balance between convergence rate and communication overhead, reducing the training loss quickly and allowing it to continue to decrease, thus achieving beneficial results.

[0068] The technical solution of the present invention has been described in detail above with reference to the embodiments shown in the accompanying drawings. However, the scope of protection of the present invention is obviously not limited to this specific embodiment. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, such as increasing the initial total number of iterations, adding or changing data augmentation techniques, etc., and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A federated learning adaptive synchronization periodic control method applied to image classification, the method involving a parameter server and a client, characterized in that: Step A: Initialize the total number of training iterations The initial value is 3, and the synchronization period of the first iteration is equal to the synchronization period of the second iteration. ,Right now The synchronization period of the second iteration is shorter than the synchronization period of the third iteration. ,Right now The synchronization cycles of the second and third iterations are used as input. As pre-trained or randomly initialized weights, in preparation for subsequent... Training in the next iteration; Step B: Before the start of each iteration, based on the number of iterations... The global loss corresponding to the weights in the previous iteration Calculate the synchronization period of this round ; obtain the synchronization period Afterwards, the parameter server will synchronize the period. and the Wheel model parameters Announced to all clients via broadcast; during the calculation of the synchronization period for each iteration. At that time, except for those already assigned values and When the number of iterations If it is greater than 3, then... Assign the following value: in Representing the The global loss corresponding to the weights of each iteration is calculated using the following formula: , The local loss of a specific client is calculated using the following formula: , Represents the current iteration round. Representing the Synchronization cycle of round iteration, Represents the number of client devices. It is its local private image data; yes The first in Data points, It is the training loss function used by the client; Step C, the client obtains the synchronization cycle. After that, each client conduct The local update calculates the... Each client in the round Local update The local SGD weights are then uploaded to the parameter server. Step D, the parameter server receives each client Local iterative updates After the local SGD weights are applied, the parameter server calculates the average of the model parameter weights for all clients using the following formula to obtain the _th_ weight. Global model parameters for each iteration: in Representing the client Local image dataset, yes Size, p i For the client The proportion of the local image dataset in the global dataset; For the first Client in round iteration In execution The local SGD weights after the next local update, i.e. the weights uploaded by the client to the parameter server; For learning rate, For the training iteration Average weight of each time, The number of clients; then, the parameter server calculates the number of clients using the following formula. Global model loss in rounds of iteration: in For the client In execution Local SGD weights after the next local update For the number of clients, Model weights The global loss is calculated to prepare for the calculation of the next round of synchronous iteration; Step E: Repeat steps BD until the target number of iterations is completed. ; Step F, Output The global model parameters obtained after the second iteration .

2. The federated learning adaptive synchronization periodic control method as described in claim 1, characterized in that: In step C, each client conduct Secondary local update, calculates client Local update Subsequent local SGD weights; The formula for calculating each client's local weight update in each iteration is as follows: in The number of training iterations. For the client, For learning rate, Update the client locally Second-rate; For the client In execution Local SGD weights after the next local update; For learning rate, According to A small batch selected randomly Calculate the stochastic gradient.

3. A system for implementing the federated learning adaptive synchronization periodic control method of claim 1, characterized in that: The system consists of n devices, one of which acts as a parameter server to initiate the learning process, perform iterative calculations, broadcast the iterative calculations to other devices acting as clients, collect and process the information uploaded by the clients; The remaining devices act as clients, performing local calculations and uploading the results to the parameter server.

Images