Improved federated learning aggregation acceleration method

Abstract

The application discloses a transverse synchronous federal learning aggregation acceleration method, and steps are as follows: a client receives a global model and a client contribution value sent by a server; the client trains a local model by using a local data set, calculates a divergence threshold value, and prepares to send the trained local model; the client optimizes the local model by using a difference matrix and a gradient compression algorithm, and sends the optimized local model to the server; the server stores all the received models in a buffer area, and calculates information entropy; the server calculates the contribution value of the last round of clients, saves the information entropy, takes all the local models from the buffer area, and obtains a new global model by weighted aggregation based on the contribution value of the local model; the server sends the contribution value of the last round and the new global model to the client, the client receives the global model sent by the server, and performs next round training. The application can improve the accuracy of a transverse synchronous federal training model, and reduce the communication resource and time cost of the client to the server.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of transverse federated learning, and is especially suitable for solving the problems of large device data difference, poor calculation efficiency, slow time response and low communication efficiency in the process of training a model by synchronous federated learning and aggregating and updating the model. BACKGROUND

[0002] With the advent of the big data era, attaching importance to data privacy and security has become a worldwide trend. Compared with the data security risks caused by uploading local data to a central server to train a model in traditional machine learning (ML), federated learning (FL) protects user data privacy and security by using an architecture in which data does not move and models move, and supports multiple parties to jointly model under the premise of data privacy and security and regulatory requirements, thereby providing a feasible method for solving data silos.

[0003] The main architecture of transverse synchronous federated learning is the federated averaging algorithm (FedAvg), which is suitable for scenarios in which the data characteristics of the participating parties overlap to a large extent, the sample spaces overlap to a small extent, the geographical distribution is close, and the communication environment is good. In the FedAvg algorithm, each participating party client (Party) first receives an initial model issued by the server, synchronously trains a local model using local data multiple times, uploads the local model to the server for aggregation of an average model, the server obtains a new global model and issues the global model to the client for a new round of model training. In the training process, the client trains a private local model locally, and uploads the encrypted model to the server to update the global model by using encryption, noise and other means.

[0004] The federated averaging algorithm can effectively fuse data features from multiple clients through multiple rounds of local model iteration, but as the data volume and diversity of user device datasets increase, the computing demand of local devices and the local model trained also increases, which can cause a large time overhead in the process of uploading the local model to the server. The time overhead of synchronous federated learning mainly includes two parts: computing time overhead and communication time overhead. The computing time overhead includes server aggregation time overhead and client local training time overhead. The communication time overhead includes the time spent by the client in sending data to the server, the time spent by the server in waiting for the client to send data, and the time spent by the server in sending data to the client. In the application scenario of federated learning, the server is usually played by a role with high computing power and stable network resources. Therefore, the computing overhead and sending overhead of the server can be almost ignored. In synchronous federated learning, reducing the training overhead and communication overhead of the server has become the main way to accelerate federated learning. Therefore, it is necessary to propose an acceleration scheme for federated learning in the aspects of client local training model and server aggregation model to improve the accuracy of horizontal synchronous federated training model and reduce the communication resources and time overhead of the client to the server. SUMMARY

[0005] In order to overcome the shortcomings of the prior art and meet the needs of the above scenarios, the purpose of the present application is to provide an improved horizontal synchronous federated learning acceleration scheme. Based on the idea of difference matrix and gradient compression, the amount of data sent by the client to the server is reduced under the overall framework of federated averaging algorithm, so as to optimize the communication efficiency of uploading the local model of the client to the server; based on D-S evidence theory, a participant optimization scheme is proposed, and a score weight is given to each participant according to the influence of the participant on the global model. In the aggregation stage, the server aggregates according to the weight coefficients of different clients instead of averaging aggregation; a dynamic updating strategy based on information entropy and divergence threshold. For the stage of sending the local model of the participant to the global server, the divergence threshold is used as a standard to measure whether to send data to the server, which has better flexibility.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0007] The horizontal synchronous federated learning aggregation acceleration method comprises the following steps:

[0008] Step 1, the client receives the global model and the client contribution value sent by the server;

[0009] Step 2, the client trains a local model using the local dataset and calculates a divergence threshold, and prepares to send the trained local model;

[0010] Step 3, the client optimizes the local model by using the difference matrix and the gradient compression algorithm, and sends the optimized local model to the server, the server stores all the received models in the buffer, and calculates the information entropy;

[0011] Step 4, the server calculates the contribution value of the last round of clients, and saves the information entropy, the server takes all the local models from the buffer, and obtains a new global model by weighted aggregation based on the contribution value of the local model;

[0012] Step 5, the server sends the contribution value of the last round and the new global model to the client, and the client receives the global model sent by the server and performs the next round of training.

[0013] The step 1, the global model is obtained by aggregating the local models of each client, the process is: the client trains locally based on the local data set to obtain a local model, the local model is updated based on the deep gradient compression algorithm and the divergence threshold, and then uploaded to the server buffer, and then the local model uploaded to the server buffer is aggregated and weighted by the contribution value of each client to obtain a new global model; the contribution value of the client is calculated based on D-S evidence theory (Dempster-Shafer evidence theory, D-S), which represents the contribution degree of the client to the global model in the last round, and is an important standard for weighted aggregation of the global model. When federated learning starts, the client participating in the training will be assigned an average contribution value; in the server aggregation stage, the server obtains the relevant parameters of each client, including the ratio of the client data set to the whole data set, the contribution value of the last round of clients, the divergence between the local model trained by the client and the global model, and the number of training rounds participated by the client so far. Considering that data heterogeneity may cause the Matthew Effect of client contribution value in the aggregation process of federated learning, the invention adopts an evidence theory algorithm based on Pearson correlation coefficient to calculate the contribution value of the new round of clients based on the relevant parameters of the client, to buffer the problem of inaccurate fusion results when there is a large conflict proposition.

[0014] The step 2, the local data set represents the data distributed in the local client, which contains the data characteristics of the local user of the client, different clients have different local data sets with different data types and sizes, and do not share data with each other; the local model is a model obtained by training the global model based on the local data set of different clients. In the initial training stage of federated learning, the server distributes a random model as the global model to the clients, each client accepts the global model distributed by the server and trains the global model based on the local data set to obtain a model with local data characteristics of the client, which is called a local model.

[0015] The step 2, the calculation method of the divergence threshold value is as follows:

[0016] In the local model training stage, the client calculates the divergence threshold value of the local model to judge whether the update of the local model is sufficient, and when all the updates are sufficient, the client sends the local model to the server to continue the local training;

[0017] When the i-th client participating in the federated learning training completes a round of local iteration, the iteration round n of the local model is recorded by a counter, and the divergence threshold value is calculated by The divergence threshold value of the client Party i is calculated, and when the participant completes a batch of local model iterations and the divergence threshold value satisfies the condition formula: The client Party i sends the local model to the server;

[0018] In the formula, n represents the local training round of the client, the character Δ represents the divergence threshold value of the model, the superscript i of Δ i represents the divergence value of the local model of the i-th client, w represents the federated learning model, w represents the local model trained by the i-th client in the t-th round, w * represents the reference model, γ i represents the contribution value of the client Party i, and the formula describes the distance between the local model and the reference model.

[0019] In the horizontal synchronization federated learning, in order to speed up the communication efficiency between the client and the server, the global round and the local round of training are set to fixed values. The global round represents the number of times of updating the server global model, and the local round represents the number of times of updating the local model by the client based on the local data set and the global model;

[0020] In the present application, all the clients participating in the training satisfy the divergence threshold value condition and after the local training round reaches the set fixed value round, the local model can be uploaded to the server buffer to complete the local model training of the current round.

[0021] The step 3, the method for optimizing the local model is as follows:

[0022] (1) The client calculates the difference matrix. It is assumed that there are M clients participating in the training in each round, i∈[1, M], and M is the maximum number of clients participating in the training. When the difference between the local model and the global model is small, that is, ε is an arbitrary constant approaching to 0, and the client still sends the local model trained locally to the server Waste of communication resources. Therefore, at this stage, the client can only send the difference between the local model and the global model to reduce the amount of data sent to the server, that is, the client sends the difference matrix between the global model and the local model to the server;

[0023] (2) Through the deep gradient compression algorithm, and send the calculation result to the server end, through the difference matrix, it can be found that when the client sends the local model to the server, there are some small changes in the matrix. The small changes in these values may have little effect on the server model in this round of update, but sending a small number still needs to occupy network bandwidth. Therefore, a deep gradient compression strategy is introduced, a compression threshold constant th is selected, and the small intermediate value is not sent during training. Instead, it is saved and added to the same position of the next round of matrix. When the intermediate value is large enough, that is, the intermediate value> th, the intermediate value is sent to the server.

[0024] The step 3, the information entropy calculation method is as follows:

[0025] The server calculates the information entropy according to the situation of the tth round of client Party i The local model with a large enough entropy value is proposed, and a global weighted aggregation operation is performed, wherein H (U) represents the information entropy of the local model uploaded by the client Party i, E (.) is a mathematical expectation, and log (.) is a logarithm operation.

[0026] The step 4, the contribution value calculation method of the last round of client is as follows:

[0027] First, the contribution value correlation matrix S of each client is calculated, and the formula is as follows:

[0028]

[0029] Second, based on the correlation matrix, the credibility cred (γ i ) of the client Party i is calculated, and the formula is as follows:

[0030]

[0031] Wherein μ represents expectation, and the contribution value γ i of the client is subscripted, then represents the mathematical expectation of the contribution value γ i of the client i, σ represents variance, and the contribution value γ i of the client is subscripted, then represents the variance of the contribution value γ i of the client i, cred represents the credibility of the client, and the corrected basic probability distribution BPA is calculated, which is defined X represents the parameters of the client, and the history of the training rounds τ participated by the client, the proportion D of the client i to the whole client data set i / D, KL-divergence fusion, and the fused result γ i As the contribution value of the client Party i to the global model in the tth round, the formula is as follows:

[0032]

[0033]

[0034] The method of step 4, weighted aggregation, to obtain a new global model is as follows:

[0035] (1) In the global model aggregation stage, when the server completes the contribution value and information entropy calculation of all participants, the contribution degree of each client and the proportion of its data set are taken as weights to perform the t+1th round of global model w t Aggregation, the formula is as follows:

[0036]

[0037] (2) After obtaining the t+1th global model w t+1 , the server sends w t and the client contribution value γ i to the client participating in the training together, and the client receives the global model w t+1 to perform a new round of training.

[0038] Compared with the existing horizontal synchronous federated learning, the method of the present application can provide the following aspects of efficiency advantage and security:

[0039] 1) Based on the idea of difference matrix and gradient compression to reduce the amount of data sent by the client to the server, to achieve the effect of optimizing the communication efficiency of the client uploading the local model to the server;

[0040] 2) Based on the D-S evidence theory, the participant optimization scheme is proposed, the score weight of each participant is given combined with the influence of the participant on the global model, and the high-quality client local model is selected for aggregation to improve the accuracy of the model;

[0041] 3) Based on a dynamic updating strategy of information entropy and divergence threshold, the aggregation of high-quality client local model and the updating of global model are accelerated. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a framework of a horizontal federated learning fault detection method based on multi-layer grouping aggregation.

[0043] Figure 2 is a visual diagram of the depth gradient method. DETAILED DESCRIPTION

[0044] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and examples.

[0045] As a paradigm of distributed machine learning, federated learning can jointly model multiple institutions under the condition of meeting user privacy protection and data security, so that participants can jointly model without sharing local data, effectively solving the "data island" problem from a technical point of view.

[0046] As an efficient federated learning sharing model framework, transverse synchronous federated learning is suitable for application scenarios with close geographical distribution of sample data and good communication environment. It updates the local model through joint iteration of multiple devices locally synchronized, and then collects the local model sent by each device to update the global model on the server side. However, the heterogeneity of user devices and the difference in data distribution will lead to computing and communication overhead, resulting in low efficiency of global model aggregation and large training time overhead. Therefore, how to optimize the aggregation and communication overhead of the global model in the transverse synchronous federated learning process to improve the overall training speed of federated learning is particularly important. The present application designs a set of transverse synchronous federated learning aggregation optimization method from local model training to global model aggregation acceleration. Through the difference matrix and depth gradient compression algorithm to compress the data of participants, a scheme based on D-S evidence theory (Dempster-Shafer evidence theory, D-S) is proposed to optimize the participants, and a dynamic delay update scheme based on divergence threshold and information entropy is designed to reduce redundant communication.

[0047] First, the necessary preliminary knowledge required to understand the present application is introduced:

[0048] 1. Transverse federated learning

[0049] Transverse federated learning is suitable for the case where the data characteristics of participants overlap more. The federated learning framework studied in the present application is mainly the federated average algorithm FedAvg, which is mainly suitable for scenarios where sample data characteristics are similar, but due to the difference between participants, the sample space overlap is less.

[0050] Suppose M and T are positive integers, representing the maximum number of participating training clients and the maximum number of iterations, respectively. Let x and y represent the data sample input and prediction result output, respectively, and i represent the client number, i∈[1,M]. In the federated average algorithm, the objective function F(w) is the loss function loss(x i ; y i between the prediction output y i and the real sample x i), the objective function can be represented as F(w) = loss(x i ,y i ; w), the loss(.) function is used to describe the degree of difference between the prediction and the actual data, F(.) is used to describe the objective function to be optimized in the horizontal federated learning process, and the symbol is the gradient, and η is the learning rate, to describe the direction of the minimum descent of the objective function F(w), and η is the step size taken in gradient descent, and ∑(.) is the accumulation operation.

[0051] In the process of training a comprehensive model by M clients with the same data structure, the following 4 steps are generally taken:

[0052] (1) Client Party i performs multiple gradient descent algorithms (Stochastic Gradient Descent, SGD) locally in the tth round: wherein, is the local model of the t-1th round, η is the learning rate of SGD, is the gradient of the objective function, and the symbol “←” represents model updating. The gradient information is hidden using security technology, and the hidden result is sent to the server;

[0053] (2) After the server receives the local models sent by each participant, the gradient is aggregated by a secure aggregation operation. In FedAVG, the server performs an average aggregation operation (aggregates the models sent by the participants, calculates the number of participants, and obtains the t+1th round global model w t+1 after averaging), and the specific algorithm formula is

[0054] (3) The server sends the aggregated t+1th round global model w t+1 to each participant after encryption;

[0055] (4) Each participant Party i updates the local model t+1 based on the t+1th round global model w sent by the server, and repeats the above steps.

[0056] 2. Synchronous update

[0057] The current research on federated learning is mostly based on a synchronous update strategy. A synchronous federated learning system usually consists of a global server and a number of participant nodes. Each participant trains a local model using local data and sends the local model to the server. The server aggregates all the local models to obtain or update the global model. In a synchronous federated learning system, it is assumed that there are M participants who have data sets with the same data structure. At the beginning of global training, the global server determines a unified prediction task target and a prediction model, and determines the parameter structure of the model, and sends it to each participant for training.

[0058] The symbol a e [0, 1] represents the selected client ratio. The training process of synchronous federated learning can be divided into four stages. After the initialization of the global model and the local model, the global server sends the global model of the current round to the local model. For the client (Party), the following stages are executed in the subsequent loop:

[0059] (1) The client receives the global model from the server. If the model is in ciphertext form, it is decrypted and used.

[0060] (2) The client uses the received global model to replace the local model, and uses the local data set to start local model iteration on the local model until the model converges or reaches the maximum iteration round.

[0061] (3) After the local model iteration is completed, the client sends the local model of the last round of local iteration to the global server. This stage encrypts or perturbs the model according to privacy requirements.

[0062] (4) The server randomly selects a part of the M participants as the selected client aM to participate in this global aggregation. After the server receives all the local models of the selected clients, it starts global model aggregation and distributes the aggregated global model to all clients.

[0063] 3. Deep gradient compression (Gradient Compression, DGC)

[0064] DGC solves the communication bandwidth problem by sending important gradients to compress the remaining gradients. In order to ensure the accuracy of the uploaded local model, DGC specifies that only gradients greater than a threshold th will be transmitted. In order to avoid information loss, gradients less than the threshold will be accumulated locally until they become large enough to be transmitted. Therefore, in the DGC process, although large gradients are immediately sent and small gradients are temporarily accumulated, small gradients will eventually become large with round accumulation and be sent to the server. The symbol th represents the gradient threshold, which is an arbitrary constant.

[0065] 4. DS Evidence Theory

[0066] Dempster-Shafer (DS) evidence theory is a method for fusion of reasoning based on uncertain information. Let θ be the Frame of Discernment (FoD) of the evidence theory, defined as θ = {H1, H2, ..., H...}. N}, where N is the number of hypotheses in the recognition system, H is a set of each hypothesis in the recognition system, and all decision schemes made by the system are a subset of the power set of the recognition framework θ.

[0067] The Basic Probability Assignment (BPA) algorithm is a mapping function m:2 under the recognition frame θ. θ →[0,1], satisfying the constraint conditions. Let φ be the empty set, and m(.) be the basic probability assignment function. Let character A be one or more propositions in the recognition frame θ, then m(A) represents the degree of support of the evidence for proposition A.

[0068] Proposition A has a belief function (Bel), which describes the degree of trust in the event that proposition A is true. Under the recognition framework θ, the belief function of proposition A is defined as follows: Among them, propositions A and B are both recognition frames 2. θ In the proposition, m is the basic probability assignment function on the recognition frame θ.

[0069] The likelihood function (PI) under the framework θ is defined as follows: the likelihood function of proposition A is defined as... A confidence interval [Bel(A),PI(A)] is obtained by calculating the confidence function Bel(A) and the likelihood function PI(A) for a certain proposition A in the recognition framework, which represents the degree of confirmation of proposition A.

[0070] Let m1 and m2 be two independent basic probability assignment functions defined on the recognition frame θ, and let A, B, and C be propositions under the recognition frame θ respectively: The Dempster-Shafer composition formula rule can then be used to fuse the basic probabilities of propositions B and C, obtain the fused result, and make a decision. Where, k = ∑ B∩C≠A m1(B)m2(C), k∈[0,1] is the evidence conflict factor.

[0071] 5. Pearson correlation coefficient

[0072] remember and Let represent the mean of the sample data, and r be the Pearson correlation coefficient, which is usually used to measure the correlation between samples. The overall Pearson correlation coefficient r between different samples is defined as the sample mean. and The product of covariance and standard deviation

[0073] 6. Information Entropy

[0074] Entropy describes the uncertainty of an event, while information entropy measures the uncertainty of information. The more ordered a system is, the lower its information entropy; conversely, the more chaotic the system, the higher its information entropy. Let character X be the information, and {x1, x2, ..., x...} n Let} be the sequence content of information X, and let H(X) represent information X = {x1, x2, ..., xn}. n}of: Where p(x) i ) represents x in the information i The probability of occurrence; log() is the logarithmic operation.

[0075] 7. KL divergence

[0076] The KL divergence is defined based on entropy. It measures the degree of difference between sample distributions. Let p and q represent two random variables, with probability distributions p(x) and q(x) respectively, and let D be the symbol. KL (p||q) represents the entropy of random variable p with respect to q. The formula for the KL divergence can then be expressed as:

[0077] like Figure 1 As shown, this invention proposes an improved horizontal synchronous federated learning acceleration scheme. The bolded parts represent the modifications made to the traditional federated averaging algorithm. These modifications are modular and can be adapted to suit specific needs in real-world scenarios. This acceleration scheme can be broken down into three main modules: reducing the computational load of training local models on the client side through algorithm optimization, reducing the amount of data sent by the client through data compression, and improving the aggregation efficiency of the server model through data filtering. This achieves accelerated horizontal synchronous client-server federated learning.

[0078] In the present application, the symbol w represents the model trained in the federated learning process, D represents the local data set of the client, i represents the serial number of the client participating in the training, and t represents the number of training rounds of federated learning. Let M and T belong to positive integers, respectively representing the maximum number of clients participating in training and the maximum number of iterations, wherein i ∈ [1, M] and t ∈ [0, T]. Therefore, the local model of the ith client participating in the tth round of training can be written as The global model aggregated by the server can be written as: w t , and the local data set of the ith client is D i . The symbol γ represents the contribution value of the local model of the client to the global model, Δ represents the divergence used to describe the difference between the models, and w* is the reference model. Therefore, the contribution value of the client i is γ i , and Δ i represents the divergence value between the client Party i and the reference model w*.

[0079] Through the data compression strategy, the difference matrix is used, and the client only sends the difference between the local model and the global model. Then, by using the deep gradient compression (DGC) idea, a compression threshold is selected. Greater than the threshold value can be sent to the server side, and less than the threshold value is accumulated to be greater than the threshold value and then sent to the client.

[0080] When the difference between the local model and the global model is small, the communication between the client and the server is unnecessary; on the contrary, when the difference between the local model and the global model is large, more communication overhead should be spent to update the model. In view of the above problem, a dynamic update acceleration strategy based on the divergence threshold is proposed, which uses the divergence threshold Δ as a dynamic measurement standard to determine whether the client needs to communicate with the server to send the local model.

[0081] The client Party i calculates the divergence threshold Δ i : In the local model training stage, the client calculates the divergence threshold of the local model to determine whether the update of the local model is sufficient. Until the update is sufficient, the client sends the local model to the server, otherwise the local training is continued. When the client i completes a round of local iteration, the iteration round n of the local model is recorded by the counter, and the divergence threshold of the client i is calculated by , wherein Δ i represents the divergence value of the local model of the ith client, represents the local model, w* represents the reference model, and γ i represents the contribution value of the client. When the participant completes a batch of local model iterations and the divergence threshold satisfies the condition: , the client can send the local model The server. The divergence threshold based on the contribution value is adjustable compared with the traditional fixed threshold standard.

[0082] In the transverse synchronization federated learning process, the local model trained by different clients using the local data set will affect the global model, therefore, selecting high-quality clients has a positive significance for the aggregation of the global model, and for such a case, the application provides a method for giving the local model weight γ based on D-S evidence theory and combining the performance of the client in the past several rounds of training process i In the aggregation stage, the server aggregates the local model according to the weight and the proportion of the data set of the client, so as to reduce the influence of poor participants (malicious participants or normal participants causing the degradation of the global model) on the precision and convergence speed of the global model. In the server aggregation stage of the local model, the application provides Where |·| calculates the number of data elements in the set, is the proportion of the data set of the client i in the total data set.

[0083] Based on the above preliminary knowledge, the execution process of the application is as follows:

[0084] 1) The client accepts the initial global model sent by the server, and the contribution value of the client.

[0085] (1.1) The client receives the global model: in the initial training stage, the global model w0 (0th round, t=0), at this time the server will randomly select a local model as the global model and start training, assuming that in the tth round of training of federated learning, the server will issue the global model w t-1 as the initial model of the tth round to M participants Party i, where t [0,T], T represents the maximum number of model training rounds, M represents the number of clients participating in training, i [1,M] is the index of the ith participant, and w represents the global or local training model.

[0086] (1.2) Denote γ i as the contribution value of the client i to the global model, and the client i receives the contribution value γ i from the server. i The contribution value γ of the client is calculated according to the D-S evidence theory formula, which is explained by (4.1).

[0087] 2) The client trains the local model using the local data set, calculates the divergence threshold, and prepares to send the trained local model.

[0088] (2.1) Client uses local dataset for local model training: the i-th client (Party i) accepts the global model w t-1 , uses local dataset for multiple gradient descent (SGD), train local model Let w* be the reference model, at the beginning of the training, the server will send the contribution value γ i of the global model with higher aggregation, weighted average operation as the reference model w*, in the actual training, the client Party i will calculate the difference between the reference model and the local model, which is used to measure whether the magnitude of the local model of the client changes to the aggregation standard. When the global model is in the first iteration, due to the lack of sufficient reference, the server will randomly select one from all participants as the reference model w*, that is, i∈[1,M]. While in the subsequent training, the reference model will be selected according to the contribution value γ i of the client Party i. The server will average aggregate the local model with higher contribution value, and take the result as the reference model of the new round, that is, Where γ i , γ j respectively represent the contribution value of the i, j-th client, for averaging.

[0089] (2.2) Client i calculates divergence threshold Δ i : In the local model training phase, the client will calculate the divergence threshold of the local model to judge whether the local model is updated enough. When all the updates are sufficient, the client will send the local model to the server for further local training. When the client Party i completes a round of local iteration, the iteration round n of the local model is recorded by the counter, and the divergence threshold of the client Party i is calculated by , where Δ i represents the divergence value of the i-th client local model, represents the local model, w represents the reference model, and γ i represents the contribution value of the client. When the participant completes a batch of local model iterations, and the divergence threshold satisfies the condition: , the client can send the local model to the server.

[0090] 3) Client uses difference matrix and gradient compression algorithm to optimize and send local model, server stores the received local model in buffer and calculates information entropy.

[0091] (3.1) Client calculates difference matrix: when the difference of single round model update of the client is small, that is, Even as ε approaches 0, the client continues to send locally trained partial models to the server. This would waste communication resources. Therefore, at this stage, the client can send only the difference between the local model and the global model to reduce the amount of data sent. That is, the client sends the difference matrix between the global model and the local model. To the server. Assume that there are M clients participating in each round of training, i∈[1,M] (M is the maximum number of clients participating in training).

[0092] (3.2) Deep gradient compression algorithm is used, and the calculation results are sent to the server: Through the difference matrix, it can be found that when the client sends the local model to the server, there are some values ​​in the matrix that change little. These values ​​that change little may have a negligible impact on the server model in this round of update, but sending a small number still requires network bandwidth. Therefore, a deep gradient compression strategy is introduced. A compression threshold constant th is selected. During the training process, the intermediate values ​​with small values ​​are not sent first. Instead, they are saved and accumulated with the intermediate values ​​in the same position of the matrix in the next round. When the intermediate value is large enough (i.e., the intermediate value > th), the intermediate value is sent to the server.

[0093] Figure 2 An intuitive expression for a deep gradient approach is given, where G denotes the gradient matrix of the local client model, and t is its index. Then G... t Let represent the gradient matrix of the local model of the client in round t. At the beginning of round t of federated learning, a compression threshold of th = 7 is selected. After client Party i completes local training, the local model is not directly compressed. Instead of sending it to the server, a deep gradient compression algorithm is executed. The algorithm's implementation details are as follows: First, the gradient matrix G from the client is... t Comparison with threshold th (G) t Represents the client-side local model The gradient matrix G t ), the gradient matrix G t All positions in the matrix greater than the threshold th are set to 1, and the remaining positions are set to 0, resulting in a mask matrix Mask. The client then sets the gradient matrix G... t The gradient is multiplied by the mask matrix Mask and the result, representing all gradients greater than the threshold thr, is sent to the server. After sending, the participants invert the mask matrix Mask and multiply it by the gradient matrix G. t Dot product yields the gradient matrix G that was not sent in this round. t And retain it until the next training round's sending phase, where G t ′ represents the gradient matrix that was not sent to the server. The mask matrix has the same dimension as the gradient matrix, and its elements are only 0 and 1, which serves as a filtering threshold.

[0094] When the next round (t+1th round) sending stage starts, the gradient matrix G t ′ of the current round (tth round) which is not sent t+1 is added to the gradient matrix G t+1 of the t+1th round, and the addition result is taken as the gradient matrix G t+1 of the t+1th round, that is, G t+1 ← G t + G i ′, the symbol "←" represents gradient matrix operation, and then the deep compression algorithm is executed. By fine-tuning the local model sent by the client through the difference matrix and the deep gradient compression algorithm, the redundancy caused by invalid update in federated learning can be avoided, and the communication cost of federated learning is reduced.

[0095] 4) The server calculates the contribution value of the last round of clients based on the D-S evidence theory, calculates the information entropy, and finds the local model that meets the information entropy condition from the buffer, and performs a weighted aggregation operation based on the contribution value to obtain a new global model.

[0096] (4.1) The server calculates the contribution of the last client based on the D-S evidence theory: at the beginning of federated learning and at each new client joining the training stage (for example, the tth round). The newly added client i will be assigned an average contribution value γ i . In the server aggregation stage, the server obtains the relevant parameters of each client i, including the proportion of the data set D i of the client Party i in the whole data set D: D i / D, the contribution value γ i of the last round of clients i, the KL divergence of the client model, and the total number of rounds τ that the client has participated in training so far. Wherein γ i (0≤γ i <1) is the contribution value score of the client Party i, which is given by the D-S evidence theory algorithm for the historical participation rounds τ, the data set proportion D i / D, and the KL-divergence fusion, τ represents the historical rounds of the client i participating in training, τ∈[0,T].

[0097] The contribution value γ i of the client is calculated according to the D-S evidence theory formula, considering that data heterogeneity may lead to Matthew Effect in the contribution value of the client in the federated learning aggregation process, the invention adopts an evidence theory algorithm based on Pearson correlation coefficient to buffer the possibility of inaccurate fusion results when there are propositions with large conflicts. Among them, the Matthew Effect refers to the polarization phenomenon when solving the client Party i using the D-S evidence theory.

[0098] where μ denotes the expectation, and γ i is the subscript of γ denotes the contribution value of client i i , and σ denotes the variance, and γ i is the subscript of γ denotes the contribution value of client i i .

[0099] First, the contribution value correlation matrix S of different clients is calculated based on section (4-1).

[0100]

[0101] where S ij denotes the correlation of client i and client i contribution value, which is calculated as follows:

[0102]

[0103] where cov(γ i ,γ j ) denotes the covariance of client i and client j contribution value, where E(.) is the mathematical expectation. It should be noted that when S ij = 0, let S ij = 0.001, which is to overcome the 0 confidence conflict of evidence theory. Through equation (4-3), the credibility cred(γ i ) of client i is calculated:

[0104]

[0105] where cred denotes the credibility of the client, and the modified basic probability distribution BPA is calculated, defined as where X represents the parameters of the client: the historical training rounds τ participated by the client, the proportion D i / D of client i and the whole client data set, KL-divergence, etc.

[0106] The parameters of the client are fused according to equations (4-4) and (4-5).

[0107]

[0108]

[0109] The fused result γ i is used as the aggregation weight of client i.

[0110] (4.2) Server calculates information entropy: the server calculates the information entropy according to the condition of the tth round of the client i The local model with a large enough entropy value is proposed, and the global weighted aggregation operation is performed.

[0111] (4.3) Global model aggregation: after the server completes the contribution value calculation of all participants and the information entropy calculation, according to formula (4-6), the global model w t of the t+1th round is aggregated according to the weight of each client.

[0112]

[0113] 5) The server sends the contribution value of the last round and the new global model to the client, the client receives the global model sent by the server, and performs the next round of training.

[0114] The server obtains the t+1th global model w t , and then w t and the client contribution value γ i are sent to the client Party i participating in the training, the client receives the global model w t , and performs a new round of training.

Claims

1. A method for federated learning aggregation acceleration with cross-synchronization, characterized in that, The steps are as follows: Step 1: The client receives the global model sent by the server and the client contribution value; Step 2: The client trains the local model using the local data set and calculates the divergence threshold, and prepares to send the trained local model; wherein, during the local model training stage, the client calculates the divergence threshold of the local model to determine whether the local model update is sufficient, and when both updates are sufficient, the client sends the local model to the server for further local training; Step 3: The client optimizes the local model using the difference matrix and gradient compression algorithm, and sends the optimized local model to the server, and the server stores all the received models in the buffer and calculates the information entropy; wherein, the method for optimizing the local model is as follows: (1) The client calculates the difference matrix, assuming that there are M clients participating in training in each round, i∈[1, M], M is the maximum number of clients participating in training, when the difference between the local model and the global model is small, that is ε is an arbitrary constant approaching 0, and the client still sends the local model trained to the server Waste of communication resources, let the client only send the difference between the local model and the global model to reduce the amount of data sent to the server, that is, the client sends the difference matrix between the global model and the local model to the server ​ (2) Through the deep gradient compression algorithm, and the calculation result is sent to the server end, a deep gradient compression strategy is introduced, a compression threshold constant th is selected, and during the training process, the intermediate value with small value is not sent, but is saved and accumulated with the intermediate value of the same position in the next round of matrix, when the intermediate value is large enough, that is, the intermediate value> th, the intermediate value is sent to the server; Step 4: The server calculates the contribution value of the last round of clients and saves the information entropy, the server takes out all the local models from the buffer, and obtains a new global model based on the contribution value of the local model weighted aggregation, the method is as follows: (1) In the global model aggregation stage, when the server completes the contribution value and information entropy calculation of all participants, the contribution degree of each client and its data set proportion are taken as weights to perform the t+1th round of global model w t Aggregation, the formula is as follows: where D i / D represents the proportion of client i to the overall client dataset (2) Server gets t+1 global model w t+1 After that, w t and client contribution value γ i are delivered to the client participating in training, and the client receives the global model w t+1 and conducts a new round of training; Step 5: The server sends the contribution value of the last round and the new global model to the client, and the client receives the global model sent by the server for the next round of training.

2. The federated learning aggregation acceleration method of claim 1, wherein, The step 1, the global model is aggregated by the local models of each client, the process is: the client trains the local model based on the local data set, the local model is updated based on the deep gradient compression algorithm and the divergence threshold, and then uploaded to the server buffer, and then the local model uploaded to the server buffer is aggregated and weighted based on the contribution value of each client to obtain a new global model; the contribution value of the client is calculated based on the D-S evidence theory, which represents the contribution degree of the client to the global model in the last round, and the clients participating in the training are assigned an average contribution value at the beginning of federated learning; In the server aggregation stage, the server obtains the relevant parameters of each client, including the ratio of the client data set to the whole data set, the contribution value of the last round of clients, the divergence between the local model trained by the client and the global model, and the number of training rounds participated by the client so far.

3. The federated learning aggregation acceleration method of claim 1, wherein, The step 2, different clients have different local data sets with different data types and sizes, and do not share data with each other.

4. The federated learning aggregation acceleration method of claim 1, wherein, The step 2, the divergence threshold is calculated as follows: Whenever the ith client Partyi participating in federated learning training completes a round of local iteration, the iteration round n of the local model is recorded by a counter, and the local model is sent to the server by the client Partyi only when the client Partyi finishes a batch of local model iterations and the divergence threshold of the client Partyi satisfies the condition: When the divergence threshold is calculated , the client Partyi sends the local model to the server. where n represents the client local training round, the character Δ represents the divergence threshold of the model, and its superscript is i, Δ i represents the divergence value of the i-th client local model, w represents the federated learning model, represents the local model trained by the i-th client in the t-th round, w * represents the reference model, γ i represents the contribution value of the client Partyi, describes the distance between the local model and the reference model.

5. The federated learning aggregation acceleration method of claim 4, wherein, In the horizontal synchronous federated learning, the global round and the local round of training are set to fixed values, the global round represents the number of times the server updates the global model, and the local round represents the number of times the client updates the local model based on the local data set and the global model; All clients participating in the training satisfy the divergence threshold condition And after the local training round reaches the set fixed value of rounds, the local model can be uploaded to the server buffer, completing the current round of local model training.

6. The federated learning aggregation acceleration method of claim 1, wherein, The step 3, the information entropy is calculated as follows: The server calculates information entropy according to the condition of the tth round of the client Partyi The local model with a large enough entropy value is proposed, and a global weighted aggregation operation is performed, wherein H(U) represents the information entropy of the local model uploaded by the client Partyi, E(.) is a mathematical expectation, and log(.) is a logarithm operation.

Images