An optimization method for federated learning system

Abstract

The present invention discloses an optimization method for a federated learning system, which belongs to the field of large-scale distributed edge intelligent learning, specifically: a server creates a global model and initializes it and sends it to all user terminals; each user terminal uses its own privacy data set for training to obtain a personalized local model, and then enters a cycle of federated learning, and each cycle is as follows: the server clusters the user terminals, selects an uplink transmission user terminal set based on the contribution of the user terminal, and each user terminal in the set compresses the local model and uploads it; the server aggregates and aligns the local model using a federated average method and an unlabeled data set to update the global model and compress it and send it to a downlink transmission user terminal set selected according to user needs; each user terminal uses the privacy data set to train again as a new personalized model, and the cycle is repeated until both the local model and the global model converge. Therefore, the present invention improves communication efficiency and realizes efficient training of federated learning.

Description

Technical Field

[0001] The present invention belongs to the field of large-scale distributed edge intelligent learning, and specifically relates to an optimization method for a federated learning system. Background Art

[0002] Machine learning can make full use of data for model training and has been widely used in fields such as computer vision, natural language processing, and communications, but it has data privacy issues. In comparison, federated learning does not require uploading or exchanging private data, which has attracted widespread attention from industry and academia and has become a hot spot in machine learning.

[0003] The computing power of edge devices in federated learning is limited, which makes it impossible to train large-scale models. In addition, unstable communication connections and limited communication resources cannot support large-scale model parameter transmission. Therefore, in order to improve communication efficiency, various model compression technologies have been widely used in federated learning.

[0004] Some researchers have conducted research on model compression technology, such as introducing ternary quantization in the model training and inference stages, and proposing a ternary federated learning protocol to reduce communication loss; using pruning to compress the model to reduce learning delays caused by limited local computing power and communication bandwidth; and proposing a distributed hierarchical tensor deep computing model and an update method for the back-propagation algorithm to improve the training efficiency of edge nodes while reducing occupied bandwidth.

[0005] However, the above work is limited to the local user end and the server using the same compression model, and does not consider the model parameter differences caused by data differences, which leads to the optimal decomposition differences. In addition, most existing compression schemes take compression at the user end, and when the server merges, the aggregated model is sent down; this scheme will cause the model to be still large when it is sent down, resulting in a large bandwidth for each user.

[0006] In order to further improve communication efficiency, user terminal selection is also one of the feasible solutions. Existing literature defines the user terminal connection probability by considering the gradient during model update and the parameter transmission distance in federated learning. The greater the user terminal connection probability, the greater the probability that the user terminal will be selected to participate in federated learning training. In addition, some literature selects user terminals by setting thresholds, such as by setting an adaptive loss threshold to achieve the purpose of dynamically adjusting the number of selected user terminals; user terminal selection is achieved by discarding user terminals with a value greater than the convergence rate threshold to improve learning performance.

[0007] However, the above works do not consider the individual needs of downlink users for the model, which leads to a waste of downlink resources and fails to meet the needs of users for rapid model updates. Summary of the invention

[0008] In response to the above problems, the present invention provides an optimization method for a federated learning system, which improves communication efficiency and realizes efficient training of federated learning while ensuring the accuracy of the federated learning model and the personalized requirements of the local model on the user side.

[0009] The optimization method of the federated learning system has the following specific steps:

[0010] Step 1: The server creates a machine learning global model S, randomly initializes the parameters of the global model S, and sends it to all user terminals;

[0011] Step 2: After receiving the model, each user terminal uses its own private data set to train it, obtains its own personalized local model and corresponding parameters, and compresses the local model using tensor decomposition to test its accuracy for the preferred data set and all types of data sets;

[0012] The private data set of each client is obtained by Dirichlet distribution simulation;

[0013] The accuracy calculation formula for the kth client on the test data set is:

[0014]

[0015] Where B is the total number of batches, is the number of samples in each batch, The number of correct labels predicted for each batch.

[0016] Step 3: The server clusters the user terminals using Bi-kmeans according to the private data distribution of each user terminal, calculates the imbalance degree of each user terminal in each cluster, selects the user terminal with the lowest imbalance degree, and puts it into the initial user terminal set A to be performed uplink transmission. Considering the accuracy of all types of the remaining user terminals, the remaining user terminals are put into the set A in descending order of accuracy until there are no remaining communication resources.

[0017] The imbalance calculation formula for the kth client is:

[0018]

[0019] in, N is the proportion of the data volume of the jth type of data contained in the kth client to the total data volume of the client, class is the total number of data types;

[0020] Step 4: Each user terminal in the user terminal set A that needs to perform uplink transmission uploads its own personalized local model and corresponding parameters, as well as the test accuracy of the preferred data set and the full-type data set to the server;

[0021] Step 5: The server puts all the received personalized local models into set C, and divides them into the same model set C1 and the different model set C2 according to the model structure. It uses the federated average method and the unlabeled dataset to aggregate and align them to obtain the updated global model S, and compresses the global model S using tensor decomposition to obtain the compressed global model S1.

[0022] For the same model set C1, the server aggregates all models using the federated average method, that is, the corresponding parameters of each model are averaged and aggregated according to their respective weights to obtain the parameter values ​​of the aggregated model;

[0023] The weight refers to the proportion of the total amount of data contained in each user terminal; at the rth iteration, the formulas for each parameter of the aggregation model are:

[0024]

[0025] in is the data size contained in the kth client, is the amount of data of the jth type contained in client k, is the total data size of all users;

[0026] For the difference model set C2, the server uses the unlabeled data {x1,x2,…,x n} for alignment; Specifically, the unlabeled data is input into each model contained in the difference model set, and the label set of each model for the unlabeled data set at the rth iteration is obtained. where c∈C2, and It is included in the tag set Y;

[0027] Similarly, the unlabeled data {x1,x2,…,x n} as the input of the aggregation model, so as to obtain the label of the data set for the aggregation model at the rth iteration and will Include the tag set Y;

[0028] Finally, entropy is used to determine the unlabeled data {x1,x2,…,x n The final tag set corresponding to} Forming an aligned training set The server uses the aligned training set to train the global model S, and compresses the global model S using tensor decomposition to obtain a compressed global model S1.

[0029] Step 6: The server determines the user terminal requirements according to the client private data distribution and test accuracy, selects the user terminal set B to participate in downlink transmission, and sends the compressed global model S1 to the user terminal set B;

[0030] Step 7: Each user terminal in the user terminal set B uses the received global model S1 as a local model, trains it with its own private data set, obtains an updated personalized local model and corresponding parameters, compresses the local model using tensor decomposition, and uploads the model compression rate and the test accuracy of the preferred data set and all types of data sets;

[0031] Step 8: The server selects a user terminal to be placed in the uplink transmission user terminal set A1 of the next cycle based on the clustering result of the user terminal and the contribution of each user terminal; returns to step 4 to enter the next cycle of federated learning until both the local model and the global model converge.

[0032] Specifically, first, in the clustering result of Bi-kmeans on the user end, the user end with the highest test accuracy for all types of data sets is selected from each cluster and put into the user end set A1 for uplink transmission;

[0033] When there are at least two clients with the highest all-type test accuracy in a cluster, the client with the lowest data imbalance is selected and put into set A1;

[0034] At the same time, consider the contribution of user k in the current rth iteration

[0035] The contribution of the user end is affected by the precision increment Δθ and the compression rate ρ; the precision increment is divided into two parts, namely the precision increment for the preferred data set And precision increments for all types of data sets

[0036] The uplink transmission user terminals A1 of the next cycle are placed in descending order of contribution until there are no remaining communication resources.

[0037] The advantages of the present invention are:

[0038] 1) An optimization method for a federated learning system. The present invention realizes efficient training of federated learning while ensuring the accuracy of the federated learning model and local personalized needs;

[0039] 2) An optimization method for a federated learning system that achieves compression on both the client and server sides, as well as alignment of the difference models after compression, saving computing and communication resources;

[0040] 3) An optimization method for a federated learning system that takes into account the client's contribution and needs, and selectively uploads and distributes models, thereby improving resource utilization while improving the efficiency of federated learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is the principle of an optimization method for a federated learning system of the present invention.

[0042] Figure 2 It is a flowchart of an optimization method for a federated learning system of the present invention.

[0043] Figure 3 It is a flow chart of the loop process in the optimization method of the federated learning system of the present invention.

[0044] Figure 4 It is a functional module diagram of an embodiment of a federated learning system device of the present invention. DETAILED DESCRIPTION

[0045] The present invention will be further described below in conjunction with the embodiments and the accompanying drawings. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications to various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0046] The present invention proposes an optimization method for a federated learning system, such as Figure 1 As shown in the figure, it mainly consists of two processes, namely the initialization process and the loop process. Specifically, the server creates a global model and initializes its parameters, and sends it to all user terminals; the server clusters each user terminal according to its data distribution for user terminal selection. Each user terminal uses the received initialized global model as its basic model, and trains the basic model using its own privacy data set to obtain a personalized local model and corresponding parameters. Then it enters the federated learning loop process, and each cycle includes the following stages: the server selects the uplink transmission user terminal set based on the clustering results and the user terminal's contribution; before uploading the local model, the user terminal will compress the local model; then, each selected user terminal transmits each local model and corresponding parameters to the server; the server uses the federated averaging method and the unlabeled data set to aggregate and align the received local models to update the global model; before sending it down, the server will compress the updated global model. Subsequently, the server selects a set of downlink transmission users according to the user's demand for the updated global model; the server sends the compressed global model to the selected user; after receiving the global model, each user uses it as a basic model, trains it with a privacy data set, and updates it to a personalized model; this cycle ends and the next cycle starts until both the local model and the global model converge. Therefore, the present invention improves communication efficiency and realizes efficient training of federated learning while ensuring the accuracy of the federated learning model and local personalized needs.

[0047] The optimization method of the federated learning system is as follows: Figure 2As shown, the specific steps are as follows:

[0048] Step 1: The server creates a global machine learning model S, randomly initializes the neural network parameters of the global model S, and sends it to all user terminals;

[0049] Step 2: After receiving the model, each user terminal uses its own private data set to train it, obtains its own personalized local model and corresponding parameters, and compresses the local model using tensor decomposition to test its accuracy for the preferred data set and all types of data sets;

[0050] The private data set of each client is obtained by Dirichlet distribution simulation, and the Dirichlet distribution formula is:

[0051]

[0052] The Tucker decomposition based on variational Bayesian matrix decomposition is used here:

[0053]

[0054] in, is the input tensor, y h′,q′,e is the output tensor, is the core tensor, and is a factor matrix, and the ranks R3 and R4 are obtained by variational Bayesian matrix decomposition. For details, see the literature: Kim YD, Park E, Yoo S, et al. Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications[J]. Computer Science, 2015, 71(2): 576-584.

[0055] The accuracy calculation formula for the kth client on the test data set is:

[0056]

[0057] Where B is the total number of batches, is the number of samples in each batch, The number of correct labels predicted for each batch.

[0058] Step 3: The server uses Bi-kmeans to cluster the user terminals according to the private data distribution of each user terminal, calculates the imbalance degree of each user terminal in each cluster, selects the user terminal with the lowest imbalance degree from each cluster and puts it into the initial user terminal set A to be performed uplink transmission, considers the full type accuracy of the remaining user terminals, and puts them into the set A in descending order of accuracy until there are no remaining communication resources;

[0059] The imbalance calculation formula for the kth client is:

[0060]

[0061] in, is the proportion of data of type j contained in the k-th client, N class is the total number of data types;

[0062] Step 4: Each user terminal in the initial user terminal set A that needs to perform uplink transmission uploads its own personalized local model and corresponding parameters, as well as the test accuracy of the preferred data set and the full-type data set to the server;

[0063] Step 5: The server puts all the received personalized local models into set C, and divides them into the same model set C1 and the different model set C2 according to the model structure. It uses the federated average method and the unlabeled dataset to aggregate and align them to obtain the updated global model S, and compresses the global model S using tensor decomposition to obtain the compressed global model S1.

[0064] For the same model set C1, the server aggregates all models using the federated average method, that is, the corresponding parameters of each model are averaged and aggregated according to their respective weights to obtain the parameter values ​​of the aggregated model;

[0065] The weight refers to the proportion of the total amount of data contained in each user terminal; at the rth iteration, the formulas for each parameter of the aggregation model are:

[0066]

[0067] in is the data size contained in the kth client, D k j is the amount of data of the jth category contained in client k, is the total data size of all users;

[0068] For the difference model set C2, the server uses the unlabeled data {x1,x2,…,x n} for alignment; Specifically, the unlabeled data is input into each model contained in the difference model set, and the label set of each model for the unlabeled data set at the rth iteration is obtained. where c∈C2, and It is included in the tag set Y;

[0069] Similarly, the unlabeled data {x1,x2,…,x n} as the input of the aggregation model, so as to obtain the label of the data set for the aggregation model at the rth iteration and will Include the tag set Y;

[0070] Finally, entropy is used to determine the unlabeled data {x1,x2,…,x n The final tag set corresponding to} Forming an aligned training set The server uses the aligned training set to train the global model S, and compresses the global model S using tensor decomposition to obtain a compressed global model S1.

[0071] Step 6: The server determines the user terminal requirements according to the client private data distribution and test accuracy, selects the user terminal set B to participate in downlink transmission, and sends the compressed global model S1 to the user terminal set B;

[0072] Step 7: Each user terminal in the user terminal set B uses the received global model S1 as a local model, trains it with its own private data set, obtains an updated personalized local model and corresponding parameters, compresses the local model using tensor decomposition, and uploads the model compression rate and the test accuracy of the preferred data set and all types of data sets;

[0073] Model compression ratio: Among them, N w is the number of parameters of the model before compression, N w is the number of parameters of the compressed model; the test accuracy formula is similar to the above.

[0074] Step 8: The server selects a user terminal to be placed in the uplink transmission user terminal set A1 of the next cycle based on the clustering result of the user terminal and the contribution of each user terminal; returns to step 4 to enter the next cycle of federated learning until both the local model and the global model converge.

[0075] Specifically, first, in the clustering result of Bi-kmeans on the user end, the user end with the highest test accuracy for all types of data sets is selected from each cluster and put into the user end set A1 for uplink transmission;

[0076] When there are at least two clients with the highest all-type test accuracy in a cluster, the client with the lowest data imbalance is selected and put into set A1;

[0077] At the same time, consider the contribution of user k in the current rth iteration

[0078] The contribution of the user end is affected by the precision increment Δθ and the compression rate ρ; the precision increment is divided into two parts, namely the precision increment for the preferred data set And precision increments for all types of data sets

[0079] The uplink transmission user terminals A1 of the next cycle are placed in descending order of contribution until there are no remaining communication resources.

[0080] like Figure 3 As shown, when the server selects the user terminal set A1 for uplink transmission, it comprehensively considers the clustering result and the contribution of the user terminal; specifically, its influencing factors include: the imbalance degree related to the number of types contained in the data of each user terminal and the amount of each type of data The accuracy increment Δθ and the model compression ratio ρ.

[0081] The reason is that in the scenario where actual communication resources are limited, if some user terminals are randomly selected to participate in federated learning, the Non-IID characteristics of the user terminal data distribution will cause the trained global model to be unsatisfactory. The global model may perform well for some types of data sets, but perform poorly for all types of data sets, which is inconsistent with the purpose of federated learning.

[0082] In order to ensure the accuracy of the global model for all types of data sets, first, in the Bi-kmeans clustering results of the user terminals, the user terminals with the highest test accuracy for all types of data sets are selected from each cluster and placed in the candidate user set; when there are at least two user terminals with the highest test accuracy for all types of data sets in a cluster, the user terminal with the lowest data imbalance is selected and placed in the candidate user set; under the premise of meeting the bandwidth constraints, they are included in the uplink user terminal set A1 in turn.

[0083] Once the client is selected, its corresponding local model and corresponding model parameters will be transmitted to the server. Under the premise of limited uplink resources, in order to maximize the contribution of the uploaded client set to the global model update, it means transmitting as many local models with better performance as possible to the server for aggregation; therefore, when selecting the client set to be transmitted uplink based on the client's contribution, the compression rate of the current user model is is an important factor to be taken into consideration, N w is the number of parameters after model compression, N w is the original model parameter.

[0084] In addition, the user-side precision increment Δθ is also an important influencing factor, where the precision increment is divided into two parts: the precision increment for the preferred data set And precision increments for all types of data sets

[0085] Therefore, the contribution of the user end is affected by the precision increment Δθ and the compression rate ρ. The contribution of the user end k in the current rth iteration is

[0086] In each iteration, the convolutional layer of the local model is decomposed using Tucker decomposition based on variational Bayesian matrix decomposition.

[0087] After the selection of the uplink user terminal set A1 and the compression of the local user terminal model are completed, each user terminal selected in this round of cycle starts to transmit each local model to the server, and the transmission time is Among them, M kr is the local model parameter of client k at the rth iteration, Compression ratio, is the uplink transmission rate. q is the quantization bit.

[0088] After the server receives the models of all the clients in the set, it divides all the received local models into two sets, namely the same model set C1 and the difference model set C2.

[0089] For the same model set C1, the server aggregates the models using the federated average method, that is, the corresponding parameters of each model are aggregated according to the proportion of the total data contained in each user terminal, so as to obtain the aggregated model The parameter values ​​of

[0090] For the difference model set C2, the server uses the unlabeled data {x1,x2,…,x n} as alignment data for model alignment. Specifically, the unlabeled data is input into the models contained in the difference model set to obtain the labels of each model for the unlabeled data set. Among them, c∈C2 is put into the label set Y as a candidate label.

[0091] Similarly, the unlabeled data {x1,x2,…,x n} as an aggregation model Input, get the label of the aggregation model for the data set And included in the candidate label set Y.

[0092] The Non-IID characteristics of the data can easily cause the local model to misjudge unseen data and obtain incorrect labels, thereby affecting the training of the global model and resulting in lower accuracy of the global model.

[0093] Entropy is used to determine the final label corresponding to the unlabeled dataset to reduce the proportion of incorrect labels in the aligned dataset. Used for global model training of the server to obtain an updated global model.

[0094] In order to reduce the pressure of downlink transmission and improve the utilization of downlink resources, the server also uses Tucker decomposition based on variational Bayesian matrix decomposition to decompose the convolutional layer of the model before sending the updated global model.

[0095] When downlink communication resources are insufficient, the number of user terminals that can receive the updated global model is limited. Therefore, it is very important to consider the needs of the user terminals. The purpose of downlink user terminal selection is to improve the test accuracy of the local model for the preferred data set under the premise of limited communication resources. In addition, the data distribution, data volume and full-type data accuracy of the selected downlink client have a potential impact on the improvement of the accuracy of the next round of global models. Therefore, the user terminal's demand for the updated global model in the current round depends on the number of classes owned, the data size and the accuracy of the local model for all types and preferred types of data. Obviously, the degree of demand is inversely proportional to all relevant items. The higher the user terminal's demand for the global model, the greater the probability that it will be selected for priority delivery.

[0096] After the downlink client set B is selected, the server will send the compressed global model to each client. The transmission time is Among them, M 0r is the parameter of the global model at the rth iteration, is the global model compression ratio, is the downlink transmission rate. After receiving the data, each user terminal will use its own private data set to train and compress the received global model to update the personalized model and improve the accuracy of the local model for the preferred data set and all types of data sets.

[0097] This cycle ends and the next cycle starts until both the local model and the global model converge.

[0098] The optimization method of the federated learning system of the present invention specifically includes a center-edge double-side compression method and an uplink and downlink asymmetric user terminal selection method;

[0099] Further improvement of the center edge double-sided compression method:

[0100] In each round of iteration, before uploading the local model, each user terminal will compress the local model to reduce the pressure of uplink transmission;

[0101] In each round of iteration, before sending the global model, the server will compress the updated global model to reduce the pressure of downlink transmission;

[0102] Before uploading and sending the updated model, the updated model is decomposed using the tensor decomposition method;

[0103] Specifically, Tucker decomposition based on variational Bayesian matrix decomposition is used to decompose the convolutional layer of the model. The rank of each model during tensor decomposition is obtained by variational Bayesian matrix decomposition, which is related to each model parameter;

[0104] In each round of iteration, the client independently compresses the local model, and some clients will retrain and compress after receiving the global model compressed by the server, resulting in inconsistent model structures of each client in the same round of iteration. Therefore, the traditional method for model aggregation, that is, directly averaging model parameters, is no longer applicable.

[0105] After receiving all uploaded models, the server divides the models into two categories, namely, the same model set and the different model set, and aggregates and aligns them using the federated average method and unlabeled dataset to obtain an updated global model;

[0106] Further improvement of the uplink and downlink asymmetric user terminal selection method:

[0107] The uploading of local models and the sending of global models are no longer targeted at all user terminals or randomly selected user terminals, but are selectively uploaded and sent according to rules, and the rules for uploading and sending are inconsistent;

[0108] The purpose of uploading the user-side model is to quickly obtain a model with high accuracy for all types on the server side;

[0109] The purpose of the server sending the updated global model is, on the one hand, to improve the accuracy of the user-side local model for its preferred data set, and on the other hand, to compensate for the poor accuracy of the user-side local model of all types caused by the Non-IID characteristics of the data, which leads to the poor performance of the global model after server aggregation and alignment.

[0110] Specifically, in each round of federated learning iteration, the server will select the user set to upload and send based on the user clustering results, the user contribution, and the demand for the updated global model; specifically:

[0111] The server comprehensively considers the clustering results and the contribution of the user end when selecting the user end set to be transmitted uplink. Specifically, the influencing factors include: the imbalance related to the number of types and the amount of each type of data contained in each user end data, the precision increment, and the model compression rate. The higher the server's contribution to the global model, the greater the probability that it will be selected to upload the model.

[0112] When determining the set of downlink user clients, the server considers the user client's demand for the updated global model. Specifically, factors affecting the demand include the number of classes owned by the user client, the data size, and the accuracy of the local model for all types and preferred types of data. The higher the user client's demand for the global model, the greater the probability that it will be selected for priority delivery.

[0113] The present invention provides an optimization method for a federated learning system based on the following Figure 4 The apparatus of the federated learning system shown specifically includes:

[0114] Receiving module: used to receive uploaded and downloaded models;

[0115] Training module: used by the user to train the local model using private data to obtain a personalized local model;

[0116] Decomposition module: used to decompose the models of the client and the server;

[0117] Transmission module: used to transmit each model and corresponding parameters;

[0118] Aggregation module: used by the server to aggregate and align the received personalized local models.

Claims

1. A method for optimizing a federated learning system, characterized in that: The specific steps are as follows: First, the server creates a global machine learning model S, and randomly initializes the parameters of the global model S, and sends it to all user terminals; after receiving the sent model, each user terminal uses its own private data set to train it, obtains its own personalized local model and corresponding parameters, and compresses the local model using tensor decomposition to test its accuracy for the preferred data set and all types of data sets; Then, the server uses Bi-kmeans to cluster the user terminals according to the private data distribution of each user terminal, calculates the imbalance degree of each user terminal in each cluster, selects the user terminal with the lowest imbalance degree, and puts it into the initial user terminal set A to be performed uplink transmission. Considering the accuracy of all types of the remaining user terminals, the remaining user terminals are put into the set A in descending order of accuracy until there are no remaining communication resources. Next, each user terminal in the user terminal set A that needs to perform uplink transmission uploads its own personalized local model and corresponding parameters, as well as the test accuracy of the preferred data set and the full-type data set to the server; The server puts all the received personalized local models into set C, and divides them into the same model set C1 and the different model set C2 according to the model structure. It uses the federated average method and the unlabeled dataset to aggregate and align them to obtain the updated global model S, and compresses the global model S using tensor decomposition to obtain the compressed global model S1. The server determines the needs of the user terminal according to the distribution of the user terminal private data and the test accuracy, selects the user terminal set B to participate in the downlink transmission, and sends the compressed global model S1 to the user terminal set B. Each user terminal in the user terminal set B uses the received global model S1 as a local model, trains it with its own private data set, obtains an updated personalized local model and corresponding parameters, compresses the local model using tensor decomposition, and uploads the model compression rate and the test accuracy of the preferred data set and all types of data sets; Finally, the server again selects the uplink transmission user terminal set A1 for the next cycle based on the clustering results of the user terminal and the contribution of each user terminal; and returns to the next cycle of federated learning until both the local model and the global model converge.

2. The optimization method of a federated learning system according to claim 1, characterized in that: The private data sets of each user terminal are obtained by Dirichlet distribution simulation.

3. The optimization method of a federated learning system according to claim 1, characterized in that: The accuracy calculation formula is: Where B is the total number of batches, is the number of samples in each batch, The number of correct labels predicted for each batch.

4. The optimization method of a federated learning system according to claim 1, characterized in that: The imbalance calculation formula is: in, is the proportion of the data volume of the jth type of data contained in the kth user end to the total data volume of the user end, N class The total number of data types.

5. The optimization method of a federated learning system according to claim 1, characterized in that: For the same model set C1, the server aggregates all models using the federated average method, that is, averages and aggregates the corresponding parameters of each model according to their respective weights, thereby obtaining the parameter values ​​of the aggregated model; The weight refers to the proportion of the total amount of data contained in each user terminal; at the rth iteration, the formulas for each parameter of the aggregation model are: in is the data size contained in the kth client, is the amount of data of the jth type contained in user terminal k, is the total data size of all users; For the difference model set C2, the server uses the unlabeled data {x1,x2,…,x n } for alignment; Specifically, the unlabeled data is input into each model contained in the difference model set, and the label set of each model for the unlabeled data set at the rth iteration is obtained. where c∈C2, and Y c r It is included in the tag set Y; Similarly, the unlabeled data {x1,x2,…,x n } as the input of the aggregation model, so as to obtain the label of the data set for the aggregation model at the rth iteration and will Include the tag set Y; Finally, entropy is used to determine the unlabeled data {x1,x2,…,x n The final tag set corresponding to} Forming an aligned training set The server uses the aligned training set to train the global model S, and compresses the global model S using tensor decomposition to obtain a compressed global model S1.

6. The optimization method of a federated learning system according to claim 1, characterized in that: The model compression ratio: Among them, N w is the number of parameters of the model before compression, is the number of parameters of the compressed model.

7. The optimization method of a federated learning system according to claim 1, characterized in that: The server selects the uplink transmission user terminal set A1 to be put into the next cycle according to the clustering of the user terminals and the contribution of each user terminal; specifically: First, in the clustering results of the user terminals, the user terminal with the highest test accuracy for all types of data sets is selected in each cluster and put into the user terminal set A1 for uplink transmission; When there are at least two clients with the highest all-type test accuracy in a cluster, the client with the lowest data imbalance is selected and put into set A1; At the same time, consider the contribution of user k in the current rth iteration The contribution of the user end is affected by the precision increment Δθ and the compression rate ρ; the precision increment is divided into two parts, namely the precision increment for the preferred data set And precision increments for all types of data sets The uplink transmission user terminals A1 of the next cycle are placed in descending order of contribution until there are no remaining communication resources.

8. The method for optimizing a federated learning system according to claim 1, wherein: The compression of the updated model by tensor decomposition occurs on the server side and the user side respectively, that is, the bilateral compression of the center and the edge is achieved. Specifically, the convolutional layer of the model is decomposed by Tucker decomposition based on the variational Bayesian matrix. The rank of each model in tensor decomposition is obtained by variational Bayesian matrix decomposition, which is related to the parameters of each model.

9. The method for optimizing a federated learning system according to claim 1, wherein: In each round of federated learning iteration, asymmetric user selection is implemented, that is, the server selects the user set for uploading and downloading based on the user clustering results, the user contribution, and the demand for the updated global model; specifically: When selecting the set of users to be transmitted uplink, the server comprehensively considers the clustering results and the contribution of the users. Specifically, the influencing factors include: the imbalance related to the number of types and the amount of each type of data contained in each user's data, the precision increment, and the model compression rate; the higher the contribution of the server to the global model, the greater the probability that it will be selected to upload the model; When determining the downlink user client set, the server considers the user client's demand for the updated global model. Specifically, the factors affecting the demand include the number of classes owned by the user client, the data size, and the accuracy of the local model for all types and preferred types of data; the higher the user client's demand for the global model, the greater the probability that it will be selected for priority delivery.

Images