An efficient federated learning sparse training method based on parallel overparameterization

Abstract

The present invention discloses an efficient federated learning sparse training method based on parallel over-parameterization, which aims to reduce the training and communication overhead of federated learning at the same time. The method belongs to the intersection of federated learning and model sparse training. Specifically comprising the following steps: S1, sparse initialization, the server sparses the original dense network; S2, find the optimal sparse network structure, by grouping the clients to perform parallel parameter exploration to achieve parallel over-parameterization, and on this basis, dynamically optimize the global sparse network structure as the training progresses; S3, sparse training, using the optimal sparse network structure found in step S2, shared by all clients and continue training until convergence. In the federated learning scenario, when the VGG11 network is used for training on the CIFAR10 dataset, the present invention can be on par with the original model in terms of accuracy using only 5% of the parameters of the original model, and reduces 71.8% of the FLOPs computing overhead and 91.3% of the communication overhead.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to the intersection of a federated learning framework algorithm and sparse training of a deep learning model, and more particularly to an efficient federated learning sparse training method based on parallel over-parameterization. Background Art

[0002] It involves two major background technologies: federated learning and sparse training of deep learning models.

[0003] 1) Federated Learning

[0004] Today, hundreds of millions of edge devices are connected to the Internet, and the massive amounts of data generated can be used to train more powerful and accurate models. In order to fully tap the value of massive edge data, federated learning is seen as a solution with the greatest potential. Its biggest advantage is that it can combine data from multiple parties while protecting the privacy and security of client data. However, traditional federated learning frameworks require powerful computing power, communication bandwidth, and storage space, which undoubtedly limits the deployment of federated learning on resource-constrained edge devices.

[0005] Some works focus on controlling the aggregation frequency of federated learning to improve communication efficiency, such as adaptively adjusting the aggregation frequency according to the resource status of the device. However, these works cannot reduce the huge training overhead of federated learning, nor can they improve the inference speed after the model is deployed.

[0006] In addition, some studies have used adaptive model pruning to reduce both computational and communication overheads. The common feature of these works is that they start from an original dense model and gradually prune the original network until the expected sparsity as training progresses. Obviously, they still need to consume a lot of resources in the early and middle stages of training.

[0007] 2) Sparse training of deep learning models

[0008] Sparse training of deep learning models is an effective method to reduce computational overhead and has made great progress. The Lottery Ticket Hypothesis (LTH) verifies that training a sparse subnetwork (a winning lottery ticket network) from scratch can achieve the same test accuracy as the original dense model. Inspired by the Lottery Ticket Hypothesis, many studies have explored the feasibility of pruning neural networks to sparseness at initialization, and using the sparse network obtained at initialization to replace the original network for training. For example, SNIP judges the importance of connections based on the sensitivity of model parameters to the loss function (connection sensitivity); GraSP uses gradient signal preservation to select sparse networks; Synflow can alleviate the problem of severe accuracy loss in extreme sparsity. However, such methods inevitably bring about a certain degree of accuracy loss, because compared with the over-parameterized original model, the sparse network found by such methods has a worse fitting ability than the original dense network.

[0009] Dynamic sparse training (DST) is considered to be a practical sparse-to-sparse training method. Its key idea is to dynamically adjust and optimize the structure of the sparse network during training. Dynamic sparse training methods usually include two parts: (1) According to the weight of the model parameters, cut off the part with the smallest absolute value; (2) re-explore new parameters according to some rules (such as SET uses random exploration; RigL chooses to prioritize exploring new parameters with larger absolute values ​​of gradients; and SNFS explores new parameters based on momentum). The benefit of dynamic sparse training is that it can be regarded as over-parameterization on a time series, that is, during the training process, continuous parameter exploration is allowed to cover all the original parameter space on the time axis. However, the direct introduction of dynamic sparse training into federated learning does not work well, mainly because it simply optimizes the sparse network based on the absolute value of the weight and cannot adapt well to federated learning with complex data distribution. Summary of the invention

[0010] The purpose of the present invention is to provide an efficient federated learning sparse training method based on parallel over-parameterization, aiming to reduce the training and communication overhead of federated learning.

[0011] The specific technical solution for achieving the purpose of the present invention is:

[0012] An efficient federated learning sparse training method based on parallel over-parameterization, characterized in that the method comprises the following steps:

[0013] S1, sparse initialization, the server sparses the original dense network;

[0014] S2. Find the optimal sparse network structure by grouping clients to perform parallel parameter exploration to achieve parallel over-parameterization, and on this basis dynamically optimize the global sparse network structure as training progresses;

[0015] S3, sparse training, using the optimal sparse network structure found in step S2, shared by all clients and continued training until convergence; wherein:

[0016] The step S1 specifically includes: before the federated learning starts, the server performs sparseness on the original dense network according to the connection sensitivity of the model parameters; the sparsity S of the model is defined as the ratio of the number of pruned parameters to the total number of parameters, using a binary mask M∈{0,1} |θ| To indicate whether the parameters of the original network are retained, that is, to represent the structure of the sparse network; starting from a given original dense network θ, before training begins, the server uses a small batch of data to send it to the original network to calculate a forward propagation, and then calculates the connection sensitivity score g(θ) of each model parameter to the loss function L:

[0017]

[0018] Where ⊙ is the Hadamard inner product, and the k parameters with the highest absolute value of sensitivity score |g(θ)| are retained to form the initial global sparse network structure M with sparsity S G , where k = (1-S-(1-S)f)×|θ|, f is the ratio of the total number of parameters reserved for exploring new parameters to the total number of parameters with sparsity S;

[0019] The goal of step S2 is to obtain the optimal sparse network structure M, which specifically includes:

[0020] S2-1, client grouping: the server randomly divides all clients into Z groups;

[0021] S2-2, Parallel over-parameterization: Dynamically evolve and update the structure of the sparse network every △T communication rounds; based on the global sparse network structure M G , the server randomly explores new parameters of proportion f in the parameter space of the original dense network for each group j to maintain the total sparsity S unchanged, thereby forming a new sparse network structure M j , and then the server will M j It is sent to each client of group j; by merging the parameters explored and covered by the collaborative cooperation between the groups, the parameter coverage rate of a single parameter exploration is Z times that of the existing dynamic sparse training method. When the explored parameters cover all the parameters of the original model, another form of over-parameterization can be achieved from a spatial perspective - parallel over-parameterization, and a parallel over-parameterized model θ is obtained. POP :

[0022] θ POP =[θ⊙M1,θ⊙M2,…,θ⊙M z ] (2)

[0023] The operator [...] means to calculate each part separately and then combine them together;

[0024] S2-3, local training: In the local training of the client, the client i in group j applies the sparse network structure M j In your local dataset X i On the other hand, the local model is iteratively updated with a learning rate η:

[0025]

[0026] When the server and client communicate, they only need to send the retained parameters according to the mask. In addition, each bit of the mask represents whether the corresponding parameter is retained. When the original model uses 32-bit floating point numbers, the mask only occupies 1 / 32 of the additional communication overhead.

[0027] S2-4. Model aggregation: Parallel over-parameterization leads to the problem of model heterogeneity. In order to preserve the unique properties of different sparse network structures, each model is divided into two components: (1) the common part parameters θ Public , the parameters in this part correspond to the global sparse structure M G , shared by all clients, so all clients participate in the aggregation of public parameters; (2) private parameters θ Private , that is, the newly explored parameters, are different for each group, so only clients in the same group participate in the aggregation; suppose there are N clients participating in the training, of which n j clients belong to group j, then the new model after group j aggregation is:

[0028]

[0029] Where |D i | represents the amount of data from client i, and Represent the public and private parameters of client i respectively;

[0030] S2-5, parameter selection: every △T communication rounds, based on the current parallel over-parameterized model θ POP , according to formula (1) to calculate |g(θ POP )|, select the top-k most potential parameters to update the global sparse network structure M G , which is used as the starting point for the next round of parameter exploration until the set round Tend until.

[0031] Step S2-2 and step S2-5 in step S2 are performed once every ΔT communication rounds until the preset round T. end Until, step S2-3 and step S2-4 are performed in each communication round; the value of f starts from α and gradually increases within the preset T end The round decay is 0 to gradually determine the structure of the global sparse network: f = 0 means that the search for the optimal sparse network is complete. At this time, the final M G This is exactly the expected optimal sparse network structure M; the sparse network structure has been used throughout the entire training process to achieve the purpose of reducing both computational and communication overheads.

[0032] The beneficial effects of the present invention include:

[0033] 1) Compared with the traditional federated learning framework, the present invention adopts a sparse network in the entire training process, which greatly reduces the computational and communication overhead.

[0034] 2) Different from the existing dynamic sparse training, the parallel over-parameterization strategy designed in the present invention effectively improves the fitting ability of sparse networks in federated learning with complex data distribution, thereby maintaining the performance of sparse networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the present invention;

[0036] Figure 2 It is a schematic diagram of the framework of the present invention;

[0037] Figure 3 This is a schematic diagram of model aggregation in the parameter exploration phase;

[0038] Figure 4 An example diagram of experimental results of the present invention in reducing computational and communication overheads;

[0039] Figure 5 The following are example graphs of experimental results with different degrees of parallel over-parameterization. DETAILED DESCRIPTION

[0040] The present invention is further described in detail in conjunction with the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the field. For ordinary technicians in the field, other drawings and other implementation methods can be obtained based on these drawings without creative work.

[0041] The present invention provides an efficient federated learning sparse training method based on parallel over-parameterization, which specifically includes:

[0042] S1: Sparse the initial dense network on the server. For a model θ with a given sparsity S, use a binary mask M∈{0,1} |θ| To indicate whether the corresponding parameters are retained, it represents the structure of the sparse network. The server needs to use a small batch of data to feed the original network before training, and then calculate the sensitivity score g(θ) of each model parameter to the loss function L:

[0043]

[0044] Where ⊙ is the Hadamard inner product, and the k parameters corresponding to the highest absolute value of the sensitivity score |g(θ)| are retained to form the initial global sparse network structure M G , where k = (1-S-(1-S)f)×|θ|, f is the ratio of the total number of parameters reserved for exploring new parameters to the total number of parameters with a sparsity of S; obviously, without affecting the accuracy of the original model, a higher sparsity S means a lower overhead can be achieved.

[0045] S2: Find the optimal sparse network, group all clients to perform parallel parameter exploration, and merge the parameters explored and covered by the collaborative cooperation between groups, so that the parameter coverage of a single parameter exploration is Z times that of the existing dynamic sparse training method. When the explored parameters cover all the parameters of the original model, another form of over-parameterization can be achieved from a spatial perspective - parallel over-parameterization, giving the optimized sparse network structure a fitting ability that is not inferior to the original dense network.

[0046] S2-1: Client grouping. The server randomly divides a large number of clients into Z groups;

[0047] S2-2: Parallel over-parameterization. The structure of the sparse network is dynamically evolved and updated every △T communication rounds. The goal of this stage is to obtain the optimal sparse network structure M. Based on the global sparse network structure M G , the server randomly explores new parameters of proportion f in the parameter space of the original dense network for each group j to maintain the total sparsity S unchanged, thereby forming a new sparse network structure M j , and then the server will M j By merging all the parameters covered by the exploration, a parallel over-parameterized model θ can be obtained. POP :

[0048] θ POP =[θ⊙M1,θ⊙M2,…,θ⊙M z ] (2)

[0049] The operator [...] means to calculate each part independently and then combine all the parts together.

[0050] S2-3: Local training. In the local training of the client, client i in group j applies its unique sparse network structure M j In your local dataset X i Above, train the local model with a learning rate η:

[0051]

[0052] When the server and client communicate, they only need to send the retained parameters according to the mask. In addition, each bit of the mask represents whether the corresponding parameter is retained. When the original model uses 32-bit floating point numbers, the mask only occupies 1 / 32 of the additional communication overhead.

[0053] S2-4: Model aggregation: Parallel over-parameterization leads to the problem of model heterogeneity; in order to preserve the unique properties of different sparse network structures, each model is divided into two components: (1) the common part parameters θ Public , the parameters in this part correspond to the global sparse structure M G , shared by all clients, so all clients participate in the aggregation of public parameters; (2) private parameters θ Private , that is, the newly explored parameters, are different for each group, so only clients in the same group participate in the aggregation. Assume that there are N clients participating in the training, of which n j clients belong to group j, then the new model after group j aggregation is:

[0054]

[0055] Where |D i | represents the amount of data from client i, and Represent the public and private parameters of client i respectively. S2-5: Parameter selection. Every △T communication rounds, based on the current parallel over-parameterized model θ POP , according to formula (1.1) calculate |g(θ POP )|, select the top-k most potential parameters to update the global sparse network structure M G , which serves as the starting point for the next round of parameter exploration.

[0056] In general, among the sub-steps in S2, S2-2 and S2-5 are executed every △T communication rounds until the preset round T. endThe two sub-steps S2-3 and S2-4 are performed in each communication round. The sparse network structure is used throughout the training process to reduce both computation and communication overhead.

[0057] Step S2-2 and step S2-5 in step S2 are performed once every ΔT communication rounds until the preset round T. end Until, step S2-3 and step S2-4 are performed in each communication round; the value of f starts from α and gradually increases within the preset T end The round decay is 0 to gradually determine the structure of the global sparse network: f = 0 means that the search for the optimal sparse network is complete. At this time, the final M G This is exactly the expected optimal sparse network structure M; the sparse network structure has been used throughout the entire training process to achieve the purpose of reducing both computational and communication overheads.

[0058] S3: Use the optimal sparse network structure M found in step S2 as the final global sparse network structure, which is shared by all clients and continues to be trained until convergence. Since the network remains highly sparse throughout the entire federated learning process, the training and communication overheads will be greatly reduced, and the inference speed of the deployed sparse model after training is also accelerated.

[0059] Example

[0060] See attached Figure 1 , which is a flow chart of the present invention.

[0061] Consider a typical federated learning framework with a total of 100 clients. In each communication round, 20 clients are randomly selected to participate in the training. Each client uses the SGD optimizer to iterate 5 times on the local dataset in each round. Figure 2 As shown, the present invention provides an efficient federated learning sparse training method based on parallel over-parameterization, including:

[0062] S1: First, the server sparses the initial dense network using a binary mask M∈{0,1} |θ| To indicate whether each parameter is retained. The server sends a small batch of 128 samples to the original network θ for training once to obtain gradient information (no need to update network parameters), and then calculates the sensitivity score g(θ) of each model parameter to the loss function L:

[0063]

[0064] The k parameters corresponding to the highest absolute value of the sensitivity score |g(θ)| are retained to form the initial global sparse network structure M G, where k = (1-S-(1-S)f)×|θ|, f is the ratio of the total number of parameters reserved for exploring new parameters to the total number of parameters with sparsity S.

[0065] S2: Optimize the sparse network structure through the collaboration of all clients. The server groups a large number of clients to perform parallel parameter exploration to achieve parallel over-parameterization. On this basis, the structure of the global sparse network is dynamically optimized as the training progresses to improve the fitting ability of the sparse model, thereby improving the accuracy of the sparse network. It includes the following sub-steps:

[0066] S2-1: The server randomly divides a large number of clients into Z groups according to the situation;

[0067] S2-2: In the parameter exploration phase, this implementation recommends updating the global sparse network structure every △T = 40 communication rounds to ensure that the newly explored parameters are updated reliably. The goal of this phase is to obtain the optimal sparse network structure M. Based on the global sparse network structure M G , the server randomly explores new parameters of proportion f for each group j to maintain the total sparsity S unchanged, thereby forming a new sparse network structure M j , and then the server will M j Sent to each client in group j. By merging all the parameters covered by the exploration, the parallel over-parameterized model θ is obtained POP :

[0068] θ POP =[θ⊙M1,θ⊙M2,…,θ⊙M z ] (2)

[0069] At t=(0,40,80,…,T end ) rounds of updating the sparse network structure, the ratio f of the number of newly explored parameters to the number of remaining parameters is controlled according to the following formula: It is recommended to use α = 0.1 to reduce the impact of model heterogeneity caused by parallel parameter exploration. When parameter exploration stops, T end round, f=0, then the final M G This is exactly the expected M. In addition, it is recommended to set a reasonable number of groups Z. When Z = 50 and S = 0.9, one exploration can cover 71% of all parameters.

[0070] S2-3: In the local training of the client, client i in group j applies its corresponding sparse network structure M j In your local dataset X i The previous iteration updates the local model 5 times:

[0071]

[0072] When the server and the client communicate, they only need to send the retained parameters and the mask according to the mask. The server and the client can identify whether the corresponding model parameters are retained through the mask.

[0073] S2-4: In order to solve the challenge of model heterogeneity brought by parallel parameter exploration, this paper adopts a new aggregation method. Figure 3 is an example to help understand. Each model can be divided into two components: public part and private part. The server aggregates the model according to the following formula:

[0074]

[0075] Note that the public part of each group's model is the same after aggregation, while the private part is different. When the parameter exploration phase is over, all clients share the same sparse network structure M, and the aggregation strategy at this time will be exactly the same as traditional federated learning.

[0076] S2-5: Every △T = 40 communication rounds, based on the current parallel over-parameterized model θ POP , according to formula (2.1) calculate |g(θ POP )|, select the top-k most potential parameters to update the global sparse network structure M G , which serves as the starting point for the next round of parameter exploration.

[0077] S3: After the parameter exploration phase, the optimal sparse network structure found in step S2 is used as the final global sparse network structure, which is shared by all clients and continues to be trained until convergence. Since the network remains highly sparse throughout the entire federated learning process, the training and communication overheads are greatly reduced, and the inference speed of the deployed sparse model after training is also greatly accelerated.

[0078] Experimental verification:

[0079] In order to verify the universality of the present invention, three different typical networks were selected for the experiment: LeNet-300-100 (MLP, a network consisting of three fully connected layers), VGG11 (deep convolutional network, in this implementation column, the last three fully connected layers are reduced to one layer), and LSTM (a single-layer RNN network with 128 units). This implementation column uses LeNet-300-100 for the MNIST classification task and trains for 1000 communication rounds; uses VGG11 for the CIFAR10 classification task and trains for 3000 communication rounds; and uses LSTM for the Sequential MNIST classification task and trains for 1000 communication rounds. For the MNIST and Sequential MNIST classification tasks, T end=400, T is used for the CIFAR10 classification task end =600.

[0080] The client local training uses the SGD optimizer, with a momentum of 0.9, a weight decay of 5e-4, and a learning rate starting from 0.1, and decaying by a factor of 10 at [0.3, 0.6, 0.8] × total rounds. In terms of data distribution, the experiment considers both uniform data distribution (IID) and non-uniform data distribution (Non-IID). The setting for non-uniform data distribution is: 1 / 5 of the clients are allowed to have data with all labels, 2 / 5 of the clients have data with half of the labels, and the remaining 2 / 5 clients only have data with 1 / 5 of the labels.

[0081] For given sparsity S=0.9 and S=0.95, the number of groups is set to 50. Table 1 reports the model test accuracy of this embodiment (abbreviated as POP-FL) for three different tasks using three typical models in two scenarios with uniform and uneven data distribution. Each data is the average and variance obtained from five experiments. In order to show the beneficial effects of the present invention, it is also compared with other existing methods, including (1) Dense: the original dense network; (2) Static: random sparse initialization, and no dynamic optimization of the sparse network structure during training; (3) SNIP: a one-time pruning method during initialization; (4) SET and RigL, existing dynamic sparse training methods, the same hyperparameter settings as in their original papers are adopted in the experiment; (5) SET-like and RigL-like: the improved SET and RigL methods according to the new sparse initialization and important parameter selection method proposed in the present invention, but without grouping the clients, can be used to verify the effectiveness of parallel over-parameterization in the present invention. The experimental results in Table 1 show that the sparse training method proposed in this invention is significantly better than the existing methods on different networks. Figure 4 The paper also reports the reduction of FLOPs computational overhead and communication overhead compared with the original dense model at different sparsity levels. For the convenience of comparison, the two overheads are normalized, i.e., "normalized overhead". When the VGG11 network is used for training on the CIFAR10 dataset, the present invention uses only 5% of the parameters to maintain the accuracy of the original model, and on this basis reduces 71.8% of the FLOPs computational overhead and 91.3% of the communication overhead.

[0082] Attached Figure 5 The performance gain effect brought by different degrees of parallel over-parameterization rate R is reported. R is equal to the ratio of the number of parameters covered by one exploration to the total number of parameters, which is controlled by the number of groups. The more groups there are, the greater the parallel over-parameterization rate R. Figure 5 The first row shows the performance trend of the sparse model with the increase of the number of groups under the two sparsities of S=0.8 and S=0.9. The second and third rows respectively plot the relationship between S=0.8 and S=0.9 and the parallel over-parameterization rate R (the value of R in the horizontal axis corresponds to the number of groups in the first row). In general, the experimental results show that as the parallel over-parameterization rate R increases, the degree of parallelization increases, and the parameter coverage increases, which significantly improves the test accuracy of the sparse model and can match or even exceed the performance of the original dense model.

[0083] Table 1 Performance evaluation of POP-FL

[0084]

[0085]

[0086] The above description is only a detailed description of the preferred embodiments and principles of the present invention. For ordinary technicians in this field, according to the ideas provided by the present invention, there will be changes in the specific implementation methods, and these changes should also be regarded as the protection scope of the present invention.

Claims

1. An efficient federated learning sparse training method based on parallel over-parameterization, characterized in that: The method comprises the following steps: S1, sparse initialization, the server sparses the original dense network; S2. Find the optimal sparse network structure by grouping clients to perform parallel parameter exploration to achieve parallel over-parameterization, and on this basis dynamically optimize the global sparse network structure as training progresses; S3, sparse training, using the optimal sparse network structure found in step S2, shared by all clients and continued training until convergence; wherein: The step S1 specifically includes: before the federated learning starts, the server performs sparseness on the original dense network according to the connection sensitivity of the model parameters; the sparsity S of the model is defined as the ratio of the number of pruned parameters to the total number of parameters, using a binary mask M∈{0,1} |θ| To indicate whether the parameters of the original network are retained, that is, to represent the structure of the sparse network; starting from a given original dense network θ, before training begins, the server uses a small batch of data to send it to the original network to calculate a forward propagation, and then calculates the connection sensitivity score g(θ) of each model parameter to the loss function L: Where ⊙ is the Hadamard inner product, and the k parameters with the highest absolute value of the sensitivity score |g(θ)| are retained to form the initial global sparse network structure M with a sparsity of (1+f)×S G , where k = (1-S-(1-S)f)×|θ|, f is the ratio of the total number of parameters reserved for exploring new parameters to the total number of parameters with sparsity S; The goal of step S2 is to obtain the optimal sparse network structure M, which specifically includes: S2-1, client grouping: the server randomly divides all clients into Z groups; S2-2, Parallel over-parameterization: Dynamically evolve and update the structure of the sparse network every ΔT communication rounds; based on the global sparse network structure M G , the server randomly explores new parameters of proportion f in the parameter space of the original dense network for each group j to maintain the total sparsity S unchanged, thereby forming a new sparse network structure M j , and then the server will M j is sent to each client of group j; by merging the parameters explored and covered by the collaborative cooperation between the groups, the parameter coverage rate of a single parameter exploration is Z times that of the existing dynamic sparse training method. When the explored parameters cover all the parameters of the original model, another form of over-parameterization can be achieved in terms of space, namely, parallel over-parameterization, and a parallel over-parameterized model θ is obtained. POP : i POP =[θ⊙M1,θ⊙M2,...,θ⊙M z ] (2) The operator [...] means to calculate each part separately and then combine them together; S2-3, local training: In the local training of the client, the client i in group j applies the sparse network structure M j In your local dataset X i On the other hand, the local model is iteratively updated with a learning rate η: When the server and client communicate, they only need to send the retained parameters according to the mask. In addition, each bit of the mask represents whether the corresponding parameter is retained. When the original model uses 32-bit floating point numbers, the mask only occupies 1 / 32 of the additional communication overhead. S2-4. Model aggregation: Parallel over-parameterization leads to the problem of model heterogeneity. In order to preserve the unique properties of different sparse network structures, each model is divided into two components: (1) the common part parameters θ Public , the parameters in this part correspond to the global sparse structure M G , shared by all clients, so all clients participate in the aggregation of public parameters; (2) private parameters θ Prikvate , that is, the newly explored parameters, are different for each group, so only clients in the same group participate in the aggregation; assuming that there are n clients participating in the training, of which n j clients belong to group j, then the new model after group j aggregation is: Where |D i | represents the amount of data from client i, and Represent the public and private parameters of client i respectively; S2-5, parameter selection: every ΔT communication rounds, based on the current parallel over-parameterized model θ POP , according to formula (1) to calculate |g(θ POP )|, select the top-k most potential parameters to update the global sparse network structure M G , which is used as the starting point for the next round of parameter exploration until the set round T end until; Step S2-2 and step S2-5 in step S2 are performed every ΔT communication rounds until the preset round T. end Until, step S2-3 and step S2-4 are performed in each communication round; the value of f starts from α and gradually increases within the preset T end The round decay is 0 to gradually determine the structure of the global sparse network: f = 0 means that the search for the optimal sparse network is complete. At this time, the final M G This is exactly the expected optimal sparse network structure M; the sparse network structure has been used throughout the entire training process to achieve the purpose of reducing both computational and communication overheads.

Images