A communication-efficient heterogeneous federated learning training method

Abstract

The application discloses a communication-efficient heterogeneous federated learning training method, which is applied to a system composed of a server and multiple clients. On the one hand, during local model training, the client optimizes and uploads a binary mask, thereby greatly reducing the communication cost of federated learning, and solving the problem that low-end devices are difficult to participate in complex federated learning model training. On the other hand, the binary mask uploaded by the client can well reflect the characteristics of data distribution, and the data-heterogeneous clients can be clustered without additional information, so that the clients with different data distributions have different models, and the model inference performance in the heterogeneous scene is improved. The local model obtained by using the method can be used for completing a preset image detection task.

Description

Technical Field

[0001] This invention belongs to the field of machine learning, specifically relating to a communication-efficient heterogeneous federated learning training method. Background Technology

[0002] Federated learning is an emerging distributed machine learning paradigm that facilitates collaborative optimization of a global model by numerous clients without sharing sensitive local data. Due to its privacy-preserving characteristics, federated learning is widely popular in both academia and industry, particularly in fields such as medical image analysis, facial recognition, and personalized recommendation systems. In federated learning, significant communication is required between the server and clients for iterative training. This communication cost becomes particularly significant when dealing with large models. For example, current trends in model architecture, such as large language models and Q-learning, involve unprecedented scales, containing billions or even trillions of learnable parameters. While complex models offer superior performance, the enormous bandwidth and throughput requirements of such large models make traditional federated learning frameworks unsuitable for training at this scale. Furthermore, data heterogeneity is common in practical federated learning, severely degrading model performance. For instance, mobile applications may collect user behavior data, such as application usage frequency, location information, and social interactions. Since user preferences and behaviors may vary due to geographical location and cultural differences, this data can exhibit significant heterogeneity across different user groups. Therefore, improving performance and communication efficiency in heterogeneous federated learning scenarios is a pressing challenge.

[0003] In their paper "Heterofl: Computation and communication efficient federated learning for heterogeneous clients (ICLR 2021)," Diao et al. proposed that the server allocates global model sub-models for local training by the client based on the client's current computing and communication resources. The server divides the client's computing and communication resources into P levels, each level corresponding to a category of global sub-model. The global sub-model reduces the input and output channels of each level by a fixed ratio according to different levels. The server sends the divided sub-models to the corresponding clients for local training and then uploads them back to the server. The server constructs a global model using specific aggregation rules, and then iteratively divides the aggregated global model again, repeating the iterative training until convergence. The shortcomings of this method are: firstly, because low-end devices are allocated smaller model structures, their inference performance is severely degraded, affecting their willingness to participate in federated learning training; secondly, high-end devices are allocated almost complete models, and their communication overhead remains high without reduction.

[0004] Shandong University disclosed a method for optimizing the performance and communication efficiency of heterogeneous federated learning in its patent application, "A Communication Method and System for Heterogeneous Federated Learning in Industrial Internet of Things" (Patent Application No.: CN202410487452.0, Authorization Announcement No.: CN118101501B). This method triggers a multi-granularity quantization module to quantize updated local model parameters. The central server receives model updates uploaded by each device node and passes the quantized model updates to the dequantization module for dequantization. Then, the central server sends the dequantized results to the stage aggregation module to aggregate model parameters, repeating the iteration until the current federated learning reaches the predetermined number of training rounds. The shortcomings of this method are: firstly, the global model after dequantization and aggregation by the central server exhibits poor inference performance on devices with small amounts of data; secondly, this method only reduces the communication overhead from client to server, not from server to client, meaning that low-end devices still cannot participate in the training of complex models.

[0005] In summary, existing methods such as gradient quantization and compression still suffer from high server-to-client communication overhead and low model performance in heterogeneous federated learning scenarios. Therefore, there is an urgent need for a communication-efficient heterogeneous federated learning training method to ensure the availability and correctness of the model when performing detection tasks in specific domains, while reducing the communication cost of federated learning. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a communication-efficient heterogeneous federated learning training method. The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] A communication-efficient heterogeneous federated learning training method is applied to a system consisting of one server and multiple clients. The method includes:

[0008] S100, the server generates the initial model weights w0 and initial weight scores s of the neural network model. g,0 The server communicates with clients to provide each client with the initial model weights w0, thereby initializing its local model; the server then uses the initial weight scores s... g,0 Calculate the initial probability mask θ g,0 Send to all clients;

[0009] S200, during the t-th round of global training iteration, the server selects K. t One client participates in the training of the federated learning model;

[0010] S300, each selected client k performs E rounds of local model training based on its local dataset, updating and optimizing its local weight scores s. k,t And upload the binary mask m k,t To the server; wherein, the local dataset includes several images;

[0011] S400, the server clusters all binary masks uploaded by clients and assigns cluster IDs to the corresponding binary masks to achieve cluster ID allocation for clients;

[0012] S500, the server performs grouping and aggregation on the binary mask of each cluster after clustering to obtain the intra-cluster probability mask and the global probability mask θ of the t-th round of global training iteration. g,t ;

[0013] S600, the server notifies the client to perform a consistency check on the allocated cluster ID;

[0014] S700, the server sends the intra-cluster probability mask to the client in the corresponding cluster and sends the global probability mask to the client that has not been assigned a cluster ID, so that the client can recover the probability mask and perform local training until the model converges.

[0015] S800, each client obtains a binary mask based on the intra-cluster probability mask, and uses the obtained binary mask and the initial model weight w0 to obtain the final sub-network model, thereby obtaining the trained local model; the local model is used to complete the preset image detection task.

[0016] In one embodiment of the present invention, S100 includes:

[0017] S110, the server uses a random seed S to generate the initial model weights w0 and initial weight scores s of the neural network model. g,0 Among them, the weight score represents the importance of the weights in the neural network model. The larger the value, the more beneficial the weight is to the model's inference.

[0018] S120, the server sends the random seed S to all clients, so that each client can build the same initial model weight w0 locally to initialize the local model;

[0019] S130, the server determines the initial weight score based on the initial weight score s. g,0 The initial probability mask θ is calculated using the logistic function. g,0 and the initial probability mask θ g,0 It is sent to each client; where the probability mask represents the probability that the weights are used in the forward propagation.

[0020] In one embodiment of the present invention, S300 includes:

[0021] S310, during the t-th round of global training iteration, each client k utilizes the logistic function Logit and the global probability mask θ received from the server. g,t-1 Calculate the weight scores s of the local model. g,t Among them, for the first round of global training, the global probability mask θ g,t-1 The initial probability mask θ g,0 ;

[0022] S320, begin the current round of local training. Each client k will assign weight scores s to its local model. k,t Using the logistic function to map to a probability mask θ k,t ; where, during the first round of local training in the t-th round of global training, the weight score s of the local model is k,t The weight score s of the local model g,t ;

[0023] S330, each client has k pairs of probability masks θ k,t Bernoulli sampling is performed to generate a binary mask m. k,t ;

[0024] S340, each client k according to the binary mask m k,t The initial model weights w0 are used to calculate the sub-network model w for forward propagation by multiplying corresponding matrix values. k,t When the value of the binary mask is 1, it means that the corresponding weight participates in the forward propagation; when the value of the binary mask is 0, it means that the corresponding weight does not participate in the forward propagation.

[0025] S350, each client k utilizes its local dataset and subnetwork model w k,t After forward propagation, its gradient value is calculated, thereby updating the local weight score s. k,t ;

[0026] S360, each client k repeats steps S320 to S350E times to complete E rounds of local training and obtain the updated and optimized weight score s. k,t ;

[0027] S370, each client k will transfer the weighted score s obtained in step S360. k,t Convert to optimized binary mask m k,t And upload it to the server.

[0028] In one embodiment of the present invention, S400 includes:

[0029] S410, the server receives Kt binary mask uploaded by each client

[0030] S420, the server from K t Q binary masks are randomly selected from the binary masks as the initial centroids of each cluster;

[0031] S430, the server assigns each binary mask to the cluster containing the nearest centroid to obtain a preliminary clustering result; wherein, the preliminary clustering result records the ID of the client within each cluster;

[0032] S440, For each cluster, calculate a centroid such that the sum of the distances from the centroid to each binary mask of the cluster is minimized, thereby obtaining the centroid of each cluster again;

[0033] S450, repeat steps S430~S440T1 times to obtain K. t A binary mask M K,t The final clustering result is obtained, and at the same time, the corresponding cluster ID is assigned to each client k based on the final clustering result.

[0034] In one embodiment of the present invention, S430 includes:

[0035] S4301, calculate each binary mask m k to the center of mass m q distance d q,k :

[0036]

[0037] Where L is the number of layers in the neural network model, and n l This represents the number of parameters in the l-th layer model. Represents the binary mask m k The parameters of the l-th layer, This represents the XOR operation;

[0038] S4302, for each binary mask m k Determine the cluster ID of the nearest centroid, denoted as:

[0039] q = argmin q∈[Q] d q,k ;

[0040] S4303, the binary mask m k Assigning it to its nearest cluster q, the operation is represented as:

[0041] C[q].insert(k);

[0042] Where C represents the clustering result, and C[q] records the IDs of all clients in cluster q.

[0043] In one embodiment of the present invention, S500 includes:

[0044] S510, the server aggregates the binary masks within each cluster q to obtain the cluster probability mask Θ after aggregating cluster q. g,t [q], the calculation formula is:

[0045] S520, the server aggregates all intra-cluster probability masks to obtain the global probability mask for the t-th round of global training iteration.

[0046] In one embodiment of the present invention, S600 includes:

[0047] S610, the server will aggregate the intra-cluster probability mask Θ g,t [q] Perform Bernoulli sampling to obtain Q cluster-specific binary masks M. t Then M t Send to each client; where M t ={m q,t |m q,t =Bern(Θ) g,t [q])} q∈[Q] Bernoulli sampling;

[0048] S620, each client uses the binary mask M within each cluster. t [q] Obtain the sub-network model of the forward propagation, and then calculate its loss value based on the local dataset. The cluster with the smallest loss value is obtained to determine the locally inferred cluster ID. Wherein, when the cluster ID inferred locally by the client is consistent with the cluster ID assigned by the server, the corresponding client will no longer participate in the clustering and cluster ID allocation process. When the cluster ID inferred locally by the client is inconsistent with the cluster ID assigned by the server, the cluster ID inferred locally by the client shall prevail.

[0049] In one embodiment of the present invention, S700 includes:

[0050] S710, the server will use integers and integers Send to the client with cluster ID q; where, This indicates the number of clients in each cluster. This represents the sum of the binary masks in each cluster;

[0051] S720, each client receives and The global probability mask θ is then recovered.g,t The model is trained locally T times to achieve convergence.

[0052] In one embodiment of the present invention, S800 includes:

[0053] S810, each client sets the intra-cluster probability mask Θ g,T [q] Perform Bernoulli sampling to obtain the binary mask m q =Bern(Θ) g,T [q]);

[0054] S820, each client according to the binary mask m q The initial model weights w0 are multiplied element-wise to generate a sub-network model with reasoning capabilities, thus forming a trained local model.

[0055] In one embodiment of the present invention, the server is a central server and the client is a medical institution server; the local dataset of the client includes the patient's medical image data and is labeled with tumor type tags;

[0056] The preset image detection task includes: the client uses a trained local model to detect the medical image data to be tested and obtain the tumor type.

[0057] The beneficial effects of this invention are:

[0058] This invention provides a communication-efficient heterogeneous federated learning training method. On one hand, during local model training, the client optimizes locally and uploads a binary mask to the server, improving upload communication efficiency (each parameter occupies 1 bit). The client downloading integer clustering masks improves download communication efficiency, allowing even low-end devices to participate in complex federated learning model training, enhancing model usability and significantly reducing communication costs. On the other hand, the binary mask uploaded by the client effectively reflects the characteristics of data distribution. Based on the similarity of the binary masks, clients with different data distributions can be estimated. Clustering clients with heterogeneous data can be performed without additional information, allowing clients with different data distributions to have different models, thus improving model inference performance in heterogeneous scenarios. By using the server to cluster and distribute models based on the binary mask, the communication efficiency and model performance of heterogeneous federated learning can be improved. Attached Figure Description

[0059] Figure 1 A flowchart illustrating a communication-efficient heterogeneous federated learning training method provided in an embodiment of the present invention;

[0060] Figure 2This is a flowchart illustrating another communication-efficient heterogeneous federated learning training method provided in an embodiment of the present invention. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0062] Federated learning enables multiple users to collaboratively train a global model without compromising privacy. However, the heterogeneity of federated learning data distribution and the limited communication bandwidth of low-end devices severely impact the convergence and performance of federated learning models. Currently, existing federated learning training methods mainly suffer from the following shortcomings:

[0063] (1) Existing technologies can only reduce the communication overhead from the client to the server by relying on compression or quantization methods. The communication overhead from the server to the client is still heavy, and the communication cost of model training is high.

[0064] (2) Existing technologies train models under heterogeneous federated learning, which have low inference performance and poor usability on low-data-volume or low-end devices.

[0065] To address the aforementioned issues, this invention provides a communication-efficient heterogeneous federated learning training method, applicable to a system consisting of one server and multiple clients. The aim is to use the trained local model to perform detection tasks within a specific domain, which may include fields such as medicine.

[0066] Please see Figure 1 and combined Figure 2 It is understood that the method may include the following steps S100 to S800:

[0067] S100, the server generates the initial model weights w0 and initial weight scores s of the neural network model. g,0 The server communicates with clients to provide each client with the initial model weights w0, thereby initializing its local model; the server then uses the initial weight scores s... g,0 Calculate the initial probability mask θ g,0 Send to all clients;

[0068] The purpose of S100 is to enable each client to generate an initial neural network model, i.e., obtain an initial local model, through server-side initialization.

[0069] In one optional implementation, S100 may include:

[0070] S110, the server uses a random seed S to generate the initial model weights w0 and initial weight scores s of the neural network model. g,0 ;

[0071] In this embodiment of the invention, the weight score represents the importance of the weights in the neural network model. The larger the value, the more beneficial the weight is to the model's inference.

[0072] The initial model weights w0 remain constant throughout the process, therefore, the initialization method of the model weights has a significant impact on the model's performance. This invention uses the "Kaiming constant" method to initialize the model weights. Specifically, the absolute value of each weight in each layer of the neural network model is a constant σ, whose sign is randomly chosen to be positive or negative. The constant σ is the standard deviation of the Kaiming normal distribution, and its calculation formula is... n l This represents the number of model parameters in the l-th layer of the neural network model. In other words, the constant σ for each layer is determined based on the number of model parameters in that layer.

[0073] Using a random seed S is a common method, which will not be discussed in detail here. The way it generates the initial model weights w0 can be random.

[0074] S120, the server sends the random seed S to all clients, so that each client can build the same initial model weight w0 locally to initialize the local model;

[0075] After the server sends the random seed S to all clients, the clients can use the random seed S to generate the initial model weights w0 of the neural network model, thereby constructing the same neural network model and realizing local model initialization.

[0076] As mentioned earlier, during local training on the client side, the initial model weights w0 remain unchanged.

[0077] S130, the server determines the initial weight score based on the initial weight score s. g,0 The initial probability mask θ is calculated using the logistic function. g,0 and the initial probability mask θ g,0 Send to each client;

[0078] Here, the probability mask represents the probability that the weights are used during forward propagation.

[0079] Specifically, calculate the initial probability mask θ g,0 The formula is: θ g,0 =Logistic(s g,0 Logistic is the logistic function. The logistic function is the function that calculates the initial weight scores s. g,0 Mapping from (-∞, +∞) to the initial probability mask θ g,0 The interval (0,1).

[0080] S200, during the t-th round of global training iteration, the server selects K. t One client participates in the training of the federated learning model;

[0081] Among them, K is selected. t Each client can be selected randomly.

[0082] S300, each selected client k performs E rounds of local model training based on its local dataset, updating and optimizing its local weight scores s. k,t And upload the binary mask m k,t The data is transmitted to the server; wherein the local dataset includes several images; the local dataset can be configured according to different image detection task requirements.

[0083] E is a natural number greater than 0, which can be set as needed.

[0084] In one optional implementation, S300 may include:

[0085] S310, during the t-th round of global training iteration, each client k utilizes the logistic function Logit and the global probability mask θ received from the server. g,t-1 Calculate the weight scores s of the local model. g,t ;

[0086] Specifically, for the first round of global training, the global probability mask θ g,t-1 The initial probability mask θ g,0 ;

[0087] Calculate the weight score s of the local model g,t The formula is:

[0088] s g,t =Logit(θ) g,t-1 );

[0089] Logit is a function that maps real numbers to the interval (0,1).

[0090] S320, begin the current round of local training. Each client k will assign weight scores s to its local model. k,t Using the logistic function to map to a probability mask θ k,t ;

[0091] The specific calculation formula is: θ k,t =Logistic(s k,t );

[0092] In the first round of local training during the t-th round of global training, the weight score s of the local model is... k,t The weight score s of the local modelg,t ;

[0093] S330, each client has k pairs of probability masks θ k,t Bernoulli sampling is performed to generate a binary mask m. k,t ;

[0094] The specific calculation formula is: m k,t =Bern(θ) k,t );

[0095] Bern represents Bernoulli sampling.

[0096] S340, each client k according to the binary mask m k,t The initial model weights w0 are used to calculate the sub-network model w for forward propagation by multiplying corresponding matrix values. k,t ;

[0097] The specific calculation formula is: w k,t =m k,t ⊙w0;

[0098] Where ⊙ represents element-wise multiplication of the matrix; when the value of the binary mask is 1, it means that the corresponding weight participates in the forward propagation, and when the value of the binary mask is 0, it means that the corresponding weight does not participate in the forward propagation.

[0099] S350, each client k utilizes its local dataset and subnetwork model w k,t After forward propagation, its gradient value is calculated, thereby updating the local weight score s. k,t ;

[0100] The specific calculation formula is as follows:

[0101] Where ← indicates that the left side is obtained by updating the right side; the right side s k,t The left side shows the original weight scores, and the right side shows the updated weight scores; η represents the learning rate, B represents the batch size (i.e., the number of samples processed simultaneously in each model training iteration), and L represents the loss function. D represents the calculation of the gradient value. k Let D represent the local dataset of client k. k,b This represents the b-th batch of samples with a batch size of Batchsize within the local dataset of client k.

[0102] S360, each client k repeats steps S320 to S350E times to complete E rounds of local training and obtain the updated and optimized weight score s. k,t ;

[0103] S370, each client k will transfer the weighted score s obtained in step S360.k,t Convert to optimized binary mask m k,t And upload it to the server.

[0104] Specifically, client k first assigns the weight score s obtained in step S360. k,t Mapped to a probability mask θ k,t The calculation formula is θ k,t =Logistic(s k,t Then, for the probability mask θ k,t Perform Bernoulli sampling to obtain the optimized binary mask m. k,t The calculation formula is m k,t =Bern(θ) k,t Finally, it is uploaded to the server.

[0105] It should be noted that the client of this invention uploads a binary mask m. k,t Each parameter in the model can be represented using 1 bit. In contrast, traditional federated learning mechanisms upload model gradients from the client, where each parameter is a floating-point number requiring 32 bits for representation. Therefore, compared to traditional federated learning mechanisms, this invention reduces the communication overhead of client uploads by 32 times.

[0106] S400, the server clusters all binary masks uploaded by clients and assigns cluster IDs to the corresponding binary masks to achieve cluster ID allocation for clients;

[0107] In one optional implementation, S400 may include:

[0108] S410, the server receives K t binary mask uploaded by each client

[0109] S420, the server from K t Q binary masks are randomly selected from the binary masks as the initial centroids of each cluster;

[0110] Step S420 is equivalent to determining Q clusters.

[0111] S430, the server assigns each binary mask to the cluster containing the nearest centroid to obtain a preliminary clustering result; wherein, the preliminary clustering result records the ID of the client within each cluster;

[0112] Specifically, the S430 may include:

[0113] S4301, calculate each binary mask m k to the center of mass m q distance d q,k:

[0114]

[0115] Where L is the number of layers in the neural network model, and n l m represents the number of parameters in the l-th layer model. l k Represents the binary mask m k The parameters of the l-th layer, This represents the XOR operation;

[0116] S4302, for each binary mask m k Determine the cluster ID of the nearest centroid, denoted as:

[0117] q = argmin q∈[Q] d q,k ;

[0118] Here, argmin represents finding the minimum value.

[0119] S4303, the binary mask m k Assigning it to its nearest cluster q, the operation is represented as:

[0120] C[q].insert(k);

[0121] Where C represents the clustering result, and C[q] records the IDs of all clients in cluster q.

[0122] S440, For each cluster, calculate a centroid such that the sum of the distances from the centroid to each binary mask of the cluster is minimized, thereby obtaining the centroid of each cluster again;

[0123] S450, repeat steps S430~S440T1 times to obtain K. t A binary mask M K,t The final clustering result is obtained, and at the same time, the corresponding cluster ID is assigned to each client k based on the final clustering result.

[0124] Specifically, the final clustering result can still be represented by C, and each client k is assigned a corresponding cluster ID q based on the final clustering result.

[0125] S500, the server performs grouping and aggregation on the binary mask of each cluster after clustering to obtain the intra-cluster probability mask and the global probability mask θ of the t-th round of global training iteration. g,t ;

[0126] In one optional implementation, S500 may include:

[0127] S510, the server aggregates the binary masks within each cluster q to obtain the cluster probability mask Θ after aggregating cluster q. g,t [q], the calculation formula is:

[0128] S520, the server aggregates all intra-cluster probability masks to obtain the global probability mask for the t-th round of global training iteration.

[0129] S600, the server notifies the client to perform a consistency check on the allocated cluster ID;

[0130] Since clustering errors may occur on the server side, this invention performs a consistency check on the cluster ID on the client side.

[0131] In one optional implementation, S600 may include:

[0132] S610, the server will aggregate the intra-cluster probability mask Θ g,t [q] Perform Bernoulli sampling to obtain Q cluster-specific binary masks M. t Then M t Send it to each client so that the client can perform a local consistency check on the cluster ID;

[0133] This step is to reduce communication overhead;

[0134] Among them, M t ={m q,t |m q,t =Bern(Θ) g,t [q])} q∈[Q] Bernoulli sampling;

[0135] S620, each client uses the binary mask M within each cluster. t [q] Obtain the sub-network model of the forward propagation, then calculate its loss value based on the local dataset, and determine the cluster ID of the cluster with the smallest loss value;

[0136] The specific calculation formula is: q' = argmin q∈Q L(X,M t [q]⊙w0);

[0137] Where q' is the locally inferred cluster ID; X is the local dataset; and L is the loss function.

[0138] When the cluster ID inferred locally by the client is consistent with the cluster ID allocated by the server, the corresponding client will no longer participate in the clustering and allocation process. When the cluster ID inferred locally by the client is inconsistent with the cluster ID allocated by the server, the cluster ID inferred locally by the client shall prevail, that is, q' will replace the originally allocated q.

[0139] Specifically, once the cluster ID inferred by the client matches the cluster ID inferred by the server, the client's cluster ID is determined to be q, and no further inferences or estimations of the client's cluster ID will be made. Otherwise, the client uses its probability mask corresponding to the estimated cluster ID to update and upload the binary mask, and the server re-evaluates the client's cluster ID until the two estimates of the cluster ID from the client and the server match.

[0140] S700, the server sends the intra-cluster probability mask to the client in the corresponding cluster and sends the global probability mask to the client that has not been assigned a cluster ID, so that the client can recover the probability mask and perform local training until the model converges.

[0141] In one optional implementation, S700 may include:

[0142] S710, the server will use integers and integers Send to the client with cluster ID q;

[0143] To further improve communication efficiency, the server can send only integers. and integers For the client with cluster ID q, where This indicates the number of clients in each cluster. This represents the sum of the binary masks in each cluster. Because... The maximum value is It can be encoded as Therefore, the server can only transmit bits to the client. Where N w This indicates the number of parameters in the binary mask.

[0144] S720, each client receives and The global probability mask θ is then recovered. g,t The model is trained locally T times to achieve convergence.

[0145] Recover the global probability mask θ g,t The calculation formula is:

[0146] T is a natural number greater than 0, which can be set as needed.

[0147] S800, each client obtains a binary mask based on the intra-cluster probability mask, and uses the obtained binary mask and the initial model weight w0 to obtain the final sub-network model, thereby obtaining the trained local model; the local model is used to complete the preset image detection task.

[0148] Step S800 involves the client generating a final sub-network model for each cluster. Specifically,

[0149] S800 may include:

[0150] S810, each client sets the intra-cluster probability mask Θ g,T [q] Perform Bernoulli sampling to obtain the binary mask m q =Bern(Θ) g,T [q]);

[0151] After the Tth training iteration, the client sets the intra-cluster probability mask Θ. g,T [q] Perform Bernoulli sampling to obtain the binary mask m q The binary mask m q It can generate sub-network models with reasoning capabilities from an initial network model.

[0152] S820, each client according to the binary mask m q The initial model weights w0 are multiplied element-wise to generate a sub-network model with reasoning capabilities, thus forming a trained local model.

[0153] The specific calculation formula for generating the subnetwork model is: w q =m q ⊙w0;

[0154] Among them, w q This generates a subnetwork model with reasoning capabilities.

[0155] As can be seen, the final model can be deployed efficiently. It only requires an initial random seed S and a binary mask m to obtain a sub-model w with reasoning capabilities. The random seed is used to construct the initial random network model parameters w0, and the binary mask is used to extract the sub-model w with reasoning capabilities from the random network model. The calculation formula is w = m ⊙ w0.

[0156] As can be seen from the above, this invention implements a highly efficient federated learning training mechanism. In this invention, the client updates weight scores locally and obtains a binary mask through Bernoulli sampling, then uploads the binary mask to the server, significantly reducing the communication efficiency between the client and the server. Simultaneously, the client can download the integer mask from the server, improving the communication efficiency between the server and the client. Furthermore, the server in this invention clusters clients with different data distributions using the binary mask, then aggregates these groups to improve the inference performance of heterogeneous federated learning models. The server obtains K... t A binary mask M K,t The final clustering result is C. Simultaneously, based on the corresponding cluster ID q assigned to client k, the intra-cluster probability mask Θ after cluster q is obtained. g,t [q]. Furthermore, this invention provides a binary mask clustering consistency check method, which improves the clustering accuracy of heterogeneous federated learning models and enhances the model's robustness. The server performs Bernoulli sampling on the aggregated intra-cluster probability masks to obtain Q intra-cluster binary masks M. t Then, it is sent to the client to perform a local consistency check on the cluster ID. If the local and server-side inference results are consistent, clustering will not be performed in subsequent iterations to reduce computational overhead; otherwise, clustering will still be performed in the next iteration.

[0157] This invention provides a communication-efficient heterogeneous federated learning training method. On one hand, during local model training, the client optimizes locally and uploads a binary mask to the server, improving upload communication efficiency (each parameter occupies 1 bit). The client downloading integer clustering masks improves download communication efficiency, allowing even low-end devices to participate in complex federated learning model training, enhancing model usability and significantly reducing communication costs. On the other hand, the binary mask uploaded by the client effectively reflects the characteristics of data distribution. Based on the similarity of the binary masks, clients with different data distributions can be estimated. Clustering clients with heterogeneous data can be performed without additional information, allowing clients with different data distributions to have different models, thus improving model inference performance in heterogeneous scenarios. By using the server to cluster and distribute models based on the binary mask, the communication efficiency and model performance of heterogeneous federated learning can be improved.

[0158] The communication-efficient heterogeneous federated learning training method provided in this invention can be applied to scenarios such as smart mobile devices, finance, healthcare and advertising recommendation systems, and multi-task collaboration of drones, to improve the communication efficiency and model performance of heterogeneous federated learning.

[0159] Taking the medical field as an example, due to privacy protection requirements, patient medical data cannot be shared. Different hospitals or medical institutions, acting as clients, can train local models using medical imaging data (such as X-rays and CT scans), patient historical medical records, biomarkers, and other data. They then upload the binary mask of their local models to a central server. The central server groups and aggregates the binary masks uploaded by each medical institution, and then sends the aggregated integer masks to different medical institutions. This iterative training process is repeated until the model converges. Because medical institutions transmit binary masks to the server, and this invention groups and aggregates different models, it can reduce the communication cost between medical institutions and the central server while improving the accuracy of model predictions for different diseases.

[0160] Local datasets can be selected and labeled according to specific testing needs in medical scenarios, thereby achieving different testing objectives.

[0161] The following is a specific example of the application of the present invention in the medical field:

[0162] The server is a central server, and the client is a medical institution server; the client's local dataset includes the patient's medical image data and is labeled with tumor type tags; wherein, the tumor type tags can be tumor size, classification grade, etc., and can be pre-labeled by professionals as needed.

[0163] The preset image detection task includes: the client uses a trained local model to detect the medical image data to be tested and obtain the tumor type.

[0164] In this example, the client trains a local model based on a local dataset and the server uses the communication-efficient heterogeneous federated learning training method of this invention. When the client obtains the medical image data of a patient to be tested, it can input its trained local model, which will output the tumor type of the medical image data to be tested, thereby completing the detection task.

[0165] Please refer to the steps described above for the specific training process, which will not be repeated here.

[0166] Similarly, the communication-efficient heterogeneous federated learning training method of this invention can also be applied to multi-UAV collaborative task execution scenarios to improve the communication efficiency and model inference capabilities of UAVs.

[0167] When drone swarms perform tasks, such as environmental monitoring, search and rescue, and drone delivery, multiple drones need to work collaboratively to complete the task. This invention allows each drone to process its sensor data and train its model locally, then upload the model's binary mask to a central server for grouping and aggregation, thereby improving communication efficiency and enhancing the overall task execution capability of the drone swarm. Specifically, a small drone swarm can use this invention to jointly train navigation and obstacle avoidance algorithms, sharing model binary masks to improve the swarm's overall obstacle avoidance capability and path planning optimization. Another small drone swarm can use sensor data (such as cameras, radar, etc.) locally to train a target recognition algorithm, and then aggregate the updated model binary masks to the central server for improvement. Through this invention, on the one hand, drones only need to periodically upload the model's binary mask, instead of large amounts of floating-point data, increasing communication efficiency from drone to central server by 32 times, significantly saving bandwidth and reducing latency, solving the problem of low communication bandwidth during drone model training; on the other hand, drones can share learning results in multi-task scenarios, utilizing the binary mask grouping and aggregation concept to improve model inference capabilities in multi-task collaborative scenarios, ensuring the availability and accuracy of drone models.

[0168] Local datasets can be selected and labeled according to specific testing needs in medical scenarios, thereby achieving different testing objectives.

[0169] The following is a specific example of environmental monitoring in the field of unmanned aerial vehicles (UAVs) according to embodiments of the present invention:

[0170] The server is a central server, and the client is an onboard computer of the UAV. The client's local dataset includes environmental image data collected by the UAV and labeled with target type tags. The target type tags can be information such as the target's category, size, and location, and can be pre-labeled by professionals as needed.

[0171] The preset image detection task includes: the client uses a trained local model to detect the image data of the environment to be tested, and obtains the target type.

[0172] In this example, the client trains a local model based on the local dataset and the server using the communication-efficient heterogeneous federated learning training method of this invention. When the client obtains new environmental image data, it can input its trained local model, which will output the target type of the environmental image data to be tested, thereby completing the detection task.

[0173] Please refer to the steps described above for the specific training process, which will not be repeated here.

[0174] In summary, compared with the prior art, the method of the present invention has the following beneficial effects:

[0175] First, this invention provides a highly efficient federated learning training method. The client only needs to upload a binary mask to the server, without needing to upload floating-point parameters, which greatly reduces the communication overhead between the client and the server. Simultaneously, the client can download an integer clustering mask from the server, further improving communication efficiency and reducing communication costs.

[0176] Second, this invention provides an efficient heterogeneous federated learning clustering method. This invention estimates clients with different data distributions based on the similarity of binary masks, clustering clients with heterogeneous data without requiring additional information. This allows clients with different data distributions to have different models, improving model inference performance in heterogeneous scenarios.

[0177] It should be noted that, in the description of this invention, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0178] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A communication-efficient heterogeneous federated learning training method, characterized in that, Applied to a system consisting of one server and multiple clients, the method includes: S100, the server generates the initial model weights w0 and initial weight scores s of the neural network model. g,0 The server communicates with clients to provide each client with the initial model weights w0, thereby initializing its local model; the server then uses the initial weight scores s... g,0 Calculate the initial probability mask θ g,0 Send to all clients; S200, during the t-th round of global training iteration, the server selects K. t One client participates in the training of the federated learning model; S300, each selected client k performs E rounds of local model training based on its local dataset, updating and optimizing its local weight scores s. k,t And upload the binary mask m k,t To the server; wherein, the local dataset includes several images; S400, the server clusters all binary masks uploaded by clients and assigns cluster IDs to the corresponding binary masks to achieve cluster ID allocation for clients; S500, the server performs grouping and aggregation on the binary mask of each cluster after clustering to obtain the intra-cluster probability mask and the global probability mask θ of the t-th round of global training iteration. g,t ; S600, the server notifies the client to perform a consistency check on the allocated cluster ID; S700, the server sends the intra-cluster probability mask to the client in the corresponding cluster and sends the global probability mask to the client that has not been assigned a cluster ID, so that the client can recover the probability mask and perform local training until the model converges. S800, each client obtains a binary mask based on the intra-cluster probability mask, and uses the obtained binary mask and the initial model weight w0 to obtain the final sub-network model, thereby obtaining the trained local model; the local model is used to complete the preset image detection task.

2. The communication-efficient heterogeneous federated learning training method according to claim 1, characterized in that, S100 includes: S110, the server uses a random seed S to generate the initial model weights w0 and initial weight scores s of the neural network model. g,0 Among them, the weight score represents the importance of the weights in the neural network model. The larger the value, the more beneficial the weight is to the model's inference. S120, the server sends the random seed S to all clients, so that each client can build the same initial model weight w0 locally to initialize the local model; S130, the server determines the initial weight score based on the initial weight score s. g,0 The initial probability mask θ is calculated using the logistic function. g,0 and the initial probability mask θ g,0 It is sent to each client; where the probability mask represents the probability that the weights are used in the forward propagation.

3. The communication-efficient heterogeneous federated learning training method according to claim 2, characterized in that, S300 includes: S310, during the t-th round of global training iteration, each client k utilizes the logistic function Logit and the global probability mask θ received from the server. g,t-1 Calculate the weight scores s of the local model. g,t Among them, for the first round of global training, the global probability mask θ g,t-1 The initial probability mask θ g,0 ; S320, begin the current round of local training. Each client k will assign weight scores s to its local model. k,t Using the logistic function to map to a probability mask θ k,t ; where, during the first round of local training in the t-th round of global training, the weight score s of the local model is k ,t The weight score s of the local model g,t ; S330, each client has k pairs of probability masks θ k,t Bernoulli sampling is performed to generate a binary mask m. k,t ; S340, each client k according to the binary mask m k,t The initial model weights w0 are used to calculate the sub-network model w for forward propagation by multiplying corresponding matrix values. k,t When the value of the binary mask is 1, it means that the corresponding weight participates in the forward propagation; when the value of the binary mask is 0, it means that the corresponding weight does not participate in the forward propagation. S350, each client k utilizes its local dataset and subnetwork model w k,t After forward propagation, its gradient value is calculated, thereby updating the local weight score s. k,t ; S360, each client k repeats steps S320 to S350E times to complete E rounds of local training and obtain the updated and optimized weight score s. k,t ; S370, each client k will transfer the weighted score s obtained in step S360. k,t Convert to optimized binary mask m k,t And upload it to the server.

4. The communication-efficient heterogeneous federated learning training method according to claim 3, characterized in that, S400 includes: S410, the server receives K t binary mask uploaded by each client S420, the server from K t Q binary masks are randomly selected from the binary masks as the initial centroids of each cluster; S430, the server assigns each binary mask to the cluster containing the nearest centroid to obtain a preliminary clustering result; wherein, the preliminary clustering result records the ID of the client within each cluster; S440, For each cluster, calculate a centroid such that the sum of the distances from the centroid to each binary mask of the cluster is minimized, thereby obtaining the centroid of each cluster again; S450, repeat steps S430~S440T1 times to obtain K. t A binary mask M K,t The final clustering result is obtained, and at the same time, the corresponding cluster ID is assigned to each client k based on the final clustering result.

5. The communication-efficient heterogeneous federated learning training method according to claim 4, characterized in that, S430 includes: S4301, calculate each binary mask m k to the center of mass m q distance d q,k : Where L is the number of layers in the neural network model, and n l This indicates the number of parameters in the l-th layer model. Represents the binary mask m k The parameters of the l-th layer, where ⊕ represents the XOR operation; S4302, for each binary mask m k Determine the cluster ID of the nearest centroid, denoted as: q=argmin q∈[Q] d q,k ; S4303, the binary mask m k Assigning it to its nearest cluster q, the operation is represented as: C[q].insert(k); Where C represents the clustering result, and C[q] records the IDs of all clients in cluster q.

6. The communication-efficient heterogeneous federated learning training method according to claim 5, characterized in that, S500 includes: S510, the server aggregates the binary masks within each cluster q to obtain the cluster probability mask Θ after aggregating cluster q. g,t [q], the calculation formula is: S520, the server aggregates all intra-cluster probability masks to obtain the global probability mask for the t-th round of global training iteration.

7. The communication-efficient heterogeneous federated learning training method according to claim 6, characterized in that, S600 includes: S610, the server will aggregate the intra-cluster probability mask Θ g,t [q] Perform Bernoulli sampling to obtain Q cluster-specific binary masks M. t Then M t Send to each client; where M t ={m q,t |m q,t =Bern(Θ) g,t [q])} q∈[Q] Bernoulli sampling; S620, each client uses the binary mask M within each cluster. t [q] Obtain the sub-network model of the forward propagation, and then calculate its loss value based on the local dataset. The cluster with the smallest loss value is obtained to determine the locally inferred cluster ID. Wherein, when the cluster ID inferred locally by the client is consistent with the cluster ID assigned by the server, the corresponding client will no longer participate in the clustering and cluster ID allocation process. When the cluster ID inferred locally by the client is inconsistent with the cluster ID assigned by the server, the cluster ID inferred locally by the client shall prevail.

8. The communication-efficient heterogeneous federated learning training method according to claim 7, characterized in that, S700 includes: S710, the server will use integers and integers Send to the client with cluster ID q; where, This indicates the number of clients in each cluster. This represents the sum of the binary masks in each cluster; S720, each client receives and Then recover the global probability mask θ g,t The model is trained locally T times to achieve convergence.

9. The communication-efficient heterogeneous federated learning training method according to claim 8, characterized in that, S800 includes: S810, each client sets the intra-cluster probability mask Θ g,T [q] Perform Bernoulli sampling to obtain the binary mask m q =Bern(Θ) g,T [q]); S820, each client according to the binary mask m q The initial model weights w0 are multiplied element-wise to generate a sub-network model with reasoning capabilities, thus forming a trained local model.

10. The communication-efficient heterogeneous federated learning training method according to claim 1 or 9, characterized in that, The server is a central server, and the client is a medical institution server; the client's local dataset includes the patient's medical image data and is labeled with tumor type tags; The preset image detection task includes: the client uses a trained local model to detect the medical image data to be tested and obtain the tumor type.

Images