Data and model heterogeneous federated learning method of global decision boundary distillation learning

Abstract

The application discloses a data and model heterogeneous federated learning method of global decision boundary distillation learning, and solves two key problems of data heterogeneity and heterogeneous model performance difference through two sub-methods. The first sub-method is aimed at the data heterogeneity problem, private data domain distribution information is fully condensed through local clustering of multiple prototypes, and irrelevant class distillation is introduced in the local supervised learning stage to alleviate local knowledge forgetting. The second sub-method is aimed at the problem of heterogeneous model performance difference, and a global decision boundary distillation learning is proposed, a global decision boundary learner is maintained and updated on the server, the local model is optimized from the perspective of the global decision boundary, and the influence of conflict information on the local model is reduced. The performance of the application is better than that of the most advanced multiple federated learning methods, and the application can better adapt to challenging scenarios, and has higher communication efficiency and better privacy reliability.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data and model heterogeneous federated learning method for global decision boundary distillation learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, the scale and quality of datasets have become key factors determining model performance. In real-world scenarios, data is often distributed across numerous terminal devices or institutions. However, increasing public concern about data privacy, coupled with increasingly stringent laws and regulations related to data sharing, poses a significant challenge to aggregating and constructing large-scale datasets. Federated learning (FL), as an emerging distributed machine learning paradigm focused on data privacy protection, enables collaborative model training without transmitting private data. This concept was first proposed by McMahan et al. at Google Research in 2017. In traditional federated learning algorithms, clients communicate and collaborate by exchanging model parameters, ensuring that private data remains local, while the server aggregates and updates the data to optimize the global model.

[0003] Federated learning has a wide range of applications, but it faces significant challenges in practical deployment, especially the heterogeneity of data and models. In real-world scenarios, private data often exhibits non-independent and identically distributed (non-IID) characteristics, causing conflicts between the local model's optimization direction and the global optimization objective, thereby reducing the overall performance of the model. To address this issue, researchers have proposed many personalized federated learning algorithms such as FedProx, FedAMP, and SCAFFOLD. These methods mainly rely on gradient-based information for personalized aggregation and mitigate the negative impact of data heterogeneity by constraining the optimization direction.

[0004] While these algorithms have achieved some success, they still rely on transmitting model parameters for collaborative training, requiring all participants to use the same model architecture. However, in real-world scenarios, participants often have varying local computing capabilities and bandwidth limitations, leading to the development of heterogeneous federated learning. This paradigm avoids sharing private model parameters and is compatible with diverse local model architectures, thus possessing greater practical application value.

[0005] Some heterogeneous federated learning algorithms leverage public datasets to facilitate communication and collaboration between heterogeneous models. These methods transmit the logits output of private models to the public dataset and utilize knowledge distillation (KD) techniques for co-training. However, their performance is highly dependent on the quality and suitability of the public dataset. Other heterogeneous federated learning algorithms (such as LG-FEDAVG and FedGH) address model heterogeneity by employing a global classifier: the server updates the global classifier and distributes it to clients to replace local classifiers. This requires repeatedly uploading local classifiers in each round, thus creating some communication overhead.

[0006] Prototype-based methods (such as FedProto and FedTGP) have been proposed to achieve heterogeneous federated learning by transmitting class prototypes from the client's local feature space and generating a global prototype to guide local model updates. However, the effectiveness of these methods depends on the quality of the global prototype. Given the heterogeneity of local models among clients, differences in their feature extraction capabilities often lead to conflicts between the global prototype and local prototypes in FedProto. Similarly, FedTGP faces an effective convergence challenge in the global prototype generation process, which limits its performance in real-world scenarios. Summary of the Invention

[0007] The present invention aims to at least partially solve one of the technical problems in the related art.

[0008] To address this, this invention proposes a heterogeneous federated learning method for global decision boundary distillation. A global decision boundary learner is established on the server side and maintained using local multi-prototypes. Local clustering-formed multi-prototypes efficiently condense the feature distribution of the private data domain, helping the global decision boundary learner learn the global decision boundary from a global perspective. During the local update phase, the distillation learning concept optimizes and guides the local model update, reducing the negative impact of differences in the capabilities of heterogeneous feature extractors and improving the performance and generalization ability of the local model. In the local supervised learning phase, irrelevant class knowledge distillation is introduced to decouple traditional knowledge distillation and alleviate the forgetting of private data domain knowledge during multi-round collaborative learning.

[0009] To achieve the above objectives, this invention proposes a heterogeneous federated learning method for global decision boundary distillation learning, comprising:

[0010] Obtain the local classifier parameters obtained by each client through local supervised training of the local classifier using local private data;

[0011] A global decision boundary learner is established based on the local classifier parameters. The learner parameters are initialized to obtain global decision boundary information, which is then sent to the client.

[0012] The client is notified to introduce irrelevant class knowledge distillation during the local training phase, select the best model in the collaborative iteration round as the teacher model, and guide the local model to capture the relationship between categories in order to obtain the optimized feature representation.

[0013] Obtain the local prototype set generated by the client based on the optimized feature representation clustering;

[0014] The local prototype set is merged into a global prototype set, and the learner training loss function is defined according to the optimization objective formula to train the global decision boundary learner. The trained learner parameters are then sent to the clients participating in the collaborative training.

[0015] The client obtains the optimized local model by optimizing the updated local classifier based on the trained learner parameters from the perspective of the global decision boundary.

[0016] The data and model heterogeneous federated learning method for global decision boundary distillation learning in this invention embodiment may also have the following additional technical features:

[0017] In one embodiment of the present invention, obtaining the local prototype set generated by the client based on the optimized feature representation clustering includes:

[0018] The client selects the initial centroid of each cluster based on the optimized clustering algorithm Kmeans++ and the probability distribution of the optimized feature representation. It then performs clustering and centroid updates on other feature vectors to obtain the clustering results. The other feature vectors include all feature vectors except those currently selected as cluster centroids.

[0019] Based on the clustering results, multiple local prototype sets are generated for each class using an unsupervised clustering algorithm.

[0020] In one embodiment of the present invention, the local supervised training loss function is:

[0021]

[0022] in, It is a sample space. It is a label space, with M clients participating in federated learning collaborative training. Each client can only access its local private dataset. The client-side local model is composed of feature extractor f. i and classifier h i Composition, feature extractor f i :X→Z, with parameters as Classifier h i :Z→Y, parameters are

[0023] The overall goal of collaborative training is:

[0024]

[0025] In one embodiment of the present invention, the feature space is input into the global decision boundary learner g, and the output is the label space, with parameter θ. g The structure is the same as h;

[0026] The server initializes the global decision boundary learner parameters using the following formula:

[0027]

[0028] After initialization, learner g learns the decision boundaries of each client for the private data domain through local clustering and multi-prototype learning, forming a global decision boundary; the optimization objective is defined as:

[0029]

[0030] Where P g P represents the collection of all client-local clustered multi-prototypes selected in the current round. g ={P 1 ∪P 2 ∪…∪P N}, where N is the number of clients participating in collaborative training in each round, and c is the actual one-hot class label corresponding to prototype p.

[0031] In one embodiment of the present invention, the local training loss of client i is defined as:

[0032]

[0033] Knowledge distillation introduces the softening probability, a temperature parameter, into the logits layer output. The calculation formula is as follows:

[0034]

[0035] Among them l s，c It is the softening probability of category c, l c , l i These are the values ​​of category c and i in the output of the model's logits layer, respectively, and the softened label is...

[0036] The distillation loss function is:

[0037]

[0038] The distillation loss is decomposed and decoupled into two parts: label-related distillation loss and irrelevant distillation loss. The decoupled loss is:

[0039]

[0040] The two parts of loss have different degrees of influence during gradient calculation:

[0041]

[0042] in This represents the logit layer output of the student model for label class c.

[0043] Define irrelevant class knowledge as follows:

[0044]

[0045] In one embodiment of the present invention, obtaining the local prototype set generated by the client based on the optimized feature representation clustering further includes:

[0046] Client i randomly selects the first prototype center point of private data class c features. After that, the others belong to the feature set. eigenvalues Probability of being selected The calculation formula is as follows:

[0047]

[0048] in for The minimum distance to the selected center point is calculated using the following formula:

[0049] Local multi-prototype clustering is represented as:

[0050]

[0051] The prototype set formed by local clustering of client i is denoted as:

[0052] In one embodiment of the present invention, the learner training loss function is:

[0053]

[0054] Where c is the prototype p corresponding to the actual one-hot class label, and g(p,θ) g ) i Let g represent the predicted probability of the learner g for the i-th class.

[0055] In one embodiment of the present invention, the loss function for optimizing the local decision boundary and the data feature space distribution is defined as:

[0056]

[0057] Where α is the global decision boundary learning ratio;

[0058] The loss of the global decision boundary learner g, which guides the local model i in optimizing its training, is defined as:

[0059]

[0060] Where σ() is the softmax function and T is the distillation temperature parameter.

[0061] The heterogeneous federated learning method for global decision boundary distillation learning in this invention has two main components in its communication overhead during the collaboration phase: the client uploading local clustering prototypes and the server sending the global decision boundary learner. The local clustering prototypes uploaded by the client are essentially multiple class representatives of each class of data in the feature space. Compared to other algorithms that upload complete local model parameters, generator parameters, or public data, these prototypes have a smaller size, and their communication overhead is negligible. The global decision boundary learner sent by the server is a lightweight, small learner with fewer parameters, and it is only sent during the model update phase. Compared to the FedAvg algorithm broadcasting complete global model parameters, it has even lower communication overhead. Therefore, this invention has higher communication efficiency.

[0062] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0063] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0064] Figure 1 This is a flowchart of a data and model heterogeneous federated learning method for global decision boundary distillation learning according to an embodiment of the present invention;

[0065] Figure 2 This is an architecture diagram of a heterogeneous federated learning method for global decision boundary distillation learning according to an embodiment of the present invention. Detailed Implementation

[0066] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0067] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0068] The following describes, with reference to the accompanying drawings, a heterogeneous federated learning method for global decision boundary distillation learning according to an embodiment of the present invention, which combines data and models.

[0069] Figure 1 This is a flowchart of a heterogeneous federated learning method for global decision boundary distillation learning according to an embodiment of the present invention. Figure 2 This is the architecture diagram of the method. (For example...) Figure 1 and Figure 2 As shown, the method includes, but is not limited to, the following steps:

[0070] S1, obtain the local classifier parameters obtained by each client through local supervised training of the local classifier using local private data.

[0071] Specifically, before collaborative training, each client needs to perform pre-training using private data for supervised learning. The local supervised training loss function can be written as:

[0072]

[0073] in, It is an instance space. It is the feature space. It is a label space. There are M clients participating in federated learning collaborative training, and each client can only access its local private dataset. The client-side local model is composed of feature extractor f. i and classifier h i Composition, Feature Extractor Its parameters are Classifier Its parameters are Feature extractors for different clients f i The structures are different, and the classifier h for all clients is different. i They have the same structure.

[0074] Meanwhile, in terms of form, the overall collaborative training objective is:

[0075]

[0076] S2, establish a global decision boundary learner based on the local classifier parameters, initialize the learner parameters to obtain global decision boundary information, and send it to the client.

[0077] Specifically, after local pre-training is complete, all clients send their local classifiers to the server. The server then establishes a global decision boundary learner, initializes its parameters, forms the global decision boundary, and sends it to the clients. The server-established global decision boundary learner g takes the feature space as input and the label space as output, with parameters θ. g Its structure is the same as h.

[0078] The server initializes the global decision boundary learner parameters using the following formula:

[0079]

[0080] After initialization, learner g learns the decision boundaries of each client for the private data domain through local clustering and multi-prototype learning, forming a global decision boundary. Its optimization objective is defined as:

[0081]

[0082] Where P g P represents the collection of all client-local clustered multi-prototypes selected in the current round. g ={P 1 ∪P 2 ∪…∪P N}, where N is the number of clients participating in collaborative training in each round, and c is the actual one-hot class label corresponding to prototype p.

[0083] S3 notifies the client to introduce irrelevant class knowledge distillation during the local training phase, selects the best model in the collaborative iteration round as the teacher model, and guides the local model to capture the relationships between categories to obtain optimized feature representations.

[0084] Specifically, during the local training phase, irrelevant class knowledge distillation is introduced into supervised learning. The optimal model from the collaborative iteration rounds is selected as the teacher model to guide the local model in capturing the relationships between classes. The local training loss for client i is defined as:

[0085]

[0086] Traditional knowledge distillation introduces the softening probability as a temperature parameter into the logits layer output. The specific calculation formula is as follows:

[0087]

[0088] Among them l s，c It is the softening probability of category c, lc , l i These are the values ​​of category c and i in the output of the model's logits layer, respectively, and the softened label is...

[0089] The distillation loss function can be written as:

[0090]

[0091] Further decomposing the distillation loss into two parts, namely the label-related distillation loss and the irrelevant distillation loss, the decoupled loss can be written as:

[0092]

[0093] Backpropagation in the model update process is based on gradient calculation. This invention explores the different degrees of influence of the two parts of loss during gradient calculation:

[0094]

[0095] in This represents the logit layer output of the student model for label class c. Based on the above formula derivation, it can be observed that the gradient value range of the label class distillation part is related to... The value is related to the magnitude of the value and depends on the performance of the teacher model.

[0096] This invention aims to help the student model learn the posterior class relations of the teacher model regarding the private data domain, thereby mitigating the forgetting of knowledge about the private data domain during multiple rounds of collaborative training. Therefore, this invention defines irrelevant class knowledge distillation as follows:

[0097]

[0098] S4, obtain the local prototype set generated by the client based on the optimized feature representation clustering.

[0099] Specifically, after local training, the selected clients are clustered to generate a local prototype set and sent to the server.

[0100] Step S4.1: Select the optimized clustering algorithm Kmeans++, and continuously select v initial centroids based on the probability distribution. Then, cluster and update the centroids for other feature vectors, repeating this process until the centroids converge or the maximum number of iterations is reached. Client i randomly selects the first prototype centroid of the private data c class features. After that, the others belong to the feature set. eigenvalues Probability of being selected The calculation formula is as follows:

[0101]

[0102] in for The minimum distance to the selected center point is calculated using the following formula:

[0103] Step S4.2: Perform local multi-prototype clustering. An unsupervised clustering algorithm generates v prototypes for each class, effectively refining the feature information of each class while preserving marginal feature information. The local prototype set is then sent to the server. Local multi-prototype clustering can be represented as:

[0104]

[0105] The prototype set formed by local clustering of client i is denoted as:

[0106] For each client, execute steps S3 and S4.

[0107] S5 merges the local prototype set into a global prototype set, defines the learner training loss function according to the optimization objective formula to train the global decision boundary learner, and sends the trained learner parameters to the clients participating in collaborative training.

[0108] Specifically, the server merges the local prototype sets received from the selected clients into a global prototype set, defines the learner training loss function according to the optimization objective formula, trains the global decision boundary learner, and sends it to the clients participating in collaborative training. The learner training loss function is:

[0109]

[0110] Where c is the prototype p corresponding to the actual one-hot class label, and g(p,θ) g ) i Let g represent the predicted probability of the learner g for the i-th class.

[0111] S6, obtain the optimized local model by optimizing the updated local classifier from the perspective of the global decision boundary based on the trained learner parameters.

[0112] Specifically, the client utilizes the received learner to guide the optimization of the local model from the perspective of the global decision boundary. The local model updates its training, and the loss function for optimizing the local decision boundary and the distribution of data feature space is defined as follows:

[0113]

[0114] Where α is the global decision boundary learning ratio, which controls the learning ratio of the local model to the global decision boundary.

[0115] The loss of the global decision boundary learner g, which guides the local model i in optimizing its training, is defined as:

[0116]

[0117] Where σ() is the softmax function and T is the distillation temperature parameter. This invention utilizes the concept of knowledge distillation, and through softened labeling, the local model learns the posterior class relationships in the global decision boundary learner g for the feature space of the private data domain, further optimizing the local decision boundary.

[0118] For each client, execute step S6.

[0119] Repeat steps S3 to S6 until the predetermined number of iterations is reached.

[0120] Furthermore, to demonstrate the effectiveness of this invention, it evaluates several commonly used image datasets for multi-class classification tasks, including CIFAR-10, CIFAR-100, TinyImageNet, and Flowers102. This invention uses four different feature extractors: a 4-layer CNN, GoogleNet, MobileNet v2, and ResNet18. During the initialization phase, each client is randomly assigned one of these four models.

[0121] Data Distribution Methods. In the experiments, this invention employed a widely used statistical heterogeneity setting to evaluate non-independent and identically distributed (non-IID) data scenarios, namely, the pathology setting and the real-world setting. In the pathology setting, this invention assigned each client two classes from CIFAR-10, ten classes from CIFAR-100, and twenty classes from Tiny-ImageNet and Flowers102 as private data. In the real-world setting, this invention used a Dirichlet distribution Dir(β) to sample each data class, with a default β value of 0.1. All datasets were split into 75% training set and 25% test set.

[0122] Baseline methods. This invention selects four popular and relevant heterogeneous federated learning algorithms as baseline methods, including: global classifier-based methods LG-FedAvg (NeurIPS2019) and FedGH (MM 2023), and prototype learning-based methods FedProto (AAAI 2022) and FedTGP (AAAI 2024).

[0123] Implementation details. In the experiments, this invention simulated 20 clients participating in federated learning, with a client participation rate of 1. In each global iteration, the number of local training rounds for each client was 1, the local model update cycle was 5, and the number of training rounds for the server-side learner was 30. The batch size was set to 10, the learning rate was 0.01, and the global training consisted of 100 iterations. The hyperparameter v was set to 5, and α was set to 0.7. The accuracy of each global experiment was the average of the results on the test sets of each client, and the best global accuracy across all experiments was reported.

[0124] Experimental results under different data distributions. Table 1 shows the accuracy (%) of the algorithm under different statistical heterogeneity settings. This invention achieves the best performance on all four datasets under both data distribution settings, with a maximum improvement of 9.27%, demonstrating its superiority. FedProto, due to differences between heterogeneous models, generates global prototypes through simple averaging, resulting in significant deviations between these prototypes and the class centers in the private data domain, thus performing poorly on datasets with more classes. On the other hand, the global prototypes generated by FedTGP during client training are difficult to converge. LG-FEDAVG and FedGH use a global classifier instead of a local classifier, but fail to effectively resolve information conflicts between clients, leading to a certain degree of local knowledge forgetting. In contrast, this invention reduces the negative impact of performance differences in heterogeneous feature extractors by optimizing the decision boundary. This algorithm optimizes the local decision boundary from a global perspective and introduces irrelevant class distillation to alleviate local knowledge forgetting, thus significantly outperforming other baseline algorithms, especially on more challenging datasets.

[0125] Table 1

[0126]

[0127] Hyperparameter Research. This invention employs two hyperparameters: the number of local prototype clusters, *v*, and the global decision boundary learning rate, *α*. Experiments were conducted on the Cifar100 and Cifar10 datasets under practical settings to investigate the impact of different hyperparameter configurations on performance. The experimental results are shown in Tables 2 and 3. Table 2 shows the accuracy (%) under different numbers of local prototype clusters, and Table 3 shows the accuracy (%) under different global decision boundary learning rates. When the number of local prototype clusters, *v*, is set to 1, the algorithm performance decreases compared to using multiple prototypes because a single prototype cannot effectively capture the distribution of the local data feature space. When the global decision boundary learning rate, *α*, is set to a higher value, it inhibits the model's learning of the local decision boundary, but the performance still outperforms all baseline algorithms. A suitable value for *α* allows the global decision boundary learner to better guide the optimization of the local model, resulting in superior performance.

[0128] Table 2

[0129] Dataset v＝1 v＝5 v＝10 CIFAR-100 44.78 45.09 44.96 CIIAR-10 87.25 87.90 87.84

[0130] Table 3

[0131] Dataset α＝0.5 α＝0.6 α＝0.7 α＝0.8 α＝0.9 CIFAR-100 44.96 44.94 45.09 44.67 44.38 CIFAR-10 87.72 87.68 87.90 87.97 87.63

[0132] Ablation experiments were conducted on four datasets to evaluate the effectiveness of the key components of this invention. The results are shown in Table 4, which represents the accuracy (%) of the ablation experiments performed using the key components of the method. The experiments show that GDBD-MP (Global Decision Boundary-Multi-Prototype) effectively optimizes the local decision boundary. Furthermore, the introduction of irrelevant class distillation enables the model to learn posterior class relationships specific to the private data domain, preserving local knowledge and further improving the performance of the local model.

[0133] Table 4

[0134]

[0135] More challenging task scenarios. To evaluate the performance of this invention in more challenging settings, the number of heterogeneous models was increased from four to eight by adding ResNet34, ResNet50, ResNet101, and ResNet152, thereby increasing the diversity among models. Experiments were conducted on the Tiny-ImageNet dataset for two data distribution settings, and the results are shown in Table 5, representing the accuracy (%) of this invention in more challenging scenarios. In multi-class few-shot learning, thanks to global decision boundary distillation, this invention maintains superior performance in both data distribution settings, with an accuracy improvement of up to 6.75%. When faced with significant differences between heterogeneous models, optimizing local decision boundaries from a global perspective, rather than using prototype learning and sharing a global head, can better adapt to more complex data distributions and mitigate the negative impact of heterogeneous model differences.

[0136] Table 5

[0137] set up Actual settings Pathology settings FadProto 24.01 32.78 FedTGP 27.53 32.29 LG-FedAvg 29.17 34.87 FedGH 26.88 34.49 This invention 30.76 35.26

[0138] This invention presents a heterogeneous federated learning method for data and models using global decision boundary distillation learning. It employs a multi-prototype approach with local clustering. The prototypes are the class centers in the feature space; this low-dimensional data is irreversible, preventing adversaries from performing data-driven reverse reasoning attacks, thus offering good privacy and reliability. The input to the global decision boundary learner is the feature space, and it primarily learns the decision boundary, not the original data. Given the security of low-dimensional data, adversaries cannot infer the original data through the global decision boundary learner. This invention improves the accuracy of the local model in cases of data heterogeneity. Global prototypes generated by simple averaging have significant discrepancies with the class centers in the feature space of the private data domain, resulting in poor performance on datasets with more data categories; global prototypes trained on clients are difficult to converge. Directly replacing the local classifier with a global classifier cannot effectively resolve conflicting information between different clients, leading to a certain degree of local knowledge forgetting. From the perspective of optimizing the decision boundary, the negative impact of performance differences of heterogeneous feature extractors is reduced. The local decision boundary is optimized from a global perspective, which improves the performance of the local model. Furthermore, irrelevant class distillation is introduced to alleviate local knowledge forgetting. Therefore, it has a greater performance advantage over other baseline algorithms on more challenging datasets.

[0139] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0140] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Images