Personalized federated learning method based on adaptive local model initialization and double knowledge distillation

Abstract

The application relates to a personalized federated learning method based on adaptive local model initialization and double knowledge distillation, and belongs to the technical field of edge computing. The method performs local model initialization on each client according to the cosine similarity between a global model and a local model, obtains an adaptively initialized local model, performs knowledge distillation on each client according to the global model and a historical local model respectively, updates the parameters of the local model through double knowledge distillation, a central server receives the local models uploaded by all clients participating in training, and obtains a final global model based on the double dynamic aggregation weights of data quality and client contribution. The application can significantly improve the precision and convergence speed of the model when dealing with data heterogeneity problems, and has good adaptability and robustness in various edge computing scenarios.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of edge computing, and relates to a personalized federated learning method based on adaptive local model initialization and double knowledge distillation. BACKGROUND

[0002] Edge Computing is a distributed computing framework that pushes computing, data storage and processing capabilities from traditional centralized data centers to edge devices closer to data sources, such as Internet of Things devices, smartphones, etc. This architecture design aims to reduce the delay of data transmission between devices and servers, improve the real-time performance and response speed of the system, and is particularly suitable for application scenarios that require low latency and real-time performance, such as intelligent manufacturing, autonomous driving, AR / VR applications and video monitoring, etc. With the continuous progress of technology and the rapid development of Internet technology, the scale and functionality of smart mobile devices, wearable devices and Internet of Things devices are growing rapidly, and they are generating massive amounts of data at all times, making the world's data grow exponentially. According to a Cisco research report, the total amount of global data will reach 402EB by the end of 2021, and human society has entered the era of big data in the Internet of Everything. The rise of big data has brought new opportunities and challenges to various industries. In recent years, deep learning has made great progress in computer vision, speech recognition and content recommendation. However, in these fields, deep learning algorithms require massive amounts of data to achieve high-performance models. Although in today's digital age, abundant data has provided great opportunities for the development of deep learning. However, in real-world scenarios, data is often scattered in different places, i.e., data exists in isolated form. If only the data of a single client is used, the trained model will be biased due to incomplete data. Moreover, due to competition between industries and privacy security issues, each client cannot directly send private data to the central server to train the model. How to effectively utilize the resources of data silos while protecting user privacy has become a new challenge in the industry.

[0003] In 2016, McMahan et al. first proposed the concept of federated learning, which is a collaborative deep learning by multiple parties without uploading local data, ensuring that the original data of each party participating in federated learning is not exposed. In the federated learning process, the client trains the model locally and then sends the trained model to the central server. The central server aggregates the models received from the clients in each communication round to obtain a global model, and sends the global model back to each client for iterative update. Due to its advantages, federated learning has been widely applied in real-world scenarios, such as recommendation systems, next-word prediction on mobile keyboards and smart medical data analysis. Therefore, federated learning is a distributed machine learning method with broad application prospects.

[0004] A key challenge in federated learning is that the data of participants is statistically heterogeneous, i.e., the data is Non-Independent and Identically Distributed (Non-IID). Non-IID data means that the data is distributed unevenly across clients, and common examples include label skew distribution and feature skew distribution. Label skew distribution means that the label distribution of the data is different between parties. Feature skew distribution means that data of the same label exhibits different features. Existing research has shown that the difference in local data distribution of clients will cause each client model to update in different directions, and the global model aggregated by the server may not perform well when running on clients, which reduces the motivation of clients to participate in federated learning.

[0005] In recent years, existing research has developed a series of personalized methods to solve the problem of non-independent and identically distributed data. FedBN introduces a batch normalization layer to stabilize the data and enhance the convergence of the model, but has the problem of low robustness. FedPer divides the model into a base layer and a personalized layer, effectively capturing statistical heterogeneity, but must consider the risk of overfitting. The FedProx method controls the distance between the local model and the global model when updating by introducing a proximal term, and the selection of the proximal term weight is challenging for the best implementation of regularization. SCAFFOLD introduces a control variable to reduce the gradient variance to improve the robustness of the local model in the environment of non-independent and identically distributed data, but the parameter tuning process of the control variable has computational complexity. FedProto uses prototypes to represent a class of data and applies prototype aggregation to federated learning under non-independent and identically distributed data. Per-FedAvg broadcasts a shared meta-model to all clients and customizes a specific local loss function for each client, and trains a personalized model for each client by minimizing the local loss function through the gradient descent algorithm, but the additional calculation required by the client locally increases the computational burden. FedMeta treats each client as a separate task, uses a meta-model instead of a global model, extracts and propagates knowledge acquired from historical tasks to achieve knowledge transfer between tasks, and improve the generalization ability to new tasks and overall performance. FedMD uses distillation technology to transfer the knowledge of the client's model to the global model, and feeds back the knowledge of the global model to the client to improve the accuracy of the model and overcome the problem of model heterogeneity and data heterogeneity, but in order to support model aggregation and knowledge distillation, a large amount of computing resources and communication bandwidth are required, which will increase the cost of computation and communication. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a personalized federated learning method based on adaptive local model initialization and double knowledge distillation.

[0007] To achieve the above object, the present application provides the following technical solutions:

[0008] A personalized federated learning method based on adaptive local model initialization and double knowledge distillation, the method comprising the following steps:

[0009] S1, building a personalized federated learning framework, which includes a plurality of clients and a central server;

[0010] S2, on each client, initializing the local model according to the cosine similarity between the global model and the local model, and obtaining the locally initialized local model;

[0011] S3, on each client, respectively performing knowledge distillation according to the global model and the historical local model, and updating the parameters of the local model through double knowledge distillation;

[0012] S4, the central server receives the local models uploaded by all clients participating in training, and obtains the final global model based on the double dynamic aggregation weights of data quality and client contribution.

[0013] Further, in step S1, the personalized federated learning framework built contains a server S and N clients, each client has a private data set {D1, D2, …, D k ,…,D N}; the local data exchange is not carried out between the clients, only the local models {W1, W2, …, W k ,…,W N} trained on the local data are exchanged with the server. The server aggregates all the local models {W1, W2, …, W k ,…,W N} uploaded by the clients to obtain the global model W g .

[0014] Further, in step S2, the following steps are included:

[0015] S21, set the parameter vectors of the global model W g and the local model W k as w1 and w2 respectively, then the formula for calculating the cosine similarity is as follows:

[0016]

[0017] where w1·w2 is the dot product of the two vectors, ‖w1‖ and ‖w2‖ are the Euclidean norms of w1 and w2 respectively; the cosine similarity is between [-1, 1], the closer the value is to 1, the more similar the two vectors are, and the closer the value is to -1, the less similar they are;

[0018] S22. The initialized local model is shown in the following equation:

[0019] W k =δ*W g +(1-δ)*W k

[0020] Here, δ represents the adaptive initialization coefficients of the global and local models, which are determined by cosine similarity.

[0021] Furthermore, in step S3, the client-side local model performs knowledge distillation based on the global model and the historical local model, respectively. Specifically, during local model training on the client side, local data is input into the global model, the initialized current local model, and the historical local model, resulting in logical values ​​predicted by the three models, which are respectively... and The client uses stochastic gradient descent, based on the total local training loss L. k Local target update local model parameters

[0022]

[0023] Where η is the learning rate. Represents the loss function L k Gradient with respect to the parameters.

[0024] Furthermore, local training on the client side includes three different losses: cross-entropy loss L... CE Loss L of global knowledge distillation GD And loss of self-knowledge distillation L SD Among them, the cross-entropy loss L CE Used to measure the current local model Differences between the actual category label y in the local data:

[0025]

[0026] Global knowledge distillation loss L GD Used to compare the distance between the output probability distributions of the student model and the teacher model in global knowledge distillation. As a teacher As a student, the global knowledge distillation loss L GD It is represented as:

[0027]

[0028] Where τ represents the temperature hyperparameter; KL(·) represents the KL divergence, which is used to measure the difference between two probability distributions;

[0029] Client self-knowledge distillation L SD In the method, the self-knowledge distillation loss L As a teacher, the self-knowledge distillation loss L As a student, the self-knowledge distillation loss L SD is expressed as:

[0030]

[0031] The total loss L of local training is obtained by using different hyperparameters to balance the strengths of global knowledge distillation and self-knowledge distillation. k is expressed as:

[0032]

[0033] wherein, α GD and β SD are parameters of the strengths of global knowledge distillation and self-knowledge distillation, respectively.

[0034] Further, in step S4, the following steps are included:

[0035] S41, the proportion of the data amount of each client in the total data amount is used to define a data amount weight:

[0036]

[0037] wherein, n k is the sample number of the client k, and N is the total number of clients;

[0038] S42, the data distribution diversity is defined by Shannon entropy:

[0039]

[0040] wherein, p i is the proportion of the label of the i-th class in the client data, and C is the total number of classes; the greater the entropy value, the more uniform the label distribution of the data, and the higher the data diversity; otherwise, the lower the data diversity;

[0041] S43, the data amount weight and the data distribution diversity are combined into a comprehensive data quality score by weighting:

[0042] Q k =A·N(D k )+B·H(P k )

[0043]

[0044] wherein, A and B are coefficients for adjusting the weights of the data amount and the data diversity, and satisfy A+B=1;

[0045] S44, according to the data quality received local model is initially aggregated, and the initial aggregation model is obtained:

[0046]

[0047] S45, define the Euclidean distance between the local model and the global model as the contribution degree of the client k to the federated learning system:

[0048]

[0049] Wherein, And Respectively represent the i-th element of the model parameter vector, d k The Euclidean distance between the client and the global model, the greater the distance, the greater the gap between the local model and the global model, the lower the contribution to the global model; and is a normal number to avoid denominator 0; The greater, the greater the contribution of the corresponding client to the federated learning system;

[0050] By normalizing the contribution of the client, the final global model is obtained on the basis of the initial aggregation model according to the dynamic aggregation weight:

[0051]

[0052] Wherein, Indicates the updated local model of the client, W t+1,g Indicates the aggregated global model, and the superscript t+1 indicates the round of global and local communication.

[0053] The beneficial effects of the present application are:

[0054] The present application determines the model update direction by the cosine similarity between the received global model and the local model through adaptive initialization, and realizes the balance between globality and individuality through global distillation and self-knowledge distillation, effectively alleviating the knowledge forgetting problem in federated learning. In addition, the aggregation weight is dynamically adjusted according to the data quality of the client when the server aggregates. The present application can significantly improve the precision and convergence speed of the model when dealing with data heterogeneity problems, and has good adaptability and robustness in various edge computing scenarios.

[0055] Other advantages, objects and features of the present application will be set forth in part in the following specification, and in part will become apparent to those skilled in the art upon examination of the following specification, or can be learned from practice of the present application. The objects and other advantages of the present application can be realized and attained by the following description. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments will be described in detail below with reference to the drawings, in which:

[0057] Fig. 1 is a basic framework diagram of federated learning;

[0058] Fig. 2 is a framework structure schematic diagram of the personalized federated learning method based on adaptive local model initialization and double knowledge distillation of the present application. DETAILED DESCRIPTION

[0059] The present application also can be implemented or applied by other different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0060] The drawings are only used for illustrative description, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application; in order to better illustrate the embodiments of the present application, some components in the drawings can be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings can be omitted.

[0061] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for illustrative description, and should not be understood as a limitation of the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0062] Please refer to Figs. 1-2 is a personalized federated learning method based on adaptive local model initialization and double knowledge distillation.

[0063] EMBODIMENT

[0064] In edge computing scenarios, data statistical heterogeneity becomes an important challenge for federated learning. Due to the differences in the data distribution collected by each edge device, such as user behavior and environmental conditions, traditional federated learning algorithms rely on a unified global model, which is difficult to adapt to personalized needs, thereby leading to a decline in model performance. To address this challenge, the present invention proposes a personalized federated learning algorithm (FedAaK) that combines adaptive local model initialization and double knowledge distillation. The method initializes the client's local model based on the global model and local data. During local training, the client uses global model distillation to retain global generalization knowledge and self-knowledge distillation to retain historical personalized knowledge. By setting different parameters to balance the strength of adaptive initialization and double knowledge distillation, the method effectively balances global and personalized performance. The server's dynamic aggregation weight is determined by data quality and contribution level. Experimental results on three different datasets show that the proposed algorithm can significantly improve the accuracy and convergence speed of the model when dealing with data heterogeneity, and has good adaptability and robustness in various edge computing scenarios.

[0065] The present embodiment first introduces related work.

[0066] Federated learning is a distributed machine learning method that does not require raw data to be centralized in a single centralized location. It collaborates with multiple clients by sharing model updates to achieve model training while protecting data privacy. The federated averaging algorithm FedAvg is a classic federated learning algorithm. The main process of classic federated learning is as follows:

[0067] (1) Client selection: Due to the limitations of client status and network conditions, the server will randomly select qualified clients for federated learning;

[0068] (2) Model initialization: The central server initializes the global model and broadcasts it to the selected clients for collaborative training;

[0069] (3) Local training: After receiving the global model, the client initializes and updates the local model, and trains the local model on the local dataset;

[0070] (4) Model upload: After local training, the client uploads the trained local model to the central server;

[0071] (5) Global aggregation: The central server receives the uploaded models from the clients and aggregates them to generate a new global model;

[0072] Iterate the above steps until the model accuracy reaches the preset accuracy or the training round reaches the preset round.

[0073] Classic frameworks of federated learning, such as Fig. 1 As shown, it includes one server S and N clients {C1, C2, ..., C...} k ,…,C N Each client has a private dataset {D1, D2, ..., D}. k ,…,D N Clients do not exchange local data with each other; they only exchange local models {W1, W2, ..., W} trained on their local data with the server. k ,…,W N The global model W is obtained by the server aggregating the local models received from all clients. The client and server undergo a total of T iterations until the model converges.

[0074] During local training, the client minimizes its local loss function value:

[0075]

[0076] Among them, l j (W k ) is the local dataset D k The loss function for sample j is typically cross-entropy loss or mean squared error loss.

[0077] The global optimization objective of federated learning is to minimize the sum of the loss functions of all clients:

[0078]

[0079] Among them, |D k | represents the number of samples in the local dataset, and |D| represents the total number of samples in the global dataset.

[0080] In traditional federated learning, it is assumed that each client's data is independent and identically distributed. However, in real-world applications, due to differences in data sources, user behavior, and usage scenarios across different clients, the data distribution is often non-independent and identically distributed. This means that there may be significant differences between datasets from different clients, including inconsistent feature and label distributions. Because of this inconsistent data distribution, the global model performs poorly on some clients and cannot adapt well to all client data, thus reducing its generalization ability. Furthermore, if local client data is limited and not widely distributed, local models are prone to overfitting to local data, making it more difficult for the global model to converge quickly to a stable optimal solution.

[0081] Personalized Federated Learning (PFL) does not require all clients to end up with the same global model, allowing each client to generate a personalized local training model on top of the global model for each client. Personalized Federated Learning can effectively address the challenges of slow convergence and accuracy decline of models on heterogeneous data, and many recent works have explored and researched it. The current common personalized federated learning methods are mainly divided into two categories:

[0082] (1) Global model personalization: After the server completes the global model aggregation, the client downloads the same global model from the server and performs additional training through local adaptation steps to achieve the purpose of personalized adjustment. This method mainly focuses on the training process on non-identically distributed data, among which data-based methods such as data augmentation or client selection can convert non-identically distributed local data into an identically distributed scenario; model-based methods such as regularization, meta-learning, and transfer learning can optimize the global model to adapt to downstream personalized learning tasks.

[0083] (2) Learning personalized model: aims to complete the personalized task during the training phase, among which structure-based methods set different personalized training structures for different data centers through parameter decoupling, knowledge distillation, etc.; similarity-based methods construct classes for clients with similar properties through multi-task learning, model difference, clustering, etc.

[0084] Since a well-performing model usually has a large size, it will lead to an increase in computational complexity and storage costs. Therefore, these models often cannot be deployed on resource-constrained mobile devices. Knowledge Distillation (KD) is introduced as a model compression method, the main idea of which is to use the output soft labels of a large pre-trained model (called the teacher model) as a target to guide the training of a smaller model (called the student model), so that the student model not only learns the information of the real label in the data, but also imitates the output of the teacher model, and thus learns the relationship between different labels. This technique is often used to compress complex deep learning models, with the goal of reducing the size and computational complexity of the model as much as possible without sacrificing model accuracy, to facilitate deployment in resource-limited environments. Soft labels are the probabilities of input belonging to each class, calculated by the softmax function σ(·):

[0085]

[0086] where q iis the logit value of the i-th class, and the logits of all classes form logits vector q, which is the output of the last fully connected layer of the model. τ is a temperature hyperparameter to control the softness of the soft labels. The value of q is between 0 and 1.

[0087] The student model uses hyperparameters α and β to control the cross-entropy loss function item L CE and the Kullback-Leibler (KL) divergence loss function item L KL to construct the knowledge distillation loss function:

[0088] L KD = α·L CE (y, q s )+ β·τ 2 ·L kL (q t ‖q s )

[0089] where q s and q t represent the logits output by the student model and the teacher model, respectively.

[0090] According to the above, the embodiment further details a personalized federated learning method based on adaptive local model initialization and double knowledge distillation of the application, as shown in Fig. 2 , which includes the following steps:

[0091] S1, build a personalized federated learning framework, which includes a plurality of clients and a central server;

[0092] S2, initialize the local model on each client according to the cosine similarity between the global model and the local model, and obtain the locally initialized local model;

[0093] S3, perform knowledge distillation on each client according to the global model and the historical local model, respectively, and update the parameters of the local model through double knowledge distillation;

[0094] S4, the central server receives the local models uploaded by all clients participating in training, and obtains the final global model based on the double dynamic aggregation weights of data quality and client contribution.

[0095] In step S1 of the embodiment, the personalized federated learning framework built contains a server S and N clients, each client has a private data set {D1, D2, …, D k ,…,D N}. The local data exchange is not performed between the clients, only the local model {W1, W2, …, Wk ,…,W N The server aggregates all local models {W1, W2, ..., W} uploaded by clients. k ,…,W N The global model W is then obtained. g .

[0096] In step S2 of this embodiment, in traditional federated learning, the server generates a global model by aggregating different client models in a heterogeneous environment, and sends the global model to the client. The client then uses the received global model as a new local model for a new round of local training. However, this global model exhibits poor generalization performance on each client. To address this issue, an adaptive initialization method is employed. Local model initialization is achieved through the cosine similarity between the global and local models, thereby correcting the model update direction. Calculating the cosine similarity between two models is typically used to compare their parameters or gradients to measure their similarity.

[0097] Global model W g With local model W k Let the parameter vectors be w1 and w2, then the formula for calculating the cosine similarity is as follows:

[0098]

[0099] Here, w1·w2 is the dot product of two vectors, and ||w1|| and ||w2|| are the Euclidean norms of w1 and w2, respectively, i.e., their lengths. The cosine similarity value is between [-1, 1]. The closer the value is to 1, the more similar the two vectors (model parameters) are, and the closer the value is to -1, the less similar they are.

[0100] The adaptive initialization coefficients δ of the global and local models are determined by cosine similarity. The initialized local model is shown in the following equation:

[0101] W k =δ*W g +(1-δ)*W k

[0102] In step S3 of this embodiment, dual knowledge distillation refers to the client-side local model performing knowledge distillation based on both the global model and the historical local model. Client-side local training includes three different losses: cross-entropy loss L... CE Loss L of global knowledge distillation GD And loss of self-knowledge distillation L SD During local model training on the client side, local data is input into the global model, the initialized current local model, and the historical local model, resulting in three types of model prediction logits. and

[0103] The difference between the current local model's output probability distribution and the ground truth class label y of the local data is measured by the cross-entropy loss:

[0104]

[0105] The distance between the student model's output probability distribution and the teacher model's output probability distribution is compared by the KL divergence, in global knowledge distillation, as the teacher and as the student, the loss L of global knowledge distillation is GD as follows:

[0106]

[0107] where τ represents the temperature hyperparameter, KL(·) represents the KL divergence, which is an index to measure the difference between two probability distributions.

[0108] In client self-knowledge distillation, as the teacher and as the student, the loss L of self-knowledge distillation is SD as follows:

[0109]

[0110] For all the above loss functions, different hyperparameters are used to balance the strength of global knowledge distillation and self-knowledge distillation, and the local training total loss is:

[0111]

[0112] where α GD and β SD are the parameters of the strength of global knowledge distillation and self-knowledge distillation, respectively.

[0113] The client updates the local model parameters k according to the local objective L

[0114]

[0115] where η is the learning rate, represents the gradient of the loss function L k with respect to the parameter

[0116] In step S4 of the embodiment, in the conventional federated learning FedAvg, the server receives the local models uploaded by all clients participating in the current round of training, and aggregates the local models using federated averaging to obtain a global model for the next round. FedAvg uses the proportion of the local data of each client in the total data to perform weighting, however, the data characteristics and class distribution of the clients may differ greatly, and using average aggregation may ignore these differences, and the model may not be able to generalize well; the computing capabilities of different clients differ greatly, and using average aggregation may dilute the model updates of some clients that contribute greatly, thereby reducing the convergence efficiency of the global model.

[0117] Therefore, the application adopts a dual dynamic aggregation weight based on data quality and client contribution, including:

[0118] The data quality is defined based on the comprehensive analysis of the data amount and the data diversity. In order to balance the data amount difference between different clients, the proportion of the data amount of each client in the total data amount is used to define the data amount weight:

[0119]

[0120] Wherein, n k is the sample number of the client k, and N is the total number of clients.

[0121] The diversity of the data label distribution reflects whether the data of the client covers enough categories, and the application defines the data distribution diversity by using Shannon entropy:

[0122]

[0123] Wherein, p i is the proportion of the i-th label in the client data, and C is the total number of categories. The greater the entropy value, the more uniform the label distribution of the data, and the higher the data diversity; on the contrary, the data tends to be in some categories, and the data diversity is lower.

[0124] The two indexes are combined into a comprehensive data quality score by weighting:

[0125] Q k =A·N(D k )+B·H(P k )

[0126]

[0127] Wherein, A and B are coefficients for adjusting the weights of the data amount and the data diversity, and satisfy A+B=1.

[0128] According to the data quality, the received local models are initially aggregated to obtain an initial aggregated model:

[0129]

[0130] The Euclidean distance between the local model and the global model is defined as the contribution degree of the client k to the federated learning system:

[0131]

[0132] wherein, and represent the i-th element of the model parameter vector, d k represents the Euclidean distance between the client and the global model, the greater the distance, the greater the gap between the local model and the global model, the lower the contribution to the global model, and ∈ is to prevent d k from being 0, so a very small positive number is added; The greater the contribution degree, the greater the contribution of the corresponding client to the federated learning system.

[0133] By normalizing the contribution degree of the client, the final global model is obtained on the basis of the initial aggregated model according to the dynamic aggregation weight:

[0134]

[0135] wherein, represents the updated local model of the client, W t+1,g represents the aggregated global model, and the superscript t+1 represents the round of global-local communication.

[0136] The present embodiment carries out experiments according to the above method.

[0137] The present embodiment carries out experiments using three different data sets:

[0138] (1) The MNIST data set has a total of 70,000 samples, each of which is a 28*28 pixel handwritten digital grayscale image. There are 10 categories of samples, which are from 0 to 9.

[0139] (2) The Cifar10 data set has a total of 60,000 samples, each of which is a 32*32 pixel RGB image (color image), and each RGB image must be divided into 3 channels (R channel, G channel, B channel). The 60,000 samples are divided into 50,000 training samples and 10,000 test samples. There are 10 categories of samples, and the label values are distinguished according to 0-9. The specific labels include: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

[0140] (3) The Cifar100 dataset and the Cifar10 dataset are the same in the number of samples, and each sample is also a 32*32 pixel RGB color image containing 3 channels. The difference is that the CIFAR-100 dataset is divided into 100 categories, and each category contains 600 samples. Among them, each category has 500 training samples and 100 test samples. In addition, the 100 subcategories in CIFAR-100 are summarized into 20 large categories. Each image is attached with a "fine" label indicating the subcategory it belongs to, and a "coarse" label indicating the large category it belongs to.

[0141] For each dataset, the Dirichlet distribution Dir(α) is used to construct the non-identically distributed data set across clients when performing experimental verification. The parameter α is used to control the degree of non-identically distributed data, and the smaller the α parameter means a higher degree of non-identically distributed data; the non-identically distributed data partition is used in this embodiment. The local data of each client contains at most s given categories, wherein s=2 for the MNIST dataset and the Cifar10 dataset, and s=10 for the Cifar100 dataset. The data is divided into training set and test set in the ratio of 0.75:0.25.

[0142] In this embodiment, the method and the comparison method of the application both use the stochastic gradient descent method to minimize the loss function. By default, the learning rate of the stochastic gradient descent is set to lr=0.005, the momentum of the new meta and the weight decay are set to 0.9 and 0.00001 respectively, the data batch size is 10, the number of clients is set to N=100 by default, the participation rate of the clients participating in the training is set to p=0.1, and the number of local updates is set to E=5 by default. The number of communication rounds T=200.

[0143] In order to make a fair comparison, the same network architecture convolutional neural network (CNN) is used for the method and all the comparison baseline methods, and the model training is performed using the NVIDIA GeForce RTX 3060Ti GPU on Pytorch in this embodiment. The method FedAaK of the application is compared with other federated learning methods, including FedAvg, FedProx and FedRep.

[0144] The experimental results are shown in Table 1, which shows the test accuracy of the four federated learning methods on the MNIST, Cifar10 and Cifar100 datasets.

[0145] Table 1

[0146]

[0147] The experimental results show that the method of the application achieves high precision in all degrees of non-independent distributed scenarios. It is at least 2% higher than the second best on Cifar10 and Cifar100. This shows that the FedAaK method based on adaptive initialization aggregation and local double knowledge distillation training can effectively balance individualization and global knowledge, greatly improving the performance of the local model. FedAak can be used to process various non-independent distributed image data, and its purpose is to provide personalized models for each client under the premise of ensuring data privacy, thereby improving the performance and adaptability of the model.

[0148] The personalized federated learning method based on adaptive local model initialization and double knowledge distillation proposed by the application can be applied to the field of medical image analysis. In medical image analysis, there may be significant differences in data distribution among different hospitals, such as different equipment, imaging technology, and patient population differences, which can cause the applicability and accuracy of the model to vary among different hospitals. Personalized federated learning can take advantage of these differences by fine-tuning the model for each hospital to improve performance. Select a public medical image dataset, such as the CheXpert or ISIC dataset. These datasets contain images from different hospitals or devices, which can be considered as different client datasets. In traditional federated learning, all clients (hospitals) share a global model; in the proposed personalized federated learning FedAaK, each client fine-tunes the global model based on its own data distribution. The accuracy of the personalized model is improved. Through personalized learning, the model can achieve relatively balanced performance on different clients, rather than overperforming or underperforming in certain hospitals. Personalized federated learning can make the model better adapt to the unique data distribution of each hospital, rather than reducing applicability due to excessive sharing.

[0149] The personalized federated learning method based on adaptive local model initialization and double knowledge distillation can also be applied to the personalized recommendation field. The recommendation system is commonly used for personalized content recommendation of different users. The traditional federated learning model generates a general model, but due to the individual differences of users, the personalized federated learning can better capture the specific preferences of each user. The public MovieLens data set is used, which contains user rating data of movies. The rating data of each user represents a local data distribution, reflecting its preferences. The recommendation system commonly used indicators are three, namely: mean square error (MSE): evaluate the error between the predicted score and the actual score; hit rate (Hit Rate): the proportion of actual user favorite content in the recommended content; diversity (Diversity): the diversity of recommended content, measures the degree of individualization. The comparison experiment results of the traditional federated learning and the proposed personalized federated learning algorithm prove that the personalized federated learning algorithm has lower mean square error on different user groups, higher hit rate, and richer diversity of recommended content, which provides empirical support for the advantages of the personalized federated learning algorithm in practical application scenarios.

[0150] The present application proposes a new adaptive initialization aggregation and local double knowledge distillation training personalized federated learning method-FedAaK, which determines the model update direction by the cosine similarity between the received global model and the local model through adaptive initialization, and realizes the balance between globality and individualization through global distillation and self-knowledge distillation, effectively alleviating the knowledge forgetting problem in federated learning. In addition, the server aggregates the aggregation weight according to the quality of the client data. A large number of experiments prove that the method of the present application is better than other compared methods, proving the effectiveness of the method.

[0151] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A personalized federated learning method based on adaptive local model initialization and double knowledge distillation, characterized in that: The method comprises the following steps: S1, building a personalized federated learning framework, including a plurality of clients and a central server; S2, initializing a local model on each client according to the cosine similarity between the global model and the local model, and obtaining the local model after adaptive initialization; S3, performing knowledge distillation on each client according to the global model and the historical local model respectively, and updating the parameters of the local model through double knowledge distillation; In step S3, the client local model is distilled according to the global model and the historical local model respectively, wherein when the local model is trained locally, the local data is input into the global model, the current local model after initialization and the historical local model, and the logical values predicted by the three models are obtained, respectively , and , the client uses the stochastic gradient descent method to update the local model parameters according to the local target total loss : wherein, is a learning rate, denotes a loss function a gradient with respect to the parameters; The client local training includes three different losses, respectively, a cross-entropy loss , a global knowledge distillation loss , and a self-knowledge distillation loss , wherein the cross-entropy loss is used to measure the difference between the current local model and the real class label of the local data . global knowledge distillation loss for comparing the distance between the output probability distribution of the student model and the output probability distribution of the teacher model in global knowledge distillation, as a teacher as a student, the global knowledge distillation loss which is expressed as: wherein, denotes a temperature hyperparameter; denotes the KL divergence, which measures the difference between two probability distributions; Client self-knowledge distillation In some embodiments, As a teacher As a student, self-knowledge distillation loss is represented as: Using different hyperparameters to trade off the strength of global knowledge distillation and self-knowledge distillation, then the local training total loss is represented as: wherein, and are parameters for global knowledge distillation and self-knowledge distillation strength, respectively; S4, the central server receives the local model uploaded by all clients participating in training, and obtains the final global model based on the double dynamic aggregation weights of data quality and client contribution degree; In step S4, the following steps are included: S41, the proportion of the data amount of each client to the total data amount is used to define the data amount weight: wherein, is the number of samples of the client , is the total number of clients; S42, the data distribution diversity is defined by Shannon entropy: wherein, is the number of classes of the first label, is the proportion of the first label in the client data, is the total number of classes; the greater the entropy value, the more uniform the label distribution of the data, the higher the data diversity; otherwise, the lower the data diversity; S43, the data amount weight and the data distribution diversity are combined into a comprehensive data quality score: wherein, and are coefficients that adjust the data volume and data diversity weights, satisfying ; S44, the received local model is initially aggregated according to the data quality, and an initial aggregation model is obtained: S45, define the Euclidean distance between the local model and the global model as the client Contribution to the federated learning system: wherein, and respectively represent the i-th element of the model parameter vector, represents the Euclidean distance between the client and the global model, the greater the distance, the greater the gap between the local model and the global model, the lower the contribution to the global model; is a positive constant to avoid the denominator 0; the greater the contribution of the corresponding client to the federated learning system.​ Through the normalization of the contribution degree of the client, the final global model is obtained based on the initial aggregation model according to the dynamic aggregation weight: wherein, denotes the updated local model of the client, denotes the aggregated global model, the superscript denotes the round of global and local communication; The samples in the data set used by the client and the central server for model training are handwritten digital images.

2. The personalized federated learning method based on adaptive local model initialization and double knowledge distillation according to claim 1, characterized in that: In step S1, in the built personalized federated learning framework, a server S and N clients are included, and each client has a private data set ; no local data exchange is performed between the clients, and only the local model after local data training is exchanged with the server ; The server aggregates all the local models uploaded by the clients The global model is obtained later .

3. The personalized federated learning method based on adaptive local model initialization and double knowledge distillation according to claim 2, characterized in that: In step S2, the following steps are included: S21, set global model with local model the parameter vectors of and the formula for calculating the cosine similarity is as follows: wherein, is the dot product of two vectors, and are the Euclidean norms of and respectively; the cosine similarity is between [-1, 1], the closer the value is to 1, the more similar the two vectors are, the closer the value is to -1, the less similar they are. S22, the initialized local model is as follows: wherein, denotes the adaptive initialization coefficients of the global and local models, which are determined by cosine similarity.

Images