Personalized federal learning method based on self-adaptive local model initialization and double knowledge distillation

Abstract

The invention relates to a personalized federal learning method based on self-adaptive local model initialization and double knowledge distillation, and belongs to the technical field of edge computing. The method comprises the following steps: performing local model initialization on each client according to cosine similarity between a global model and a local model to obtain a self-adaptively initialized local model; respectively performing knowledge distillation on each client according to the global model and the historical local model, and updating parameters of the local model through double knowledge distillation; and the central server receives local models uploaded by all clients participating in training, and obtains a final global model based on dual dynamic aggregation weights of data quality and client contribution degrees. According to the method, the precision and convergence speed of the model can be remarkably improved when the data heterogeneity problem is processed, and good adaptability and robustness are shown in various edge calculation scenes.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology and relates to a personalized federated learning method based on adaptive local model initialization and dual knowledge distillation. Background Art

[0002] Edge computing is a distributed computing framework that pushes computing, data storage and processing capabilities from traditional centralized data centers to edge devices closer to data sources, such as IoT devices and smartphones. This architecture is designed to reduce the latency of data transmission between devices and servers, improve the real-time performance and response speed of the system, and is particularly suitable for application scenarios with high requirements for low latency and real-time performance, such as smart manufacturing, autonomous driving, AR / VR applications and video surveillance. With the continuous advancement of science and technology and the rapid development of Internet technology, the scale and functions of smart mobile devices, wearable devices and IoT devices are constantly growing, and massive amounts of data are generated all the time, causing the data in the world to show an exponential explosion. According to a research report by Cisco, the total amount of global data will reach 402EB by the end of 2021, and human society has entered the era of big data where everything is connected. The rise of big data has brought new opportunities and challenges to all walks of life. In recent years, deep learning has made great progress in computer vision, speech recognition and content recommendation. However, in these fields, deep learning algorithms require massive amounts of data to obtain high-performance models. Although in today's digital age, abundant data provides huge opportunities for the development of deep learning. However, in real-world scenarios, data is often scattered in different places, that is, data exists in isolated forms. If only the data of a single client is used, the trained model will be biased due to incomplete data. And due to competition among various industries and privacy and security issues, each client cannot directly send private data to the central server to train the model. How to effectively utilize the resources of data islands while protecting user privacy has gradually become a new challenge in the industry.

[0003] In 2016, McMahan et al. first proposed the concept of federated learning, which is to conduct collaborative deep learning by multiple parties without uploading local data, ensuring that the original data of the parties involved in federated learning is not exposed. In the process of federated learning, the client trains the model locally and then sends the trained model to the central server. After receiving the client's model in each communication round, the central server aggregates it to obtain a global model and sends the global model back to each client for iterative update. Due to its advantages, federated learning has been widely used in real-world scenarios, such as next word prediction on mobile keyboards of recommendation systems and smart medical data analysis. Therefore, federated learning is a distributed machine learning method with broad application prospects.

[0004] A key challenge in federated learning is that the data of the participants is statistically heterogeneous, that is, the data is non-independent and identically distributed (Non-IID). Non-IID refers to the imbalanced distribution of data on the client, and the common ones are label skewed distribution and feature skewed distribution. Label skewed distribution means that the label distribution of data is different between parties. Feature distribution skew means that the same type of labeled data presents different characteristics. Existing research has shown that differences in the distribution of local data on the client will cause each client model to be updated in different directions. Finally, the global model aggregated by the server may perform poorly when running on the client and cannot be applied to all clients, thereby reducing the motivation of the client to participate in federated learning.

[0005] In recent years, existing research has developed a series of personalized methods to solve the problem of non-IID data. FedBN introduces a batch normalization layer to stabilize data and enhance model convergence, but it has the problem of low robustness. FedPer divides the model into a base layer and a personalized layer to effectively capture statistical heterogeneity, but the risk of overfitting must be considered. The FedProx method controls the distance between the local model and the global model when updating by introducing proximal terms. The selection of the proximal term weight is challenging for the optimal implementation of the regularization effect. SCAFFOLD introduces control variables to reduce gradient variance and improve the robustness of the local model in the environment where data is not independent and identically distributed, but the parameter adjustment process of the control variable is computationally complex. FedProto uses prototypes to represent a class of data and applies prototype aggregation to joint learning under non-IID data. Per-FedAvg broadcasts a shared meta-model to all clients and customizes a unique local loss function for each client. The local loss function is minimized by the gradient descent algorithm to train a personalized model for each client, but the client needs to perform additional calculations locally, which increases the computational burden. FedMeta treats each client as a separate task, replaces the global model with a meta-model, refines and disseminates knowledge acquired from historical tasks, realizes knowledge transfer between tasks, and improves the generalization ability and overall performance of new tasks. FedMD uses distillation technology to transfer the client's model knowledge to the global model, and feeds back the knowledge of the global model to the client to improve model accuracy and overcome the problems of model heterogeneity and data heterogeneity. However, in order to support model aggregation and knowledge distillation, a large amount of computing resources and communication bandwidth are required, which will increase computing and communication costs. Summary of the invention

[0006] In view of this, the object of the present invention is to provide a personalized federated learning method based on adaptive local model initialization and dual knowledge distillation.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A personalized federated learning method based on adaptive local model initialization and dual knowledge distillation, the method comprising the following steps:

[0009] S1. Build a personalized federated learning framework, which includes several clients and a central server;

[0010] S2, on each client, initialize the local model according to the cosine similarity between the global model and the local model, and obtain the local model after adaptive initialization;

[0011] S3. Perform knowledge distillation on each client based on the global model and the historical local model, and update the parameters of the local model through double knowledge distillation.

[0012] S4. The central server receives the local models uploaded by all clients participating in the training, and obtains the final global model based on the dual dynamic aggregation weights of data quality and client contribution.

[0013] Furthermore, in step S1, the personalized federated learning framework is constructed, which includes a server S and N clients, each of which has a private dataset {D1, D2, …, D k ,…,D N}; The clients do not exchange local data with each other, but only exchange the local models {W1,W2,…,W k ,…,W N The server aggregates all local models {W1,W2,…,W k ,…,W N} and then get the global model W g .

[0014] Further, in step S2, the following steps are specifically included:

[0015] S21. Setting the global model W g With the local model W k The parameter vectors are w1 and w2 respectively, then the formula for calculating cosine similarity is as follows:

[0016]

[0017] Among them, w1·w2 is the dot product of two vectors, ‖w1‖ and ‖w2‖ are the Euclidean norms of w1 and w2 respectively; the cosine similarity is between [-1, 1], the closer the value is to 1, the more similar the two vectors are, and the closer the value is to -1, the less similar they are;

[0018] S22, the local model after initialization is as follows:

[0019] W k =δ*W g +(1-δ)*W k

[0020] Among them, δ represents the adaptive initialization coefficient of the global model and the local model, which is determined by the cosine similarity.

[0021] Further, in step S3, the client local model performs knowledge distillation according to the global model and the historical local model, wherein when the client performs local model training, the local data is input into the global model, the initialized current local model and the historical local model, and the logical values ​​predicted by the three models are obtained, which are respectively and The client uses the stochastic gradient descent method to train the local total loss L k Update local model parameters with local objectives

[0022]

[0023] Where η is the learning rate, Denotes the loss function L k The gradient with respect to the parameters.

[0024] Furthermore, client local training includes three different losses, namely cross entropy loss L CE , the loss of global knowledge distillation L GD and the self-knowledge distillation loss L SD , where the cross entropy loss L CE Used to measure the current local model The difference between the local data real category label y is:

[0025]

[0026] Global knowledge distillation loss L GD Used to compare the distance between the output probability distribution of the student model and the output probability distribution of the teacher model. In global knowledge distillation, As a teacher As a student, the global knowledge distillation loss L GD , which is expressed as:

[0027]

[0028] Where τ represents the temperature hyperparameter; KL(·) represents the KL divergence, which is used to measure the difference between two probability distributions;

[0029] Client Self-Knowledge Distillation SD In As a teacher As a student, the self-knowledge distillation loss L SD It is expressed as:

[0030]

[0031] Using different hyperparameters to weigh the strength of global knowledge distillation and self-knowledge distillation, the total local training loss L k It is expressed as:

[0032]

[0033] Among them, α GD and β SD are the parameters for the strength of global knowledge distillation and self-knowledge distillation, respectively.

[0034] Further, in step S4, the following steps are included:

[0035] S41. The data volume weight is defined by the ratio of the data volume of each client to the total data volume:

[0036]

[0037] Among them, n k is the number of samples of client k, N is the total number of clients;

[0038] S42. Define data distribution diversity through Shannon entropy:

[0039]

[0040] Among them, p i is the proportion of the i-th category label in the client data, and C is the total number of categories; the larger the entropy value, the more uniform the data label distribution and the higher the data diversity; conversely, the lower the data diversity;

[0041] S43. Combine the data volume weight and data distribution diversity weight into a comprehensive data quality score:

[0042] Q k =A·N(D k )+B·H(P k )

[0043]

[0044] Among them, A and B are coefficients for adjusting the weights of data volume and data diversity, satisfying A+B=1;

[0045] S44. Perform initial aggregation on the received local models according to data quality to obtain an initial aggregation model:

[0046]

[0047] S45. Define the Euclidean distance between the local model and the global model as the contribution of client k to the federated learning system:

[0048]

[0049] in, and Represents the i-th element of the model parameter vector, d k Represents the Euclidean distance between the client and the global model. The larger the distance, the greater the gap between the local model and the global model, and the lower the contribution to the global model. ∈ is a positive constant to avoid the denominator of 0; The larger it is, the greater the contribution of the corresponding client to the federated learning system;

[0050] By normalizing the client contribution, based on the initial aggregation model, the final global model is obtained according to the dynamic aggregation weight:

[0051]

[0052] in, represents the updated local model of the client, W t+1,g It represents the global model after aggregation, and the superscript t+1 represents the round of global and local communication.

[0053] The beneficial effects of the present invention are:

[0054] The present invention uses the cosine similarity between the received global model and the local model to determine the model update direction through adaptive initialization, and achieves a balance between globality and personalization through global distillation and self-knowledge distillation, effectively alleviating the problem of knowledge forgetting in federated learning. In addition, the server dynamically adjusts the aggregation weight according to the client data quality during aggregation. The present invention can significantly improve the accuracy and convergence speed of the model when dealing with data heterogeneity problems, and shows good adaptability and robustness in a variety of edge computing scenarios.

[0055] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 This is the basic framework diagram of federated learning;

[0058] Figure 2 Schematic diagram of the framework structure of the personalized federated learning method based on adaptive local model initialization and dual knowledge distillation of the present invention. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0060] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0061] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0062] See also Figure 1-2 , a personalized federated learning method based on adaptive local model initialization and dual knowledge distillation.

[0063] Example

[0064] In edge computing scenarios, data statistical heterogeneity has become an important challenge for federated learning. Since the data distribution collected by each edge device is often different, such as user behavior, environmental conditions, etc., traditional federated learning algorithms rely on a unified global model and are difficult to adapt to personalized needs, which leads to a decrease in model performance. To address this challenge, the present invention proposes a personalized federated learning algorithm (FedAaK) with adaptive local model initialization and dual knowledge distillation. This method performs client adaptive local model initialization based on the global model combined with local data. During local training on the client, global model distillation is used to retain global generalized knowledge and self-knowledge distillation is used to retain historical personalized knowledge. This method effectively balances global and personalized performance by setting different parameters to weigh the strength of adaptive initialization and dual knowledge distillation. The dynamic aggregation weight of the server is determined by data quality and contribution. Experimental results on three different data sets show that the proposed algorithm can significantly improve the accuracy and convergence speed of the model when dealing with data heterogeneity problems, and shows good adaptability and robustness in a variety of edge computing scenarios.

[0065] This embodiment first introduces related work.

[0066] Federated learning is a distributed machine learning method that does not require the original data to be centralized in a single location. It performs collaborative learning by sharing model updates among multiple clients, thereby achieving model training while protecting data privacy. The FedAvg algorithm is a classic federated learning algorithm. The main process of classic federated learning is as follows:

[0067] (1) Client selection: Due to the limitations of client status and network conditions, the server will randomly select qualified clients for federated learning;

[0068] (2) Model initialization: The central server initializes the global model and broadcasts the global model to the clients that choose to perform collaborative training;

[0069] (3) Local training: After receiving the global model, the client initializes and updates the local model and trains the local model on the local dataset;

[0070] (4) Model upload: After local training, the client uploads the trained local model to the central server;

[0071] (5) Global aggregation: The central server receives the models uploaded by the client and aggregates them to generate a new round of global models;

[0072] Iterate the training according to the above steps until the model accuracy reaches the preset accuracy or the training rounds reach the preset rounds.

[0073] The classic framework of federated learning is Figure 1 As shown, it contains a server S and N clients {C1,C2,…,C k ,…,C N}, each client has a private dataset {D1,D2,…,D k ,…,D N The clients do not exchange local data with each other, but only exchange local models {W1, W2, …, W k ,…,W N The global model W is obtained by the server aggregating the local models received from all clients, and the client and server perform T rounds of iterations until the model converges.

[0074] When training locally, the client minimizes its local loss function:

[0075]

[0076] Among them, l j (W k ) is the local dataset D k The loss function of sample j in is generally cross entropy loss or mean square error loss.

[0077] The global optimization goal of federated learning is to minimize the sum of the loss functions of all clients:

[0078]

[0079] Among them, |D k | represents the sample size of the local dataset, and |D| represents the total number of global samples.

[0080] In traditional federated learning, it is assumed that each client's data is independent and identically distributed. However, in actual applications, due to differences in data sources, user behaviors, and usage scenarios of different clients, data distribution is often not independent and identically distributed. This means that there may be significant differences between data sets of different clients, including inconsistent feature distribution and label distribution. Due to inconsistent data distribution, the global model performs poorly on some clients and cannot adapt well to all client data, which will reduce the generalization ability of the global model. In addition, if the local client data is small and not widely distributed, it will cause the local model to easily overfit to the local data, making it more difficult for the global model to quickly converge to a stable optimal solution.

[0081] Personalized Federated Learning (PFL) does not require all clients to obtain the same global model in the end, but allows each client to generate a personalized local training model for each client based on the global model. Personalized Federated Learning can effectively address the challenges of slow convergence and reduced accuracy of models on heterogeneous data, and many works have explored and studied it recently. Currently, common personalized federated learning methods are mainly divided into two categories:

[0082] (1) Global model personalization: After the global model aggregation is completed on the server, the client downloads the same global model from the server and performs additional training through the local adaptation step to achieve the purpose of personalized adjustment. This method mainly focuses on the training process on non-IID data. Data-based methods such as data enhancement or client selection can transform non-IID local data into IID scenarios; model-based methods such as regularization, meta-learning, transfer learning, etc. can optimize the global model to adapt to downstream personalized learning tasks.

[0083] (2) Learning personalized models: Aims to complete the personalized task during the training phase. The structure-based method sets different personalized training structures for different data centers through parameter decoupling and knowledge distillation. The similarity-based method constructs classes for clients with similar properties through multi-task learning, model difference, clustering, etc.

[0084] Since models with good performance are usually very large, this will lead to increased computational complexity and storage costs. Therefore, these models are often not deployable on resource-constrained mobile devices. Knowledge Distillation (KD) was introduced as a model compression method. Its main idea is to use the output soft labels of a large pre-trained model (called the teacher model) as the target to guide the training of a smaller model (called the student model), so that the student model not only learns the information of the real labels in the data, but also imitates the output of the teacher model, and then learns the relationship between different labels. This technology is often used to compress complex deep learning models. The purpose is to reduce the size and computational complexity of the model without sacrificing the accuracy of the model as much as possible, so as to facilitate deployment in resource-limited environments. The soft label is the probability that the input belongs to each category, which is calculated by the softmax function σ(·):

[0085]

[0086] Among them, q iis the logit value of the i-th class. The logits of all classes form the logits vector q, which is the output of the last fully connected layer of the model. τ is a temperature hyperparameter used to control the softness or hardness of the soft label. The value of q is between 0 and 1.

[0087] The student model uses hyperparameters α and β to control the cross entropy loss function term L CE and the Kullback-Leibler (KL) divergence loss function term L KL To construct the knowledge distillation loss function:

[0088] L KD =α·L CE (y,q s )+β·τ 2 ·L kL (q t ‖q s )

[0089] Among them, q s and q t Represent the logits output by the student model and the teacher model respectively.

[0090] Based on the above content, this embodiment further introduces in detail a personalized federated learning method based on adaptive local model initialization and dual knowledge distillation of the present invention. Figure 2 As shown, it includes the following steps:

[0091] S1. Build a personalized federated learning framework, which includes several clients and a central server;

[0092] S2, on each client, initialize the local model according to the cosine similarity between the global model and the local model, and obtain the local model after adaptive initialization;

[0093] S3. Perform knowledge distillation on each client based on the global model and the historical local model, and update the parameters of the local model through double knowledge distillation.

[0094] S4. The central server receives the local models uploaded by all clients participating in the training, and obtains the final global model based on the dual dynamic aggregation weights of data quality and client contribution.

[0095] In step S1 of this embodiment, the personalized federated learning framework is constructed, including a server S and N clients, each of which has a private data set {D1, D2, ..., D k ,…,D N The clients do not exchange local data with each other, but only exchange local models {W1, W2, …, Wk ,…,W N The server aggregates all local models {W1,W2,…,W k ,…,W N} and then get the global model W g .

[0096] In step S2 of this embodiment, in traditional federated learning, the server generates a global model by aggregating different client models in a heterogeneous environment, and sends the global model to the client, and the client uses the received global model as the new local model for a new round of local training. However, the generalization effect of this global model on each client is very poor. In order to solve this problem, an adaptive initialization method is used to initialize the local model through the cosine similarity between the global model and the local model to achieve the correction of the model update direction. Calculating the cosine similarity between two models is usually used to compare their parameters or gradients to measure the similarity between them.

[0097] Global Model W g With the local model W k The parameter vectors are w1 and w2 respectively, then the formula for calculating cosine similarity is as follows:

[0098]

[0099] Among them, w1·w2 is the dot product of two vectors, ‖w1‖ and ‖w2‖ are the Euclidean norms of w1 and w2, that is, their lengths. The value of cosine similarity is between [-1, 1]. The closer the value is to 1, the more similar the two vectors (model parameters) are, and the closer the value is to -1, the less similar they are.

[0100] The adaptive initialization coefficient δ of the global model and the local model is determined by cosine similarity, and the initialized local model is shown as follows:

[0101] W k =δ*W g +(1-δ)*W k

[0102] In step S3 of this embodiment, dual knowledge distillation refers to the client local model performing knowledge distillation based on the global model and the historical local model. The client local training includes three different losses, namely, cross entropy loss L CE , the loss of global knowledge distillation L GD and the self-knowledge distillation loss L SD When the client performs local model training, the local data is input into the global model, the initialized current local model, and the historical local model, and the logical values ​​(logits) predicted by the three models are obtained, which are and

[0103] Use cross entropy loss to measure the current local model The difference between the local data real category label y is:

[0104]

[0105] Use KL divergence to compare the distance between the output probability distribution of the student model and the output probability distribution of the teacher model. In global knowledge distillation, As a teacher As a student, the loss L for global knowledge distillation GD As shown below:

[0106]

[0107] Here, τ represents the temperature hyperparameter and KL(·) represents the KL divergence, which is a measure of the difference between two probability distributions.

[0108] In client-side self-knowledge distillation, As a teacher As a student, the loss L for self-knowledge distillation SD As shown below:

[0109]

[0110] For all the above loss functions, different hyperparameters are used to weigh the strength of global knowledge distillation and self-knowledge distillation, and the total local training loss is:

[0111]

[0112] Among them, α GD and β SD are the parameters for the strength of global knowledge distillation and self-knowledge distillation, respectively.

[0113] The client uses the stochastic gradient descent (SGD) method, according to the L k Update local model parameters with local objectives

[0114]

[0115] Where η is the learning rate, Denotes the loss function L k The gradient with respect to the parameters.

[0116] In step S4 of this embodiment, in traditional federated learning FedAvg, the server receives the local models uploaded by all clients participating in this round of training, and uses federated average aggregation to obtain a new round of global models. FedAvg uses the proportion of local data of each client to the total amount of data for weighting. However, the data characteristics and category distribution of the clients may vary greatly. Using average aggregation will ignore these differences, and the model may not generalize well. The computing power of different clients varies greatly. Using average aggregation may dilute the model updates of some clients with greater contributions, thereby reducing the convergence efficiency of the global model.

[0117] Therefore, the present invention adopts dual dynamic aggregation weights based on data quality and client contribution, including:

[0118] Data quality is defined by a comprehensive analysis of data volume and data diversity. In order to balance the data volume differences between different clients, the data volume weight is defined by the proportion of each client's data volume to the total data volume:

[0119]

[0120] Among them, n k is the number of samples of client k, and N is the total number of clients.

[0121] The diversity of data label distribution reflects whether the client data covers enough categories. The present invention defines data distribution diversity through Shannon entropy:

[0122]

[0123] Among them, p i is the proportion of the i-th category label in the client data, and C is the total number of categories. The larger the entropy value, the more evenly the data label distribution is, and the higher the data diversity is; conversely, the more the data tends to certain categories, the lower the data diversity is.

[0124] These two indicators are weighted and combined into a comprehensive data quality score:

[0125] Q k =A·N(D k )+B·H(P k )

[0126]

[0127] Among them, A and B are coefficients for adjusting the weights of data volume and data diversity, satisfying A+B=1.

[0128] The received local models are initially aggregated according to the data quality to obtain the initial aggregated model:

[0129]

[0130] Define the Euclidean distance between the local model and the global model as the contribution of client k to the federated learning system:

[0131]

[0132] in, and Represents the i-th element of the model parameter vector, d k Represents the Euclidean distance between the client and the global model. The larger the distance, the greater the gap between the local model and the global model, and the lower the contribution to the global model. ∈ is to prevent d k is 0, so a small positive number is added; The larger it is, the greater the contribution of the corresponding client to the federated learning system.

[0133] By normalizing the client contribution, based on the initial aggregation model, the final global model is obtained according to the dynamic aggregation weight:

[0134]

[0135] in, represents the updated local model of the client, W t+1,g It represents the global model after aggregation, and the superscript t+1 represents the round of global and local communication.

[0136] This embodiment carried out experiments according to the above method.

[0137] This example uses three different data sets for experiments:

[0138] (1) The MNIST dataset has a total of 70,000 samples, each of which is a 28*28 pixel grayscale image of a handwritten digit. There are 10 types of samples, from digits 0 to 9.

[0139] (2) The Cifar10 dataset has a total of 60,000 samples, each of which is a 32*32 pixel RGB image (color image), and each RGB image must be divided into 3 channels (R channel, G channel, B channel). These 60,000 samples are divided into 50,000 training samples and 10,000 test samples. There are 10 categories of samples, and the label values ​​are distinguished by 0-9. Specific labels include: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

[0140] (3) The Cifar100 dataset and the Cifar10 dataset have the same number of samples, both containing 60,000 samples, and each sample is also a 32*32 pixel RGB color image with 3 channels. The difference is that the CIFAR-100 dataset is divided into 100 categories, each containing 600 samples. Among them, each category has 500 training samples and 100 test samples. In addition, the 100 subcategories in CIFAR-100 are summarized into 20 large categories. Each image is accompanied by a "fine" label, indicating the subcategory it belongs to, and a "coarse" label, indicating the large category it belongs to.

[0141] For each data set, this embodiment uses Dirichlet distribution Dir(α) to construct non-independent and identically distributed data sets across clients when conducting experimental verification. The parameter α is used to control the degree of non-independent and identically distributed data. The smaller the α parameter is, the higher the degree of non-independent and identically distributed data is. This embodiment uses pathological non-independent and identically distributed data partitioning. The local data of each client contains at most s categories given in advance, where MNIST data set and Cifar10 data set have s=2 and Cifar100 data set has s=10. The data is divided into training set and test set in a ratio of 0.75:0.25.

[0142] In this embodiment, the method and comparison method of the present invention both use the stochastic gradient descent method to minimize the loss function. By default, the learning rate of stochastic gradient descent is set to lr=0.005, the new element momentum and weight decay are set to 0.9 and 0.00001 respectively, the data batch size is 10, the number of clients is set to N=100 by default, the client participation rate participating in the training is set to ρ=0.1, the number of local updates is set to E=5 by default, and the number of communication rounds is T=200.

[0143] For fair comparison, this method and all the baseline methods use the same network architecture, convolutional neural network (CNN), and this example uses the NVIDIA GeForce RTX 3060Ti GPU on Pytorch for model training. This example compares the method FedAaK of the present invention with other federated learning methods, including FedAvg, FedProx, and FedRep.

[0144] The experimental results are shown in Table 1, which shows the test accuracy of the four federated learning methods on three datasets: MNIST, Cifar10 and Cifar100.

[0145] Table 1

[0146]

[0147] The experimental results show that the method of the present invention achieves high accuracy in all degrees of non-IID scenarios. It is at least 2% higher than the second best on Cifar10 and Cifar100. This shows that the FedAaK method based on adaptive initialization aggregation and local dual knowledge distillation training can effectively balance personalization and global knowledge, greatly improving the performance of the local model. FedAak can be used to process various types of non-IID image data. Its purpose is to provide a personalized model for each client while ensuring data privacy, thereby improving the performance and adaptability of the model.

[0148] The personalized federated learning method based on adaptive local model initialization and dual knowledge distillation proposed in the present invention can be applied to the field of medical image analysis. In medical image analysis, there may be significant differences in data distribution in different hospitals, such as different equipment, imaging technology and patient population differences, which will lead to different applicability and accuracy of the model in different hospitals. Personalized federated learning can take advantage of this difference and improve performance by fine-tuning the model separately in each hospital. Select a public medical image dataset, such as CheXpert or ISIC dataset. These datasets contain images from different hospitals or equipment and can be regarded as datasets of different clients. In traditional federated learning, all clients (hospitals) share a global model; in the proposed personalized federated learning FedAaK, each client performs personalized fine-tuning on the basis of the global model according to its own data distribution. The accuracy of the personalized model on a specific client is improved. Through personalized learning, the model can achieve relatively balanced performance on different clients, rather than being overly superior or underperforming in certain specific hospitals. Personalized federated learning can make the model better adapt to the unique data distribution of each hospital without reducing its applicability due to excessive sharing.

[0149] The personalized federated learning method based on adaptive local model initialization and dual knowledge distillation proposed in the present invention can also be applied to the field of personalized recommendation. Recommendation systems are often used for personalized content recommendations for different users. The traditional federated learning model generates a general model, but due to individual differences among users, personalized federated learning can better capture the specific preferences of each user. The public MovieLens dataset is used, which contains user rating data for movies. The rating data of each user represents a local data distribution, reflecting his or her preferences. There are three commonly used indicators for recommendation systems, namely: mean square error (MSE): the error between the evaluation prediction score and the actual score; hit rate: the proportion of content that actual users like in the recommended content; diversity: the diversity of recommended content, measuring the degree of personalization. The comparative experimental results of traditional federated learning and the proposed personalized federated learning algorithm prove that the personalized federated learning algorithm has a lower mean square error, a higher hit rate, and a richer diversity of recommendations on different user groups, providing empirical support for the advantages of the personalized federated learning algorithm in practical application scenarios.

[0150] The present invention proposes a new personalized federated learning method of adaptive initialization aggregation and local dual knowledge distillation training, FedAaK. This method uses the cosine similarity between the received global model and the local model to determine the model update direction through adaptive initialization, and achieves a balance between globality and personalization through global distillation and self-knowledge distillation, effectively alleviating the problem of knowledge forgetting in federated learning. In addition, the server dynamically adjusts the aggregation weight according to the client data quality during aggregation. A large number of experiments have shown that the method of the present invention is superior to other compared methods, proving the effectiveness of the method.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A personalized federated learning method based on adaptive local model initialization and dual knowledge distillation, characterized by: The method comprises the following steps: S1. Build a personalized federated learning framework, which includes several clients and a central server; S2, on each client, initialize the local model according to the cosine similarity between the global model and the local model, and obtain the local model after adaptive initialization; S3. Perform knowledge distillation on each client based on the global model and the historical local model, and update the parameters of the local model through double knowledge distillation. S4. The central server receives the local models uploaded by all clients participating in the training, and obtains the final global model based on the dual dynamic aggregation weights of data quality and client contribution.

2. The personalized federated learning method based on adaptive local model initialization and dual knowledge distillation according to claim 1, characterized in that: In step S1, the personalized federated learning framework is constructed, which includes a server S and N clients. Each client has a private dataset {D1, D2, ..., D k , ..., D N }; The clients do not exchange local data with each other, but only exchange the local models {W1, W2, ..., W k , ..., W N }; The server aggregates all local models {W1, W2, ..., W k , ..., W N } and then get the global model W g .

3. The personalized federated learning method based on adaptive local model initialization and dual knowledge distillation according to claim 2, characterized in that: In step S2, the following steps are specifically included: S21. Setting the global model W g With the local model W k The parameter vectors are w1 and w2 respectively, then the formula for calculating cosine similarity is as follows: Among them, w1·w2 is the dot product of two vectors, ||w1|| and ||w2|| are the Euclidean norms of w1 and w2 respectively; the cosine similarity is between [-1, 1], the closer the value is to 1, the more similar the two vectors are, and the closer the value is to -1, the less similar they are; S22, the local model after initialization is as follows: W k =δ*W g +(1-d)*W k Among them, δ represents the adaptive initialization coefficient of the global model and the local model, which is determined by the cosine similarity.

4. The personalized federated learning method based on adaptive local model initialization and dual knowledge distillation according to claim 3, characterized in that: In step S3, the client local model performs knowledge distillation according to the global model and the historical local model. When the client performs local model training, the local data is input into the global model, the initialized current local model and the historical local model, and the logical values ​​predicted by the three models are obtained, which are and The client uses the stochastic gradient descent method to train the local total loss L k Update local model parameters with local objectives Where η is the learning rate, Denotes the loss function L k The gradient with respect to the parameters.

5. The personalized federated learning method based on adaptive local model initialization and dual knowledge distillation according to claim 4, characterized in that: Client local training includes three different losses: cross entropy loss L CE , the loss of global knowledge distillation L GD and the self-knowledge distillation loss L SD , where the cross entropy loss L CE Used to measure the current local model The difference between the local data real category label y is: Global knowledge distillation loss L GD Used to compare the distance between the output probability distribution of the student model and the output probability distribution of the teacher model. In global knowledge distillation, As a teacher As a student, the global knowledge distillation loss L GD , which is expressed as: Where τ represents the temperature hyperparameter; KL(·) represents the KL divergence, which is used to measure the difference between two probability distributions; Client Self-Knowledge Distillation SD In As a teacher As a student, the self-knowledge distillation loss L SD It is expressed as: Using different hyperparameters to weigh the strength of global knowledge distillation and self-knowledge distillation, the total local training loss L k It is expressed as: Among them, α GD and β SD are the parameters for the strength of global knowledge distillation and self-knowledge distillation, respectively.

6. The personalized federated learning method based on adaptive local model initialization and dual knowledge distillation according to claim 4, characterized in that: In step S4, the following steps are included: S41. The data volume weight is defined by the ratio of the data volume of each client to the total data volume: Among them, n k is the number of samples of client k, N is the total number of clients; S42. Define data distribution diversity through Shannon entropy: Among them, p i is the proportion of the i-th category label in the client data, and C is the total number of categories; the larger the entropy value, the more uniform the data label distribution and the higher the data diversity; conversely, the lower the data diversity; S43. Combine the data volume weight and data distribution diversity weight into a comprehensive data quality score: Q k =A·N(D k )+B·H(P k ) Among them, A and B are coefficients for adjusting the weights of data volume and data diversity, satisfying A+B=1; S44. Perform initial aggregation on the received local models according to data quality to obtain an initial aggregation model: S45. Define the Euclidean distance between the local model and the global model as the contribution of client k to the federated learning system: in, and Represents the i-th element of the model parameter vector, d k Represents the Euclidean distance between the client and the global model. The larger the distance, the greater the gap between the local model and the global model, and the lower the contribution to the global model. ∈ is a positive constant to avoid the denominator of 0; The larger it is, the greater the contribution of the corresponding client to the federated learning system; By normalizing the client contribution, based on the initial aggregation model, the final global model is obtained according to the dynamic aggregation weight: in, represents the updated local model of the client, W t+1,g It represents the global model after aggregation, and the superscript t+1 represents the round of global and local communication.

Images