A communication-efficient privacy-preserving personalized federated learning method

Abstract

The application discloses a communication-efficient privacy-preserving personalized federated learning method. The application studies a personalized federated learning based on feature fusion mutual learning, which can realize communication-efficient personalized learning by interactive training of shared models, private models and fusion models on the client. Specifically, only the shared model is shared with the global model to reduce communication cost, while the private model can be personalized, and the fusion model can adaptively fuse local knowledge and global knowledge at different stages. Secondly, in order to further reduce the communication cost and enhance the privacy of the gradient, the application designs a privacy protection method based on gradient compression. The method can well realize privacy protection and lightweight compression by constructing a chaotic encryption circulant measurement matrix. In addition, the application also proposes an adaptive iterative hard threshold algorithm based on sparsity to improve flexibility and reconstruction performance.

Description

Technical Field

[0001] This invention belongs to the fields of privacy protection and federated learning, specifically relating to a privacy-preserving personalized federated learning method with high communication efficiency. Background Technology

[0002] Today, with the improvement of device computing power, many excellent models have been developed to extract latent patterns from the massive amounts of data generated daily, which has driven the rapid development of deep learning (DL). However, traditional centralized deep learning suffers from communication congestion due to massive data transmission, making it impossible to effectively guarantee service quality. Moreover, privacy leaks during data collection are a major threat. Federated learning (FL), as a promising paradigm in distributed deep learning, has made significant progress in terms of privacy protection and communication overhead.

[0003] While functional programming (FL) offers more advantages than centralized deep learning (DL), the model update process still incurs significant communication overhead, especially when clients have large-scale models. Furthermore, many privacy-preserving, communication-efficient FL schemes employ redundant encryption algorithms, leading to further computational costs. To address these issues, current research focuses on three main approaches.

[0004] The first type of approach improves communication efficiency by reducing the number of communication rounds, selecting some participating clients, or minimizing communication time. While this approach helps improve communication efficiency, it does not fundamentally reduce the data size of the model parameters; that is, a massive number of parameters still exist in a single round of communication, especially when the client model is large.

[0005] The second type of approach reduces the amount of data uploaded or downloaded using various compression methods. This approach does indeed save communication costs by reducing communication bits. However, it cannot achieve efficient training performance when the data is non-IID. Furthermore, the assumption that all client models are homogeneous limits client personalization.

[0006] To overcome the challenges posed by the heterogeneity of datasets and models, a third type of communication-efficient approach was proposed, which can improve model performance under heterogeneous datasets and models. This personalized approach adapts well to the distribution of Non-IID data and the heterogeneity of models, and provides many insights into the clever use of KD. However, it does not reasonably integrate local and global knowledge, making it difficult to achieve stable model training for clients with Non-IID datasets.

[0007] From the three schemes mentioned above, especially the most representative and advanced research in the third category, it can be seen that KD not only helps reduce communication costs but also gives FL (Flexible Interchange Function) good heterogeneity fault tolerance. However, a small model does not necessarily mean fewer parameters. Some schemes only transmit the logit output of the last layer to further reduce communication traffic, but this can lead to insufficient knowledge transfer. At the same time, it is also crucial to enable clients to better absorb knowledge from global knowledge without conflicting with local knowledge. Furthermore, although efficient communication FLs with privacy-preserving methods have been developed in the first two categories, they typically introduce additional encryption techniques, which increases computational costs and places higher demands on hardware. Therefore, it is necessary to design a more efficient and personalized FL with privacy protection. Summary of the Invention

[0008] Purpose of the invention: In view of the above-mentioned defects in the existing technology, the purpose of this invention is to design a privacy-preserving personalized federated learning method with high communication efficiency, which enables the client to complete personalized learning with low communication cost, overcomes the performance loss problem caused by model heterogeneity and data heterogeneity, achieves privacy protection and lightweight compression, and improves the flexibility of reconstruction.

[0009] Technical Solution: To achieve the above objectives, this invention proposes a communication-efficient, privacy-preserving, personalized federated learning method, which includes the following steps:

[0010] (1) Based on the different privacy protection needs of different clients, a personalized federated learning network model with privacy protection is designed. The model consists of privacy clients, public clients and a central server.

[0011] (2) All clients learn knowledge through their own neural networks and combine the knowledge learned from the local dataset with global knowledge to achieve personalized learning. After the client completes training of the local neural network using the local dataset, the privacy client compresses the gradients of its neural network with the gradients of the public client's neural network in groups to reduce communication overhead and protect gradient information. The privacy client uploads the compressed gradients to the central server for global aggregation.

[0012] (3) The central server decompresses and reconstructs the compressed gradients uploaded by the client, performs global aggregation on the reconstructed gradients, updates the global neural network, and finally distributes the model parameters of the global neural network to each client.

[0013] Furthermore, step (1) includes the following: The central server is configured with N private clients and M public clients. Private clients i∈{1,2,…N} are randomly selected from the public group {1,2,…M}. iA new group is formed from 10 public clients, and all clients are divided into N groups. Privacy clients and public clients train their neural networks using their own local datasets and collaborate with each other through federated learning by sharing neural network gradients with the assistance of a central server.

[0014] Furthermore, step (2) includes the following:

[0015] Step 2.1), client i utilizes its local dataset D i The neural network was trained, which consisted of three sub-models: a private model, a shared model, and a fusion model. The private model extracted features from the local dataset. Feature extraction from shared models Will After bridging layer and After concatenation, the data is input into a fusion model to complete feature fusion and obtain fused features. The three sub-models learn from each other through knowledge distillation to complete local training. The loss of each sub-model consists of two parts: one is the cross-entropy loss between the predicted hard target and the ground truth. Another is the Kullback-Leibler (KL) divergence of soft targets among sub-models. Design a time-independent balance weight α pair Scaling is applied, and this weight gradually increases from 0 to 1 via a Gaussian process, i.e.:

[0016]

[0017] Where t and t stop These represent the current iteration and the stopping rum-up iteration count of α, respectively. When the entire training process reaches a preset stable state, the value of α is fixed at 1. The training losses for the private model, shared model, and fusion model are... They are defined as follows:

[0018]

[0019]

[0020]

[0021] in, and represent the hard objectives of the private model, shared model, and fusion model, respectively. These are their soft targets, where T is the distillation temperature that controls the softness, and when T = 1, soft() is the softmax function;

[0022] In the formula It is a weighted set of soft objectives for private and shared models, defined as:

[0023]

[0024] Where β∈[0.5,1] is the set tradeoff factor, and the rum-up method is used to dynamically adjust (1-β) from 0.5 to 1 over time. It is and The weighted integration feature;

[0025] Based on the above analysis, the total loss model for local client i is defined as follows:

[0026]

[0027] The gradients of the shared model are uploaded to a central server for global aggregation. i for:

[0028]

[0029] in, This represents the model parameters shared by client i with respect to sample b in the local dataset, n. i It is D i The number of samples, This represents the training loss of the shared model in client i with respect to sample b in the local dataset;

[0030] The private model and the fusion model are updated only locally, and the specific process is as follows:

[0031]

[0032]

[0033] Where ε is the learning rate of the private model and the fusion model, These are the parameters of the private model and the fused model in client i, respectively. These are the model parameters of the private model and the fusion model in client i with respect to sample b in the local dataset, respectively.

[0034] Step 2.2): When all clients have completed local training, the privacy client will perform group aggregation of gradients, then the gradient g of privacy client i will be... i Updated to:

[0035]

[0036] in, This represents the gradient of public client k, which is in the same group as privacy client i.

[0037] Step 2.3) will be applied to the aggregated gradient g. i Compress it.

[0038] Furthermore, the local dataset uses either the MNIST or CIFAR10 classification dataset.

[0039] Furthermore, the specific method for step 2.3) is as follows:

[0040] In the privacy client i, assume the gradient g of the j-th layer i,j The dimension is D i,j ×1, using the measurement matrix to measure the gradient g i Compress and measure matrix Φ i The generation method is as follows;

[0041] 1) Generate a mapping of length D based on Chebyshev. i,j ×d chaotic sequence The (n+1)th element z of the chaotic sequence n+1 The generated expression is:

[0042] z n+1 =cos(q·arccosz) n )

[0043] Where q is the Chebyshev order, z n Let q be the nth element of the chaotic sequence, ∈ [3,5], with initial value z0 ∈ [0,1]. Select a sampling interval d and perform interval sampling on the above chaotic sequence. Let this sequence be the j-th layer measurement matrix. The first line, The standard deviation is σ i,j ;

[0044] 2) For the product generated in step 1), Execute C i,j -1 left circular shifts to generate dimension C i,j Φ i,j ,Right now:

[0045]

[0046] Among them, C i,j It is a constant and satisfies C i,j <<D i,j , for Φ i,j Normalization is performed, that is:

[0047]

[0048] When a∈(0,1) is a constant, σ i,j ×C i,j Let be the normalization coefficient, then ||Φ i,j ||2<1;

[0049] 3) Repeat the above two steps to generate gradient g. i Measurement matrix Φ for each layer i,j The final measurement matrix Φ is formed. i , where Φ i It satisfies the restricted isometric property;

[0050] Privacy client i based on Φ i Perform gradient compression, the compressed gradient Represented as:

[0051]

[0052] Among them, A i =Φ i DCT i For the perception matrix, DCT i Let g′ represent the sparse basis matrix after the Discrete Cosine Transform (DCT). i It is a sparse coefficient vector.

[0053] Furthermore, the specific method for step (3) is as follows:

[0054] After receiving all compressed gradients, the central server reconstructs them, performs global aggregation of the reconstructed gradients, and updates the global neural network.

[0055] The reconstruction goal is to reconstruct based on a given compression gradient. and perception matrix A i Solving for the sparse coefficient vector g′ i The reconstruction is achieved by optimizing the following L2 norm-based minimization problem:

[0056]

[0057] st‖g′ i ||0≤S i

[0058] The constraint condition states that the number of non-zero values ​​in the reconstructed gradient does not exceed the sparsity S. i S i The specific estimation method is defined as follows:

[0059]

[0060] Among them, B i G representsi The number of elements in the solution and the number of iterations in the entire solution process are closely related to the sparsity, i.e., the number of iterations is... symbol Indicates rounding down;

[0061] During the optimization process, g′ i Updated to:

[0062]

[0063] Where l represents the l-th iteration l≤S i , It is a nonlinear operator function that will Except for the one with the largest absolute value All elements except the first element are set to zero, defined as:

[0064]

[0065] Where λ represents the pair of pairs ... After sorting by the absolute values ​​of the elements, the first... One maximum value;

[0066] go through The optimal sparse coefficient vector is obtained after the second iteration. The reconstructed gradient is then...

[0067] Global gradient aggregation is performed, and the global neural network is updated. The update process of the global neural network is as follows:

[0068]

[0069] Where, ω t These are the parameters of the global neural network, where η represents the learning rate of the global model, and the total sample size is... n i It is D i The number of samples.

[0070] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0071] 1. This invention designs a personalized federated learning method based on feature fusion and mutual learning, which can provide efficient communication services for clients under Non-IID datasets and heterogeneous models. Clients can adaptively integrate local and global knowledge at different training stages and cleverly learn the feature extraction and classification capabilities of the global model. Simultaneously, each client extracts a small-scale shared model to participate in aggregation, which effectively alleviates communication pressure.

[0072] 2. This invention proposes a gradient compression privacy protection method that is easier to implement in hardware. Simultaneously, a novel measurement matrix is ​​designed by combining the ideas of chaotic mapping and cyclic matrices. More importantly, the protection strength can be improved by frequently changing the key at a low cost, preventing the key from being cracked by attackers. In particular, CS, as a compression technique, can significantly reduce the communication traffic between the client and the central server with small errors;

[0073] 3. This invention proposes a sparsity-based adaptive iterative hard thresholding (SAIHT) algorithm to minimize the error caused by gradient compression. It can adaptively adjust the number of iterations according to the sparsity of the gradient, thereby improving the reconstruction performance. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of a privacy-preserving personalized federated learning network model designed in this invention;

[0075] Figure 2 This is a schematic diagram of the personalized federated learning framework with high communication efficiency and privacy protection designed in this invention.

[0076] Figure 3 This is a comparison chart of the accuracy of the method of the present invention with other methods at different compression ratios;

[0077] Figure 4 This is a performance comparison chart of the SAIHT algorithm of this invention with other reconstruction algorithms;

[0078] Figure 5 This is a comparison chart of the tag reconstruction accuracy of the method of the present invention and other methods under different attacks;

[0079] Figure 6 This is a comparison chart of the reasoning performance of the method of this invention with that of an honest but curious central server under other methods. Detailed Implementation

[0080] The technical solution of the present invention will be further described in detail below with reference to examples and accompanying drawings, so as to make the technical solution of the present invention easier to understand and master.

[0081] This invention discloses a privacy-preserving personalized federated learning method with high communication efficiency, comprising the following steps:

[0082] (1) Based on the different privacy protection needs of different clients, a personalized FL network model with privacy protection is designed, which includes privacy clients, public clients and central servers;

[0083] like Figure 1 As shown, there are 10 clients within a central server, of which N = 5 are private clients and M = 5 are public clients. Furthermore, the private clients i ∈ {1, 2, ..., N} are randomly selected from the public group {1, 2, ..., M}. A new group is formed, so all clients are divided into 5 groups. Each client trains its neural network using its own local dataset and collaborates with the central server by sharing neural network gradients through federated learning.

[0084] (2) All clients learn knowledge through their own neural networks and combine the knowledge learned from their local datasets with global knowledge to achieve personalized learning. Specifically, after a client trains its local neural network using its local dataset, the privacy client groups and aggregates its neural network gradients with those of the public client's neural network and then performs gradient compression to reduce communication overhead and protect gradient information. Finally, the privacy client uploads the compressed gradients to the central server for global aggregation.

[0085] The specific steps are as follows:

[0086] Step 2.1) Client i utilizes its local dataset (e.g., classification datasets such as MNIST and CIFAR10) D i The neural network is trained by means of three sub-models: a private model, a shared model, and a fusion model. First, the private model extracts features from the local dataset. Feature extraction from shared models Next, After bridging layer and After concatenation, the data is input into a fusion model to complete feature fusion and obtain fused features. Finally, the three sub-models complete local training by learning from each other based on knowledge distillation (KD). The loss of each sub-model consists of two parts: one is the cross-entropy loss between the predicted hard target and the ground-truth. Another is the Kullback-Leibler (KL) divergence of soft targets among sub-models. To balance the impact of these two parts on training, a time-independent balancing weight α was designed to... Scaling is applied, and this weight gradually increases from 0 to 1 via a Gaussian process, i.e.:

[0087]

[0088] Where t and t stopThese represent the current iteration and the stopping rum-up iteration of α, respectively. When the entire training process is relatively stable, the value of α is fixed at 1. The training losses of the private model, shared model, and fusion model are respectively... They are defined as follows:

[0089]

[0090]

[0091]

[0092] in, and represent the hard objectives of the private model, shared model, and fusion model, respectively. These are their soft targets, where T is the distillation temperature that controls the softness, and when T = 1, soft() is the softmax function;

[0093] In the formula It is a weighted set of soft objectives for private and shared models, defined as:

[0094]

[0095] Where β∈[0.5,1] is the set trade-off factor, and the rum-up method is also used here to dynamically adjust (1-β) from 0.5 to 1 over time. for and The weighted integration features.

[0096] When α and β are fixed (α = 1, β = 0.5), the private and shared models have the same guiding ability for the fusion model, but the shared model performs poorly in the initial stage, which may weaken the learning effect of the fusion model. Simultaneously, a large KL value in the early stages can cause the fusion model to deviate from the correct learning direction. Therefore, the learning result with α = 1 and β = 0.5 is the worst. When the values ​​of α or β are dynamically adjusted with training time, knowledge can be reasonably transferred between models, and the model performance can be improved to varying degrees. Moreover, as can be seen from the embedding graph, this method is the most stable during training. When T = 1, soft() is the softmax function. Increasing the temperature within a certain range softens the labels to obtain more dark knowledge and expand the exploration range during training; therefore, the accuracy of T = 3 is higher than that of T = 1. However, when the temperature is too high, over-softening occurs, making it impossible for the model to clearly identify the correct knowledge and disrupting the learning direction. Therefore, in subsequent experiments, T is set to 3.

[0097] Based on the above analysis, the total loss model for local client i is defined as follows:

[0098]

[0099] The gradients of the shared model are uploaded to a central server for global aggregation. i for:

[0100]

[0101] in, This represents the model parameters shared by client i with respect to sample b in the local dataset, n. i It is D i The number of samples, This represents the training loss of the shared model in client i with respect to sample b in the local dataset;

[0102] The private model and the fusion model are updated only locally, and the specific process is as follows:

[0103]

[0104]

[0105] Where ε is the learning rate of the private model and the fusion model, These are the parameters of the private model and the fused model in client i, respectively. These are the model parameters of the private model and the fusion model in client i with respect to sample b in the local dataset, respectively.

[0106] Step 2.2) When all clients have completed local training, the privacy client will perform group aggregation of gradients, then the gradient g of privacy client i will be... i Updated to:

[0107]

[0108] in, This represents the gradient of public client k, which is in the same group as privacy client i.

[0109] Step 2.3) To further reduce communication overhead and resist inference attacks, the aggregated gradient g will be... i Compress it.

[0110] First, we need to analyze g. i Sparsification is performed as follows:

[0111] g i ←DCT i g′ i ,

[0112] Among them, DCT i This represents the sparse basis matrix after DCT.

[0113] Next, layer-by-layer CS is used to reduce the computational complexity of compression and reconstruction. In the privacy client i, assume the gradient g of the j-th layer... i,j The dimension is D i,j ×1, using the measurement matrix to measure the gradient g i Compress and measure matrix Φ i The generation method is as follows:

[0114] 1) Generate a mapping of length D based on Chebyshev. i,j The chaotic sequence [z0, z] of ×d d ,…,z (Di,j-1)×d The (n+1)th element z of the chaotic sequence n+1 The generated expression is:

[0115] z n+1 =cos(q·arccosz) n )

[0116] Where q is the Chebyshev order, z n Let q be the nth element of the chaotic sequence, ∈ [3,5], with initial value z0 ∈ [0,1]. Choose a suitable sampling interval d and perform interval sampling on the above chaotic sequence. This sequence is set as the j-th layer measurement matrix. The first line, The standard deviation is σ i,j .

[0117] 2) For the product generated in step 1), Execute C i,j -1 left circular shifts to generate dimension C i,j Φ i,j ,Right now:

[0118]

[0119] Where C i,j It is a constant and satisfies C i,j <<D i,j To facilitate reconstruction, for Φ i,j Normalization is performed, that is:

[0120]

[0121] When a∈(0,1) is a constant, σ i,j ×C i,j Let be the normalization coefficient, then ||Φ i,j ||2<1.

[0122] 3) Repeat the above two steps to obtain the gradient g. i Measurement matrix Φi , where Φ i It satisfies the restricted isometric property.

[0123] Finally, the privacy client i is based on Φ i Perform gradient compression, the compressed gradient Represented as

[0124]

[0125] Among them, A i =Φ i DCT i Let g′ be the perception matrix. i It is a sparse coefficient vector.

[0126] From the above expression for generating chaotic sequences, it can be seen that z n The value range of is [-1, 1], which may change the sign of the gradient, thus confusing attackers launching inference attacks. Therefore, the measurement matrix Φ i It can be used as a key to encrypt gradients.

[0127] During training, assume the total number of iterations I = 500, the distillation temperature T = 3, and the number of rum-up iterations at which the rum-up stops t. stop =200. In the model, the optimizer SGD is selected, with both local and global learning rates set to ε = η = 0.001. Based on experience, the Chebyshev order q = 3.99 and the sampling interval d = 5 are set, which will achieve a good chaotic state. Furthermore, the average accuracy of all client-side fusion models is calculated as the evaluation object for the final model accuracy in the scheme.

[0128] (3) The central server decompresses the compressed gradients received from the clients. Next, it globally aggregates the reconstructed gradients and updates the global neural network. Finally, it distributes the parameters of the global neural network to each client.

[0129] The reconstruction goal is to reconstruct based on a given compression gradient. and perception matrix A i Solving for the sparse coefficient vector g′ i The reconstruction is achieved by optimizing the following L2 norm-based minimization problem:

[0130]

[0131] st‖g′ i ||0≤S i

[0132] The constraint condition states that the number of non-zero values ​​in the reconstructed gradient does not exceed the sparsity S. i Si The specific estimation method is defined as follows:

[0133]

[0134] Among them, B i G represents i The number of elements in the solution. Specifically, the number of iterations in the entire solution process is closely related to the sparsity, i.e., the number of iterations is... symbol This indicates rounding down to the nearest integer.

[0135] During the optimization process, g′ i Updated to:

[0136]

[0137] Where l represents the l-th iteration l≤S i , It is a nonlinear operator function that will Except for the one with the largest absolute value All elements except the first element are set to zero, defined as:

[0138]

[0139] Where λ represents the pair of pairs ... After sorting by the absolute values ​​of the elements, the first... One maximum value;

[0140] go through The optimal sparse coefficient vector is obtained after the second iteration. The reconstructed gradient is then...

[0141] Finally, the gradients are aggregated globally and the global neural network is updated. The update process of the global neural network is as follows:

[0142]

[0143] Where, ω t These are the parameters of the global neural network, where η represents the learning rate of the global model, and the total sample size is...

[0144] Finally, the specific implementation of the method of this invention verifies the advantages of this invention in terms of communication cost and privacy protection, that is, it can better meet the performance and privacy protection needs of clients participating in federated learning. Figure 3The accuracy of different schemes at different compression ratios is described. As the compression ratio increases from 0 to 0.8, the model accuracy of all three schemes decreases, but even at a compression ratio of 0.8, the accuracy decreases by a maximum of only 3.76%. This is because most gradients in the model are close to zero, especially during the stable training phase. The "Sparsification" scheme performs coarse-grained sparsification of gradients without reconstruction, ignoring some important gradients, thus resulting in the lowest accuracy. Notably, even at a compression ratio as high as 0.8, the accuracy of the present invention is only sacrificed by 2.57%. Therefore, the present invention achieves a significant reduction in communication traffic with high model accuracy.

[0145] In addition, Figure 4 This paper compares the performance of the SAIHT algorithm with the traditional IHT algorithm when using Gaussian and chaotic measurement matrices. As shown by the four curves in the figure, when the measurement matrix is ​​the same, the SAIHT algorithm achieves better gradient reconstruction and obtains better model accuracy than IHT. Furthermore, both reconstruction algorithms based on normalized measurement matrices (i.e., "SAITHT-C" and "IHT-C") outperform those based on Gaussian measurement matrices (i.e., "SAIHT-G" and "IHT-G"). Moreover, unlike random Gaussian measurement matrices, the combination of chaotic techniques and cyclic matrices in the SAIHT algorithm allows for frequent key replacement only by changing the initial values ​​and chaotic parameters. Therefore, it offers efficient privacy protection and lightweight communication.

[0146] To verify the privacy protection performance of "PPGC", in Figure 5-6 The paper demonstrates the attacker's inference results on labels and training images. This invention simulates two representative attacks on the gradients of threat models: Gradient Inversion (GI) and Deep Leakage from Gradients (DLG), and performs GI and DLG attacks on MNSIT and CIFAR 10 respectively. Figure 5 The result is the average after 100 simulations, showing the accuracy of label reconstruction under different attacks. This invention categorizes attackers into external attackers and honest but curious central servers, assuming prior knowledge of the image categories in the dataset, but lack of detailed data for specific clients. The probability of an attacker successfully inferring image labels is compared across three methods, where "GA" and "GA w / o" differ only in whether grouping aggregation is performed. Because external attackers and the honest but curious central server in "Sparsification" have the same information, their probabilities of inferring labels are equal. Figure 5It is evident that the defense capability of "Sparsification" is far inferior to that of the proposed defense method, with its tag reconstruction accuracy being approximately 2.5 times that of the present invention. Furthermore, a comparison between "GA" and "GA w / o" reveals that the reconstruction accuracy of "GA" is lower than that of "GA w / o." Therefore, gradient grouping and aggregation can protect the gradients of the privacy client, which is... Figure 5 The second picture provides strong evidence for this.

[0147] In fact, in the worst-case scenario, it is very difficult for an attacker to know the image categories in the dataset beforehand. To more intuitively demonstrate the protective effect of this invention on client-side training data, in Figure 6 The text demonstrates the visual reasoning effects of an honest but curious central server, which is most likely to infer client data, under different defense methods. Figure 6 The original image in (a) is a truck (labeled 9). Through iterative inference using DLG, the central server can easily and gradually infer the original image. However, under the defenses of "sparseness" and "PPGC," the truck image cannot be visually identified. Similarly, Figure 6 (b) demonstrates the inference effect of the central server using GI on three MNIST images (digits 6, 7, and 4). The proposed "PPGC" also achieves better protection for vision, but under the defense of "Sparsification", the true label can be roughly recovered.

[0148] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the present invention.

Claims

1. A personalized federated learning method with high communication efficiency and privacy protection, characterized in that, The method includes the following steps: (1) Based on the different privacy protection needs of different clients, a personalized federated learning network model with privacy protection is designed. The model consists of privacy clients, public clients and a central server. (2) All clients learn knowledge through their own neural networks and combine the knowledge learned from the local dataset with global knowledge to achieve personalized learning. After the client completes training of the local neural network using the local dataset, the privacy client compresses the gradients of its neural network with the gradients of the public client's neural network in groups to reduce communication overhead and protect gradient information. The privacy client uploads the compressed gradients to the central server for global aggregation. (3) The central server decompresses and reconstructs the compressed gradients uploaded by the client, performs global aggregation on the reconstructed gradients, updates the global neural network, and finally distributes the model parameters of the global neural network to each client. The specific method for compressing the aggregated gradient is as follows: In the privacy client i, assume the gradient of the j-th layer The dimension is Using the measurement matrix to measure the gradient Compress and measure the matrix. The generation method is as follows; 1) Generate a length of [length] based on Chebyshev mapping. chaotic sequence The chaotic sequence of the th element The generated expression is: ； Where q is the Chebyshev order. The nth element of the chaotic sequence initial value Select a sampling interval d and perform interval sampling on the above chaotic sequence. This sequence is set as the j-th layer measurement matrix. The first line, The standard deviation is ; 2) For the product generated in step 1), implement The left circular shift is repeated to generate a dimension of 1. of ,Right now: ； in, It is a constant and satisfies right Normalization is performed, that is: ； when When it is a constant, Let be the normalization coefficient, then, ; 3) Repeat the above two steps to generate gradients. Measurement matrix for each layer The final measurement matrix is ​​formed. ,in, It satisfies the restricted isometric property; Privacy client i based Perform gradient compression, the compressed gradient Represented as: ； in, For the perception matrix, This represents the sparse basis matrix after discrete cosine transform. It is a sparse coefficient vector; The specific method for step (3) is as follows: After receiving all compressed gradients, the central server reconstructs them, performs global aggregation of the reconstructed gradients, and updates the global neural network. The reconstruction goal is to reconstruct based on a given compression gradient. and perception matrix Solving for the sparse coefficient vector The reconstruction is achieved by optimizing the following L2 norm-based minimization problem: ； ； The constraint condition indicates that the number of non-zero values ​​in the reconstructed gradient does not exceed [a certain limit]. , The specific estimation method is defined as follows: ； in, express The number of elements in the solution and the number of iterations in the entire solution process are closely related to the sparsity, i.e., the number of iterations is... ,symbol" " indicates rounding down; During the optimization process, Updated to: ； in, Indicates the l-th iteration , It is a nonlinear operator function that will Except for the one with the largest absolute value All elements except the first element are set to zero, defined as: ； in, Indicated as to After sorting by the absolute values ​​of the elements, the first... One maximum value; go through The optimal sparse coefficient vector is obtained after the second iteration. Then the reconstructed gradient is ; Global gradient aggregation is performed, and the global neural network is updated. The update process of the global neural network is as follows: ； in, These are the parameters of the global neural network. This represents the learning rate of the global model, with a total sample size of [value missing]. , yes The number of samples.

2. The personalized federated learning method with high communication efficiency and privacy protection according to claim 1, characterized in that, Step (1) includes the following: setting the scope of the central server to include A privacy client and Public client, private client From public group Random selection A new group is formed by several public clients, and all clients are divided into... Groups Privacy clients and public clients train their neural networks using their own local datasets and achieve collaboration between clients through federated learning by sharing neural network gradients with the assistance of a central server.

3. The personalized federated learning method with high communication efficiency and privacy protection according to claim 2, characterized in that, Step (2) includes the following: Step 2.1), Client i utilizes its local dataset The neural network was trained, which consisted of three sub-models: a private model, a shared model, and a fusion model. The private model extracted features from the local dataset. Features extracted by the shared model ;Will After bridging layer and After concatenation, the data is input into a fusion model to complete feature fusion and obtain fused features. The three sub-models learn from each other through knowledge distillation to complete local training. The loss of each sub-model consists of two parts: one is the cross-entropy loss between the predicted hard target and the ground truth. The other is the Kullback-Leibler divergence of soft targets among sub-models. Design a time-independent balance weight. right Scaling is applied, and this weight gradually increases from 0 to 1 via a Gaussian process, i.e.: ； in, and They are The current iteration and the number of rum-up iterations to stop, when the entire training process reaches a preset stable state. The value is fixed at 1, representing the training loss of the private model, shared model, and fusion model. , , They are defined as follows: ； ； ； in, These represent the hard objectives of the private model, the shared model, and the fusion model, respectively. , , They are their soft targets, among which, To control the distillation temperature for softness, when hour, It is the softmax function; In the formula It is a weighted set of soft objectives for private and shared models, defined as: ； in, It is a set of trade-off factors, using the rum-up method to dynamically adjust over time. Adjusted from 0.5 to 1. , it is and Weighted integration features; Based on the above analysis, the total loss model for local client i is defined as follows: ； The gradients of the shared model are uploaded to a central server for global aggregation. for: ； in, This represents the model parameters of the shared model in client i with respect to sample b in the local dataset. yes The number of samples, This represents the training loss of the shared model in client i with respect to sample b in the local dataset; The private model and the fusion model are updated only locally, and the specific process is as follows: ； ； in, It is the learning rate of the private model and the fusion model. These are the parameters of the private model and the fused model in client i, respectively. These are the model parameters of the private model and the fusion model in client i with respect to sample b in the local dataset, respectively. Step 2.2), when all clients have completed local training, the privacy client will perform group aggregation of gradients, then the gradient of privacy client i will be... Updated to: ； in, The gradient of public client k in the same group as privacy client i; Step 2.3) will be applied to the aggregated gradient described above. Compress it.

4. The personalized federated learning method with high communication efficiency and privacy protection according to claim 3, characterized in that, The local dataset uses either the MNIST or CIFAR10 classification dataset.

Images