Personalized Federated Learning Blockchain-Enhanced Cyclic Distillation Guided Channel Decoupling Framework

Abstract

This invention provides a blockchain-enhanced cyclic distillation-guided channel decoupling framework for personalized federated learning. The framework includes the following components: establishing a distillation-guided channel decoupling federated learning framework; integrating the blockchain into the federated learning process; employing cyclic distillation; and introducing self-distillation into personalized federated learning. This invention offers the advantages of high efficiency and security in personalized machine learning models, with low privacy leakage risk.

Description

Technical Field

[0001] This invention relates to the field of federated learning framework technology, and more particularly to a blockchain-enhanced cyclic distillation guided channel decoupling framework for personalized federated learning. Background Technology

[0002] Federated learning (FL) is a machine learning framework that has made significant progress in recent years. Federated learning involves many clients collaboratively training a model under the coordination of a central server, while maintaining the distribution of training data. This means it can be trained on a large scale on the devices that generate the data, and this sensitive data remains with its owners—collected and trained locally. After local training, the central training coordinator obtains each node's training contribution by acquiring updates to the distributed model, but does not access the actual sensitive data, thus embodying the principles of centralized collection and data minimization.

[0003] However, the heterogeneity of data distribution among customers still brings many problems. Current federated learning frameworks still face difficulties in achieving secure and transparent data sharing among participants and are prone to privacy leaks. Therefore, it is urgent to find ways to reduce the risk of privacy leaks. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a blockchain-enhanced cyclic distillation guided channel decoupling framework for personalized federated learning, which has the advantages of high efficiency and security of personalized machine learning models and low risk of privacy leakage.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0006] A personalized federated learning blockchain-enhanced cyclic distillation bootstrap channel decoupling framework, the framework comprising the following components:

[0007] Establish a distillation-guided channel decoupling federated learning framework;

[0008] Integrating blockchain into the federated learning process;

[0009] We employ cyclic distillation and introduce self-distillation into personalized federated learning.

[0010] The present invention is further configured such that: when establishing the distillation-guided channel decoupling federated learning framework, a federated learning framework with N distributed devices is designed, where each device D = {D1, D2, ..., D...} N} holds a local dataset D i ,in Indicates input, This indicates the corresponding label.

[0011] The present invention is further configured such that: when establishing a distillation-guided channel decoupling federated learning framework, a teacher model T(x; W) is defined. T ) and student model S(x; W S ), where W T W represents the weights of the teacher model. S The weights of the student model are represented by [value 1]; channel decoupling is applied to the above model to obtain M decoupled sub-models, denoted as [value 2]. Where m = 1, 2, ..., M; minimizing the loss L considering the local dataset. i (W S ) and L d (W S The overall loss function L(W) S ):

[0012] L(W S )=∑ i [αL i (W S )+(1-α)L d (W S )],

[0013] Where α∈[0,1] are the weight parameters balancing the local dataset loss and distillation loss, L d (W S ) is the distillation loss between the student model and the decoupled teacher sub-model, and

[0014]

[0015] The present invention is further configured such that: when integrating a blockchain into the federated learning process, a blockchain B is established, which contains a series of blocks B = {b1, b2, ..., b}. k}, each block b k All contain a set of model update transactions Each block b k Each transaction in Corresponding to a local model update The consensus mechanism C(·) is used for verification; model update transactions are aggregated in each block, and the global model weights are updated as follows:

[0016]

[0017] Among them W k This indicates that in processing block b k The global model weights obtained after the transactions.

[0018] The present invention is further configured to: employ cyclic distillation and introduce self-distillation into personalized federated learning; during local training, cyclic distillation is used to process w i That is, apply consistent regularization between personalized weights and w0, i.e., shared weights;

[0019] By w i The subnet parameterized by w0 is represented as By (w i The network composed of (w0) is represented as For each input sample x i ,from and Collect yi separately L and Overall forecast Minimize the true label y i Cross-entropy loss function L CD The loss from cyclic distillation is defined as follows:

[0020]

[0021] Where KL(.) represents the Kullback-Leibler divergence, which is related to w i Applying consistent regularization between w0 and w0, the total loss function is:

[0022] L = L CE +λ·L CD ,

[0023] The balance coefficient λ is set to 1.

[0024] The present invention is further configured such that, in order to stabilize training performance, an exponential moving average (EMA) scheme is used for the personalized channel w. i The local weight updates in the process are smoothed out at time t, w i The EMA has been updated to:

[0025]

[0026] in This is the original update of the overall loss function, with the smoothing coefficient β. t It depends on the current global period number.

[0027] The present invention has the following advantages:

[0028] 1. Model Personalization: The framework proposed in this application transfers knowledge from the global model to the local model through distillation-guided channel decoupling, improving the performance of the local model on client-specific data distributions. Compared with traditional FL systems, this approach provides more accurate and relevant results.

[0029] 2. Negative Transfer Mitigation: The framework proposed in this application mitigates negative transfer by reducing the gap between the local and global weight learning representations, addressing the problem caused by differences in client data distribution. This helps avoid the issue of poor model performance on different client data, a common problem in traditional FL systems.

[0030] 3. Data Privacy, Security, and Auditability: The framework proposed in this application provides data privacy, security, and auditability by using blockchain technology. This ensures that the model training process is secure, client-side local data remains private, and the model training process is transparent and verifiable. This approach establishes trust among participants and ensures compliance with data privacy regulations. Attached Figure Description

[0031] Figure 1 This is a schematic diagram illustrating the use of a blockchain and federated learning framework for model personalization in an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] A personalized federated learning blockchain-enhanced cyclic distillation guided channel decoupling framework, such as Figure 1 As shown, this is a global model for personalized non-independent identically distributed federated learning. Specifically, it involves assigning adaptive proportions of learnable personalized weights layer by layer from top to bottom.

[0034] In this way, the framework aims to achieve greater personalization capabilities for both simple and complex patterns (such as image and label-level personalization). A uniform personalization partitioning ratio is defined to determine the precise proportion of personalized channels in each layer. Channel parameters with a ratio of p are trained locally without aggregation through a blockchain server, resulting in different private weights, denoted by w. i This indicates that the subscript is associated with the client ID, and the remaining (1-p) percent of the shared weight, denoted as w0, will be trained using a common federated learning algorithm.

[0035] This application's personalized federated learning blockchain-enhanced cyclic distillation bootstrap channel decoupling framework includes the following components:

[0036] Establish a distillation-guided channel decoupling federated learning framework;

[0037] Integrating blockchain into the federated learning process;

[0038] We employ cyclic distillation and introduce self-distillation into personalized federated learning.

[0039] When establishing a distillation-guided channel decoupling federated learning framework, a federated learning framework with N distributed devices is designed, where each device D = {D1, D2, ..., D...} N} holds a local dataset D i ,in Indicates input, This represents the corresponding label. And it defines the teacher model T(x; W). T ) and student model S(x; W S ), where W T W represents the weights of the teacher model. S The weights of the student model are represented by [value 1]; channel decoupling is applied to the above model to obtain M decoupled sub-models, denoted as [value 2]. Where m = 1, 2, ..., M; minimizing the loss L considering the local dataset. i (W S ) and L d (W S The overall loss function L(W) S ):

[0040] L(W S )=∑ i [αL i (W S )+(1-α)L d (W S )],

[0041] Where α∈[0,1] are the weight parameters balancing the local dataset loss and distillation loss, L d (W S ) is the distillation loss between the student model and the decoupled teacher sub-model, and

[0042]

[0043] To integrate blockchain into the federated learning process, a blockchain B is established, containing a series of blocks B = {b1, b2, ..., b}. k}, each block b k All contain a set of model update transactions Each block b k Each transaction in Corresponding to a local model update The consensus mechanism C(·) is used for verification; model update transactions are aggregated in each block, and the global model weights are updated as follows:

[0044]

[0045] Among them W kThis indicates that in processing block b k The global model weights obtained after the transactions.

[0046] The goal is to design a secure and efficient consensus mechanism C(·) to ensure the integrity and privacy of model updates while incentivizing participating devices.

[0047] In summary, the formulation of the methodological problem of distillation-guided channel decoupling federated learning and blockchain integration aims to minimize the overall loss function L(W). S Considering the loss L of the local dataset i (W S ) and distillation loss L d (W S The global model weights are optimized using a blockchain-based consensus mechanism C(·), while secure and privacy-preserving model updates are ensured through this mechanism.

[0048] The following is a personalized feedback learning algorithm integrating cyclic distillation and blockchain:

[0049] Algorithm 1:

[0050]

[0051]

[0052] When employing cyclic distillation and introducing self-distillation into personalized federated learning, cyclic distillation is used to adjust the frequency of w during local training. i That is, apply consistent regularization between personalized weights and w0, i.e., shared weights;

[0053] By w i The subnet parameterized by w0 is represented as By (w i The network composed of (w0) is represented as For each input sample x i ,from and Collect yi separately L and Overall forecast Minimize the true label y i Cross-entropy loss function L CD The loss from cyclic distillation is defined as follows:

[0054]

[0055] Where KL(.) represents the Kullback-Leibler divergence, which is related to w i Applying consistent regularization between w0 and w0, the total loss function is:

[0056] L = LCE +λ·L CD ,

[0057] The balance coefficient λ is set to 1.

[0058] To stabilize training performance, an exponential moving average (EMA) scheme is used for individual channels w. i The local weight updates in the process are smoothed out at time t, w i The EMA has been updated to:

[0059]

[0060] in This is the original update of the overall loss function, with the smoothing coefficient β. t It depends on the current global period number.

[0061] This application includes a distillation-guided channel decoupling technique that combines knowledge distillation and channel decoupling to improve the personalization of models in federated learning. It addresses the challenge of learning from data distributions from different clients by leveraging a global model as a "teacher" to impart knowledge to local "student" models. By employing cyclic distillation, the framework imposes consistency regularization between personalized and shared weights during local training, reducing the gap between the learned representations of local and global weights. It also includes blockchain fusion: integrating blockchain technology into the federated learning system provides a decentralized and secure method for maintaining a distributed ledger of data, enhancing data privacy by keeping client-specific local data confidential, neither exposing it to other participants nor storing it on the blockchain. Blockchain fusion also improves security against attacks and data manipulation and ensures that the training process is transparent, verifiable, and auditable.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A personalized federated learning blockchain-enhanced cyclic distillation guided channel decoupling method, characterized by: The method includes the following parts: Establish a distillation-guided channel decoupling federated learning framework; Integrating blockchain into the federated learning process; Circular distillation is employed, and self-distillation is introduced into personalized federated learning; When establishing a distillation-guided channel decoupling federated learning framework, a federated learning framework with N distributed devices is designed, where each device... Holding a local dataset D i ,in , Indicates input, Indicates the corresponding label; When establishing a distillation-guided channel-decoupled federated learning framework, a teacher model is defined. and student model ,in The weights of the teacher model are represented. The weights of the student model are represented by [value 1]; channel decoupling is applied to the above model to obtain M decoupled sub-models, denoted as [value 2]. Where m = 1, 2, ..., M; minimize the loss considering the local dataset. and Overall loss function : , in, These are the weight parameters that balance the local dataset loss and the distillation loss. It is the distillation loss between the student model and the decoupled teacher sub-model, and ； When integrating blockchain into the federated learning process, a blockchain B is established, containing a series of blocks. Each block All contain a set of model update transactions Each block Each transaction in Corresponding to a local model update and use consensus mechanism Validation is performed; model update transactions are aggregated in each block, and the global model weights are updated as follows: in Indicates processing block The global model weights obtained after the transactions; Cyclic distillation is employed, and self-distillation is introduced into personalized federated learning. During local training, cyclic distillation is used to... That is, personalized weights and That is, apply consistent regularization to the shared weights; Depend on The parameterized subnet is represented as , ,Depend on The network is represented as , For each input sample ,from and Collected separately and Overall forecast Minimize and True Labels Cross-entropy loss function The loss from cyclic distillation is defined as follows: in Denotes the Kullback-Leibler divergence, which is related to... and Applying consistency regularization between them, the total loss function is: Balance coefficient Set to 1; To stabilize training performance, an exponential moving average (EMA) scheme is used for individual channels. Local weight updates in time are smoothed out. , The EMA has been updated to: in It is the original update of the overall loss function, and the smoothing coefficient. It depends on the current global period number.

Images