Trusted decentralized federal learning system for mobile electronic equipment
By adopting a decentralized federated learning system based on knowledge extraction in the mobile electronic device ecosystem, the heterogeneity problem in the artificial intelligence Internet of Things scenario is solved, learning efficiency and privacy protection capabilities are improved, and the convergence and efficiency of the decentralized training process are ensured.
Patent Information
- Application Number
- CN202510214321.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-25
- Publication Date
- 2025-06-27
AI Technical Summary
In the large-scale artificial intelligence Internet of Things scenario, the decentralized federated learning framework faces heterogeneity problems, low communication efficiency, large-scale problems, etc. In particular, heterogeneity problems include different learning models, non-independent and same-distributed data distributions and unbalanced data sets, resulting in low learning efficiency and privacy protection problems.
A trusted decentralized federated learning system based on knowledge extraction is proposed. By conducting local training in the mobile electronic device ecosystem, model aggregation and update using a point-to-point method, knowledge is captured from the teacher model using soft prediction, and teacher model and student model are combined through joint loss function, and knowledge extraction weights are iteratively adjusted to adapt to heterogeneity problems.
It effectively overcomes the problem of heterogeneity, improves learning efficiency, solves privacy protection problems, adjusts the weight of knowledge extraction through real-time learning performance, and ensures the convergence and efficiency of the decentralized training process.
Smart Images
Figure CN120216983A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of the Internet of Things, and in particular to a trusted decentralized federated learning system for mobile electronic devices. Background Art
[0002] With the rapid development of the Internet of Things (IoT), the landscape of mobile electronic devices has been completely transformed, ushering in an era where intelligent, interconnected devices enhance our daily lives. In this evolving ecosystem, IoT-driven mobile electronic devices, such as smartphones, smartwatches, and other intelligent devices, have the ability to collect and analyze data at the network edge. The essence of these intelligent mobile devices lies in their ability to operate in data-rich environments, leveraging extensive local data to provide personalized and responsive user experiences. Integrating artificial intelligence technologies into this IoT-driven landscape transforms these devices into intelligent ecosystems that can predict user needs, optimize energy consumption, and enhance security. However, communication bandwidth limitations and data privacy risks have become key challenges. Traditional methods of offloading large datasets to remote cloud servers for artificial intelligence processing are becoming increasingly impractical, fraught with vulnerabilities in bandwidth limitations and data leakage.
[0003] IoT-driven mobile electronic devices can collect and analyze data to improve functionality and user experience, and are increasingly becoming part of edge computing networks. Decentralized federated learning is regarded as a promising artificial intelligence IoT framework that utilizes the data generated by these interconnected mobile devices without compromising user privacy, performing computations near the data sources rather than in a centralized cloud data center. However, there are many obstacles and bottlenecks in deploying decentralized learning frameworks in large-scale artificial intelligence IoT scenarios, such as heterogeneity issues, communication efficiency, large-scale problems, etc. Specifically, heterogeneity issues include different learning models, non-independent and identically distributed data distributions, and imbalanced datasets. Summary of the Invention
[0004] The present invention is to overcome the above-mentioned deficiencies in the prior art, and provides a trusted decentralized federated learning system for mobile electronic devices that can solve heterogeneity problems and protect data privacy.
[0005] To achieve the above object, the present invention adopts the following technical solutions:
[0006] A trusted decentralized federated learning system for mobile electronic devices, based on sustainable and adaptive decentralized federated learning of knowledge extraction, where decentralized federated learning refers to the process of local training in a mobile electronic device ecosystem, adopting a peer-to-peer method, and specifically includes the following steps:
[0007] (1) Establishment of a decentralized federated learning framework: For a pair of learners, two local clients exchange their local learning models, and then use a method based on weighted averaging to aggregate the local learning model, i.e., the student model, with the received trained learning model, i.e., the teacher model, and train the aggregated model using the local dataset, iteratively; the local client processes data samples on the local mobile electronic device;
[0008] (2) Knowledge extraction: Use soft prediction to capture the learned knowledge from the teacher model, and combine the teacher model and the student model through a joint loss function to facilitate the training of the student model; in each pair of learners, one local client finds a pair of learners and then performs model updates; the local client processes data samples on the local device;
[0009] (3) Real-time evaluation: Iteratively use real-time learning performance to determine the knowledge extraction weights of the student model and the teacher model; design three algorithms to implement, including the original decentralized federated learning DFL, knowledge extraction-based decentralized federated learning KDFL for different learning models, and knowledge extraction-based decentralized federated learning with adaptive weights KDFLAW.
[0010] The present invention proposes a sustainable and adaptive knowledge extraction-based decentralized federated learning framework. Knowledge extraction is one of the transfer learning techniques, which uses soft prediction to capture the learned knowledge from a trained model. Then, it combines two local learning models (the teacher model and the student model) through a joint loss function to facilitate the training process of the student model. In this sense, we first introduce knowledge extraction into the decentralized learning framework to improve learning efficiency. Among them, in each pair of learners, one local client finds a pair of learners and then performs model sharing. To address the privacy protection issue, the local client processes data samples on the local device. We formulate a decentralized optimization problem and formally propose various heterogeneity problems. The proposed framework includes two components, including a knowledge extraction algorithm for hardware heterogeneity and a real-time evaluation module to adapt to the client drift (data heterogeneity) problem during the training process. Specifically, to mitigate the performance loss caused by data heterogeneity, we iteratively use real-time learning performance (e.g., loss, average accuracy, etc.) to determine the weights of the student model and the teacher model. We design three algorithms to implement the proposed framework, including the original decentralized federated learning (DFL), the knowledge extraction-based decentralized federated learning framework (KDFL) for different learning models, and the knowledge extraction-based decentralized federated learning framework with adaptive weights (KDFLAW).
[0011] Preferably, in step (1), federated learning is one of the representative distributed learning frameworks, aiming to obtain a shared machine deep learning model on a large number of distributed artificial intelligence Internet of Things devices; in each iteration, it proceeds as model aggregation and model update, where the central parameter server collects all received local updates to determine the latest global learning update, and uses a weighted average-based method to send the latest global learning update back to the local clients, and the local clients will use the latest received global learning update and their local resources to update this model; represent the original federated learning as
[0012]
[0013] where F() represents the loss function; w is the model parameter, representing the model weight vector in the local node set N; w * represents the model weight optimization parameter in the local node set N; in the original federated learning, the central parameter server uses a weighted average-based method to determine the global loss function, and represent it as follows:
[0014]
[0015] where |N| represents the size of the local node set, which also refers to the number of local clients; F n () represents the loss function of local client n; |D| represents the number of training data samples from the local node set N, and local nodes and local clients are interchangeable; |D n | represents the number of device training samples on local node n; specifically, |D| is calculated as follows:
[0016]
[0017] In each iteration, the central parameter server updates the global learning model after receiving the local updates, and represent the process as follows:
[0018]
[0019] where u n t represents the local model update of local node n, and t represents the iteration index; it is proposed that local nodes use a gradient descent-based method to generate local updates on their own datasets, and represent it as follows:
[0020]
[0021] where x n ,y n respectively represent the data sample and label information of local client n; w nRepresent the model parameters of the local node n; obtain the local model update u n t , which is expressed as follows:
[0022]
[0023] Compared with the original federated learning, the central parameter server is not available in decentralized federated learning; decentralized federated learning will use a peer-to-peer communication link for the training process, which distributes model aggregation and model updates on each local node; the optimization problem of decentralized federated learning aims to train a set of learning models based on the local datasets of users, and decentralized federated learning is expressed as follows:
[0024]
[0025] where W represents the set of local training weights, including w1…w N ; thus, the sum of the loss functions F n (w n , D n ) will not consider the local node weights.
[0026] Preferably, in step (2), knowledge extraction is implemented using a joint loss function, where the teacher model generates soft predictions to assist the student model; to obtain soft predictions, the teacher model uses the Softmax-T function to convert the Logit unit into soft predictions, and Softmax-T is expressed as follows:
[0027]
[0028] where T represents the temperature in knowledge extraction, z k represents the k-th label in the Logit unit, and P represents the number of labels; for the joint loss function, the two loss functions are combined by using a weight-based method, and the joint loss function is expressed as follows:
[0029] F = (1 - α)F su + αF ta , (9)
[0030] where α represents the knowledge extraction weight, which is used to determine the ratio between the teacher model α and the student model 1 - α; F ta and F su represent the loss functions of the teacher model and the student model respectively.
[0031] Preferably, in step (3), the decentralized federated learning framework uses knowledge extraction to transfer the extracted knowledge between two local clients, which is expressed as follows:
[0032]
[0033] Design a real-time evaluation module to adjust the weights of knowledge extraction, and develop three algorithms, including the original decentralized federated learning DFL, the knowledge extraction-based decentralized federated learning KDFL for different learning models, and the knowledge extraction-based decentralized federated learning with adaptive weights KDFLAW. The basic working processes of the algorithms are as follows:
[0034] (i) The local client sets the initialization of decentralized federated learning, including the local dataset and parameters;
[0035] (ii) All local clients will perform a one-time local training process;
[0036] (iii) Each local client will find local learning pairs in a peer-to-peer manner;
[0037] (iv) The local client will exit the learning program;
[0038] Among them, step (iii) will be repeated until the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value.
[0039] Preferably, the specific working process of the decentralized federated learning DFL is as follows:
[0040] (i) Initialize the training task of decentralized federated learning and configure relevant parameters;
[0041] (ii) If it is the first iteration, perform a one-time local training process on the local client;
[0042] (iii) If it is not the first iteration, each local client finds local learning pairs, exchanges and pairs the local node m, i.e., w m with the learning model w n and performs model aggregation on w m and w n using the weight averaging method |D| = |D m | + |D n | to update the weights of w m and w n and repeat; Repeat;
[0043] (iv) If the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value, exit the learning program.
[0044] Preferably, the specific working process of the knowledge extraction-based decentralized federated learning KDFL for different learning models is as follows:
[0045] (i) Initialize the training task of decentralized federated learning and configure relevant parameters;
[0046] (ii) If it is the first iteration, perform a one-time local training process on the local clients;
[0047] (iii) If it is not the first iteration, each local client finds a local learning pair, exchanges and pairs the local node m, i.e., w m with the learning model w n and performs model aggregation on w m and w n Set the joint loss function based on the knowledge extraction method as shown in Equation 9, solve the optimization problem as shown in Equation 10, and repeat;
[0048] (iv) If the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value, exit the learning program.
[0049] Preferably, the specific workflow of the decentralized federated learning KDFLAW based on adaptive knowledge extraction is as follows:
[0050] (i) Initialize the training task of decentralized federated learning and configure relevant parameters;
[0051] (ii) If it is the first iteration, perform a one-time local training process on the local clients;
[0052] (iii) If it is not the first iteration, each local client finds a local learning pair, exchanges and pairs the local node m, i.e., w m with the learning model w n and performs model aggregation on w m and w n Set the joint loss function based on the knowledge extraction method as shown in Equation 9, test the aggregated model w m to determine the knowledge extraction weights of the aggregated model w m and solve the optimization problem as shown in Equation 10, and repeat;
[0053] (iv) If the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value, exit the learning program.
[0054] The beneficial effects of the present invention are: overcoming various heterogeneity problems, including different learning models, non-independent and identically distributed data distributions, and imbalanced data sets; improving learning efficiency and solving the privacy protection problem; using real-time learning performance to determine the knowledge extraction weights between the teacher model and the student model, confirming the convergence of the decentralized training process, verifying the efficiency and effectiveness of the proposed framework, and improving the learning performance in various heterogeneous scenarios. Brief Description of the Drawings
[0055] Figure 1 is a schematic diagram of the mobile electronic device ecosystem in the present invention. Detailed Embodiments
[0056] The present invention will be further described below in conjunction with the drawings and detailed embodiments.
[0057] As Figure 1 in the described embodiment, a trusted decentralized federated learning system for mobile electronic devices, sustainable and adaptive decentralized federated learning based on knowledge extraction, and decentralized federated learning refers to the process of local training in the mobile electronic device ecosystem, adopting a peer-to-peer method, specifically including the following steps:
[0058] (1) Establishment of the decentralized federated learning framework: For pairwise learning, two local clients exchange their local learning models, and then use the method based on weighted averaging to aggregate the local learning model, that is, the student model, with the received trained learning model, that is, the teacher model, and use the local dataset to train the aggregated model, and iterate; the local client will process the data samples on the mobile electronic device locally.
[0059] Federated learning is one of the representative distributed learning frameworks, aiming to obtain a shared machine deep learning model on a large number of distributed artificial intelligence Internet of Things devices. In each iteration, it proceeds as model aggregation and model update. The central parameter server collects all the received local updates to determine the latest global learning update, and uses the method based on weighted averaging to send the latest global learning update back to the local client. The local client will use the latest received global learning update and its local resources to update this model. The local nodes and local clients in the present invention are interchangeable. Mathematically, the original federated learning is represented as
[0060]
[0061] where F() represents the loss function; w is the model parameter, representing the model weight vector in the local node set N; w * represents the model weight optimization parameter in the local node set N; in the original federated learning, the central parameter server uses the method based on weighted averaging to determine the global loss function, and represents it as follows:
[0062]
[0063] where |N| represents the size of the local node set, and also refers to the number of local clients; F n() represents the loss function of the local client n; |D| represents the number of training data samples from the local node set N, where local nodes and local clients are interchangeable; |D n | represents the number of device training samples on the local node n; specifically, |D| is calculated as follows:
[0064]
[0065] In each iteration, the central parameter server updates the global learning model after receiving the local updates, and the process is represented as follows:
[0066]
[0067] where, u n t represents the local model update of the local node n, and t represents the iteration index; it is proposed that the local node uses a gradient descent-based method to generate the local update on its own dataset, which is represented as follows:
[0068]
[0069] where, x n , y n represent the data sample and label information of the local client n respectively; w n represents the model parameter of the local node n; it is easy to obtain the local model update u n t , which is represented as follows:
[0070]
[0071] Compared with the original federated learning, decentralized federated learning provides a more flexible topology, where the central parameter server is not available in decentralized federated learning. Decentralized federated learning will use peer-to-peer communication links for the training process, which distributes model aggregation and model updates on each local node. The optimization problem of decentralized federated learning aims to train a set of learning models based on the local datasets of users, and decentralized federated learning is represented as follows:
[0072]
[0073] where, W represents the set of local training weights, including w1…w N ; obviously, thus, the sum of the loss functions F n (w n , D n ) will not consider the local node weights. On the other hand, due to the nature of the decentralized topology, finding the optimal solution may be a challenging task without compromising the privacy policy.
[0074] (2) Knowledge extraction: Use soft prediction to capture the learned knowledge from the teacher model, and combine the teacher model and the student model through a joint loss function to facilitate the training of the student model; in each learning pair, a local client will find a learning pair and then perform model updates; the local client processes data samples on a local processing device.
[0075] Knowledge extraction can reduce model complexity and computational costs while effectively transmitting and compressing knowledge. It achieves this by having the student model mimic the output of the teacher model, facilitating the flexible aggregation of heterogeneous local client models. As Figure 1 (the small figure in the lower right corner) shows, knowledge extraction uses a joint loss function to achieve this goal, where the teacher model (a trained learning model) can generate soft predictions to assist the student model (a blank learning model). To obtain soft predictions, the teacher model uses the Softmax-T function to convert the Logit unit into a soft prediction, and represent Softmax-T as follows:
[0076]
[0077] where, T represents the temperature in knowledge extraction (a hyperparameter), z k represents the k-th label in the Logit unit, P represents the number of labels; for the joint loss function, by using a weight-based method to combine the two loss functions, represent the joint loss function as follows:
[0078] F = (1 - α)F su + αF ta , (9)
[0079] where, α represents the knowledge extraction weight, used to determine the ratio between the teacher model α and the student model 1 - α; F ta and F su represent the loss functions of the teacher model and the student model respectively.
[0080] (3) Real-time evaluation: Iteratively use real-time learning performance to determine the knowledge extraction weights of the student model and the teacher model; design three algorithms to implement, including the original decentralized federated learning DFL, knowledge extraction-based decentralized federated learning KDFL for different learning models, and adaptive knowledge extraction-based decentralized federated learning KDFLAW.
[0081] We analyzed the remaining problems of decentralized federated learning in real-world Internet of Things (IoT) scenarios. To address these issues, we proposed an adaptive knowledge extraction-based decentralized federated learning framework. The heterogeneity of the AIoT scenarios includes different learning models, non-independent and identically distributed (non-IID) data distributions, and imbalanced datasets. In most existing works, decentralized federated learning is deployed under homogeneous conditions, i.e., all local clients adopt the same learning model in model aggregation. However, the real-world AIoT scenarios introduce a large amount of device hardware diversity, making it non-negligible to support various learning models. In this sense, different learning models cannot be ignored. Decentralized federated learning is troubled by the problem of data heterogeneity, which has a negative impact on model aggregation. Specifically, both non-IID data distributions and imbalanced datasets can have a negative impact on the decentralized training process, such as average learning accuracy, convergence, etc. The reason behind the above problems is that the aggregation method based on weight averaging amplifies model divergence. For example, in the worst case, a client with a large number of local data samples can easily dominate the training process and bias the model aggregation in an unfair direction. To this end, we propose a sustainable and adaptive knowledge extraction-based decentralized federated learning framework, which uses knowledge extraction to transfer the extracted knowledge between two local clients, and it is represented as follows:
[0082]
[0083] To further improve the robustness of the proposed framework in heterogeneous scenarios, we designed a real-time evaluation module to adjust the weights of knowledge extraction, which can alleviate the client drift problem. We developed three algorithms, including the original decentralized federated learning (DFL), knowledge extraction-based decentralized federated learning (KDFL) for different learning models, and knowledge extraction-based adaptive weighted decentralized federated learning (KDFLAW). The basic workflow of the algorithms is as follows:
[0084] (i) Local clients set the initialization of decentralized federated learning, including local datasets, parameters, etc.;
[0085] (ii) All local clients will perform a one-time local training process;
[0086] (iii) Each local client will find local learning pairs in a peer-to-peer manner;
[0087] (iv) Local clients will exit the learning program;
[0088] Among them, step (iii) will be repeated until the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value. The proposed algorithms (KDFL and KDFLAW) can support the decentralized training process with different (identical) training models. To clearly illustrate our method, we provide the work processes of three algorithms in DFL, KDFL, and KDFLAW. We assume that each local client uses the loss value as the local performance metric to evaluate the received model.
[0089] The specific work process for decentralized federated learning DFL is as follows:
[0090] (i) Initialize the training task of decentralized federated learning and configure relevant parameters; the parameters include learning rate, batch size, number of epochs, model initialization, etc.;
[0091] (ii) If it is the first iteration (t = 0), then perform a one-time local training process on the local client;
[0092] (iii) If it is not the first iteration, then each local client finds the local learning pair, pairs the local node m, i.e., w m with the learning model w n for exchange pairing, aggregates w m and w n using the weighted average method |D| = |D m | + |D n | to update the weights of w m and w n and repeat; Repeat;
[0093] (iv) If the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value, then exit the learning program.
[0094] The specific work process for knowledge extraction-based decentralized federated learning KDFL with different learning models is as follows:
[0095] (i) Initialize the training task of decentralized federated learning and configure relevant parameters; the parameters include learning rate, batch size, number of epochs, model initialization, knowledge extraction weights, etc.;
[0096] (ii) If it is the first iteration (t = 0), then perform a one-time local training process on the local client;
[0097] (iii) If it is not the first iteration, then each local client finds the local learning pair, pairs the local node m, i.e., w m with the learning model w n for exchange pairing, pairs w m and wn Perform model aggregation, set the joint loss function based on the knowledge extraction method as shown in Equation 9, solve the optimization problem as shown in Equation 10, and repeat;
[0098] (iv) If the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value, then exit the learning program.
[0099] The specific workflow of the decentralized federated learning KDFLAW based on adaptive knowledge extraction is as follows:
[0100] (i) Initialize the training task of decentralized federated learning and configure relevant parameters; the parameters include learning rate, batch size, number of epochs, model initialization, knowledge extraction weight, etc.;
[0101] (ii) If it is the first iteration (t = 0), then perform a one-time local training process on the local client.
[0102] (iii) If it is not the first iteration, then each local client finds a local learning pair, pairs the local node m, i.e., w m with the learning model w n for exchange pairing, perform model aggregation on w m and w n set the joint loss function based on the knowledge extraction method as shown in Equation 9, test the aggregated model w m to determine the knowledge extraction weight of the aggregated model w m solve the optimization problem as shown in Equation 10, and repeat;
[0103] (iv) If the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value, then exit the learning program.
[0104] Decentralized federated learning is a viable solution that enables artificial intelligence technologies to access local resources to train learning models without compromising privacy issues. As shown in Figure 1 , decentralized federated learning will perform local training processes in a mobile electronic device ecosystem using a peer-to-peer approach even without a central parameter server. It is different from the original federated learning, which forces local learning groups to train a global learning model by allowing each local client to have its own preferences in its own learning goals. Specifically, decentralized federated learning includes two main steps, including model aggregation and model update. For a pair of learning models, two local clients exchange their local learning models and then aggregate the local learning models with the received learning models using a weighted average-based method. Model update aims to train the aggregated model using the local dataset and iterate.
[0105] However, the heterogeneity and large-scale issues of AIoT devices pose many challenges to the deployment of decentralized federated learning frameworks in real-world AIoT scenarios. The heterogeneity of AIoT systems can come from multiple domains, including hardware heterogeneity, data heterogeneity, etc.
[0106] Hardware heterogeneity: AIoT devices are composed of many hardware components with different performances because these products are designed to meet various working conditions. Performance differences, such as computing power and communication protocols, may affect the efficiency of decentralized federated learning. For example, if a local AIoT device does not have a GPU, it will struggle during the training process of deep learning models.
[0107] Data heterogeneity: In data heterogeneity, non-independent and identically distributed data distributions and imbalanced data sets are two major problems. Non-independent and identically distributed data distributions indicate that AIoT devices with local working scenarios collect data samples from different data distributions. Non-independent and identically distributed data distributions lead to the client drift problem in the decentralized training process. Imbalanced data sets indicate differences in data sizes. Therefore, imbalanced data sets may cause learning fairness issues in the decentralized training process, making the learning model biased towards a selfish direction.
[0108] In this invention, we propose a trustworthy and adaptive decentralized federated learning framework based on knowledge extraction to overcome many heterogeneity and large-scale issues. To achieve this goal, we formulate a decentralized optimization problem and formally present various heterogeneity problems. The proposed framework includes two components, including a knowledge extraction-based algorithm to transfer knowledge between different learning models and a real-time evaluation module to adapt to the client drift problem. We derive the convergence analysis, and the convergence analysis shows that the convergence of the proposed framework is guaranteed. Finally, to confirm the effectiveness and efficiency of the proposed framework, we conduct extensive experiments on three public data sets to verify the effectiveness and efficiency of the proposed framework in large-scale IoT scenarios; in addition, we introduce several baseline methods and set various scenarios for the experiments; the evaluation results clearly show that the proposed framework can significantly improve learning performance on various metrics, outperform existing state-of-the-art baseline methods, and enhance the feasibility of decentralized federated learning in various large-scale AIoT scenarios.
Claims
1. A trusted decentralized federated learning system for mobile electronic devices, characterized in that: Sustainable and adaptive decentralized federated learning based on knowledge extraction. Decentralized federated learning refers to the process of local training in the mobile electronic device ecosystem in a peer-to-peer manner, which includes the following steps: (1) Establishment of a decentralized federated learning framework: For a pair of learning, two local clients exchange their local learning models, and then use a weighted average method to aggregate the local learning model, i.e., the student model, with the received trained learning model, i.e., the teacher model. The aggregated model is trained using the local dataset in an iterative manner. The local client processes the data samples on the mobile electronic device locally. (2) Knowledge extraction: Use soft prediction to capture the learned knowledge from the teacher model and combine the teacher model and the student model through a joint loss function to facilitate the training of the student model; In each learning pair, a local client finds a learning pair and then performs a model update; the local client processes the data samples locally on the device; (3) Real-time evaluation: Iteratively use the real-time learning performance to determine the knowledge extraction weights of the student model and the teacher model; Three algorithms are designed to implement this, including the original decentralized federated learning DFL, decentralized federated learning KDFL based on knowledge extraction for different learning models, and decentralized federated learning KDFLAW based on adaptive knowledge extraction.
2. A trusted decentralized federated learning system for mobile electronic devices according to claim 1, characterized in that: In step (1), federated learning is one of the representative distributed learning frameworks, which aims to obtain a shared machine deep learning model on a large number of distributed AI IoT devices; In each iteration, it is performed as model aggregation and model update. The central parameter server collects all received local updates to determine the latest global learning update and sends the latest global learning update back to the local client using a weighted average-based method. The local client will use the latest received global learning update and its local resources to update this model; the original federated learning is represented as Where F() represents the loss function; w is the model parameter, which represents the model weight vector in the local node set N; w * Represents the model weight optimization parameter in the local node set N; in the original federated learning, the central parameter server uses a weighted average-based method to determine the global loss function, which is expressed as follows: Where |N| represents the size of the local node set, which also refers to the number of local clients; F n () represents the loss function of the local client n; |D| represents the number of training data samples from the local node set N, and the local node and the local client are interchangeable; |D n | represents the number of device training samples on the local node n; specifically, |D| is calculated as follows: In each iteration, the central parameter server updates the global learning model after receiving local updates, and its process is expressed as follows: Among them, u n t represents the local model update of the local node n, and t represents the iteration index. It is proposed that the local node generates local updates on its own dataset using a gradient descent-based method, which is expressed as follows: Among them, x n ,y n Respectively represent the data samples and label information of the local client n; w n Represents the model parameters of the local node n; obtains the local model update u n t , which can be expressed as follows: Compared with the original federated learning, the central parameter server is not available in decentralized federated learning; decentralized federated learning will use a point-to-point communication link for the training process, which distributes model aggregation and model updates on each local node; the optimization problem of decentralized federated learning aims to train a set of learning models based on the user's local data set, and decentralized federated learning is expressed as follows: Where W represents the local training weight set, including w1…w N ; In this way, the loss function F n (w n ,D n ) does not take local node weights into account.
3. A trusted decentralized federated learning system for mobile electronic devices according to claim 1 or 2, characterized in that: In step (2), knowledge extraction is implemented using a joint loss function, where the teacher model generates soft predictions to assist the student model; in order to obtain soft predictions, the teacher model uses the Softmax-T function to convert the Logit unit into a soft prediction. Softmax-T is expressed as follows: Where T represents the temperature in knowledge extraction, z k represents the kth label in the Logit unit, and P represents the number of labels; for the joint loss function, the two loss functions are combined by using a weight-based method, and the joint loss function is expressed as follows: F=(1-a)F su +αF ta , (9) Among them, α represents the knowledge extraction weight, which is used to determine the ratio between the teacher model α and the student model 1-α; F ta and F su Represent the loss functions of the teacher model and the student model respectively.
4. A trusted decentralized federated learning system for mobile electronic devices according to claim 3, characterized in that: In step (3), the decentralized federated learning framework uses knowledge extraction to transfer the extracted knowledge between two local clients, which is expressed as follows: A real-time evaluation module is designed to adjust the weight of knowledge extraction. Three algorithms are developed, including the original decentralized federated learning DFL, decentralized federated learning KDFL based on knowledge extraction for different learning models, and decentralized federated learning KDFLAW based on adaptive knowledge extraction. The basic workflow of the algorithm is as follows: (i) The local client sets up the initialization of decentralized federated learning, including local datasets and parameters; (ii) All local clients will undergo a one-time local training process; (iii) Each local client will find a local learning pair in a peer-to-peer manner; (iv) The local client will exit the learning process; Among them, step (iii) will be repeated until the local client consumes all resource budgets or the accuracy of the local learning model approaches the set value.
5. A trusted decentralized federated learning system for mobile electronic devices according to claim 4, characterized in that: The specific workflow for decentralized federated learning DFL is as follows: (i) Initialize the decentralized federated learning training task and configure relevant parameters; (ii) If it is the first iteration, a one-time local training process is performed on the local client; (iii) If it is not the first iteration, each local client finds a local learning pair and sets the local node m to w m With the learning model w n Exchange pairing, m and w n Perform model aggregation and use weighted averaging method |D|=|D m |+|D n |Update w m and w n Weight Repeat; (iv) If the local client consumes all resource budget or the accuracy of the local learning model is close to the set value, the learning process is exited.
6. A trusted decentralized federated learning system for mobile electronic devices according to claim 4, characterized in that: The specific workflow of decentralized federated learning KDFL based on knowledge extraction for different learning models is as follows: (i) Initialize the decentralized federated learning training task and configure relevant parameters; (ii) If it is the first iteration, a one-time local training process is performed on the local client; (iii) If it is not the first iteration, each local client finds a local learning pair and sets the local node m to w m With the learning model w n Exchange pairing, m and w n Perform model aggregation, set the joint loss function based on the knowledge extraction method as shown in Formula 9, solve the optimization problem as shown in Formula 10, and repeat; (iv) If the local client consumes all resource budget or the accuracy of the local learning model is close to the set value, the learning process is exited.
7. A trusted decentralized federated learning system for mobile electronic devices according to claim 4, characterized in that: The specific workflow of decentralized federated learning KDFLAW based on adaptive knowledge extraction is as follows: (i) Initialize the decentralized federated learning training task and configure relevant parameters; (ii) If it is the first iteration, a one-time local training process is performed on the local client; (iii) If it is not the first iteration, each local client finds a local learning pair and sets the local node m to w m With the learning model w n Exchange pairing, m and w n Perform model aggregation and set the joint loss function based on the knowledge extraction method as shown in Formula 9. m Test and determine the aggregated model w m The knowledge extraction weights are used to solve the optimization problem, see formula 10, and repeat the process; (iv) If the local client consumes all resource budget or the accuracy of the local learning model is close to the set value, the learning process is exited.