Self-adaptive decentralized small sample federated learning framework based on knowledge distillation driving

Abstract

The invention belongs to the technical field of federated learning, and relates to a knowledge distillation driven self-adaptive decentralized small sample federated learning framework, which comprises the following steps of: firstly, proposing a knowledge offset-based strategy, measuring a knowledge offset degree between a student model and a teacher model, optimizing local model updating of each node, and reducing differences between models; the overall performance is improved; secondly, constructing a neighbor relation graph among nodes by using a graph neural network, guiding a knowledge distillation process, and enhancing the generalization ability of the model; and finally, designing an adaptive frequency adjustment algorithm, and dynamically adjusting the model update frequency according to the model offset degree and the communication cost so as to reduce unnecessary communication overhead and improve the training efficiency. Compared with a traditional federated learning method, the framework has the advantages that the classification accuracy is remarkably improved and the communication cost is reduced on a Mini-Image Net data set and a Tiered-Image Net data set.

Description

Technical Field

[0001] The present invention belongs to the technical field of federated learning, and specifically relates to an adaptive decentralized small-sample federated learning framework driven by knowledge distillation. Background Art

[0002] In today's machine learning field, data privacy and security have become critical issues. With the rapid growth of data volume and the wide distribution of data, traditional centralized data storage and processing methods face more and more challenges, especially in the learning scenario of small sample image data. Decentralized Federated Learning (DFL), as an emerging distributed learning method, avoids the risks of privacy leakage and data abuse by retaining data on local devices instead of centralized storage. However, in the decentralized learning of small sample image data, there are still a series of problems that need to be solved. Decentralized federated learning of small sample images faces the problems of data sparsity and data heterogeneity. Since each node can only obtain a small amount of sample data and the data distribution of different nodes is significantly different, each node cannot fully learn representative data features. Especially in image tasks, a small amount of data cannot cover the diversity of image features, which makes the model easy to overfit or underfit, and it is difficult to generalize to unseen data in the global model. In addition, the high-dimensional characteristics and complex intrinsic structure of image data make it more difficult to train the model under small sample conditions. This data sparsity not only affects the training effect of each node, but also makes it impossible to effectively guarantee the consistency and accuracy of the global model among multiple nodes.

[0003] Secondly, the existing DFL method is still insufficient in optimizing communication and computing costs. In decentralized learning, frequent communication and model updates between nodes are inevitable, but this mechanism often leads to a large computing and communication burden in small sample scenarios. Each time the model is updated, each node needs to exchange a large amount of parameter and gradient information, which not only increases the communication cost, but also places high demands on the computing resources of the node. For nodes with limited computing power such as edge devices, this communication and computing burden may lead to reduced system efficiency and even interrupt the learning process. Therefore, in small sample image federated learning, how to reduce communication and computing costs and ensure efficient model updates has become a key challenge.

[0004] In addition, due to the lack of global coordination between nodes in decentralized federated learning, there are often large differences between the local models of each node. Especially in small sample learning scenarios, data imbalance and sample scarcity between nodes make the model update direction of each node inconsistent, resulting in performance fluctuations in the global model. Model inconsistency not only makes it difficult for the global model to converge quickly, but may also affect the reliability and accuracy of the model in practical applications. In image tasks, this consistency problem is particularly prominent because the complexity and diversity of image data make knowledge transfer between nodes more difficult and model fusion and adjustment more complicated.

[0005] Decentralized federated learning is an important branch of federated learning, which aims to eliminate the dependence on central servers to achieve more robust and flexible distributed learning. Traditional federated learning (FL) relies on central servers to coordinate model updates and parameter aggregation between clients, while decentralized federated learning solves the single point failure problem in centralized solutions by allowing clients to communicate directly with each other, while reducing communication bottlenecks and delays.

[0006] Under the DFL framework, clients can autonomously decide with which other clients to share updates, and propagate and aggregate parameters through a graph-based communication topology. SpreadGNN is a typical case in decentralized federated learning, which proposes a decentralized periodic averaged stochastic gradient descent method (DPA-SGD), where each client periodically exchanges model updates with its neighbors to complete collaborative training. This approach performs well when facing non-independent and identically distributed (non-IID) data.

[0007] Although DFL has significant advantages, it also faces some challenges. For example, how to design the optimal communication topology to maximize model performance and minimize communication costs is a current research hotspot. In addition, due to the asynchrony and local update mechanism of decentralized learning, how to ensure the convergence and consistency of the global model remains an important issue. To address these issues, researchers have proposed a variety of graph-based decentralized optimization methods, such as DFL-EC and GCFL+, which significantly improve the performance of decentralized federated learning by designing flexible topologies and adaptive aggregation strategies.

[0008] In the framework of decentralized federated small-sample learning, one of the main technical challenges is how to effectively share and transfer knowledge in a decentralized environment. Traditional federated learning relies on a central server to coordinate communication and parameter exchange between clients, while each client in a decentralized framework needs to communicate directly with its neighbors. Therefore, how to design an efficient graph structure communication topology has become one of the key issues in decentralized federated small-sample learning. The paper proposes a graph-based small-sample method, which introduces the idea of ​​small-sample learning into decentralized federated learning, enabling the model to quickly adapt to different tasks from a small amount of data from multiple nodes, thereby enhancing its generalization ability.

[0009] The measurement of knowledge deviation is a key technology in decentralized federated learning. By evaluating the knowledge difference between the student model and the teacher model, the local model update process of each node is optimized. FedTAD is a topology-aware data-independent knowledge distillation technology that improves the performance of global graph neural networks by enhancing reliable knowledge transfer in subgraph federated learning.

[0010] The FedEC framework solves the problem of knowledge graph completeness in federated learning by embedding contrastive learning and integration methods and utilizing knowledge learned from different aspects.

[0011] The literature uses a large teacher model trained on a central server to distill knowledge to individual user models, optimizes resource allocation and training strategies, and ultimately improves user accuracy and reduces overall latency. In addition, the literature studies the impact of knowledge complexity and differences in learning behavior between teachers and students on distillation efficiency.

[0012] Graph Neural Networks (GNNs) excel in processing graph-structured data. Chauhan proposed a classification prediction method based on a few-sample graph spectrum. In the meta-training phase, a feature extractor is trained to extract features from the graph, and an attention network is used to predict the actual class label. In the meta-testing phase, the attention network is adjusted to quickly adapt to new tasks. Although Chauhan proposed a new meta-learning architecture for graph classification, there are still some limitations. FedGraph-KD is a method that combines federated learning and knowledge distillation to solve the model heterogeneity problem in federated graph neural networks. This method enhances the consistency and robustness of the model by distilling knowledge. In addition, the FairGKD method learns fair graph neural networks through partial knowledge distillation without accessing all information, significantly improving the fairness of GNN while maintaining practicality.

[0013] ASFGNN proposes a learning model for automatically separating federated graph neural networks. By dividing the training process into client message passing and federated loss calculation, it combines Bayesian optimization for hyperparameter adjustment, thereby improving the efficiency of parameter adjustment. The literature is also an innovative attempt to build an interpretable graph neural network (EGNN). It introduces explicit "contribution" weights into the model through knowledge distillation, thereby improving the interpretability and performance of the model. However, its scalability in processing high-dimensional and complex data still needs to be improved. Summary of the invention

[0014] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an adaptive decentralized small sample federated learning framework driven by knowledge distillation.

[0015] In order to achieve the purpose of the present invention, the present invention will be implemented by adopting the technical solution described below.

[0016] An adaptive decentralized small sample federated learning framework driven by knowledge distillation, the adaptive decentralized small sample federated learning framework comprising at least three clients, each of which has a decentralized federated learning strategy based on knowledge offset, a knowledge distillation process guided by a graph neural network, and an adaptive frequency adjustment algorithm, wherein:

[0017] The decentralized federated learning strategy based on knowledge offset measures and adjusts the knowledge offset between different nodes to achieve the optimization goal of minimizing the weighted sum of each node's local loss and knowledge offset loss. The update process of each node's local model is optimized according to the optimization goal, and the updated local models of each node are aggregated into a global model through local communication between nodes.

[0018] The knowledge distillation process guided by the graph neural network uses the graph neural network to construct a neighbor relationship graph between nodes, uses the GNN model to extract the relationship features between nodes, and uses the extracted relationship features to perform knowledge distillation to guide the training process of the local model; wherein: the GNN model includes a message passing mechanism and a node aggregation mechanism, the message passing mechanism is used to collect information from neighbor nodes; the node aggregation mechanism is used to aggregate the node's message and its own features to update the node's status;

[0019] An adaptive frequency adjustment algorithm dynamically adjusts the frequency of model updates according to the model's offset and communication cost, minimizing communication and computing overheads while ensuring model consistency; wherein: the communication cost dynamic adjustment model includes a frequency adjustment function, which is used to determine whether the node needs to update the model in each iteration; in the process of dynamically adjusting the model update frequency based on the communication cost, an incremental model update strategy is introduced, stipulating that each time the model is updated, the node only transmits the change in its model parameters relative to the last update.

[0020] As a preferred solution of the present invention, the knowledge offset is the knowledge offset between node i and its neighbor node set N(i), which is defined as:

[0021]

[0022] Where KL(·||·) represents the Kullback-Leibler divergence, T is the temperature parameter used to adjust the smoothness of the softmax output, and h i and h j Represent the feature representation of node i and its neighbor node j respectively.

[0023] As a preferred embodiment of the present invention, the optimization target is expressed as:

[0024]

[0025] Among them, L local (w i ,D i ) indicates that node i is in the local dataset D i The loss on , α is a weight parameter used to balance the local loss and knowledge deviation loss.

[0026] As a preferred solution of the present invention, the updating process comprises the following steps:

[0027] S41, node i uses local data set D i and the current model parameters w i , calculate the local loss L local (w i ,D i ), and update the model parameters by gradient descent:

[0028]

[0029] Where η is the learning rate;

[0030] S42, calculate the knowledge deviation KDD between node i and its neighboring nodes i , and adjust the model parameters according to the deviation; by minimizing the knowledge deviation loss, we can get the following update formula:

[0031]

[0032] Where β is the adjustment step size, is the gradient of knowledge bias loss with respect to model parameters;

[0033] S43. Combine the gradients of local loss and knowledge deviation loss to update comprehensive model parameters:

[0034]

[0035] As a preferred solution of the present invention, the local model aggregation is that each node sends its local model parameters to neighboring nodes after completing the local model update. The neighboring nodes perform weighted averaging based on the received local model parameters and update their own model parameters for aggregation. The aggregation formula is:

[0036]

[0037] in, represents the model parameters of node i in round t+1, represents the model parameters of neighbor node j in round t.

[0038] As a preferred solution of the present invention, the expression of the message transmission mechanism is:

[0039]

[0040] in, represents the message representation of node i at level l+1, MSG(·) is the message function, e ij is the edge feature between node i and node j. For node i, its feature is represented by h i , the feature of the neighbor node is represented by h j , where j∈N(i) represents the set of all nodes directly connected to node i;

[0041] The expression of the node aggregation mechanism is:

[0042]

[0043] Where AGG(·) is the aggregation function, and the aggregation method is weighted average.

[0044] As a preferred embodiment of the present invention, the forward propagation process of the GNN model is as follows:

[0045]

[0046] Among them, σ(·) is the activation function, W (l) is the weight matrix of the lth layer, are the features of nodes i and j at layer l respectively.

[0047] As a preferred embodiment of the present invention, the distillation loss function of the knowledge distillation is:

[0048] L KD (h i ,w i)=KL(softmax(h i / T)||softmax(f(w i ) / T)) (10)

[0049] Among them, KL(·||·) represents the KL divergence, T is the temperature parameter used to adjust the smoothness of the softmax output, and f(w i ) is the model output of node i.

[0050] As a preferred solution of the present invention, the adaptive frequency adjustment algorithm comprises the following steps:

[0051] S101. Define model deviation Δ i is the local model w of node i i With the global model The difference between ; the model deviation is expressed as:

[0052]

[0053] Among them, N i represents the set of neighbor nodes of node i, and ||·|| represents the Euclidean distance. Model deviation Δ i The larger the value is, the greater the difference between the local model of node i and the models of its neighboring nodes is, and more frequent model updates are needed to ensure consistency;

[0054] S102. Define communication cost C i is the overhead required for node i to synchronize the model under the current network conditions; the communication cost is expressed as:

[0055] C i =α 1 T i，j +β 1 S i，j (12)

[0056] Among them, T i,j represents the communication delay between node i and its neighbor node j, S i,j represents the bandwidth consumption between node i and its neighbor node j, α 1 , β 1 is a weight parameter used to balance the impact of delay and bandwidth; the communication cost C i The higher the value, the greater the overhead of model synchronization under the current network conditions, and the synchronization frequency needs to be reduced to save resources;

[0057] In order to dynamically adjust the frequency of model updates for communication cost, a frequency adjustment function τ is introduced. i , which is used to determine whether node i needs to update the model in each iteration; the frequency adjustment function is expressed as:

[0058]

[0059] Among them, τ min and τ max Represent the minimum and maximum update frequency, Δ max and C max Represent the maximum values ​​of model deviation and communication cost respectively;

[0060] In this way, the model update frequency of node i will be dynamically adjusted according to the changes in model deviation and communication cost, thereby minimizing communication and computational overhead while ensuring model consistency;

[0061] In each iteration, node i first calculates its model deviation Δ i and communication cost C i , and then use the frequency adjustment function τ i Determine whether to update the model: If τ i Greater than a certain threshold τ thresh , then node i updates the model, otherwise skips this update.

[0062] As a preferred solution of the present invention, the incremental model update strategy is that each time the model is updated, node i only needs to transmit the change Δw of its model parameters relative to the last update. i :

[0063]

[0064] in, and denote the model parameters of node i at the tth and t-1th iterations respectively.

[0065] Beneficial Effects

[0066] The present invention aims to solve the main problems existing in small sample DFL and improve the consistency, communication efficiency and overall performance of the model.

[0067] First, a DFL strategy based on knowledge offset is proposed. By measuring the knowledge offset between the student model and the teacher model, the local model update process of each node is optimized. In this way, DFL can transfer more consistent knowledge between different nodes, reduce the differences between models, and thus improve the performance of the overall model.

[0068] Secondly, a knowledge distillation method based on graph neural network is proposed. By constructing a neighbor relationship graph between nodes, the knowledge distillation process is guided. Through the GNN model, each node can not only use its own data for training, but also learn from neighbor nodes, thereby improving the generalization ability of the model.

[0069] Finally, to address the communication and computing cost issues in DFL, an adaptive frequency adjustment algorithm was designed. This algorithm reduces unnecessary communication overhead and improves the overall training efficiency by dynamically adjusting the model update frequency. At the same time, by introducing a trade-off mechanism between model offset and communication cost, it ensures that communication efficiency is maximized while ensuring model consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a structural schematic diagram of the framework of the present invention;

[0071] Figure 2 This is a topological diagram of the decentralized federated learning in the present invention;

[0072] Figure 3 This is a graph showing how the communication cost varies with the number of clients. DETAILED DESCRIPTION

[0073] The present invention is further described in conjunction with the accompanying drawings and embodiments.

[0074] As an embodiment of the present invention, Figures 1 to 3 As shown, an adaptive decentralized small sample federated learning framework driven by knowledge distillation is provided, wherein the adaptive decentralized small sample federated learning framework includes at least three clients, each of which has a decentralized federated learning strategy based on knowledge offset, a knowledge distillation process guided by a graph neural network, and an adaptive frequency adjustment algorithm, wherein:

[0075] The decentralized federated learning strategy based on knowledge offset measures and adjusts the knowledge offset between different nodes to achieve the optimization goal of minimizing the weighted sum of each node's local loss and knowledge offset loss. The update process of each node's local model is optimized according to the optimization goal, and the updated local models of each node are aggregated into a global model through local communication between nodes.

[0076] The knowledge distillation process guided by the graph neural network uses the graph neural network to construct a neighbor relationship graph between nodes, uses the GNN model to extract the relationship features between nodes, and uses the extracted relationship features to perform knowledge distillation to guide the training process of the local model; wherein: the GNN model includes a message passing mechanism and a node aggregation mechanism, the message passing mechanism is used to collect information from neighbor nodes; the node aggregation mechanism is used to aggregate the node's message and its own features to update the node's status;

[0077] An adaptive frequency adjustment algorithm dynamically adjusts the frequency of model updates according to the model's offset and communication cost, minimizing communication and computing overheads while ensuring model consistency; wherein: the communication cost dynamic adjustment model includes a frequency adjustment function, which is used to determine whether the node needs to update the model in each iteration; in the process of dynamically adjusting the model update frequency based on the communication cost, an incremental model update strategy is introduced, stipulating that each time the model is updated, the node only transmits the change in its model parameters relative to the last update.

[0078] In decentralized federated learning, the models of different nodes may differ significantly due to differences in data distribution and training processes. This model inconsistency can lead to fluctuations and instability in global model performance. Therefore, a decentralized federated learning strategy based on knowledge offset is proposed. By measuring and adjusting the knowledge offset between different nodes, the model update process of each node is optimized to achieve the purpose of improving the overall model consistency and performance.

[0079] Knowledge Distillation Divergence (KDD) is used to measure the difference in knowledge representation between the student model and the teacher model. Specifically, for node i and its neighbor node set N(i), its knowledge distillation can be defined as:

[0080]

[0081] Where KL(·||·) represents the Kullback-Leibler divergence, T is the temperature parameter used to adjust the smoothness of the softmax output, and h i and h j Represent the feature representation of node i and its neighbor node j respectively.

[0082] By calculating the knowledge deviation, we can evaluate the consistency of each node and its neighbor nodes in knowledge representation. A large knowledge deviation indicates a large knowledge difference between nodes, and appropriate adjustments should be made to reduce this difference. In decentralized federated learning, our optimization goal is to minimize the weighted sum of each node's local loss and knowledge deviation loss. Specifically, for node i, its optimization goal can be expressed as:

[0083]

[0084] Among them, L local (w i ,D i ) indicates that node i is in the local dataset D i The loss on , α is a weight parameter used to balance the local loss and knowledge deviation loss.

[0085] In the process of updating the local model of each node, we need to consider both the training error of the local data and the knowledge consistency with neighboring nodes. The specific update process is as follows.

[0086] Node i uses the local dataset D i and the current model parameters w i , calculate the local loss L local (w i ,D i ), and update the model parameters by gradient descent:

[0087]

[0088] Where η is the learning rate.

[0089] Then calculate the knowledge deviation KDD between node i and its neighboring nodes i , and adjust the model parameters according to the deviation. By minimizing the knowledge deviation loss, we can get the following update formula:

[0090]

[0091] Where β is the adjustment step size, is the gradient of the knowledge bias loss with respect to the model parameters.

[0092] Finally, the gradients of the local loss and the knowledge offset loss are combined to update the comprehensive model parameters:

[0093]

[0094] This comprehensive update strategy can ensure the accuracy of the model on local data while reducing the knowledge gap with neighboring nodes, thereby improving the consistency and performance of the overall model.

[0095] In decentralized federated learning, the aggregation of global models does not rely on a central server, but is achieved through local communication between nodes. Specifically, after completing the local model update, each node sends its model parameters to neighboring nodes, and the neighboring nodes perform weighted averaging based on the received parameters to update their own model parameters. The specific aggregation formula is:

[0096]

[0097] in, represents the model parameters of node i in round t+1, represents the model parameters of neighbor node j in round t.

[0098] Through this distributed model aggregation strategy, the decentralized federated learning strategy based on knowledge offset that we proposed can effectively avoid the bottleneck problem of the central server, effectively solve the problems of model consistency and performance fluctuation, and at the same time improve the robustness and scalability of the model.

[0099] Graph Neural Networks (GNNs) perform well in processing graph structured data, and are particularly suitable for modeling complex relationships between nodes. Knowledge distillation (KD), as a model compression technology, transfers the knowledge of a large-scale teacher model to a small-scale student model, thereby reducing the consumption of computing resources while maintaining high performance. This paper proposes a knowledge distillation method guided by graph neural networks to improve the overall performance of decentralized federated learning (DFL).

[0100] In the framework of the present invention, each node is regarded as a node in the graph, and the edges between nodes represent the relationship between them.

[0101] First, we use graph neural networks to build a neighbor relationship graph between nodes, use GNN to refine the relationship features between nodes, and then use these features to perform knowledge distillation to guide the training process of the local model.

[0102] The GNN model constructed by the present invention includes two main parts: message passing mechanism and node aggregation mechanism. The message passing mechanism is used to collect information from neighboring nodes. For node i, its feature representation is h i , the feature of the neighbor node is represented by h j , where j∈N(i) represents the set of all nodes directly connected to node i. The message passing mechanism can be expressed as:

[0103]

[0104] in, represents the message representation of node i at level l+1, MSG(·) is the message function, e ij is the edge feature between node i and node j.

[0105] The aggregation mechanism is used to merge the node's message and its own features to update the node's status. The feature update formula for node i is:

[0106]

[0107] Among them, AGG(·) is the aggregation function, and the aggregation method is weighted average. Combining the above two mechanisms, the forward propagation process of GNN is as follows:

[0108]

[0109] Among them, σ(·) is the activation function, W (l) is the weight matrix of the lth layer, are the features of nodes i and j at layer l respectively. In the process of knowledge distillation, the present invention extracts the node features h i As the input of the teacher model, the knowledge transfer effect between the teacher model and the student model is measured by the Kullback-Leibler (KL) divergence. Specifically, the distillation loss function can be expressed as:

[0110] L KD (h i ,w i )=KL(softmax(h i / T)||softmax(f(w i ) / T)) (10)

[0111] Among them, KL(·||·) represents the KL divergence, T is the temperature parameter used to adjust the smoothness of the softmax output, and f(w i ) is the model output of node i.

[0112] By minimizing the distillation loss, the knowledge extracted by GNN can be effectively transferred to the local model, thereby improving the generalization ability and robustness of the model. Through the above methods, our graph neural network guided knowledge distillation process can effectively utilize the relationship characteristics between nodes and improve the performance of the model in small sample and heterogeneous data scenarios. It can not only improve the overall model consistency and robustness of decentralized federated learning, but also significantly reduce communication and computing costs, and improve the efficiency and scalability of the system.

[0113] In decentralized federated learning (DFL), communication and computation costs are key factors affecting system efficiency and performance. Traditional federated learning methods usually rely on fixed communication and model update frequencies, which can easily lead to resource waste and unstable performance in heterogeneous network environments and dynamic data distribution. To address these issues, we designed an adaptive frequency adjustment algorithm that optimizes communication and computation costs by dynamically adjusting the model update frequency while ensuring model consistency and performance.

[0114] The core idea of ​​the adaptive frequency adjustment algorithm is to dynamically adjust the frequency of model updates based on the model offset and communication cost, thereby reducing unnecessary communication overhead. Specifically, the model offset is used to measure the difference between the current model and the global model, while the communication cost reflects the overhead of model synchronization under the current network environment. By considering these two factors comprehensively, we can determine the optimal frequency for each node to update the model at different time points.

[0115] First, define the model deviation Δ i is the local model w of node i i With the global model The model deviation can be expressed as:

[0116]

[0117] Among them, N i represents the set of neighbor nodes of node i, and ||·|| represents the Euclidean distance. Model deviation Δ i The larger it is, the greater the difference between the local model of node i and the models of its neighboring nodes, and more frequent model updates are needed to ensure consistency.

[0118] Next, we define the communication cost C i is the cost required for node i to synchronize the model under the current network conditions. The communication cost can be expressed as:

[0119] C i =α 1 T i，j +β 1 S i，j (12)

[0120] Among them, T i,j represents the communication delay between node i and its neighbor node j, S i,j represents the bandwidth consumption between node i and its neighbor node j, α 1 , β 1 is a weight parameter used to balance the impact of latency and bandwidth. Communication cost C i The higher it is, the greater the overhead of model synchronization under the current network conditions, and the synchronization frequency needs to be reduced to save resources.

[0121] In order to dynamically adjust the model update frequency, we introduce a frequency adjustment function τ i , which is used to determine whether node i needs to update the model in each iteration. The frequency adjustment function can be expressed as:

[0122]

[0123] Among them, τ min and τ max Represent the minimum and maximum update frequency, Δ max and C max denote the maximum values ​​of model deviation and communication cost respectively. In this way, the model update frequency of node i will be dynamically adjusted according to the changes of model deviation and communication cost, thereby minimizing communication and computational overhead while ensuring model consistency.

[0124] In each iteration, node i first calculates its model deviation Δ i and communication cost C i , and then use the frequency adjustment function τ i Determine whether to update the model. If τ i Greater than a certain threshold τ thresh , node i updates the model, otherwise it skips the update. This adaptive frequency adjustment mechanism can effectively reduce unnecessary model synchronization and improve the overall efficiency of the system.

[0125] In addition, to further optimize the communication cost, we also introduced an incremental model update strategy. Traditional federated learning methods usually require the transmission of complete model parameters, which will bring huge communication overhead in the case of large-scale models and high-frequency updates. The incremental model update strategy only transmits the changes in model parameters, thereby greatly reducing the amount of communication data. Specifically, each time the model is updated, node i only needs to transmit the change in its model parameters relative to the last update Δw i :

[0126]

[0127] in, and They represent the model parameters of node i at the tth and t-1th iterations respectively. Incremental model updating not only reduces the amount of communication data, but also effectively reduces transmission delay and bandwidth consumption, thereby further improving the communication efficiency of the system.

[0128] The adaptive frequency adjustment algorithm can dynamically adjust the model update frequency according to the model deviation and communication cost, thereby minimizing the communication and computing overhead while ensuring the consistency of the model. This algorithm not only improves the overall efficiency of the decentralized federated learning framework, but also enhances the adaptability and robustness of the system in a heterogeneous network environment.

[0129] Convergence Analysis of Adaptive Frequency Adjustment

[0130] In order to ensure the effectiveness and stability of the adaptive frequency adjustment algorithm in decentralized federated learning (DFL), we need to analyze its convergence. This paper will prove the convergence of the adaptive frequency adjustment algorithm from a theoretical perspective and analyze its performance under different conditions.

[0131] First, we make the following basic assumptions:

[0132] Assumption 1. The model loss function L(w) is continuously differentiable and strongly convex in the model parameter space. This means that there exist constants μ>0 and L>0 such that for any w 1 and w2 have:

[0133] μ||w 1 -w 2 || 2 ≤<▽L(w 1 )-▽L(w 2 ),w 1 -w 2 >≤L||w 1 -w 2 || 2 (15)

[0134] Assumption 2. Gradient is bounded:

[0135] Assume that the gradient of each node i is ▽L i (w) is bounded, that is, there exists a constant G such that for any w and node i:

[0136] ||▽L i (w)||≤G (16)

[0137] Assumption 3. Communication delay and bandwidth:

[0138] Assume that the communication delay T i,j and bandwidth S i,j It is bounded during the entire training process, that is, there is a constant T max and S max So that:

[0139] 0≤T i,j ≤T max ,0≤S i,j ≤S max (17)

[0140] We consider the model update process of node i, the goal is to minimize the weighted sum of the local loss function and the knowledge deviation loss. The optimization objective of node i can be expressed as:

[0141]

[0142] Among them, the local loss function and knowledge deviation loss KDD i They are:

[0143]

[0144]

[0145] In each iteration, the model update process of node i can be expressed as:

[0146]

[0147] Among them, the learning rate η is a small positive number.

[0148] According to Assumption 1, the loss function is strongly convex, and there exist constants μ>0 and L>0 satisfying:

[0149]

[0150] Among them, w * It is the global optimal solution.

[0151] Using the gradient descent update formula, we have:

[0152]

[0153] Expanding the square term on the right side and using the assumptions of strong convexity and bounded gradients yields:

[0154]

[0155] Due to L total is strongly convex, and the gradient is bounded, we have:

[0156]

[0157] Substitute these into the update formula and choose an appropriate learning rate η so that You can get:

[0158]

[0159] Let γ = 1-2ημ + η 2 L, due to It can be guaranteed that 0<γ<1. Therefore, we can get:

[0160]

[0161] This means that at each iteration, the model parameters w i will converge to the global optimal solution w at a certain convergence rate γ * Through the adaptive frequency adjustment algorithm, node i adjusts the frequency according to the model deviation Δ in each iteration. i and communication cost C i Dynamically adjust the model update frequency to reduce unnecessary communication overhead. i >τ threshThe frequency of communication and bandwidth consumption can be significantly reduced because the model update is performed in real time. At the same time, the incremental model update strategy further reduces the amount of communication data and optimizes the overall communication cost. Through the above analysis, we have proved the convergence of the adaptive frequency adjustment algorithm in decentralized federated learning, and by dynamically adjusting the model update frequency and adopting the incremental model update strategy, we can effectively reduce the communication and computing costs and improve the overall efficiency and performance of the system.

[0162] Experimental Section

[0163] In order to verify the effectiveness of the proposed decentralized federated learning framework and its various innovative technologies, the present invention designed and conducted multiple groups of experiments, including federated learning comparison experiments, ablation experiments, experiments on the impact of adaptive frequency optimization algorithms, and experiments on the impact of different topological structures on decentralized federated learning. The following is a detailed experimental setup and result analysis.

[0164] Experimental setup

[0165] We conducted experiments using the resnet12 model on the Mini-ImageNet and Tiered-ImageNet datasets, respectively. The details are consistent with the model structure proposed in TADAM. The stochastic gradient descent algorithm was used as the optimizer, with an initial learning rate of 0.1, which was adjusted according to the predetermined design during training: for MiniImageNet, the learning rate was reduced at the 12,000th, 14,000th, and 16,000th times. For TieredImageNet, the learning rate was halved every 24,000 episodes. For all experiments, 2,000 episodes of the model were tested; during training, 4 episodes were selected per batch. On each dataset, we simulated a decentralized distributed data scenario, dividing the dataset into 4 nodes and 8 nodes, with the topology shown in the following figure. Figure 2 As shown in the figure, each node only has part of the data, and the experiments are conducted when the number of samples in each new category is 5 and the number of new categories to be processed is 1 (5-way 1-shot) and the number of samples in each new category is 5 and the number of new categories to be processed is 5 (5-way 5-shot). In order to ensure the fairness and comparability of the experiment, we compared the performance of traditional federated learning methods (such as FedAvg, FedMeta) and the Lstm model with the proposed DFL framework under the same conditions. We used similar dataset partitioning and training parameters to ensure that each method was evaluated in the same environment. Classification accuracy and communication cost are the main indicators for evaluating model performance. Classification accuracy is used to measure the prediction ability of the model on the test set, and communication cost represents the amount of data transmission between nodes after convergence during the entire training process.

[0166] Federated Learning Comparative Experiment

[0167] In order to evaluate the performance difference of the proposed decentralized federated learning (DFL) framework relative to the traditional federated learning (FedAvg) method, we selected two representative datasets, Mini-ImageNet and Tiered-ImageNet, and conducted detailed comparative experiments under 4 nodes. The experiment aims to verify the effectiveness of the DFL framework in image classification tasks.

[0168] Table 1 Comparative experimental results

[0169]

[0170] Table 1 shows the experimental results of 5-way 1-shot and 5-way 5-shot classification tasks on the Mini-ImageNet and Tiered-ImageNet datasets. From the data in the table, it can be seen that the classification accuracy of the proposed DFL framework (labeled as "Ours") is better than the traditional LSTM model, FedMeta method and FedAvg method in both task settings, which fully proves the effectiveness of our method.

[0171] On the Mini-ImageNet dataset, the LSTM model has a classification accuracy of 53.15% in the 5-way 1-shot task and 67.24% in the 5-way 5-shot task, which is inferior to the FedAvg and DFL frameworks. This shows that although LSTM can capture local time series features, its generalization ability is limited when faced with small samples and heterogeneous data distributions. The FedAvg method has a classification accuracy of 52.75% in the 5-way 1-shot task, slightly lower than LSTM and FedMeta, and a classification accuracy of 68.32% in the 5-way 5-shot task, which exceeds LSTM, but is still lower than the 69.14% of the DFL framework. In contrast, the DFL framework improves 2.2 and 1.9 percentage points over LSTM in the 5-way 1-shot and 5-way 5-shot tasks, respectively, showing stronger generalization ability and adaptability in small sample learning scenarios.

[0172] On the Tiered-ImageNet dataset, the performance of LSTM and FedMeta is similar to that on the Mini-ImageNet dataset. The classification accuracy of the LSTM model in the 5-way 1-shot and 5-way 5-shot tasks is 56.94% and 69.24%, respectively, and the classification accuracy of the FedAvg method is 56.54% and 72.23%, respectively. Although the FedAvg method has a certain improvement over the LSTM in the 5-way 5-shot task, both are significantly lower than the DFL framework. The classification accuracy of the DFL framework in the 5-way 1-shot task is 59.14%,

[0173] It is 2.2 percentage points higher than LSTM, and the classification accuracy in the 5-way 5-shot task reaches 74.14%, which is 1.91 percentage points higher than FedAvg. This shows that the DFL framework can still maintain its robustness and efficiency when processing the Tiered-ImageNet dataset with larger data scale and more complex distribution.

[0174] Overall, the experimental results show that both LSTM and FedAvg methods have certain limitations when dealing with small samples and heterogeneous data distributions, especially in terms of classification accuracy and global consistency, which are inferior to FedMe eta and DFL frameworks. In contrast, the DFL framework has shown advantages in classification accuracy and model consistency by introducing graph neural networks to guide knowledge distillation, adaptive frequency adjustment algorithms, and strategies based on knowledge shift.

[0175] The relationship between the communication cost of each method and the number of clients in 9000 rounds of experiments under different numbers of clients (4, 8, 16, 32). It can be observed that as the number of clients increases, the communication cost shows an upward trend, but the growth rate and performance of different methods are significantly different, such as Figure 3 shown.

[0176] First, DFL (marked as "ours") maintains the lowest communication cost in all client number scenarios, and its growth trend is relatively gentle. This shows that the proposed DFL framework performs well in reducing communication overhead, especially in scenarios with a large number of clients, and its communication optimization strategy can effectively alleviate the communication pressure caused by the increase in nodes. In contrast, the communication costs of other methods are significantly higher than DFL, which shows that traditional methods are difficult to effectively reduce the communication burden when dealing with large-scale client scenarios. Although the communication cost of FedAvg is close to DFL in scenarios with a small number of clients, its communication cost rises rapidly as the number of clients increases, showing a strong dependence on global model aggregation. The communication costs of LSTM and FedMeta in all client number scenarios are higher than DFL and FedAvg, especially when the number of clients is large, the communication cost of the LSTM method is the highest, indicating that its model design fails to fully optimize the communication efficiency in distributed scenarios.

[0177] From the trend point of view, the curves of FedMeta and FedAvg intersect in some node scenarios, which shows that FedMeta can control communication costs more effectively than FedAvg in some scenarios, but this optimization effect gradually weakens under large-scale client numbers. DFL always shows the lowest communication cost in all scenarios, thanks to its adaptive frequency adjustment algorithm and decentralized design, which effectively reduces the communication requirements across nodes. The experimental results fully verify the advantages of the DFL framework in communication cost optimization. Especially in the scenario where the number of clients increases, the communication cost of DFL increases slowly, showing strong scalability and communication efficiency optimization capabilities.

[0178] Ablation experiment

[0179] In order to further verify the effectiveness of each component in the proposed decentralized federated learning framework, we designed and conducted ablation experiments. By removing the key components in the DFL framework, the strategies of the graph neural network-guided knowledge distillation and the adaptive frequency adjustment algorithm, we observed their impact on the overall performance and evaluated the contribution of each component in improving the model performance. In order to ensure the rationality and comparability of the ablation experiment results, we adopted a training-to-convergence approach in the experimental setting and set a unified convergence standard for each experimental group: when the loss function of the validation set does not decrease significantly within 10 consecutive rounds, the model is considered to have converged and the training process stops. This method ensures that each model can achieve its best performance and fully reflects the impact of each component on the model performance.

[0180] However, different models may require different training rounds to converge, which will lead to differences in communication costs and affect the fairness of the comparison. To address this challenge, we focus on the differences in communication costs in the result analysis and compare the communication resources consumed by each experimental group when reaching convergence.

[0181] In addition, in order to balance the number of training rounds and convergence, we set an upper limit of 9000 for the maximum number of training rounds. For example, if the model still does not reach the convergence standard within the preset maximum number of training rounds, the training process will be forced to stop. This not only controls the training time and communication costs, but also allows the model to converge in advance if conditions permit, ensuring the fairness of the experiment and the reliability of the results.

[0182] Through the above experimental settings, we are able to reasonably evaluate the contribution of each key component to the model performance and communication cost, and verify the effectiveness of the proposed decentralized federated learning framework.

[0183]

[0184]

[0185] From the above experimental results, it can be seen that the complete DFL method achieved the highest classification accuracy on both the Mini-ImageNet and Tiered-ImageNet datasets (55.35% and 59.14%, respectively), while the communication cost was also well controlled. In contrast, after removing the knowledge distillation guided by the graph neural network, the classification accuracy decreased (reduced to 53.87% and 58.01%, respectively), indicating that the optimization of the knowledge transfer process by the graph neural network effectively improved the model performance. At the same time, after removing the adaptive frequency adjustment algorithm, although the classification accuracy did not change much, the communication cost increased significantly (increased to 139 and 275, respectively), proving the role of the adaptive frequency adjustment algorithm in effectively reducing communication costs.

[0186] The impact of topology on decentralized federated learning

[0187] In order to evaluate the impact of different topologies on the performance of the decentralized federated learning (DFL) framework, we designed and conducted detailed experiments. Different topologies include star networks, mesh networks, ring networks, and random networks. By comparing the performance of these different topologies on the Mini-ImageNet and Tiered-ImageNet datasets, we can better understand the adaptability and efficiency of different network shapes for the DFL framework.

[0188] On each data set of 4 nodes, we simulated a decentralized distributed data scenario, dividing the data set into multiple nodes, each of which only has part of the data. The specific designs of different topologies are as follows:

[0189] Star network: A central node is connected to multiple peripheral nodes, and all communications are carried out through the central node.

[0190] Mesh network: Each node is connected to multiple other nodes, forming a dense network structure.

[0191] Ring network: Each node is connected only to the two nodes before and after it, forming a closed ring.

[0192] Random network: Each node is randomly connected to several other nodes, forming an irregular topology.

[0193]

[0194] From the above results, it can be seen that the mesh network structure performs best in the decentralized federated learning framework. Its efficient multi-path communication can significantly improve the efficiency of information exchange between nodes and reduce communication delays and bandwidth consumption. Although the star network structure has a high classification accuracy, its central node is prone to form a communication bottleneck, affecting the overall efficiency. The ring network structure has a single communication path, resulting in poor model consistency and convergence speed. The random network structure has better flexibility and adaptability, and has shown relatively stable performance on different data sets.

[0195] These experimental results show that different topologies have a significant impact on the performance of the decentralized federated learning framework. Choosing a suitable topology can effectively improve the classification accuracy of the model, shorten the training time, and reduce the communication cost, thereby improving the overall efficiency and robustness of the DFL framework.

[0196] Experimental Conclusion

[0197] Aiming at the problems of poor model consistency and high communication cost in decentralized federated learning (DFL), this paper proposes an improved DFL framework. Firstly, the present invention guides knowledge distillation through graph neural network to enhance the classification performance of the model; secondly, an adaptive frequency adjustment algorithm is designed to dynamically adjust the model update frequency according to the model offset and communication cost, thereby optimizing the system efficiency; finally, the strategy based on knowledge offset further improves the model consistency between different nodes.

[0198] Experimental results show that the improved DFL framework significantly improves classification accuracy, reduces communication costs, and shortens training time on Mini-ImageNet and Tiered-ImageNet datasets. Future work will further explore more efficient topology optimization methods and adaptive learning algorithms to adapt to more complex and dynamic distributed learning environments.

Claims

1. An adaptive decentralized small sample federated learning framework driven by knowledge distillation, the adaptive decentralized small sample federated learning framework comprising at least three clients, characterized in that: Each client has a decentralized federated learning strategy based on knowledge offset, a knowledge distillation process guided by a graph neural network, and an adaptive frequency adjustment algorithm, where: The decentralized federated learning strategy based on knowledge offset measures and adjusts the knowledge offset between different nodes to achieve the optimization goal of minimizing the weighted sum of each node's local loss and knowledge offset loss. The update process of each node's local model is optimized according to the optimization goal, and the updated local models of each node are aggregated into a global model through local communication between nodes. The knowledge distillation process guided by the graph neural network uses the graph neural network to construct a neighbor relationship graph between nodes, uses the GNN model to extract the relationship features between nodes, and uses the extracted relationship features to perform knowledge distillation to guide the training process of the local model; wherein: the GNN model includes a message passing mechanism and a node aggregation mechanism, the message passing mechanism is used to collect information from neighbor nodes; the node aggregation mechanism is used to aggregate the node's message and its own features to update the node's status; An adaptive frequency adjustment algorithm dynamically adjusts the frequency of model updates according to the model's offset and communication cost, minimizing communication and computing overheads while ensuring model consistency; wherein: the communication cost dynamic adjustment model includes a frequency adjustment function, which is used to determine whether the node needs to update the model in each iteration; in the process of dynamically adjusting the model update frequency based on the communication cost, an incremental model update strategy is introduced, stipulating that each time the model is updated, the node only transmits the change in its model parameters relative to the last update.

2. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The knowledge offset is the knowledge offset between node i and its neighbor node set N(i), which is defined as: Where KL(·||·) represents the Kullback-Leibler divergence, T is the temperature parameter used to adjust the smoothness of the softmax output, and h i and h j Represent the feature representation of node i and its neighbor node j respectively.

3. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The optimization objective is expressed as: Among them, L local (w i ,D i ) indicates that node i is in the local dataset D i The loss on , α is a weight parameter used to balance the local loss and knowledge deviation loss.

4. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The updating process comprises the following steps: S41, node i uses local data set D i and the current model parameters w i , calculate the local loss L local (w i ,D i ), and update the model parameters by gradient descent: Where η is the learning rate; S42, calculate the knowledge deviation KDD between node i and its neighboring nodes i , and adjust the model parameters according to the deviation; by minimizing the knowledge deviation loss, we can get the following update formula: Where β is the adjustment step size, is the gradient of knowledge bias loss with respect to model parameters; S43. Combine the gradients of local loss and knowledge deviation loss to update comprehensive model parameters:

5. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The local model aggregation is that each node sends its local model parameters to neighboring nodes after completing the local model update. The neighboring nodes perform weighted averaging based on the received local model parameters and update their own model parameters for aggregation. The aggregation formula is: in, represents the model parameters of node i in round t+1, represents the model parameters of neighbor node j in round t.

6. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The expression of the message passing mechanism is: in, represents the message representation of node i at level l+1, MSG(·) is the message function, e ij is the edge feature between node i and node j. For node i, its feature is represented by h i , the feature of the neighbor node is represented by h j , where j∈N(i) represents the set of all nodes directly connected to node i; The expression of the node aggregation mechanism is: Where AGG(·) is the aggregation function, and the aggregation method is weighted average.

7. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The forward propagation process of the GNN model is as follows: Among them, σ(·) is the activation function, W (l) is the weight matrix of the lth layer, are the features of nodes i and j at layer l respectively.

8. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The distillation loss function of the knowledge distillation is: L KD (h i ,w i )=KL(softmax(h i / T)||softmax(f(w i ) / T)) (10) Among them, KL(·||·) represents the KL divergence, T is the temperature parameter used to adjust the smoothness of the softmax output, and f(w i ) is the model output of node i.

9. According to claim 1, the adaptive decentralized small sample federated learning framework driven by knowledge distillation is characterized in that: The adaptive frequency adjustment algorithm comprises the following steps: S101. Define model deviation Δ i is the local model w of node i i With the global model The difference between ; the model deviation is expressed as: Among them, N i represents the set of neighbor nodes of node i, and ||·|| represents the Euclidean distance. Model deviation Δ i The larger the value is, the greater the difference between the local model of node i and the models of its neighboring nodes is, and more frequent model updates are needed to ensure consistency; S102. Define communication cost C i is the overhead required for node i to synchronize the model under the current network conditions; the communication cost is expressed as: C i =α1T i，j +β1S i，j (12) Among them, T i,j represents the communication delay between node i and its neighbor node j, S i,j represents the bandwidth consumption between node i and its neighbor node j, α1, β1 are weight parameters used to balance the impact of delay and bandwidth; the communication cost C i The higher the value, the greater the overhead of model synchronization under the current network conditions, and the synchronization frequency needs to be reduced to save resources; In order to dynamically adjust the frequency of model updates for communication cost, a frequency adjustment function τ is introduced. i , which is used to determine whether node i needs to update the model in each iteration; the frequency adjustment function is expressed as: Among them, τ min and τ max Respectively represent the minimum and maximum update frequency, Δ max and C max Represent the maximum values ​​of model deviation and communication cost respectively; In this way, the model update frequency of node i will be dynamically adjusted according to the changes in model deviation and communication cost, thereby minimizing communication and computational overhead while ensuring model consistency; In each iteration, node i first calculates its model deviation Δ i and communication cost C i , and then use the frequency adjustment function τ i Determine whether to update the model: If τ i Greater than a certain threshold τ thresh , then node i updates the model, otherwise skips this update.

10. The adaptive decentralized small sample federated learning framework driven by knowledge distillation according to claim 9, characterized in that: The incremental model update strategy is that each time the model is updated, node i only needs to transmit the change Δw of its model parameters relative to the last update i : in, and denote the model parameters of node i at the tth and t-1th iterations respectively.

Images