Adaptive decentralized federated learning framework based on knowledge distillation driving

Abstract

The application belongs to the technical field of federated learning, and relates to a self-adaptive centralized small sample federated learning method based on knowledge distillation driving. First, a strategy based on knowledge offset is proposed. By measuring the knowledge offset degree between the student model and the teacher model, the local model update of each node is optimized, the difference between the models is reduced, and the overall performance is improved. Second, a graph neural network is used to construct a neighbor relationship graph between nodes to guide the knowledge distillation process and enhance the generalization ability of the model. Finally, an adaptive frequency adjustment algorithm is designed to dynamically adjust the model update frequency according to the model offset degree and communication cost, so as to reduce unnecessary communication overhead and improve training efficiency. Compared with the traditional federated learning method, the framework significantly improves the classification accuracy and reduces the communication cost on the Mini-ImageNet and Tiered-ImageNet data sets.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of federated learning, and particularly relates to a self-adaptive decentralized small-sample federated learning framework based on knowledge distillation driving. BACKGROUND

[0002] In the field of machine learning today, data privacy and security have become crucial issues. With the rapid growth of data volume and the wide distribution of data, traditional centralized data storage and processing methods are facing increasing challenges, especially in the learning scenario of small-sample image data. Decentralized federated learning (DFL) as a new emerging distributed learning method, by keeping data on local devices rather than centralized storage, avoids the risk of privacy leakage and data misuse. However, in the decentralized learning of small-sample image data, there are still a series of problems to be solved. Decentralized small-sample image federated learning faces the problems of data sparsity and data heterogeneity. Since each node can only obtain a small amount of sample data, and the data distribution of different nodes has significant differences, each node cannot fully learn the representative data features. Especially in image tasks, a small amount of data cannot cover the diversity of image features, which makes the model prone to overfitting or underfitting, and it is difficult to generalize to unseen data in the global model. In addition, the high-dimensional characteristics and complex internal structure of image data make it more difficult to train the model under the condition of small sample. This data sparsity not only affects the training effect of each node, but also leads to the consistency and accuracy of the global model among multiple nodes cannot be effectively guaranteed.

[0003] Secondly, the existing DFL methods are still insufficient in optimizing communication and computation costs. In decentralized learning, frequent communication and model updating between nodes are inevitable, but this mechanism often leads to a large computational and communication burden in the small-sample scenario. At each model update, each node needs to exchange a large amount of parameter and gradient information, which not only increases the communication cost, but also puts higher requirements on the computing resources of the nodes. For edge devices and other nodes with limited computing power, this communication and computation burden may lead to reduced system efficiency, and even cause the learning process to be interrupted. Therefore, in small-sample image federated learning, how to reduce the communication and computation cost and ensure efficient model updating has become a key challenge.

[0004] Furthermore, due to the lack of global coordination among nodes in decentralized federated learning, significant differences often exist between the local models of each node. Particularly in few-shot learning scenarios, data imbalance and sample scarcity between nodes cause inconsistent model update directions, leading to performance fluctuations in the global model. This model inconsistency not only makes it difficult for the global model to converge quickly but can also affect the reliability and accuracy of the model in practical applications. This consistency problem is particularly prominent in image tasks because the complexity and diversity of image data make knowledge transfer between nodes more difficult, and model fusion and tuning more complex.

[0005] Decentralized federated learning (DFL) is an important branch of federated learning that aims to eliminate the dependence on a central server, enabling more robust and flexible distributed learning. Traditional federated learning (FL) relies on a central server to coordinate model updates and parameter aggregation among clients, while decentralized federated learning solves the single point of failure problem in centralized solutions by allowing clients to communicate directly with each other, while also reducing communication bottlenecks and latency.

[0006] Within the DFL framework, clients can autonomously decide which other clients to share updates with, and propagate and aggregate parameters through a graph-based communication topology. SpreadGNN is a typical example of decentralized federated learning, proposing a decentralized periodic average stochastic gradient descent method (DPA-SGD) that enables collaborative training by having each client periodically exchange model updates with its neighbors. This approach performs exceptionally well with non-independent and identically distributed (non-IID) data.

[0007] Despite its significant advantages, Decentralized Federated Learning (DFL) also faces several challenges. For example, designing the optimal communication topology to maximize model performance while minimizing communication costs is a current research hotspot. Furthermore, ensuring the convergence and consistency of the global model remains a crucial issue due to the asynchronous nature and local update mechanism of decentralized learning. To address these challenges, researchers have proposed various graph-based decentralized optimization methods, such as DFL-EC and GCFL+. These methods significantly improve the performance of decentralized federated learning by designing flexible topologies and adaptive aggregation strategies.

[0008] One of the main technical challenges in decentralized federated few-shot learning is how to effectively achieve knowledge sharing and transfer in a decentralized environment. Traditional federated learning relies on a central server to coordinate communication and parameter exchange between clients, while each client in a decentralized framework needs to communicate directly with its neighbors. Therefore, designing an efficient graph-based communication topology has become a key issue in decentralized federated few-shot learning. This literature proposes a graph-based few-shot method, introducing the idea of ​​few-shot learning into decentralized federated learning, enabling the model to quickly adapt to different tasks from a small amount of data across multiple nodes, thereby enhancing its generalization ability.

[0009] Measuring knowledge shift is a key technique in decentralized federated learning. By evaluating the knowledge differences between student and teacher models, the local model update process of each node is optimized. FedTAD is a topology-aware, data-agnostic knowledge distillation technique that improves the performance of global graph neural networks by enhancing reliable knowledge transfer in subgraph federated learning.

[0010] The FedEC framework addresses the knowledge graph integrity problem in federated learning by embedding contrastive learning and ensemble methods, leveraging knowledge learned from different sources.

[0011] This paper utilizes a large teacher model trained on a central server to distill knowledge into individual user models, optimizing resource allocation and training strategies to ultimately improve user accuracy and reduce overall latency. Furthermore, the paper investigates the impact of knowledge complexity and differences in teacher and student learning behaviors on distillation efficiency.

[0012] Graph Neural Networks (GNNs) excel at handling graph-structured data. Chauhan proposed a classification prediction method based on few-shot graphs, training a feature extractor in the meta-training phase to extract features from the graph, and using an attention network to predict the actual class label. In the meta-testing phase, the attention network is tuned to quickly adapt to new tasks. While Chauhan proposed a novel meta-learning architecture for graph classification, some limitations remain. FedGraph-KD, a method combining federated learning and knowledge distillation, aims to address the model heterogeneity problem in federated graph neural networks. This method enhances model consistency and robustness by distilling knowledge. Furthermore, the FairGKD method learns a fair graph neural network through partial knowledge distillation, without needing access to all information, significantly improving the fairness of GNNs while maintaining practicality.

[0013] ASFGNN proposes an automatic separation learning model for federated graph neural networks. It improves parameter tuning efficiency by dividing the training process into client-side message passing and federated loss calculation, combined with Bayesian optimization for hyperparameter tuning. The literature on constructing interpretable graph neural networks (EGNNs) is also an innovative attempt. It introduces explicit "contribution" weights into the model through knowledge distillation, thereby improving the model's interpretability and performance; however, its scalability in handling high-dimensional and complex data still needs improvement. Summary of the Invention

[0014] The purpose of this invention is to overcome the shortcomings of the prior art and provide an adaptive decentralized few-shot federated learning framework based on knowledge distillation.

[0015] To achieve the objectives of this invention, the present invention will be implemented using the technical solutions described below.

[0016] An adaptive decentralized few-shot federated learning framework driven by knowledge distillation is disclosed. The framework comprises at least three clients, each possessing a knowledge-biased decentralized federated learning strategy, a graph neural network-guided knowledge distillation process, and an adaptive frequency adjustment algorithm.

[0017] The decentralized federated learning strategy based on knowledge bias measures and adjusts the knowledge bias between different nodes to achieve the optimization objective of minimizing the weighted sum of the local loss and knowledge bias loss of each node. Based on the optimization objective, the update process of the local model of each node is optimized, and the updated local models of each node are aggregated into a global model through local communication between nodes.

[0018] The knowledge distillation process guided by graph neural networks (GNNs) constructs a neighbor relationship graph between nodes, extracts relationship features between nodes using a GNN model, and uses the extracted relationship features for knowledge distillation to guide the training process of the local model. Specifically, the GNN model includes a message passing mechanism and a node aggregation mechanism. The message passing mechanism is used to collect information from neighboring nodes; the node aggregation mechanism is used to aggregate the messages and features of a node to update the node's state.

[0019] An adaptive frequency adjustment algorithm dynamically adjusts the model update frequency based on the model offset and communication cost, minimizing communication and computation overhead while ensuring model consistency. Specifically, the communication cost dynamic adjustment model includes a frequency adjustment function to determine whether a node needs to update its model in each iteration. During the process of dynamically adjusting the model update frequency based on communication cost, an incremental model update strategy is introduced, stipulating that each node only transmits the changes in its model parameters relative to the previous update during each model update.

[0020] As a preferred embodiment of the present invention, the knowledge offset is a node and its neighboring node set Knowledge bias is defined as:

[0021] (1)

[0022] in, Indicates the Kullback-Leibler divergence. This is a temperature parameter used to adjust the smoothness of the softmax output. and Representing nodes respectively and its neighboring nodes The characteristics are represented.

[0023] As a preferred embodiment of the present invention, the optimization objective is expressed as:

[0024] (2)

[0025] in, Represents a node In local dataset The losses on It is a weighting parameter used to balance local loss and knowledge shift loss.

[0026] As a preferred embodiment of the present invention, the updating process includes the following steps:

[0027] S41, Node Using local datasets and current model parameters Calculate local loss And update the model parameters using gradient descent:

[0028] (3)

[0029] in, It is the learning rate;

[0030] S42, Computation Node Knowledge offset from its neighboring nodes And adjust the model parameters according to the knowledge shift; by minimizing the knowledge shift loss, we can obtain the following update formula:

[0031] (4)

[0032] in, It's about adjusting the step size. It is the gradient of the knowledge shift loss with respect to the model parameters;

[0033] S43. Combine the gradients of the local loss and the knowledge shift loss to perform a comprehensive model parameter update:

[0034] (5)

[0035] In a preferred embodiment of the present invention, the local model aggregation involves each node sending its local model parameters to neighboring nodes after completing its local model update. The neighboring nodes then perform a weighted average of the received local model parameters and update their own model parameters accordingly. The aggregation formula is as follows:

[0036] (6)

[0037] in, Represents a node exist Wheel model parameters, Representing neighboring nodes In the Wheel model parameters.

[0038] As a preferred embodiment of the present invention, the expression of the message passing mechanism is:

[0039] (7)

[0040] in, Represents a node In the The message from the layer indicates that, It is a message function. It is a node and nodes Edge features between nodes Its characteristics are represented as The characteristics of neighboring nodes are represented as ,in , representing all nodes The set of directly connected nodes;

[0041] The expression for the node aggregation mechanism is:

[0042] (8)

[0043] in, It is an aggregation function, and the aggregation method is weighted average.

[0044] As a preferred embodiment of the present invention, the forward propagation process of the GNN model is as follows:

[0045] (9)

[0046] in, It is an activation function. It is the first The weight matrix of the layer, These are the features of nodes i and j at layer l, respectively.

[0047] As a preferred embodiment of the present invention, the distillation loss function of the knowledge distillation is:

[0048] (10)

[0049] in, Denotes KL divergence, This is a temperature parameter used to adjust the smoothness of the softmax output. It is a node The model output.

[0050] As a preferred embodiment of the present invention, the adaptive frequency adjustment algorithm includes the following steps:

[0051] S101, Define model offset For nodes local model With global model The difference between them; the model offset is expressed as:

[0052] (11)

[0053] in, Represents a node The set of neighboring nodes, Represents the Euclidean distance. Model offset. The larger the value, the more likely it is to be a node. The greater the difference between the local model and the models of its neighboring nodes, the more frequently the model needs to be updated to ensure consistency.

[0054] S102, Define communication cost For nodes The overhead required to perform a model synchronization under current network conditions; the communication cost is expressed as:

[0055] (12)

[0056] in, Represents a node and its neighboring nodes Communication delay between them Represents a node and its neighboring nodes Bandwidth consumption between These are weighting parameters used to balance the effects of latency and bandwidth; communication cost. The higher the value, the greater the overhead of model synchronization under the current network conditions, and the more necessary it is to reduce the synchronization frequency to save resources.

[0057] To dynamically adjust the model update frequency to reduce communication costs, a frequency adjustment function is introduced. Used to determine nodes Whether a model update is needed in each iteration; the frequency adjustment function is expressed as:

[0058] (13)

[0059] in, and These represent the minimum and maximum update frequencies, respectively. and These represent the maximum values ​​of model offset and communication cost, respectively.

[0060] In this way, the node The model update frequency will be dynamically adjusted according to changes in model offset and communication cost, thereby minimizing communication and computation overhead while ensuring model consistency.

[0061] In each iteration, the node First, calculate its model offset. and communication costs Then use the frequency adjustment function Determine whether to perform a model update: If Greater than a certain threshold Then the node Perform a model update; otherwise, skip this update.

[0062] As a preferred embodiment of the present invention, the incremental model update strategy is specifically a node... Each time the model is updated, only the changes in its model parameters relative to the previous update need to be transmitted. :

[0063] (14)

[0064] in, and Representing nodes respectively In the Second and third Model parameters for the next iteration.

[0065] Beneficial effects

[0066] This invention aims to address the main problems existing in small-sample DFL, and improve model consistency, communication efficiency and overall performance.

[0067] First, a knowledge-biased DFL strategy is proposed. By measuring the knowledge bias between the student model and the teacher model, the local model update process of each node is optimized. In this way, DFL can transfer more consistent knowledge between different nodes, reduce the differences between models, and thus improve the overall model performance.

[0068] Secondly, a knowledge extraction method based on graph neural networks is proposed. By constructing a neighbor relationship graph between nodes to guide the knowledge distillation process, each node can not only use its own data for training, but also learn from neighboring nodes, thereby improving the generalization ability of the model.

[0069] Finally, to address the communication and computation costs in DFL, an adaptive frequency adjustment algorithm is designed. This algorithm reduces unnecessary communication overhead and improves overall training efficiency by dynamically adjusting the model update frequency. At the same time, by introducing a trade-off mechanism between model offset and communication cost, it ensures that communication efficiency is maximized while maintaining model consistency. Attached Figure Description

[0070] Fig. 1 This is a schematic diagram of the structure of the framework described in this invention;

[0071] Fig. 2 This is a diagram of the decentralized federated learning topology in this invention;

[0072] Fig. 3 This is a graph showing the relationship between communication costs and the number of clients. Detailed Implementation

[0073] The present invention will be further described in conjunction with the accompanying drawings and embodiments.

[0074] As an embodiment of the present invention, such as Figs. 1 to 3 As shown, an adaptive decentralized few-shot federated learning framework based on knowledge distillation is presented. This framework includes at least three clients, each possessing a knowledge-biased decentralized federated learning strategy, a graph neural network-guided knowledge distillation process, and an adaptive frequency adjustment algorithm.

[0075] The decentralized federated learning strategy based on knowledge bias measures and adjusts the knowledge bias between different nodes to achieve the optimization objective of minimizing the weighted sum of the local loss and knowledge bias loss of each node. Based on the optimization objective, the update process of the local model of each node is optimized, and the updated local models of each node are aggregated into a global model through local communication between nodes.

[0076] The knowledge distillation process guided by graph neural networks (GNNs) constructs a neighbor relationship graph between nodes, extracts relationship features between nodes using a GNN model, and uses the extracted relationship features for knowledge distillation to guide the training process of the local model. Specifically, the GNN model includes a message passing mechanism and a node aggregation mechanism. The message passing mechanism is used to collect information from neighboring nodes; the node aggregation mechanism is used to aggregate the messages and features of a node to update the node's state.

[0077] An adaptive frequency adjustment algorithm dynamically adjusts the model update frequency based on the model offset and communication cost, minimizing communication and computation overhead while ensuring model consistency. Specifically, the communication cost dynamic adjustment model includes a frequency adjustment function to determine whether a node needs to update its model in each iteration. During the dynamic adjustment of the model update frequency based on communication cost, an incremental model update strategy is introduced, stipulating that each node only transmits the changes in its model parameters relative to the previous update during each model update.

[0078] In decentralized federated learning, models across different nodes can exhibit significant differences due to variations in data distribution and training processes. This inconsistency leads to fluctuations and instability in the overall model performance. Therefore, this paper proposes a knowledge-bias-based decentralized federated learning strategy. By measuring and adjusting the knowledge bias between different nodes, the strategy optimizes the model update process of each node, thereby improving overall model consistency and performance.

[0079] Knowledge Distillation Divergence (KDD) measures the difference in knowledge representation between the student model and the teacher model. Specifically, for nodes... and its neighboring node set Its knowledge bias can be defined as:

[0080] (1)

[0081] in, Indicates the Kullback-Leibler divergence. This is a temperature parameter used to adjust the smoothness of the softmax output. and Representing nodes respectively and its neighboring nodes The characteristics are represented.

[0082] By calculating the knowledge offset, we can evaluate the consistency of each node with its neighboring nodes in terms of knowledge representation. A large knowledge offset indicates a significant difference in knowledge between nodes, and appropriate adjustments can be made to reduce this difference. In decentralized federated learning, our optimization objective is to minimize the weighted sum of the local loss and the knowledge offset loss for each node. Specifically, for a node... Its optimization objective can be expressed as:

[0083] (2)

[0084] in, Represents a node In local dataset The losses on It is a weighting parameter used to balance local loss and knowledge shift loss.

[0085] During the local model update process at each node, we need to consider both the training error of the local data and the knowledge consistency with neighboring nodes. The specific update process is as follows.

[0086] node Using local datasets and current model parameters Calculate local loss And update the model parameters using gradient descent:

[0087] (3)

[0088] in, It is the learning rate.

[0089] Next, calculate the nodes. Knowledge offset from its neighboring nodes And adjust the model parameters according to the knowledge shift. By minimizing the knowledge shift loss, we can obtain the following update formula:

[0090] (4)

[0091] in, It's about adjusting the step size. It is the gradient of the knowledge shift loss with respect to the model parameters.

[0092] Finally, the gradients of the local loss and the knowledge shift loss are combined to perform a comprehensive model parameter update:

[0093] (5)

[0094] This comprehensive update strategy ensures the accuracy of the model on local data while reducing knowledge differences with neighboring nodes, thereby improving the overall consistency and performance of the model.

[0095] In decentralized federated learning, the aggregation of the global model does not rely on a central server, but is achieved through local communication between nodes. Specifically, after each node completes its local model update, it sends its model parameters to its neighboring nodes. The neighboring nodes then perform a weighted average of the received parameters and update their own model parameters accordingly. The specific aggregation formula is as follows:

[0096] (6)

[0097] in, Represents a node exist Wheel model parameters, Representing neighboring nodes In the Wheel model parameters.

[0098] By employing this distributed model aggregation strategy, our proposed decentralized federated learning strategy based on knowledge shift can effectively avoid the bottleneck problem of central servers, effectively solve the problems of model consistency and performance fluctuation, and improve the robustness and scalability of the model.

[0099] Graph Neural Networks (GNNs) excel at processing graph-structured data, particularly suitable for modeling complex relationships between nodes. Knowledge distillation (KD), as a model compression technique, transfers knowledge from a large-scale teacher model to a small-scale student model, allowing the model to maintain high performance while reducing computational resource consumption. This invention proposes a knowledge distillation method guided by graph neural networks to improve the overall performance of decentralized federated learning (DFL).

[0100] In the framework of this invention, each node is regarded as a node in a graph, and the edges between nodes represent the relationships between them.

[0101] First, we use graph neural networks to construct a neighbor relationship graph between nodes, extract the relationship features between nodes through GNN, and then use these features to perform knowledge distillation, thereby guiding the training process of the local model.

[0102] The GNN model constructed in this invention comprises two main parts: a message passing mechanism and a node aggregation mechanism. The message passing mechanism is used to collect information from neighboring nodes. For nodes... Its characteristics are represented as The characteristics of neighboring nodes are represented as ,in , representing all nodes A set of directly connected nodes. The message passing mechanism can be represented as:

[0103] (7)

[0104] in, Represents a node In the The message from the layer indicates that, It is a message function. It is a node and nodes Edge features between them.

[0105] Aggregation mechanisms are used to combine a node's messages and its own characteristics to update the node's state. The feature update formula is:

[0106] (8)

[0107] in, It is an aggregation function, and the aggregation method is a weighted average. Combining the above two mechanisms, the forward propagation process of GNN is as follows:

[0108] (9)

[0109] in, It is an activation function. It is the first The weight matrix of the layer, These are the features of nodes i and j at layer l, respectively. In the knowledge distillation process, this invention extracts the node features from the GNN. As input to the teacher model, the knowledge transfer effect between the teacher and student models is measured using the Kullback-Leibler (KL) divergence. Specifically, the distillation loss function can be expressed as:

[0110] (10)

[0111] in, Denotes KL divergence, This is a temperature parameter used to adjust the smoothness of the softmax output. It is a node The model output.

[0112] By minimizing the distillation loss, the knowledge extracted by the GNN can be effectively transferred to the local model, thereby improving the model's generalization ability and robustness. Using the above method, our graph neural network-guided knowledge distillation process can effectively utilize the relational features between nodes, improving the model's performance in scenarios with small samples and heterogeneous data. This not only improves the overall model consistency and robustness of decentralized federated learning but also significantly reduces communication and computation costs, enhancing system efficiency and scalability.

[0113] In decentralized federated learning (DFL), communication and computation costs are key factors affecting system efficiency and performance. Traditional federated learning methods typically rely on fixed communication and model update frequencies, which can easily lead to resource waste and performance instability in heterogeneous network environments and dynamic data distributions. To address these issues, we design an adaptive frequency adjustment algorithm that optimizes communication and computation costs by dynamically adjusting the model update frequency, while ensuring model consistency and performance.

[0114] The core idea of ​​the adaptive frequency adjustment algorithm is to dynamically adjust the model update frequency based on the model offset and communication cost, thereby reducing unnecessary communication overhead. Specifically, model offset measures the difference between the current model and the global model, while communication cost reflects the overhead of model synchronization in the current network environment. By comprehensively considering these two factors, we can determine the optimal frequency for each node to update the model at different times.

[0115] First, define the model offset. For nodes local model With global model The difference between them. The model offset can be expressed as:

[0116] (11)

[0117] in, Represents a node The set of neighboring nodes, Represents the Euclidean distance. Model offset. The larger the value, the more likely it is to be a node. The greater the difference between the local model and the models of its neighboring nodes, the more frequently the model needs to be updated to ensure consistency.

[0118] Next, we define the communication cost. For nodes The overhead required to perform a model synchronization under current network conditions. Communication cost can be expressed as:

[0119] (12)

[0120] in, Represents a node and its neighboring nodes Communication delay between them Represents a node and its neighboring nodes Bandwidth consumption between This is a weighting parameter used to balance the impact of latency and bandwidth. Communication cost. The higher the value, the greater the overhead of model synchronization under the current network conditions, and the more necessary it is to reduce the synchronization frequency to save resources.

[0121] To dynamically adjust the model update frequency, we introduce a frequency adjustment function. Used to determine nodes Does the model need to be updated in each iteration? The frequency adjustment function can be expressed as:

[0122] (13)

[0123] in, and These represent the minimum and maximum update frequencies, respectively. and These represent the maximum values ​​of model offset and communication cost, respectively. In this way, the node... The model update frequency will be dynamically adjusted according to changes in model offset and communication cost, thereby minimizing communication and computation overhead while ensuring model consistency.

[0124] In each iteration, the node First, calculate its model offset. and communication costs Then use the frequency adjustment function Determine whether to update the model. If Greater than a certain threshold Then the node If the model is not updated, then the update is skipped. This adaptive frequency adjustment mechanism can effectively reduce unnecessary model synchronization and improve the overall efficiency of the system.

[0125] Furthermore, to further optimize communication costs, we introduced an incremental model update strategy. Traditional federated learning methods typically require the transmission of complete model parameters, which incurs significant communication overhead in cases of large-scale models and frequent updates. The incremental model update strategy, however, only transmits the changes in model parameters, thereby significantly reducing the amount of communication data. Specifically, nodes... Each time the model is updated, only the changes in its model parameters relative to the previous update need to be transmitted. :

[0126] (14)

[0127] in, and Representing nodes respectively In the Second and third The model parameters for the next iteration. Incremental model updates not only reduce the amount of communication data, but also effectively reduce transmission latency and bandwidth consumption, thereby further improving the system's communication efficiency.

[0128] The adaptive frequency adjustment algorithm dynamically adjusts the model update frequency based on model offset and communication costs, thereby minimizing communication and computational overhead while ensuring model consistency. This algorithm not only improves the overall efficiency of the decentralized federated learning framework but also enhances the system's adaptability and robustness in heterogeneous network environments.

[0129] Convergence analysis of adaptive frequency adjustment

[0130] To ensure the effectiveness and stability of the adaptive frequency adjustment algorithm in decentralized federated learning (DFL), we need to analyze its convergence. This invention will theoretically prove the convergence of the adaptive frequency adjustment algorithm and analyze its performance under different conditions.

[0131] First, we make the following basic assumptions:

[0132] Assumption 1. Model loss function It is continuously differentiable and strongly convex in the model parameter space. This implies the existence of constants. and , such that for any and have:

[0133] (15)

[0134] Assumption 2. The gradient is bounded:

[0135] Assuming each node gradient Bounded, meaning there exists a constant. Makes any and nodes have:

[0136] (16)

[0137] Assumption 3. Communication delay and bandwidth:

[0138] Assuming communication delay and bandwidth The training process is bounded, meaning there exists a constant. and Make:

[0139] (17)

[0140] We consider nodes The model update process aims to minimize the weighted sum of the local loss function and the knowledge shift loss. Node The optimization objective can be expressed as:

[0141] (18)

[0142] Among them, the local loss function and knowledge shift loss They are respectively:

[0143] (19)

[0144] (20)

[0145] In each iteration, the node The model update process can be represented as:

[0146] (twenty one)

[0147] Among them, learning rate It is a small positive number.

[0148] According to hypothesis 1, the loss function is strongly convex and has a constant. and satisfy:

[0149] (twenty two)

[0150] in, It is the globally optimal solution.

[0151] Using the gradient descent update formula, we have:

[0152] (twenty three)

[0153] Expanding the squared terms on the right and using the assumptions of strong convexity and bounded gradients, we can obtain:

[0154] (twenty four)

[0155] because It is strongly convex and its gradient is bounded. We have:

[0156]

[0157] (25)

[0158] Substitute these values ​​into the update formula and choose an appropriate learning rate. Make We can obtain:

[0159] (26)

[0160] make ,because It can guarantee Therefore, we can conclude that:

[0161] (27)

[0162] This indicates that in each iteration, the model parameters They will all converge at a certain rate Converging to the global optimum Through an adaptive frequency adjustment algorithm, the node In each iteration, based on the model offset and communication costs Dynamically adjust the model update frequency to reduce unnecessary communication overhead. Since each model update only occurs once... This approach, which performs updates in real-time, significantly reduces communication frequency and bandwidth consumption. Furthermore, the incremental model update strategy further reduces the amount of communication data, optimizing overall communication costs. Through the above analysis, we have demonstrated the convergence of the adaptive frequency adjustment algorithm in decentralized federated learning. Moreover, by dynamically adjusting the model update frequency and employing an incremental model update strategy, we can effectively reduce communication and computational costs, improving the overall efficiency and performance of the system.

[0163] Experimental Section

[0164] To verify the effectiveness of the proposed decentralized federated learning framework and its various innovative technologies, this invention designed and conducted multiple sets of experiments, including federated learning comparison experiments, ablation experiments, experiments on the impact of adaptive frequency optimization algorithms, and experiments on the impact of different topologies on decentralized federated learning. The following are detailed experimental setups and results analysis.

[0165] Experimental setup

[0166] We conducted experiments on the Mini-ImageNet and TieredImageNet datasets using the ResNet12 model, with details consistent with the model architecture proposed in TADAM. We used stochastic gradient descent as the optimizer with an initial learning rate of 0.1, which was adjusted during training according to a predetermined design: for MiniImageNet, the learning rate was reduced at iterations 12000, 14000, and 16000; for TieredImageNet, the learning rate was halved every 24000 episodes. For all experiments, the model was tested on 2000 episodes; during training, 4 episodes were selected per batch. On each dataset, we simulated a decentralized distributed data scenario, dividing the dataset into 4-node and 8-node partitions, with the following topology: Fig. 2 As shown, each node possesses only partial data, and experiments are conducted with 5 samples per new category and 1 new category to be processed (5-way 1-shot) and 5 samples per new category and 5 new categories to be processed (5-way 5-shot). To ensure fairness and comparability, we compare the performance of traditional federated learning methods (such as FedAvg and FedMeta) and the Lstm model with the proposed DFL framework under the same conditions. We use similar dataset partitions and training parameters to ensure that each method is evaluated in a similar environment. Classification accuracy and communication cost are the main metrics for evaluating model performance. Classification accuracy measures the model's predictive ability on the test set, while communication cost represents the amount of data transferred between nodes after convergence throughout the training process.

[0167] Federated Learning Comparative Experiment

[0168] To evaluate the performance difference between the proposed decentralized federated learning (DFL) framework and the traditional federated learning (FedAvg) method, we selected two representative datasets—Mini-ImageNet and Tiered-ImageNet—and conducted detailed comparative experiments with four nodes. The experiments aimed to verify the effectiveness of the DFL framework in image classification tasks.

[0169] Table 1 Comparative Experimental Results

[0170] Table 1 presents the experimental results for 5-way 1-shot and 5-way 5-shot classification tasks on the Mini-ImageNet and Tiered-ImageNet datasets. The data in the table clearly shows that the proposed DFL framework (labeled "Ours") outperforms traditional LSTM models, the FedMeta method, and the FedAvg method in both task settings, fully demonstrating the effectiveness of our method.

[0171] On the Mini-ImageNet dataset, the LSTM model achieves a classification accuracy of 53.15% in the 5-way 1-shot task and 67.24% in the 5-way 5-shot task, underperforming the FedAvg and DFL frameworks. This indicates that while LSTM can capture local time-series features, its generalization ability is limited when facing small sample sizes and heterogeneous data distributions. The FedAvg method achieves a classification accuracy of 52.75% in the 5-way 1-shot task, slightly lower than LSTM and FedMeta, and a classification accuracy of 68.32% in the 5-way 5-shot task, surpassing LSTM but still lower than the DFL framework's 69.14%. In contrast, the DFL framework improves performance by 2.2 and 1.9 percentage points over LSTM in the 5-way 1-shot and 5-way 5-shot tasks, respectively, demonstrating stronger generalization ability and adaptability in small-sample learning scenarios.

[0172] On the Tiered-ImageNet dataset, the performance of LSTM and FedMeta is similar to that on the Mini-ImageNet dataset. The LSTM model achieves classification accuracies of 56.94% and 69.24% in the 5-way 1-shot and 5-way 5-shot tasks, respectively, while the FedAvg method achieves accuracies of 56.54% and 72.23%, respectively. Although the FedAvg method shows some improvement over LSTM in the 5-way 5-shot task, both are significantly lower than the DFL framework. The DFL framework achieves a classification accuracy of 59.14% in the 5-way 1-shot task.

[0173] It improves upon LSTM by 2.2 percentage points, and achieves a classification accuracy of 74.14% in the 5-way 5-shot task, which is 1.91 percentage points higher than FedAvg. This demonstrates that the DFL framework maintains its robustness and efficiency when handling the larger and more complex Tiered-ImageNet dataset.

[0174] Overall, the experimental results show that both LSTM and FedAvg methods have certain limitations when handling small sample sizes and heterogeneous data distributions, particularly in classification accuracy and global consistency compared to the FedMeta and DFL frameworks. In contrast, the DFL framework demonstrates advantages in multiple aspects, including classification accuracy and model consistency, by introducing graph neural networks to guide knowledge distillation, adaptive frequency adjustment algorithms, and knowledge shift-based strategies.

[0175] The relationship between communication cost and client number for each method over 9000 rounds of experiments was investigated with varying client numbers (4, 8, 16, 32). It can be observed that communication cost increases with the number of clients, but the rate of increase and performance differ significantly among different methods. Fig. 3 As shown.

[0176] First, DFL (labeled "ours") maintains the lowest communication cost across all client count scenarios, and its growth trend is relatively gradual. This indicates that the proposed DFL framework performs excellently in reducing communication overhead, especially in scenarios with a large number of clients, where its communication optimization strategy effectively alleviates the communication pressure caused by the increase in nodes. In contrast, the communication costs of other methods are significantly higher than DFL, indicating that traditional methods struggle to efficiently reduce the communication burden when dealing with large-scale client scenarios. While FedAvg's communication cost is close to DFL's in scenarios with a small number of clients, its communication cost rises rapidly with the increase in the number of clients, demonstrating a strong dependence on global model aggregation. LSTM and FedMeta have higher communication costs than DFL and FedAvg across all client count scenarios, especially with LSTM having the highest communication cost when the number of clients is large, indicating that its model design has not adequately optimized communication efficiency in distributed scenarios.

[0177] The trends show that the curves for FedMeta and FedAvg intersect in some node scenarios, indicating that FedMeta can control communication costs more effectively than FedAvg in certain situations, but this optimization effect gradually weakens with a large number of clients. DFL, on the other hand, consistently exhibits the lowest communication cost across all scenarios, thanks to its adaptive frequency adjustment algorithm and decentralized design, which effectively reduces cross-node communication requirements. Experimental results fully validate the advantages of the DFL framework in communication cost optimization. Especially in scenarios with an increasing number of clients, DFL's communication cost increases more slowly, demonstrating strong scalability and communication efficiency optimization capabilities.

[0178] ablation experiment

[0179] To further validate the effectiveness of each component in the proposed decentralized federated learning (DFL) framework, we designed and conducted ablation experiments. By sequentially removing key components from the DFL framework—the graph neural network-guided knowledge distillation and adaptive frequency adjustment algorithms—we observed their impact on overall performance and evaluated the contribution of each component to improving model performance. To ensure the reasonableness and comparability of the ablation experiment results, we adopted a training-to-convergence approach in our experimental setup, setting a uniform convergence criterion for each experimental group: the model was considered converged when the loss function on the validation set did not significantly decrease within 10 consecutive rounds, and the training process was then stopped. This method ensured that each model reached its optimal performance, fully reflecting the impact of each component on model performance.

[0180] However, different models may require different training epochs to converge, leading to differences in communication costs and affecting the fairness of the comparison. To address this challenge, we focused on the differences in communication costs in our results analysis, comparing the communication resources consumed by each experimental group when reaching convergence.

[0181] Furthermore, to balance training epochs and convergence, we set a maximum limit of 9000 training epochs. For example, if the model fails to reach the convergence criterion within the preset maximum number of training epochs, the training process will be forcibly stopped. This controls training time and communication costs while allowing the model to converge earlier if conditions permit, ensuring the fairness of the experiment and the reliability of the results.

[0182] Through the above experimental setup, we can reasonably evaluate the contribution of each key component to model performance and communication cost, and verify the effectiveness of the proposed decentralized federated learning framework.

[0183] The experimental results show that the complete DFL method achieved the highest classification accuracy on both the Mini-ImageNet and Tiered-ImageNet datasets (55.35% and 59.14%, respectively), while also effectively controlling communication costs. In contrast, removing the knowledge distillation guided by the graph neural network resulted in a decrease in classification accuracy (to 53.87% and 58.01%, respectively), indicating that the optimization of the knowledge transfer process by the graph neural network effectively improved model performance. Furthermore, while removing the adaptive frequency adjustment algorithm did not significantly change the classification accuracy, it significantly increased communication costs (to 139 and 275, respectively), demonstrating the effectiveness of the adaptive frequency adjustment algorithm in reducing communication costs.

[0184] The impact of topology on decentralized federated learning

[0185] To evaluate the impact of different topologies on the performance of the decentralized federated learning (DFL) framework, we designed and conducted detailed experiments. The different topologies included star networks, mesh networks, ring networks, and random networks. By comparing the performance of these different topologies on the Mini-ImageNet and Tiered-ImageNet datasets, we were able to better understand the adaptability and efficiency of different network shapes to the DFL framework.

[0186] For each dataset across four nodes, we simulated a decentralized distributed data scenario, dividing the dataset into multiple nodes, each containing only a subset of the data. The specific designs for different topologies are as follows:

[0187] Star network: A central node is connected to multiple peripheral nodes, and all communication is conducted through the central node.

[0188] Mesh network: Each node is connected to multiple other nodes, forming a dense network structure.

[0189] Ring network: Each node is connected to only the two nodes before and after it, forming a closed loop.

[0190] Random networks: Each node is randomly connected to several other nodes, forming an irregular topology.

[0191] The results show that the mesh network structure performs best in the decentralized federated learning framework. Its efficient multi-path communication significantly improves the information exchange efficiency between nodes and reduces communication latency and bandwidth consumption. While the star network structure has high classification accuracy, its central node easily becomes a communication bottleneck, affecting overall efficiency. The ring network structure suffers from poor model consistency and convergence speed due to its single communication path. The random network structure, on the other hand, exhibits good flexibility and adaptability, demonstrating relatively stable performance across different datasets.

[0192] These experimental results demonstrate that different topologies have a significant impact on the performance of decentralized federated learning (DFL) frameworks. Choosing an appropriate topology can effectively improve the model's classification accuracy, shorten training time, and reduce communication costs, thereby enhancing the overall efficiency and robustness of the DFL framework.

[0193] Experimental conclusions

[0194] To address the issues of poor model consistency and high communication costs in decentralized federated learning (DFL), this invention proposes an improved DFL framework. First, this invention enhances the model's classification performance by using graph neural networks to guide knowledge distillation. Second, an adaptive frequency adjustment algorithm is designed to dynamically adjust the model update frequency based on model offset and communication costs, thereby optimizing system efficiency. Finally, a knowledge offset-based strategy further improves model consistency among different nodes.

[0195] Experimental results show that the improved DFL framework significantly improves classification accuracy, reduces communication costs, and shortens training time on the Mini-ImageNet and Tiered-ImageNet datasets. Future work will further explore more efficient topology optimization methods and adaptive learning algorithms to adapt to more complex and dynamic distributed learning environments.

Claims

1. A knowledge distillation-driven adaptive decentralized few-shot federated learning method, wherein the adaptive decentralized few-shot federated learning method includes at least three clients, characterized in that... Each client features a knowledge-biased decentralized federated learning strategy, a graph neural network-guided knowledge distillation process, and an adaptive frequency adjustment algorithm, where: The decentralized federated learning strategy based on knowledge shift measures and adjusts the knowledge shift between different nodes to minimize the weighted sum of the local loss and knowledge shift loss of each node. Based on this optimization objective, the update process of the local model of each node is optimized, and the updated local models of each node are aggregated into a global model through local communication between nodes. Here, the knowledge shift is a node's... and its neighboring node set Knowledge bias is defined as: (1) in, Indicates the Kullback-Leibler divergence. This is a temperature parameter used to adjust the smoothness of the softmax output. and Representing nodes respectively and its neighboring nodes Feature representation; The knowledge distillation process guided by graph neural networks (GNNs) constructs a neighbor relationship graph between nodes, extracts relationship features between nodes using a GNN model, and uses the extracted relationship features for knowledge distillation to guide the training process of the local model. Specifically, the GNN model includes a message passing mechanism and a node aggregation mechanism. The message passing mechanism is used to collect information from neighboring nodes; the node aggregation mechanism is used to aggregate the messages and features of a node to update the node's state. An adaptive frequency adjustment algorithm dynamically adjusts the model update frequency based on model offset and communication cost, minimizing communication and computational overhead while ensuring model consistency. Specifically, the communication cost dynamic adjustment model includes a frequency adjustment function to determine whether a node needs to update its model in each iteration. During this process, an incremental model update strategy is introduced, stipulating that each node only transmits the changes in its model parameters relative to the previous update during each update. The adaptive frequency adjustment algorithm includes the following steps: S11, Define model offset For nodes local model With global model The difference between them; the model offset is expressed as: (11) in, Represents a node The set of neighboring nodes, Represents Euclidean distance, model offset The larger the value, the more likely it is to be a node. The greater the difference between the local model and the models of its neighboring nodes, the more frequently the model needs to be updated to ensure consistency. S12, Define communication cost For nodes The overhead required to perform a model synchronization under current network conditions; the communication cost is expressed as: (12) in, Represents a node and its neighboring nodes Communication delay between them Represents a node and its neighboring nodes Bandwidth consumption between These are weighting parameters used to balance the effects of latency and bandwidth; communication cost. The higher the value, the greater the overhead of model synchronization under the current network conditions, and the more necessary it is to reduce the synchronization frequency to save resources. To dynamically adjust the model update frequency to reduce communication costs, a frequency adjustment function is introduced. Used to determine nodes Whether a model update is needed in each iteration; the frequency adjustment function is expressed as: (13) in, and These represent the minimum and maximum update frequencies, respectively. and These represent the maximum values ​​of model offset and communication cost, respectively. In this way, the node The model update frequency will be dynamically adjusted according to changes in model offset and communication cost, thereby minimizing communication and computation overhead while ensuring model consistency. In each iteration, the node First, calculate its model offset. and communication costs Then use the frequency adjustment function Determine whether to perform a model update: If Greater than a certain threshold Then the node Perform a model update; otherwise, skip this update.

2. The adaptive decentralized few-shot federated learning method based on knowledge distillation as described in claim 1, characterized in that, The optimization objective is expressed as: (2) in, Represents a node In local dataset The losses on It is a weighting parameter used to balance local loss and knowledge shift loss.

3. The adaptive decentralized few-shot federated learning method based on knowledge distillation as described in claim 1, characterized in that, The update process includes the following steps: S31, Node Using local datasets and current model parameters Calculate local loss And update the model parameters using gradient descent: (3) in, It is the learning rate; S32, Computation Node Knowledge offset from its neighboring nodes And adjust the model parameters according to the knowledge shift; by minimizing the knowledge shift loss, we can obtain the following update formula: (4) in, It's about adjusting the step size. It is the gradient of the knowledge shift loss with respect to the model parameters; S33. Combine the gradients of the local loss and the knowledge shift loss to perform a comprehensive model parameter update: (5)。 4. The adaptive decentralized few-shot federated learning method based on knowledge distillation as described in claim 1, characterized in that, The local model aggregation involves each node sending its local model parameters to its neighboring nodes after updating its local model. The neighboring nodes then perform a weighted average of the received local model parameters and update their own model parameters accordingly. The aggregation formula is as follows: (6) in, Represents a node exist Wheel model parameters, Representing neighboring nodes In the Wheel model parameters.

5. The adaptive decentralized few-shot federated learning method based on knowledge distillation as described in claim 1, characterized in that, The expression for the message passing mechanism is: (7) in, Represents a node In the The message from the layer indicates that, It is a message function. It is a node and nodes Edge features between nodes Its characteristics are represented as The characteristics of neighboring nodes are represented as ,in , representing all nodes The set of directly connected nodes; The expression for the node aggregation mechanism is: (8) in, It is an aggregation function, and the aggregation method is weighted average.

6. The adaptive decentralized few-shot federated learning method based on knowledge distillation as described in claim 5, characterized in that, The forward propagation process of the GNN model is as follows: (9) in, It is an activation function. It is the first The weight matrix of the layer, These are the features of nodes i and j at layer l, respectively.

7. The adaptive decentralized few-shot federated learning method based on knowledge distillation as described in claim 1, characterized in that, The distillation loss function for knowledge distillation is: (10) in, Denotes KL divergence, This is a temperature parameter used to adjust the smoothness of the softmax output. It is a node The model output.

8. The adaptive decentralized few-shot federated learning method based on knowledge distillation as described in claim 1, characterized in that, The incremental model update strategy is specifically for nodes. Each time the model is updated, only the changes in its model parameters relative to the previous update need to be transmitted. : (14) in, and Representing nodes respectively In the Second and third Model parameters for the next iteration.

Images