A cloud-edge federated learning method and system, and a storage medium

Abstract

The application discloses a cloud-edge federated learning method, system and storage medium, and belongs to the field of model training optimization. First, the cloud constructs a dynamic clustering mechanism based on the model features and data distribution information uploaded by the edge terminal, reduces the statistical heterogeneity within the group, and the edge terminal performs local training and intra-group model aggregation according to the cloud clustering result, thereby improving the consistency and adaptability of the local cluster model. Second, a decoupled knowledge transfer mechanism is adopted, the global model and the local cluster model are respectively decoupled into feature layers and classification layers, hierarchical knowledge alignment is performed in the distillation process, the learning ability of the local model to intermediate feature expression and classification decision boundary is strengthened, and therefore the convergence speed and generalization performance of the model in a heterogeneous data environment are improved. Further, the technical problems of model performance decline and weak generalization ability caused by data heterogeneity in cloud-edge federation in the prior art are solved.

Description

Technical Field

[0001] This application relates to the field of model training optimization technology, and in particular to a cloud-edge federated learning method, system, and storage medium. Background Technology

[0002] With the deep integration of cloud computing and edge computing, federated learning has gradually evolved into a distributed learning paradigm that integrates cloud resources, namely Cloud-Edge Federated Learning (CEFL). In this architecture, the cloud is responsible for global model coordination and aggregation, while edge nodes participate in local training as data providers. However, compared with traditional federated scenarios, the heterogeneity of computing resources, network conditions, and data distribution is more significant in cloud-edge environments. This not only exacerbates the complexity and harmfulness of data heterogeneity but may also lead to a decline in global model performance and even model convergence problems. Therefore, in resource-heterogeneous cloud-edge environments, building an efficient and robust heterogeneous federated learning framework to overcome existing technical bottlenecks has become an important research topic in this field.

[0003] While cloud-edge federated learning (CEFL) has demonstrated significant potential in protecting privacy and improving data utilization, it still faces severe heterogeneity challenges in practical deployments. CEFL heterogeneity can be broadly categorized into three types: statistical heterogeneity, model heterogeneity, and system heterogeneity. Statistical heterogeneity, being the most critical and challenging issue, refers to the significant differences in data distribution among participants due to variations in data sources, usage scenarios, or user behavior. These differences not only cause shifts in terminal model updates, leading to performance fluctuations in the global model during aggregation, but can also result in training instability, decreased model accuracy, and even convergence failure. However, in real-world cloud-edge scenarios, data collected by edge devices is often highly personalized and localized. Examples include medical image data collected from different hospitals, operational logs generated by industrial equipment, and driving behavior recorded by vehicle terminals, all exhibiting significant non-independent and identically distributed (Non-IID) characteristics. Statistical heterogeneity can lead to biased terminal updates, causing the model's gradient direction to deviate from the global optimum. This results in slower convergence, increased training epochs, and in some extreme cases, excessive differences between local clusters can cause aggregation failure, significantly weakening the generalization ability of the global model. To address this challenge, recent research has focused on strategies to mitigate statistical heterogeneity, aiming to improve the stability and performance of federated learning in complex and heterogeneous environments.

[0004] To mitigate statistical heterogeneity, current research primarily focuses on two directions: terminal optimization strategies and server optimization strategies. Terminal optimization strategies are based on terminal data resources and reduce the impact of heterogeneity on the global model by adjusting the local model training process. Common methods include terminal selection, local calibration, and clustering. Terminal selection is a commonly used method to address statistical heterogeneity. Some researchers have proposed improving efficiency based on probability conditions according to the current model and the data on the terminal, while others have proposed prioritizing terminals with higher local losses to participate in aggregation, utilizing their more representative gradient information to accelerate convergence and effectively alleviate the data heterogeneity problem. However, these methods often ignore the collaborative information between terminals, resulting in limited performance improvement. Another type of method is the terminal grouping strategy based on clustering. For example, some researchers have proposed cluster federated learning, which uses clustering algorithms to group terminals according to data distribution, allowing for collaborative optimization within groups and reducing heterogeneity interference. Some researchers have proposed FlexCFL, which effectively addresses data heterogeneity through elastic grouping and dynamic migration, achieving a balance between accuracy and communication efficiency. However, its performance depends on the quality of the initial grouping and has high computational and synchronization overhead. To further improve the performance of clustered federated learning, scholars have proposed ROCFL, which, through a data quality-aware mechanism and dynamic resource allocation strategy, improves accuracy and robustness in complex heterogeneous scenarios while balancing privacy protection and system efficiency. However, this method still faces challenges such as strong clustering sensitivity and difficulty in hyperparameter tuning. While terminal optimization strategies have achieved positive results in improving local training adaptability and reducing the impact of device heterogeneity, they still have certain limitations. For example, terminal selection often ignores the collaborative potential between devices, and clustering strategies are sensitive to initial grouping, hyperparameter settings, and computational costs, making it difficult to maintain stable performance in large-scale, highly heterogeneous real-world environments. More importantly, terminal optimization mainly affects the local model, lacking systematic control over global model aggregation and knowledge transfer, making it difficult to completely solve the model bias problem caused by statistical differences. Therefore, in recent years, the academic community has gradually shifted its focus to server-side optimization strategies, hoping to further mitigate the impact of data heterogeneity on model performance from a global perspective.

[0005] Compared to terminal optimization strategies, server-side optimization methods address statistical heterogeneity from the perspective of global model aggregation and knowledge sharing. Common methods include adjusting aggregation strategies, data sharing, and knowledge distillation. Traditional aggregation strategies balance heterogeneity by weighted averaging of terminal model parameters. While ensuring fair weight allocation, this can negatively impact global model performance when terminal models with large datasets have poor accuracy. Data sharing methods reduce the differences in data distribution between terminals by uploading shared data or generating synthetic samples. While effectively mitigating distribution bias, this inevitably introduces potential privacy risks, often requiring additional protection mechanisms such as differential privacy. Knowledge distillation, initially used for model compression and training heterogeneous models, has been widely applied in federated learning to address statistical heterogeneity. This method utilizes a teacher-student model architecture, using a high-performance teacher model to guide student model training, thereby improving the model's generalization ability. Knowledge distillation can be broadly categorized into statistical distillation, feature-based distillation, and response-based distillation. Among these, response-based methods (i.e., logits or probability distributions) have attracted significant attention due to their high knowledge transfer efficiency. Some scholars have proposed the FedMD method, which achieves heterogeneous federated learning through knowledge distillation and public dataset mediation, allowing different models to collaborate without sharing data or architecture, indirectly mitigating data heterogeneity. However, its effectiveness is highly dependent on the quality and representativeness of the public dataset. Other scholars have proposed FedGEMS, which adopts a protocol matching FedMD and further improves upon it. It addresses data heterogeneity in federated learning through a selective knowledge fusion strategy, aggregating knowledge from multiple terminals using a server model and suppressing the impact of low-quality data through weight allocation. However, it still faces challenges related to reliance on public data and computational overhead. To avoid relying on public datasets to address statistical heterogeneity, some scholars have proposed exchanging logits to address this issue. Logits capture important features of each category, allowing student models to mimic the logits distribution (softened probabilities) of teacher models, thus conveying richer knowledge. Building on this, some scholars have proposed the FedCD method, which generates high-quality teacher logits through server ensemble learning as the target for knowledge distillation across all terminals, effectively addressing heterogeneity. Some scholars have proposed that FedNTD, during local training, introduces a non-true class distillation loss in addition to cross-entropy loss. This mitigates the forgetting problem caused by local training by preserving the global model's knowledge of non-locally distributed data, thereby reducing the distribution bias of the global model. However, this method ignores the logits generated by the true class when calculating the Softmax score of the KD-based loss. Other scholars have found that in existing frameworks, using an inaccurate global model for a specific classification may mislead the training of local cluster models. To address this, the FedCAD method is proposed, which dynamically allocates class-adaptive distillation weights. When the global model has a higher weight in a certain class, the local cluster model learns more knowledge.Building upon this, some scholars have proposed the FedSSD method, which achieves dynamic knowledge transfer through a class-sample two-layer adaptive distillation mechanism. By evaluating the class confidence of the global model on public datasets and the prediction reliability of local samples, it effectively suppresses bias caused by differences in data distribution. While FedSSD boasts superior performance, it relies on public datasets for class weight initialization, and the calculation of sample-level confidence adds additional overhead. Although server-side optimization strategies demonstrate significant advantages in improving global model consistency and reducing distribution bias, they also suffer from high computational costs, reliance on public data, and significant privacy risks. Particularly in scenarios without public data, the effectiveness of distillation strategies is limited, impacting the overall system efficiency.

[0006] In summary, both existing terminal and server-side optimization strategies have made positive progress in mitigating the statistical heterogeneity problem in federated learning. Terminal strategies effectively reduce model drift risk through local training process adjustments, clustering, and personalized design, while server-side strategies improve the consistency and generalization ability of the global model through aggregation optimization and knowledge distillation. However, both methods suffer from problems such as strong dependence on public data, insufficient cloud-edge collaboration, weak cross-layer optimization mechanisms, and poor adaptability to complex heterogeneous environments. Especially in practical cloud-edge collaboration scenarios, it is difficult to simultaneously address the needs of terminal resource constraints, intra-group variability suppression, and global knowledge consistency, thus limiting the improvement of overall model performance.

[0007] It is evident that statistical heterogeneity is one of the key challenges limiting the performance of federated learning. Although cloud-edge federated learning frameworks based on knowledge distillation can alleviate this problem to some extent, the stability and generalization of model training are still significantly affected in practical applications.

[0008] There is currently no effective solution to the technical problems of degraded model training performance and weak generalization ability caused by data heterogeneity in cloud-edge federation, as mentioned above in the existing technologies. Summary of the Invention

[0009] The embodiments of this disclosure provide a cloud-edge federated learning method, system, and storage medium to at least solve the technical problems of decreased model training performance and weak generalization ability caused by data heterogeneity in cloud-edge federation in the prior art.

[0010] According to one aspect of the present disclosure, a cloud-edge federated learning method is provided, comprising: Step S1: The cloud server initializes a global model and receives model feature vectors uploaded by all terminals; Step S2: At the beginning of each communication round, the cloud server determines whether the clustering trigger condition is met: if it is met, it clusters all terminals into K clusters based on the latest received model feature vectors and sends the terminal list of its managed clusters and the global model to the edge server; if it is not met, it directly sends the global model to the edge server; Step S3: The edge server initializes the local cluster model of its managed clusters based on the global model and sends the local cluster model and the global model to the terminals within the clusters; Step S4: The terminals iteratively train the local model using a dual-teacher decoupled distillation mechanism, in each iteration: (a) calculate the cross-entropy loss of the local model; (b) combine the global model and the local cluster model. (a) Using the local model as the student model and the teacher model as the teacher model, calculate the distillation loss of the student model and each teacher model in the intermediate feature layer and classification layer respectively; (c) Weight the losses calculated in steps (a) and (b) to obtain the total loss; (d) Update the local model parameters based on the total loss; Step S5: After the terminal training is completed, extract the model feature vector of the local model and upload it to the cloud server, and upload the updated local model parameters to the corresponding edge server; Step S6: The edge server weights and aggregates the local model parameters uploaded by the terminal in the cluster, generates a new local cluster model and uploads it to the cloud server; Step S7: The cloud server aggregates all local cluster models and updates the global model; Step S8: If the current communication round has not reached the preset total round, repeat steps S2 to S7; otherwise, output the terminal clustering results and the final global model.

[0011] According to another aspect of the present disclosure, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0012] This application designs two main mechanisms: an adaptive clustering module driven by terminal model features and a collaborative optimization module based on hierarchical decoupled distillation. First, the cloud (cloud server) constructs a dynamic clustering mechanism triggered on demand (conditional judgment) based on model feature vectors uploaded by edge terminals (reflecting model state and data distribution heterogeneity), reducing intra-group statistical heterogeneity. The edge (including edge servers and terminals) performs local training and intra-group model aggregation based on the cloud clustering results, improving the consistency and adaptability of local cluster models. Second, a hierarchical (feature layer and classification layer) decoupled knowledge transfer mechanism is adopted. By using the global model and local cluster models as teacher models respectively, during local training on the terminal, the student model (local model) simultaneously aligns its knowledge with both teachers at the feature layer and classification layer, strengthening the local model's ability to learn intermediate feature representations and classification decision boundaries, thereby improving the convergence speed and generalization performance of the model in heterogeneous data environments. This solves the technical problems of model performance degradation and weak generalization ability caused by data heterogeneity in cloud-edge federation in existing technologies. Attached Figure Description

[0013] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this application, illustrate exemplary embodiments of this disclosure and are used to explain this disclosure, but do not constitute an undue limitation of this disclosure. In the drawings:

[0014] Figure 1 This is a flowchart of the cloud-edge federated learning method according to the embodiments of this application;

[0015] Figure 2 This is a framework diagram of the cloud-edge federated learning system according to the embodiments of this application;

[0016] Figure 3 This is a schematic diagram of a feature-driven adaptive clustering module according to an embodiment of this application;

[0017] Figure 4 This is a schematic diagram of a collaborative optimization module based on decoupled distillation according to an embodiment of this application;

[0018] Figure 5 This is a schematic diagram illustrating the heterogeneity partitioning according to the embodiments of this application;

[0019] Figure 6 This is a schematic diagram illustrating the accuracy of various methods for different heterogeneities in Fashion-MNIST according to embodiments of this application;

[0020] Figure 7 This is a schematic diagram illustrating the accuracy of various methods for different heterogeneities of CIFAR-10 according to the embodiments of this application;

[0021] Figure 8This is a diagram illustrating the effect of distillation temperature according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Example

[0025] According to this embodiment, a method embodiment of cloud-edge federated learning is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] According to the first aspect of this embodiment, a cloud-edge federated learning method is provided. Figure 1 A flowchart illustrating the method is shown below. (Refer to...) Figure 1 As shown, the method includes:

[0027] Step S1: The cloud server initializes the global model and receives model feature vectors uploaded by all terminals;

[0028] Step S2: At the beginning of each communication round, the cloud server determines whether the clustering trigger condition is met: if it is met, it clusters all terminals into K clusters based on the latest received model feature vectors and sends the terminal list and global model of the managed clusters to the edge server; if it is not met, it directly sends the global model to the edge server.

[0029] Step S3: The edge server initializes the local cluster model of its managed cluster based on the global model, and distributes the local cluster model and the global model to the terminals within the cluster;

[0030] Step S4: The terminal uses a dual-teacher decoupled distillation mechanism to iteratively train the local model. In each iteration: (a) calculate the cross-entropy loss of the local model; (b) use the global model and the local cluster model as teacher models and the local model as student models, and calculate the distillation loss of the student model and each teacher model in the intermediate feature layer and classification layer respectively; (c) weight the losses calculated in steps (a) and (b) to obtain the total loss; (d) update the parameters of the local model based on the total loss.

[0031] Step S5: After the terminal training is completed, extract the model feature vector of the local model and upload it to the cloud server, and upload the updated local model parameters to the corresponding edge server.

[0032] Step S6: The edge server weights and aggregates the local model parameters uploaded by the terminals within the cluster, generates a new local cluster model, and uploads it to the cloud server;

[0033] Step S7: The cloud server aggregates all local cluster models and updates the global model;

[0034] Step S8: If the current communication round has not reached the preset total round, repeat steps S2 to S7; otherwise, output the terminal clustering results and the final global model.

[0035] In this embodiment of the invention, when the federated learning task starts, a global model is initialized via a cloud server, receiving model feature vectors uploaded by all terminals (corresponding to step S1). Specifically, the cloud server is responsible for generating and initializing a basic version of the global machine learning model (e.g., the initial weights of a deep neural network). Subsequently, the cloud server receives model feature vectors uploaded by all terminal devices participating in the federated learning. These feature vectors are not raw data, but rather key information extracted from the terminal's local model (which may be the model after the previous training round or the initial model) that can characterize the model state, data distribution heterogeneity, and local knowledge characteristics. Thus, initializing the global model provides the starting point for learning, and the fact that all terminals upload model feature vectors (rather than raw data) is the core privacy protection mechanism of federated learning, avoiding the privacy leakage risk brought about by data centralization. Furthermore, these model feature vectors provide a data-driven basis for subsequent steps (especially clustering), enabling the cloud server to perceive the heterogeneity of the terminal model state.

[0036] Next, at the start of each communication round, the cloud server determines whether the clustering trigger conditions are met: if met, it clusters all terminals into K clusters based on the latest received model feature vectors and sends the terminal list and global model of the managed clusters to the edge servers; if not met, it directly sends the global model to the edge servers (corresponding to step S2). Specifically, at the start of each round of federated learning communication, the cloud server first checks the preset clustering trigger conditions (e.g., whether the current round is a fixed-period round (e.g., every 5 rounds), whether the change in the average cosine similarity of the terminal model feature vectors exceeds a threshold, or whether the model performance improvement after the previous round of aggregation has stagnated). If the conditions are met, it uses the latest batch of received model feature vectors and employs a clustering algorithm (e.g., K-Means, spectral clustering) to divide all terminal devices into K clusters. After division, the cloud server notifies each edge server of the terminal device list of the cluster it manages and sends the current version of the global model to each edge server. If the clustering conditions are not met, the cloud server directly broadcasts the latest global model to all edge servers. Therefore, by introducing a dynamic clustering mechanism triggered on demand, the computational overhead of clustering in every round is avoided, and clustering is only triggered when conditions are met (such as a significant increase in model differences or performance bottlenecks). Furthermore, clustering is based on model feature vectors (reflecting data distribution / model state), grouping terminals with similar data characteristics or model update directions, laying the foundation for subsequent cluster-level personalization (local cluster models). By delegating management tasks to edge servers, the load on cloud servers is reduced, and edge computing resources are utilized, achieving more efficient cloud-edge collaboration.

[0037] Next, the edge server initializes the local cluster model of its managed cluster based on the global model, and distributes both the local cluster model and the global model to the terminals within the cluster (corresponding to step S3). Specifically, after receiving information from the cloud server (whether it contains a cluster list and the global model, or only the global model), the edge server creates a local cluster model for each cluster it manages. This local cluster model is typically initialized as a copy of the currently received global model. Then, the edge server sends this newly initialized local cluster model along with the current global model to all terminal devices within that cluster. The global model represents the overall knowledge, ensuring basic consistency in learning across all terminals. The local cluster model serves as the carrier for cluster-level personalization; its initialization based on the global model guarantees consistency of the starting point and the inheritance of knowledge. By simultaneously distributing both to the terminals, the necessary teacher model is provided for subsequent dual-teacher distillation, laying the foundation for personalized terminal learning.

[0038] Subsequently, the terminal employs a dual-teacher decoupled distillation mechanism to iteratively train the local model (corresponding to step S4). Specifically, after receiving the global model and the local cluster model, the terminal device uses local private data to perform multiple rounds of iterative training on its local model. The training employs a dual-teacher decoupled distillation mechanism: (a) Calculate the standard cross-entropy loss of the local model on the local data to measure the difference between the model's prediction and the true label. (b) Treat the global model and the local cluster model as two independent teacher models, and the local model as the student model. Calculate the distillation loss of the student model and each teacher model on the specified intermediate feature layer (such as the output of a convolutional layer) and the final classification layer (such as the output of Softmax) to capture the signals that the student model learns from the two teachers in terms of feature representation and output probability distribution. (c) Combine the cross-entropy loss obtained in step (a) and the multiple distillation losses related to the two teachers obtained in step (b) according to preset weights to obtain the final total loss. (d) Use an optimizer (such as SGD) to backpropagate based on the total loss and update the parameters of the local model. The dual-teacher distillation mechanism allows the local model to learn simultaneously from global knowledge (generalization) and intra-cluster personalized knowledge (adapting to local similar data distributions). Decoupling is achieved by separately calculating the losses of the feature layer and the classification layer, enabling the model to more precisely mimic the teacher's knowledge at different levels. This collaboratively optimizes the local model, strengthens its ability to learn intermediate feature representations and classification decision boundaries, improves the convergence speed and generalization performance of the model in heterogeneous data environments, and effectively alleviates the local model drift problem caused by Non-IID data.

[0039] Subsequently, after the terminal training is completed, the model feature vector of the local model is extracted and uploaded to the cloud server, and the updated local model parameters are uploaded to the corresponding edge server (corresponding to step S5). Specifically, after completing local iterative training, the terminal device extracts the model feature vector from its updated local model, then uploads this newly extracted model feature vector to the cloud server, and uploads the updated local model parameters to the edge server corresponding to its cluster. This completes the terminal-side computation and communication for this round. By uploading the model feature vector to the cloud server, the latest basis is provided for the next round (or future rounds) of clustering, enabling the cloud server to continuously perceive changes in the terminal model's state. By uploading the local model parameters to the edge server, data support is provided for subsequent intra-cluster model aggregation.

[0040] Next, the edge server weighted aggregates the local model parameters uploaded by terminals within the cluster to generate a new local cluster model, which is then uploaded to the cloud server (corresponding to step S6). Specifically, the edge server collects the updated local model parameters uploaded by all terminals within its managed cluster. Then, the edge server aggregates these local model parameters using a weighted average (e.g., assigning weights based on the amount of local data from each terminal) to form a new local cluster model for that cluster. Finally, the edge server uploads this newly generated local cluster model parameter to the cloud server. This achieves cluster model aggregation at the edge layer, generating a better new local cluster model that represents the common knowledge of the cluster by integrating the learning results of all terminals within the cluster. Furthermore, by dividing the aggregation to the edge layer, the communication volume of directly uploading to the cloud server is significantly reduced (only one aggregated model needs to be uploaded instead of multiple terminal models), and edge computing resources are fully utilized.

[0041] Next, the cloud server aggregates all local cluster models and updates the global model (corresponding to step S7). Specifically, the cloud server receives new local cluster models representing their respective clusters from all edge servers. Then, the cloud server aggregates these local cluster models using a weighted average (e.g., assigning weights based on the total amount of terminal data within each cluster or the number of clusters) to obtain the updated global model. This achieves global model aggregation in the cloud, integrating knowledge from different clusters (representing different data distribution patterns) and fusing cluster-level personalized information to update a more powerful and generalizable global model, ensuring the continuous evolution and unification of global knowledge.

[0042] Finally, if the current communication round has not reached the preset total number of rounds, steps S2 to S7 are repeated; otherwise, the terminal clustering results and the final global model are output (corresponding to step S8). Specifically, the cloud server checks whether the current communication round counter has reached the preset total number of training rounds. If not, the round counter is incremented by 1, and the process jumps back to step S2 to start a new round of federated learning iterations (including possible clustering, model distribution, local training, aggregation, etc.). If the total number of rounds has been reached, the entire federated learning training process ends, and the terminal clustering results formed by the last clustering (i.e., the cluster to which each terminal belongs) and the final global model obtained from the last update are finally output. This realizes the cyclical iteration of the entire federated learning process. Through multiple iterations, the model (global and local clusters) is continuously optimized. The final output clustering results provide insights into the similarity of terminal data distribution, which helps to understand the system or conduct subsequent analysis / services. The final global model is the core result of federated learning collaborative training and can be directly used for downstream tasks or as a base model.

[0043] As described in the background section, existing terminal and server optimization strategies have made positive progress in mitigating the statistical heterogeneity problem in federated learning. Terminal strategies effectively reduce model drift risk through local training process adjustments, clustering, and personalized design, while server strategies improve the consistency and generalization ability of the global model through aggregation optimization and knowledge distillation. However, both methods suffer from problems such as strong dependence on public data, insufficient cloud-edge collaboration, weak cross-layer optimization mechanisms, and poor adaptability to complex heterogeneous environments. Especially in practical cloud-edge collaboration scenarios, it is difficult to simultaneously address the needs of terminal resource constraints, intra-group variability suppression, and global knowledge consistency, thus limiting the improvement of overall model performance. It is evident that statistical heterogeneity is one of the key challenges restricting the performance of federated learning. Although cloud-edge federated learning frameworks based on knowledge distillation can alleviate this problem to some extent, the stability and generalization of model training are still significantly affected in practical applications.

[0044] In view of this, this application designs two major mechanisms: an adaptive clustering module driven by terminal model features and a collaborative optimization module based on hierarchical decoupled distillation. First, the cloud (cloud server) constructs a dynamic clustering mechanism triggered on demand (conditional judgment) based on the model feature vectors uploaded by the edge terminal (reflecting the heterogeneity of model state and data distribution), reducing intra-group statistical heterogeneity. The edge (including edge servers and terminals) performs local training and intra-group model aggregation based on the cloud clustering results, improving the consistency and adaptability of local cluster models. Second, a hierarchical (feature layer and classification layer) decoupled knowledge transfer mechanism is adopted. By using the global model and local cluster models as teacher models respectively, during local training on the terminal, the student model (local model) simultaneously aligns its knowledge with both teachers at the feature layer and classification layer, strengthening the local model's ability to learn intermediate feature representations and classification decision boundaries, thereby improving the convergence speed and generalization performance of the model in heterogeneous data environments. This solves the technical problems of model performance degradation and weak generalization ability caused by data heterogeneity in cloud-edge federation in existing technologies.

[0045] Figure 2This paper demonstrates the overall architecture of the cloud-edge collaborative federated learning system based on decoupled distillation proposed in this application. The system consists of two parts: a feature-driven adaptive clustering module and a decoupled distillation-based collaborative optimization module. The feature-driven adaptive clustering module dynamically performs clustering in the cloud based on model features and data distribution information uploaded by the terminals, balancing data similarity and communication overhead to achieve a global clustering strategy. Meanwhile, at the edge, each terminal device trains its local model based on the clustering results distributed from the cloud and performs model aggregation within its group to obtain local cluster models, thereby effectively reducing intra-group heterogeneity and improving intra-group model consistency. This module, with the cloud leading the global clustering decision and the edge performing local optimization and aggregation, works collaboratively to optimize the overall system performance.

[0046] Combination Figure 2 As shown, the collaborative optimization module based on decoupled distillation mainly designs a decoupled knowledge transfer mechanism to fully leverage the collaborative advantages of the global perspective in the cloud and the local perspective in the edge. Specifically, in each round of federated training, the cloud (i.e., the cloud server) aggregates the cluster models uploaded by each cluster to form a new global model, which is then distributed to all edge clusters as a unified global teacher model. On the edge side, the local cluster models within each cluster serve as cluster teacher models, and the terminal local models within each cluster are regarded as student models. Knowledge distillation is performed separately with the global teacher model and the local cluster teacher models distributed by the cloud to obtain more generalizable feature representations and decision boundaries. To improve the effectiveness and fine-grained control of knowledge transfer, the distillation process is decoupled into two levels: feature layer distillation and Softmax layer distillation. Feature layer distillation strengthens the model's ability to learn the intrinsic structure of the data by aligning intermediate feature representations; Softmax layer distillation constrains the model's output probability distribution, improving the accuracy and stability of classification decisions, and realizing the effective transfer and sharing of knowledge at different levels, thereby improving the overall performance of the cloud-edge federated learning system.

[0047] In this embodiment of the invention, to address the problem that traditional clustering methods cannot accurately reflect the true similarity between terminals due to the significant heterogeneity of terminal data distribution and rapid changes in model parameters in cloud-edge federated learning scenarios, a feature-driven adaptive clustering method is proposed, such as... Figure 3 As shown. First, a distributed data awareness mechanism is constructed, enabling each terminal node to recognize local data distribution characteristics. With model behavior Indicator feature vector extraction The data is then uploaded to a cloud server for clustering. Next, the cloud server uses a similarity metric to cluster all terminal nodes based on the received feature vectors, and distributes the clustering results to the edge server. Each terminal node then trains its model using its local private dataset based on the distributed global model parameters. Finally, the edge server uses a weighted aggregation method to aggregate the model parameters uploaded by the terminals within the cluster, obtaining a local model within the cluster. This model is retained locally for local collaborative optimization and uploaded to the cloud server for global collaborative optimization.

[0048] In the cloud-edge collaborative federated learning framework, a dynamic clustering mechanism based on data distribution characteristics constructs a similarity metric by analyzing the local data distribution characteristics of terminal nodes (such as category distribution and feature mean) and model update behavior characteristics, grouping isomorphic terminal devices into the same cluster. Compared to traditional clustering methods based on model parameters, this clustering strategy can more accurately reflect the true distribution similarity between terminals. Within each cluster, terminal devices generate personalized model updates through local training, and then perform intra-cluster federated aggregation to form a local cluster model. The cluster model can better adapt to the data characteristics of terminals within each cluster, thereby improving the model's generalization ability and overall performance.

[0049] To this end, feature construction is performed. After all terminal models have been trained locally, the model feature vectors are extracted and uploaded. .

[0050] (1)

[0051] in, Indicates the first i The terminal in the first t The statistical characteristics of local data in each training round, such as the proportion of each class of samples, the mean and variance of the data, are used to represent the distribution of terminal data. Indicates the first i The terminal in the first t The difference between the local model within a cluster in each training round and the global model in the previous training round is used to measure the direction of model updates. The above features are normalized to form a low-dimensional feature vector.

[0052] After collecting features uploaded by all terminals, the cloud server uses the K-means algorithm for clustering. , indicating the first k Terminals within a cluster exhibit similar data and model behaviors. Indicates the number of clusters, This indicates that all terminals within the cluster are at the 1st... t Feature vectors uploaded in each communication round Cluster k The set of terminals; k=1~K;

[0053] To evaluate the clustering effect, the global silhouette coefficient is used, and the calculation formula is as follows:

[0054] (2)

[0055] in, Indicates terminal i The average distance to other terminals within its cluster; Indicates terminal i The average distance to the nearest neighbor cluster; N Indicates terminal i The total number of all terminals within the cluster. When A higher value indicates that in the first... t In a single communication round, the clustering structure is good and the clustering is reasonable; otherwise, there may be mis-clustering or fuzzy structure.

[0056] During training, the data distribution and model parameters of the terminal devices may dynamically change over time. This change may stem from temporal drift of the data collected by the terminal devices, task changes, or overfitting trends in the local model, making it difficult for the statically partitioned cluster structure to maintain its effectiveness in the long term, thus affecting the model's convergence speed and final performance. To address this, this application designs an adaptive dynamic clustering strategy that periodically evaluates changes in terminal behavior and dynamically adjusts the cluster structure to adapt to distribution drift and improve federated optimization performance.

[0057] To reasonably control the timing of cluster structure reconstruction, the following triggering conditions are introduced to comprehensively evaluate the evolution of terminal behavior and the current clustering effect:

[0058] (1) Intra-cluster model heterogeneity: By calculating the average variance of the model parameters or feature representations of each terminal within a cluster, when this variance exceeds a set threshold, it indicates a significant increase in intra-cluster heterogeneity, requiring re-clustering; or

[0059] (2) Inter-cluster contour coefficient: The contour coefficient is used to measure the rationality of the current clustering structure. If the index drops by more than the preset ratio after multiple rounds of local training, it indicates that the clustering effect has degraded and the clustering structure needs to be reconstructed.

[0060] To address the potential additional communication and computational burdens associated with the dynamic evaluation process, the system employs a fixed evaluation cycle. Specifically, feature drift detection and clustering quality evaluation are performed once every n rounds of communication. This strategy ensures that the clustering structure exhibits good dynamic adaptability and optimization performance while controlling system resource consumption.

[0061] After the terminals complete a new round of clustering, the system adopts the following optimization strategy within each cluster. To alleviate the communication pressure between the cloud server and the terminal nodes, the terminal nodes train on their local private data and upload the local model to the cluster center (the corresponding edge server). The edge server then performs intra-cluster weighted aggregation of the terminal model parameters based on the amount of local data on the terminal, resulting in a local cluster model that is uploaded to the cloud server for global aggregation. The intra-cluster aggregation strategy is as follows:

[0062] (3)

[0063] in, Indicates the first k The local cluster model after each cluster update; Indicates the first k The first cluster within the cluster i Local data volume per terminal; Cluster k The total amount of data from all terminals within the system; Cluster k The set of terminals; Indicates the first k The first cluster within the cluster i Local model parameters for each terminal.

[0064] In non-independent and identically distributed environments, local models on different terminals exhibit significant differences in feature representation capabilities and decision boundaries. This makes it difficult to effectively integrate cross-terminal knowledge through direct aggregation within the parameter space, and may even lead to model degradation or insufficient generalization ability. However, knowledge distillation does not fuse parameters but learns knowledge from the soft objectives output by the teacher model. To effectively alleviate data heterogeneity in cloud-edge federated scenarios and take into account the differences between individual terminals, this application proposes a collaborative optimization module based on decoupled distillation, such as... Figure 4 As shown. First, the terminal node receives the global model from the edge server as its local initial student model. First, the model is trained. Second, during training, a dual-model approach, using a global model and a local cluster model, is employed to collaboratively transfer knowledge. A decoupling distillation strategy is introduced to decouple the model into an intermediate feature layer and a softmax layer, where feature alignment and class distillation are performed separately to achieve multi-level knowledge transfer. Feature loss is calculated in the intermediate feature layer. To achieve feature alignment, target class distillation is performed simultaneously in the Softmax layer to maintain the discriminative ability of the main classes, while non-target class distillation enhances the discrimination of negative samples, and the overall decoupled distillation loss is calculated. Finally, the cloud server collects the local cluster models uploaded by the edge servers, performs global aggregation, distributes the updated global model to all edge nodes, and further broadcasts it to the terminal nodes to enter the next round of training.

[0065] The collaborative optimization module design for decoupled distillation is based on the distillation concept. In traditional knowledge distillation teacher models, the softened output is typically in the Softmax layer of the model network. Compared to intermediate feature layers, this layer possesses stronger semantic information, as shown in the following equation:

[0066] (4)

[0067] in, It is the softened non-target category The probability, The teacher model applies to non-target categories. i The logit probability. This refers to the distillation temperature coefficient. Traditional logit distillation only fits the distribution by calculating the KL divergence between the teacher and student models, but it does not fully consider the role of the softmax layer in the distillation process. Therefore, the softmax layer is divided into two parts: target class and non-target class. The specific decoupling process is as follows:

[0068] Assume the teacher model is The student model is The total number of sample categories is From the above formula, we can see that Representing non-target categories in the teacher model The probability, It is the non-target category in the student model. To ensure the student model learns the knowledge of the teacher model while preserving its adaptability to a specific cluster dataset, the probability is calculated by minimizing the KL divergence between the student and teacher models. The formula for representing the loss between the two is as follows:

[0069] (5)

[0070] in, This indicates that the teacher model is related to the target category. t The predicted probability; Indicates an index that is not the target category; Represents the target category in the student model t The probability value, The total probability value for non-target categories, and also defined Define the classification probability of a single non-target class in the teacher model. Define the total probability value of non-target categories in the teacher model. Define the total probability of the student model for non-target categories. For student models, for a single non-target category i The probability of is then:

[0071] (6)

[0072] in, This indicates that the teacher model is related to the target category. t The score output; This indicates that the teacher model applies to non-target categories. j The logits output (i.e., the score output); The classification layer of the teacher model represents the non-target categories. i The logits output; The classification layer of the teacher model represents the non-target categories. k The score output; This indicates that the student model is related to the target category. t The score output; The classification layer of the student model represents the non-target category. j The logits output; The classification layer of the student model represents the non-target category. i The logits output; The classification layer of the student model represents the non-target category. k The output of logits; index i , j and k Used to distinguish variables; Indicates the total number of sample categories; It is the distillation temperature coefficient; Indicates a non-target class index, and Same meaning;

[0073] Equation (5) can be rewritten as:

[0074] (7)

[0075] in, Target Class Distillation Loss (TCKD) measures the difference between the teacher and student models at the logits level, which helps the student model learn the decision boundary of the teacher model. It is denoted as [missing information]. . Non-target class distillation loss (NCKD) measures the difference in probability distribution between the teacher and student models, but this measurement is weighted, with weights of 1 / 2. This is the complement of the teacher model's predicted probability for the target category. It also implies that the more uncertain the teacher model's prediction of the target category (i.e., the more uncertain the prediction), the more likely the teacher model is to predict the target category correctly. The smaller the value, the more importance is placed on the student model's learning in that category, denoted as . Simplify the above formula further:

[0076] (8)

[0077] From the above formula, we can see that TCKD and There is a coupling relationship, when When this change occurs, it not only affects the weight of the second term but may also indirectly affect the learning process of the student model. This is because the student model needs to pay more attention to learning non-target categories when the teacher model's prediction uncertainty is high. Therefore, further optimization of the equation yields the decoupled knowledge distillation loss. ,in and The formula used to balance target and non-target classes is as follows:

[0078] (9)

[0079] However, when the teacher model becomes more complex and the data heterogeneity is greater, relying solely on the output layer knowledge may prevent the student model from fully capturing the knowledge of the teacher model. Therefore, this application also introduces intermediate layer feature loss. By calculating the distance between the intermediate feature layers of the two models as a partial loss, the student model can capture the high-level feature representation of the teacher model. The specific calculation formula is as follows:

[0080] (10)

[0081] in, This represents the feature representation of the intermediate feature layer in the teacher model. The feature representation of the intermediate feature layer corresponding to the student model;

[0082] Cross-entropy loss also plays a crucial role in model training, measuring the difference between the probability distribution and the true probability distribution, with the aim of minimizing the model's predictions. With real labels y The difference between them enhances the model's ability to identify data features and improves the overall performance of the model. The formula is as follows:

[0083] (11)

[0084] in, The local model's predicted values ​​for the sample data. The true labels for the sample data.

[0085] A dual-teacher collaborative knowledge transfer mechanism, employing both a global model and a local cluster model, is adopted, with decoupled distillation performed at the intermediate feature layer and the Softmax output layer, respectively. The global model, acting as a unified teacher model, helps improve the overall generalization ability of the student model; while the local cluster model better preserves the feature representation capabilities of terminals within the same cluster, enhancing the model's adaptability to local data distributions. Based on this, two distillation losses are designed: a global model distillation loss... Distillation loss with intra-cluster teacher model The calculation formula is as follows:

[0086] (12)

[0087] in, This represents the total distillation loss for both the local and global models. This represents the total distillation loss for both the local model and the local cluster model. Distillation loss in the intermediate feature layer for the student model and the global model as the teacher model; Distillation loss in the intermediate feature layer for the student model and the local cluster model as the teacher model; Distillation loss at the classification layer for the student model and the global model as the teacher model; Distillation loss at the classification layer for the student model and the local cluster model as the teacher model; and These are hyperparameters used to balance the distillation losses of the local and global models in the intermediate feature layers and classification layers; and This is a hyperparameter used to balance the distillation loss of the local model and the local cluster model in the intermediate feature layer and the classification layer.

[0088] In summary, the loss function of the entire model in this application can be expressed in three parts: cross-entropy loss, global model distillation loss, and so on. Distillation loss of intra-cluster teacher model It can be expressed as the following formula, using hyperparameters. and To balance the overall loss:

[0089] (13)

[0090] in, The cross-entropy loss is for the local model; This represents the total distillation loss for both the local and global models. This represents the total distillation loss for both the local model and the local cluster model. and These are hyperparameters used to balance the overall loss.

[0091] Furthermore, the detailed steps of the cloud-edge federated learning method proposed in this application are shown in Algorithm 1:

[0092] The input data for Algorithm 1 includes: dataset { }, Number of terminals N, Local sample batch size B, Learning rate heterogeneity Local training epoch (local_epoch), communication epoch (E), global model ;

[0093] The output data of Algorithm 1 includes: the terminal clustering result formed by the last clustering. The final global model obtained from the last update ;

[0094] In Algorithm 1, the cloud server performs the following operations:

[0095] Initialize global model ;

[0096] All terminals upload model parameters and feature vectors;

[0097] Dynamic clustering is performed based on the judgment criteria to form... K Cluster ;

[0098] For communication rounds e = 1, 2, ... E, perform the following operations:

[0099] Deploy global model To the edge server;

[0100] Collect all cluster models;

[0101] Global model update ;

[0102] End the loop;

[0103] In Algorithm 1, the edge server performs the following operations:

[0104] 1. For clusters belonging to this edge server K= Perform the following operations:

[0105] 2. Initialize the model ;

[0106] 3. Collect local model parameters uploaded by the terminal;

[0107] 4. Weighted aggregation terminal model as a local cluster model ;

[0108] 5. Upload To the cloud server, and retain it locally as a cluster model;

[0109] 6. End the loop.

[0110] In Algorithm 1, the terminal performs the following operations:

[0111] 7. Initialize the local model ;

[0112] 8. For communication rounds e=1,2,...,E, perform the following operations:

[0113] 9. For local training epochs t = 1, 2, ..., local_epoch, perform the following operations:

[0114] 10. Perform forward propagation and calculate cross-entropy loss. ;

[0115] 11. Calculation of distillation losses in cluster models ;

[0116] 12. Calculation and Global Model Distillation Loss ;

[0117] 13. ;

[0118] 14. End the loop;

[0119] 15. Extract terminal feature vectors ;

[0120] 16. Upload To the cloud server;

[0121] 17. Upload local model parameters To the edge server;

[0122] 18. End the loop.

[0123] To verify the effectiveness of this method, this application uses the Fashion-MNIST and CIFAR-10 datasets as experimental datasets. Fashion-MNIST focuses on the classification of clothing images and contains 70,000 28x28 grayscale images in 10 categories, with 7,000 images in each category. These categories include T-shirts, pants, shoes, dresses, coats, shirts, hats, scarves, skirts, and bags. The CIFAR-10 dataset contains 60,000 32×32 RGB images of 10 categories of colored objects (such as airplanes, cars, birds, etc.), of which 50,000 are training samples and 10,000 are test samples. Detailed datasets are shown in Table 1 below.

[0124] Table 1 Dataset Information Table

[0125]

[0126] This application simulates a cloud-edge federated framework, comprising one cloud server, five edge servers, and 30 terminals. Each terminal uses the SGD optimizer with a learning rate of 0.01, 10 training epochs, and 100 communication epochs. Different models are used for different datasets. For the Fashion-MNIST dataset, three 3x3 convolutional layers, three 2x2 pooling layers, one Dropout layer, and two fully connected layers are used. For the CIFAR-10 dataset, three 5x5 convolutional layers, three 2x2 pooling layers, and two fully connected layers are used. All experiments are performed on high-performance computing nodes equipped with 4×NVIDIA A30 GPUs (24GB VRAM / card). The node configuration includes an Intel Xeon Gold 6342 processor (2.80GHz), 251GB of memory, and Ubuntu 22.04LTS with NVIDIA CUDA 11.7 operating system. The experimental software used is Python 3.12.7 and Torch 2.4.0.

[0127] To simulate the control of data heterogeneity in a cloud-edge federated environment, a Dirichlet distribution is used to partition the dataset, and its probability density function is shown below:

[0128] (14)

[0129] Where parameters Used to quantify the heterogeneity level of the terminal dataset. When When the value is small, the data distribution will exhibit significant non-uniformity, meaning that the category preferences of different terminals differ significantly; conversely, when the value is large... As the value increases, the data distribution tends to be more uniform, and the distribution of various types of samples across different terminals becomes more balanced. Figure 5 Displayed on the CIFAR-10 dataset The experimental results of the division, among which A value of 0.5 indicates that the data partition is approximately independent and identically distributed. =0.3 or =0.1 significantly enhanced heterogeneity, while A value of 0.01 represents an extremely heterogeneous scenario, where some users completely lack data for certain categories, resulting in local data being highly biased towards a few categories, while data for other users is sparsely distributed. In a scatter plot, the size of the points reflects the number of samples each terminal has in a specific category.

[0130] The primary evaluation metric in this experiment is classification accuracy, which measures the model's overall classification performance on the test set. The value range is [0, 100%], with higher values ​​indicating better model performance. Specifically, it is defined as follows:

[0131] (15)

[0132] To verify the effectiveness of the proposed Cloud edge collaborative federated learning method based on Decoupling Distillation (CEFLDD) in statistically heterogeneous scenarios, data distributions were divided according to four different levels of heterogeneity on two benchmark datasets: Fashion-MNIST and CIFAR-10. =0.01, 0.1, 0.3, 0.5, constructing continuous test scenarios ranging from extremely heterogeneous to relatively homogeneous. Accuracy is compared with FedAvg, FedDistill, pFedSD, FedNTD, and FedGKD methods.

[0133] FedAvg is a classic federated learning method where the terminal trains the model locally, then uploads the model parameters to the server for simple averaging to update the global model. However, it is prone to performance degradation or slow convergence in scenarios with non-independent and identically distributed data.

[0134] FedDistill proposes a groundbreaking distillation-based baseline approach, primarily consisting of Federated Distillation (FD) and Federated Augmentation (FAug). FD reduces communication overhead by exchanging model outputs instead of parameters, while FAug leverages generative adversarial networks to locally generate missing data to correct non-independent and identically distributed data, thereby improving model performance.

[0135] pFedSD: This method alleviates the problem of personalized knowledge forgetting caused by initialization in federated learning through self-knowledge distillation. In each round of communication, this method retains the personalized model from the previous round and uses knowledge distillation to pass historical personalized knowledge to the current local model, thus achieving a better balance between personalization and generalization.

[0136] FedNTD: This method mitigates the performance degradation caused by data heterogeneity by preserving the knowledge of the global model through knowledge distillation only on non-real categories during local training.

[0137] FedGKD: This method reduces terminal drift by integrating historical global models and using knowledge distillation techniques to guide local training. Its core is to enhance the local model's ability to capture global features through multi-round integration and self-distillation of global models, without requiring additional data or model modifications, thus significantly improving convergence and model accuracy in non-IID scenarios.

[0138] CEFLDD: Compared to traditional knowledge distillation methods, the CEFLDD two-level collaborative optimization framework takes the decoupled distillation of the global model and local models as its core innovation, which not only ensures the effectiveness of knowledge transfer but also significantly improves the model's generalization ability. CEFLDD dynamically constructs the optimal clustering strategy through a cloud-based feature-driven adaptive clustering module; at the edge, through the decoupled distillation mechanism of the feature layer and the Softmax layer, terminal devices can learn common features from the global teacher model and obtain personalized decision-making capabilities from the local teacher model.

[0139] To evaluate the performance of CEFLDD on the Dirichlet distribution, experiments were conducted with different degrees of heterogeneity on the two datasets, namely... =0.01, 0.1, 0.3, and 0.5, the specific experimental results on the Fashion-MNIST dataset are shown in Table 2 and... Figure 6 The CEFLDD method demonstrates superior performance across different levels of data heterogeneity. The highest accuracy was achieved at all values, especially in highly heterogeneous environments. In scenarios with a coefficient of 0.01, FedGKD shows a significant improvement over other methods and also leads in other heterogeneous scenarios, demonstrating good generalization ability and adaptability to non-IID data. In highly heterogeneous data distribution scenarios, FedGKD's performance is somewhat affected, mainly because it relies on global gradient information. When the terminal data distribution is highly skewed, gradient conflicts may occur, affecting the model's convergence stability. pFedSD also has limitations in such scenarios because when the terminal encounters only a small number of class samples in multiple rounds, the retained historical personalized model is prone to local overfitting, and the knowledge transferred through self-distillation may be highly biased. Compared to the previous two methods, FedNTD performs more stably in scenarios with low heterogeneity. Its advantage lies in simultaneously considering non-true class information and retaining the global model. Compared with the above methods, the method proposed in this application is more robust in highly heterogeneous environments: by comprehensively introducing the intermediate feature layer and Softmax layer information of the global model and the intra-cluster model, effective collaborative distillation of multi-level knowledge is achieved, thereby enhancing the adaptability and expressive power of the sub-model under non-independent and identically distributed data.

[0140] Table 2. Results of the Fashion-MNIST experiment comparing different heterogeneities.

[0141]

[0142] On the CIFAR-10 dataset, the CEFLDD method outperforms other comparative algorithms at different levels of heterogeneity. Specific experimental results are shown in Table 3 and... Figure 7 This is mainly due to CEFLDD's joint consideration of feature differences between terminals and multi-level knowledge transfer mechanisms, effectively improving the model's generalization ability in non-independent and identically distributed scenarios. In contrast, the traditional FedAvg algorithm uses a simple weighted average strategy, which fails to fully characterize the differences in data distribution between terminals, thus significantly reducing its performance in highly heterogeneous environments. Although FedDistill introduces a knowledge distillation mechanism, its distillation process is limited to between terminals and lacks effective integration of global knowledge on the server side, limiting its performance in complex tasks. pFedSD emphasizes personalized modeling, which can improve local adaptability, but its global collaborative ability is weak, easily leading to model dispersion in highly imbalanced scenarios. FedNTD and FedGKD enhance the model's generalization ability with complex distillation strategies, alleviating the challenges brought by data heterogeneity to some extent, but still struggle to effectively capture key global features when the data distribution is extremely skewed. However, CEFLDD adapts to the structural and semantic differences between terminals through a dynamic clustering strategy and combines intra-cluster and global dual distillation mechanisms to achieve multi-level knowledge fusion, thus exhibiting superior performance and robustness under various task types and heterogeneous data conditions.

[0143] Table 3. Results of CIFAR-10 experiments comparing different heterogeneities.

[0144]

[0145] Distillation temperature is a crucial hyperparameter for controlling the smoothness of soft targets in knowledge distillation. An appropriate distillation temperature can improve knowledge transfer efficiency and enhance the generalization ability of student models. If the temperature is too low, it cannot effectively transfer the potential knowledge of the teacher model, while if the temperature is too high, it may introduce noise. Therefore, this application also conducts experiments to adjust the key parameter T in the knowledge distillation process, using values ​​in the range {1, 3, 5}, with the aim of finding the optimal distillation temperature. Figure 8 The results show that there is an optimal temperature value for each dataset. The optimal temperature is T=1 for the Fashion-MNIST dataset and T=3 for the CIFAR-10 dataset.

[0146] In summary, the CEFLDD method proposed in this application demonstrates superior performance in federated learning scenarios with varying degrees of statistical heterogeneity, fully validating the effectiveness of the decoupled distillation training strategy in improving model generalization ability and stability. Especially under conditions of extremely non-independent data distribution (pFed = 0.01), CEFLDD exhibits stronger adaptability and robustness compared to existing methods (such as FedAvg, FedDistill, pFedSD, FedNTD, and FedGKD), significantly mitigating model degradation caused by heterogeneity. As the degree of heterogeneity increases, the adaptive clustering and decoupled distillation mechanisms employed by CEFLDD effectively improve intra-group model consistency, enhance feature extraction and classification capabilities, and effectively promote the model's generalization ability and adaptability.

[0147] In addition, refer to Figure 1 As shown, according to a second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0148] Therefore, to address the problems of model performance degradation and weak generalization ability caused by data heterogeneity in cloud-edge federated learning, this application proposes a cloud-edge collaborative federated learning method based on decoupled distillation. This method uses a feature-driven adaptive clustering module to dynamically perform clustering based on data distribution and model features in the cloud, achieving intra-cluster collaborative optimization at the edge. The collaborative optimization module based on decoupled distillation decouples knowledge distillation into dual transfer between the feature layer and the Softmax layer, realizing hierarchical transfer of intermediate features and output layer knowledge, thus improving the model's representation ability and decision performance. Experimental results show that on the Fashion-MNIST and CIFAR-10 datasets divided by Dirichlet, CEFLDD achieves accuracies of 80.30% and 50.94% respectively in the extreme non-independent identically distributed scenario (α=0.01), outperforming comparative methods such as FedAvg, FedDistill, pFedSD, FedNTD, and FedGKD, verifying the effectiveness and robustness of this method in cross-heterogeneous scenarios.

[0149] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, as some steps can be performed in other orders or simultaneously according to the present invention. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention. The above descriptions are merely preferred embodiments of the present invention. It should be pointed out that those skilled in the art can make several improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A cloud-edge federated learning method, characterized in that, include: Step S1: The cloud server initializes the global model and receives model feature vectors uploaded by all terminals; Step S2: At the beginning of each communication round, the cloud server determines whether the clustering trigger condition is met: if it is met, it clusters all terminals into K clusters based on the latest received model feature vectors and sends the terminal list and global model of the managed clusters to the edge server; if it is not met, it directly sends the global model to the edge server. Step S3: The edge server initializes the local cluster model of its managed cluster based on the global model, and distributes the local cluster model and the global model to the terminals within the cluster; Step S4: The terminal uses a dual-teacher decoupled distillation mechanism to iteratively train the local model. In each iteration: (a) calculate the cross-entropy loss of the local model; (b) use the global model and the local cluster model as teacher models and the local model as student models, and calculate the distillation loss of the student model and each teacher model in the intermediate feature layer and classification layer respectively; (c) weight the losses calculated in steps (a) and (b) to obtain the total loss; (d) update the parameters of the local model based on the total loss. Step S5: After the terminal training is completed, extract the model feature vector of the local model and upload it to the cloud server, and upload the updated local model parameters to the corresponding edge server. Step S6: The edge server weights and aggregates the local model parameters uploaded by the terminals within the cluster, generates a new local cluster model, and uploads it to the cloud server; Step S7: The cloud server aggregates all local cluster models and updates the global model; Step S8: If the current communication round has not reached the preset total round, repeat steps S2 to S7; otherwise, output the terminal clustering results and the final global model.

2. The method according to claim 1, characterized in that, The cloud server determines whether the clustering trigger condition is met by following these steps: The cloud server calculates the average variance of the local model parameters or model feature vectors of each terminal within the cluster; When the average variance exceeds a set threshold, the cloud server determines that the clustering trigger condition is met; otherwise, it does not.

3. The method according to claim 1, characterized in that, The cloud server determines whether the clustering trigger condition is met by following these steps: The global profile coefficient in the t-th communication round is calculated using the following formula. : ； in, Indicates terminal i The average distance to other terminals within its cluster; Indicates terminal i The average distance to the nearest neighbor cluster; N Indicates terminal i The number of all terminals within the cluster; If the decrease in the global contour coefficient after a preset number of consecutive communication rounds exceeds a preset proportion, the cloud server determines that the clustering trigger condition is met; otherwise, it is not.

4. The method according to claim 1, characterized in that, The cross-entropy loss of the local model is calculated using the following formula. : ； in, The local model's predicted values ​​for the sample data. The true labels for the sample data.

5. The method according to claim 1, characterized in that, The distillation loss of the student model and each teacher model at the intermediate feature layer is calculated using the following formula. : ； in, This represents the feature representation of the intermediate feature layer in the teacher model. This represents the feature representation of the intermediate feature layer corresponding to the student model.

6. The method according to claim 1, characterized in that, The distillation loss of the student model and each teacher model at the classification layer is calculated using the following formula. : ； ； ； ； in, The target class distillation loss is used for the student model and the teacher model. For the non-target class distillation loss of the student model and the teacher model; and The weights are used to balance the distillation losses of the target class and the non-target class; This indicates that the teacher model is related to the target category. t The predicted probability; Indicates an index that is not the target category; Represents the target category in the student model t The probability value, This represents the total probability value of non-target categories in the teacher model. This represents the total probability of the student model for non-target categories; For a single non-target category in the teacher model i The classification probability, This indicates that the student model applies to a single non-target category. i The probability of; This indicates that the teacher model is related to the target category. t The score output; The classification layer of the teacher model represents the non-target categories. j The score output; The classification layer of the teacher model represents the non-target categories. i The score output; The classification layer of the teacher model represents the non-target categories. k The score output; This indicates that the student model is related to the target category. t The score output; The classification layer of the student model represents the non-target category. j The score output; The classification layer of the student model represents the non-target category. i The score output; The classification layer of the student model represents the non-target category. k Score output; subscript i , j and k Used to distinguish variables; Indicates the total number of sample categories; It is the distillation temperature coefficient.

7. The method according to claim 1, characterized in that, The total loss is obtained by weighting and combining the losses calculated in steps (a) and (b) using the following formula. : ； ； in, The cross-entropy loss is for the local model; This represents the total distillation loss for both the local and global models. This represents the total distillation loss for both the local model and the local cluster model. and These are hyperparameters used to balance the overall loss; Distillation loss in the intermediate feature layer for the student model and the global model as the teacher model; Distillation loss in the intermediate feature layer for the student model and the local cluster model as the teacher model; Distillation loss at the classification layer for the student model and the global model as the teacher model; Distillation loss at the classification layer for the student model and the local cluster model as the teacher model; and These are hyperparameters used to balance the distillation losses of the local and global models in the intermediate feature layers and classification layers; and This is a hyperparameter used to balance the distillation loss of the local model and the local cluster model in the intermediate feature layer and the classification layer.

8. The method according to claim 1, characterized in that, The edge server generates a new local cluster model by weighting and aggregating the local model parameters uploaded by terminals within the cluster using the following formula: ； in, Indicates the first k The local cluster model after each cluster update; Indicates the first k The first cluster within the cluster i Local data volume per terminal; Cluster k The total amount of data from all terminals within the system; Cluster k The set of terminals; Indicates the first k The first cluster within the cluster i Local model parameters for each terminal.

9. The method according to claim 1, characterized in that, The cloud server aggregates all local cluster models and updates the global model using the following formula: ； in, This represents the updated global model; K This represents the number of clusters obtained from the latest clustering of all terminals; Indicates the first k The latest local cluster model uploaded to the cloud server.

10. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, the method described in any one of claims 1 to 9 is performed by a processor.

Images