Adaptive bidirectional distillation training method and system for heterogeneous federated learning, and medium

Abstract

This application relates to an adaptive bidirectional distillation training method, system, and medium for heterogeneous federated learning. The method is applied to a distributed system composed of a server and heterogeneous clients with multiple computing powers. The server sets up a global proxy model, and the clients configure heterogeneous local private models and local copies of the proxy model. The server initializes and distributes parameters. After the clients update their copies, they predict and calculate confidence scores on local non-IID private training data, generating adaptive dynamic distillation weights positively correlated with the classification difficulty of the samples. Bidirectional knowledge distillation is then performed between the two models, and parameters are updated. The clients upload their copy parameters, and the server aggregates and updates them using a federated averaging algorithm, iterating until the models converge. This method does not rely on auxiliary datasets, improves the efficiency of knowledge transfer between models, alleviates the bias in heterogeneous training of data and models, reduces communication overhead, and significantly improves the aggregation performance and generalization ability of the federated model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an adaptive bidirectional distillation training method, system, and medium for heterogeneous federated learning. Background Technology

[0002] Federated learning is a distributed machine learning paradigm that enables "data to remain still while the model moves." It allows multiple data holders to collaboratively train and share models while protecting their private data, effectively solving the "data silos" and privacy leakage problems of traditional centralized machine learning. It has become a core technology for cross-device and cross-organizational collaborative modeling.

[0003] Heterogeneous federated learning, as an important branch of federated learning, breaks through the limitations of homogeneous models in traditional federated learning. It allows clients with different computing power to train personalized heterogeneous models based on their own hardware resources and local data distribution, greatly improving the adaptability of federated learning in real-world scenarios and becoming a current research hotspot. However, existing heterogeneous federated learning methods still have many technical shortcomings, making it difficult to balance knowledge transfer efficiency, model aggregation effect, and scenario adaptability: some methods rely on high-quality public datasets, but in practice, it is difficult to obtain public datasets that are suitable for all clients, which can easily lead to insufficient model generalization ability; methods using prototype-like aggregation have limited knowledge representation capabilities and cannot fully capture complex data distributions, resulting in insufficient knowledge transfer between clients; methods based on surrogate models often use unidirectional knowledge distillation and fixed distillation weights, which have asymmetric problems of low knowledge transfer efficiency and cannot adapt to the difficulty of sample classification. Weak samples are prone to introducing noise that interferes with model training; at the same time, existing methods do not effectively screen distillation samples, and redundant training increases the computing power consumption of clients and reduces the calibration effect on model bias caused by data heterogeneity.

[0004] Therefore, there is an urgent need to propose a new heterogeneous federated learning method that can achieve efficient knowledge transfer between proxy models and local private models without relying on public datasets, adapt to the difficulty of sample classification and the heterogeneous characteristics of data, improve the aggregation performance, robustness and generalization ability of heterogeneous federated learning models, and meet the collaborative training needs of real-world heterogeneous scenarios. Summary of the Invention

[0005] Therefore, it is necessary to provide an adaptive bidirectional distillation training method, system, and medium for heterogeneous federated learning that can improve the efficiency of knowledge transfer between models in heterogeneous federated learning and effectively alleviate the training bias caused by data and model heterogeneity, in order to address the above-mentioned technical problems.

[0006] An adaptive bidirectional distillation training method for heterogeneous federated learning is disclosed. The method is applied to a distributed computer system consisting of a server and multiple clients. The server is equipped with a global proxy model, and each client is a computing terminal with heterogeneous computing power, and is equipped with a heterogeneous local private model and a local copy of the global proxy model. The method includes:

[0007] The server initializes the global proxy model and sends the initial model parameters to each of the clients participating in the training. Each client updates its local copy of the global proxy model according to the initial model parameters. In each of the aforementioned clients, a local copy of the global proxy model is used to make predictions based on locally stored non-IID distribution private training data, and the sample prediction confidence is calculated. Based on the sample prediction confidence, an adaptive dynamic distillation weight that is positively correlated with the sample classification difficulty is calculated. Based on the adaptive dynamic distillation weight, a bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model, and the parameters of the local copy of the global proxy model and the local private model are updated respectively. Each client uploads the updated model parameters of the local copy of the global proxy model to the server; The server uses a federated averaging algorithm to aggregate the model parameters uploaded by all clients, updates the global proxy model, and then distributes the updated global proxy model parameters to each client participating in the training. The process of local client updates and server aggregation is repeated until the global proxy model of the server converges, thus completing the training of the heterogeneous federated learning model.

[0008] In one embodiment, calculating the sample prediction confidence includes: The logit value of the sample is output through a local copy of the global proxy model; The logit value is subjected to softmax normalization to obtain the class prediction probability of the sample; The maximum value among the predicted probabilities of the categories is selected as the prediction confidence of the sample.

[0009] In one embodiment, the step of calculating the adaptive dynamic distillation weights that are positively correlated with the sample classification difficulty based on the sample prediction confidence includes: Low-confidence samples with prediction confidence levels less than or equal to a confidence threshold are filtered out using an indicator function; The adaptive dynamic distillation weight is calculated based on the predicted confidence of the low-confidence samples, and the lower the predicted confidence of the low-confidence samples, the larger the corresponding adaptive dynamic distillation weight value.

[0010] In one embodiment, the formula for calculating the adaptive dynamic distillation weight is: ; In the above formula, This indicates adaptive dynamic distillation weights. This represents the total number of samples in the private training data. Indicates an indicator function, Indicates the first The prediction confidence of each sample This indicates the preset confidence threshold.

[0011] In one embodiment, the bidirectional knowledge distillation of the logit layer includes: The logit layer output knowledge of the local copy of the global proxy model is passed to the local private model, and the logit layer output knowledge of the local private model is fed back to the local copy of the global proxy model, realizing the bidirectional interactive transmission of global common knowledge and local personalized knowledge.

[0012] In one embodiment, the bidirectional knowledge distillation measures the difference in output between the local copy of the global proxy model and the local private model at the logit layer using KL divergence. The loss function of the bidirectional knowledge distillation is constructed by combining cross-entropy loss and KL divergence distillation loss, and the total loss function of the local private model and the local copy of the global proxy model are both multiplied by the adaptive dynamic distillation weights.

[0013] In one embodiment, the global agent model is a lightweight deep learning model, including at least one of a 4-layer convolutional neural network and MobileNetV2; The local private model is a heterogeneous deep learning model adapted to the client's computing power, including at least one of the ResNet series models, ViT series models, and basic convolutional neural networks.

[0014] This application also provides an adaptive bidirectional distillation training system for heterogeneous federated learning. The system includes servers that communicate with each other and multiple clients with heterogeneous computing power. The servers are configured with a global proxy model, and each client is configured with a heterogeneous local private model and a local copy of the global proxy model. The server is configured as follows: Initialize the global proxy model and send the initial model parameters of the global proxy model to each of the clients participating in the training; The system receives updated model parameters from local copies of the global proxy model uploaded by each client, and uses a federated average algorithm to weight and aggregate the model parameters to update the global proxy model. The updated global proxy model parameters are sent to each of the clients participating in the training. The process of local client update and server aggregation is repeated until the global proxy model converges, thus completing the training of the heterogeneous federated learning model. Each of the aforementioned clients is configured as follows: Update the local copy of the global proxy model based on the initial model parameters issued by the server; The local copy of the global proxy model is used to predict non-IID distributed private training data stored locally, and the prediction confidence of the samples is calculated. Based on the sample prediction confidence, calculate the adaptive dynamic distillation weights that are positively correlated with the sample classification difficulty; Based on the adaptive dynamic distillation weights, bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model to update the parameters of the local copy of the global proxy model and the local private model respectively. Upload the model parameters of the updated local copy of the global proxy model to the server.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described adaptive bidirectional distillation training method for heterogeneous federated learning.

[0016] The aforementioned adaptive bidirectional distillation training method, system, and medium for heterogeneous federated learning, in a distributed computer system consisting of a server and multiple clients, involves a server equipped with a global proxy model. Each client is a computing terminal with heterogeneous computing power and is equipped with a heterogeneous local private model and a local copy of the global proxy model. The server initializes the global proxy model and distributes the initial model parameters to each client participating in the training. Each client updates its local copy of the global proxy model based on the initial model parameters. Within each client, the local copy of the global proxy model is used to perform predictions based on locally stored non-IID distributed private training data, and the sample prediction confidence is calculated. The calculation of the sample prediction confidence is correlated with the sample classification difficulty. The relevant adaptive dynamic distillation weights are used, and bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model based on the adaptive dynamic distillation weights. The parameters of the local copy of the global proxy model and the local private model are updated respectively. Each client uploads the updated model parameters of the local copy of the global proxy model to the server. The server uses a federated averaging algorithm to aggregate the model parameters uploaded by all clients, updates the global proxy model, and then distributes the updated model parameters of the global proxy model to each client participating in the training. The process of client local update and server aggregation is repeated until the global proxy model of the server converges, and the training of the heterogeneous federated learning model is completed.

[0017] Beneficial effects: This method constructs a bidirectional distillation mechanism within the client, building a local copy of the global proxy model and a heterogeneous local private model. It dynamically calculates adaptive distillation weights based on the sample prediction confidence, accurately focusing on low-confidence, difficult samples for reinforcement training. This effectively improves the knowledge transfer efficiency between models in heterogeneous federated learning, alleviates the training bias caused by data heterogeneity and model heterogeneity, and achieves stable convergence without relying on public datasets. While reducing communication overhead and client computing power consumption, it significantly improves the aggregation performance and generalization ability of the federated model. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an adaptive bidirectional distillation training method for heterogeneous federated learning in one embodiment. Figure 2 This is a schematic diagram of the overall process of the adaptive bidirectional distillation training method for heterogeneous federated learning in one embodiment. Figure 3 This is a diagram illustrating the performance comparison of heterogeneous federated methods on the AUCDDD dataset in an experiment. Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] To address the problems in existing technologies, such as low knowledge transfer efficiency in heterogeneous federated learning, training bias caused by heterogeneous data and models, poor scenario adaptability due to reliance on auxiliary datasets, and high communication overhead, this application addresses these issues. Figure 1 As shown, an adaptive bidirectional distillation training method for heterogeneous federated learning is provided. This method is applied to a distributed computer system consisting of a server and multiple clients. The server is equipped with a global proxy model, and each client is a computing terminal with heterogeneous computing power, and is equipped with a heterogeneous local private model and a local copy of the global proxy model. The method specifically includes the following steps: In step S100, the server initializes the global proxy model and sends the initial model parameters to each client participating in the training. Each client updates its local copy of the global proxy model according to the initial model parameters.

[0021] In step S110, in each client, the local copy of the global proxy model is used to make predictions based on the locally stored non-IID distribution private training data, and the sample prediction confidence is calculated. Based on the sample prediction confidence, adaptive dynamic distillation weights that are positively correlated with the sample classification difficulty are calculated. Based on the adaptive dynamic distillation weights, bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model, and the parameters of the local copy of the global proxy model and the local private model are updated respectively.

[0022] In step S120, each client uploads the updated model parameters of the local copy of the global proxy model to the server.

[0023] Step S130: The server uses a federated averaging algorithm to aggregate the model parameters uploaded by all clients, updates the global proxy model, and then distributes the updated global proxy model parameters to each client participating in the training. The process of local client updates and server aggregation is repeated until the server's global proxy model converges, thus completing the training of the heterogeneous federated learning model.

[0024] To more clearly explain the technical principles and implementation details of this method, we will first explain the heterogeneous federated learning scenarios to which this method is applicable and the definition of the core problems.

[0025] Specifically, in the context of heterogeneous federated learning models, the system architecture and problem definition of this method are as follows: assuming there are... There are 1 client, and each client stores its private training dataset locally. ,remember For the size of the client's dataset, This represents the total size of the datasets across all clients. To protect data privacy, raw data is never transmitted across devices. Additionally, each client needs to maintain two types of models: a heterogeneous local private personalized model (i.e., a local private model with a set of parameters). ) and the local copy of the global proxy model (parameter set) The server is only responsible for aggregating and distributing global proxy model parameters and does not participate in any local data interaction. The entire federated system achieves collaborative learning by iteratively optimizing the global objective function.

[0026] Furthermore, the global objective function is expressed as: ; In the above formula, Weighting coefficients for data size. For the client The local training objective.

[0027] In step S100, the distributed computer system architecture includes two core modules: local update (step S110) and server update (step S130). Local update includes bidirectional knowledge distillation, confidence quantification filtering, and adaptive dynamic distillation. Each client locally maintains two types of models: a local private personalized model (local private model) and a global proxy model (a local copy of the global proxy model). Assume the entire training process includes... Round iteration, in each round During updates, the client will base its updates on its private dataset. Train two models. Note that the local private model structure varies depending on the client, i.e. Furthermore, the client-side private model is always maintained locally. In contrast, the global proxy model adopts a unified structure, and its number of parameters is usually less than that of the local private model, in order to reduce the communication overhead of parameter transmission.

[0028] In step S110, the client primarily completes local model adaptive training and bidirectional knowledge interaction. In this step, the client does not rely on any external datasets or server intervention, but only uses local private data to complete confidence assessment, dynamic weight calculation, and bidirectional distillation updates. Key samples that are difficult to classify are identified through sample confidence, and distillation weights matching the sample difficulty are automatically assigned. This enables the global proxy model and the local private model to achieve knowledge exchange and mutual optimization at the logit layer. This allows the local model to absorb global common features to improve generalization, while the proxy model captures local personalized features to improve adaptability. This fundamentally improves the efficiency of knowledge transfer between heterogeneous models, alleviates training bias caused by data and model heterogeneity, and ensures that local data does not leave the domain, meeting the privacy and security requirements of federated learning.

[0029] In this embodiment, a local copy of the global proxy model is used to perform forward inference on the local non-IID distributed private training data, and the logit value of the sample is output. For logit value The softmax normalization process is performed to obtain the class prediction probability of the sample. The maximum probability is taken as the prediction confidence of the sample. The specific process is as follows: ; In the above formula, Indicates the prediction confidence level. This represents the predicted class probability of the sample.

[0030] In this embodiment, the adaptive dynamic distillation weight, which is positively correlated with the sample classification difficulty, is calculated based on the sample prediction confidence. This includes: filtering out low-confidence samples with prediction confidence less than or equal to a confidence threshold using an indicator function; calculating the adaptive dynamic distillation weight based on the prediction confidence of the low-confidence samples; and the lower the prediction confidence of the low-confidence samples, the larger the corresponding adaptive dynamic distillation weight value.

[0031] Specifically, the confidence threshold is set based on the percentile of the confidence value. Samples with confidence scores below a threshold are selected as key enhancement samples for the surrogate model. The filter mask is defined as follows: ; In the above formula, This represents an indicator function that takes a value of 1 when a certain condition is met, indicating that the sample requires additional reinforcement learning from the surrogate model. In batches... In the process, mixed samples that require reinforcement learning are selected. The surrogate model uses these filtered samples to perform distillation learning with the private model, thereby obtaining the reinforcement loss after confidence screening.

[0032] Furthermore, after obtaining the prediction confidence, on the one hand, it is considered that the prediction confidence of the surrogate model is negatively correlated with the sample classification difficulty: samples with lower confidence are often "difficult samples" that are easily confused by the model and have blurred classification boundaries. These samples require more reinforcement learning through distillation. On the other hand, the distillation intensity should be positively correlated with the sample classification difficulty. Difficult samples need to be assigned higher distillation weights to strengthen the knowledge transfer between the private model and the surrogate model, while easy samples are assigned lower weights to avoid model overfitting or training redundancy caused by excessive distillation. To solve the problem that fixed-weight distillation cannot adapt to the differences in sample classification difficulty, an adaptive weight allocation strategy based on the prediction confidence of the global surrogate model is adopted. The adaptive dynamic weight is the key parameter in the proposed framework for regulating the amount of knowledge transferred from the global surrogate model to the client's local private personalized model. By quantifying the classification uncertainty of the samples, the intensity of knowledge distillation is dynamically adjusted to ensure that only high-quality knowledge samples are transferred, avoiding noise interference from the weak global surrogate model to interfere with the training of the private model. The formula for calculating the adaptive dynamic distillation weight is as follows: ; In the above formula, This indicates adaptive dynamic distillation weights. This represents the total number of samples in the private training data. Indicates an indicator function, Indicates the first The prediction confidence of each sample This indicates the preset confidence threshold.

[0033] In this embodiment, based on the aforementioned adaptive dynamic distillation weighting technique, the distillation loss of the local copy of the global proxy model is expressed as follows: ; The distillation loss of the local private model is expressed as: ; In the above formula, This is the KL divergence term corresponding to two-way knowledge distillation.

[0034] In this embodiment, bidirectional knowledge distillation is introduced as the core scheme for local client updates. This allows the client to train a local, private, personalized model tailored to its own data distribution characteristics, thereby replacing the unidirectional parameter update paradigm in traditional federated learning. The server maintains only a global proxy model, while the client contains two models: a global proxy model, which serves as the core medium for knowledge transfer between the local client and the central server; and a local, private, personalized model, designed based on the local client's own data distribution characteristics.

[0035] Specifically, during the bidirectional knowledge distillation of the logit layer: the knowledge output from the logit layer of the local copy of the global proxy model is passed to the local private model, while the knowledge output from the logit layer of the local private model is fed back to the local copy of the global proxy model, realizing the bidirectional interactive transfer of global common knowledge and local personalized knowledge. The implementation of this bidirectional knowledge distillation process is ultimately mathematically expressed and the model optimized by constructing a total loss function. Specifically: passing the knowledge from the local copy of the global proxy model to the local private model involves calculating the distillation loss using KL divergence, enabling the local private model to learn the common features of the global proxy model. Feeding the knowledge from the local private model back to the local copy of the global proxy model also involves calculating the distillation loss using reverse KL divergence, allowing the global proxy model to absorb the personalized features of the local private model.

[0036] In this embodiment, bidirectional knowledge distillation measures the output difference between the local copy of the global proxy model and the local private model at the logit layer using KL divergence. The loss function of bidirectional knowledge distillation is constructed by combining cross-entropy loss and KL divergence distillation loss. The total loss function of both the local private model and the local copy of the global proxy model is multiplied by the adaptive dynamic distillation weights.

[0037] Specifically, the total loss function of the local private model is expressed as: ; The loss function for the local replica in the global proxy model is expressed as: ; In the above formula, and The cross-entropy loss is used for the local private personalized model and the global proxy model, respectively, based on local data. and `<parameter>` is a hyperparameter used to balance the weight ratio of cross-entropy loss and knowledge distillation loss. This mechanism not only allows the global proxy model to pass global common knowledge to the local personalized model, but also supports the feedback of the unique knowledge of the local personalized model to the globally adapted model. In such cases, this step degenerates into a typical FedAvg algorithm. Furthermore, this method omits the weighted average term from FedAvg. The calculation is due to the sample size for each client. This information should not be exposed to a central server, as it could become a breakthrough for attackers to steal privacy. Furthermore, differences in sample size could lead to clients with larger sample sizes having an excessively large proportion in model training, raising fairness issues.

[0038] Furthermore, from the server's perspective, the global proxy model uses a federated averaging algorithm to aggregate the model parameters uploaded by each client to complete iterative updates, with the goal of fitting the joint distribution of data from all clients. This forms a global model with strong generalization capabilities. From the client's perspective, the local private personalized model continuously operates on the private dataset. The training is performed on the local private personalized model, and global common knowledge is extracted from the sub-models during each round of communication. Throughout the federated training process, the local private personalized model... It is always stored locally on the client and will not be replaced, used to fit personalized distributions on private data. .

[0039] During training, the entire system is started by an initial global proxy model maintained by a central server. Simultaneously, all clients initialize their local private personalized models. Subsequently, all clients derive global sub-models based on the global model and perform local update operations. It's important to note that if the global model is not a complete model, the derived global model must be concatenated with the adaptation layer to construct the global sub-model (e.g., if the global model only contains convolutional layers of a convolutional neural network (CNN), the global sub-model is composed of these convolutional layers and fully connected layers). Unlike traditional federated learning, each client does not directly train on the copy of the global model. Instead, multiple rounds of bidirectional mutual learning are performed between the global sub-model and the local private personalized model to complete the local update.

[0040] Specifically, the global proxy model is a lightweight deep learning model, including at least one of a 4-layer convolutional neural network and MobileNetV2. The local private model is a heterogeneous deep learning model adapted to the client's computing power, including at least one of ResNet series models, ViT series models, and basic convolutional neural networks.

[0041] Step S130 represents the global aggregation and iterative optimization closed loop of the heterogeneous federated learning training process of this method. This step, led by the server, plays a crucial role in model aggregation, global updates, and iterative scheduling. Its core function is to weighted aggregate the local replica parameters of the global proxy model uploaded by each client, which have already undergone local bidirectional distillation optimization. Through a federated averaging algorithm, the local knowledge scattered across each client is fused into unified global knowledge, forming a new generation of global proxy model with greater generalization capabilities. The updated model parameters are then distributed to each client, providing a better initial model for the next round of local training, thus forming a stable training closed loop of local optimization and global aggregation iteration.

[0042] By continuously repeating this process until the model converges, this approach can effectively utilize the diversity of distributed data to continuously improve the robustness and convergence speed of the global model, while ensuring the efficiency, stability, and large-scale deployment of the federated learning system.

[0043] Specifically, after each round of local training, each client receives updated parameters for its local private personalized model and proxy model. Subsequently, each client updates the model parameters of the local template of the global proxy model. The models are uploaded to the server, where they are integrated into a new global agent model using traditional federated learning aggregation methods. This new global proxy model will serve as the initial global proxy model for the next round of client communication, i.e. .

[0044] Furthermore, the aggregation formula for the global proxy model is as follows: ; In this embodiment, the entire method alternates between client-side updates and server-side aggregation, iterating continuously until the local private personalized model converges. The complete method architecture is as follows: Figure 2 As shown in the figure. Furthermore, this paper also proposes a code that can implement the above method, as shown in Algorithm 1.

[0045] Algorithm 1: Adaptive Bidirectional Knowledge Distillation (FedABKD) Algorithm Given: Each client and its local private model Private datasets Initial unified global proxy model Client engagement rate Learning rate Total number of communication iterations Local training rounds ; Request: The trained client-side local private model and global proxy model ; 1: Server execution: 2: for each round do; 3: Initialize the global proxy model ; 4: Based on participation rate Randomly select a subset of clients ; 5: Change the current global proxy model Distribute to subset All clients in; 6: for Client do in parallel 7: Client update / / Local update on the client; 8:end for 9: Update the global proxy model: ; 10:end for 11: Client Update: 12: Initialize the client Local private personalized model ; 13: Copy the local copy of the global proxy model: ; 14:for epoch do 15: In private data Above, for local copy With private model Perform two-way knowledge distillation training; 16:end for 17: Client The updated global proxy model will be copied locally. Uploaded to the server; 18:return .

[0046] The heterogeneous federated learning adaptive bidirectional distillation training method proposed in this application has strong engineering feasibility and practical application value. It can be widely applied to various distributed intelligent training scenarios with sensitive data privacy and heterogeneous terminal computing power, especially suitable for fields such as intelligent transportation and medical image analysis that require protection of local data privacy and have significant differences in terminal device configurations. Among them, its application in the field of intelligent transportation safety is particularly typical. Addressing the core pain points in the current training of driver distraction behavior recognition models, such as heterogeneous computing power of different vehicle terminals (e.g., private cars, ride-hailing vehicles, freight vehicles), the inability to transmit driver distraction behavior data (including distraction driving videos, physiological feature data, etc.) locally collected by each vehicle as privacy data, and the non-IID characteristics of data distribution, this method can be directly deployed in a distributed system composed of cloud servers and multiple types of vehicle terminals. Meanwhile, to verify the effectiveness and feasibility of the method, the experimental part uses a real driver distraction behavior dataset for comparative experiments. This dataset contains a variety of typical distraction scenarios (such as making and receiving phone calls, looking down at the screen, and driving while fatigued), covering driver behavior data under different lighting and road conditions. It can realistically simulate the local data distribution characteristics of actual vehicle terminals, providing reliable experimental support and data basis for the practical application of the method.

[0047] In this method, the effectiveness of the method is also demonstrated through experiments.

[0048] First, comparative experiments on heterogeneous federated learning of models under a series of heterogeneous data scenarios were conducted, and the analysis of heterogeneous label scenarios is shown in Table 1.

[0049] Table 1. Performance evaluation of heterogeneous federated methods for different models on the SFDDD and 3MDAD datasets in a heterogeneous label scenario.

[0050] Under this setting, the performance of each method was affected to varying degrees, but FedABKD (the method proposed in this paper) consistently maintained the best performance, fully validating its ability to handle imbalanced label distribution. On the SFDDD dataset, FedABKD achieved an accuracy (Acc) of 85.52%, a 2.30 percentage point improvement compared to the classic baseline LG-FedAvg (83.22%), and a 0.50 percentage point improvement compared to the second-best method FedTGP (84.02%). Furthermore, FedABKD achieved the best F1 score, accuracy, and AUC, with its AUC on par with FedTGP, further demonstrating the stability of its classification performance. This advantage stems from FedABKD's adaptive knowledge distillation mechanism, which effectively mitigates local bias caused by label heterogeneity. Through cross-client knowledge transfer, it calibrates the classification decision boundary of the local model, reducing the negative impact of class imbalance on model performance. On the 3MDAD dataset, FedABKD demonstrates a more significant advantage, achieving an accuracy of 88.88%, a 2.07 percentage point improvement over the second-best method, FedMKD (86.81%), and a 2.45 percentage point improvement over LG-FedAvg. It achieves the best F1 score, accuracy, and AUC, with its AUC only slightly lower than FedMKD (97.14%), but its overall performance is superior. The 3MDAD dataset contains diverse data from multiple vehicle models and camera perspectives. Even in scenarios with heterogeneous labels, the inherent heterogeneity of local data is more pronounced. FedABKD, through the synergy of feature distillation and decision distillation, effectively aligns the feature representations and classification outputs of different clients, reducing heterogeneity conflicts between clients and thus exhibiting stronger robustness in highly complex data scenarios.

[0051] Under this setting, the performance of each method was affected to varying degrees, but FedABKD consistently maintained the best performance, fully validating its ability to handle imbalanced label distribution. On the SFDDD dataset, FedABKD's AUC value demonstrated the stability of its classification performance, achieving the best results in all other metrics, including accuracy. This advantage stems from FedABKD's adaptive bidirectional knowledge distillation mechanism. This mechanism facilitates bidirectional knowledge transfer between the private model and the surrogate model: on one hand, the surrogate model transfers global common knowledge to the private model, calibrating its classification decision boundary; on the other hand, the private model feeds back its local-specific knowledge to the surrogate model, enriching the global model's understanding of data distribution. The synergistic effect of these two mechanisms effectively mitigates local bias caused by label heterogeneity. In particular, on the 3MDAD dataset, FedABKD's advantage is even more significant, achieving an accuracy of 88.88% and being the best in all metrics, including F1 score. The 3MDAD dataset contains differences in data collection from multiple vehicle models and camera perspectives. Even in scenarios with heterogeneous labels, the inherent heterogeneity of local data is more prominent. FedABKD, through cross-architecture knowledge transfer, can effectively align feature representations and classification outputs from different clients, reducing the overlapping conflicts between model heterogeneity and data heterogeneity, thus exhibiting stronger robustness in high-complexity data scenarios.

[0052] However, methods that directly employ static prototype transfer or generative knowledge distillation often struggle to adapt to class distribution shifts caused by heterogeneous labels. Methods like FedProto and FedGen perform relatively poorly precisely because they rely on fixed prototype representations or generator-synthesized data, failing to dynamically adjust knowledge transfer strategies based on the actual class distribution of each client. In contrast, FedABKD's adaptive distillation weight mechanism dynamically adjusts the weight ratio of feature distillation and decision distillation based on local data distribution and model confidence. This strengthens global knowledge guidance in clients with missing classes and preserves local characteristics in clients with sufficient classes, thus achieving a better balance between personalization and generalization.

[0053] Table 2. Performance evaluation of heterogeneous federation methods for different models on the SFDDD and 3MDAD datasets in scenarios with heterogeneous data types.

[0054] Furthermore, an analysis of heterogeneous data types was conducted. Table 2 shows the performance of the heterogeneous federated learning method on the SFDDD and 3MDAD datasets under these conditions. Overall, the proposed FedABKD method achieves the best performance across all evaluation dimensions on both datasets, validating its strong robustness in scenarios with heterogeneous data types.

[0055] Unlike heterogeneous label scenarios, the core challenge of heterogeneous data sets lies in the extreme imbalance of sample sizes, leading to significant differences in the convergence speed of local models and consequently parameter conflicts during global aggregation. On the 3MDAD dataset, the overall performance of each method is superior to that of the SFDDD dataset, demonstrating the impact of dataset characteristics on heterogeneous federated methods. FedABKD continues to significantly outperform. Comparing the results of the two datasets reveals that FedABKD, through its adaptive heterogeneous adaptation mechanism, consistently demonstrates its advantages across different datasets. This validates its ability to effectively balance the contradiction between local data scarcity and global knowledge aggregation in scenarios with uneven data distribution, achieving superior feature representation and classification decision performance.

[0056] like Figure 3 As shown, on the AUCDDD dataset, the performance of various methods is generally lower than that of the SFDDD and 3MDAD datasets, while FedABKD still shows a significant advantage, achieving an accuracy of 72.40%, which is significantly better than all the compared methods. This result indicates that FedABKD's adaptive dynamic bidirectional distillation mechanism can effectively cope with the challenges of multiple heterogeneous scenarios. When label heterogeneity and quantity heterogeneity coexist, traditional methods often face the dual dilemma of knowledge transfer distortion and local overfitting: on the one hand, model heterogeneity leads to feature space misalignment, making it difficult to transfer knowledge effectively; on the other hand, data heterogeneity leads to local training bias, making it difficult for the model to converge to the global optimum. FedABKD achieves cross-architecture semantic alignment at the feature level and cross-client distribution calibration at the decision level by adaptively adjusting the weights of knowledge distillation, thereby balancing local data characteristics and global model consistency and achieving better generalization performance.

[0057] In summary, FedABKD significantly outperforms existing advanced federated learning methods in heterogeneous data scenarios. Compared to traditional federated learning methods such as LG-FedAvg and FedProx, FedABKD introduces a knowledge distillation mechanism, mitigating data heterogeneity without relying on complex regularization constraints. Compared to knowledge distillation-based methods such as FedKD and FedMKD, FedABKD employs a bi-branch distillation design, simultaneously achieving knowledge alignment at the feature and decision levels, avoiding the limitations of single distillation methods. Compared to prototype-based methods such as FedTGP and FedKTL, FedABKD avoids explicit propagation of prototype information, exhibiting stronger scenario adaptability through end-to-end adaptive learning.

[0058] The experimental results across three datasets demonstrate that FedABKD significantly outperforms existing state-of-the-art heterogeneous federated learning methods in heterogeneous data scenarios. Compared to traditional federated learning methods such as LG-FedAvg and FedProx, FedABKD introduces a knowledge distillation mechanism, mitigating data heterogeneity without relying on complex regularization constraints. Compared to knowledge distillation-based methods like FedKD and FedMKD, FedABKD employs a bi-branch distillation design, simultaneously achieving knowledge alignment at the feature and decision levels, avoiding the limitations of single distillation methods. Compared to prototype-based methods like FedTGP and FedKTL, FedABKD avoids the transmission of prototype information, exhibiting stronger scenario adaptability.

[0059] To further evaluate the performance of FedABKD in heterogeneous model scenarios, two heterogeneous model settings (heterogeneous feature extractor and heterogeneous model) were designed in the experiment. Comparative experiments were conducted on the SFDDD dataset with heterogeneous labels to explore the impact of different levels of model heterogeneity on algorithm performance. The experiment still compared FedABKD with several advanced heterogeneous federated learning algorithms, focusing on analyzing its performance under different heterogeneous model configurations. The experimental results are shown in Table 3.

[0060] Table 3. Test accuracy (%) of the SFDDD dataset under different levels of model heterogeneity.

[0061] Analysis of Heterogeneous Feature Extractor Scenarios: Under this setting, as the complexity of the feature extractor increases (HtFE3→HtFE9), the accuracy of each method generally shows a downward trend. This is because the higher the heterogeneity of the feature extractors, the greater the difference in feature representations between different clients, and the greater the difficulty of federated aggregation. However, FedABKD maintains the best performance under all heterogeneous feature extractor configurations. In highly challenging, highly heterogeneous scenarios (HtFE8, HtFE9), FedABKD still maintains its lead, with an accuracy of 85.52% under the HtFE8 configuration and 86.40% under the HtFE9 configuration, both showing improvements compared to the second-best methods. This result indicates that FedABKD's adaptive bidirectional knowledge distillation mechanism can effectively align the output features of different heterogeneous feature extractors, reduce feature representation differences, and maintain good performance stability even in highly heterogeneous scenarios. This is because the bidirectional distillation mechanism can learn the common semantic representation of feature extractors with different architectures: the proxy model passes global consensus knowledge to the private model, and the private model feeds back local data characteristics to the proxy model. The synergy between the two effectively alleviates the knowledge transfer barrier caused by the heterogeneity of feature extractors.

[0062] Heterogeneous model scenario analysis: Heterogeneous model settings further increase the complexity of federated learning, with different clients using completely different model structures (Res34-HtC4, HtFE8-HtC4, HtM). 10 This places higher demands on the algorithm's cross-model knowledge transfer capabilities. In this scenario, FedABKD still demonstrates significant advantages: with the Res34-HtC4 configuration, the accuracy reaches 99.23%; with the HtFE8-HtC4 configuration, the accuracy is 85.20%; and with HtM... 10 Under this configuration, the accuracy reached 89.19%. Unlike heterogeneous feature extractor scenarios, the challenge in heterogeneous model scenarios lies in the fact that differences in the classification head structure lead to output space dimensions being consistent but semantic mapping relationships differing. FedABKD performs exceptionally well in this scenario because its bidirectional knowledge distillation mechanism does not rely on a specific model structure and can adaptively extract common knowledge from different models. Through mutual learning between the proxy model and the private model, effective cross-model knowledge aggregation is achieved, mitigating the performance degradation caused by model heterogeneity. It is worth noting that in HtM... 10 In this configuration, FedABKD, for the first time, introduces the ViT architecture, forming a heterogeneous ensemble with CNN models, yet still maintains leading performance, validating its adaptability to different model paradigms (CNN and Visual Transformer). Experimental results from both heterogeneous model scenarios demonstrate that FedABKD exhibits excellent performance under varying degrees of model heterogeneity, effectively addressing the challenges posed by heterogeneous feature extractors and heterogeneous complete models. Compared to existing methods, FedABKD's core advantage lies in its adaptive bidirectional knowledge distillation design, which dynamically adjusts the weights of knowledge distillation based on the degree of model heterogeneity. Experimental results fully validate its flexibility and practicality in real-world applications.

[0063] In the aforementioned adaptive bidirectional distillation training method for heterogeneous federated learning, a unified lightweight proxy model is used as the medium for information exchange between clients. This retains the advantages of proxy models in terms of privacy protection and knowledge sharing while avoiding dependence on public datasets. To enhance learning and generalization capabilities in heterogeneous data environments, this method first employs a confidence-based screening mechanism: by identifying samples with low confidence in the proxy model, reinforcement distillation is performed in conjunction with the private model, thereby optimizing the aggregation performance of the proxy model. To improve the efficiency of knowledge transfer between the proxy model and the private model, a bidirectional knowledge distillation method is introduced during local updates, enabling the private and proxy models to learn from each other. Furthermore, considering that fixed-weight distillation cannot adapt to differences in sample classification difficulty, adaptive dynamic distillation is introduced to improve the efficiency of asymmetric knowledge transfer between the proxy model and the private model, ensuring that only high-quality knowledge is transferred and avoiding noise interference from weak proxy models in the training of the private model. Finally, each client only needs to upload the proxy model to the server to complete the aggregation. Extensive experiments on multiple real-world driver distraction behavior datasets demonstrate that the FedABKD framework exhibits superior performance compared to existing mainstream methods in heterogeneous environments. The research results confirm the efficiency of the framework and demonstrate its practicality and robustness in real-world applications.

[0064] Specifically, this method proposes a federated learning framework applicable to both heterogeneous and homogeneous models. This framework achieves efficient collaborative learning while protecting user data privacy, allowing clients to train personalized models and exhibiting high adaptability and practicality, making it suitable for application scenarios with privacy requirements. It eliminates dependence on public data, effectively mitigating the negative impact of public data initialization in heterogeneous model learning. This method guides efficient knowledge transfer through bidirectional distillation between the proxy model and the local model at the logit layer, effectively alleviating the asymmetry problem of low knowledge transfer efficiency between the proxy model and the local model, thereby improving the framework's robustness and generalization ability in heterogeneous environments.

[0065] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0066] In one embodiment, an adaptive bidirectional distillation training system for heterogeneous federated learning is provided. The system includes servers that communicate with each other and multiple clients with heterogeneous computing power. The servers are configured with a global proxy model, and each client is configured with a heterogeneous local private model and a local copy of the global proxy model. The server is configured as follows: Initialize the global proxy model and send the initial model parameters of the global proxy model to each of the clients participating in the training; The system receives updated model parameters from local copies of the global proxy model uploaded by each client, and uses a federated average algorithm to weight and aggregate the model parameters to update the global proxy model. The updated global proxy model parameters are sent to each of the clients participating in the training. The process of local client update and server aggregation is repeated until the global proxy model converges, thus completing the training of the heterogeneous federated learning model. Each of the aforementioned clients is configured as follows: Update the local copy of the global proxy model based on the initial model parameters issued by the server; The local copy of the global proxy model is used to predict non-IID distributed private training data stored locally, and the prediction confidence of the samples is calculated. Based on the sample prediction confidence, calculate the adaptive dynamic distillation weights that are positively correlated with the sample classification difficulty; Based on the adaptive dynamic distillation weights, bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model to update the parameters of the local copy of the global proxy model and the local private model respectively. Upload the model parameters of the updated local copy of the global proxy model to the server.

[0067] For specific limitations on the adaptive bidirectional distillation training system for heterogeneous federated learning mentioned above, please refer to the limitations on the corresponding adaptive bidirectional distillation training method for heterogeneous federated learning mentioned above, which will not be repeated here.

[0068] Each functional module in the above system can be implemented entirely or partially through software, hardware, or a combination thereof. The hardware can be a computer device including a processor and memory. Each module can be embedded in the processor in hardware form or independent of the processor, or it can be stored in the memory in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0069] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computational and control capabilities to support the training and inference operations of heterogeneous federated learning models. The memory includes a non-volatile storage medium and internal memory; the non-volatile storage medium stores an operating system and a computer program corresponding to the heterogeneous federated learning adaptive bidirectional distillation training method of this application; the internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium, ensuring efficient execution of parameter calculation and data caching during model training. The network interface of the computer device is used to communicate with external servers or other clients via network connection to realize the distribution, uploading, and data interaction of model parameters. When the computer program is executed by the processor, it implements the heterogeneous federated learning adaptive bidirectional distillation training method described in this application. The computer device's display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad set on the computer device's casing, or an external keyboard, touchpad, or mouse, etc., used to perform operations such as setting model parameters and triggering training instructions.

[0070] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0071] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: The server initializes the global proxy model and sends the initial model parameters to each of the clients participating in the training. Each client updates its local copy of the global proxy model according to the initial model parameters. In each of the aforementioned clients, a local copy of the global proxy model is used to make predictions based on locally stored non-IID distribution private training data, and the sample prediction confidence is calculated. Based on the sample prediction confidence, an adaptive dynamic distillation weight that is positively correlated with the sample classification difficulty is calculated. Based on the adaptive dynamic distillation weight, a bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model, and the parameters of the local copy of the global proxy model and the local private model are updated respectively. Each client uploads the updated model parameters of the local copy of the global proxy model to the server; The server uses a federated averaging algorithm to aggregate the model parameters uploaded by all clients, updates the global proxy model, and then distributes the updated global proxy model parameters to each client participating in the training. The process of local client updates and server aggregation is repeated until the global proxy model of the server converges, thus completing the training of the heterogeneous federated learning model.

[0072] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: The server initializes the global proxy model and sends the initial model parameters to each of the clients participating in the training. Each client updates its local copy of the global proxy model according to the initial model parameters. In each of the aforementioned clients, a local copy of the global proxy model is used to make predictions based on locally stored non-IID distribution private training data, and the sample prediction confidence is calculated. Based on the sample prediction confidence, an adaptive dynamic distillation weight that is positively correlated with the sample classification difficulty is calculated. Based on the adaptive dynamic distillation weight, a bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model, and the parameters of the local copy of the global proxy model and the local private model are updated respectively. Each client uploads the updated model parameters of the local copy of the global proxy model to the server; The server uses a federated averaging algorithm to aggregate the model parameters uploaded by all clients, updates the global proxy model, and then distributes the updated global proxy model parameters to each client participating in the training. The process of local client updates and server aggregation is repeated until the global proxy model of the server converges, thus completing the training of the heterogeneous federated learning model.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An adaptive bidirectional distillation training method for heterogeneous federated learning, characterized in that, The method is applied to a distributed computer system consisting of a server and multiple clients. The server is configured with a global proxy model, and each client is a computing terminal with heterogeneous computing power, and is configured with a heterogeneous local private model and a local copy of the global proxy model. The method includes: The server initializes the global proxy model and sends the initial model parameters to each of the clients participating in the training. Each client updates its local copy of the global proxy model according to the initial model parameters. In each of the aforementioned clients, a local copy of the global proxy model is used to make predictions based on locally stored non-IID distribution private training data, and the sample prediction confidence is calculated. Based on the sample prediction confidence, an adaptive dynamic distillation weight that is positively correlated with the sample classification difficulty is calculated. Based on the adaptive dynamic distillation weight, a bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model, and the parameters of the local copy of the global proxy model and the local private model are updated respectively. Each client uploads the updated model parameters of the local copy of the global proxy model to the server; The server uses a federated averaging algorithm to aggregate the model parameters uploaded by all clients, updates the global proxy model, and then distributes the updated global proxy model parameters to each client participating in the training. The process of local client updates and server aggregation is repeated until the global proxy model of the server converges, thus completing the training of the heterogeneous federated learning model.

2. The adaptive bidirectional distillation training method for heterogeneous federated learning according to claim 1, characterized in that, The calculation of sample prediction confidence includes: The logit value of the sample is output through a local copy of the global proxy model; The logit value is subjected to softmax normalization to obtain the class prediction probability of the sample; The maximum value among the predicted probabilities of the categories is selected as the prediction confidence of the sample.

3. The adaptive bidirectional distillation training method for heterogeneous federated learning according to claim 1, characterized in that, The adaptive dynamic distillation weights, which are positively correlated with the sample classification difficulty, are calculated based on the sample prediction confidence, including: Low-confidence samples with prediction confidence levels less than or equal to a confidence threshold are filtered out using an indicator function; The adaptive dynamic distillation weight is calculated based on the predicted confidence of the low-confidence samples, and the lower the predicted confidence of the low-confidence samples, the larger the corresponding adaptive dynamic distillation weight value.

4. The adaptive bidirectional distillation training method for heterogeneous federated learning according to claim 3, characterized in that, The formula for calculating the adaptive dynamic distillation weight is as follows: ； In the above formula, This indicates adaptive dynamic distillation weights. This represents the total number of samples in the private training data. Indicates an indicator function, Indicates the first The prediction confidence level for each sample This indicates the preset confidence threshold.

5. The adaptive bidirectional distillation training method for heterogeneous federated learning according to claim 1, characterized in that, The bidirectional knowledge distillation of the logit layer includes: The logit layer output knowledge of the local copy of the global proxy model is passed to the local private model, and the logit layer output knowledge of the local private model is fed back to the local copy of the global proxy model, realizing the bidirectional interactive transmission of global common knowledge and local personalized knowledge.

6. The adaptive bidirectional distillation training method for heterogeneous federated learning according to claim 5, characterized in that, The bidirectional knowledge distillation measures the difference in output between the local copy of the global proxy model and the local private model at the logit layer using KL divergence. The loss function of the bidirectional knowledge distillation is constructed by combining cross-entropy loss and KL divergence distillation loss, and the total loss function of the local private model and the local copy of the global proxy model are both multiplied by the adaptive dynamic distillation weights.

7. The adaptive bidirectional distillation training method for heterogeneous federated learning according to claim 1, characterized in that, The global agent model is a lightweight deep learning model, including at least one of a 4-layer convolutional neural network and MobileNetV2; The local private model is a heterogeneous deep learning model adapted to the client's computing power, including at least one of the ResNet series models, ViT series models, and basic convolutional neural networks.

8. An adaptive bidirectional distillation training system for heterogeneous federated learning, characterized in that, The system includes servers that communicate with each other and multiple clients with heterogeneous computing power. The servers are configured with a global proxy model, and each client is configured with a heterogeneous local private model and a local copy of the global proxy model. The server is configured as follows: Initialize the global proxy model and send the initial model parameters of the global proxy model to each of the clients participating in the training; The system receives updated model parameters from local copies of the global proxy model uploaded by each client, and uses a federated average algorithm to weight and aggregate the model parameters to update the global proxy model. The updated global proxy model parameters are sent to each of the clients participating in the training. The process of local client update and server aggregation is repeated until the global proxy model converges, thus completing the training of the heterogeneous federated learning model. Each of the aforementioned clients is configured as follows: Update the local copy of the global proxy model based on the initial model parameters issued by the server; The local copy of the global proxy model is used to predict non-IID distributed private training data stored locally, and the prediction confidence of the samples is calculated. Based on the sample prediction confidence, calculate the adaptive dynamic distillation weights that are positively correlated with the sample classification difficulty; Based on the adaptive dynamic distillation weights, bidirectional knowledge distillation of the logit layer is performed between the local copy of the global proxy model and the local private model to update the parameters of the local copy of the global proxy model and the local private model respectively. Upload the model parameters of the updated local copy of the global proxy model to the server.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images