Heterogeneous federated learning method based on spatio-temporal data distillation

Abstract

The invention discloses a heterogeneous federal learning method based on spatio-temporal data distillation, which is used for solving the problems that knowledge forgetting occurs when a new global model is trained due to the fact that the time problem of knowledge retention is not fully solved in the related technology, and meanwhile unique spatio-temporal characteristics of data of each client are not fully considered. And the efficiency and convergence of federal learning are poor. The server side receives local coded data which is sent by each client side and is obtained after coding based on local data, carries out data-free global distillation to obtain a first global model, and distributes the first global model to each client side; the client performs partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment and optimization, generates a local model and sends the local model back to the server; and the server side updates the first global model through a data-free global distillation strategy according to each local model to obtain a second global model, and distributes the second global model to each client side for next round of training.

Description

Technical Field

[0001] The present invention relates to the field of personalized federated learning technology, and in particular to a heterogeneous federated learning method based on spatiotemporal data distillation, a heterogeneous federated learning device based on spatiotemporal data distillation, an electronic device and a storage medium. Background Art

[0002] Federated Learning (FL) is a distributed machine learning method. It allows multiple clients to collaborate to train a shared global model without directly exchanging data. This method avoids the centralized processing of sensitive data and the risk of potential leakage, and has significant advantages in data privacy protection. Although federated learning provides a new solution for data privacy protection, it faces the challenge of data heterogeneity in practical applications. Since the data distribution of clients may differ significantly, this leads to inconsistent performance of the global model on different clients. Client drift may even occur. That is, the difference between the local model and the global model is too large, affecting model performance and convergence.

[0003] To overcome the data heterogeneity problem in federated learning, the concept of personalized federated learning came into being. Personalized federated learning aims to customize the model for each client to capture and utilize its unique data characteristics. This approach attempts to provide more accurate prediction results for each client while maintaining the generalization ability of the global model. However, personalized federated learning faces the challenge of how to effectively integrate and learn the personalized data of the client without sacrificing global generalization.

[0004] Current technologies have provided a variety of solutions to the challenges in personalized federated learning. These include using model aggregation techniques to explain differences in client data distribution. And applying knowledge distillation techniques to transfer knowledge from global models to local models. Although these methods have alleviated the impact of data heterogeneity to a certain extent, they have not yet fully addressed the needs of personalized learning. The current technical means have failed to fully address the time problem of knowledge retention, resulting in knowledge forgetting when training a new global model. In addition, current methods have also failed to fully consider the unique spatiotemporal characteristics of each client's data, resulting in poor efficiency and convergence of federated learning. Summary of the invention

[0005] The present invention provides a heterogeneous federated learning method based on spatiotemporal data distillation, a heterogeneous federated learning device based on spatiotemporal data distillation, an electronic device and a storage medium, which are used to solve or partially solve the technical problems that the related technology fails to fully solve the time problem of knowledge retention, resulting in knowledge forgetting when training a new global model, and fails to fully consider the unique spatiotemporal characteristics of each client data, resulting in poor efficiency and convergence of federated learning.

[0006] The present invention provides a heterogeneous federated learning method based on spatiotemporal data distillation, which is applied to a heterogeneous federated learning system. The heterogeneous federated learning system includes at least one server end, and each of the server ends is communicatively connected with multiple clients. The heterogeneous federated learning method includes:

[0007] The server receives local encoded data sent by each of the clients and obtained by encoding based on local data, performs data-free global distillation according to each of the local encoded data to obtain a first global model, and distributes the first global model to each of the clients;

[0008] The client performs partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment optimization, generates a local model based on the partial parameter local distillation result and the non-real time distillation result, and sends the local model to the server;

[0009] The server updates the first global model through a data-free global distillation strategy based on each of the local models to obtain a second global model, and distributes the second global model to each of the clients for the next round of federated learning training.

[0010] Optionally, the server receives local encoded data sent by each of the clients and obtained by encoding based on local data, and performs data-free global distillation according to each of the local encoded data to obtain a first global model, including:

[0011] Receiving local encoded data sent by each of the clients after encoding based on local data;

[0012] Using a generator network constructed based on a meta-learning method to train each of the local encoding data to learn the common features of each of the local encoding data;

[0013] Based on the common feature learning results, outputting a virtual meta-learning global dataset through a trained generator network, wherein the meta-learning global dataset includes a plurality of common representation pseudo samples;

[0014] A global model is initialized as a student model, the trained generator network is used as a teacher model, and the multiple common representation pseudo samples are used as outputs of the teacher model for distillation. The initialized global model is optimized by minimizing the difference between the outputs of the student model and the teacher model to obtain a first global model.

[0015] Optionally, in each of the clients, according to the local data and the first global model, partial parameter local distillation and non-real time distillation are performed in combination with dynamic temperature adjustment optimization, and a process of generating a local model based on the partial parameter local distillation result and the non-real time distillation result includes:

[0016] Obtaining a meta-learning global dataset from the server;

[0017] According to the local data, the meta-learning global data set and the first global model, partial parameter local distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a partially distilled local sub-model;

[0018] According to the meta-learning global data set and the first global model, non-real time distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a non-real distillation local sub-model;

[0019] The partially distilled local sub-model is added to the non-true distilled local sub-model to obtain a local model.

[0020] Optionally, performing local distillation of some parameters according to the local data, the meta-learning global data set and the first global model in combination with an optimal distillation temperature obtained based on a dynamic temperature adjustment mechanism to obtain a partially distilled local sub-model includes:

[0021] Using the first global model as a local model to be trained of the client, and dividing it into a feature extraction module and a prediction module;

[0022] Obtaining a prediction copy module of the prediction module by copying, and constructing a feature extraction local model according to the prediction copy module and the feature extraction module;

[0023] Inputting the local data into the feature extraction local model, calculating the cross entropy loss between the model prediction result and the true label, and updating the feature extraction module in the feature extraction local model while keeping the prediction replication module unchanged by optimizing the cross entropy loss to obtain a feature extraction update module;

[0024] Obtain a feature extraction replication module of the feature extraction module by replication, and construct a prediction local model according to the feature extraction replication module and the prediction module;

[0025] The meta-learning global data set is input into the prediction local model and the first global model respectively, knowledge distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism, and the knowledge distillation loss is solved. By optimizing the knowledge distillation loss, the prediction module in the prediction local model is updated while keeping the feature extraction replication module unchanged, so as to obtain a prediction update module;

[0026] The feature extraction update module and the prediction update module are integrated into the first global model to obtain a partially distilled local sub-model.

[0027] Optionally, performing non-real time distillation according to the meta-learning global dataset and the first global model in combination with an optimal distillation temperature obtained based on a dynamic temperature adjustment mechanism to obtain a non-real distillation local sub-model includes:

[0028] Using the first global model as the local model to be trained of the client, and calculating the performance loss of the local model to be trained on the meta-learning global dataset;

[0029] Using the local model to be trained to perform prediction based on the memory replay mechanism, and calculating the local retention loss;

[0030] Based on the performance loss and the local preservation loss, and taking into account the knowledge forgetting metric, a non-true time total loss is constructed;

[0031] Taking minimizing the non-true total time loss as the optimization goal, dual-objective knowledge distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to optimize the model parameters of the local model to be trained and obtain a non-true distillation local sub-model.

[0032] Optionally, the process of obtaining the optimal distillation temperature based on the dynamic temperature adjustment mechanism comprises:

[0033] Based on the specified task loss and knowledge distillation loss of the student model of the client during the knowledge distillation process, construct a total loss function of the student model;

[0034] Temperature prediction training is performed by adversarially optimizing the optimization path of the student model. During the training process, the difficulty of the knowledge distillation task is adjusted by a scaling factor, and minimizing the total loss of the student model is used as the optimization goal. The total loss function of the student model is solved to obtain the predicted distillation temperature.

[0035] A nonlinear activation process is performed according to the predicted distillation temperature to obtain an actual distillation temperature, and the actual distillation temperature is used as an optimal distillation temperature for a partial parameter local distillation process and a non-real time distillation process.

[0036] Optionally, the process of updating the first global model by a data-free global distillation strategy according to each of the local models to obtain the second global model in the server side includes:

[0037] Using the first global model as a student model and each of the local models as a teacher model;

[0038] Comprehensively considering the cross entropy loss and knowledge distillation loss of the server in the knowledge distillation process, constructing a dual-loss global function to measure the difference between the prediction accuracy of the first global model on the meta-learning global dataset and the sum of the outputs of each of the local models;

[0039] Taking minimizing the dual-loss global function as the optimization goal, the first global model is updated through knowledge distillation to obtain a second global model.

[0040] The present invention also provides a heterogeneous federated learning device based on spatiotemporal data distillation, which is applied to a heterogeneous federated learning system. The heterogeneous federated learning system includes at least one server end, each of which is communicatively connected to a plurality of clients; the heterogeneous federated learning device includes:

[0041] A global model generating unit, configured to receive, at the server end, local encoded data obtained by encoding based on local data sent by each of the clients, perform data-free global distillation according to each of the local encoded data to obtain a first global model, and distribute the first global model to each of the clients;

[0042] A local model generating unit, configured for the client to perform partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment optimization, generate a local model based on the partial parameter local distillation result and the non-real time distillation result, and send the local model to the server;

[0043] A global model updating unit is used for the server to update the first global model according to each of the local models through a data-free global distillation strategy to obtain a second global model, and distribute the second global model to each of the clients for the next round of federated learning training.

[0044] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0045] The memory is used to store program code and transmit the program code to the processor;

[0046] The processor is used to execute the heterogeneous federated learning method based on spatiotemporal data distillation as described in any of the above items according to the instructions in the program code.

[0047] The present invention also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the heterogeneous federated learning method based on spatiotemporal data distillation as described in any of the above items.

[0048] It can be seen from the above technical solutions that the present invention has the following advantages:

[0049] A heterogeneous federated learning method based on spatiotemporal data distillation is provided. The server receives the local encoded data obtained by encoding based on the local data sent by each client, performs data-free global distillation according to each local encoded data, obtains the first global model, and distributes the first global model to each client; the client performs partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment optimization, generates a local model based on the partial parameter local distillation results and the non-real time distillation results, and sends the local model to the server; the server updates the first global model according to each local model through the data-free global distillation strategy to obtain the second global model, and distributes the second global model to each client for the next round of federated learning training. Therefore, by combining the spatiotemporal data distillation of partial parameter local distillation and non-real time distillation, historical knowledge and spatial knowledge will not be forgotten, and can be used to guide the training of the current model, thereby improving the efficiency and convergence of federated learning. Extracting the unique data features of each client through data-free distillation can alleviate data heterogeneity and effectively solve the client drift problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 This is a schematic diagram of the structure of a heterogeneous federated learning system based on spatiotemporal data distillation;

[0052] Figure 2 A flowchart of the steps of a heterogeneous federated learning method based on spatiotemporal data distillation;

[0053] Figure 3 The overall architecture diagram of a personalized heterogeneous federated learning framework;

[0054] Figure 4 A schematic diagram of a data-free global distillation update process on a server;

[0055] Figure 5 Flowchart for performing local distillation of some parameters for a client;

[0056] Figure 6 A schematic diagram of a non-real time distillation;

[0057] Figure 7 The structural block diagram of a heterogeneous federated learning device based on spatiotemporal data distillation. DETAILED DESCRIPTION

[0058] The embodiments of the present invention provide a heterogeneous federated learning method based on spatiotemporal data distillation, a heterogeneous federated learning device based on spatiotemporal data distillation, an electronic device and a storage medium, which are used to solve or partially solve the technical problem that the related technology fails to fully solve the time problem of knowledge retention, resulting in knowledge forgetting when training a new global model, and fails to fully consider the unique spatiotemporal characteristics of each client data, resulting in poor efficiency and convergence of federated learning.

[0059] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] As an example, current technologies provide a variety of solutions to the challenges in personalized federated learning. These include using model aggregation techniques to explain differences in client data distribution. And applying knowledge distillation techniques to transfer knowledge from global models to local models. Although these methods have alleviated the impact of data heterogeneity to a certain extent, they still fail to fully address the needs of personalized learning. The current technical means fail to fully address the time problem of knowledge retention, resulting in knowledge forgetting when training new global models. In addition, current methods also fail to fully consider the unique spatiotemporal characteristics of each client's data, resulting in poor efficiency and convergence of federated learning.

[0061] Therefore, one of the core invention points of the embodiment of the present invention is: in order to solve the client drift caused by the data heterogeneity in federated learning, as well as the problems of knowledge retention and the unique spatiotemporal characteristics of client data not being considered in the system applying personalized federated learning, a personalized heterogeneous federated learning system and method based on spatiotemporal data distillation are proposed. On the one hand, by combining spatiotemporal data distillation with partial parameter local distillation and non-real time distillation, historical knowledge and spatial knowledge will not be forgotten, which can be used to guide the training of the current model and improve the efficiency and convergence of federated learning. On the other hand, by extracting the unique data features of each client through data-free distillation, data heterogeneity can be alleviated, and the client drift problem can be effectively solved. The technical solution of the present invention can not only effectively solve the data heterogeneity problem in traditional federated learning and reduce client drift, but also enhance the personalized adaptability of the model and alleviate the phenomenon of knowledge forgetting.

[0062] Reference Figure 1 , showing a structural schematic diagram of a heterogeneous federated learning system based on spatiotemporal data distillation provided by an embodiment of the present invention.

[0063] Combination Figure 1 The heterogeneous federated learning system 100 includes at least one server 101, each of which is connected to multiple clients 102 via a wireless or wired network. The server 101 is responsible for initializing the global model and building a generator network, while the client 102 uses local data to perform personalized model training. In the embodiment of the present invention, each client 102 optimizes its local model through a two-way distillation strategy, including parameter sharing in the spatial dimension and knowledge retention in the temporal dimension, combined with a dynamic temperature adjustment mechanism.

[0064] Based on the heterogeneous federated learning system introduced in the above embodiment, refer to Figure 2 , shows a flowchart of a heterogeneous federated learning method based on spatiotemporal data distillation provided by an embodiment of the present invention, which may specifically include the following steps:

[0065] Step 201, the server receives local encoded data sent by each of the clients and obtained by encoding based on local data, performs data-free global distillation according to each of the local encoded data to obtain a first global model, and distributes the first global model to each of the clients;

[0066] In order to enable those skilled in the art to better understand the technical solution of the present invention, Figure 3 The overall architecture diagram of a personalized heterogeneous federated learning framework provided by an embodiment of the present invention is shown. Figure 3 The heterogeneous federated learning method based on spatiotemporal data distillation is explained.

[0067] In the Federated Spatio-Temporal Knowledge Distillation (FedSTKD) framework, each client is responsible for maintaining and updating its own local model, while the server is responsible for maintaining the global model and aggregating model updates from all clients. In this process, the client trains the model with local data and sends the model update to the server instead of sharing the data directly. This improves data privacy. The server collects local model updates from all clients and updates the global model through an aggregation algorithm, namely Federated Averaging (FL-Avg).

[0068] In the initialization phase, the server first receives the local encoded data sent by each client after encoding based on the local data. It is understandable that the local data of each client can be any data that can be used for model training and federated learning. Such as various types of text data, audio data, image data or video data collected or generated based on different actual application scenarios. Then the server constructs a generator network through a meta-learning method, which can learn the common features of different client data and generate a set of pseudo samples. The server then uses these pseudo samples for data-free global distillation and generates an initial global model based on the distillation results. The global model is then distributed to all clients as the starting point for local training of each client.

[0069] Figure 4 A schematic diagram of a data-free global distillation update process on a server;

[0070] In some embodiments, the server receives local encoded data sent by each client and obtained by encoding based on local data, and performs data-free global distillation according to each local encoded data to obtain a first global model, including the following sub-steps S01 to S04:

[0071] Step S01: receiving local encoded data sent by each client after encoding based on local data;

[0072] Each client Initialize a local model . While maintaining the local dataset . Where x represents the data and y represents the true label of the data. The server uses the meta-learning method to build a generator network .

[0073] The goal of the generator network is to learn a common latent space so that data from different clients can be represented in this space. The generator network can be a neural network that accepts a latent code As input, it outputs a sample with a similar distribution to the input data.

[0074] Each client Using its local model For local data Encode and get the potential code As local encoding data. These latent encodings Sent to the server.

[0075] Step S02: using a generator network constructed based on a meta-learning method to train each local encoding data to learn the common features of each local encoding data;

[0076] In order to deal with the problem of data heterogeneity, the embodiment of the present invention defines an optimization problem, which aims to identify the parameters by closely aligning the data points in the latent space. . This enables the generator to effectively capture common features. The goal of the optimization problem is to minimize the distance between latent codes and the distance between the generated data and the actual data Among them, the common features can be expressed as:

[0077]

[0078] Potential encoding for each client , the server uses a generator network These encodings are trained to learn common features of the data.

[0079] Based on common feature learning, the generator network can output pseudo samples . Pseudo sample Designed to represent a common representation of all client data. Under the guidance of the loss function Sub-gradient descent can get new parameter values In this way, composite gradients can be introduced into the training of the generator and this challenge can be addressed by simplifying the efficiency of backpropagation. The process can be expressed as:

[0080]

[0081] in, Representatives in The gradient obtained in the iteration.

[0082] Step S03: Based on the common feature learning result, a virtual meta-learning global data set is output through the trained generator network, where the meta-learning global data set includes a plurality of common representation pseudo samples;

[0083] After the previous steps, the server can get the trained generator network. At the same time, the server can also use the generator network to output the representation of the pseudo sample. The representations of these pseudo samples constitute a virtual meta-learning global dataset Since the pseudo samples are generated after learning the common features, the dataset does not depend on any real data. Next, the server will meta-learn the global dataset Used together with the current global model for the data-free global distillation process.

[0084] Step S04: Initialize a global model as the student model, use the trained generator network as the teacher model, and distill multiple common representation pseudo samples as the output of the teacher model. By minimizing the difference between the output of the student model and the teacher model, the initialized global model is optimized to obtain the first global model.

[0085] The server initializes a global model . Initialize the global model As a student model, meta-learning the global dataset The pseudo samples in are used as the output of the teacher model for distillation. By minimizing the difference between the output of the student model and the teacher model (generator network), the server is able to optimize the global model to better adapt to the common features across clients. The core goal of this process is to improve the generalization ability of the global model by leveraging the common features learned in the generator network.

[0086] Step 202: the client performs partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment optimization, generates a local model based on the partial parameter local distillation results and the non-real time distillation results, and sends the local model to the server;

[0087] After the client receives the global model sent by the server (to distinguish it from the global model obtained after subsequent aggregation and update, the global model is defined as the first global model), it can perform local distillation of some parameters according to local data, as well as non-real knowledge distillation in the time dimension (i.e., non-real time distillation). In this process, the dynamic temperature adjustment module is used to optimize the knowledge distillation effect so that the model can effectively learn from the global model. At the same time, it can also maintain the adaptability of the model to local data.

[0088] In some embodiments, in each client, partial parameter local distillation and non-real time distillation are performed according to local data and the first global model in combination with dynamic temperature adjustment optimization, and a process of generating a local model based on the partial parameter local distillation result and the non-real time distillation result includes the following sub-steps S11 to S14:

[0089] Step S11: Obtain the meta-learning global dataset from the server ;

[0090] Step S12: performing local distillation of some parameters according to the local data, the meta-learning global data set and the first global model in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a partially distilled local sub-model;

[0091] In the specific implementation, Figure 5 A flowchart of a client performing local distillation of some parameters provided by an embodiment of the present invention is shown. Figure 5 The process of performing local distillation of some parameters according to the local data, the meta-learning global data set and the first global model in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a partially distilled local sub-model may include the following sub-steps S121 to S126:

[0092] Step S121: using the first global model as a local model to be trained on the client, and dividing it into a feature extraction module and a prediction module;

[0093] The first The parameters of the first global model received by the client from the server in round (when t=0, it means the first round of training) As a local model The first global model is divided into two independent modules: feature extraction module and prediction module .

[0094] The role of the feature extraction module is to capture the universal feature representation of the data. It focuses more on the universal laws in global and local data. The prediction module focuses on capturing the high-level prediction capabilities learned in the global model. Through module division, the client can optimize these two modules separately for different learning goals. This can better balance the relationship between global knowledge and local personalization.

[0095] Step S122: obtaining a prediction copy module of the prediction module by copying, and constructing a feature extraction local model according to the prediction copy module and the feature extraction module;

[0096] Different from the distillation of the prediction module, the optimization of the feature extraction module mainly depends on the local dataset. At this stage, the client copies the initial prediction module for , and fix the copied prediction module , only update the feature extraction module Based on feature extraction module And the prediction replication module , you can build a local model for feature extraction ={ , .

[0097] Step S123: inputting local data into the feature extraction local model, calculating the cross entropy loss between the model prediction result and the true label, and updating the feature extraction module in the feature extraction local model while keeping the prediction replication module unchanged by optimizing the cross entropy loss to obtain a feature extraction update module;

[0098] Local data Input to feature extraction local model ={ , In the client, the model's prediction results are calculated The cross entropy loss between the real label and the real label. Among them, the optimization of cross entropy loss enables the feature extraction module to better learn the fine-grained features of local data, thereby enhancing the performance of the model on local data. The cross entropy loss of this process is defined as follows:

[0099]

[0100] Step S124: obtaining a feature extraction replication module of the feature extraction module by replication, and constructing a prediction local model according to the feature extraction replication module and the prediction module;

[0101] For the prediction module, the client The prediction module is distilled using the optimal distillation temperature T obtained from the dynamic temperature adjustment mechanism to absorb the shared knowledge contained in the global model. In this process, the client uses the generator network Generated Meta-Learning Global Dataset . In the previous step, the generator network Client-side latent encoding Generate a batch of fake samples These samples reflect the characteristics of global data and can help the client better align the prediction distribution of the local model with the global model during the distillation process.

[0102] During the distillation process, the client first copies the feature extraction module of the local model for (For the convenience of formula expression, the t in the tth round is omitted in the following) After that, the copied feature extraction module is fixed , only update the prediction module Based on the prediction module And feature extraction replication module , you can build a predictive local model .

[0103] Step S125: inputting the meta-learning global data set into the prediction local model and the first global model respectively, performing knowledge distillation in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism, and solving the knowledge distillation loss. By optimizing the knowledge distillation loss, while keeping the feature extraction replication module unchanged, the prediction module in the prediction local model is updated to obtain a prediction update module;

[0104] Meta-learning global dataset are sequentially input into the local model and the first global model In the process, the optimal distillation temperature T obtained in the dynamic temperature adjustment mechanism is combined to perform knowledge distillation, and the local models are obtained respectively. The prediction results and global model The prediction results The final knowledge distillation loss in this step is as follows:

[0105]

[0106] in, represents the softmax of the result; T represents the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism. In this process, the distillation loss enables the local model to learn the shared prediction ability from the global model.

[0107] Step S126: Integrate the feature extraction update module and the prediction update module into the first global model to obtain a partially distilled local sub-model.

[0108] After completing independent training for the prediction module and feature extraction module, the client integrates the updates of these two submodules into , obtain the partially distilled local sub-model after local distillation This updating process takes into account both the knowledge learned from the global model and the characteristics of the local dataset, so that the local model can capture the commonalities of the global data while adapting to the individuality of the local data.

[0109] In addition, to continuously improve the long-term learning ability of the model, the client can also maintain a knowledge base to store the knowledge gained from the past local parameter distillation process. This knowledge can be reused in subsequent distillations, helping the model to quickly adapt and learn when faced with new local data.

[0110] Step S13: performing non-real time distillation according to the meta-learning global data set and the first global model in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a non-real distillation local sub-model;

[0111] In the specific implementation, Figure 6 FIG. 2 is a schematic diagram of a non-real time distillation provided by an embodiment of the present invention. Figure 6 , according to the meta-learning global data set and the first global model, the process of performing non-real time distillation in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain the non-real distillation local sub-model includes the following sub-steps S131 to S134:

[0112] Step S131: using the first global model as the local model to be trained of the client, and calculating the performance loss of the local model to be trained on the meta-learning global data set;

[0113] Similar to the previous example, each client can maintain a knowledge base. The knowledge base is used to store the knowledge obtained by the client model in the previous non-real time distillation process. This knowledge includes the parameters, feature representations, and other useful information of the historical model. This information can be reused in future learning processes to improve the learning efficiency and accuracy of the model.

[0114] The local model to be trained is used for local distillation of the previous parameters. Distinguish, and define the local model to be trained in the non-real time distillation process as .

[0115] In order to carry out the subsequent non-real time distillation, we first define a knowledge forgetting metric . Knowledge Forgetting Measures It measures the gap between the peak accuracy of each class during learning and its final accuracy.

[0116]

[0117] in, Indicates the number of categories; Indicates The accuracy of each category in round t; It is The accuracy of each category in round T.

[0118] In a heterogeneous federated learning system, each client uses the pseudo sample based on the common features obtained in step S02 to update its model. These pseudo samples represent knowledge replayed from the knowledge base and are used to simulate data for new learning tasks. The client computes the local model for round t Performance loss on these fake samples . It reflects the model's ability to adapt to new knowledge. The loss function can be cross entropy loss or other loss functions suitable for specific tasks. The loss is calculated as:

[0119]

[0120] in, Represents a pseudo sample The real label, Indicates that the client is in the first Local model of the wheel; Represents the loss function, which is used to calculate the model output and the true label The real difference between represents the expected value, for a random variable in a probability distribution, is the long-term average value of the random variable.

[0121] Step S132: using the local model to be trained to perform prediction based on the memory replay mechanism, and calculating the local retention loss;

[0122] To strengthen the model’s memory of old knowledge, the client implements a memory replay mechanism. This includes storing pseudo samples generated in previous iterations and replaying them in subsequent distillation processes. The client uses the local model again. In these replayed pseudo samples Predictions are made on the y-value pair and the associated loss is calculated, which is called the local preservation loss. The implementation of the memory replay mechanism helps the model to keep the memory of previous tasks while learning new tasks. Local Preservation Loss The calculation formula is:

[0123]

[0124] in, represents the previous pseudo sample selected from the memory bank; Indicate here The real label.

[0125] Step S133: constructing a non-true time total loss based on the performance loss and the local retention loss while taking into account the knowledge forgetting metric;

[0126] The total loss of non-true time is expressed as follows:

[0127]

[0128] in, Is a weight factor used to balance the importance of new knowledge adaptability and old knowledge retention. Knowledge forgetting measure is directly added to the total loss so that the model does not forget the knowledge of old tasks when learning new tasks. Represents the performance of the model on the current task. represents the performance of the model on the replayed pseudo samples. Together, these two provide the long-term learning effect of the model.

[0129] Step S134: Taking minimizing the total non-true time loss as the optimization goal, dual-objective knowledge distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to optimize the model parameters of the local model to be trained and obtain a non-true distillation local sub-model.

[0130] This step mainly realizes dual-objective distillation and model update. The optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism is used in the distillation process. Perform knowledge distillation. In the time dimension, the client performs dual-objective distillation by minimizing the total loss of non-true time. , optimize the model parameters , and obtain the non-true distillation local model after non-true time distillation .

[0131] Step S14: Add the partially distilled local sub-model to the non-true distilled local sub-model to obtain a local model.

[0132] After each round of communication, the client will partially distill the local sub-model With non-true distillation local model Add together to get the final local model Each client will local model The model updates are sent to the server, which uses an aggregation algorithm, such as federated averaging, to merge the model updates from all clients and send the updated global model to each client.

[0133] The entire federated learning process is iterative, and each round may involve adjustments to the distillation temperature to optimize the model learning effect and adaptability.

[0134] In some embodiments, the process of obtaining the optimal distillation temperature based on the dynamic temperature adjustment mechanism includes the following sub-steps S21 to S23:

[0135] Step S21: constructing a total loss function of the student model based on the specified task loss and knowledge distillation loss of the student model during the knowledge distillation process of the client;

[0136] The total loss function of the student model includes the loss of the specified task (i.e., task-specific) and knowledge distillation loss Among them, the distillation temperature is adjusted dynamically. The total loss function can be expressed as:

[0137]

[0138] in, and Represent the outputs of the student model and the teacher model respectively; represents the true label; and Represent the parameters of the student model and the teacher model respectively; Indicates the distillation temperature.

[0139] Step S22: Perform temperature prediction training by adversarial student model optimization path. During the training process, adjust the difficulty of the knowledge distillation task by scaling factors, and minimize the total loss of the student model as the optimization goal. Solve the total loss function of the student model to obtain the predicted distillation temperature.

[0140] The temperature prediction is trained by adversarially optimizing the student model to find the distillation temperature that maximizes the performance of the student model, i.e., predicting the distillation temperature. It can be expressed as:

[0141]

[0142] By scaling factor Adjust the difficulty of the knowledge distillation task. It can be expressed as:

[0143]

[0144] Among them, m represents the current epoch; and express the boundaries of express The epoch number of the change.

[0145] Step S23: Perform nonlinear activation processing according to the predicted distillation temperature to obtain the actual distillation temperature, and use the actual distillation temperature as the optimal distillation temperature for the partial parameter local distillation process and the non-real time distillation process.

[0146] Predicted temperature After the nonlinear activation processing of the Sigmoid activation function, it is used to determine the actual distillation temperature T. It can be expressed as:

[0147]

[0148] in, Indicates the basal temperature; Indicates the preset temperature range.

[0149] In step 203, the server updates the first global model through a data-free global distillation strategy according to each of the local models to obtain a second global model, and distributes the second global model to each of the clients for the next round of federated learning training.

[0150] After receiving the local models returned by all clients, the server updates the current global model through the data-free global distillation strategy. After completing the model update, the new global model is distributed to each client to continue the next round of training until the set threshold is reached (such as reaching the preset training rounds, or the model loss is reduced to meet the expected requirements, etc.).

[0151] In some embodiments, the process of updating the first global model by the data-free global distillation strategy according to each local model in the server to obtain the second global model includes the following sub-steps S31 to S33:

[0152] Step S31: using the first global model as a student model and each local model as a teacher model;

[0153] The first global model As a student model, each local model As a teacher model, it updates the global model through knowledge distillation and data-free global distillation strategies.

[0154] Step S32: comprehensively considering the cross entropy loss and knowledge distillation loss of the server in the knowledge distillation process, constructing a dual-loss global function to measure the difference between the prediction accuracy of the first global model on the meta-learning global dataset and the sum of the outputs of each local model;

[0155] In this step, a dual loss objective function is defined, including cross entropy loss and knowledge distillation loss , to measure the first global model on the meta-learning global dataset The difference between the prediction accuracy on and all local model outputs and.

[0156] Step S33: Taking minimizing the dual-loss global function as the optimization goal, the first global model is updated through knowledge distillation to obtain the second global model.

[0157] Minimizing the dual loss objective function can be expressed as:

[0158]

[0159] Updating the first global model through knowledge distillation , and obtain the second global model , and the second global model Sent back to each client as the local model for a new round of training.

[0160] In an embodiment of the present invention, an application method of a personalized heterogeneous federated learning system based on spatiotemporal data distillation is proposed. On the one hand, through spatiotemporal data distillation, historical knowledge and spatial knowledge will not be forgotten, and can be used to guide the training of the current model to improve the efficiency and convergence of federated learning. On the other hand, by extracting the unique data features of each client through data-free distillation, data heterogeneity can be alleviated, and the client drift problem can be effectively solved. The technical solution of the present invention can not only effectively solve the data heterogeneity problem in traditional federated learning and reduce client drift, but also enhance the personalized adaptability of the model and alleviate the phenomenon of knowledge forgetting.

[0161] Reference Figure 7 , shows a structural block diagram of a heterogeneous federated learning device based on spatiotemporal data distillation provided by an embodiment of the present invention, which is applied to the heterogeneous federated learning system introduced in the above embodiment. The heterogeneous federated learning device may specifically include:

[0162] A global model generating unit 701 is configured to receive, at the server end, local encoded data obtained by encoding based on local data sent by each of the clients, perform data-free global distillation according to each of the local encoded data to obtain a first global model, and distribute the first global model to each of the clients;

[0163] A local model generation unit 702 is used for the client to perform partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment optimization, generate a local model based on the partial parameter local distillation results and the non-real time distillation results, and send the local model to the server;

[0164] The global model updating unit 703 is used for the server to update the first global model according to each of the local models through the data-free global distillation strategy, obtain the second global model, and distribute the second global model to each of the clients for the next round of federated learning training.

[0165] In an optional embodiment, the global model generating unit 701 includes:

[0166] A local encoding data receiving unit, used for receiving local encoding data sent by each of the clients after encoding based on local data;

[0167] A common feature learning unit, used for training each of the local encoded data using a generator network constructed based on a meta-learning method to learn common features of each of the local encoded data;

[0168] A meta-learning global data set generating unit, configured to output a virtual meta-learning global data set through a trained generator network based on a common feature learning result, wherein the meta-learning global data set includes a plurality of common representation pseudo samples;

[0169] The initial global model optimization unit is used to initialize a global model as a student model, use the trained generator network as a teacher model, and use the multiple common representation pseudo samples as the output of the teacher model for distillation, and optimize the initialized global model by minimizing the difference between the output of the student model and the teacher model to obtain a first global model.

[0170] In an optional embodiment, the local model generation unit 702 in each of the clients includes:

[0171] A meta-learning global data set acquisition unit, used to acquire a meta-learning global data set from the server;

[0172] A partial parameter local distillation unit, configured to perform partial parameter local distillation according to the local data, the meta-learning global data set, and the first global model in combination with an optimal distillation temperature obtained based on a dynamic temperature adjustment mechanism, to obtain a partially distilled local sub-model;

[0173] A non-real time distillation unit, configured to perform non-real time distillation according to the meta-learning global dataset and the first global model in combination with an optimal distillation temperature obtained based on a dynamic temperature adjustment mechanism, to obtain a non-real distillation local sub-model;

[0174] The local sub-model adding unit is used to add the partially distilled local sub-model and the non-true distilled local sub-model to obtain a local model.

[0175] In an optional embodiment, the partial parameter local distillation unit comprises:

[0176] A model division unit, used to use the first global model as a local model to be trained of the client, and divide the model into a feature extraction module and a prediction module;

[0177] A feature extraction local model building unit, used for obtaining a prediction copy module of the prediction module by copying, and building a feature extraction local model according to the prediction copy module and the feature extraction module;

[0178] A feature extraction module updating unit, configured to input the local data into the feature extraction local model, calculate the cross entropy loss between the model prediction result and the true label, and update the feature extraction module in the feature extraction local model while keeping the prediction copy module unchanged by optimizing the cross entropy loss, so as to obtain a feature extraction update module;

[0179] A prediction local model building unit, used for obtaining a feature extraction copy module of the feature extraction module by copying, and building a prediction local model according to the feature extraction copy module and the prediction module;

[0180] A prediction module updating unit, used for inputting the meta-learning global data set into the prediction local model and the first global model respectively, performing knowledge distillation in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism, and solving the knowledge distillation loss, and updating the prediction module in the prediction local model while keeping the feature extraction replication module unchanged by optimizing the knowledge distillation loss to obtain a prediction update module;

[0181] The partially distilled local sub-model construction unit is used to integrate the feature extraction update module and the prediction update module into the first global model to obtain a partially distilled local sub-model.

[0182] In an optional embodiment, the non-real time distillation unit comprises:

[0183] a performance loss calculation unit, configured to use the first global model as a local model to be trained of the client, and calculate the performance loss of the local model to be trained on the meta-learning global data set;

[0184] A local retention loss calculation unit, used to use the local model to be trained to perform prediction based on a memory replay mechanism and calculate the local retention loss;

[0185] A non-true time total loss construction unit, used for constructing a non-true time total loss based on the performance loss and the local retention loss while considering a knowledge forgetting metric;

[0186] The non-true distillation local sub-model construction unit is used to perform dual-objective knowledge distillation with minimizing the non-true total time loss as the optimization goal, combined with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism, so as to optimize the model parameters of the local model to be trained and obtain the non-true distillation local sub-model.

[0187] In an optional embodiment, the heterogeneous federated learning device further includes:

[0188] A student model total loss function construction unit, used to construct a student model total loss function based on the specified task loss and knowledge distillation loss of the student model of the client during the knowledge distillation process;

[0189] A predicted distillation temperature calculation unit is used to perform temperature prediction training by adversarial student model optimization path. During the training process, the difficulty of the knowledge distillation task is adjusted by a scaling factor, and minimizing the total loss of the student model is used as the optimization goal. The total loss function of the student model is solved to obtain the predicted distillation temperature;

[0190] The actual distillation temperature calculation unit is used to perform nonlinear activation processing according to the predicted distillation temperature to obtain the actual distillation temperature, and use the actual distillation temperature as the optimal distillation temperature for the partial parameter local distillation process and the non-real time distillation process.

[0191] In an optional embodiment, the global model updating unit 703 includes:

[0192] A model definition unit, configured to use the first global model as a student model and each of the local models as a teacher model;

[0193] A dual-loss global function construction unit, used to comprehensively consider the cross entropy loss and knowledge distillation loss of the server in the knowledge distillation process, and construct a dual-loss global function to measure the difference between the prediction accuracy of the first global model on the meta-learning global data set and the sum of the outputs of each of the local models;

[0194] The second global model generating unit is used to update the first global model through knowledge distillation with minimizing the dual-loss global function as the optimization goal to obtain the second global model.

[0195] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the aforementioned method embodiment.

[0196] It should be noted that, in order to enable those skilled in the art to better distinguish data of the same type but with different actual meanings, some technical features are distinguished by the first and the second in the embodiments of the present invention. The first and the second are only used for data distinction and have no other special meanings. It can be understood that the present invention is not limited to this.

[0197] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:

[0198] The memory is used to store the program code and transmit the program code to the processor;

[0199] The processor is used to execute the heterogeneous federated learning method based on spatiotemporal data distillation of any embodiment of the present invention according to the instructions in the program code.

[0200] An embodiment of the present invention also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the heterogeneous federated learning method based on spatiotemporal data distillation of any embodiment of the present invention.

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0202] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0203] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0206] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A heterogeneous federated learning method based on spatiotemporal data distillation, characterized in that: Applied to a heterogeneous federated learning system, the heterogeneous federated learning system includes at least one server end, each of the server ends is communicatively connected with a plurality of clients; The heterogeneous federated learning method includes: The server receives local encoded data sent by each of the clients and obtained by encoding based on local data, performs data-free global distillation according to each of the local encoded data to obtain a first global model, and distributes the first global model to each of the clients; The client performs partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment optimization, generates a local model based on the partial parameter local distillation result and the non-real time distillation result, and sends the local model to the server; The server updates the first global model through a data-free global distillation strategy based on each of the local models to obtain a second global model, and distributes the second global model to each of the clients for the next round of federated learning training.

2. The heterogeneous federated learning method based on spatiotemporal data distillation according to claim 1, characterized in that: The process of receiving, in the server side, local encoded data obtained by encoding based on local data sent by each of the clients, and performing data-free global distillation according to each of the local encoded data to obtain a first global model includes: Receiving local encoded data sent by each of the clients after encoding based on local data; Using a generator network constructed based on a meta-learning method to train each of the local encoding data to learn the common features of each of the local encoding data; Based on the common feature learning results, outputting a virtual meta-learning global dataset through a trained generator network, wherein the meta-learning global dataset includes a plurality of common representation pseudo samples; A global model is initialized as a student model, the trained generator network is used as a teacher model, and the multiple common representation pseudo samples are used as outputs of the teacher model for distillation. The initialized global model is optimized by minimizing the difference between the outputs of the student model and the teacher model to obtain a first global model.

3. The heterogeneous federated learning method based on spatiotemporal data distillation according to claim 2 is characterized in that: In each of the clients, according to the local data and the first global model, partial parameter local distillation and non-real time distillation are performed in combination with dynamic temperature adjustment optimization, and a local model is generated based on the partial parameter local distillation results and the non-real time distillation results, including: Obtaining a meta-learning global dataset from the server; According to the local data, the meta-learning global data set and the first global model, local distillation of some parameters is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a partially distilled local sub-model; According to the meta-learning global dataset and the first global model, non-real time distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a non-real distillation local sub-model; The partially distilled local sub-model is added to the non-true distilled local sub-model to obtain a local model.

4. The heterogeneous federated learning method based on spatiotemporal data distillation according to claim 3 is characterized in that: The method of performing local distillation of some parameters according to the local data, the meta-learning global data set and the first global model in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a partially distilled local sub-model includes: Using the first global model as a local model to be trained of the client, and dividing it into a feature extraction module and a prediction module; Obtaining a prediction copy module of the prediction module by copying, and constructing a feature extraction local model according to the prediction copy module and the feature extraction module; Inputting the local data into the feature extraction local model, calculating the cross entropy loss between the model prediction result and the true label, and updating the feature extraction module in the feature extraction local model while keeping the prediction replication module unchanged by optimizing the cross entropy loss to obtain a feature extraction update module; Obtain a feature extraction replication module of the feature extraction module by replication, and construct a prediction local model according to the feature extraction replication module and the prediction module; The meta-learning global data set is input into the prediction local model and the first global model respectively, knowledge distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism, and the knowledge distillation loss is solved. By optimizing the knowledge distillation loss, the prediction module in the prediction local model is updated while keeping the feature extraction replication module unchanged, so as to obtain a prediction update module; The feature extraction update module and the prediction update module are integrated into the first global model to obtain a partially distilled local sub-model.

5. The heterogeneous federated learning method based on spatiotemporal data distillation according to claim 3 is characterized in that: The non-real time distillation is performed according to the meta-learning global data set and the first global model in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to obtain a non-real distillation local sub-model, including: Using the first global model as the local model to be trained of the client, and calculating the performance loss of the local model to be trained on the meta-learning global dataset; Using the local model to be trained to perform prediction based on the memory replay mechanism, and calculating the local retention loss; Based on the performance loss and the local preservation loss, and taking into account the knowledge forgetting metric, a non-true time total loss is constructed; Taking minimizing the non-true total time loss as the optimization goal, dual-objective knowledge distillation is performed in combination with the optimal distillation temperature obtained based on the dynamic temperature adjustment mechanism to optimize the model parameters of the local model to be trained and obtain a non-true distillation local sub-model.

6. The heterogeneous federated learning method based on spatiotemporal data distillation according to any one of claims 3 to 5, characterized in that: The process of obtaining the optimal distillation temperature based on the dynamic temperature adjustment mechanism includes: Based on the specified task loss and knowledge distillation loss of the student model of the client during the knowledge distillation process, construct a total loss function of the student model; Temperature prediction training is performed by adversarially optimizing the optimization path of the student model. During the training process, the difficulty of the knowledge distillation task is adjusted by a scaling factor, and minimizing the total loss of the student model is used as the optimization goal. The total loss function of the student model is solved to obtain the predicted distillation temperature. A nonlinear activation process is performed according to the predicted distillation temperature to obtain an actual distillation temperature, and the actual distillation temperature is used as an optimal distillation temperature for a partial parameter local distillation process and a non-real time distillation process.

7. The heterogeneous federated learning method based on spatiotemporal data distillation according to claim 2, characterized in that: The process of updating the first global model by the data-free global distillation strategy according to each of the local models in the server to obtain the second global model includes: Using the first global model as a student model and each of the local models as a teacher model; Comprehensively considering the cross entropy loss and knowledge distillation loss of the server in the knowledge distillation process, constructing a dual-loss global function to measure the difference between the prediction accuracy of the first global model on the meta-learning global dataset and the sum of the outputs of each of the local models; Taking minimizing the dual-loss global function as the optimization goal, the first global model is updated through knowledge distillation to obtain a second global model.

8. A heterogeneous federated learning device based on spatiotemporal data distillation, characterized in that: Applied to a heterogeneous federated learning system, the heterogeneous federated learning system includes at least one server end, each of the server ends is communicatively connected with a plurality of clients; The heterogeneous federated learning device includes: A global model generating unit, configured to receive, at the server end, local encoded data obtained by encoding based on local data sent by each of the clients, perform data-free global distillation according to each of the local encoded data to obtain a first global model, and distribute the first global model to each of the clients; A local model generating unit, configured for the client to perform partial parameter local distillation and non-real time distillation according to the local data and the first global model in combination with dynamic temperature adjustment optimization, generate a local model based on the partial parameter local distillation result and the non-real time distillation result, and send the local model to the server; A global model updating unit is used for the server to update the first global model according to each of the local models through a data-free global distillation strategy to obtain a second global model, and distribute the second global model to each of the clients for the next round of federated learning training.

9. An electronic device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the heterogeneous federated learning method based on spatiotemporal data distillation according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the heterogeneous federated learning method based on spatiotemporal data distillation as described in any one of claims 1-7.

Images