A federated mutual learning model training method for non-independent and identically distributed data

Through the combination of group mutual learning and distillation technology, the problem of federated learning communication bandwidth limitation and low model generation efficiency under non-independent and same-distributed data is solved, and efficient model training and global model generation are realized, suitable for heterogeneous data environments .

CN114091667BActive Publication Date: 2024-11-19BEIJING INST OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111386087.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-22
Publication Date
2024-11-19
Estimated Expiration
2041-11-22

AI Technical Summary

Technical Problem

In a non-independent and homogeneous data environment, federated learning faces the problems of communication bandwidth limitations and low model generation efficiency, and existing methods are difficult to effectively reduce communication costs and improve training performance.

Method used

Using a combination of group mutual learning and distillation technology, the initialized global model parameters are sent through the server to the intermediate client. The intermediate client generates and transmits them to the edge client. The edge client updates the model parameters locally, the intermediate client and the edge client. The terminal conducts multiple rounds of training, uses KL divergence and distillation learning methods to update the model, reduce the number of direct interactions between the server and the client, only transmits the model output, and reduces bandwidth usage.

Benefits of technology

The effect of federated learning on heterogeneous data is improved, the model training time is shortened, the upstream and downstream bandwidth asymmetry problem is solved, global and local personalized models are generated, and communication costs are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114091667B_ABST
    Figure CN114091667B_ABST
Patent Text Reader

Abstract

The present invention provides a federated mutual learning model training method for non-independent and identically distributed data, including: S1, sending initial global model parameters to an intermediate client, the intermediate client generates intermediate client model parameters, S2, the edge client generates edge client model parameters using a local data set; S3, the intermediate client and the edge client update parameters using a mutual learning method; S4, uploading the probability prediction value output by the intermediate client model to a server, the server updates the global model and the intermediate client model using a distillation technique; S5, executing steps S3 to S4 until the model meets the convergence condition, obtaining the final intermediate client model, edge client model and global model, and then the server broadcasts the final global model to all edge clients. In the present invention, the problems of federated learning communication bandwidth limitation and model generation of non-independent and identically distributed data are solved by group mutual learning and knowledge distillation methods.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning, and in particular to a federated mutual learning model training method for non-independent and identically distributed data. Background Art

[0002] With the development of artificial intelligence and big data, information resources can be transmitted at high speed in distributed networks, realizing the global management of physical information. Cloud computing and edge computing have promoted the development of deep learning. Deep learning models usually contain millions of parameters. Generally, expanding the scale of neural networks can effectively improve the accuracy of models. However, in distributed networks, resource-constrained edge devices cannot deploy large neural network models, which will lead to communication delays and denial of access to low-priority user terminals. At this time, the powerful computing power and storage space of the central server are needed. In real life, data collection, exchange and ownership between enterprises are strictly regulated by laws and regulations. It is very difficult to obtain large amounts of high-quality training data, and the problem of privacy leakage is becoming increasingly prominent.

[0003] Federated learning is a distributed framework that decouples data and models. It can solve the problems of data silos and privacy protection. The entire process does not require the data of the participants to be centralized in a central storage point. The joint modeling of all participants can be achieved without leaving the local data. In the federated learning framework, there is generally a central computing party that is responsible for collecting model parameter information transmitted by other parties, and after updating the corresponding algorithm, returning tasks to all parties, iterating repeatedly until convergence, and finally building an effective global model. During the entire process, neither the client nor the server has the right to obtain and control the data of other clients. The normal use of client devices is not affected during the construction of the federated learning model. The trained federated learning model can be shared and deployed among all data participants, and has broad application prospects in the fields of smart healthcare, financial insurance, and smart Internet of Things.

[0004] In real-world scenarios, data distribution across devices may be statistically heterogeneous, including situations such as skewed label assignments, skewed feature distributions, and skewed quantities. This paper mainly studies how federated learning can be trained on non-independent and identically distributed data. Traditional federated averaging algorithms are trained on non-independent and identically distributed data and are easily affected by discrete values, which introduces large variances and results in suboptimal global models. For example, data heterogeneity can lead to slow convergence of the global model, increase the number of communications between the server and the client, and the number of training rounds on the client side. Federated learning may be several orders of magnitude slower than traditional distributed systems in terms of network bandwidth. Bandwidth limitations can cause client connections to fail or be interrupted, and clients with limited network access or failed connections will be discarded during the communication process. The number of nodes may be several orders of magnitude more than that of a distributed system. Deploying a large number of users places more stringent requirements on latency and computing resources. As the number of communication rounds and the number of users increase, the communication overhead problem becomes more prominent. For the above issues, relevant research has been conducted from a variety of perspectives:

[0005] To address the communication overhead issue, cloud servers can access more data, but the communication overhead is too high and the delay is long. Compared with cloud-based federated learning, client-edge-cloud hierarchical federated learning can reduce direct communication between clients and cloud servers by introducing intermediate edge servers, shorten model training time and reduce energy consumption of terminal devices. However, the number of clients and the amount of data accessed by edge servers are limited, and heterogeneity between devices and data will bring more obvious communication problems. The researchers proposed a model training method for multi-center aggregated federated learning, attempting to cluster clients with similar tasks into multiple centers for training, thereby accelerating the convergence speed of each center and improving the accuracy of the model in each center. By increasing the client's computing power and reducing the number of communication exchanges between the server and the client, it is more conducive to rapid model generation. Methods such as compressing the model parameter scale and pruning the network are used to reduce the exchange of a large number of model parameters during the communication process. Knowledge distillation technology uses a large and complex teacher network to help a small and simple student network train the model. The student network can not only learn the information of the real label, but also learn the relationship information of different labels. The small model is easy to deploy and is used to solve the bandwidth limitation problem of federated learning.

[0006] In response to the problem of data heterogeneity, traditional integration methods and federated average algorithms are easily affected by outliers. In the case of cross-device heterogeneous data, the model will be offset during the training process, and the final global model cannot improve the training model of all participants. Researchers have proposed sharing some global data subsets to reduce the degree of difference in heterogeneous data across devices, but there is a hidden danger of privacy leakage. In order to reduce the risk of privacy leakage, a large amount of redundant data will be added to the public data set and pseudo samples will be generated using adversarial networks; knowledge distillation and generative adversarial network technology will be combined to complete the missing samples of each device in other devices, and the data distribution of each client will be converted into independent and identically distributed data to construct an effective global model, but it will increase the communication cost; some scholars also believe that clients with similar tasks can be trained together to generate multiple models for different tasks.

[0007] The above mentioned methods explore the federated learning of non-IID data, but the following problems remain unsolved:

[0008] 1. Neural networks generate a large number of model parameters. Bandwidth limitations affect parameter transmission and model training. Compared with cloud-based federated learning, the client-edge-cloud hierarchical federated learning system introduces an intermediate edge server, which can reduce direct communication between the client and the cloud server and reduce model training time. However, a large number of parameters still need to be transmitted, and there is still a lack of solutions that can reduce communication costs.

[0009] 2. In recent years, knowledge distillation technology has been combined with federated learning. However, when processing non-independent and identically distributed data, it is necessary to pre-generate a teacher model on a suitable proxy dataset, but it is difficult to find a suitable dataset to generate a teacher model. At present, there is still a lack of federated learning methods that can solve the high communication cost and training performance loss encountered when processing non-independent and identically distributed data. Summary of the invention

[0010] In view of the shortcomings of the prior art, in order to solve the communication bandwidth limitation and model generation problems encountered by federated learning when training non-independent and identically distributed data, the present invention constructs a model training method for federated mutual learning of heterogeneous data, and adopts a method combining group mutual learning and distillation technology. The group mutual learning method can reduce the requirements for pre-generated teacher models, and the model outputs are exchanged between student networks to avoid transmitting a large number of parameters, reduce the number of direct interactions between the server and the client, improve the effect of federated learning on training models of heterogeneous data and solve the problem of asymmetric upstream and downstream bandwidth speeds. It generates both global models and local personalized models, which can be selected by users in practical applications.

[0011] To achieve the above object, the present invention is implemented through the following technical solutions:

[0012] A federated mutual learning model training method for non-independent and identically distributed data, comprising:

[0013] S1. The server sends the initialized global model parameters to the intermediate clients of each group. The intermediate clients generate the intermediate client model parameters using the intermediate client data set and send the generated intermediate client model parameters to the edge clients in the group.

[0014] S2, the edge client receives the intermediate client model parameters, and the edge client generates the edge client model parameters using the local data set;

[0015] S3, the intermediate clients and edge clients in the group use the mutual learning method to perform multiple rounds of training to update the intermediate client model parameters and the edge client model parameters;

[0016] S4, the intermediate clients of all groups upload the label category probability prediction values ​​of the intermediate client models to the server and update the global model parameters;

[0017] S5. Repeat steps S3-S4 until the convergence condition is met, and the intermediate client model, edge client model and global model are obtained. The server broadcasts the final global model generated by training to all edge clients.

[0018] Furthermore, the global model, the intermediate client model, and the edge client model are neural network models.

[0019] Furthermore, in step S2, the intermediate client and the edge client respectively update the model parameters using the stochastic gradient descent algorithm on the local data set.

[0020] Furthermore, in step S3, the mutual learning method includes:

[0021] S31. All edge clients in the group are regarded as student networks. The output of all edge client models in round t is C is the number of edge clients, and the probability prediction value of the label category is calculated and transmitted to the intermediate client; i represents the i-th intermediate client, and j represents the j-th edge client connected to the i-th intermediate client;

[0022] S32, calculation of the intermediate client of group i c represents the edge client, represents the average value of the probability prediction value of the calculated label category, and the KL divergence D between the intermediate client model of the i-th group and the student network of the edge client model within the group KL , the formula is: m represents the intermediate client, represents the probability prediction value of the label category of the intermediate client in the tth round of the i-th group. The loss function of the intermediate client model in the i-th group is

[0023] S33, the intermediate client of the i-th group updates the intermediate client model parameters using the stochastic gradient descent algorithm on the intermediate client data set

[0024] S34, the j-th edge client of the i-th group calculates the KL divergence D KL And the loss function The formula is:

[0025] S35. The edge client updates the edge client model parameters using the stochastic gradient descent algorithm on the edge client data set.

[0026] S36: After executing N rounds of S31-S35, all intermediate clients calculate the probability prediction value of the label category and transmitted to the server.

[0027] Furthermore, a temperature parameter T is added to the Softmax function of the edge client model to adjust the probability distribution of the output. The edge client calculates the probability prediction value of the corresponding label category.

[0028] Furthermore, the server receives the probability prediction value of the corresponding label category from the intermediate client model Output of the intermediate client i represents the i-th intermediate client, m represents the type of intermediate client, and the distillation learning method is used to update the global model. The distillation learning method includes:

[0029] S41. The server calculates the loss function of the global model z represents the number of interaction rounds between the server and the intermediate client, and the formula is:

[0030]

[0031] S42. Update the global model parameters using the gradient descent algorithm on the local data set of the server: z is the number of rounds of mutual learning between the intermediate client and the edge client;

[0032] S43, each group of intermediate clients calculates the KL divergence and loss function respectively, the formula is:

[0033] In the present invention, the problem of insufficient bandwidth utilization is solved, and the model time of federated learning training of non-independent and identically distributed data is shortened. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 It is a schematic diagram of the grouping of intermediate clients and edge clients;

[0036] Figure 2 is a flowchart of a federated mutual learning model training method according to an embodiment of the present invention;

[0037] Figure 3 is a flow chart of a mutual learning method according to an embodiment of the present invention;

[0038] Figure 4 Schematic diagram of a distillation method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The method provided by the present invention can be executed in a mobile terminal, a computer terminal or a similar computing device, and each participant has a number of training data that they hope to use to train the model. Figure 1 As shown, the intermediate clients are grouped, each group includes an intermediate client and multiple edge clients, the edge client can be multiple devices, and the intermediate client can access a certain number of edge clients. The present invention proposes a federated mutual learning model training method for non-independent and identically distributed data, such as Figure 2 As shown, including:

[0041] S1. The server establishes communication with the intermediate client. The server sends the initialized global model parameters to the intermediate client. The intermediate client model parameters are generated on the local data set of the intermediate client and sent to the edge client.

[0042] S2. After grouping, the intermediate clients of each group establish communication with the edge clients in the group, the edge clients receive the intermediate client model parameters, and the edge clients generate edge client model parameters on the local data set;

[0043] S3, the intermediate client and the edge client use a mutual learning algorithm to update the intermediate client model parameters and the edge client model parameters;

[0044] S4, the intermediate client uploads the probability prediction value of the label category of the intermediate client model to the server, and the server updates the global model using the distillation technology;

[0045] S5. Repeat steps S3-S4 until the convergence condition is met, and the intermediate client model, edge client model and global model are obtained. Then the server broadcasts the final global model to all edge clients.

[0046] exist Figure 1 In the figure, the server is represented by s, the intermediate client is represented by m, and the edge client is represented by c. The global model, the intermediate client model, and the edge client model are all neural network models.

[0047] In step S1, the server seeks client participation according to the node (intermediate client and edge client) status and sends modeling tasks to the intermediate client and edge client. 0 represents offline and 1 represents online. Intermediate client m,i∈{1,…,M}, edge client c,j∈{1,…,C}, when Class i=1, Class i j = 1, the intermediate client m and the edge client c belong to the same group. Global model parameters When z = 0, is the initial global model, the intermediate client model parameters represents the model parameters updated by the tth intermediate client, the edge client model parameters represents the model parameters updated at the tth time by the jth edge client in the i-th group.

[0048] The intermediate client and edge client feedback the joint modeling response according to their own needs. The data sets of edge clients and intermediate clients in each group meet non-independent and identically distributed requirements. For example, the edge clients in each group only contain one or more disjoint data label classes. Sent to the intermediate client.

[0049] Taking the i-th group as an example, the intermediate client m∈{1,…,M} receives the global model parameters After that, Initialized as intermediate client parameters, with Indicates that in the local dataset (where x i is the data sample feature, y i is the true label of the sample, n is the number of samples) uses the stochastic gradient descent algorithm to update the intermediate client model parameters The intermediate client model parameters are passed to the edge clients c∈{1,…,C} within the group.

[0050] In step S2, the edge client receives the global model parameters After that, initialize the edge client model parameters Edge clients in local datasets (where x i is the data sample feature, y i is the true label of the sample, n is the number of samples, and the sample types between edge clients are disjoint and satisfy non-independent and identically distributed) and the stochastic gradient descent algorithm is used to generate the edge client model parameters Where η is the learning rate, Represents the first update gradient of the jth edge client in the i-th group.

[0051] In step S3, the mutual learning algorithm is used to update the model parameters of the intermediate client and the edge client. Figure 3 As shown, the specific description of the mutual learning algorithm is:

[0052] S31. All edge clients c in the group (intermediate clients and corresponding edge clients are a group) are regarded as a student network (student network is a term in mutual learning), which is marked as i here. j , represents the j edge clients in the i-th group, the output of the edge client model Calculate the probability prediction value of the label category The soft label of the output is transformed by adding the temperature parameter T in the Softmax function (the activation function of the output layer of the edge client model) and transmitted to the intermediate client;

[0053] S32, calculation on the intermediate client KL divergence D between the intermediate client model and the edge client student network KL , used to measure the degree of difference between data distributions, the formula is:

[0054] The loss function of the intermediate client model is

[0055] S33, the intermediate client updates the intermediate client model parameters using the stochastic gradient descent algorithm on the local data set

[0056] S34, edge client calculates divergence D KL , the loss function of the edge client model is calculated on the edge client, and the formula is:

[0057] S35. The edge client updates the edge client model parameters using the stochastic gradient descent algorithm on the edge client data set.

[0058] S36, steps S31-S35 are a mutual learning update process between the intermediate client and the edge client. After N updates (N is manually set), the output of the intermediate client is updated. Transfer to the server.

[0059] In step S4, the server receives the probability prediction value of the label category from the intermediate client model Use distillation learning algorithm to update the global model, such as Figure 4 As shown, the distillation learning method includes:

[0060] S41. The server calculates the global model loss function The formula is:

[0061]

[0062] S42, using the gradient descent algorithm on the local data set of the server to update the global model parameters

[0063] S43, the intermediate client calculates the intermediate client model loss function on the local data set:

[0064] In step S5, steps S3 to S4 are repeated until the global model, the intermediate client model, and the edge client model meet the convergence conditions, and finally the intermediate client model parameters are generated. Edge Client Model Parameters and global model parameters The global model parameters Distribute to all edge clients.

[0065] Example 1

[0066] According to the method steps provided in this embodiment, a model training method for solving non-independent and identically distributed federated learning is introduced, taking the MNIST image classification task as an example. Each client is a data owner, and at least one terminal participates in the training, as well as the intermediate client. The sample data held by each data owner can be the same data set or a different data set. The server and client databases store the MNIST data set. After manual segmentation, the edge clients in each group only contain one or more non-overlapping data labels, satisfying the non-independent and identically distributed situation. The server, intermediate client and edge client use the LeNet-5 convolutional neural network (input layer dimension 28×28, convolution layer includes 6 5×5 convolution kernels, maximum pooling layer includes 1 2×2 kernel, convolution layer includes 16 5×5 convolution kernels, maximum pooling layer includes 1 2×2 kernel, convolution layer includes 120 5×5 convolution kernels, fully connected layer includes 84 neurons, output layer dimension 1×10), the input vector is a 28X28 pixel image converted into a 10-dimensional vector output by LeNet-5, the predicted maximum probability value of the output layer is the corresponding label value, for example [0.1028, -0.0138, -0.0664, -0.0575, -0.0387, 0.0269, 0.1911, 0.0022, -0.0441, -0.0797], indicating that the soft label prediction result is 6. The intermediate client, edge client and server all use this neural network. The specific steps are as follows:

[0067] In step S1, the server is represented by S and the intermediate client m i ,i=1,2, edge client uses c i,i=1,2,3,4, the intermediate clients and edge clients are grouped, the first group includes m1,c1,c2, and the second group includes m2,c3,c4. The server seeks client participation according to the node status and sends modeling tasks to the edge clients. The dataset of edge client c1 contains 1000 images with labels 1 and 2, the dataset of edge client c2 contains 500 images with labels 3 and 4, the dataset of edge client c3 contains 1000 images with labels 5 and 6, the dataset of edge client c4 contains 2000 images with labels 7 and 8, the dataset of intermediate client m1 contains 1000 images with labels 1, 3, 5, the dataset of intermediate client m2 contains 1000 images with labels 2, 4, 6, and the dataset of server S contains 2000 images with labels 1 and 9. The server generates the initial model The server will initialize the model parameters Transmitted to the intermediate client, taking the first group as an example, the intermediate client generates a model on the local dataset

[0068] In step S2, taking m1, c1, c2 as an example, the intermediate client model parameters Transmitted to edge clients c1, c2, the edge clients generate local models based on the local data set

[0069] In step S3, the output of edge clients c1, c2 is Calculate the probability prediction value of the label category Transmitted to the intermediate client m1, the intermediate client calculates And the loss function Update intermediate client model parameters The loss function of each edge client model is calculated as Here I j =1,2, this is a mutual learning update process between the intermediate clients and edge clients in the group. After N rounds of updates, the probability prediction values ​​of the label categories of all intermediate clients are Transfer to the server.

[0070] In step S4, the server receives the probability prediction value of the tag category Using the distillation learning algorithm, the server calculates the server model loss function as The intermediate client calculates the model loss function as:

[0071] In step S5, after S3-S4 are repeatedly executed and convergence conditions are met, the final global model is generated on the server. The final global model generated by training is broadcast to all edge clients.

[0072] The above steps are the whole process of the embodiment. The whole process can generate an edge client model, an intermediate client model and a global client model. When the edge client generation accuracy is low, the global model can be selected. When the edge client local model accuracy is high, the local model is selected.

[0073] The present invention proposes a federated mutual learning model training method for non-independent and identically distributed data, which reduces the direct communication between edge clients and servers during the training process. Although the number of edge clients accessed by intermediate clients is limited, the communication efficiency is higher. Compared with traditional methods, the more complex the network, the more parameters are generated. Parameters refer to the weight values ​​on the neural network nodes at each moment. The sum of parameters is the number of nodes multiplied by the number of parameters on the nodes. Transmitting a large number of parameters during the training process will occupy a large amount of bandwidth. Compared with the mutual learning method, which is independent of the complexity of the model structure, only the output of the model is transmitted during the training process, which reduces the transmission of a large number of model parameters, makes full use of the upstream and downstream bandwidths between clients, and weakens the influence of bandwidth limitation on federated learning training. Edge clients use distillation algorithms to jointly train the global model, and can dynamically adjust the parameter changes of each group of intermediate clients caused by non-independent and identically distributed data, thereby solving the influence of non-independent and identically distributed data on federated learning model training.

[0074] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A federated mutual learning model training method for non-independent and identically distributed data, characterized in that: The method comprises: S1. The server sends the initialized global model parameters to the intermediate clients of each group. The intermediate clients generate the intermediate client model parameters using the intermediate client data set and send the generated intermediate client model parameters to the edge clients in the group. S2, the edge client receives the intermediate client model parameters, and the edge client generates the edge client model parameters using the local data set; S3, the intermediate clients and edge clients in the group use the mutual learning method to perform multiple rounds of training to update the intermediate client model parameters and the edge client model parameters; S4, the intermediate clients of all groups upload the label category probability prediction values ​​of the intermediate client models to the server and update the global model parameters; S5. Repeat steps S3-S4 until the convergence condition is met, and the intermediate client model, the edge client model and the global model are obtained. The server broadcasts the generated final global model to all edge clients. In step S3, the mutual learning method includes: S31. Record the output of all edge client models in the group at round t as C is the number of edge clients, and the probability prediction value of the label category is calculated and transmitted to the intermediate client; i represents the i-th intermediate client, and j represents the j-th edge client connected to the i-th intermediate client; S32, calculation of the intermediate client of group i c represents the edge client, and the KL divergence D between the intermediate client model of the i-th group and the edge client model within the group is calculated. KL and the loss function of the intermediate client model of group i S33, the intermediate client of the i-th group updates the intermediate client model parameters using the stochastic gradient descent algorithm on the intermediate client data set S34, the j-th edge client of the i-th group calculates the KL divergence D KL And the loss function S35. The edge client updates the edge client model parameters using the stochastic gradient descent algorithm on the edge client data set. S36: After executing N rounds of S31-S35, all intermediate clients calculate the probability prediction value of the label category and transmit to the server; In step S32, the divergence formula is: m represents the intermediate client, represents the probability prediction value of the label category of the intermediate client in the tth round of the i-th group. The loss function of the intermediate client model in the i-th group is In step S34, the loss function is calculated as: The temperature parameter T is added to the Softmax function of the edge client model to adjust the probability distribution of the output. The edge client calculates the probability prediction value of the corresponding label category 2. The model training method according to claim 1, characterized in that: The global model, intermediate client model, and edge client model are neural network models.

3. The model training method according to claim 1, characterized in that: In step S2, the intermediate client and the edge client respectively update the model parameters using the stochastic gradient descent algorithm on the local data set.

4. The model training method according to claim 1, characterized in that: The server receives the probability prediction value of the corresponding label category from the intermediate client model Output of the intermediate client i represents the i-th intermediate client, m represents the type of intermediate client, and the distillation learning method is used to update the global model. The distillation learning method includes: S41. The server calculates the loss function of the global model z represents the number of interaction rounds between the server and the intermediate client, and the formula is: S42. Update the global model parameters using the gradient descent algorithm on the local data set of the server: z is the number of rounds of mutual learning between the intermediate client and the edge client; S43, each group of intermediate clients calculates the KL divergence and loss function respectively, the formula is: