Packet integrated distillation federated learning method in edge network

Abstract

The application discloses a packet integrated distillation federated learning method in an edge network and relates to the technical field of machine learning, and comprises the following steps: grouping determined Client equipment, each group comprising a Leader equipment and a plurality of attender equipment; performing N rounds of integrated distillation federated training on a global model for each group of Client equipment; the Leader equipment in each group of Client equipment returns the obtained global model parameters to an edge server; the edge server aggregates all the received global model parameters, then tests the aggregated global model parameters, and if the obtained federated learning Loss data converges, the federated learning ends, otherwise, the aggregated global model parameters are sent to the Leader equipment, and the Leader equipment updates the locally most recently trained global model according to the aggregated global model parameters. The application sets different model training for end equipment with different computing resources, then uses an integrated distillation method to perform knowledge migration, improves the accuracy of the model, and avoids complex model training of the server.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to a grouping integrated distillation federated learning method in an edge network. BACKGROUND

[0002] With the improvement of mobile device performance and the development of edge network, a large number of computing resources are configured at the edge, and a centralized data center is no longer necessary. Model learning and data training can be completed in a distributed manner, and the network burden caused by uploading a large number of original data is avoided. Moreover, it has the advantage of being able to utilize real data of a large area of user clusters to promote large-scale data training and model learning. However, distributed learning in edge computing faces the problems of data security and privacy protection. The privacy protection feature and distributed model computing capability of federated learning are exactly adapted to the edge network scenario. The application of federated learning to the edge network has been practiced in transportation, shopping and medical treatment, etc., and has a wide research and application prospect.

[0003] Federated learning is a distributed machine learning method that can independently complete model training on each device without collecting user data, and then encrypt the gradient data for privacy protection and transmit it to the center node server (federal center). Finally, the center node updates the gradient (global model) according to the aggregation result and returns it to each device to complete the gradient and model update of each device, solving the problem of user data island. Currently, many fields of machine learning have introduced federated learning, such as federated transfer learning, federated reinforcement learning, and federated secure tree. Ordinary federated learning algorithms such as FedAvg or FedSGD use local SGD updates for training steps by all agents and build a global model based on the aggregation of agent models without sharing any private data. Although federated learning can avoid many privacy issues and economic costs brought by traditional centralized machine learning and data collection, it also brings a lot of open problems and challenges. Especially in the case of limited resources of edge devices in edge computing environment, federated network not only faces great challenges in data heterogeneity, but also faces great challenges in device heterogeneity and model heterogeneity. And because the model aggregated at the server side in the current federated learning process usually needs to maintain the same structure as the model distributedly trained at the client side, but generally speaking, the number of devices that can have enough computing resources to train the large model required by the server side is always small, so it is possible that the number of clients participating in federated learning is not enough, resulting in insufficient training parties and training data for federated learning, and thus not achieving good results. The global model cannot meet the needs of most users in the cluster. However, if the server side compromises to set a smaller model as the global model for training in order to obtain more participants and improve data quantity, it also cannot achieve the desired high accuracy effect. In view of this dilemma faced by federated learning, the method of knowledge distillation is used to solve these problems.

[0004] Hinton et al. proposed knowledge distillation, which uses the soft targets of the teacher network as part of the loss function to induce the training of the student network and achieve knowledge transfer. Yim et al. use a matrix to characterize the feature relationship between layers, then use the L2 loss function to reduce the difference between the teacher model and the student model, and let the student model learn this method, rather than just using the target loss function for knowledge transfer. Heo et al. use an adversarial attack strategy to convert benchmark class samples to target class samples, and the adversarial generated samples induce the training of the student network, thereby effectively improving the student network's ability to identify decision boundaries. However, hard labels can cause the model to overfit, so Yang et al. proposed a more reasonable method, which did not calculate the additional loss of all classes, but selected a few classes with the highest confidence score to soften the label and improve the model's generalization performance.

[0005] In recent years, some scholars have combined federated learning and knowledge distillation to solve the problems of data heterogeneity, device heterogeneity, and model heterogeneity in federated learning. In terms of data heterogeneity, Jeong et al. proposed a distributed model federated distillation training algorithm that can generate data using a generative adversarial network to solve the problem of heterogeneous data samples generated by users. In terms of model heterogeneity, Li et al. proposed a device personalized model design scheme, FedMD, and used knowledge distillation technology for transfer learning to migrate the knowledge learned by local models to the global model. Lin et al. also used knowledge distillation technology to propose a federated learning scheme, FedDF, that can fuse multiple models to solve the problem of model heterogeneity, and also verified the effectiveness of the method in the case of data heterogeneity.

[0006] Currently, some people have introduced ensemble distillation into federated learning methods, such as a two-level federated learning mechanism with knowledge distillation. In this method, the server initializes a global model and distributes a public dataset to each participant. Then, the server distributes a small model that meets the computing capacity of each participant. Each participant then trains the local model using the public dataset and their own private dataset. The server then performs knowledge distillation on the server side using the ensemble distillation method to migrate the knowledge learned by the participants to the global model. The server returns the distilled loss to each participant for the next round of learning. In this method, the server needs to participate in training the model like the user side. Since the Logits are calculated on the server side, and the Logits contain the knowledge learned by the user models, integrating many user-collected Logits will consume a lot of server resources for calculation.

[0007] Existing literature also proposes a federated learning method for heterogeneous models (FedMD), which uses knowledge distillation technology as a "translator" to reach a consensus on the knowledge in each personalized model. In this method, each participant in federated learning, including the server, can initialize their own personalized model. Then, they train the model using the public dataset and their own private dataset. The server then performs knowledge distillation using the ensemble distillation method to migrate the knowledge. The server calculates the consensus loss function and distributes it to all participants. The participants then update their local models based on the loss function. In this way, each participant can obtain a personalized model, and the server can calculate a global model. In this method, the server also needs to perform complex ensemble distillation and model calculation processes, which will consume a lot of server resources.

[0008] Although the two solutions can solve the problem that the participants in the edge environment cannot participate in federated learning due to insufficient computing power to a certain extent, resources are very scarce in the edge environment, whether on the server side or on the device side, so concentrating such a large amount of calculation on the server will affect the service capability of the edge computing server and also reduce the running efficiency of the whole system.

[0009] It can be seen that the prior art still has the following defects:

[0010] (1) The edge server in the federated learning with knowledge distillation increases the large amount of calculation and the problem of excessive resource consumption;

[0011] (2) The edge device does not have enough resources to train a large neural network, resulting in insufficient participating users in federated learning and insufficient data for federated learning to participate in training, so that a global model with appropriate and sufficient precision cannot be obtained. SUMMARY

[0012] The present application provides a grouping integrated distillation federated learning method in an edge network, which can alleviate the above problems.

[0013] In order to alleviate the above problems, the technical scheme adopted by the present application is as follows:

[0014] The present application provides a grouping integrated distillation federated learning method in an edge network, comprising the following steps:

[0015] S1, the edge server determines a plurality of Client devices that need to participate in federated learning;

[0016] S2, the edge server divides all the Client devices into a plurality of groups according to the computing resources, each group including a Leader device and a plurality of attender devices, and the Leader device has sufficient computing resources to calculate the global model;

[0017] S3, the edge server initializes a global model for the Leader device to train locally, and a small model for the attender device to train, the small model meeting the computing capacity of all the Client devices;

[0018] S4, for each group of Client devices, the global model is trained by N rounds of integrated distillation federated training;

[0019] S5, the Leader device in each group of Client devices returns the global model parameters obtained by training to the edge server;

[0020] S6, the edge server aggregates all the global model parameters received for processing;

[0021] S7, the edge server tests the global model parameters after aggregation processing to obtain federated learning Loss data;

[0022] S8, if the federated learning Loss data converges, the federated learning ends, otherwise the global model parameters after aggregation processing are issued to the Leader device, the Leader device updates the locally last trained global model according to the global model parameters after aggregation processing, and then jumps to step S4.

[0023] In a preferred embodiment of the present application, in step S3, the edge server needs to first issue the global model and the small model to the Leader device of each group, and the Leader device of each group broadcasts the small model to the attender device in the group.

[0024] In a preferred embodiment of the present application, in step S4, each round of integrated distillation federated training specifically includes the following steps:

[0025] S41, the Leader device trains the global model according to the local Loss data and the local picture data, and obtains a first set of labels corresponding to a plurality of training categories, and the attender device trains the small model according to the local Loss data and the local picture data, and obtains a second set of labels corresponding to a plurality of training categories;

[0026] S42, the Leader device calculates the first group of Soft targets through the Softmax function according to the first set of labels corresponding to a plurality of training categories, and the attender device calculates the second group of Soft targets through the Softmax function according to the second set of labels, and sends them to the Leader device;

[0027] S43, the Leader device integrates all the Soft targets at present, and forms a plurality of matrices according to categories;

[0028] S44, the Leader device performs weighted linear combination on the matrix of each category, and then selects the Soft target with the smallest loss as the output value z t of the category;

[0029] S45, the Leader device returns the output value z t of each category to the attender device;

[0030] S46, each Client device updates the local model according to the output value z tAnd the local calculation Softtarget, the local Loss data is calculated and updated.

[0031] Compared with the prior art, the present application has the beneficial effects that:

[0032] The method can integrate knowledge in multiple models, so that edge devices with insufficient computing resources can train a small model that matches their computing and storage resources while migrating knowledge to the global model.

[0033] The experimental results show that the method of setting different models for end devices with different computing resources, then training and using integrated distillation to migrate knowledge improves the accuracy of the model, while avoiding complex model training on the server.

[0034] To make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the following embodiments of the present application are described in detail below, and the accompanying drawings are described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, it should be understood that the following drawings only show some embodiments of the present application, therefore should not be regarded as limiting the scope, for those skilled in the art, without creative labor, other related drawings can also be obtained from these drawings.

[0036] Figure 1 is a flowchart of the grouping integrated distillation federated learning method described in the present application;

[0037] Figure 2 is a flowchart of one round of integrated distillation federated training of the present application;

[0038] Figure 3 is the model accuracy obtained by using the FedAvg method and the FedEKD method under the condition of insufficient participants, and using the FedAvg method under the condition of sufficient participants in the Cifar10 data set environment;

[0039] Figure 4 is the model accuracy obtained by using the FedAvg method and the FedEKD method under the condition of insufficient participants, and using the FedAvg method under the condition of sufficient participants in the Cifar100 data set environment;

[0040] Figure 5 is the global model accuracy of FedEKD under the condition of lacking enough users to calculate the global model, compared with the global model accuracy of FedAvg under the condition of having enough users to train the global model, and the global model accuracy of the global model trained by completely using the centralized training method;

[0041] Figure 6 is the global model accuracy of FedEKD under the condition of lacking enough users to calculate the global model, compared with the global model accuracy of FedAvg under the condition of having enough users to train the global model, and the global model accuracy of the global model trained by completely using the centralized training method;

[0042] Figure 7 is the comparison of the amount of parameter data required to be processed by the server between the method of the present application and other methods using knowledge distillation technology for federated learning. DETAILED DESCRIPTION

[0043] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0044] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0045] EMBODIMENTS

[0046] Please refer to Figure 1 and Figure 2 The embodiments of the present application provide a grouping integrated distillation federated learning method FedEKD in an edge network, comprising the following steps:

[0047] S1, the edge server determines a plurality of Client devices that need to participate in federated learning.

[0048] S2, the edge server divides all the Client devices into a plurality of groups according to the computing resources, each group including one Leader device and a plurality of attender devices, and the Leader device has sufficient computing resources to calculate the global model.

[0049] S3, the edge server initializes a global model for the leader device to train and a small model for the attender device to train, and the small model meets the computing capacity of all client devices.

[0050] After initializing the global model and the small model, the edge server needs to first distribute the global model and the small model to the leader device in each group of client devices, and the leader device in each group broadcasts the small model to the attender device in the group.

[0051] S4, for each group of client devices, the global model is subjected to N rounds of integrated distillation federated training, and each round of integrated distillation federated training specifically includes the following steps:

[0052] S41, the leader device trains the global model according to the local loss data and the local picture data, and obtains a first set of labels corresponding to a plurality of training categories, and the attender device trains the small model according to the local loss data and the local picture data, and obtains a second set of labels corresponding to a plurality of training categories.

[0053] In the present application, the first set of labels corresponding to a plurality of training categories is represented as

[0054] z L =(z1,z2,z3,…,z m ),

[0055] The second set of labels corresponding to a plurality of training categories is represented as

[0056]

[0057] S42, the leader device calculates the first group of Soft targets through the Softmax function according to the first set of labels corresponding to a plurality of training categories, and the attender device calculates the second group of Soft targets through the Softmax function according to the second set of labels corresponding to a plurality of training categories, and sends them to the leader device.

[0058] In step S42, the Softmax function is

[0059]

[0060] Wherein, q i is the probability of a certain category, i.e. the Soft target calculated by the device, z i is the prediction value of the i-th category, exp(z i / T) is the LOGITS of the model in a certain category, For the model in all categories of LOGITS, T is a temperature hyperparameter.

[0061] The first group of Soft targets is obtained by bringing the first group of label sets corresponding to several training categories into the Softmax function.

[0062] Q1 = (q1, q2, q3,..., q m )

[0063] The second group of Soft targets is obtained by bringing the second group of label sets corresponding to several training categories into the Softmax function.

[0064] Q2 = (q1, q2, q3,..., q m )

[0065] S43, the Leader device integrates all the current local Soft targets, and forms several matrices according to categories, as follows:

[0066] Q = (Q1, Q2, Q3,..., Q m )

[0067] S44, the Leader device performs weighted linear combination on the matrix of each category, and then selects the Soft target with the minimum loss as the output value z t of the category. In the present application, the formula for performing weighted linear combination on the matrix of each category is as follows:

[0068] Q ′ = αQ = (α1Q1, α2Q2,..., α m Q m ), α ∈ R m , α = (α1, α2,..., α m ) and

[0069] Q ′ is obtained by weighted linear combination.

[0070] The calculation formula of the output value z t is as follows:

[0071] z t = minL CE (Q, y i ),

[0072] L CE = H(z s , y s )

[0073] where y iis the class label of the i-th class of data, H() represents the cross-entropy function, z t is the output value of the integrated distillation calculated.

[0074] S45, the Leader device returns the output value z t to the attender device.

[0075] S46, each Client device calculates the updated local Loss data based on the output value z t and the locally calculated Softtarget.

[0076] In the present application, the calculation formula of the local Loss data is as follows:

[0077]

[0078] wherein, is the Soft target calculated locally by each device, wherein represents the loss value obtained by performing the cross-entropy function on the output obtained by training and the local class label by each device, represents the output z t obtained by using the integrated distillation, and the loss value obtained by performing the KL divergence function on the local Softtarget and then multiplying by the square of the temperature hyperparameter T.

[0079] S5, the Leader device in each group of Client devices returns the trained global model parameters to the edge server.

[0080] S6, the edge server aggregates all the received global model parameters according to the following formula,

[0081]

[0082] wherein, represents the global model parameters returned by the i-th Leader device in the t-th round of federated learning training, represents the parameters of the global model aggregated by the i-th device to be received in the t+1-th round, and n is the total number of Leader devices.

[0083] S7, the edge server tests the aggregated global model parameters to obtain the federated learning Loss data.

[0084] S8, if the federal learning Loss data converges, the federal learning ends, otherwise the aggregated global model parameters are issued to the Leader device, the Leader device updates the locally latest trained global model according to the aggregated global model parameters, that is, reduces the model gradient, and then jumps to step S4.

[0085] Comparative example

[0086] The comparative example of the present application provides a federal learning method FedAvg in an edge network, comprising the following steps:

[0087] S1, the edge server randomly selects a plurality of Client devices that need to participate in federal learning.

[0088] S2, the edge server initializes a global model for the Client device to train locally

[0089] After initializing the global model, the edge server directly broadcasts the global model to all Client devices participating in this round of federal learning

[0090] S3, for each Client device, (1, N) rounds of federal learning training of the global model can be performed, each round comprising the following steps:

[0091] S31, the edge server randomly selects the Client device that accepts model parameter uploading in this round, and the selected device performs the federal learning training in this round.

[0092] S32, the Client device trains the global model according to the local Loss data and the local picture data, and obtains a label set corresponding to a plurality of training categories,

[0093] In this comparative example, the label set corresponding to a plurality of training categories is represented as

[0094] z L =(z1,z2,z3,…,z m ),

[0095] S33, the Client device converts the label set of each training category obtained using the Softmax function without adding the temperature hyperparameter T into a probability q i , that is:

[0096]

[0097] S34, the Client device tests each probability q i obtained using the local picture test data set, and obtains a Loss value using the cross-entropy function:

[0098] L CE= H(z i , y i )

[0099] where z i is the label value obtained by probability prediction and y i is the actual label value, the loss value of the next round of local training is obtained, if the value of the training round is not a multiple of 5, jump to S31;

[0100] S4, if the value of the training round is a multiple of 5, each Client device transmits the global model parameters obtained by training to the edge server

[0101] S5, the edge server aggregates all the global model parameters received according to the following formula,

[0102]

[0103] represents the global model parameters returned by the i-th leader device in the t-th round.

[0104] S6, the edge server tests the global model parameters after the aggregation processing, and obtains the federal learning loss data.

[0105] S7, if the federal learning loss data converges, the federal learning ends, otherwise the global model parameters after the aggregation processing are issued to the leader device, the leader device updates the locally most recently trained global model according to the global model parameters after the aggregation processing, that is, reduces the model gradient, and then jumps to step S3.

[0106] In order to verify the technical effect of the method of the application, the following provides an experimental analysis case.

[0107] Experimental data

[0108] To verify the effectiveness of the method we proposed, we verify our method by using the data set commonly used in machine learning image classification, we designed a comparative experiment on two data sets of Cifar100 and Cifar10 to analyze our method specifically. Among them, the Cifar10 data set is composed of 60000 32x32 color images, and the images have 10 classes, and each class has 6000 images. Respectively contain 50000 training images and 10000 test images. Because the Cifar10 data set can quickly evaluate the ability of the algorithm, it is widely used. At the same time, we also use the Cifar100 data set to further prove the ability of the algorithm we propose, the Cifar100 data set contains 100 classes, each class contains 600 32x32 color images, and each class has 500 training sets and 100 test sets. The number of images in each class of this data set is more in line with the edge computing scenario we want to apply.

[0109] In the experiment, we tested our method using Cifar10 and Cifar100 data sets, respectively. Among them, we mainly test the accuracy of the global model we obtain for the image classification task, and analyze the effectiveness of the method we propose to increase the number of participating users and improve the ability of the global model in the edge computing environment.

[0110] In the experiment, we simulated 10 edge devices for federated learning, of which only 2 devices have the ability to calculate the global model Global Model required for federated learning, and the remaining 8 devices only have the ability to calculate smaller models. We distribute the model through the server, and each participant can choose different models for local training, but participants who can calculate the global model must choose the global model for local training. Each participant selects its own private data set for training, and then performs knowledge transfer through ensemble distillation. We use the Adam optimizer and initialize the learning rate to 0.05, and train for 20 epochs per round. We split the Cifar data so that each client uses the split data set as its private data set for training.

[0111] Compare the above examples and comparative examples, as follows:

[0112] 1. Evaluation index and benchmark model

[0113] We use the commonly used evaluation index TOP-1 accuracy to evaluate the performance of our proposed method, which is widely used to evaluate the performance of different models. We compare the proposed method FedEKD with the basic federated learning algorithm FedAvg and the improved FedProx for data heterogeneity and device heterogeneity.

[0114] In the comparison, we use ResNet20, ResNet8, ShuffleV1 and Vgg8 four models for training.

[0115] 2、Experimental results and analysis

[0116] First, we compare the federated learning under different number of participating users. As can be seen from the table, in the federated learning environment, if only a few users can participate in the federated training process, the efficiency of the global model obtained by directly federated learning through ordinary FedAvg is very low (the table is the accuracy of federated learning training in a federated learning environment with 10 users, of which only 2 users have the ability to train global models). Table 1 is the accuracy of each global model trained by FedAvg in the edge computing environment lacking training global model devices under different data set environments.

[0117] Table 1

[0118]

[0119] But in our FedEKD algorithm, we can get a global model with higher accuracy by using multiple miniaturized models to enable more users to participate in the process. Through Figure 3 and Figure 4 It can be seen that in the same environment, our method enables more users to participate in the federated learning process, enriches the private data needed for federated learning, and improves the accuracy of the model (the right dotted line in the figure represents the federated learning training accuracy of 2 out of 10 users participating in training, and the upper right dotted line represents the federated learning training accuracy of all users participating in training in a federated learning environment with 10 users).

[0120] Meanwhile, the global model formed by our FedEKD algorithm has little difference in accuracy compared to the FedAvg algorithm when there are enough users to calculate the global model, but the convergence speed of the global model in our method is faster. In different experimental environments, 10 different experimental devices select different models sent by the server according to their own conditions, but at least two devices must select to train the global model (because in our method we do not want the server to perform model operations, only parameter aggregation, if the model constructed on the server side also participates in integrated distillation, the server side also needs to perform operations).

[0121] As shown in Figure 5 and Figure 6 , it can be seen that in the case of sufficient edge devices at the edge to calculate the global model, the training results of our model can converge faster compared to FedAvg in the Cifar10 and Cifar100 data set environment.

[0122] Then, we compare the global model formed by the participants using different models with the global model formed by the participants using the same model, and we find that the global model we finally form is not much affected by the depth and type of the participant's model, that is, whether the participants use the same model or completely different models, the difference in accuracy of the final global model is less than 0.3. Therefore, in reality, the server only needs to initialize two models according to the computing power of the participants to obtain a good global model. Table 2 compares the accuracy of the global model calculated by different small models under the FedEKD algorithm in the absence of sufficient users to calculate the global model in different data set environments.

[0123] Table 2

[0124]

[0125] Finally, we compare the amount of data processed by the edge server in our method with the amount of data processed by the server in the Bi-level and FedMD methods, as shown in Figure 7 , we use ResNet20 as the global model and ResNet8 as the small model in Bi-Level, and use ResNet20 as the global model and ResNet8, ShuffleV1 and Vgg8 as personalized models in FedMD to count the number of parameters processed by the server. According to the experimental results, we can see that we have significantly reduced the amount of data processed by the server, while avoiding the complex work of model calculation by the server.

[0126] By comparing the examples and comparative examples, it can be found that the knowledge distillation technology has very strong practicability in solving the device heterogeneous characteristics and data heterogeneous characteristics of federated learning. The grouping integrated distillation federated learning method FedEKD in the edge network provided by the application can make the end device select the model according to the computing power of the end device, and can use the knowledge distillation technology to transfer the knowledge between the models, and can ensure that the resources of the server side can be used to provide services for the users as much as possible without being used for complex model calculation. At the same time, we effectively verify the ability of the method of the application by comparison, and can greatly improve the accuracy of the global model under the condition that only a few users have the computing power of the global model. At the same time, we compare the influence of different small models on the global model, and find that only two models need to be constructed in the method to meet the requirements of the global model.

[0127] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A grouped ensemble distillation federated learning method in edge networks, characterized in that, Includes the following steps: S1. The edge server identifies several client devices that need to participate in federated learning; S2. The edge server divides all client devices into several groups based on computing resources. Each group includes one leader device and several attendant devices. The leader device has sufficient computing resources to compute the global model. S3. The edge server initializes a global model for Leader devices to train and a small model for Attendant devices to train locally. The small model meets the computing power of all Client devices. S4. For each group of Client devices, N rounds of ensemble distillation federated training are performed on the global model; each round of ensemble distillation federated training specifically includes the following steps: S41. The Leader device trains a global model based on local Loss data and local image data, and obtains the first set of labels corresponding to several training categories. The Attender device trains a small model based on local Loss data and local image data, and obtains the second set of labels corresponding to several training categories. S42. The Leader device calculates the first set of soft targets using the Softmax function based on the first set of labels corresponding to several training categories. The Attender device calculates the second set of soft targets using the Softmax function based on the second set of labels corresponding to the second training categories and sends them to the Leader device. S43, the Leader device integrates all currently local Soft targets and organizes them into several matrices according to categories; S44. The Leader device performs a weighted linear combination of the matrices for each category, and then selects the Soft target with the minimum loss as the output value for that category. ; S45, the Leader device will output the value for each category. Returned to the atender device; S46, Output value of each Client device based on each category And the locally calculated Soft target, and calculate and update the local Loss data; S5. The Leader device in each Client device group returns the trained global model parameters to the edge server. S6. The edge server aggregates all received global model parameters. S7. The edge server tests the aggregated global model parameters to obtain federated learning loss data. S8. If the federated learning loss data converges, the federated learning ends; otherwise, the aggregated global model parameters are sent to the Leader device. The Leader device updates its most recently trained global model according to the aggregated global model parameters, and then proceeds to step S4.

2. The grouped ensemble distillation federated learning method according to claim 1, characterized in that, In step S3, the edge server first needs to distribute the global model and the small model to the Leader device of each group, and then the Leader device of each group broadcasts the small model to the attendant device of its group.

3. The grouped ensemble distillation federated learning method according to claim 1, characterized in that, In step S41, the training methods for both the global model and the small model include data preprocessing and inputting it into the neural network model, loss calculation, gradient descent, and parameter optimization.

4. The grouped ensemble distillation federated learning method according to claim 3, characterized in that, In step S42, the Softmax function is: in, The probability of a certain category, i.e., the soft target calculated by the device. Let be the predicted value for the i-th category. For the model's LOGITS in a certain category, Here, LOGITS represents the model across all categories, and T is the temperature hyperparameter.

5. The grouped ensemble distillation federated learning method according to claim 4, characterized in that, In step S46, the formula for calculating local loss data is as follows: in, , This refers to the soft target calculated locally on each device, where... This represents the loss value obtained by performing a cross-entropy function on the output trained on each device and its local class label. This indicates the output obtained using integrated distillation. Multiply the KL divergence function by the local soft-target and then by the temperature hyperparameter. The loss value is obtained by squaring the square of the loss.

6. The grouped ensemble distillation federated learning method according to claim 5, characterized in that, In step S6, the polymerization process uses the following formula: in, This represents the global model parameters returned by the i-th leader device in the t-th round of federated learning training. This represents the parameters that the i-th device will receive after global model aggregation in round t+1, where n is the total number of leader devices.

7. The grouped ensemble distillation federated learning method according to claim 6, characterized in that, In step S7, the method for testing the aggregated global model parameters is to test the accuracy of the global model parameters on the test set by using the test set data and labels.

Images