Heterogeneous edge-end collaborative learning method based on prototype and edge-end system

Abstract

The invention discloses a prototype-based heterogeneous edge-end collaborative learning method and an edge-end system, and aims to solve challenges caused by end-side data isomerism and edge-end model isomerism in a plurality of edge-end collaborative scenes. According to the method, global prototype aggregation training and a global prototype-based data generation mechanism are introduced, so that collaborative learning of heterogeneous edge-end models is realized, and diversified edge-end scene requirements are effectively met. According to the method, time and resource consumption required by equipment training in a scene with limited side end resources are considered. The prototype is introduced into a unique knowledge transmission mode, so that the knowledge sharing burden in traditional edge-end collaborative learning is greatly reduced. According to the method, the limitation that heterogeneous model training cannot be carried out based on a prototype in a traditional method is broken through, the application barrier of the heterogeneous model in edge-end cooperation is broken through, and resource consumption needed by training is remarkably reduced in an edge-end scene with resource limitation. According to the method, the training performance of the edge-end heterogeneous model is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing, and specifically relates to a prototype-based heterogeneous edge-end collaborative learning method and an edge-end system. Background Art

[0002] Cloud-edge-end collaborative learning is the current mainstream end-device training architecture. It uses end-device resources to train locally, and edge servers perform small-scale knowledge aggregation, which is ultimately processed and distributed by the cloud. Although this architecture effectively utilizes the resources of end devices, it still has problems with high communication latency and heavy transmission burden. Edge-end collaborative AI technology migrates cloud functions to edge servers, and trains models together through continuous knowledge sharing between edge servers and end devices. Both edge servers and end devices perform model training and reasoning. The larger the model scale of the edge server, the greater the potential for assisting end-device reasoning. Communicating only with edge servers instead of the cloud can reduce communication latency and improve user experience in many commercial and industrial scenarios.

[0003] In practice, the knowledge transmission burden of edge collaboration must be considered because it directly affects latency and energy consumption, further determining whether resource-limited edge devices can complete the expected training tasks. Current methods train by transmitting gradients and models, intermediate results such as logits, and processed training data. However, since the model needs to be shared in each round, the transmission burden of gradient and model sharing is huge and cannot support heterogeneous model training. In addition, although logits can improve the performance of heterogeneous models, its transmission cost is proportional to the amount of data, which is usually unacceptable to end devices. Processing training data transmission to train models inevitably incurs high transmission costs and potential user privacy leakage. In contrast, the prototype is calculated as the average feature vector of data belonging to the same class, which greatly reduces the transmission burden. In addition, it is also used for few-shot learning because it can cope with data heterogeneity, demonstrating its great potential in edge collaborative heterogeneous learning. However, current prototype-based research only focuses on improving the performance of edge models, rather than edge heterogeneous models, ignoring resource-constrained scenarios with multiple edges and multiple end devices. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a prototype-based edge collaborative learning method, which is suitable for application in an edge system composed of multiple end-side devices and multiple edge-side servers. The purpose is to reduce the transmission burden of edge collaborative learning and realize collaborative learning of edge heterogeneous models by only transmitting lightweight prototypes.

[0005] The invention also provides a corresponding edge system.

[0006] In order to achieve the above invention object, the technical solution of the present invention is as follows:

[0007] In a first aspect, a prototype-based heterogeneous edge-end collaborative learning method is provided, the method comprising:

[0008] (1) The terminal device and all edge servers confirm their ownership relationship, and all edge servers randomly select a server as the aggregation edge server;

[0009] (2) The terminal device initializes the local model, performs local training in combination with the global prototype issued in the previous round, and sends the locally trained terminal prototype to the corresponding edge server. The terminal prototype is the average value of the feature vectors of all samples divided by category after the local model training of the terminal device;

[0010] (3) The edge server aggregates the terminal prototypes of all terminal devices within the jurisdiction into edge prototypes, and sends the edge prototypes to the aggregation edge server. The edge prototypes are obtained by aggregating the terminal prototypes according to the category distribution of the terminal device data;

[0011] (4) Aggregate edge servers according to the category distribution of edge servers and aggregate the edge models of all edge servers into a global prototype;

[0012] (5) The aggregation edge server sends the global prototype to the edge server, and the edge server sends the global prototype to the terminal device;

[0013] (6) All edge servers use the global prototype and the pre-trained generative model to generate data on the specified category prototype and train the edge model;

[0014] (7) Determine whether the edge model meets the convergence conditions. If not, return to step (2) to continue training; otherwise, terminate the training.

[0015] Furthermore, the terminal device initializes the local model and performs local training based on the global prototype issued in the previous round, including:

[0016] The terminal device initializes the local model, prepares training data, and interacts with the edge server to determine whether there is a global prototype. If so, the global prototype is used as the target to approach during training and combined with local data for training. If there is no global prototype, only local data is used for training.

[0017] Furthermore, each terminal device performs local training on its own local data to minimize its own loss function. The loss function is calculated as in is the cross entropy loss function; is the output of the terminal device model for sample data x, and y is the label corresponding to sample data x; Represents the global prototype and end prototype The distance between them; λ is the control parameter of the loss.

[0018] Furthermore, the terminal prototype calculation formula generated by the terminal device is: in is the prototype of the jth class sent by terminal device n to edge server e; the subscript ne represents that terminal device n is associated with edge server e; and f ne (x) respectively represent the j-th type of data owned by the terminal device n and the feature vector of the data x after feature extraction.

[0019] Furthermore, the calculation formula for the edge server to generate the edge prototype is: in Represents the prototype vector of the jth class corresponding to the edge server e; N e and Each represents the set of terminal devices under the jurisdiction of edge server e and the corresponding number of samples of the jth class.

[0020] Furthermore, the aggregation formula of the global prototype of the aggregation edge server is in represents the number of samples of the jth class owned by edge server e, and E is the set of edge servers.

[0021] Furthermore, all edge servers use the global prototypes and pre-trained generative models to generate data on the prototypes of the specified category, including:

[0022] All edge servers use Gaussian distribution to randomly generate several noise vectors, combine several noise vectors and global prototypes into new feature vectors, and provide these new feature vectors as input to the pre-trained generative model to generate data by category. The generated data will be used to train heterogeneous edge models on edge servers.

[0023] In a second aspect, an edge system is provided, including a plurality of terminal devices and a plurality of edge servers, wherein the plurality of terminal devices and the plurality of edge servers complete model training by collaboration;

[0024] In one round of iteration, the terminal device is configured to: initialize the local model, perform local training in combination with the global prototype sent in the previous round, and send the locally trained terminal prototype to the corresponding edge server to which it belongs. The terminal prototype is the average value of the feature vectors of all samples divided by category after the local model of the terminal device is trained and calculated one by one by category;

[0025] The edge server is configured to: aggregate the end prototypes of all terminal devices within the jurisdiction into edge prototypes, and send the edge prototypes to an aggregation edge server, wherein the edge prototype is obtained by aggregating the end prototypes according to the category distribution of the terminal device data, and the aggregation edge server is a server randomly selected from a number of edge servers; and receive the global prototype from the aggregation edge server and send the global prototype to the terminal device; and generate data on the specified category prototype using a pre-trained generation model according to the global prototype to train the edge model;

[0026] The aggregation edge server is configured to: aggregate the edge models of all edge servers into a global prototype according to the category distribution of the edge servers; and send the global prototype to the edge servers.

[0027] Beneficial effects: The present invention proposes a method for realizing heterogeneous edge-end collaborative learning in a framework of multiple terminal devices and multiple edge servers, relying only on lightweight prototype transmission. The method first obtains the end prototype through the terminal device, and then converges to obtain the edge prototype, and the edge prototype is then converged into a global prototype. The global prototype can not only be used for the training of the end device but also as the central vector of the multidimensional space, mixed with random noise to obtain different vectors. Then, the generative model processes these vectors to generate samples, and these generated samples are used to train the edge model. The method relies on aggregated prototypes and prototype-based data generation to complete heterogeneous model training, and has the following advantages compared with the prior art:

[0028] (1) This method has low communication cost and is suitable for edge communication. The cost of transmitting the prototype is at least reduced to 1 / 20 of the cost of transmitting the model / gradient / sample. This greatly saves the communication overhead of edge collaborative learning and allows more clients to interact with one edge when bandwidth is limited.

[0029] (2) The prototype transmitted by this method is the average of the sum of the features of multiple samples. After obtaining the prototype, the attacker cannot infer the features of a single sample from the aggregated prototype. Thanks to the transmission of only the prototype, the attack methods based on the model / gradient / sample will not cause privacy damage to the collaborative learning of the present invention;

[0030] (3) The present invention has good scalability. It not only improves the performance of heterogeneous edge models by transmitting lightweight prototypes, but can also be combined with other logits-based methods to further improve performance. In addition, it can replace the edge learning components in the current cloud architecture and support the collaboration of heterogeneous models in various industries by sharing prototypes, bringing new ideas to the knowledge sharing method of large and small models. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the following drawings are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 Flowchart of the prototype-based edge collaborative learning method.

[0033] Figure 2 The following is an architectural diagram of the prototype-based edge collaborative learning method.

[0034] Figure 3 The figure is a comparison chart of the terminal-side model accuracy of the prototype-based edge collaborative learning method on the comparison dataset when the Dirichlet distribution parameter is 0.1.

[0035] Figure 4 The figure is a comparison chart of the edge-side model accuracy of the prototype-based edge collaborative learning method on the comparison dataset when the Dirichlet distribution parameter is 0.1.

[0036] Figure 5 A comparison chart of the communication burden of the prototype-based edge collaborative learning method during one round of communication. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described in conjunction with specific embodiments and corresponding drawings. At the same time, it should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, the modifications of various equivalent forms of the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0038] The present invention belongs to the field of edge computing, and focuses on designing a collaborative learning method applied in the framework of multiple terminal devices and multiple edge servers. Combining the characteristics of lightweight and representative class prototypes, the collaborative training of heterogeneous edge models is completed by using global prototype aggregation auxiliary training and prototype-based data generation. Assume that the scenario of the heterogeneous edge model is as follows: there are E: = {1, ..., E} edge servers and N: = {1, ..., N} terminal devices, and the model on the edge server (edge) and the model on the terminal device (end) are heterogeneous. The model on the edge is defined as ω e , define the model on the end as ω n Different ends and edges perform collaborative training. Since the edge servers have strong computing power, different edge servers can communicate with each other to transmit prototypes. Each edge server only covers a part of the end devices. Each edge server e has its own set of end devices Ne , a total of |N e | terminals. Each terminal device has a data set of Where X i represents the dataset of the i-th client, Y i Indicates the label corresponding to the dataset owned by the i-th client. There are a total of Each terminal device has a different number of classes, namely |C n |. The following will demonstrate the implementation process of the prototype-based edge collaborative learning method through a specific example. The whole process is as follows Figure 1 As shown, the specific implementation is:

[0039] Step 1: First, obtain the topology of the current network structure. Figure 2 For example, six terminal devices correspond to two edge servers respectively. Dirichlet distribution is used to simulate the heterogeneity of data. The parameter is set to 1 to randomly generate a corresponding training data set for each device. After the random data generation is completed, each terminal device is equipped with a training data set and a test set of the CIFAR-100 data set. The first three terminal devices correspond to the first edge server, and the last three edge servers correspond to the second edge server. Each terminal device runs the ResNet-8 model. The model running on each edge server is ResNet-34. The generation model on each edge server is a pre-trained StyleGAN. The right edge server is randomly selected as the aggregation edge server (also referred to as the aggregation server in the present invention). In addition to shouldering the responsibilities of the normal edge server, it also needs to collect and aggregate edge prototypes, and send the aggregated global prototypes to other edge servers.

[0040] Step 2: All terminals randomly initialize their local models, the global number of initial training rounds is 100 rounds, the batch_size is 10, the learning rate is set to 0.005 and the learning rate decays at a rate of 10% after 20 rounds. The terminal checks whether the local receives the global prototype sent from the edge server to which it belongs. If the global prototype is not received, local training is carried out normally. If the global prototype is detected, the prototype loss is increased on the basis of the local cross entropy, so that the prototype corresponding to the category in the local data set is close to the global prototype during the training process. After completing the training, all terminals divide the feature vectors of all samples by category, and calculate the average value by category one by one to obtain the terminal prototype. In addition, all terminals need to test the accuracy information of the training model on the test set and send it to the corresponding edge server for recording.

[0041] Among them, each terminal device runs local training on its own local data to minimize the loss function, and the loss function is calculated as follows: in is the cross entropy loss function. It is the output of the terminal device for the sample data x. Measured global prototype and terminal prototype (terminal prototype) The distance between them. λ is the control parameter of the loss. If the terminal device does not receive the global prototype, the loss function is Represents the calculation of cross entropy loss using the end model, where y is the label corresponding to the sample data x.

[0042] The terminal prototype calculation formula generated by the terminal device is: in It is the prototype of the jth class sent by terminal device n to edge server e. The subscript ne represents that terminal device n is associated with edge server e. and f ne (x) respectively represent the j-th type of data owned by the terminal device n and the feature vector of the data x after feature extraction.

[0043] Step 3: After all terminal devices obtain the terminal prototype, they send their own terminal prototype to the corresponding edge server according to the topology determined in step 1. For example, terminal device n will send terminal prototype To the corresponding edge server e. At this time, all end devices record the size of the end prototype, and this information will be given to the edge server as a transmission burden metric for communication burden statistics.

[0044] Step 4: All edge servers receive the terminal prototypes of the terminal devices, aggregate the terminal prototypes by category according to the category distribution of the terminal device data under their jurisdiction, and obtain the edge prototypes. Then all edge servers will send the edge prototypes to the random aggregation edge server determined in step 1. The aggregation edge server will calculate the category distribution of each edge server and aggregate the edge prototypes into a global prototype according to the category distribution information.

[0045] The calculation formula for edge server to generate edge prototype is: in Represents the prototype vector of the jth class corresponding to edge server e. N e and Each represents the set of terminal devices under the jurisdiction of edge server e and the corresponding number of samples of the jth class.

[0046] The formula for aggregating the global prototype by the aggregating edge server is: in Represents the number of samples of the jth class owned by edge server e.

[0047] Step 5: The global prototype is sent by the aggregation edge server to other edge servers, and then the edge server sends the global prototype to the corresponding terminal device for the next training. Similarly, all edge servers need to record the size of the global prototype, and this information will be given to the aggregation edge server as a transmission burden metric for communication burden statistics.

[0048] Step 6: All edge servers use the received global prototype as the basis vector and use the normal distribution to randomly generate 1,000 feature vectors of the same dimension as the global prototype. These feature vectors correspond to the 100 classes of CIFAR-100. Then the generated random noise vector is added to the global vector to represent the different feature vectors in the sample space. All edge servers feed these generated feature vectors as input to the pre-trained generative model StyleGAN, generating 1,000 samples for 100 categories, with 10 samples for each category. These samples are then used to train the edge model. The vector generated by the normal distribution is denoted as The different eigenvectors generated are Generative Models Generate a large number of samples X g The process is represented as Each edge server uses the generated samples to train the heterogeneous model. The number of training rounds and the number of generated samples can be determined according to the idle computing power of the edge server. In this example, the global number of rounds of edge server training is also set to 100 rounds, the batch_size is 10, the learning rate is set to 0.005 and the learning rate decays at a rate of 10% after 20 rounds.

[0049] Step 7: Determine whether the current terminal model and edge server model have converged. If the edge model does not converge, return to step 2 and repeat the above process. Otherwise, the training ends and a trained heterogeneous model is obtained. After the training is completed, the model is tested on the CIFAR-100 test data set to obtain the test accuracy and record it.

[0050] Furthermore, the performance of the prototype-based edge-end collaborative learning method described in the present invention was verified through simulation experiments. The prototype-based edge-end collaborative learning method was implemented using Python language and Pytorch framework, and HierAVG, FedAgg and FedGKT methods were selected for comparison. HierAVG is a two-layer framework of edge-end collaboration to collaboratively train a homogeneous model through model and gradient transmission. FedGKT supports the training of edge-end heterogeneous models by sharing feature maps through the knowledge distillation method. FedAgg processes the raw data to generate bridge samples, and then the edge model can be used to train a larger edge model. The comparison criteria include the average test accuracy of the edge model and the highest test accuracy of the edge model after convergence or 100 rounds of communication. All programs are run on a standard Linux server equipped with Intel(R) Xeon(R) Gold5218R CPU@2.10GHz, 64GB RAM, 1TB SSD, GeForce RTX 3060, and Ubuntu. The experimental results are as follows Figures 3 to 5 As shown, Figure 3 It is demonstrated that the present invention (PBT) enables the end device to achieve the highest test accuracy on the CIFAR-100 dataset, which is at least 35.08%, 25.02% and 3.17% higher than the highest accuracy in HierFAVG, FedAgg and FedGKT. Figure 4 It is demonstrated that the present invention achieves the highest accuracy of 78.12%, 53.16% and 20.31% on the MNIST, CIFAR-10 and CIFAR-100 datasets, respectively, which is at least 7.96% higher than other methods. Figure 5 The communication burden of a single round of edge knowledge sharing for all methods is shown. Thanks to the transmission of the global prototype vector, the results show that this method only requires a transmission burden of 0.11M, which is the lowest among all methods.

[0051] In summary, this prototype-based edge collaborative learning method effectively realizes the sharing of knowledge on the edge side by using prototypes as a means of knowledge sharing. Using prototypes for knowledge transmission significantly reduces the burden of knowledge transfer in a single round of transmission, thereby greatly improving the overall speed of edge collaborative learning. In addition, the sharing of prototypes will not have a negative impact on the training of the edge model, but can further enhance the training effect of the edge model through data generation. Through the transmission of prototypes, a rich source of knowledge can be continuously provided to the system throughout the life cycle of the edge collaborative learning system. This method can quickly advance the model training process after the data set is built on the edge side. More importantly, while performing prototype aggregation, the training of the local model will not be disturbed or delayed. Through experimental verification, this method has shown good adaptability on multiple different data sets and can effectively support the training of heterogeneous edge models. This shows that the prototype-based edge collaborative learning method has application potential and application value in a variety of application scenarios.

[0052] The present invention also provides an edge system, comprising a plurality of terminal devices and a plurality of edge servers, wherein the plurality of terminal devices and the plurality of edge servers complete model training by collaboration;

[0053] In one round of iteration, the terminal device is configured to: initialize the local model, perform local training in combination with the global prototype sent in the previous round, and send the locally trained terminal prototype to the corresponding edge server to which it belongs. The terminal prototype is the average value of the feature vectors of all samples divided by category after the local model of the terminal device is trained and calculated one by one by category;

[0054] The edge server is configured to: aggregate the end prototypes of all terminal devices within the jurisdiction into edge prototypes, and send the edge prototypes to an aggregation edge server, wherein the edge prototype is obtained by aggregating the end prototypes according to the category distribution of the terminal device data, and the aggregation edge server is a server randomly selected from a number of edge servers; and receive the global prototype from the aggregation edge server and send the global prototype to the terminal device; and generate data on the specified category prototype using a pre-trained generation model according to the global prototype to train the edge model;

[0055] The aggregation edge server is configured to: aggregate the edge models of all edge servers into a global prototype according to the category distribution of the edge servers; and send the global prototype to the edge servers.

[0056] The specific implementations of the training process of the terminal device and the edge server, the aggregation process of the edge server, and the aggregation process of the aggregated edge server can refer to the description of the aforementioned method embodiment, which will not be repeated here.

Claims

1. A prototype-based heterogeneous edge collaborative learning method, characterized in that: The following steps are involved: (1) The terminal device and all edge servers confirm their ownership relationship, and all edge servers randomly select a server as the aggregation edge server; (2) The terminal device initializes the local model, performs local training in combination with the global prototype issued in the previous round, and sends the locally trained terminal prototype to the corresponding edge server. The terminal prototype is the average value of the feature vectors of all samples divided by category after the local model training of the terminal device; (3) The edge server aggregates the terminal prototypes of all terminal devices within the jurisdiction into edge prototypes, and sends the edge prototypes to the aggregation edge server. The edge prototypes are obtained by aggregating the terminal prototypes according to the category distribution of the terminal device data; (4) Aggregate edge servers according to the category distribution of edge servers and aggregate the edge models of all edge servers into a global prototype; (5) The aggregation edge server sends the global prototype to the edge server, and the edge server sends the global prototype to the terminal device; (6) All edge servers use the global prototype and the pre-trained generative model to generate data on the specified category prototype and train the edge model; (7) Determine whether the edge model meets the convergence conditions. If not, return to step (2) to continue training; otherwise, terminate the training.

2. The method according to claim 1, characterized in that The terminal device initializes the local model and performs local training based on the global prototype issued in the previous round, including: The terminal device initializes the local model, prepares training data, and interacts with the edge server to determine whether there is a global prototype. If so, the global prototype is used as the target to approach during training and combined with local data for training. If there is no global prototype, only local data is used for training.

3. The method according to claim 2, characterized in that Each terminal device performs local training on its own local data to minimize its own loss function. The loss function is calculated as in is the cross entropy loss function; is the output of the terminal device model for sample data x, and y is the label corresponding to sample data x; Represents the global prototype and end prototype The distance between them; λ is the control parameter of the loss.

4. The method according to claim 1, characterized in that: The terminal prototype calculation formula generated by the terminal device is: in is the prototype of the jth class sent by terminal device n to edge server e; the subscript ne represents that terminal device n is associated with edge server e; and f ne (x) respectively represent the j-th type of data owned by the terminal device n and the feature vector of the data x after feature extraction.

5. The method according to claim 1, characterized in that The calculation formula for edge server to generate edge prototype is: in Represents the prototype vector of the jth class corresponding to the edge server e; N e and Each represents the set of terminal devices under the jurisdiction of edge server e and the corresponding number of samples of the jth class.

6. The method according to claim 5, characterized in that The aggregation formula for the global prototype of the aggregation edge server is: in represents the number of samples of the jth class owned by edge server e, and E is the set of edge servers.

7. The method according to claim 1, characterized in that All edge servers use global prototypes and pre-trained generative models to generate data on prototypes of specified categories, including: All edge servers use Gaussian distribution to randomly generate several noise vectors, combine several noise vectors and global prototypes into new feature vectors, and provide these new feature vectors as input to the pre-trained generative model to generate data by category. The generated data will be used to train heterogeneous edge models on edge servers.

8. An edge system, characterized in that: It includes a plurality of terminal devices and a plurality of edge servers, wherein the plurality of terminal devices and the plurality of edge servers complete model training by collaboration; In one round of iteration, the terminal device is configured to: initialize the local model, perform local training in combination with the global prototype sent in the previous round, and send the locally trained terminal prototype to the corresponding edge server to which it belongs. The terminal prototype is the average value of the feature vectors of all samples divided by category after the local model of the terminal device is trained and calculated one by one by category; The edge server is configured to: aggregate the terminal prototypes of all terminal devices within the jurisdiction into edge prototypes, and send the edge prototypes to an aggregation edge server, wherein the edge prototypes are obtained by aggregating the terminal prototypes according to the category distribution of the terminal device data, and the aggregation edge server is a server randomly selected from a number of edge servers; and receiving a global prototype from an aggregation edge server and sending the global prototype to a terminal device; and generating data on a specified category prototype using a pre-trained generation model based on the global prototype to train an edge model; The aggregation edge server is configured to: aggregate the edge models of all edge servers into a global prototype according to the category distribution of the edge servers; and send the global prototype to the edge servers.

Images