A federated knowledge distillation method, apparatus, device and storage medium

Abstract

The application discloses a federal knowledge distillation method and device, equipment and storage medium, and relates to the technical field of machine learning, which comprises the following steps: training a local model by using a private data set of each user end to obtain a trained local model; inputting the same public data into each trained local model to obtain model output results, and uploading all the model output results to a central server, so that the central server trains a discriminator model by using the model output results; calculating the average value of all the model output results by the central server to obtain an average model output, and downloading the average model output and the gradient of the loss function of the discriminator model with respect to the user end local model output to the corresponding user end, so that the user end performs knowledge migration on the user end local model by using the average model output and the gradient. The discriminator model located in the central server can realize flexible and efficient knowledge migration among multiple user ends, and reduce communication cost.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a federated knowledge distillation method, apparatus, device, and storage medium. Background Art

[0002] With the rapid development of mobile communications technology and artificial intelligence, users are increasingly demanding intelligent applications. Federated learning has emerged to fully utilize user data widely distributed at the edge of the network. In federated learning, each user first trains a local model using private data and sends the model parameters to a central server. The central server then aggregates the global model by averaging the model parameters across all users. This global model is then broadcast to all users to continue local model training. Through periodic local model training and global model aggregation, federated learning enables collaborative machine learning across a large number of user devices while ensuring the privacy of user data.

[0003] However, traditional federated learning has the following limitations: First, when the user model is large, transmitting model parameters to the central server will consume a lot of communication time, thereby reducing the speed of overall model training; second, traditional federated learning requires all users to have the same model structure in order to average model parameters. However, in actual scenarios, user devices have different computing and storage resource capacities and support different local model structures. Therefore, traditional federated learning methods are difficult to apply to heterogeneous user local models and model privacy requirements.

[0004] Federated knowledge distillation, on the other hand, can achieve knowledge transfer between black-box heterogeneous models by exchanging user local model outputs with smaller data volumes, utilizing knowledge distillation technology to ensure user model privacy and reduce communication costs. Unlike traditional federated learning, users upload their local model outputs for the same public data to a central server. Knowledge distillation is then performed using the average output of all user models calculated by the central server. This minimizes the distance between each user's local model output and the global average output of all users, enabling knowledge transfer from other user models to the local model, ultimately resulting in a user model with high accuracy on global data. However, in real life, data collected by user devices is often non-independent and identically distributed. Simply averaging the outputs of all users' local models fails to reflect the true global data characteristics, and knowledge distillation based on the average output of all users' local models severely limits the performance of the final federated knowledge distillation model. Therefore, it is urgent to design new federated knowledge distillation methods to enable flexible and efficient inter-user knowledge transfer, taking into account the heterogeneous models and data characteristics of users. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a federated knowledge distillation method, apparatus, device, and storage medium that can achieve flexible and efficient knowledge migration between multiple clients and reduce communication costs. The specific solution is as follows:

[0006] In a first aspect, the present application discloses a federated knowledge distillation method, comprising:

[0007] Using the private dataset of each user terminal to train the local model of the user terminal, respectively, to obtain multiple trained local models;

[0008] Inputting the same public data into each of the trained local models to obtain corresponding multiple model output results, and uploading all the model output results to a central server so that the central server can train a discriminator model using the model output results;

[0009] The central server calculates the average value of all the model output results to obtain the average model output, and sends the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal uses the average model output and the gradient to perform knowledge transfer on the trained local model.

[0010] Optionally, performing knowledge transfer on the trained local model using the average model output and the gradient includes:

[0011] The trained local model is trained by minimizing a loss function using the average model output and the gradient.

[0012] Optionally, after performing knowledge transfer on the trained local model using the average model output and the gradient, the method further includes:

[0013] Determine whether all current user-side local models converge. If so, stop executing the knowledge migration process. If not, periodically execute the knowledge migration process until all user-side local models converge.

[0014] Optionally, the private dataset includes input features and corresponding data labels.

[0015] Optionally, the discriminator model is used to identify the user end to which the user-end local model belongs, and to form adversarial learning with the user-end local model training so that all the user-end local models have the same model output probability distribution, and to classify the data labels corresponding to the outputs of the user-end local models to expand the decision boundaries between different data categories.

[0016] In a second aspect, the present application discloses a federated knowledge distillation method, which is applied to a user end and includes:

[0017] Use the local private dataset to train the local model to obtain the trained local model;

[0018] Inputting the same public data into the trained local model to obtain a model output result;

[0019] Uploading the model output results to a central server so that the central server can train a discriminator model using the model output results and the model output results uploaded by other users, and calculating the average of all model output results to obtain an average model output;

[0020] Receive the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output returned by the central server, and perform knowledge transfer on the trained local model based on the average model output and the gradient.

[0021] In a third aspect, the present application discloses a federated knowledge distillation method, which is applied to a central server and includes:

[0022] Obtaining model output results obtained by inputting the same public data uploaded by all user terminals into a trained local model located at the user terminal; wherein the trained local model is a model obtained by training the local model of the user terminal based on a private dataset;

[0023] Using the model output results to train a discriminator model;

[0024] Calculate the average value of all the model output results to obtain the average model output, and send the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal uses the average model output and the gradient to perform knowledge transfer on the trained local model.

[0025] In a fourth aspect, the present application discloses a federal knowledge distillation device, comprising:

[0026] A model training module is used to train the local model of each user terminal using the private data set of the user terminal to obtain multiple trained local models;

[0027] A common data input module, used to input the same common data into each of the trained local models to obtain corresponding multiple model output results;

[0028] A model output result uploading module is used to upload all the model output results to a central server so that the central server can use the model output results to train the discriminator model;

[0029] A mean calculation module, configured to calculate the mean of all the model output results through the central server to obtain an average model output;

[0030] A data delivery module, configured to deliver the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal;

[0031] A knowledge transfer module is used for the user terminal to perform knowledge transfer on the trained local model using the average model output and the gradient.

[0032] In a fifth aspect, the present application discloses an electronic device comprising a processor and a memory; wherein, the processor implements the aforementioned federated knowledge distillation method when executing a computer program stored in the memory.

[0033] In a sixth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned federated knowledge distillation method is implemented.

[0034] It can be seen that the present application first uses the private data set of each user terminal to train the local model of the user terminal to obtain multiple trained local models, and then inputs the same public data into each of the trained local models to obtain corresponding multiple model output results, and uploads all the model output results to the central server so that the central server can use the model output results to train the discriminator model, and then calculates the average value of all the model output results through the central server to obtain the average model output, and sends the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal can use the average model output and the gradient to perform knowledge transfer on the trained local model. The present application uses the discriminator model located on the central server to achieve flexible and efficient knowledge transfer between multiple user terminals and reduce communication costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0036] Figure 1 This is a flow chart of a federated knowledge distillation method disclosed in this application;

[0037] Figure 2 A block diagram of a specific federated knowledge distillation method disclosed in this application;

[0038] Figure 3 A flowchart of a specific federated knowledge distillation method disclosed in this application;

[0039] Figure 4 A flowchart of a specific federated knowledge distillation method disclosed in this application;

[0040] Figure 5 This is a schematic diagram of the structure of a federal knowledge distillation device disclosed in this application;

[0041] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0043] This application embodiment discloses a federated knowledge distillation method, see Figure 1 As shown, the method includes:

[0044] Step S11: using the private data set of each user terminal to train the local model of the user terminal, to obtain multiple trained local models.

[0045] In this embodiment, each user terminal's private data set is first used to train its own local model to obtain multiple trained local models. That is, the user terminal trains its own local model without sharing local data. The private data set may include input features and corresponding data labels. For details, see Figure 2 As shown, Figure 2 A specific block diagram of the federated knowledge distillation method is shown, in which any user terminal n has a local private dataset Private datasets A single data sample in can be composed of input features x and corresponding data labels y. Assume that the local model of any user terminal n is f n (x,θ n ), then the corresponding model output probability distribution is p n (x,θn ), that is, f n (x,θ n ) is the Softmax function output. Among them, θ n represents the model parameters of the local model of any user terminal n, then the model parameter set of all users is In addition, the private dataset of all users can be expressed as

[0046] Step S12: Input the same public data into each of the trained local models to obtain corresponding multiple model output results, and upload all the model output results to the central server so that the central server can use the model output results to train the discriminator model.

[0047] In this embodiment, after the local model of each user terminal is trained using the private data set of each user terminal to obtain multiple trained local models, the same public data pre-set for each user terminal is input into each of the above-mentioned trained local models, and then the corresponding multiple model output results are obtained through model calculation. Then, all the above-mentioned model output results are uploaded to the central server. After the central server obtains the above-mentioned model output results, a discriminator model is trained using the above-mentioned model output results. It should be pointed out that all users in this embodiment jointly maintain a public data set P. When each user terminal inputs the public data set P into the trained local model obtained by training using the local private data set, the output results p of each model can be obtained. n , and then output the model results p of each user end n As the input of the discriminator model, where the model outputs the result p n It consists of two parts: a (p n (x,θ n ), w), and h c (p n (x,θ n ), w), respectively correspond to the two tasks of the discriminator model: the recognition model output result p n The corresponding user terminal n and the real data label y, where w is the model parameter of the discriminator model. Therefore, the objective function of the discriminator model can be expressed as:

[0048]

[0049] In this embodiment, the discriminator model is used to identify the user end to which the user end local model belongs, and to form adversarial learning with the user end local model training, so that all the user end local models have the same model output probability distribution, and to classify the data labels corresponding to the output of the user end local model, so as to expand the decision boundary between different data categories. It should be pointed out that the discriminator model in this embodiment has two tasks. The first task is to identify the user end to which the output result of the user end local model belongs, and to form adversarial learning with the user end local model training, so as to promote all user end local models to have the same output probability distribution and realize flexible knowledge transfer between users; the other task is to classify the data labels corresponding to the output of the user end local model, so as to expand the decision boundary between different data categories, thereby realizing efficient knowledge transfer between users.

[0050] Step S13: The central server calculates the average value of all the model output results to obtain the average model output, and sends the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal uses the average model output and the gradient to perform knowledge transfer on the trained local model.

[0051] In this embodiment, after the discriminator model is trained using the model output results, the central server is used to calculate the average of all the model output results to obtain the average model output. The average model output and the gradient of the loss function of the discriminator model relative to the trained local model output are then sent to the corresponding user terminal. After the user terminal obtains the average model output and the gradient, the average model output and the gradient are used to perform knowledge transfer on the trained local model. That is, the user terminal can update the local model based on the feedback results of the discriminator and the average model output to perform knowledge transfer. Specifically, the objective function of any user terminal n is:

[0052]

[0053] in, represents the mean of the probability distribution of the local model outputs of other user terminals except user terminal n, and β and γ represent the weight coefficients between different loss functions.

[0054] Specifically, the knowledge transfer of the trained local model using the average model output and the gradient may include: training the trained local model to minimize the loss function using the average model output and the gradient. The global average minimization loss function can be expressed as:

[0055]

[0056] In this embodiment, after performing knowledge transfer on the trained local model using the average model output and the gradient, the process may further include: determining whether all current user-side local models have converged, and if so, stopping the knowledge transfer process; if not, periodically performing the knowledge transfer process until all user-side local models have converged. Figure 2 As shown, each round of model training includes user-side operations and server-side operations. During each round of model training, it is determined whether all current user-side local models have converged. If they have converged, the entire knowledge transfer process is stopped. If they have not converged, the knowledge transfer process is executed periodically until all user-side local models converge. That is, the local model training of each user side and the global knowledge distillation of all user sides are executed periodically until all user-side local models converge.

[0057] It can be seen that the embodiment of the present application first uses the private data set of each user terminal to train the local model of the user terminal to obtain multiple trained local models, and then inputs the same public data into each of the trained local models to obtain corresponding multiple model output results, and uploads all the model output results to the central server so that the central server can use the model output results to train the discriminator model, and then calculates the average of all the model output results through the central server to obtain the average model output, and sends the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal can use the average model output and the gradient to perform knowledge transfer on the trained local model. The embodiment of the present application proposes a federated knowledge distillation scheme based on adversarial learning, which uses the discriminator model located on the central server to measure the efficiency of knowledge transfer between user terminals, and at the same time feeds back to the user terminal to guide the user's local model training, thereby realizing flexible and efficient knowledge transfer between multiple user terminals, and ultimately enabling each user terminal to obtain a local model with high accuracy on the global private data set.

[0058] This application embodiment discloses a specific federated knowledge distillation method applied to the user side, see Figure 3 As shown, the method includes:

[0059] Step S21: Train the local model using the local private dataset to obtain a trained local model.

[0060] In this embodiment, each user terminal has a different private data set stored locally. When federated knowledge distillation is required, each user terminal first trains its local model using the local private data set to obtain a trained local model corresponding to each user terminal.

[0061] Step S22: input the same public data into the trained local model to obtain a model output result.

[0062] In this embodiment, after each user terminal trains the local model using the local private data set to obtain a trained local model, the same public data previously set for all user terminals is further input into the trained local model to obtain a corresponding model output result.

[0063] Step S23: upload the model output result to the central server, so that the central server trains a discriminator model using the model output result and other model output results uploaded by the user terminals, and calculates the average of all model output results to obtain an average model output.

[0064] In this embodiment, after the same public data is input into the trained local model to obtain a model output result, the model output result can be further uploaded to the central server. When the central server obtains the model output results uploaded by each user terminal, a discriminator model is trained using all the model output results, and the average of all the model output results is calculated to obtain a corresponding average model output. Then, the loss function of the trained discriminator model with respect to the gradient of the output of the trained local model and the calculated average model output is returned to each user terminal.

[0065] Step S24: receive the average model output and the loss function of the discriminator model with respect to the gradient of the output of the trained local model returned by the central server, and perform knowledge transfer on the trained local model based on the average model output and the gradient.

[0066] In this embodiment, when the user terminal receives the average model output and the loss function of the discriminator model with respect to the gradient of the output of the trained local model sent by the central server, the trained local model can be knowledge transferred based on the average model output and the gradient.

[0067] Wherein, the more specific processing process of the above steps can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0068] It can be seen that the embodiment of the present application is applied to the user end, and the local model is first trained using a local private data set to obtain a trained local model, and then the same public data is input into the trained local model to obtain a model output result, and then the model output result is uploaded to the central server, so that the central server can use the model output result and the model output result uploaded by other user ends to train the discriminator model, and calculate the average of all model output results to obtain the average model output, and then receive the average model output returned by the central server and the gradient of the loss function of the discriminator model relative to the trained local model output, and perform knowledge transfer on the trained local model based on the average model output and the gradient. An embodiment of the present application provides a flexible and efficient federated knowledge distillation solution for local model heterogeneity and non-independent and identically distributed data between user terminals. In view of the heterogeneity of user terminal models, knowledge transfer between user terminals is achieved through knowledge distillation without the need to share user model parameters. In view of data heterogeneity, the knowledge transfer method based on the average output of all user terminal local models in the existing federated knowledge distillation is abandoned. In this embodiment of the present application, a discriminator model is trained on a central server to measure the efficiency of knowledge transfer, and to guide flexible and efficient knowledge transfer between user terminals.

[0069] This application embodiment discloses a specific federated knowledge distillation method applied to a central server, see Figure 4 As shown, the method includes:

[0070] Step S31: Obtain the model output results obtained by inputting the same public data uploaded by all user terminals into the trained local model located at the user terminal; wherein the trained local model is a model obtained by training the local model of the user terminal based on a private data set.

[0071] In this embodiment, the model output results uploaded by all user terminals can be obtained through the central server. It should be pointed out that the model output result is the model output result obtained by each user terminal inputting the same public data into its own trained local model. The trained local model is the model obtained after the user terminal uses the local private data set to train the local model.

[0072] Step S32: using the model output results to train a discriminator model.

[0073] In this embodiment, after the central server obtains the model output results uploaded by all user terminals by inputting the same public data into the trained local model located at the user terminal, it further trains a discriminator model using all the received model output results.

[0074] Step S33: Calculate the average value of all the model output results to obtain the average model output, and send the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal uses the average model output and the gradient to perform knowledge transfer on the trained local model.

[0075] In this embodiment, after a discriminator model is trained using the model output results, the average of all the above model output results is calculated to obtain an average model output, and then the above average model output and the gradient of the loss function of the above discriminator model relative to the above trained local model output are sent to the corresponding user terminal. When the user terminal obtains the average model output and the gradient sent by the above central server, the above average model output and the gradient are further used to perform knowledge transfer on the trained local model.

[0076] For more specific processing procedures of the above steps, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0077] It can be seen that the embodiment of the present application is applied to the central server, which first obtains the model output results uploaded by all user terminals by inputting the same public data into the trained local model located at the user terminal, and then uses the model output results to train the discriminator model, and then calculates the average value of all the model output results to obtain the average model output, and sends the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal uses the average model output and the gradient to perform knowledge transfer on the trained local model. Compared with the traditional federated learning method, the embodiment of the present application takes into account the black-box heterogeneous model characteristics between user terminals, and realizes knowledge transfer and collaborative learning between multiple user terminals through knowledge distillation. In addition, there is no need to share model parameters between user terminals, and only the model output with a smaller amount of data needs to be shared. Therefore, the present application has lower communication costs; furthermore, compared with the traditional federated knowledge distillation method, the non-independent and identically distributed data characteristics between user terminals are taken into account, and a discriminator is trained on the server side to form adversarial learning with the user terminal, thereby realizing flexible knowledge transfer between user terminals; in addition, by designing a multi-task discriminator model, the decision boundary between data categories can be further increased, and efficient knowledge transfer between user terminals can be realized, so that each user can obtain higher model accuracy.

[0078] Correspondingly, the present application embodiment also discloses a federal knowledge distillation device, see Figure 5 As shown, the device includes:

[0079] The model training module 11 is used to train the local model of each user terminal using the private data set of the user terminal to obtain multiple trained local models;

[0080] A common data input module 12 is used to input the same common data into each of the trained local models to obtain corresponding multiple model output results;

[0081] A model output result uploading module 13 is used to upload all the model output results to a central server so that the central server can use the model output results to train a discriminator model;

[0082] The mean calculation module 14 is used to calculate the mean of all the model output results through the central server to obtain the average model output;

[0083] A data sending module 15 is configured to send the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal;

[0084] The knowledge transfer module 16 is used for the user terminal to perform knowledge transfer on the trained local model using the average model output and the gradient.

[0085] Among them, the specific work processes of the above modules can refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0086] It can be seen that in the embodiment of the present application, the local model of the user terminal is first trained using the private data set of each user terminal to obtain multiple trained local models, and then the same public data is input into each of the trained local models to obtain corresponding multiple model output results, and all the model output results are uploaded to the central server so that the central server can use the model output results to train the discriminator model, and then the central server calculates the average value of all the model output results to obtain the average model output, and sends the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal, so that the user terminal can use the average model output and the gradient to perform knowledge transfer on the trained local model. The embodiment of the present application uses the discriminator model located on the central server to achieve flexible and efficient knowledge transfer between multiple user terminals and reduce communication costs.

[0087] In some specific embodiments, the knowledge transfer module 16 may specifically include:

[0088] A loss function minimization training unit is used to use the average model output and the gradient to perform loss function minimization training on the trained local model.

[0089] In some specific embodiments, the knowledge transfer module 16 may further include:

[0090] A judgment unit, used to judge whether all current user-side local models have converged;

[0091] A stop execution unit, configured to stop executing the knowledge transfer process if convergence occurs;

[0092] The execution unit is used to periodically execute the knowledge migration process if convergence does not occur until all user-side local models converge.

[0093] In some specific embodiments, the private dataset includes input features and corresponding data labels.

[0094] In some specific embodiments, the discriminator model is used to identify the user end to which the user-end local model belongs, and to form adversarial learning with the user-end local model training so that all the user-end local models have the same model output probability distribution, and to classify the data labels corresponding to the outputs of the user-end local models to expand the decision boundaries between different data categories.

[0095] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0096] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the federated knowledge distillation method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may be a computer.

[0097] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0098] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0099] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including computer programs capable of implementing the federated knowledge distillation method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of performing other specific tasks.

[0100] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned federated knowledge distillation method. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0102] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0104] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0105] The above is a detailed introduction to the federated knowledge distillation method, device, equipment and storage medium provided by this application. Specific examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core ideas. At the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A method for knowledge transfer between user terminals located at the edge of a network, characterized in that: include: Using the private dataset of each user terminal to train the local model of the user terminal, respectively, to obtain multiple trained local models; The private data set is user data distributed on the edge of the network; The same public data is input into each of the trained local models to obtain corresponding multiple model output results, and all the model output results are uploaded to a central server so that the central server can use the model output results to train a discriminator model; the public data is a public data set jointly maintained by each of the user terminals on the network edge, and the discriminator model is used to perform a first task and a second task; wherein the first task is to identify the user terminal to which each model output result belongs, and to form adversarial learning with the local model of the user terminal on the network edge, and the second task is to classify the data labels corresponding to the local model output of the user terminal; Calculating the average of all the model output results by the central server to obtain an average model output, and sending the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal; The user terminal uses the average model output and the gradient to perform knowledge migration on the trained local model to complete the knowledge migration between the user terminals on the edge of the network.

2. The method for knowledge transfer between user terminals located at the edge of a network according to claim 1, characterized in that: The performing knowledge transfer on the trained local model by using the average model output and the gradient includes: The trained local model is trained by minimizing a loss function using the average model output and the gradient.

3. The method for knowledge transfer between user terminals located at the edge of a network according to claim 1, characterized in that: After performing knowledge transfer on the trained local model using the average model output and the gradient, the method further includes: Determine whether all current user-side local models converge. If so, stop executing the knowledge migration process. If not, periodically execute the knowledge migration process until all user-side local models converge.

4. The method for knowledge transfer between user terminals located at the edge of a network according to any one of claims 1 to 3, characterized in that: The private dataset includes input features and corresponding data labels.

5. The method for knowledge transfer between user terminals located at the edge of a network according to claim 4, characterized in that: The discriminator model is used to identify the user end to which the user end local model belongs, and to form adversarial learning with the user end local model training so that all the user end local models have the same model output probability distribution, and to classify the data labels corresponding to the output of the user end local model to expand the decision boundary between different data categories.

6. A knowledge migration device between user terminals located at the edge of a network, characterized in that: include: A model training module is used to train the local model of each user terminal using the private data set of the user terminal to obtain multiple trained local models; The private data set is user data distributed on the edge of the network; A public data input module, configured to input the same public data into each of the trained local models to obtain corresponding multiple model output results; the public data is a public data set jointly maintained by each of the user terminals on the network edge; A model output result uploading module is used to upload all the model output results to a central server so that the central server can use the model output results to train a discriminator model; the discriminator model is used to perform a first task and a second task; wherein the first task is to identify the user end to which each model output result belongs and to form adversarial learning with the local model of the user end at the network edge; the second task is to classify the data labels corresponding to the local model output of the user end; A mean calculation module, configured to calculate the mean of all the model output results through the central server to obtain an average model output; A data delivery module, configured to deliver the average model output and the gradient of the loss function of the discriminator model relative to the trained local model output to the corresponding user terminal; A knowledge transfer module is used to perform knowledge transfer on the trained local model through the user terminal using the average model output and the gradient, so as to complete the knowledge transfer between the user terminals on the edge of the network.

7. An electronic device, characterized in that: It comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, it implements the knowledge migration method between user terminals located at the edge of the network as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, the knowledge migration method between user terminals located at the edge of the network as described in any one of claims 1 to 5 is implemented.

Images