Classification prediction model training method, classification prediction method, device and storage medium

Abstract

The present invention discloses a classification prediction model training method, classification prediction method, device, and storage medium. The training method includes each client using a local training data set to train a local classification prediction model, calculate prototypes of various types of data in the local training data set, and calculate soft decisions of a public data set; a central server aggregating all prototypes and all soft decisions, constructing an optimization objective function using the aggregated prototypes and aggregated soft decisions, and training a global classification prediction model to calculate soft decisions of the public data set; each client using the received soft decisions and the public data set to train the local classification prediction model; when the number of loop rounds equals the set number of loop rounds, the trained local classification prediction models and the global classification prediction model are obtained. The present invention can reduce communication overhead between the server and the client, achieve personalized model architecture, and improve model accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of privacy computing technology, and in particular relates to a classification prediction model training method, a classification prediction method, an electronic device and a storage medium based on a federated knowledge distillation algorithm. Background Art

[0002] Due to the rapid development of mobile devices (such as phones, watches, and computers) and advances in sensor technology, a vast amount of data (private user data, such as personal images) is being collected by edge devices. With the rapid advancement of artificial intelligence (AI), this private data is often aggregated and stored in the cloud, where it is used in conjunction with machine learning and deep learning models to enable various intelligent applications. However, uploading sensitive raw data to the cloud over the network and centrally processing private data there poses serious privacy risks for data donors. Driven by the need to protect data privacy and security, the concept of federated learning has emerged. Unlike centralized learning, federated learning supports collaborative learning of a global model across distributed computing nodes using local data. Raw data is not sent to the cloud; only learned global model updates are submitted to the cloud for aggregation. The global model on the cloud is then updated and sent back to the distributed computing nodes for the next round of iteration. This iterative approach allows global models to be learned without compromising user privacy. In addition to improving data privacy, federated learning offers numerous other benefits, such as increased security, autonomy, and efficiency.

[0003] With the development of federated learning, many new challenges have also emerged. The main challenges come from two aspects:

[0004] (1) Traditional federated learning algorithms share model parameters at each iteration, which means that the communication overhead is too large. Existing deep learning models may have millions of parameters. For example, MobileBRET is a deep learning model structure for natural language processing tasks with 25 million parameters, corresponding to a memory size of 96MB. Mobile devices at the edge are often subject to bandwidth constraints. Each round of communication requires exchanging 96MB of information, which is challenging for mobile devices. This makes many mobile devices unable to participate in federated learning tasks that require large parameter interactions.

[0005] (2) The heterogeneity problem poses a huge challenge to deploying federated learning systems in real-world scenarios. On the one hand, there is the problem of model heterogeneity. Most mobile devices participating in federated learning tasks have different computing resources and bandwidth resources. Mobile devices do not have sufficient bandwidth or computing power to train large deep learning models. This means that different participants may need models with different architectures for training, and the federated learning architecture based on model parameter interaction cannot meet the needs of participants using models with different architectures. On the other hand, there is the problem of data heterogeneity. The local data distribution of each mobile device participating in the federated learning task is not globally independent and identically distributed. Simply aggregating the model parameters of the mobile device client may hinder the convergence of the model.

[0006] Based on the above constraints, the global model trained by the federated learning task may not have high accuracy in real practice. Summary of the Invention

[0007] The purpose of the present invention is to provide a classification prediction model training method, classification prediction method, device and storage medium based on the federated knowledge distillation algorithm to solve the problems of excessive communication overhead of traditional federated learning algorithms, inability of traditional federated learning algorithms to meet the needs of participants using different architecture models, and inability to improve model accuracy due to data heterogeneity.

[0008] The present invention solves the above technical problems through the following technical solutions: a classification prediction model training method based on a federated knowledge distillation algorithm, comprising the following steps:

[0009] Step 1: Construct a central server and N clients C = {C 1 ,C 2 ,...,C i ,...,C N}, where N≥2, C i represents the i-th client;

[0010] Step 2: Each of the client C i Both construct local training datasets D with annotations locally i and local classification prediction model X i , and let the cycle round t = 1;

[0011] Step 3: Each of the client C i Use local training dataset D locally i For the local classification prediction model X i Perform iterative training and use the trained local classification prediction model X i Calculate the local training data set D i Prototypes of various types of data Among them, the prototype of class k data Refers to the local classification prediction model X for class k data i Output the average value of the feature vector;

[0012] Each of the clients C i Utilize the trained local classification prediction model X locally i Calculate the public dataset D without annotations P Soft decision Wherein, the soft decision Refers to the local classification prediction model X i The predicted output of

[0013] Step 4: All clients C will calculate their own prototypes and soft decision-making Sending to the central server;

[0014] Step 5: The central server aggregates all received prototypes and soft decisions of each class to obtain aggregated prototypes and aggregated soft decisions; uses the aggregated prototypes and aggregated soft decisions to construct an optimization objective function, and uses the public dataset D P Iteratively train the constructed global classification prediction model using the optimized objective function;

[0015] The unannotated public dataset D is calculated using the trained global classification prediction model. P Soft decision Wherein, the soft decision Refers to the prediction output of the global classification prediction model;

[0016] Step 6: The central server sends the soft decision Sent to each of the client c i ;

[0017] Step 7: Each of the client C i Using the received soft decisions and public dataset D P For the local classification prediction model X i Conduct iterative training;

[0018] Step 8: Determine whether the cycle round t is equal to the set round. If so, obtain the trained local classification prediction model X i and the global classification prediction model; otherwise, let t=t+1 and jump to step 3.

[0019] Furthermore, both the local classification prediction model and the global classification prediction model adopt a deep residual network model.

[0020] Furthermore, for the client C i , the prototype of class k data The specific calculation formula is:

[0021]

[0022] Among them, D k represents a dataset with class k, R w (·) represents the local classification prediction model X i The input layer and hidden layer network, (x j ,y j )∈D k Represents the dataset D k All data in x j represents the jth input sample, y j Represents the input sample x j Corresponding annotations.

[0023] Furthermore, the aggregation formula for aggregating all prototypes of class k is:

[0024]

[0025] Among them, N k represents the number of clients that have prototypes of class k, P k Represents the prototype of class k after aggregation;

[0026] The aggregation formula for aggregating all soft decisions is:

[0027]

[0028] in, represents the soft decision after aggregation.

[0029] Furthermore, the specific expression of the optimization objective function constructed using the aggregated prototypes and aggregated soft decisions is:

[0030]

[0031]

[0032] Among them, (x j ,k)∈D P Denotes the unlabeled public dataset D P , k represents the sample x j The predicted label of all soft decisions Determined by the distribution after aggregation; L2(·) is the relative entropy loss function; C i ∈C N Represents all clients; αj Represents client C i Soft decision The weight of represents the soft decision of the global classification prediction model; L M (·) represents the root mean square loss function; N k represents the number of clients with prototypes of class k; λ represents a hyperparameter; Represents client C i Prototype based on class k data; Represents sample x j The hidden layer output of the global classification prediction model; M represents the number of categories in the classification task.

[0033] Based on the same inventive concept, the present invention further provides a classification prediction method based on a classification prediction model, wherein the classification prediction model includes a global classification prediction model and N local classification prediction models, wherein the global classification prediction model and the local classification prediction model are trained by any of the above-mentioned classification prediction model training methods based on the federated knowledge distillation algorithm, and the classification prediction method includes the following steps:

[0034] Obtain data to be classified;

[0035] The classification prediction model is used to perform classification prediction on the data to be classified to obtain the category of the data to be classified.

[0036] Based on the same inventive concept, the present invention further provides an electronic device, comprising:

[0037] memory for storing computer programs;

[0038] A processor is used to implement the steps of the classification prediction model training method based on the federated knowledge distillation algorithm described above, or implement the steps of the classification prediction method based on the classification prediction model described above when executing the computer program.

[0039] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the classification prediction model training method based on the federated knowledge distillation algorithm described above, or implements the steps of the classification prediction method based on the classification prediction model described above.

[0040] Beneficial effects

[0041] Compared with the prior art, the advantages of the present invention are:

[0042] The present invention provides a classification prediction model training method, classification prediction method, electronic device and storage medium. The client's private data and local classification prediction model are stored locally on the client, ensuring the privacy and security of private data. Knowledge distillation is used to improve traditional federated learning based on model parameter interaction to soft decision interaction based on model output, greatly reducing the communication overhead between the server and the client. At the same time, the client and server are allowed to select a model with a suitable architecture based on their own bandwidth resources and computing resources, thereby realizing the personalization of the model architecture.

[0043] At the same time, the present invention also alleviates the problem of difficulty in improving model accuracy due to the high heterogeneity of client private data through the prototype network, greatly improving the model accuracy. The federated learning framework using the method of the present invention is stable and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a flow chart of knowledge distillation in an embodiment of the present invention;

[0046] Figure 2 This is a flow chart of the federated knowledge distillation algorithm in an embodiment of the present invention;

[0047] Figure 3 This is a flow chart of a classification prediction model training method based on a federated knowledge distillation algorithm in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following is a clear and complete description of the technical solutions of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0049] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0050] In order to solve the problems of high communication overhead of traditional federated learning algorithms based on model parameter interaction, the inability of traditional federated learning algorithms to meet the needs of participants using models with different architectures, and the inability to improve model accuracy due to data heterogeneity, the present invention applies knowledge distillation and prototype networks to federated learning.

[0051] Knowledge distillation is a method used in machine learning algorithms for model compression. Its purpose is to use a pre-trained teacher model to inject its knowledge into an untrained student model. Knowledge distillation differs from standard model training, which attempts to make the model's predictions match the true label values of each sample (for example, [cat, dog] = [0, 1]). However, knowledge distillation attempts to make the student model's predictions match the teacher model's predictions for the same sample, namely logits (soft decisions), for example, [cat, dog] = [0.3, 0.7]. These logits contain more information than the true label values, allowing for faster training of the student model compared to standard models.

[0052] Consider a simple neural network, which includes an input layer, a hidden layer and an output layer. Let F w (·) represents the entire network, R w (·) represents the input layer and hidden layer network, O w (·) represents the output layer network. In the multi-classification problem, for a sample x j , the output layer will eventually output a classification prediction result In knowledge distillation, the student model updates the weights of its own neural network by minimizing its own loss function and distillation regularizer. The optimization goal of the student model is as follows:

[0053]

[0054] Among them, (x j ,y j )∈D represents a public dataset, λ represents a fixed hyperparameter, and y j Represents sample x j The true label value, L1(·) represents the cross entropy loss function, L2(·) represents the relative entropy loss function, represents the output logits (soft decisions) of the teacher model, Represents the output logits (soft decision) of the student model, and its process is as follows Figure 1 shown.

[0055] Based on the knowledge distillation described above, it is applied to federated learning. In this case, the server and client in the federated learning framework no longer exchange model parameters, but rather the model's soft decisions. A public dataset is also deployed globally to transfer model knowledge. However, due to the data privacy requirements of federated learning, the public dataset is unlabeled data. The basic process of the federated knowledge distillation algorithm is as follows:

[0056] (1) Each client trains a model locally using private data and uploads the logistic (soft decision) of the client model on the public dataset to the server in each round of communication;

[0057] (2) After the server receives the logits (soft decisions) uploaded by all clients participating in federated learning, it aggregates them to obtain the final logits (soft decisions). At this time, the server model serves as the student model, and all client models serve as teacher models for knowledge distillation training;

[0058] (3) After the knowledge distillation training is completed, the server sends the logits (soft decisions) of the server model (student model) for the public dataset to each client;

[0059] (4) After the client receives the logits (soft decisions) from the server, the client model becomes the student model and the server model becomes the teacher model. Knowledge distillation training is performed again. After training, the client returns to step (1) to start the next round of federated learning training until global convergence.

[0060] Since the public dataset is unlabeled data, the optimization goal of the server model during knowledge distillation training becomes:

[0061]

[0062] Among them, x j ∈D P represents an unlabeled public dataset, C i ∈C N Represents all clients participating in federated learning, For client C i Model logits (soft decision), α i For client C i The weights of the model logits (soft decisions), is the server model logits (soft decision), L2(·) is the relative entropy loss function, and its flow chart is as follows Figure 2 shown.

[0063] By applying knowledge distillation to federated learning, the interaction between models is no longer about model parameters but about model output logits (soft decisions), which greatly reduces communication overhead. After using knowledge distillation, the server and client, as well as between clients, do not need to use models with the same architecture. This also greatly increases the personalization capabilities of the federated learning framework. Each device (client and server) can choose a model with a different architecture based on its own situation.

[0064] However, when using knowledge distillation in a federated learning framework, the distribution of private data between clients is often non-iid (non-independent and identically distributed), which can slow model training convergence and make it difficult to improve model accuracy. This is because when clients' private data is highly heterogeneous, the client-side model's output logits (soft decisions) can be overconfident. This causes the aggregated logits (soft decisions) on the server to contain erroneous knowledge. Using these logits (soft decisions) to train the server model inevitably leads to slow model training convergence and low model accuracy.

[0065] For example, two clients are participating in federated learning. Client A's private data contains images of dogs, cats, and airplanes; client B's private data contains images of cats, frogs, and airplanes. After clients A and B each train their models using their private data, a dog image appears in the public dataset. Since client A's private dataset contains dog images, the distribution of client A's output logits (soft decisions) for this image will correctly favor the dog class. However, since client B's private dataset does not contain dog images but does contain cat images, it is likely that client B's model will incorrectly predict the dog image as a cat. Consequently, the distribution of client B's model output logits (soft decisions) will be biased towards the cat class. Aggregating the logits (soft decisions) output by clients A and B's models will likely result in a relatively even distribution between cats and dogs, slowing convergence of the server model and preventing improvement in model accuracy.

[0066] Based on the above problems, the present invention proposes to use prototype networks to alleviate such phenomena. Prototype networks refer to the average representation of embedded vectors of a certain type of data in the feature space, which is an abstract feature representation of a type of data. Consider a simple neural network, which includes an input layer, a hidden layer and an output layer. Let F w (·) represents the entire network, R w (·) represents the input layer and hidden layer network, O w(·) represents the output layer network, then the prototype network of class k can be expressed as

[0067]

[0068] Among them, D k represents a dataset of class k, (x j ,y j )∈D k Represents the dataset D k All data in.

[0069] The prototype network can distinguish the differences between each class from the feature space level. When applied to federated learning, the server aggregates the client's class prototype network based on private data. Although the client's prototype network is slightly different due to the heterogeneity of data distribution, the server can obtain more abstract feature representations about the class. By learning these abstract feature representations, it can alleviate the model training problem caused by overconfident logits (soft decisions).

[0070] Therefore, an embodiment of the present invention provides a classification prediction model training method based on a federated knowledge distillation algorithm, comprising the following steps:

[0071] Step 1: Construct a central server and N clients C = {C 1 ,C 2 ,...,C i ,...,C N}, where N≥2, C i Represents the i-th client.

[0072] In this embodiment, N=100, and the client is a watch, a mobile phone, a computer, or an IPAD.

[0073] Step 2: Each of the client C i Both construct local training datasets D with annotations locally i and local classification prediction model X i , and let the cycle round t=1.

[0074] Local training dataset D i That is client C i The dataset consists of private data (such as image data), the local classification prediction model X i That is, on the client C iEach client has its own local training dataset and local classification prediction model. These models can have the same or different architectures. These clients participate in federated learning to train a powerful server model for classification tasks, such as image classification, while protecting private data.

[0075] In this embodiment, the local classification prediction model X i Use a deep residual network model, such as the ResNet11 model.

[0076] Step 3: Each of the client C i Use local training dataset D locally i For the local classification prediction model X i Perform iterative training and use the trained local classification prediction model X i Calculate the local training data set D i Prototypes of various types of data Among them, the prototype of class k data Refers to the local classification prediction model X for class k data i Output the mean of the eigenvectors.

[0077] On each client C i , using the local training dataset D i For the local classification prediction model X i Perform iterative training as the existing training process. For client C i , the prototype of class k data The specific calculation formula is:

[0078]

[0079] Among them, D k represents a dataset with class k, R w (·) represents the local classification prediction model X i The input layer and hidden layer network, (x j ,y j )∈D k Represents the dataset D k All data in x j represents the jth input sample, y j Represents the input sample x j The prototype is the average value of the output feature vector of a certain type of image data. For example, client 1 has 100 dog image data and 100 cat image data. Then, these 100 dog image data are input into the local classification prediction model respectively, and 100 model R are obtained. wThe output feature vector of (·) is averaged to get the prototype of dog, and the same is true for the prototypes of other categories.

[0080] Each of the clients C i Utilize the trained local classification prediction model X locally i Calculate the public dataset D without annotations P Soft decision Wherein, the soft decision Refers to the local classification prediction model X i The predicted output.

[0081] The public dataset D P Sample x in j Input to each client C i Local classification prediction model X i In the example, we get the corresponding client C i Public dataset D P Soft decision That is, the local classification prediction model X i For the public dataset D P The predicted output of the sample in .

[0082] In this embodiment, the public dataset D P The CIFAR-10 universal object color image dataset is used. The number of target categories corresponding to this dataset is M = 10, namely: airplane, car, bird, cat, deer, dog, frog, horse, ship and truck. The pixel size of each image is 32×32, and there are 6000 images for each target category.

[0083] The public dataset D P An image is used as a local classification prediction model X i Input, local classification prediction model X i The output of the local classification prediction model X i The prediction result for this image is an array of ten items, each representing the model's predicted distribution for each of the ten categories. For example, if we input an image of a dog, the model will output an array: [..., 0.1, 0.2, 0.5, ...], where the data in the array represents the model's predicted probability for a particular class.

[0084] Step 4: All clients C will calculate their own prototypes and soft decision-making Send to the central server.

[0085] Step 5: The central server aggregates all received prototypes and soft decisions of each class to obtain aggregated prototypes and aggregated soft decisions; uses the aggregated prototypes and aggregated soft decisions to construct an optimization objective function, and uses the public dataset D P The constructed global classification prediction model is iteratively trained with the optimized objective function.

[0086] Aggregation includes average aggregation and weighted aggregation. This embodiment adopts average aggregation. The aggregation formula for aggregating all prototypes of class k is:

[0087]

[0088] Among them, N k represents the number of clients that have prototypes of class k, P k Represents the prototype of class k after aggregation.

[0089] The aggregation formula for aggregating all soft decisions is:

[0090]

[0091] in, represents the soft decision after aggregation.

[0092] The specific expression of the optimization objective function constructed using the aggregated prototypes and aggregated soft decisions is:

[0093]

[0094]

[0095] Among them, (x j ,k)∈D P Denotes the unlabeled public dataset D P , k represents the sample x j The predicted label of all soft decisions Determined by the distribution after aggregation; L2(·) is the relative entropy loss function; C i ∈C N Represents all clients; α i Represents client C i Soft decision The weight of represents the soft decision of the global classification prediction model; L M (·) represents the root mean square loss function; N k represents the number of clients with prototypes of class k; λ represents a hyperparameter; Represents client C i Prototype based on class k data; Represents sample xj The hidden layer output of the global classification prediction model; M represents the number of categories in the classification task.

[0096] In this embodiment, the global classification prediction model adopts a deep residual network model, such as the ResNet56 model.

[0097] The unannotated public dataset D is calculated using the trained global classification prediction model. P Soft decision Wherein, the soft decision Refers to the prediction output of the global classification prediction model.

[0098] Step 6: The central server sends the soft decision Sent to each of the client c i .

[0099] Step 7: Each of the client C i Using the received soft decisions and public dataset D P For the local classification prediction model X i Perform iterative training.

[0100] Step 8: Determine whether the cycle round t is equal to the set round. If so, obtain the trained local classification prediction model X i and the global classification prediction model; otherwise, let t=t+1 and jump to step 3.

[0101] Based on the same inventive concept, the present invention further provides a classification prediction method based on a classification prediction model, wherein the classification prediction model includes a global classification prediction model and N local classification prediction models, wherein the global classification prediction model and the local classification prediction model are trained by the classification prediction model training method based on the federated knowledge distillation algorithm. The classification prediction method includes the following steps:

[0102] Step 1: Obtain the data to be classified;

[0103] Step 2: Use the classification prediction model to perform classification prediction on the data to be classified to obtain the category of the data to be classified.

[0104] In the present invention, the client's private data and local classification prediction model are stored locally on the client, ensuring the privacy and security of private data; knowledge distillation is used to improve the traditional federated learning based on model parameter interaction to soft decision interaction based on model output, which greatly reduces the communication overhead between the server and the client. At the same time, it allows the client and server to select a model with a suitable architecture based on their own bandwidth resources and computing resources, thereby realizing the personalization of the model architecture.

[0105] At the same time, the present invention also alleviates the problem of difficulty in improving model accuracy due to the high heterogeneity of client private data through the prototype network, greatly improving the model accuracy. The federated learning framework using the method of the present invention is stable and efficient.

[0106] The above disclosure is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, and they should all be covered by the scope of protection of the present invention.

Claims

1. A classification prediction model training method based on the federated knowledge distillation algorithm, characterized in that: The following steps are involved: Step 1: Construct a central server and N clients C = {C 1 ,C 2 ,...,C i ,...,C N }, where N≥2, C i represents the i-th client; Step 2: Each of the client C i Both construct local training datasets D with annotations locally i and local classification prediction model X i , and let the cycle round t = 1; where the local training data set D i is the image data; Step 3: Each of the client C i Use local training dataset D locally i For the local classification prediction model X i Perform iterative training and use the trained local classification prediction model X i Calculate the local training data set D i Prototypes of various types of data Among them, the prototype of class k data Refers to the local classification prediction model X for class k data i Output the average value of the feature vector; Each of the clients C i Utilize the trained local classification prediction model X locally i Calculate the public dataset D without annotations P Soft decision Wherein, the soft decision Refers to the local classification prediction model X i The predicted output of Step 4: All clients C will calculate their own prototypes and soft decision-making Sending to the central server; Step 5: The central server aggregates all received prototypes and soft decisions of each class to obtain aggregated prototypes and aggregated soft decisions; uses the aggregated prototypes and aggregated soft decisions to construct an optimization objective function, and uses the public dataset d P Iteratively train the constructed global classification prediction model using the optimized objective function; The unannotated public dataset D is calculated using the trained global classification prediction model. P Soft decision Wherein, the soft decision Refers to the prediction output of the global classification prediction model; Step 6: The central server sends the soft decision Sent to each of the client c i ; Step 7: Each of the client C i Using the received soft decisions and public dataset D P For the local classification prediction model X i Conduct iterative training; Step 8: Determine whether the cycle round t is equal to the set round. If so, obtain the trained local classification prediction model X i and the global classification prediction model; otherwise, let t=t+1 and jump to step 3.

2. The classification prediction model training method based on the federated knowledge distillation algorithm according to claim 1 is characterized in that: The local classification prediction model and the global classification prediction model both adopt a deep residual network model.

3. The classification prediction model training method based on the federated knowledge distillation algorithm according to claim 1 is characterized in that: For the client C i , the prototype of class k data The specific calculation formula is: Among them, D k represents a dataset with class k, R w (·) represents the local classification prediction model X i The input layer and hidden layer network, (x j ,y j )∈D k Represents the dataset D k All data in x j represents the jth input sample, y j Represents the input sample x j Corresponding annotations.

4. The classification prediction model training method based on the federated knowledge distillation algorithm according to claim 1 is characterized in that: The aggregation formula for aggregating all prototypes of class k is: Among them, N k represents the number of clients that have prototypes of class k, P k Represents the prototype of class k after aggregation; The aggregation formula for aggregating all soft decisions is: in, represents the soft decision after aggregation.

5. The classification prediction model training method based on the federated knowledge distillation algorithm according to any one of claims 1 to 4, characterized in that: The specific expression of the optimization objective function constructed using the aggregated prototypes and aggregated soft decisions is: Among them, (x j ,k)∈D P Denotes the unlabeled public dataset D P , k represents the sample x j The predicted label of all soft decisions Determined by the distribution after aggregation; L2(·) is the relative entropy loss function; C i ∈C N Indicates all clients; i Represents client C i Soft decision The weight of represents the soft decision of the global classification prediction model; L M (·) represents the root mean square loss function; N k represents the number of clients with prototypes of class k; λ represents a hyperparameter; Represents client C i Prototype based on class k data; Represents sample x j The hidden layer output of the global classification prediction model; M represents the number of categories in the classification task.

6. A classification prediction method based on a classification prediction model, characterized in that: The classification prediction model includes a global classification prediction model and N local classification prediction models, wherein the global classification prediction model and the local classification prediction model are trained by the classification prediction model training method based on the federated knowledge distillation algorithm according to any one of claims 1 to 5, and the classification prediction method includes the following steps: Obtain data to be classified; The classification prediction model is used to perform classification prediction on the data to be classified to obtain the category of the data to be classified.

7. An electronic device, characterized in that: The device comprises: memory for storing computer programs; A processor, configured to implement the steps of the classification prediction model training method based on the federated knowledge distillation algorithm described in any one of claims 1 to 5 when executing the computer program, or implement the steps of the classification prediction method based on the classification prediction model described in claim 6.

8. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements the steps of the classification prediction model training method based on the federated knowledge distillation algorithm described in any one of claims 1 to 5, or implements the steps of the classification prediction method based on the classification prediction model described in claim 6.

Images