Self-adaptive personalized federal learning method based on knowledge distillation

Abstract

The invention belongs to the field of federated learning technology application, and particularly relates to a knowledge distillation-based self-adaptive personalized federated learning method, which comprises the following steps that: a client generates pseudo data according to local data through a variational automatic encoder, and performs data enhancement on the pseudo data; carrying out intensive training on a teacher model deployed in the server by utilizing the enhanced data; the student model deployed at the client generates a soft prediction label based on an input sample and uploads the soft prediction label to the server, and the server dynamically weights the soft prediction label of each client based on the confidence of the soft prediction label; performing further iterative training by using the dynamically weighted soft label to generate a personalized label, and issuing the personalized label to a corresponding client; the client constructs distillation loss according to the received personalized label to train a student model; according to the method, the federal model performance can be improved in a federal learning scene of data isomerism and model isomerism, the personalized learning effect of the client is enhanced, and the personalized demand of the client is met.

Description

Technical Field

[0001] The present invention belongs to the field of application of federated learning technology, and in particular relates to an adaptive personalized federated learning method based on knowledge distillation. Background Art

[0002] With the further development of big data, people's awareness and concern about data privacy are also increasing; therefore, federated learning has received widespread attention since its proposal and has been applied in some scenarios. The effect of traditional federated learning depends on the model structure and data distribution of the client. However, in actual federated scenarios, the data distribution and model structure of the client are often heterogeneous, and it is difficult for us to train a global model suitable for all clients through federated learning. Therefore, personalized federated learning has become a hot topic in recent research. At present, personalized federated learning has made great progress in the fields of healthcare, financial services, etc.

[0003] The purpose of personalized federated learning (PFL) is to build a personalized model based on the data of the participants while ensuring data privacy. Compared with traditional federated learning, PFL focuses more on providing a personalized model for each participant rather than a unified global model. Currently, researchers divide personalized federation into two categories, global model personalization and learning personalized models. Global model personalization aims to train a global model with high generalization performance, and then fine-tune it on the client to achieve personalization. Learning personalized models is to train a separate personalized model on each client.

[0004] In recent years, more and more scholars have conducted research on personalized federated learning in the field of federated learning. The research areas are mainly based on multi-task learning, knowledge distillation and transfer learning. Based on multi-task learning, it mainly learns the independent model of each node and uses an arbitrary convex loss function to train an independent weight vector for each node; and considers the correlation between node models to solve statistical problems in the federated environment and improve sample capacity. Based on knowledge distillation, it mainly focuses on federated learning in model heterogeneous scenarios, and uses soft prediction labels for exchange instead of model parameters to make federated learning more flexible. Based on transfer learning, it mainly uses the similarity between data, tasks, or models to apply the model learned in the source field to the target field.

[0005] Furthermore, the process of dynamically weighting the soft prediction labels of each client based on the confidence of the soft prediction labels includes:

[0006] Calculate the confidence of the client based on the client's data quality and the client's historical performance;

[0007] Based on the attention mechanism, the attention score is calculated according to the confidence of each client, and the soft label of the corresponding client is weighted according to the attention score.

[0008] Although scholars have achieved a lot of results in personalized federated learning, there are still some challenges:

[0009] 1. The generalization ability of the teacher model is insufficient in data heterogeneity scenarios. In federated learning, the data distribution of the clients participating in federated learning is very different, and there is also the problem of data privacy. Therefore, there is a problem of insufficient generalization ability when building the teacher model.

[0010] 2. Client drift caused by uneven aggregation of client soft prediction labels. Aggregating soft prediction labels from each client in each round of learning has the problem of uneven aggregation, which causes client drift. Therefore, how to effectively balance and aggregate client soft labels has become an important challenge in federated learning.

[0011] 3. The problem of lack of personalized knowledge of local models in heterogeneous environments. In heterogeneous scenarios, local models need the assistance of global knowledge to complete personalization on private data sets. How to effectively obtain global knowledge to assist local clients in personalization is also a challenge people face. Summary of the invention

[0012] To solve the above problems, the present invention provides an adaptive personalized federated learning method for knowledge distillation, comprising the following steps:

[0013] The client generates pseudo data based on local data through a variational autoencoder, uploads the pseudo data to the server, and uses hybrid shearing technology to enhance the pseudo data;

[0014] The teacher model deployed on the server is trained with data augmentation, and the random jitter dropout regularization technique is used during training to enhance the robustness of the teacher model.

[0015] The student model deployed on the client generates soft prediction labels based on the input samples and uploads them to the server. The server dynamically weights the soft prediction labels of each client based on the confidence of the soft prediction labels.

[0016] Use the dynamically weighted soft labels for further iterative training and generate personalized labels, which are then sent to the corresponding clients.

[0017] The client calculates the inter-class prediction loss and intra-class prediction loss based on the received personalized labels, and constructs the distillation loss based on the inter-class prediction loss and intra-class prediction loss to train the student model;

[0018] Determine whether the iteration threshold of federated learning has been reached. If so, end the training and fix the network parameters of each client student model. If not, repeat the above steps.

[0019] Compared with the prior art, the present invention can improve the performance of federated models in federated learning scenarios with heterogeneous data and models, enhance the personalized learning effect of clients, and further meet the personalized needs of clients. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the adaptive personalized federated learning framework based on knowledge distillation of the present invention;

[0021] Figure 2 This is a flow chart of the adaptive personalized federated learning method based on knowledge distillation of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The present invention provides an adaptive personalized federated learning method for knowledge distillation, comprising the following steps:

[0024] The client generates pseudo data based on local data through a variational autoencoder, uploads the pseudo data to the server, and uses hybrid shearing technology to enhance the pseudo data;

[0025] The teacher model deployed on the server is trained with data augmentation, and the random jitter dropout regularization technique is used during training to enhance the robustness of the teacher model.

[0026] The student model deployed on the client generates soft prediction labels based on the input samples and uploads them to the server. The server dynamically weights the soft prediction labels of each client based on the confidence of the soft prediction labels.

[0027] Use the dynamically weighted soft labels for further iterative training and generate personalized labels, which are then sent to the corresponding clients.

[0028] The client calculates the inter-class prediction loss and intra-class prediction loss based on the received personalized labels, and constructs the distillation loss based on the inter-class prediction loss and intra-class prediction loss to train the student model;

[0029] Determine whether the iteration threshold of federated learning has been reached. If so, end the training and fix the network parameters of each client student model. If not, repeat the above steps.

[0030] In existing federated learning technologies, clients participating in federated learning have data heterogeneity and model heterogeneity problems, making it difficult to train a model suitable for all clients through federated learning. In existing technologies, knowledge distillation is used to allow clients to use different model architectures, but there is a problem of insufficient generalization ability of the teacher model in scenarios with data heterogeneity and privacy requirements; at the same time, the uneven aggregation of soft prediction labels on the client may also cause client drift problems, and there is also a problem of lack of personalized knowledge of local models in heterogeneous environments. In heterogeneous scenarios, how to effectively obtain global knowledge to assist local clients in personalization is also a challenge.

[0031] In view of the above problems, the present invention improves the training and distillation process of the teacher model, and proposes an adaptive personalized federated learning method based on knowledge distillation. The method generates a pseudo data set by using a VAE generation model on each client model, and obtains an enhanced teacher model through the generated pseudo data set and data enhancement without directly contacting the real data, thereby improving the generalization ability of the teacher model; by introducing an attention mechanism, the confidence of each soft prediction label is evaluated at the same time, and the weight during aggregation is dynamically adjusted to alleviate the problem of model client drift, and the teacher model is used to generate intra-class labels and inter-class labels according to the data characteristics of each client, thereby giving the client model the right to freely match the output of the teacher model, and the similarity with the previous round of model is added to the loss to achieve personalized customization of the local model.

[0032] This paper proposes an adaptive personalized federated learning method based on knowledge distillation. This method uses VAE and data enhancement to obtain an enhanced teacher model without contacting real data. At the same time, a dynamic weighting method based on the attention mechanism is used in the aggregation process to alleviate the client drift problem. Finally, personalized soft prediction tags are used to achieve personalized model customization of local clients. Figure 1 As shown, the present invention mainly includes 3 parts:

[0033] 1. N clients generate pseudo data sets based on local data through variational autoencoders, use mixed shearing technology for data enhancement, and input the enhanced samples into the teacher model for enhanced training; the pseudo data sets and data enhancement enable the teacher model to have stronger client data distribution perception and generalization capabilities, and the random jitter drop regularization technology is used to enhance the robustness of the teacher;

[0034] Second, the client generates soft prediction labels based on the input samples and uploads them to the server. The soft prediction labels of different clients are dynamically weighted through the attention mechanism. The teacher model is further iterated based on the weighted soft prediction labels to adaptively learn the personalized features of each client.

[0035] 3. Different data distributions of different clients require different knowledge to be obtained from the teacher model. Therefore, in the present invention, the teacher model generates personalized soft prediction labels through inter-class prediction and intra-class prediction. The personalized soft prediction labels are sent to each client through the server, and the client model is trained iteratively based on the personalized soft prediction labels and the model parameters of the previous round. In this way, a personalized model that is more suitable for the local data distribution is obtained.

[0036] In one embodiment, a specific process of a personalized federated learning method based on parameter stratification is as follows: Figure 2 As shown, the following steps are included:

[0037] S1, N clients generate pseudo data sets based on local data through variational autoencoders, use hybrid shearing technology to enhance data, and input the enhanced samples into the teacher model for enhanced training;

[0038] S2. Through pseudo data sets and data enhancement, the teacher model has stronger client data distribution perception and generalization capabilities, and the random jitter drop regularization technology is used to enhance the robustness of the teacher model;

[0039] S3. The client generates soft prediction labels based on the input samples and uploads them to the server. The soft prediction labels of different clients are dynamically weighted through the attention mechanism.

[0040] S4. The teacher model is further iterated based on the weighted soft prediction labels, so that it adaptively learns the personalized features of each client and generates personalized soft prediction labels through inter-class prediction and intra-class prediction;

[0041] S5. Personalized soft prediction labels are sent to each client through the server, and the inter-class prediction and intra-class prediction losses are added to the distillation loss, which can give the client model the right to adaptively match the output of the teacher model. In addition, an additional regularization term is added to the distillation loss. This regularization term mainly controls the similarity between the local model of the previous round and the local model of the current round.

[0042] S6. Determine whether the iteration threshold of federated learning is reached. If so, proceed to step S7, otherwise return to step S3;

[0043] S7. Fix the client model parameters to obtain a personalized model suitable for local needs.

[0044] In this embodiment, each client is trained on a local dataset using VAE, and the generated pseudo dataset The enhanced teacher model is obtained by data augmentation without direct contact with real data, thereby improving the generalization ability of the teacher model. The specific process includes: In order to ensure that the samples newly generated by VAE are used The data distribution of the generated sample is similar to that of the original sample x, and the similarity evaluation index L between the generated sample and the original sample is mse (mean square error function) is introduced into the training loss function.

[0045] z=μ+εσ,ε~N(0,1)

[0046]

[0047] in, represents the loss function of the variational autoencoder, represents the new sample generated by the variational autoencoder based on the original sample x, θ represents the parameters of the encoder in the variational autoencoder, represents the parameters of the decoder in the variational autoencoder; Representation sample and the mean square error function between x and sample x; KL(·) represents the divergence; N(μ, σ) represents a Gaussian distribution with expectation μ and variance σ, N(0, 1) represents a Gaussian distribution with expectation 0 and variance 1, and the expectation μ and variance σ are learnable parameters; z is the latent variable of the variational autoencoder.

[0048] Secondly, CutMix is ​​used for data enhancement, and the mixed samples are input into the teacher model for enhanced training, namely:

[0049]

[0050] in, Represents two different images x i , x′i; ⊙ represents pixel-by-pixel multiplication; M represents a binary mask whose size is the same as the image x′ i Similarly, the mask is a random mask, and the number of regions with values ​​of 1 in the mask is related to λ. For example, if λ is 0.7, 70% of the elements in the mask M have values ​​of 1. Represents two different images x i , x′ i Label y i , y′ i The new label obtained by mixing; λ is a random number that obeys the (0, 1) uniform distribution.

[0051] The ShakeDrop (random jitter drop regularization) module is embedded in the teacher model to enhance the robustness of the teacher model. After adding the regularization term, the goal of model training is not only to reduce the task loss, but also to limit the size of the model weight to avoid overfitting problems caused by overly complex models, thereby achieving the purpose of enhancing the generalization ability of the teacher model. Then define the loss function L Mix , as shown below:

[0052]

[0053] In the above formula, L Mix (x i ,y i , x′ i , y′ i ; θ′) represents the loss function when the teacher model uses the data after data enhancement for intensive training, x i , x′ i Represents the data sample before data enhancement, x′ i , y′ i Represents data sample x i , x′ i The corresponding labels, θ′ represents the parameters of the teacher model; L CE (x i ,y i ; θ′) indicates that the input data sample in the teacher model is x i , the true label is y i The cross entropy loss when ; represents the mixed sample loss, According to the data sample x i , x′ i The data samples obtained after data enhancement; λ is a random number that obeys the (0, 1) uniform distribution; Indicates that the data sample input in the teacher model is The true label is y i The cross entropy loss when ; Indicates that the data sample input in the teacher model is The true label is y′ i The cross entropy loss when L SD (θ′) denotes the random jitter dropout regularization term.

[0054] The process of dynamically weighting different clients through the attention mechanism includes:

[0055] S31. First, evaluate the confidence of the soft prediction label from the client. The confidence of the soft prediction label C k and the client's data quality Q k , historical performance kIt mainly reflects the reliability of the client in the learning process of the teacher model, namely:

[0056]

[0057] C k =α·Q k +β·H k

[0058]

[0059] Among them, C k is the confidence of client k; Q k is the data quality of client k; H k is the historical performance of client k. In this embodiment, the accuracy of client k’s prediction in the last round of training is as the historical representation of client k; α and β are Q k , H k The weight factor of ; missing represents the number of erroneous data in the client data; outliers represents the number of missing values ​​in the client data, and total represents the total number of client data.

[0060] S32, attention score A i It is calculated through the soft attention mechanism. The soft attention mechanism usually uses the softmax function to process the confidence score to calculate the attention score of each client. The specific expression is as follows:

[0061]

[0062] Each local model M n exist Perform training to warm up, and then Conduct training, including is the generated global pseudo dataset, D i Represents the private dataset of the i-th client, and the local soft prediction obtained in each round of training Upload to the server to form a soft prediction set Then the soft prediction sets from different devices are fused, which is specifically expressed as:

[0063]

[0064] Among them, Y g Represents the data after the soft prediction sets from different devices are integrated; Represents the prediction result of the i-th sample of the t-th training local model.

[0065] The teacher model is further iterated based on the weighted soft prediction to adaptively learn the personalized features of each client, and the process of generating personalized soft prediction labels through inter-class prediction and intra-class prediction includes:

[0066] S41, the process of further iterating the teacher model based on the weighted soft prediction label to adaptively learn the personalized features of each client is:

[0067] The weighted soft predictions are transferred to the teacher model by distillation. The specific process is as follows:

[0068]

[0069] In the above formula, KL represents KL(·) divergence, T represents the temperature parameter of distillation, softmax() represents the softmax activation function, and Y g represents the soft prediction from the client after being aggregated by the server, Y t Represents the soft prediction of the teacher model. Based on the above method, this embodiment specifically defines the loss function as follows:

[0070] L θ =L CE (x i ,y i ;θ)+λ1L UD

[0071] From the above formula, we can see that the loss function mainly consists of two parts, one part L CE (x i ,y i ; θ) is the cross entropy loss, and the other part is the distillation loss L UD , λ1 is the weight.

[0072] S42. Due to the heterogeneity of client data, consider that the client acquires knowledge from two aspects of the teacher model: one is the prediction distribution of the teacher model for different categories, and the other is the prediction distribution for different instances under the same category. The specific measurement formula is as follows:

[0073]

[0074] Where B represents the training batch size; represents the probability distribution of the predicted labels of the i-th batch of training data of the student model, represents the probability distribution of the predicted labels of the i-th batch of training data of the teacher model; represents the prediction of the i-th batch of training data by the student model, represents the prediction of the i-th batch of training data by the teacher model; represents the probability distribution of the j-th category prediction label of the student model, represents the probability distribution of the j-th category prediction label of the teacher model; represents the j-th category prediction of the student model, represents the j-th category prediction of the teacher model.

[0075] This embodiment adopts d p (,) is used to measure the correlation between two coefficients. In this embodiment, the calculation is based on the Pearson coefficient. The measurement method of any two vectors, such as vector x and vector y, is expressed as:

[0076] d p (x, y) = 1 - ρ p (x, y)

[0077]

[0078] In the above formula, ρ p (x, y) represents the Pearson correlation coefficient of vector x and vector y, Cov(x, y) is the covariance of vector x and vector y, Std(x) and Std(y) represent the standard deviation of vector x and vector y respectively. Respectively represent the mean of the elements in vector x and vector y, x i ,y i Respectively represent the i-th element in vector x and vector y; A represents the number of measurement types, for example, in the calculation When the measurement type is the training batch size, in the calculation , the measure type is the number of categories.

[0079] In this embodiment, the training iteration process of the client model based on the personalized soft prediction label and the model parameters of the previous round includes:

[0080] Adding the inter-class and intra-class losses from step S4 to the distillation loss can give the client model the right to adaptively match the output of the teacher model;

[0081] In addition, consider adding an additional regularization term to the distillation loss. This regularization term mainly controls the local model of the previous round. and the local model of the current round The similarity between them is mainly to ensure the personalization of the local model; this regularization term can be expressed as:

[0082]

[0083] Among them, ||·|| means finding the norm.

[0084] In summary, the distillation loss is as follows:

[0085]

[0086] in, Represents the loss function of the student model for the tth training; represents the cross entropy loss of client k during the tth training process; L out Represents the prediction distribution loss for different categories; L in represents the predicted distribution of the same category under different instances; L m represents a regularization term, which in the present invention is the norm of the difference between the network parameters of the student model in the current training round and the network parameters of the student model in the previous training round; β, γ, and μ are all loss weights. By using the above loss function, different clients can obtain different knowledge from the teacher model, and the purpose of local model personalization can be achieved.

[0087] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An adaptive personalized federated learning method based on knowledge distillation, characterized in that: The specific steps include: The client generates pseudo data based on local data through a variational autoencoder, uploads the pseudo data to the server, and uses hybrid shearing technology to enhance the pseudo data; The teacher model deployed on the server is trained with data augmentation, and the random jitter dropout regularization technique is used during training to enhance the robustness of the teacher model. The student model deployed on the client generates soft prediction labels based on the input samples and uploads them to the server. The server dynamically weights the soft prediction labels of each client based on the confidence of the soft prediction labels. Use the dynamically weighted soft labels for further iterative training and generate personalized labels, which are then sent to the corresponding clients. The client calculates the inter-class prediction loss and intra-class prediction loss based on the received personalized labels, and constructs the distillation loss based on the inter-class prediction loss and intra-class prediction loss to train the student model; Determine whether the iteration threshold of federated learning has been reached. If so, end the training and fix the network parameters of each client student model. If not, repeat the above steps.

2. The adaptive personalized federated learning method based on knowledge distillation according to claim 1, characterized in that: When training the variational autoencoder, the similarity index between the newly generated samples and the original samples is introduced into the loss function. The loss function of the variational autoencoder is expressed as: in, represents the loss function of the variational autoencoder, represents the new sample generated by the variational autoencoder based on the original sample x, θ represents the parameters of the encoder in the variational autoencoder, represents the parameters of the decoder in the variational autoencoder; Representation sample The mean square error function between the expected value and the sample x; KL(·) represents the divergence; N(μ, σ) represents a Gaussian distribution with an expected value of μ and a variance of σ, and N(0, 1) represents a Gaussian distribution with an expected value of 0 and a variance of 1. The expected value μ and the variance σ are learnable parameters.

3. The adaptive personalized federated learning method based on knowledge distillation according to claim 1, characterized in that: The process of data enhancement on pseudo data using mixed shearing technology is expressed as: in, Represents two different images x i , x′ i The new image obtained by mixing; ⊙ represents pixel-by-pixel multiplication; M represents a binary mask; Represents two different images x i , x′ i The labels yi, y′ i The new label obtained by mixing; λ is a random number that obeys the (0, 1) uniform distribution.

4. The adaptive personalized federated learning method based on knowledge distillation according to claim 1, characterized in that: The teacher model deployed on the server is trained using the data after data enhancement. The loss function during training is expressed as: Among them, L Mix (x i ,y i , x′ i , y′ i ; θ′) represents the loss function when the teacher model uses the data after data enhancement for intensive training, x i , x′ i Represents the data sample before data enhancement, x′ i , y′ i Represents data sample x i , x′ i The corresponding labels, θ′ represents the parameters of the teacher model; L CE (x i ,y i ; θ′) indicates that the input data sample in the teacher model is x i , the true label is y i The cross entropy loss when ; represents the mixed sample loss, According to the data sample x i , x′ i The data samples obtained after data enhancement; λ is a random number that obeys the (0, 1) uniform distribution; Indicates that the data sample input in the teacher model is The cross entropy loss when the true label is yi; Indicates that the data sample input in the teacher model is The true label is y′ i The cross entropy loss when L SD (θ′) denotes the random jitter dropout regularization term.

5. The adaptive personalized federated learning method based on knowledge distillation according to claim 1, characterized in that: The process of dynamically weighting the soft prediction labels of each client based on the confidence of the soft prediction labels includes: Calculate the confidence of the client based on the client's data quality and the client's historical performance; Based on the attention mechanism, the attention score is calculated according to the confidence of each client, and the soft label of the corresponding client is weighted according to the attention score.

6. The adaptive personalized federated learning method based on knowledge distillation according to claim 5, characterized in that: The client's confidence is expressed as: C k =α·Q k +β·H k Among them, C k is the confidence of client k; Q k is the data quality of client k; H k is the historical performance of client k; α and β are Q k , H k The weight factor of ; missing represents the number of erroneous data in the client data; outliers represents the number of missing values ​​in the client data, and total represents the total number of client data.

7. The adaptive personalized federated learning method based on knowledge distillation according to claim 1, characterized in that: The teacher model uses the dynamically weighted soft labels for further iterative training, and the loss function during training is expressed as: L θ =L CE (x i ,y i ;θ)+λ1L UD Among them, L θ represents the loss function of the teacher model; L CE (x i ,y i ; θ) indicates that the teacher model has a network parameter of θ and an input sample of x i , the true label is y i The cross entropy loss when L UD is the distillation loss of the teacher model, λ1 is the weight of the distillation loss; T is the temperature parameter of the distillation; KL(·) means to find the divergence; softmax(·) means the softmax activation function; Y g represents the soft prediction from the client after being aggregated by the server; Y t represents the soft prediction of the teacher model.

8. The adaptive personalized federated learning method based on knowledge distillation according to claim 1, characterized in that: The distillation loss is expressed as: in, represents the loss function of the t-th training of the student model of the k-th client; represents the cross entropy loss of the student model of the kth client in the tth round of federated learning; L out Represents the prediction distribution loss for different categories; L in represents the predicted distribution of the same category under different instances; L m represents a regularization term, that is, the norm of the difference between the network parameters of the student model in the current training round and the network parameters of the student model in the previous round; β, γ, and μ are all loss weights.

9. The adaptive personalized federated learning method based on knowledge distillation according to claim 8, characterized in that: About the prediction distribution loss L of different categories out It is expressed as: About the predicted distribution L of the same category under different instances in It is expressed as: Among them, B represents the training batch size, C represents the number of categories; d p (x, y) represents the measure of vector x and vector y, expressed as d p (x, y) = 1 - ρ p (x,y),ρ p (x, y) represents the Pearson correlation coefficient between vector x and vector y; represents the probability distribution of the predicted labels of the i-th batch of training data of the student model, represents the probability distribution of the predicted labels of the i-th batch of training data of the teacher model; represents the prediction of the i-th batch of training data by the student model, represents the prediction of the i-th batch of training data by the teacher model; represents the probability distribution of the j-th category prediction label of the student model, represents the probability distribution of the j-th category prediction label of the teacher model; represents the j-th category prediction of the student model, represents the j-th category prediction of the teacher model.

Images