A Data Transaction Approach Based on Multi-Constraint Unsupervised Federated Knowledge Distillation

Abstract

This invention proposes a data trading method based on multi-constraint unsupervised federated knowledge distillation. By constructing various constraints, including logit consistency, batch statistical features, and information entropy constraints, it achieves efficient and accurate knowledge transfer from multiple teacher models to student models while ensuring image data privacy. Furthermore, batch-level statistical constraints make the model more stable when processing different batches of image data. In addition, the data holder only uploads the trained model parameters, without sharing the original image data, thus guaranteeing data privacy. This enhances the performance of student models in image tasks and solves the problems of insufficient knowledge transfer, poor model stability, and difficulties in integrating knowledge from multiple teachers in traditional knowledge distillation for image data trading, showing promising application prospects.

Description

Technical Field

[0001] This invention relates to the field of data transaction technology, and in particular to a data transaction method based on multi-constraint unsupervised federated knowledge distillation. Background Technology

[0002] In the field of data trading technology, especially in scenarios involving image data, data privacy protection and efficient utilization have always been critical issues that urgently need to be addressed. Federated learning-based methods directly trade models rather than data, avoiding data leakage caused by output transmission. However, different local models exhibit heterogeneity, and how to integrate models with different architectures during data trading is a key problem that needs to be solved. Knowledge distillation is a model optimization and knowledge transfer method that can transfer knowledge to student models with different architectures, providing new possibilities for data trading. However, traditional knowledge distillation methods have significant shortcomings in data trading applications. Most of them only focus on the similarity of the model output probability distribution. This single constraint is too simplistic and fails to fully exploit the complex feature information in image data. Image data contains rich features such as texture, shape, and color. Relying solely on output probability distribution constraints, student models cannot fully learn the key knowledge in teacher models, resulting in poor model accuracy and generalization ability. For example, in data trading scenarios for medical image diagnosis, a knowledge distillation model with a single constraint may not accurately identify the image features of some rare diseases, thus affecting the accuracy of diagnosis. Furthermore, when multiple data holders are involved (i.e., multi-teacher model scenarios), effectively integrating the knowledge from multiple teacher models for distillation becomes another challenge. Image data from different data holders differ in distribution, features, and other aspects. Traditional methods cannot fully integrate knowledge from these different sources, making it difficult for student models to learn comprehensive and accurate knowledge, thus limiting the improvement of model performance.

[0003] In summary, existing knowledge distillation methods cannot adequately meet the requirements of knowledge transfer accuracy, model stability, and multi-teacher knowledge integration in image data trading. There is an urgent need for a new method to address these issues, promote the development of image data trading, and maximize the utilization of image data value. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a data trading method based on multi-constraint unsupervised federated knowledge distillation. By constructing multiple constraints, this method enables efficient and accurate knowledge transfer from multiple teacher models to student models while ensuring image data privacy. This enhances the performance of student models in image tasks and solves the problems of insufficient knowledge transfer, poor model stability, and difficulty in integrating multi-teacher knowledge in traditional knowledge distillation for image data trading.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The data transaction method based on multi-constraint unsupervised federated knowledge distillation includes the following steps:

[0007] Buyer's student model Receive teacher models uploaded by K data holders ;

[0008] Construct the total loss function To minimize the total loss function To achieve the overall optimization goal, knowledge distillation is performed;

[0009] The total loss function It integrates logit (the inverse function of the logistic function) loss, batch-level statistical loss, and information entropy loss.

[0010] Furthermore, the knowledge distillation specifically involves using the Adam algorithm to process the student model. parameters Update to minimize the total loss function The update process is as follows:

[0011]

[0012] in, It's the learning rate. It is the total loss function Regarding student model parameters The gradient is used to update the parameters of the student model through multiple iterations until the preset number of iterations is reached.

[0013] Furthermore, the total loss function for:

[0014]

[0015] in, , , For weight parameters, For logit loss, For batch-level statistical loss, This represents the loss of information entropy.

[0016] Furthermore, multi-constraint knowledge distillation ensures that the probability distributions (logit) output by the student model and the teacher model remain consistent. The logit loss... The calculation method is as follows:

[0017] Buyer's unlabeled dataset A sample As a teacher model The input, processed by the teacher model The processed logit is denoted as Calculate the student model for The processed logit is denoted as ; and All are of length Feature vectors; the mean squared error loss is used to measure the difference between the student model and the teacher model. Differences in output logit:

[0018]

[0019] Because there are multiple teacher models, logit loss for:

[0020] .

[0021] Furthermore, at the statistical level, the statistical characteristics of the teacher and student model outputs are kept consistent. This involves the batch-level statistical loss. The calculation method is as follows:

[0022] Teacher Model Given an input of batch size B, the output logit set is: The student model also obtains the logit output set as follows: ; Calculate the teacher model separately The first group of the output logit set The second set of statistical features and logit sets output by the student model Statistical characteristics of order, of which the first group Statistical characteristics and second group The total order of the statistical features is the same, and the statistical features of each order are obtained by adjusting the power order which has the same order.

[0023] Student Model and Teacher Model The statistical loss at the batch level is as follows:

[0024]

[0025] in, , , , , These are the weights of different statistical characteristic constraints. , Teacher Model The first-order feature mean of the student model, , Teacher Model The second-order feature variance of the student model, , Teacher Model The third-order feature skewness of the student model, , Teacher Model The fourth-order feature kurtosis of the student model, and Teacher Model and student model Statistical characteristics of order;

[0026] Because a K-teacher model is used, the statistical loss is at the batch level. The overall definition is as follows:

[0027] .

[0028] Furthermore, the first-order feature mean Second-order characteristic variance Third-order eigenskewness and fourth-order characteristic kurtosis for:

[0029]

[0030]

[0031]

[0032] ;

[0033] The first-order characteristic mean Second-order characteristic variance Third-order eigenskewness and fourth-order characteristic kurtosis for:

[0034]

[0035]

[0036]

[0037] .

[0038] Furthermore, the student model learns the distribution characteristics of features in the teacher model. The information entropy loss... The definition is as follows:

[0039]

[0040] in, For a teacher model based on a batch Output the information entropy of the logit collection:

[0041]

[0042] in, For teacher model Under this batch size, the first The probability of each logit value occurring is calculated by counting the number of repetitions if the logit values ​​are repeated; otherwise, the probability of each logit value occurring is... ;

[0043] The information entropy of the logit set output by the student model based on a batch:

[0044]

[0045] in, For the student model at this batch size, the first The probability of each logit value occurring is calculated by counting the number of repetitions if the logit values ​​are repeated; otherwise, the probability of each logit value occurring is... .

[0046] Compared with existing technologies, the advantages of this invention are as follows: This invention achieves accurate knowledge transfer from a multi-teacher model to a student model through multiple constraints, including logit consistency, batch statistical features, and information entropy constraints. Furthermore, batch-level statistical constraints make the model more stable when processing different batches of image data. In addition, the data holder only uploads the trained model parameters, without sharing the original image data, thus ensuring data privacy. Attached Figure Description

[0047] Figure 1 This is a flowchart of the data transaction method based on multi-constraint unsupervised federated knowledge distillation according to the present invention. Detailed Implementation

[0048] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings.

[0049] like Figure 1 As shown, this embodiment of the invention provides a data transaction method based on multi-constraint unsupervised federated knowledge distillation, including the following steps:

[0050] First, prepare the teacher model: Based on the characteristics of their own image dataset, the data holder selects a suitable deep neural network architecture for local model training. During training, stochastic gradient descent (SGD) is used to optimize the model parameters, with cross-entropy loss function... Optimize as the objective function. Data holder. The local dataset is ,in The total number of data holders, It is an image sample. These are the corresponding tags. It is the number of samples in the local dataset of data holder i. It is the index of the sample. Based on the above definition, the local model of data holder i Loss function during training in round t for:

[0051]

[0052] in Image samples After local model The cross-entropy loss between the output and the label is used to update the local model through multiple iterations. The parameters until the local model convergence.

[0053] This embodiment has ten data holders ( Each data holder possesses image data from different sources. Considering the balance between model computational efficiency and performance, all 10 data holders chose MobileNet as their local model architecture. This is because MobileNet has a lightweight network structure, significantly reducing the number of parameters and computational cost through depthwise separable convolutions, maintaining good feature extraction capabilities even under resource constraints, making it suitable for processing large-scale image data.

[0054] During training, the SGD optimization algorithm is used to update the model parameters, with a learning rate of 0.001 and the cross-entropy loss function as the optimization objective. It is assumed that each data holder's local dataset contains an average of... The dataset contains 10 images, each with a corresponding label. During training, data is input into the model in batches of 64. After multiple rounds of iterative training, the model is considered successful when its loss value on the local validation set decreases by less than 1% for 10 consecutive rounds. At this point, the model is considered to have converged, and local model training is complete. After the 10 data holders have completed their local model training, they upload the trained model parameters to the buyer (server). The buyer receives these models and uses them as 10 teacher models. .

[0055] The buyer uses teacher models provided by multiple data holders to construct a multi-constraint knowledge distillation loss, with the buyer's model serving as the student model. The knowledge distillation process in the buyer's unlabeled dataset The above is carried out, among which yes The number of samples, yes The One sample.

[0056] In this embodiment, the buyer possesses a dataset containing 20,000 unlabeled images. In the process of knowledge distillation, each time from... Select batch size as Image samples were used. These samples were then input into 10 teacher models. and student model The logit output is obtained. In the implementation, the student model also uses MobileNet, and the model outputs a one-dimensional vector of size 200 as the logit. The logit difference is calculated according to the formula to obtain the logit loss. This is used to measure the consistency between the logit outputs of the student and teacher models.

[0057] The mean squared error loss is used to measure the difference between the student model and the teacher model. Differences in output logit:

[0058]

[0059] Because there are multiple teacher models, logit loss for:

[0060] .

[0061] Simultaneously, the statistical characteristics of the logit for the teacher and student models at this batch size, such as mean, variance, skewness, and kurtosis, are calculated, and a batch-level statistical loss is constructed based on these characteristics. To ensure that the statistical characteristics output by the teacher and student models are similar during batch data processing, the weighting coefficients for statistical information of different orders are set as follows:

[0062]

[0063] Student Model and Teacher Model The statistical loss at the batch level is as follows:

[0064]

[0065] Because a K-teacher model is used, the statistical loss is at the batch level. The overall definition is as follows:

[0066] .

[0067] Information entropy is calculated based on the logit set of teacher and student models in a batch. The teacher model is calculated separately according to a predetermined formula. Information entropy Information entropy of the student model This leads to information entropy loss. This prompts the student model to learn the distribution characteristics of the teacher model's features.

[0068]

[0069] Set weight parameters , , Construct the overall loss function:

[0070]

[0071] Update the student model parameters and learning rate using the Adam optimization algorithm. Set to 0.0001, the update process is as follows:

[0072]

[0073] After each training epoch, record the loss value and accuracy on the validation set (if a validation set exists), among other metrics. Set the maximum number of iterations to 400 and observe the trend of the loss value during training. If the overall loss function... The decrease was less than Training will stop when the maximum number of iterations is reached.

[0074] This invention achieves accurate knowledge transfer from a multi-teacher model to a student model by constructing multiple constraints based on logit consistency, batch statistical features, and information entropy. Data holders only upload model parameters without sharing the original image data, ensuring data privacy. In terms of performance improvement, the student model achieves a 2.37% higher accuracy in image classification tasks compared to traditional single-constraint knowledge distillation methods. In a multi-teacher scenario with 10 data holders, while single-level constraints converge after 600 training rounds, the method of this invention converges in only 400 rounds.

[0075] Finally, it should be noted that the above embodiments are intended to illustrate the technical solutions of the present invention and do not constitute any limitation on the present invention. Those skilled in the art should fully understand that modifications to the technical solutions described in the foregoing embodiments or equivalent substitutions for any part or all of the technical features are entirely feasible. Such modifications or substitutions, as long as they do not depart from the scope of protection defined by the claims of the present invention, should be considered reasonable extensions of the present invention.

Claims

1. A data transaction method based on multi-constraint unsupervised federated knowledge distillation, characterized in that: Includes the following steps: Buyer's student model Receive teacher models uploaded by K data holders ;in, The teacher model trained by data holder i on its local model based on the characteristics of its own image dataset. Construct the total loss function To minimize the total loss function To achieve the overall optimization goal, knowledge distillation is performed; The total loss function It integrates logit loss, batch-level statistical loss, and information entropy loss; The total loss function for: in, , , For weight parameters, For logit loss, For batch-level statistical loss, This is due to information entropy loss; The logit loss The calculation method is as follows: Buyer's unlabeled dataset A sample As a teacher model The input, processed by the teacher model The processed logit is denoted as Calculate the student model for The processed logit is denoted as ; and All are of length Feature vectors; the mean squared error loss is used to measure the difference between the student model and the teacher model. Differences in output logit: Because there are multiple teacher models, logit loss for: ； Batch-level statistical loss The calculation method is as follows: Teacher Model Given an input of batch size B, the output logit set is: The student model also obtains the logit output set as follows: ; Calculate the teacher model separately The first group of the output logit set The second set of statistical features and logit sets output by the student model Statistical characteristics of order, of which the first group Statistical characteristics and second group The total order of the statistical features is the same, and the statistical features of each order are obtained by adjusting the power order which has the same order. Student Model and Teacher Model The statistical loss at the batch level is as follows: in, , , , , These are the weights of different statistical characteristic constraints. , Teacher Model The first-order feature mean of the student model, , Teacher Model The second-order feature variance of the student model, , Teacher Model The third-order feature skewness of the student model, , Teacher Model The fourth-order feature kurtosis of the student model, and Teacher Model and student model Statistical characteristics of order; Because a K-teacher model is used, the statistical loss is at the batch level. The overall definition is as follows: ； The information entropy loss The definition is as follows: in, For a teacher model based on a batch Output the information entropy of the logit collection: in, For teacher model Under this batch size, the first The probability of each logit value occurring is calculated as follows: if logit values ​​are repeated, the number of repetitions needs to be counted to calculate the probability; if there are no repetitions, the probability of each logit value occurring is... ; The information entropy of the logit set output by the student model based on a batch: in, For the student model at this batch size, the first The probability of each logit value occurring is calculated as follows: if logit values ​​are repeated, the number of repetitions needs to be counted to calculate the probability; if there are no repetitions, the probability of each logit value occurring is... .

2. The method according to claim 1, characterized in that, The knowledge distillation specifically involves using the Adam algorithm to process the student model. parameters Update to minimize the total loss function The update process is as follows: in, It's the learning rate. It is the total loss function Regarding student model parameters The gradient is used to update the parameters of the student model through multiple iterations until the preset number of iterations is reached.

3. The method according to claim 1, characterized in that, The first-order characteristic mean Second-order characteristic variance Third-order eigenskewness and fourth-order characteristic kurtosis for: ； The first-order characteristic mean Second-order characteristic variance Third-order eigenskewness and fourth-order characteristic kurtosis for: 。

Images