Data transaction method based on multi-constraint unsupervised federated knowledge distillation

Abstract

According to the data transaction method based on multi-constraint unsupervised federated knowledge distillation, by constructing various constraint conditions, logit consistency, batch statistical features and information entropy constraints, on the premise that image data privacy is ensured, efficient and accurate knowledge migration from multiple teacher models to student models is achieved, and the data transaction efficiency is improved. And the batch-level statistical constraint enables the model to be more stable when processing different batches of image data. Besides, the data holder only uploads the trained model parameters, and does not need to share original image data, so that the data privacy is guaranteed. The performance of a student model in an image task is enhanced, the problems of insufficient knowledge transmission, poor model stability and difficulty in integration of multi-teacher knowledge in image data transaction of traditional knowledge distillation are solved, and the method has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of data trading technology, and in particular to a data trading method based on multi-constraint unsupervised federated knowledge distillation. Background Art

[0002] In the field of data trading technology, especially in scenarios involving image data, data privacy protection and efficient utilization remain key challenges. Federated learning-based approaches, which directly trade models rather than data, can avoid data leakage caused by output transmission. However, different local models exhibit heterogeneity, necessitating the integration of models with different architectures during data trading. Knowledge distillation, a model optimization and knowledge transfer method, can transfer knowledge to student models with different architectures, offering new possibilities for data trading. However, traditional knowledge distillation methods have significant drawbacks in data trading applications. Most focus solely on the similarity of model output probability distributions. This single constraint approach is overly simplistic and fails to fully exploit the complex feature information in image data. Image data contains a rich set of features, such as texture, shape, and color. Relying solely on output probability distribution constraints, the student model cannot fully learn the key knowledge from the teacher model, resulting in poor accuracy and generalization. For example, in data trading scenarios for medical image diagnosis, a single-constraint knowledge distillation model may not accurately identify the image features of some rare conditions, thus impacting diagnostic accuracy. Furthermore, when multiple data holders are involved (i.e., multiple teacher models are used), effectively integrating the knowledge from these multiple teacher models for distillation presents another challenge. Image data from different data holders differs in distribution and features, and traditional methods are unable to fully integrate knowledge from these diverse sources. This makes it difficult for the student model to learn comprehensive and accurate knowledge, limiting performance improvements.

[0003] In summary, existing knowledge distillation methods cannot well meet the requirements of knowledge transfer accuracy, model stability and multi-teacher knowledge integration in image data trading. There is an urgent need for a new method to solve these problems in order to promote the development of image data trading and maximize the utilization of image data value. Summary of the Invention

[0004] In response to the shortcomings of existing technologies, the present invention proposes a data trading method based on multi-constraint unsupervised federated knowledge distillation. By constructing multiple constraints, while ensuring the privacy of image data, efficient and accurate knowledge transfer from multiple teacher models to student models is achieved, thereby enhancing the performance of student models in image tasks and solving the problems of insufficient knowledge transfer, poor model stability and difficulty in integrating knowledge from multiple teachers in traditional knowledge distillation in image data trading.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: The data trading method based on multi-constrained unsupervised federated knowledge distillation includes the following steps: Buyer's Student Model Receive teacher models uploaded by K data holders ; Constructing the total loss function , to minimize the total loss function Perform knowledge distillation for the overall optimization goal; The total loss function It integrates logit (inverse function of the logistic function) loss, batch-level statistical loss, and information entropy loss.

[0006] Furthermore, the knowledge distillation is specifically: using the Adam algorithm to distill the student model Parameters Update to minimize the total loss function , the update process is as follows: in, is the learning rate, is the total loss function About the student model parameters The gradient of the student model is updated through multiple iterations until the preset number of iterations is reached.

[0007] Furthermore, the total loss function for: in, 、 、 is the weight parameter, is the logit loss, is the statistical loss at the batch level, is the information entropy loss.

[0008] Furthermore, multi-constraint knowledge distillation ensures that the probability distribution (logit) output by the student model and the teacher model remains consistent. The calculation method is as follows: The buyer's unlabeled dataset A sample of As a teacher model The input of the teacher model The processed logit is recorded as , calculate the student model for The processed logit is recorded as ; and The length is The feature vector of the student model is measured by the mean square error loss. Output the difference in logits: Since there are multiple teacher models, the logit loss for: .

[0009] Furthermore, the statistical features of the teacher and student models are kept consistent at the statistical level. The calculation method is as follows: Set up teacher model The logit set output under a batch size of B is , the student model also obtains the output set of logit as ; Calculate the teacher model separately The first group of logit sets output The second set of logit sets of order statistics and student model outputs Order statistical characteristics, where the first group Order statistics and the second group The total order of the statistical features of the orders is the same, and the statistical features of each order are calculated by adjusting the power level with the same order as that of the order; Student model and teacher model The statistical loss at the batch level is as follows: in, 、 、 、 、 are the weights of different statistical feature constraints, 、 Teacher Model and the first-order eigenmean of the student model, 、 Teacher Model and the second-order characteristic variance of the student model, 、 Teacher Model and the third-order characteristic skewness of the Student model, 、 Teacher Model and the fourth-order characteristic kurtosis of the Student model, and Teacher Model and student models Order statistical characteristics; Since K teacher models are used, the batch-level statistical loss The overall definition is as follows: .

[0010] Furthermore, the first-order characteristic mean , the second-order characteristic variance , the third-order characteristic skewness and the fourth-order characteristic kurtosis for: ; The first-order eigenvalue , the second-order characteristic variance , the third-order characteristic skewness and the fourth-order characteristic kurtosis for: .

[0011] Furthermore, the student model learns the distribution characteristics of the features in the teacher model. The definition is as follows: in, For a teacher model based on a batch Output the information entropy of the logit set: in, Model for teachers At this batch size The probability of a logit value appearing. If the logit value is repeated, the number of repetitions needs to be counted to calculate the probability; if there is no repetition, the probability of each logit value appearing is ; The information entropy of the logit set output by the student model based on a batch is: in, The student model is the first The probability of a logit value appearing. If the logit value is repeated, the number of repetitions needs to be counted to calculate the probability; if there is no repetition, the probability of each logit value appearing is .

[0012] Compared with existing technologies, this invention offers the following advantages: It utilizes multiple constraints, including logit consistency, batch statistical features, and information entropy constraints, to achieve precise knowledge transfer from multiple teacher models to a student model. Furthermore, batch-level statistical constraints make the model more stable when processing different batches of image data. Furthermore, data holders only upload trained model parameters, without sharing raw image data, thus ensuring data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of the data trading method based on multi-constraint unsupervised federated knowledge distillation of the present invention. DETAILED DESCRIPTION

[0014] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, an embodiment of the present invention provides a data trading method based on multi-constraint unsupervised federated knowledge distillation, including the following steps: First, prepare the teacher model: the data holder selects a suitable deep neural network architecture for local model training based on the characteristics of its own image dataset. During the training process, stochastic gradient descent (SGD) is used to optimize the model parameters, with the cross entropy loss function Optimize as the objective function. Data holder The local dataset is ,in is the total number of data holders, is an image sample, is the corresponding label, is the number of samples in the local dataset of data holder i, is the index of the sample. Based on the above definition, the local model of data holder i The loss function in the tth round of training for: in is an image sample Through the local model The cross entropy loss between the output and the label is used to update the local model through multiple iterations Parameters until the local model convergence.

[0016] This embodiment has ten data holders ( ), each data holder possesses image data from different sources. Considering the balance between model computational efficiency and performance, all ten data holders selected MobileNet as their local model architecture. MobileNet's lightweight network structure significantly reduces the number of parameters and computational effort through depthwise separable convolutions. Even with limited resources, it maintains strong feature extraction capabilities, making it suitable for processing large-scale image data.

[0017] During training, the SGD optimization algorithm is used to update the model parameters, the learning rate is set to 0.001, and the cross entropy loss function is used as the optimization target. Assume that the local dataset of each data holder contains an average of images, and the images have corresponding labels. During the training process, the data is input into the model for training in batches, and the size of each batch is set to 64. After multiple rounds of iterative training, when the loss value of the model on the local validation set decreases by less than 10 rounds in a row, When , the model is considered to have converged and the local model training is completed. After the 10 data holders complete the local model training, they upload the trained model parameters to the buyer (server). The buyer receives these models and uses them as 10 teacher models. .

[0018] The buyer uses the teacher models provided by multiple data holders to build a multi-constrained knowledge distillation loss, and the buyer model serves as the student model. , the knowledge distillation process is based on the buyer’s unlabeled dataset On, where yes The number of samples, yes No. samples.

[0019] In this example, the buyer has a dataset of 20,000 unlabeled images ( ). In the process of knowledge distillation, each time from The batch size is selected as The image samples are input into 10 teacher models respectively. and student models , get the logit output. In the implementation, the student model also uses MobileNet, and the logit output of the model is a one-dimensional vector of size 200. Calculate the logit difference according to the formula to get the logit loss , to measure the consistency of the logit output of the student and teacher models.

[0020] The student model and the teacher model are measured by the mean squared error loss Output the difference in logits: Since there are multiple teacher models, the logit loss for: .

[0021] At the same time, the statistical characteristics of the logit such as mean, variance, skewness and kurtosis of the teacher and student models under this batch are calculated, and the statistical loss at the batch level is constructed based on these characteristics. , ensuring that the statistical characteristics of the teacher and student models are similar when processing batch data. The weight coefficients of statistical information of different orders are set as follows: Student model and teacher model The statistical loss at the batch level is as follows: Since K teacher models are used, the batch-level statistical loss The overall definition is as follows: .

[0022] The information entropy is calculated based on the logit set of the teacher and student models in a batch. The teacher model is calculated according to the established formula. Information entropy and the information entropy of the student model , and then get the information entropy loss , which encourages the student model to learn the distribution characteristics of the teacher model features.

[0023] Setting weight parameters 、 、 , construct the overall loss function: Use Adam optimization algorithm to update student model parameters and learning rate Set to 0.0001, the update process is as follows: After each round of training, record the loss value of the current round and the accuracy on the validation set (if there is a validation set) and other indicators. Set the maximum number of iterations to 400 times. During the training process, observe the trend of the loss value. If the overall loss function is The decline is less than , or when the maximum number of iterations is reached, the training stops.

[0024] The present invention realizes accurate knowledge transfer from multi-teacher models to student models by constructing multiple constraints of logit consistency, batch statistical features and information entropy. The data holder only uploads model parameters without sharing the original image data, thus ensuring data privacy. In terms of effect improvement, the accuracy of the student model in image classification tasks is improved by 2.37% compared with the traditional single-constraint knowledge distillation method. In a multi-teacher scenario with 10 data holders, the single-level constraint converges in 600 rounds of training, while the method of the present invention can converge in 400 rounds.

[0025] Finally, it should be noted that the above embodiments are intended to illustrate the technical solutions of the present invention and do not constitute any form of limitation of the present invention. Those skilled in the art should fully understand that it is entirely feasible to modify the technical solutions described in the above embodiments or to replace any or all of the technical features with equivalents. Such modifications or replacements, as long as they do not deviate from the scope of protection defined by the claims of the present invention, should be considered reasonable extensions of the present invention.

Claims

1. A data trading method based on multi-constrained unsupervised federated knowledge distillation, characterized by: The steps include: Buyer's Student Model Receive teacher models uploaded by K data holders ; Constructing the total loss function , to minimize the total loss function Perform knowledge distillation for the overall optimization goal; The total loss function It integrates logit loss, batch-level statistical loss, and information entropy loss.

2. The method according to claim 1, characterized in that The knowledge distillation is specifically: using the Adam algorithm to distill the student model Parameters Update to minimize the total loss function , the update process is as follows: in, is the learning rate, is the total loss function About the student model parameters The gradient of the student model is updated through multiple iterations until the preset number of iterations is reached.

3. The method according to claim 1, characterized in that The total loss function for: in, 、 、 is the weight parameter, is the logit loss, is the statistical loss at the batch level, is the information entropy loss.

4. The method according to claim 3, characterized in that The logit loss The calculation method is as follows: The buyer's unlabeled dataset A sample of As a teacher model The input of the teacher model The processed logit is recorded as , calculate the student model for The processed logit is recorded as ; and The length is The feature vector of the student model is measured by the mean square error loss. Output the difference in logits: Since there are multiple teacher models, the logit loss for: 。 5. The method according to claim 3, characterized in that: The batch-level statistical loss The calculation method is as follows: Set up teacher model The logit set output under a batch size of B is , the student model also obtains the output set of logit as ; Calculate the teacher model separately The first group of logit sets output The second set of logit sets of order statistics and student model outputs Order statistical characteristics, where the first group Order statistics and the second group The total order of the statistical features of the orders is the same, and the statistical features of each order are calculated by adjusting the power level with the same order as that of the order; Student model and teacher model The statistical loss at the batch level is as follows: in, 、 、 、 、 are the weights of different statistical feature constraints, 、 Teacher Model and the first-order eigenmean of the student model, 、 Teacher Model and the second-order characteristic variance of the student model, 、 Teacher Model and the third-order characteristic skewness of the Student model, 、 Teacher Model and the fourth-order characteristic kurtosis of the Student model, and Teacher Model and student models Order statistical characteristics; Since K teacher models are used, the batch-level statistical loss The overall definition is as follows: 。 6. The method according to claim 5, characterized in that The first-order eigenvalue , the second-order characteristic variance , the third-order characteristic skewness and the fourth-order characteristic kurtosis for: ； The first-order eigenvalue , the second-order characteristic variance , the third-order characteristic skewness and the fourth-order characteristic kurtosis for: 。 7. The method according to claim 3, characterized in that The information entropy loss The definition is as follows: in, For a teacher model based on a batch Output the information entropy of the logit set: in, Model for teachers At this batch size The probability of a logit value appearing. If the logit value is repeated, the number of repetitions needs to be counted to calculate the probability; if there is no repetition, the probability of each logit value appearing is ; The information entropy of the logit set output by the student model based on a batch is: in, The student model is the first The probability of a logit value appearing. If the logit value is repeated, the number of repetitions needs to be counted to calculate the probability; if there is no repetition, the probability of each logit value appearing is .

Images