Deep forgery identification method and system based on knowledge back distillation

Abstract

The invention discloses a deep counterfeiting identification method and system based on knowledge back distillation, and belongs to the technical field of computer vision. The method comprises the following steps: estimating feature variation distribution of a teacher model by using a Gaussian mixture model, and training a student model under the constraint of back distillation loss and classification loss guided by the feature variation distribution; wherein after a trained student model Mi is obtained, the trained student model Mi is used as a teacher model to obtain a trained student model Mi + 1; and after obtaining teacher features t and student features si of the to-be-detected image based on a teacher model and a trained student model Mi, fusing the teacher features t and the student features si, and obtaining a counterfeit identification result of the to-be-detected image according to the fused features. According to the invention, the generalization performance of deep counterfeiting identification can be enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically to a deepfake authentication method and system based on knowledge distillation. Background Art

[0002] With the rapid development of deepfake technology, the number of crisis events caused by its abuse has been increasing. These technologies can synthesize highly persuasive pictures, videos, and audio of real people, which has led to a significant increase in manipulated content. Unfortunately, the abuse of these tools has led to an increasing number of crisis events, including stealing personal information by bypassing facial recognition systems, identity manipulation for fraud, etc. These events pose a serious threat to personal privacy and social security. Therefore, the need for an efficient and generalizable deepfake authentication solution has become urgent.

[0003] Existing forgery discriminators often exhibit poor performance when faced with previously unseen forgery methods. Previous studies have shown that different forgery methods leave unique, content-independent fingerprints in the generated images. Although these specific features are useful for identifying specific generation methods, they tend to weaken the generalization ability of the model. This results in forgery discriminators often overfitting to specific cues in the training set rather than learning general, generalizable features, which limits their adaptability to cross-domain data. To address the generalization problem caused by overfitting to specific features, previous forgery authentication methods have mainly focused on data augmentation and feature augmentation techniques. The core idea of these augmentation techniques is to design more generalizable representations for the input data or the learned features. Although these methods have significantly improved the generalization performance, they usually require heuristics and careful design.

[0004] Researchers rethought the reasons for overfitting and found that the overfitting features with low loss not only reduce the adaptability to cross-domain data but also inhibit the model's ability to learn more widely applicable information. Therefore, if forgery discriminators can bypass the influence of overfitting features with low loss, they can learn more generalizable features. Summary of the Invention

[0005] The present invention discloses a deepfake authentication method and system based on knowledge distillation, which can enhance the generalization performance of deepfake authentication.

[0006] To achieve the above object, the technical solution of the present invention includes the following content.

[0007] A deepfake authentication method based on knowledge distillation, the method comprising:

[0008] Estimate the variational distribution of the features of the teacher model using a Gaussian mixture model, and train the student model under the constraints of the anti-distillation loss and the classification loss guided by the variational distribution of the features; wherein, after obtaining the trained student model M i , use the trained student model M i as the teacher model to obtain the trained student model M i+1 ;

[0009] Based on the teacher model and the trained student model M i obtain the teacher features t and the student features s of the image to be detected i After that, fuse the teacher features t and the student features s i and obtain the forgery discrimination result of the image to be detected according to the fused features.

[0010] Further, training the student model under the constraints of the anti-distillation loss and the classification loss guided by the variational distribution of the features includes:

[0011] Use the student model to obtain a probability distribution sample u of the student features of the training samples j ;

[0012] Based on the variational distribution and the probability distribution sample u j , obtain the cross-entropy between the student feature probability distribution and the teacher feature probability distribution;

[0013] On each training batch, obtain the anti-distillation loss by maximizing the cross-entropy between the student feature probability distribution and the teacher feature probability distribution;

[0014] On each training batch, use the training samples to obtain the classification cross-entropy loss of the student model;

[0015] Based on the anti-distillation loss and the classification cross-entropy loss, perform backpropagation to update the parameters of the student model.

[0016] Further, fusing the teacher features t and the student features s i and obtaining the forgery discrimination result of the image to be detected according to the fused features includes:

[0017] Obtain the fusion weights;

[0018] Weight the teacher features t and the student features s according to the fusion weights i to obtain the fused features;

[0019] Based on the fused features, perform classification to obtain the forgery discrimination result of the image to be detected.

[0020] Further, the calculation process of the fusion weights includes:

[0021] Obtain the feature variational distribution q of the image to be detected in the teacher model and the trained student model M i where z ∈ [t, i], t represents the teacher model, and i represents the trained student model M z ; i ;

[0022] Calculate the mean μ z and variance z of the feature variational distribution q

[0023] According to the mean μ z , calculate the average mean

[0024] Based on the mean μ z ; the variance and the average mean obtain the dispersion degree η z of the feature variational distribution p z ;

[0025] Based on the dispersion degree η z , obtain the fusion weight α i of the teacher model and the trained student model M z .

[0026] A deepfake discrimination system based on knowledge distillation, the system includes:

[0027] A training module for estimating the feature variational distribution of the teacher model using a Gaussian mixture model and training the student model under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution; wherein, after obtaining the trained student model M i , use this trained student model M i as the teacher model to obtain the trained student model M i+1 ;

[0028] An inference module for obtaining the teacher feature t and student feature s of the image to be detected based on the teacher model and the trained student model M i and then fusing the teacher feature t and the student feature s i and obtaining the forgery discrimination result of the image to be detected according to the fused feature. i

[0029] Further, training the student model under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution includes:

[0030] Using the student model to obtain the probability distribution sampling u of the student features of the training samples​j ;

[0031] Sample \(u\) based on the variational distribution of the features and the probability distribution j , obtaining the cross-entropy between the student feature probability distribution and the teacher feature probability distribution;

[0032] On each training batch, obtain the anti-distillation loss by maximizing the cross-entropy between the student feature probability distribution and the teacher feature probability distribution;

[0033] On each training batch, obtain the classification cross-entropy loss of the student model using the training samples;

[0034] Based on the anti-distillation loss and the classification cross-entropy loss, perform backpropagation to update the parameters of the student model.

[0035] Furthermore, fuse the teacher feature \(t\) and the student feature \(s\) i and obtain the forgery discrimination result of the image to be detected according to the fused feature, including:

[0036] Obtain the fusion weight;

[0037] Weight the teacher feature \(t\) and the student feature \(s\) according to the fusion weight i to obtain the fused feature;

[0038] Based on the fused feature, perform classification to obtain the forgery discrimination result of the image to be detected.

[0039] Furthermore, the calculation process of the fusion weight includes:

[0040] Obtain the variational distribution \(q\) of the features of the image to be detected in the teacher model and the trained student model \(M\) i where \(z\in[t, i]\), \(t\) represents the teacher model, and \(i\) represents the trained student model \(M\) z ; i ;

[0041] Calculate the mean \(\mu\) of the variational distribution \(q\) of the features z and the variance z ;

[0042] According to the mean \(\mu\) z , calculate the average mean

[0043] Based on the mean \(\mu\) z ; the variance and the average mean obtain the degree of dispersion \(\eta\) of the variational distribution \(p\) of the features z ; z ;

[0044] According to the dispersion degree η z , the fusion weight α of the teacher model and the trained student model M i is obtained z .

[0045] An electronic device, the electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for deepfake authentication based on knowledge distillation as described in any one of the above is implemented.

[0046] A computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, the method for deepfake authentication based on knowledge distillation as described in any one of the above is implemented.

[0047] The core of the KND technology lies in exploring the knowledge space unknown to the teacher model through the student model while maintaining the learning of the target task, thereby improving the model's ability to identify unseen forgery techniques. This method not only improves the generalization ability of the model but also reduces the over-reliance on specific forgery clues, enabling the model to maintain high identification performance when facing new and unknown forgery techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of deepfake authentication using the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0049] The following will further elaborate on the present invention through specific implementation cases and drawings.

[0050] The method for deepfake authentication based on knowledge distillation of the present invention utilizes the proposed Knowledge Negative Distillation (KND) technology to construct a framework of a teacher model and a student model, automatically reducing the impact of low-loss overfitting features on learning more general information, and is applicable to improving the generalization ability of deepfake authentication. Among them, there are two models in the framework of the present invention: a teacher model and a student model. The teacher model is a pre-trained model that may perform well on specific tasks but may overfit to specific features in the training data. The student model aims to learn more extensive and generalizable features to improve cross-domain performance.

[0051] Such as Figure 1As shown in the figure, the method of the present invention is divided into two parts: training and testing. In the training stage, the student network is reverse distilled according to the feature distribution of the teacher model estimated by the GMM. In the testing stage, the teacher features and the student features are input into the distribution-guided feature fusion module, and the weighted fusion features are input into their respective classifiers, and the average value of the classification results is used as the final discrimination result. Specifically, the deepfake discrimination method based on knowledge reverse distillation of the present invention includes the following steps 1 to 3.

[0052] Step 1: Estimate the variational distribution of the features of the teacher model using the Gaussian mixture model.

[0053] The Gaussian Mixture Model (GMM) is used to estimate the probability distribution of the features of the teacher model. This distribution represents the knowledge known by the teacher model. The feature probability distribution of the teacher model is used as the basis for the student model to learn.

[0054] Step 2: Train the student model under the constraints of the reverse distillation loss and the classification loss guided by the variational distribution of the features.

[0055] The student model is trained to maximize the cross-entropy between its feature probability distribution and the feature probability distribution of the teacher model. This process encourages the student model to explore those knowledge areas not covered by the teacher model, that is, those features with a low probability of occurrence in the probability distribution of the teacher model.

[0056] The cross-entropy H(p s of the student feature probability distribution and the teacher feature p t is expressed as Equation (1). s ,p t )

[0057] H(p s ,p t ) = ∑ N -p s (u j ) log p t (u j ) (1)

[0058] where u j is the probability distribution sampling of the student feature s, N is the number of probability distribution samplings of the student feature, j = 1, 2,..., N, and the Kronecker function δ(s, u of the student feature probability distribution is defined as Equation (2). j )

[0059]

[0060] When training the student model, the present invention samples according to the probability distribution of the student features u jEquation (1) can be written as Equation (3).

[0061]

[0062] p t The variational distribution q obtained using the Gaussian mixture model t is used instead. Maximizing Equation (3) can be formulated as minimizing the KND loss function Equation (4).

[0063] L KND (s) = ∑ s log q t (s) (4)

[0064] where ∑ s log q t (s) represents the sum of the KND losses of all training data in a batch.

[0065] In addition, the student model is also constrained by the categorical cross-entropy loss Equation (5),

[0066] L CLS (p) = ∑ p (ylogp+(1 - y)log(1 - p)) (5)

[0067] where p is the prediction value of the classifier, y is the label, (ylogp+(1 - y)log(1 - p) represents the categorical cross-entropy loss of one training data, and ∑ p (ylogp+(1 - y)log(1 - p)) represents the sum of the categorical cross-entropy losses of all training data in a batch.

[0068] The KND loss function and the categorical cross-entropy loss function are combined using the weight λ to obtain the overall loss function Equation (6) for training the student model.

[0069] L(s,p) = L KND (s) + L CLS (p) (6)

[0070] In addition, in an embodiment of the present invention, the trained student model can also be used as a new teacher model for recursively training a new student model. This process can dynamically expand the generalization ability of the model, but as the fusion effect converges, the effect of recursive training will tend to be stable.

[0071] Step 3: Based on the teacher model and the trained student model M i Obtain the teacher feature t and the student feature s of the image to be detected i After that, the teacher feature t and the student feature s iFuse them and obtain the forgery discrimination result of the image to be detected according to the fused features.

[0072] Step 3.1: Obtain the fusion weights.

[0073] The fusion weights can be obtained by manual setting. However, to avoid the blurring and unclear problems caused by directly averaging the features of different models, the present invention also discloses a weighted average algorithm for different model features guided by the dispersion of the feature probability distribution. By controlling the temperature parameter of the softmax operation, the fusion strategy can be adjusted to optimize the model performance.

[0074] For n models (including the original teacher model t and n - 1 trained student models), the feature t of the z-th model z is obtained by the forward propagation of the input data through the model encoder. The discrete degree η of the feature variational distribution q z is measured by Equation (7): z It is measured by Equation (7):

[0075]

[0076] where μ z , represent the mean and variance of the feature distribution of the z-th model. The present invention uses the temperature coefficient τ to control the fusion weight α (i) . α (i) is obtained by Equation (8):

[0077]

[0078] Step 3.2: Weight the teacher feature t and the student feature s i according to the fusion weights to obtain the fused feature f.

[0079] The fused feature f is obtained by Equation (9).

[0080]

[0081] where t z ∈[t, s i .

[0082] Step 3.3: Classify based on the fused feature to obtain the forgery discrimination result of the image to be detected.

[0083] The classification process of the present invention is to send the fused feature f into the classifier of each model z respectively, and then average the classification results, and obtain the forgery discrimination result of the image to be detected according to the average result.

[0084] In summary, different from traditional knowledge distillation, in knowledge anti-distillation, the teacher model acts as a "negative example", encouraging the student model to explore the knowledge unknown to the teacher in the dataset while solving the forgery discrimination task. The present invention believes that different probability distributions represent different types of information, so the cross-entropy between the probability distributions of the student and teacher features is maximized to increase the difference between them. By doing so, the student model automatically avoids being affected by the low-loss overfitting features learned by the teacher model, and thus can focus on more generalizable features.

[0085] The present invention first uses a Gaussian mixture model (GMM) to estimate the probability distribution of the teacher model features as the representation of the knowledge known to the teacher. Then, the student model is trained to maximize the cross-entropy between its feature probability distribution and the feature probability distribution of the teacher model, thereby encouraging the student model to explore the knowledge areas not covered by the teacher model. During this process, the student model extracts features by forward inputting data and learns the target task, while being guided by the cross-entropy loss to increase the difference from the teacher model features. Then, the features of the student model are used as an extensible component to be fused with the features of the teacher model, and this process is guided by their respective probability distributions. In addition, the present invention also proposes a recursive training method that allows the trained student model to be used as a new teacher model for training subsequent student models, thereby dynamically expanding the generalization ability of the model. Finally, through the distribution-guided feature fusion technology, the features of multiple models are combined to further improve the performance of the model in the deep forgery discrimination task. By this method, the performance of the model in the unknown forgery technology discrimination task can be effectively improved, and more accurate deep forgery discrimination can be achieved.

[0086] Next, two specific experiments are used to illustrate the deep forgery discrimination method based on knowledge anti-distillation provided by the present invention.

[0087] Example 1 Generalization test of face forgery discrimination based on knowledge anti-distillation.

[0088] Taking FF++ as the training set and Celeb-DF-v1 as the test set as an example:

[0089] 1) First, use the Xception network to fully train on the training set to extract the features of face forgery images and real images;

[0090] 2) Use a Gaussian mixture model (GMM) to estimate the probability distribution of the teacher model features as the representation of the knowledge known to the teacher;

[0091] 3) Train the student model to maximize the cross-entropy between its feature probability distribution and the feature probability distribution of the teacher model estimated in 2), encouraging the student model to explore the knowledge areas not covered by the teacher model;

[0092] 4) In the testing stage, perform distribution-guided feature fusion on the features of the student model and the teacher model to obtain a more comprehensive facial forgery feature representation;

[0093] 5) Conduct tests on Celeb-DF-v1. Input the weighted fusion features obtained in 4) into their respective classifiers, and take the average of the classification results as the final discrimination result;

[0094] 6) Compare the results of the method of the present invention with those of other methods. The obtained AUC (area under the ROC curve) for facial forgery discrimination is shown in Table 1.

[0095] Table 1. AUC for cross-dataset testing on the Celeb-DF-v1 dataset

[0096] Training dataset Test dataset The method of the present invention UCF SPSL FF++ Celeb-DF-v1 0.881 0.779 0.815

[0097] In Table 1, UCF and SPSL are the comparison methods, and both of these methods are methods for improving the generalization of facial forgery discrimination. Therefore, they can be compared with the method of the present invention.

[0098] According to the results in Table 1, it is shown that the method of the present invention can significantly improve the generalization effect of facial forgery discrimination.

[0099] Example 2 Generalization test for synthetic image discrimination based on knowledge distillation.

[0100] Taking the data generated by ProGAN in ForenSynths as the training set and the data generated by DALLE as the test set as an example:

[0101] 1) First, fully train using the Xception network on the training set to extract the features of synthetic images and real images;

[0102] 2) Use the Gaussian mixture model (GMM) to estimate the probability distribution of the teacher model features as the representation of the teacher's known knowledge;

[0103] 3) Train the student model to maximize the cross-entropy between its feature probability distribution and the estimated teacher model feature probability distribution in 2), encouraging the student model to explore the knowledge areas not covered by the teacher model;

[0104] 4) In the testing stage, perform distribution-guided feature fusion on the features of the student model and the teacher model to obtain a more comprehensive synthetic image feature representation;

[0105] 5) Conduct tests on DALLE. Input the weighted fusion features obtained in 4) into their respective classifiers, and take the average of the classification results as the final discrimination result;

[0106] 6) Compare the results of the method of the present invention with those of other methods. The obtained discrimination accuracies are shown in Table 2.

[0107] Table 2. Accuracy of cross-dataset testing on the DALLE dataset

[0108] Training dataset Test dataset The method of the present invention F3Net Ojha ProGAN DALLE 0.906 0.716 0.895

[0109] In Table 2, F3Net and Ojha are the comparative methods, and both of these methods are methods for improving the generalization of synthetic image discrimination. Therefore, the method of the present invention can be used for comparison.

[0110] According to the results in Table 2, it is shown that the method of the present invention can significantly improve the generalization effect of synthetic image discrimination.

[0111] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those of ordinary skill in the art can modify or equivalently replace the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to what is described in the claims.

Claims

1. A deep fake identification method based on knowledge de-distillation, characterized in that: The method comprises: The Gaussian mixture model is used to estimate the characteristic variation distribution of the teacher model, and the student model is trained under the constraints of the anti-distillation loss and classification loss guided by the characteristic variation distribution; wherein, after obtaining the trained student model M i Afterwards, the trained student model M i As the teacher model, to obtain the trained student model M i+1 ; Based on the teacher model and the trained student model M i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s i The fusion is performed, and a forgery identification result of the image to be detected is obtained according to the fusion feature.

2. The method according to claim 1, characterized in that Under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution, the student model is trained, including: Use the student model to obtain the probability distribution sampling u of the student characteristics of the training samples j ; Based on the characteristic variational distribution and the probability distribution sampling u j , get the cross entropy of the student feature probability distribution and the teacher feature probability distribution; On each training batch, the anti-distillation loss is obtained by maximizing the cross entropy between the student feature probability distribution and the teacher feature probability distribution; On each training batch, the classification cross entropy loss of the student model is obtained using the training samples; Back-propagation is performed based on the anti-distillation loss and the classification cross entropy loss to update the parameters of the student model.

3. The method according to claim 1, characterized in that: The teacher feature t and the student feature s i The fusion is performed, and a forgery identification result of the image to be detected is obtained according to the fusion feature, including: Get fusion weight; The teacher feature t and the student feature s are combined according to the fusion weight i Perform weighting to obtain fusion features; Classification is performed based on the fusion features to obtain a forgery identification result of the image to be detected.

4. The method according to claim 3, characterized in that The calculation process of the fusion weight includes: Get the image to be detected in the teacher model and the trained student model M i The characteristic variational distribution q in z , z∈[t,i], t represents the teacher model, i represents the trained student model M i ; Calculate the characteristic variational distribution q z The mean μ z and variance According to the mean μ z , calculate the average mean Based on the mean μ z ; The variance and the average mean Get the characteristic variation distribution p z The degree of dispersion η z ; According to the discrete degree η z , get the teacher model and the trained student model M i The fusion weight α z .

5. A deep fake identification system based on knowledge anti-distillation, characterized in that: The system comprises: The training module is used to estimate the characteristic variation distribution of the teacher model using the Gaussian mixture model, and train the student model under the constraints of the anti-distillation loss and classification loss guided by the characteristic variation distribution; wherein, after obtaining the trained student model M i Afterwards, the trained student model M i As the teacher model, to obtain the trained student model M i+1 ; The reasoning module is used to calculate the inference value based on the teacher model and the trained student model M. i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s i The fusion is performed, and a forgery identification result of the image to be detected is obtained according to the fusion feature.

6. The system according to claim 5, characterized in that Under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution, the student model is trained, including: Use the student model to obtain the probability distribution sampling u of the student characteristics of the training samples j ; Based on the characteristic variational distribution and the probability distribution sampling u j , get the cross entropy of the student feature probability distribution and the teacher feature probability distribution; On each training batch, the anti-distillation loss is obtained by maximizing the cross entropy between the student feature probability distribution and the teacher feature probability distribution; On each training batch, the classification cross entropy loss of the student model is obtained using the training samples; Back-propagation is performed based on the anti-distillation loss and the classification cross entropy loss to update the parameters of the student model.

7. The system according to claim 5, characterized in that The teacher feature t and the student feature s i The fusion is performed, and a forgery identification result of the image to be detected is obtained according to the fusion feature, including: Get fusion weight; The teacher feature t and the student feature s are fused according to the fusion weight i Perform weighting to obtain fusion features; Classification is performed based on the fusion features to obtain a forgery identification result of the image to be detected.

8. The system according to claim 7, characterized in that The calculation process of the fusion weight includes: Get the image to be detected in the teacher model and the trained student model M i The characteristic variational distribution q in z , z∈[t,i], t represents the teacher model, i represents the trained student model M i ; Calculate the characteristic variational distribution q z The mean μ z and variance According to the mean μ z , calculate the average mean Based on the mean μ z ; The variance and the average mean Get the characteristic variation distribution p z The degree of dispersion η z ; According to the discrete degree η z , get the teacher model and the trained student model M i The fusion weight α z .

9. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the deep counterfeit identification method based on knowledge de-distillation as described in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the deep fake identification method based on knowledge de-distillation as described in any one of claims 1 to 4.

Images