Deep fake identification method and system based on knowledge de-distillation

Abstract

The present invention discloses a deep fake identification method and system based on knowledge de-distillation, which belongs to the field of computer vision technology. The method comprises: using a Gaussian mixture model to estimate the characteristic variation distribution of the teacher model, and training the student model under the de-distillation loss and classification loss constraints guided by the characteristic variation distribution; wherein, after obtaining the trained student model M i Afterwards, the trained student model M i As the teacher model, to obtain the trained student model M i+1 ; Based on the teacher model and the trained student model M i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s i The fusion is performed, and a forgery identification result of the image to be detected is obtained based on the fusion features. The present invention can enhance the generalization performance of deep forgery identification.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a deep fake identification method and system based on knowledge de-distillation. Background Art

[0002] With the rapid development of deepfake technology, the number of crisis incidents caused by its misuse is increasing. These technologies can synthesize highly convincing images, videos, and audio of real people, leading to a significant increase in manipulated content. Unfortunately, the misuse of these tools has led to a growing number of crisis incidents, including bypassing facial recognition systems to steal personal information and manipulating identities for fraud. These incidents pose a serious threat to personal privacy and social security. Therefore, the need for efficient and generalizable deepfake detection solutions has become urgent.

[0003] Existing forgery discriminators often perform poorly when faced with previously unseen forgery methods. Previous studies have shown that different forgery methods leave unique, content-independent fingerprints in the generated images. While these specific features are useful for identifying specific generation methods, they often weaken the model's generalization ability. This causes forgery discriminators to overfit to specific cues in the training set rather than learning universal, generalizable features, which limits their adaptability to cross-domain data. To address the generalization problem caused by overfitting to specific features, previous forgery identification methods have mainly focused on data augmentation and feature enhancement techniques. The core idea of ​​these enhancement techniques is to design more generalizable representations for the input data or learned features. Although these methods significantly improve generalization performance, they usually require heuristics and careful design.

[0004] Researchers rethought the causes of overfitting and found that low-loss overfitting features not only reduce adaptability to cross-domain data, but also inhibit the model's ability to learn more widely applicable information. Therefore, if forged discriminators can bypass the influence of low-loss overfitting features, they can learn more generalizable features. Summary of the Invention

[0005] The present invention discloses a deep fake identification method and system based on knowledge de-distillation, which can enhance the generalization performance of deep fake identification.

[0006] To achieve the above objectives, the technical solution of the present invention includes the following contents.

[0007] A deep fake identification method based on knowledge de-distillation, the method comprising:

[0008] The Gaussian mixture model is used to estimate the characteristic variation distribution of the teacher model, and the student model is trained under the anti-distillation loss and classification loss constraints guided by the characteristic variation distribution; wherein, after obtaining the trained student model M i Afterwards, the trained student model M i As the teacher model, to obtain the trained student model M i+1 ;

[0009] Based on the teacher model and the trained student model M i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s i The fusion is performed, and a forgery identification result of the image to be detected is obtained according to the fusion feature.

[0010] Furthermore, under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution, the student model is trained, including:

[0011] Use the student model to obtain the probability distribution sampling u of the student features of the training sample j ;

[0012] Sampling u based on the characteristic variational distribution and the probability distribution j , get the cross entropy of the probability distribution of student features and the probability distribution of teacher features;

[0013] On each training batch, the anti-distillation loss is obtained by maximizing the cross entropy between the probability distribution of the student features and the probability distribution of the teacher features;

[0014] On each training batch, the classification cross entropy loss of the student model is obtained using the training samples;

[0015] Backpropagation is performed based on the dedistillation loss and the classification cross entropy loss to update the parameters of the student model.

[0016] Furthermore, the teacher feature t and the student feature s i Performing fusion and obtaining a forgery identification result of the image to be detected based on the fusion features, including:

[0017] Get fusion weight;

[0018] The teacher feature t and the student feature s are combined according to the fusion weight i Perform weighting to obtain fusion features;

[0019] Classification is performed based on the fusion features to obtain a forgery identification result of the image to be detected.

[0020] Furthermore, the calculation process of the fusion weight includes:

[0021] Get the image to be detected in the teacher model and the trained student model M i The characteristic variational distribution q in z , z∈[t,i], t represents the teacher model, i represents the trained student model M i ;

[0022] Calculate the characteristic variation distribution q z The mean μ z and variance

[0023] According to the mean μ z , calculate the average mean

[0024] Based on the mean μ z ; The variance and the average mean Get the characteristic variation distribution p z The degree of dispersion η z ;

[0025] According to the discrete degree η z , get the teacher model and the trained student model M i The fusion weight α z .

[0026] A deep fake identification system based on knowledge de-distillation, the system comprising:

[0027] The training module is used to estimate the characteristic variation distribution of the teacher model using the Gaussian mixture model, and train the student model under the constraints of the anti-distillation loss and classification loss guided by the characteristic variation distribution; wherein, after obtaining the trained student model M i Afterwards, the trained student model M i As the teacher model, to obtain the trained student model M i+1 ;

[0028] Inference module, used for inference based on teacher model and trained student model M i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s i The fusion is performed, and a forgery identification result of the image to be detected is obtained according to the fusion feature.

[0029] Furthermore, under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution, the student model is trained, including:

[0030] Use the student model to obtain the probability distribution sampling u of the student features of the training samplej ;

[0031] Sampling u based on the characteristic variational distribution and the probability distribution j , get the cross entropy of the probability distribution of student features and the probability distribution of teacher features;

[0032] On each training batch, the anti-distillation loss is obtained by maximizing the cross entropy between the probability distribution of the student features and the probability distribution of the teacher features;

[0033] On each training batch, the classification cross entropy loss of the student model is obtained using the training samples;

[0034] Backpropagation is performed based on the dedistillation loss and the classification cross entropy loss to update the parameters of the student model.

[0035] Furthermore, the teacher feature t and the student feature s i Performing fusion and obtaining a forgery identification result of the image to be detected based on the fusion features, including:

[0036] Get fusion weight;

[0037] The teacher feature t and the student feature s are combined according to the fusion weight i Perform weighting to obtain fusion features;

[0038] Classification is performed based on the fusion features to obtain a forgery identification result of the image to be detected.

[0039] Furthermore, the calculation process of the fusion weight includes:

[0040] Get the image to be detected in the teacher model and the trained student model M i The characteristic variational distribution q in z , z∈[t,i], t represents the teacher model, i represents the trained student model M i ;

[0041] Calculate the characteristic variation distribution q z The mean μ z and variance

[0042] According to the mean μ z , calculate the average mean

[0043] Based on the mean μ z ; The variance and the average mean Get the characteristic variation distribution p z The degree of dispersion η z ;

[0044] According to the discrete degree η z , get the teacher model and the trained student model M i The fusion weight α z .

[0045] An electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements any of the above-mentioned deep fake identification methods based on knowledge de-distillation.

[0046] A computer-readable storage medium, characterized in that computer program instructions are stored on the computer-readable storage medium, and when the computer program instructions are executed by a processor, any of the deep fake identification methods based on knowledge dedistillation as described above is implemented.

[0047] The core of KND technology lies in using the student model to explore the unknown knowledge space of the teacher model while maintaining learning of the target task, thereby improving the model's ability to identify unseen forgery techniques. This approach not only improves the model's generalization ability but also reduces its over-reliance on specific forgery cues, allowing the model to maintain high identification performance when faced with new and unknown forgery techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of deep fake identification using the method of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below through specific implementation cases and drawings.

[0050] The deep fake identification method based on knowledge de-distillation of the present invention utilizes the proposed knowledge de-distillation (KND) technology to construct a framework of teacher model and student model, automatically reducing the impact of low-loss overfitting features on learning more generalized information, and is suitable for improving the generalization ability of deep fake identification. Among them, there are two models in the framework of the present invention: a teacher model and a student model. The teacher model is a pre-trained model that may perform well on a specific task, but may overfit to specific features in the training data. The student model aims to learn a wider range of more generalized features to improve cross-domain performance.

[0051] like Figure 1As shown, the method of the present invention is divided into two parts: training and testing. In the training phase, the student network is de-distilled based on the distribution of teacher model features estimated by GMM. In the testing phase, the teacher and student features are input into the distribution-guided feature fusion module, the weighted fused features are input into their respective classifiers, and the average of the classification results is used as the final identification result. Specifically, the deepfake identification method of the present invention based on knowledge de-distillation includes the following steps 1 to 3.

[0052] Step 1: Estimate the characteristic variational distribution of the teacher model using a Gaussian mixture model.

[0053] A Gaussian Mixture Model (GMM) is used to estimate the probability distribution of the teacher model's features. This distribution represents the knowledge known by the teacher model. The probability distribution of the teacher model's features is used as the basis for the student model's learning.

[0054] Step 2: Train the student model under the constraints of dedistillation loss and classification loss guided by feature variational distribution.

[0055] The student model is trained to maximize the cross entropy between its feature probability distribution and the teacher model’s feature probability distribution. This process encourages the student model to explore knowledge areas not covered by the teacher model, that is, features that have a low probability of appearing in the teacher model’s probability distribution.

[0056] Student characteristic probability distribution p s and teacher characteristics p t The cross entropy H(p s ,p t ) is expressed as formula (1).

[0057] H(p s ,p t )=∑ N -p s (u j )logp t (u j ) (1)

[0058] Among them, u j is the probability distribution sampling of student characteristics s, N is the number of probability distribution sampling of student characteristics, j=1,2,…,N, the probability distribution of student characteristics Kronecker function δ(s,u j ) is defined as formula (2).

[0059]

[0060] When training the student model, the present invention samples u according to the probability distribution of student characteristics. jFormula (1) can be written as Formula (3).

[0061]

[0062] p t Variational distribution q obtained using the Gaussian mixture model t Instead, maximizing Equation (3) can be formulated as minimizing the KND loss function Equation (4).

[0063] L KND (s)=∑ s log q t (s) (4)

[0064] Among them, ∑ s log q t (s) represents the sum of KND losses of all training data in a batch.

[0065] In addition, the student model is also constrained by the classification cross entropy loss formula (5),

[0066] L CLS (p)=∑ p (ylogp+(1-y)log(1-p)) (5)

[0067] Where p is the predicted value of the classifier, y is the label, (ylogp+(1-y)log(1-p) represents the classification cross entropy loss of a training data, ∑ p (ylogp+(1-y)log(1-p)) represents the sum of the categorical cross entropy losses for all training data in a batch.

[0068] By combining the weight λ with the KND loss function and the classification cross entropy loss function, we can obtain the overall loss function formula (6) for training the student model.

[0069] L(s,p)=L KND (s)+L CLS (p) (6)

[0070] In addition, in one embodiment of the present invention, the trained student model can also be used as a new teacher model to recursively train a new student model. This process can dynamically expand the generalization ability of the model, but as the fusion effect converges, the effect of recursive training will tend to stabilize.

[0071] Step 3: Based on the teacher model and the trained student model M i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s iThe fusion is performed, and a forgery identification result of the image to be detected is obtained according to the fusion feature.

[0072] Step 3.1: Get the fusion weights.

[0073] This fusion weight can be manually set, but to avoid the blurring and unclearness caused by directly averaging the features of different models, the present invention also discloses a weighted averaging algorithm that uses the dispersion of the feature probability distribution to guide the weighted averaging of different model features. By controlling the temperature parameter of the softmax operation, the fusion strategy can be adjusted to optimize model performance.

[0074] For n models (including the original teacher model t and n-1 trained student models), the feature t of the zth model z It is obtained by forward propagation of the input data through the model encoder. Feature variation distribution q z The degree of dispersion η z Use formula (7) to measure:

[0075]

[0076] Among them, μ z , Represents the mean and variance of the z-th model feature distribution. The present invention uses the temperature coefficient τ to control the weight α of feature fusion (i) α (i) From formula (8), we can get:

[0077]

[0078] Step 3.2: Combine teacher features t and student features s according to the fusion weights i Perform weighted addition to obtain the fusion feature f.

[0079] The fusion feature f is obtained by formula (9).

[0080]

[0081] Among them, t z ∈[t,s i ].

[0082] Step 3.3: Classify based on the fused features to obtain the forgery identification result of the image to be detected.

[0083] The classification process of the present invention is to send the fusion feature f to the classifier of each model z respectively, then perform average processing on the classification results, and obtain the forgery identification result of the image to be detected based on the average processing result.

[0084] In summary, unlike traditional knowledge distillation, in knowledge de-distillation, the teacher model acts as a "negative example," encouraging the student model to explore knowledge unknown to the teacher in the dataset while simultaneously solving the forgery detection task. This paper considers that different probability distributions represent different types of information and therefore maximizes the cross-entropy between the student and teacher feature probability distributions to increase the difference between them. By doing so, the student model automatically avoids being influenced by the low-loss, overfitting features learned by the teacher model, allowing it to focus on more generalizable features.

[0085] The present invention first uses a Gaussian mixture model (GMM) to estimate the probability distribution of the teacher model's features as a representation of the teacher's known knowledge. Then, the student model is trained to maximize the cross entropy between its feature probability distribution and the teacher model's feature probability distribution, thereby encouraging the student model to explore knowledge areas not covered by the teacher model. In this process, the student model extracts features and learns the target task by forward inputting data, while being guided by the cross entropy loss to increase the difference with the teacher model's features. Then, the features of the student model are fused with the features of the teacher model as scalable components, and this process is guided by their respective probability distributions. In addition, the present invention also proposes a recursive training method, which allows the trained student model to be used as a new teacher model for training subsequent student models, thereby dynamically expanding the generalization ability of the model. Finally, through the distribution-guided feature fusion technology, the features of multiple models are combined to further improve the performance of the model in the deep fake identification task. Through this method, the performance of the model in the unknown counterfeiting technology identification task can be effectively improved, and more accurate deep fake identification can be achieved.

[0086] Below, two specific experiments are used to illustrate the deep fake identification method based on knowledge de-distillation provided by the present invention.

[0087] Example 1 Generalization test of facial forgery identification based on knowledge de-distillation.

[0088] Take FF++ as the training set and Celeb-DF-v1 as the test set as an example:

[0089] 1) First, use the Xception network to fully train on the training set to extract the features of the fake and real facial images;

[0090] 2) Using Gaussian mixture model (GMM) to estimate the probability distribution of teacher model features as a representation of the teacher’s known knowledge;

[0091] 3) Train the student model to maximize the cross entropy between its feature probability distribution and the teacher model feature probability distribution estimated in 2), encouraging the student model to explore knowledge areas not covered by the teacher model;

[0092] 4) In the testing phase, the features of the student model and the teacher model are subjected to distribution-guided feature fusion to obtain a more comprehensive representation of facial forgery features;

[0093] 5) Test on Celeb-DF-v1, input the weighted fusion features obtained in 4) into each classifier, and take the average of the classification results as the final identification result;

[0094] 6) Comparing the results of the method of the present invention with those of other methods, the obtained AUC (area under the ROC curve) for facial forgery identification is shown in Table 1.

[0095] Table 1. AUC of cross-dataset tests on the Celeb-DF-v1 dataset

[0096] Training dataset Test dataset Method of the present invention UCF SPSL FF++ Celeb-DF-v1 0.881 0.779 0.815

[0097] In Table 1, UCF and SPSL are compared methods. Both methods are methods for improving generalization in facial forgery identification. Therefore, the method of the present invention can be used for comparison.

[0098] According to the results in Table 1, it is shown that the method of the present invention can significantly improve the generalization effect of facial forgery identification.

[0099] Example 2: Generalization test of synthetic image identification based on knowledge dedistillation.

[0100] Take the data generated by ProGAN in ForenSynths as the training set and the data generated by DALLE as the test set as an example:

[0101] 1) First, use the Xception network to fully train on the training set to extract the features of synthetic images and real images;

[0102] 2) Using Gaussian mixture model (GMM) to estimate the probability distribution of teacher model features as a representation of the teacher’s known knowledge;

[0103] 3) Train the student model to maximize the cross entropy between its feature probability distribution and the teacher model feature probability distribution estimated in 2), encouraging the student model to explore knowledge areas not covered by the teacher model;

[0104] 4) In the testing phase, the features of the student model and the teacher model are fused using distribution-guided feature fusion to obtain a more comprehensive synthetic image feature representation;

[0105] 5) Test on DALLE, input the weighted fusion features obtained in 4) into each classifier, and take the average of the classification results as the final identification result;

[0106] 6) Comparing the results of the method of the present invention with those of other methods, the identification accuracy rates obtained are shown in Table 2.

[0107] Table 2. Accuracy of cross-dataset tests on the DALLE dataset

[0108] Training dataset Test dataset Method of the present invention F3Net Ojha ProGAN DALLE 0.906 0.716 0.895

[0109] In Table 2, F3Net and Ojha are compared methods. Both methods are methods for improving generalization by synthesizing image identification. Therefore, the method of the present invention can be used for comparison.

[0110] According to the results in Table 2, it is shown that the method of the present invention can significantly improve the generalization effect of synthetic image identification.

[0111] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Those skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be based on the claims.

Claims

1. A deep fake identification method based on knowledge de-distillation, characterized by: The method comprises: The Gaussian mixture model is used to estimate the characteristic variation distribution of the teacher model, and the student model is trained under the anti-distillation loss and classification loss constraints guided by the characteristic variation distribution; wherein, after obtaining the trained student model M i Afterwards, the trained student model M i As the teacher model, to obtain the trained student model M i+1 ; Based on the teacher model and the trained student model M i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s i Performing fusion, and obtaining a forgery identification result of the image to be detected based on the fusion features; Wherein, under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution, the student model is trained, including: Use the student model to obtain the probability distribution sampling u of the student features of the training sample j ; Sampling u based on the characteristic variational distribution and the probability distribution j , get the cross entropy of the probability distribution of student features and the probability distribution of teacher features; On each training batch, the anti-distillation loss is obtained by maximizing the cross entropy between the probability distribution of the student features and the probability distribution of the teacher features; On each training batch, the classification cross entropy loss of the student model is obtained using the training samples; Backpropagation is performed based on the dedistillation loss and the classification cross entropy loss to update the parameters of the student model.

2. The method according to claim 1, characterized in that The teacher feature t and the student feature s i Performing fusion and obtaining a forgery identification result of the image to be detected based on the fusion features, including: Get fusion weight; The teacher feature t and the student feature s are combined according to the fusion weight i Perform weighting to obtain fusion features; Classification is performed based on the fusion features to obtain a forgery identification result of the image to be detected.

3. The method according to claim 2, characterized in that The calculation process of the fusion weight includes: Get the image to be detected in the teacher model and the trained student model M i The characteristic variational distribution q in z , z∈[t,i], t represents the teacher model, i represents the trained student model M i ; Calculate the characteristic variation distribution q z The mean μ z and variance According to the mean μ z , calculate the average mean Based on the mean μ z ; The variance and the average mean Get the characteristic variation distribution p z The degree of dispersion η z ; According to the discrete degree η z , get the teacher model and the trained student model M i The fusion weight α z .

4. A deep fake identification system based on knowledge de-distillation, characterized by: The system comprises: The training module is used to estimate the characteristic variation distribution of the teacher model using the Gaussian mixture model, and train the student model under the constraints of the anti-distillation loss and classification loss guided by the characteristic variation distribution; wherein, after obtaining the trained student model M i Afterwards, the trained student model M i As the teacher model, to obtain the trained student model M i+1 ; Inference module, used for inference based on teacher model and trained student model M i Get the teacher feature t and student feature s of the image to be detected i Then, the teacher feature t and the student feature s i Performing fusion, and obtaining a forgery identification result of the image to be detected based on the fusion features; Wherein, under the constraints of the anti-distillation loss and classification loss guided by the feature variational distribution, the student model is trained, including: Use the student model to obtain the probability distribution sampling u of the student features of the training sample j ; Sampling u based on the characteristic variational distribution and the probability distribution j , get the cross entropy of the probability distribution of student features and the probability distribution of teacher features; On each training batch, the anti-distillation loss is obtained by maximizing the cross entropy between the probability distribution of the student features and the probability distribution of the teacher features; On each training batch, the classification cross entropy loss of the student model is obtained using the training samples; Backpropagation is performed based on the dedistillation loss and the classification cross entropy loss to update the parameters of the student model.

5. The system according to claim 4, characterized in that The teacher feature t and the student feature s i Performing fusion and obtaining a forgery identification result of the image to be detected based on the fusion features, including: Get fusion weight; The teacher feature t and the student feature s are combined according to the fusion weight i Perform weighting to obtain fusion features; Classification is performed based on the fusion features to obtain a forgery identification result of the image to be detected.

6. The system according to claim 5, characterized in that The calculation process of the fusion weight includes: Get the image to be detected in the teacher model and the trained student model M i The characteristic variational distribution q in z , z∈[t,i], t represents the teacher model, i represents the trained student model M i ; Calculate the characteristic variation distribution q z The mean μ z and variance According to the mean μ z , calculate the average mean Based on the mean μ z ; The variance and the average mean Get the characteristic variation distribution p z The degree of dispersion η z ; According to the discrete degree η z , get the teacher model and the trained student model M i The fusion weight α z .

7. An electronic device, characterized in that: The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the deep fake identification method based on knowledge de-distillation as described in any one of claims 1-3.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the deep fake identification method based on knowledge de-distillation as described in any one of claims 1 to 3.

Images