A deep fake image detection method based on knowledge distillation and domain adversarial training

Abstract

The present application relates to the field of image detection, and provides a deep fake image detection method based on knowledge distillation and domain adversarial training, comprising: constructing a source domain data set and a target domain data set; using the source domain data set to perform source domain training of a model to obtain a trained basic model; using the source domain data set and the target domain data set and the trained basic model to perform model migration training based on knowledge distillation and domain adversarial training to obtain a deep fake image detection model; when a new type of deep fake image appears and the amount of labeled data of the new type of deep fake image cannot perform domain adversarial training, using the labeled data of the new type of deep fake image to fine-tune the deep fake image detection model. The present application combines different transfer learning methods, on the one hand, improves the detection accuracy of the deep fake image detection model on known types of fake images, and on the other hand, quickly adapts to the situation of a small amount of new type of labeled data.

Description

Technical Field

[0001] This invention relates to the field of image detection, and more specifically, to a deepfake image detection method based on knowledge distillation and domain adversarial training. Background Technology

[0002] With the development of generative artificial intelligence technology, deepfake technology has become one of the most popular multimedia tampering techniques. For example, FaceSwap replaces faces in images or videos with faces from different sets of images or videos; Face2Face transfers facial expressions from a source image or video to a target image or video while preserving the target person's identity. These techniques can be easily used for deceptive advertising and other malicious purposes, and are highly susceptible to exploitation by fraudsters, causing irreparable damage. Researching deepfake image detection is of great significance in effectively preventing the misuse of this technology.

[0003] Deepfake image detection is generally defined as a binary classification task that predicts whether a given image is real or fake. Existing detection methods mainly use traditional handcrafted features, leverage unique biometric features of faces, and use hierarchical feature representations automatically learned by deep convolutional neural networks. While these methods have achieved certain detection performance, they generally suffer from the following two problems:

[0004] First, there are various methods of deepfake. For example, the FaceForensics++ dataset, which is commonly used in academia, includes four types of deepfake data: DeepFake, Face2Face, Faceswap, and NeuralTextures. Typical deep neural network models are trained on datasets that combine multiple deepfake images, often resulting in poor detection performance. The detection results for various types of forgery vary, possibly because the excessive diversity within the dataset makes it difficult for the model to distinguish the key differences between forged and real images.

[0005] Secondly, the detection model has poor generalization ability. This means that when presented with new types of deepfake images, even if they are semantically similar, the model's detection performance is significantly affected. This is mainly because deep neural networks may overfit to features useful for a specific task, and therefore cannot be transferred to detecting deepfake images generated using different techniques.

[0006] To address these two issues, academia has conducted much valuable research, with transfer learning being a typical approach. Studies have found that when training datasets contain various deepfake image datasets, compared to merging training, training the detection model entirely on a single deepfake image dataset first allows the model to gain a detailed understanding of the differences between genuine and fake images of a single deepfake type. Then, transfer learning is used to expand its knowledge of different deepfake image types, achieving better detection results. To improve the detection capability for new types of deepfake images through transfer learning, typical approaches include model (local) fine-tuning transfer, feature-supervised (parameter regularization, feature constraints, knowledge distillation) model fine-tuning transfer, and domain adaptation. While these methods can transfer knowledge to the target domain to some extent with relatively limited target domain data, current research typically involves single-step transfer learning from the source domain to the target domain, still facing the catastrophic forgetting problem. That is, when transferring the model to the target domain, knowledge from the source domain may be forgotten, leading to a significant drop in detection performance. Summary of the Invention

[0007] The present invention aims to provide a deepfake image detection method based on knowledge distillation and domain adversarial training to solve the problems existing in the current deepfake image detection.

[0008] This invention provides a deepfake image detection method based on knowledge distillation and domain adversarial training, comprising:

[0009] Construct the dataset, including the source domain dataset and the destination domain dataset;

[0010] The model is trained in the source domain using the source domain dataset to obtain a well-trained base model;

[0011] Using source domain datasets, target domain datasets, and a pre-trained base model, a model transfer training based on knowledge distillation and domain adversarial training is performed to obtain a deepfake image detection model; the deepfake image detection model is used for deepfake image detection.

[0012] When a new type of deepfake image emerges and the amount of labeled data for this new type of deepfake image is insufficient for domain adversarial training, the labeled data for this new type of deepfake image is used to fine-tune the deepfake image detection model.

[0013] In a preferred embodiment, the step of training the model in the source domain using the source domain dataset includes:

[0014] Choose a base model;

[0015] The base model is used to perform true / false classification training on the source domain dataset to obtain a trained base model; the true / false classification loss function for true / false classification training is the classification cross-entropy loss.

[0016] In a preferred embodiment, the step of performing model transfer training based on knowledge distillation and domain adversarial training using the source domain dataset, the target domain dataset, and the trained base model includes:

[0017] Teacher and student models are built based on the pre-trained base model;

[0018] The teacher model and student model are subjected to model transfer training based on knowledge distillation and domain adversarial training on the source domain dataset and the destination domain dataset to obtain a deepfake image detection model.

[0019] In a preferred embodiment, the construction of the teacher model and student model based on the trained base model includes:

[0020] The trained base model is divided into interconnected feature extractors. F True / False Classifier C ;

[0021] A copy of the trained base model is made and its parameters are frozen to serve as the teacher model.

[0022] Feature extractor of the trained base model F Add a domain classifier D Later used as a student model; among which, the domain classifier D No operations are performed during forward propagation, and the feature extractor is updated during backward propagation. F When dealing with parameters, the gradient needs to be reversed.

[0023] In a preferred embodiment, the loss function for model transfer training based on knowledge distillation and domain adversarial training includes classification cross-entropy loss, domain adversarial training loss, feature representation loss, and knowledge distillation loss.

[0024] In a preferred embodiment, the loss function for fine-tuning the deepfake image detection model includes classification cross-entropy loss, feature representation loss, and knowledge distillation loss.

[0025] In a preferred embodiment, classification cross-entropy loss Represented as:

[0026]

[0027] in, N It is the number of samples input to the model. It is the first iThe true and false labels of each sample It is the first i The predicted probability of a sample.

[0028] In a preferred embodiment, domain adversarial training loss Represented as:

[0029]

[0030] in, This represents the samples input to the model. N It is the number of samples input to the model. It is the first i Domain labels for each sample, F Indicates feature extractor, Indicates the sample Extracted features This represents the gradient reversal function, whose input is a set of samples. Extracted features , The domain classifier takes the output of the gradient inversion function as its input.

[0031] In a preferred embodiment, feature representation loss Defined as:

[0032]

[0033] in, The maximum value among the vectors after softmax normalization of image features on the student model. Corresponding image features; The maximum value among the vectors after softmax normalization of image features on the teacher model. The corresponding image features; the range of values ​​for the vectors obtained by softmax normalization of the image features in the student and teacher models is as follows: The value interval is v.

[0034] In a preferred embodiment, knowledge distillation loss for:

[0035]

[0036]

[0037]

[0038] in, For the temperature-based softmax function in the student model, For the temperature-based softmax function in the teacher model; For temperature, N It is the number of samples input to the model. The student model is the first i The unnormalized scores of each sample The student model is the first j The unnormalized scores of each sample The teacher model is for the first i The unnormalized scores of each sample The teacher model is for the first j The unnormalized scores of each sample; It is an exponential function with the natural constant e as its base.

[0039] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0040] This invention cleverly combines the advantages of different transfer learning methods, using a unified model framework to improve the detection accuracy of deepfake image detection models for known types of forged images, while also rapidly adapting to situations where only a very small number of new types of labeled data appear. Specifically, during model transfer training, the ability of domain adversarial training to extract general features is fully utilized, while the supervision of knowledge distillation enhances the stability of domain adversarial training, thereby improving the model's detection accuracy for known types of forged images. During model use, if a new type of forged image appears with only a very small amount of labeled data, the model can be fine-tuned under the supervision of knowledge distillation to quickly adapt to the new type of data, avoiding overfitting. Attached Figure Description

[0041] Figure 1 The flowchart below provides a deepfake image detection method based on knowledge distillation and domain adversarial training, which is an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram illustrating the principle of model transfer training based on knowledge distillation and domain adversarial training in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0046] Current deepfake image detection faces two major challenges: first, training a single model based on merging data from multiple deepfake types often fails to achieve satisfactory detection results; second, the detection model's generalization ability is poor when faced with new types of deepfake images. However, the data situations differ between the two scenarios. Model training utilizes known deepfake types with labeled and abundant data; while dealing with new types of deepfake images, labeled data may be scarce or plentiful, and unlabeled data may also exist. Research indicates that transfer learning may be a solution to these two problems. However, current research on deepfake image detection based on transfer learning only covers single-pass transfer learning from the source domain to the target domain, still facing the problem of catastrophic forgetting. Training detection models for multiple types of deepfake images through transfer learning, or adapting to constantly emerging new types of deepfake images, requires multiple transfer learning sessions from the source domain model to the target domain. This presents a greater challenge in achieving knowledge transfer while avoiding catastrophic forgetting.

[0047] In view of this, this invention extensively combines the advantages of existing transfer learning methods and proposes a deepfake image detection method based on knowledge distillation and domain adversarial training, aiming to solve the problems currently faced by deepfake image detection with a unified model framework. The design principle of this invention is as follows:

[0048] (1) Model fine-tuning transfer learning is a commonly used technique in transfer learning. In deepfake image detection, it involves adjusting some or all parameters of the source domain model to adapt it to the target domain data. Model fine-tuning is suitable for labeled data, requiring less training time and allowing for rapid adaptation to new data types through incremental learning. However, for new data types with only a very small amount of labeled data, careful design of the fine-tuning method is necessary; otherwise, it is easy to fall into local suboptimal solutions, overfitting the target domain-specific features while destroying the general features learned from the source domain. Therefore, model fine-tuning is not suitable for multiple target domain transfer learning operations. Model fine-tuning usually requires some feature supervision (parameter regularization, feature constraints, knowledge distillation) to minimize the forgetting of source domain features.

[0049] (2) Domain adversarial training uses a domain classifier to distinguish whether features come from the source or target domain. The backbone network attempts to learn features that are useful for the task and are domain-inseparable. Adversarial training reduces the distribution differences between the source and target domains while the backbone network learns feature representations, making it difficult for the domain classifier to distinguish between the two domains. Domain adversarial training can utilize both a small amount of labeled data and a large amount of unlabeled data. Domain adversarial training eliminates inter-domain differences by obfuscating the features of the domain discriminator, thereby improving transfer performance. Furthermore, because source domain data is involved in the training, it is less likely to destroy the general features learned from the source domain. Extending domain adversarial training may be more suitable for multiple target domain transfer learning. However, domain adversarial training is unstable, has a long training time, and cannot quickly and dynamically adapt to new types of forged data. The deepfake image detection method based on knowledge distillation and domain adversarial training aims to: fully utilize the ability of domain adversarial training to extract general features during model training, and improve the stability of domain adversarial training through the supervision of knowledge distillation, thereby improving the model's detection accuracy for known forged images; and in the use of the model, if a new type of image forgery appears and there is only a very small amount of labeled data, the model can be fine-tuned under the supervision of knowledge distillation to quickly adapt to the new type of data.

[0050] Based on the above design principles, such as Figure 1 As shown, this embodiment of the invention provides a deepfake image detection method based on knowledge distillation and domain adversarial training, comprising:

[0051] S100, construct the dataset, including the source domain dataset and the destination domain dataset;

[0052] S200: Use the source domain dataset to train the model in the source domain to obtain a well-trained base model;

[0053] S300: Using the source domain dataset, the target domain dataset, and the trained base model, perform model transfer training based on knowledge distillation and domain adversarial training to obtain a deepfake image detection model; the deepfake image detection model is used for deepfake image detection.

[0054] S400: When a new type of deepfake image appears and the amount of labeled data for this new type of deepfake image is insufficient for domain adversarial training, the labeled data for this new type of deepfake image is used to fine-tune the deepfake image detection model.

[0055] Therefore, this invention cleverly combines the advantages of different transfer learning methods and uses a unified model framework to improve the detection accuracy of deepfake image detection models for known types of forged images on the one hand, and to quickly adapt to situations where only a very small number of new types of labeled data appear. Specifically, during model transfer training, the ability of domain adversarial training to extract general features is fully utilized, while the supervision of knowledge distillation is used to improve the stability of domain adversarial training, thereby improving the model's detection accuracy for known types of forged images. In model usage, if a new type of forged image appears and there is only a very small amount of labeled data, the model can be fine-tuned under the supervision of knowledge distillation to quickly adapt to the new type of data and avoid overfitting.

[0056] The following details the specific implementation of the deepfake image detection method based on knowledge distillation and domain adversarial training.

[0057] S100, Construct the dataset, including the source domain dataset and the destination domain dataset:

[0058] In this embodiment, the real and fake videos in the FaceForensics++ dataset are processed to form a source domain dataset and a destination domain dataset. Specifically, the four types of deepfake videos (DeepFake, Face2Face, Faceswap, and NeuralTextures) and real videos in the FaceForensics++ dataset are processed by extracting frames using the FFmpeg library. At the same time, the MTCNN (Multi-task Cascaded Convolutional Networks) algorithm is used to crop out the faces in each extracted frame, forming a four-type deepfake image dataset that contains both real and fake images. From the four types of deepfake image datasets, one type of deepfake image dataset is selected as the source domain dataset, such as the DeepFake dataset, and the other type of deepfake image dataset is selected as the destination domain dataset, such as the Face2Face dataset.

[0059] S200: Use the source domain dataset to train the model in the source domain to obtain a trained base model.

[0060] S201, Select a base model. In this embodiment, the Xception network is selected as the base model.

[0061] S202, using the aforementioned basic model ( Figure 2 The purple portion (in the middle) is used to train a true / false classification model on the source domain dataset to obtain a trained base model. The true / false classification loss function used for training is the classification cross-entropy loss. The definition is as follows:

[0062]

[0063] in, N It is the number of samples input to the model. It is the first i The true and false labels of each sample It is the first i The predicted probability of a sample.

[0064] S300 utilizes the source domain dataset, the target domain dataset, and the pre-trained base model to perform model transfer training based on knowledge distillation and domain adversarial training, resulting in a deepfake image detection model:

[0065] To extend a model's detection capabilities in the target domain, model transfer training based on knowledge distillation and domain adversarial training is necessary. The dataset used for model transfer training is a combination of source and target domain data. Specifically:

[0066] S301, Construct teacher model (T) and student model (S) based on the trained base model.

[0067] First, the trained base model ( Figure 2 The purple portion is divided into interconnected feature extractors. F True / False Classifier C This embodiment uses the Xception network as an example, taking the fully connected layer (FC) of the Xception network as the true / false classifier. C The network preceding the fully connected (FC) layer in the Xception network serves as a feature extractor. F The image feature representation vector f is the output of the Xception network before the fully connected layer FC, and the vector dimension is 2048.

[0068] Next, a copy of the trained base model is made and its parameters are frozen to serve as the teacher model. Figure 2 (Middle gray area). The supervisory role of the teacher model on the student model is mainly reflected in the feature representation loss. and knowledge distillation loss .

[0069] Then, in the feature extractor of the trained base model F Add a domain classifier D ( Figure 2 The yellow portion in the middle is used as the student model. That is to say, the student model, in addition to the pre-trained base model (…), is used as the student model. Figure 2 In addition to the purple part, it also includes a domain classifier. D ( Figure 2 (Middle yellow section). Domain classifier DThis refers to a given input sample Domain classifier D It can predict the domain labels of samples. Here, the label refers to either the source domain or the destination domain. Domain adversarial training refers to the feature extractor. F It will be updated in a competitive manner, including two competing objectives:

[0070] In terms of true / false classification, feature extractors F True / False Classifier C It will be updated with the aim of correctly classifying the authenticity of the data (labeled source and destination domain data);

[0071] In terms of domain classification, domain classifiers D The update will be performed with the aim of correctly classifying which domain a given sample comes from.

[0072] But feature extractor F This will enable the domain classifier D The feature extractor cannot be updated for the purpose of correctly classifying the neighborhood of a given sample. This will ensure that the feature extractor... F Domain-invariant features are learned. To achieve domain adversarial training, the domain classifier... D No operations are required during forward propagation; the feature extractor is updated during backward propagation. F When dealing with the parameters, it is necessary to reverse the gradient, that is, to take the negative gradient. Here, the gradient reversal function is defined as follows: At this point, the input is the feature extractor. F For the sample Extracted features Its forward propagation and backward propagation behaviors are represented as follows:

[0073]

[0074]

[0075] in, The gradient reversal function represents the pair of gradients. The derivative, Represents the identity matrix. Used to control the intensity of adversarial training.

[0076] For deepfake image detection, real samples show almost no significant differences between the source and destination domains. If the goal is to eliminate domain differences in low-level features (such as texture and lighting), align mid-level semantic features (such as facial parts), and preserve high-level task-relevant features, this can be achieved by applying gradient inversion of varying intensities to different network layers. α 浅层 > α 中层 >α 高层 , α 浅层 For shallow strength, α 中层 For medium layer strength, α 高层 This represents high-level intensity. Therefore, the domain adversarial training loss... for:

[0077]

[0078] in, This represents the samples input to the model. N It is the number of samples input to the model. It is the first i Domain labels for each sample, F Indicates feature extractor, Indicates the sample Extracted features This represents the gradient reversal function, whose input is a set of samples. Extracted features , The domain classifier takes the output of the gradient inversion function as its input.

[0079] Feature representation loss It is designed based on the assumption that there must be similar features between different types of deepfake images. The teacher model trained on the source domain dataset can help the student model learn the target domain with fewer data samples. The image features on the teacher model are represented as... The image features on the student model are represented as After performing softmax normalization on these image features, the maximum value in the vector is taken. The range of values ​​for the vectors obtained by softmax normalization of image features from the student and teacher models. Image features within the region are stored at predetermined intervals. Segmenting the features helps minimize domain shifts during the learning process. Therefore, the feature representation loss... Defined as:

[0080]

[0081] in, The maximum value among the vectors after softmax normalization of image features on the student model. Corresponding image features; The maximum value among the vectors after softmax normalization of image features on the teacher model. The corresponding image features.

[0082] Knowledge distillation loss It measures the difference between the student model's output and the teacher model's soft labels. Assumption The student model is the first i The unnormalized score (logit) of each sample. The student model is the first j The unnormalized scores of each sample The teacher model is for the first i The unnormalized score (logit) of each sample. The teacher model is for the first j The unnormalized scores of each sample are represented by the temperature-controlled softmax function as follows:

[0083] The temperature-based softmax function in the student model is expressed as:

[0084]

[0085] The temperature-based softmax function in the teacher model is expressed as:

[0086]

[0087] in, For temperature, a larger temperature parameter can help the student model learn the knowledge of the teacher model better by making the probability distribution smoother; It is an exponential function with the natural constant e as its base. Therefore, knowledge distillation loss... Represented as:

[0088]

[0089] In summary, the loss function for model transfer training based on knowledge distillation and domain adversarial training is... The loss includes classification cross-entropy loss, domain adversarial training loss, feature representation loss, and knowledge distillation loss, expressed as:

[0090]

[0091] in, , and All of these are hyperparameters.

[0092] It should be noted that at this point, the model has only been trained in two domains: the source domain and the destination domain. If training is to continue, the domain classifier needs to be expanded to three classes for further training, and so on.

[0093] S400, when a new type of deepfake image emerges and the amount of labeled data for this new type of deepfake image is insufficient for domain adversarial training, the deepfake image detection model is fine-tuned using labeled data (i.e., a very small amount of labeled data) from this new type of deepfake image. Since domain adversarial training is not possible, the loss function used for fine-tuning the deepfake image detection model is... The loss includes classification cross-entropy loss, feature representation loss, and knowledge distillation loss, expressed as:

[0094]

[0095] At this point, due to the supervisory role of feature representation loss and knowledge distillation loss, overfitting can be effectively avoided with limited labeled data. In this embodiment, stochastic gradient descent is used to optimize the model during training and fine-tuning.

[0096] Based on the same technical concept, embodiments of the present invention also provide an electronic device that can implement the deepfake image detection method based on knowledge distillation and domain adversarial training provided in the above embodiments of the present invention. In one embodiment, the electronic device can be a server, a terminal device, or other electronic device. Figure 3 As shown, the electronic device may include:

[0097] At least one processor and a memory connected to the at least one processor. In this embodiment of the invention, the specific connection medium between the processor and the memory is not limited. Figure 3 The example used is the connection between the processor and memory via a bus. The bus... Figure 3 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. Buses can be divided into address buses, data buses, control buses, etc., but for ease of representation, [the specific bus type is not shown here]. Figure 3 The processor is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, a processor can also be called a controller; there are no restrictions on the name.

[0098] In this embodiment of the invention, the memory stores instructions that can be executed by at least one processor. By executing the instructions stored in the memory, at least one processor can execute the deepfake image detection method based on knowledge distillation and domain adversarial training discussed above.

[0099] The processor is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory and calling data stored in memory, it can monitor the device's various functions and process data, thereby enabling overall monitoring of the device.

[0100] In an alternative design, the processor may include one or more processing units. The processor may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor. In some embodiments, the processor and memory may be implemented on the same chip; in some embodiments, they may also be implemented separately on separate chips.

[0101] The processor can be a general-purpose processor, such as a CPU, digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the deepfake image detection method based on knowledge distillation and domain adversarial training disclosed in the embodiments of this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0102] Memory, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory can include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic memory, magnetic disk, optical disk, etc. Memory is any other medium capable of carrying or storing desired program code in the form of instructions or data structures, and accessible by a computer, but is not limited thereto. In embodiments of the present invention, memory can also be a circuit or any other device capable of implementing storage functions, used to store program instructions and / or data.

[0103] By designing and programming the processor, the code corresponding to the deepfake image detection method based on knowledge distillation and domain adversarial training described in the foregoing embodiments can be embedded into the chip, enabling the chip to execute the steps of the method described in the foregoing embodiments during runtime. How to design and program the processor is a technique well-known to those skilled in the art and will not be elaborated upon here.

[0104] Based on the same inventive concept, embodiments of the present invention also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a deep forgery image detection method based on knowledge distillation and domain adversarial training as described above.

[0105] In some alternative embodiments, the present invention also provides a method for detecting deepfake images based on knowledge distillation and domain adversarial training, which can also be implemented as a program product comprising program code that, when the program product is run on a device, causes the control device to perform the steps in the method for detecting deepfake images based on knowledge distillation and domain adversarial training according to various exemplary embodiments of the present invention as described above.

[0106] It should be noted that although several units or sub-units of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Furthermore, although the operation of the method of the invention is described in a specific order in the drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0107] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0108] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a server, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] Program code for performing the operations of this invention can be written using any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0110] In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deepfake image detection method based on knowledge distillation and domain adversarial training, characterized in that, include: Construct the dataset, including the source domain dataset and the destination domain dataset; The model is trained in the source domain using the source domain dataset to obtain a well-trained base model; Using the source domain dataset, the target domain dataset, and the trained base model, a model transfer training based on knowledge distillation and domain adversarial training is performed to obtain a deepfake image detection model; the deepfake image detection model is used for deepfake image detection. When a new type of deepfake image emerges and the amount of labeled data for this new type of deepfake image is insufficient for domain adversarial training, the labeled data for this new type of deepfake image is used to fine-tune the deepfake image detection model. The process of performing model transfer training based on knowledge distillation and domain adversarial training using source domain datasets, target domain datasets, and a pre-trained base model includes: Teacher and student models are built based on the pre-trained base model; The teacher model and student model are subjected to model transfer training based on knowledge distillation and domain adversarial training on the source domain dataset and the destination domain dataset to obtain a deepfake image detection model. The construction of teacher and student models based on the pre-trained base model includes: The trained base model is divided into interconnected feature extractors. F True / False Classifier C ; A copy of the trained base model is made and its parameters are frozen to serve as the teacher model. Feature extractor of the trained base model F Add a domain classifier D Later used as a student model; among which, the domain classifier D No operations are performed during forward propagation, and the feature extractor is updated during backward propagation. F When dealing with parameters, the gradient needs to be reversed.

2. The deepfake image detection method based on knowledge distillation and domain adversarial training according to claim 1, characterized in that, The method of training the model in the source domain using the source domain dataset includes: Choose a base model; The base model is used to perform true / false classification training on the source domain dataset to obtain a trained base model; the true / false classification loss function for true / false classification training is the classification cross-entropy loss.

3. The deepfake image detection method based on knowledge distillation and domain adversarial training according to claim 2, characterized in that, The loss functions for model transfer training based on knowledge distillation and domain adversarial training include classification cross-entropy loss, domain adversarial training loss, feature representation loss, and knowledge distillation loss.

4. The deepfake image detection method based on knowledge distillation and domain adversarial training according to claim 3, characterized in that, Loss functions for fine-tuning deepfake image detection models include classification cross-entropy loss, feature representation loss, and knowledge distillation loss.

5. The deepfake image detection method based on knowledge distillation and domain adversarial training according to claim 4, characterized in that, Classification cross-entropy loss Represented as: in, N It is the number of samples input to the model. It is the first i The true and false labels of each sample It is the first i The predicted probability of a sample.

6. The deepfake image detection method based on knowledge distillation and domain adversarial training according to claim 4, characterized in that, Domain adversarial training losses Represented as: in, This represents the samples input to the model. N It is the number of samples input to the model. It is the first i Domain labels for each sample, F Indicates feature extractor, Indicates the sample Extracted features This represents the gradient reversal function, whose input is a set of samples. Extracted features , The domain classifier takes the output of the gradient inversion function as its input.

7. The deepfake image detection method based on knowledge distillation and domain adversarial training according to claim 4, characterized in that, Feature representation loss Defined as: in, The maximum value among the vectors after softmax normalization of image features on the student model. Corresponding image features; The maximum value among the vectors after softmax normalization of image features on the teacher model. The corresponding image features; the range of values ​​for the vectors obtained by softmax normalization of the image features in the student and teacher models is as follows: The value interval is v.

8. The deepfake image detection method based on knowledge distillation and domain adversarial training according to claim 4, characterized in that, Knowledge distillation loss for: in, For the temperature-based softmax function in the student model, For the temperature-based softmax function in the teacher model; For temperature, N It is the number of samples input to the model. The student model is the first i The unnormalized scores of each sample The student model is the first j The unnormalized scores of each sample The teacher model is for the first i The unnormalized scores of each sample The teacher model is for the first j The unnormalized scores of each sample; It is an exponential function with the natural constant e as its base.

Images