Deepfake detection method, apparatus, device and readable storage medium

Abstract

The application discloses a deep fake detection method, device and equipment and a readable storage medium. The method comprises the following steps: acquiring training data; constructing a quadruple sample by using the training data, wherein the quadruple sample comprises an anchor sample, an enhanced positive sample, a difficult negative sample and an irrelevant negative sample; the semantic similarity between the enhanced positive sample, the difficult negative sample, the irrelevant negative sample and the anchor sample satisfies a preset ordinal relationship; determining a loss function for model training according to the quadruple sample; optimizing a feature extraction network by using the loss function to obtain a converged feature extraction network; and detecting an image to be detected by using the feature extraction network. The application converts the fake detection into a fine list sorting problem, enables the feature extraction network to learn a structured feature space capable of measuring the authenticity through network optimization, and realizes accurate and robust detection of deep fake.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a deepfake detection method, apparatus, device, and readable storage medium. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, deepfake technology has become increasingly sophisticated. The images or videos it generates are now visually indistinguishable from real ones, posing a serious threat to personal privacy and social security. Therefore, developing efficient and reliable deepfake detection technology has become an urgent need in the field of cybersecurity.

[0003] Currently, the mainstream technologies for deepfake detection mainly revolve around deep neural networks, broadly categorized into two paradigms. The first is a binary classification-based paradigm, which simplifies the detection task into a binary classification problem of determining whether an input sample is "real" or "fake." These methods typically rely on the powerful feature extraction capabilities of deep networks, attempting to find subtle traces left by the forgery process in images. However, this method is prone to overfitting to traces of specific forgery methods in the training data. When faced with novel forgery techniques not present in the training set, or images post-processed through social media compression, encoding, or decoding, its generalization ability and robustness significantly decrease, leading to inaccurate detection results.

[0004] The second approach is based on metric learning, which aims to improve the model's generalization ability by learning a more discriminative feature space. The core of this method is a region-level contrastive loss function designed using a precise "fusion mask" that identifies the forged regions. Its goal is to increase the feature distance between the "source face region" and the "target face region" within the same forged sample in the feature space, while simultaneously decreasing the feature distance between the "target face region" and the original real target face. However, this approach heavily relies on the prior information of the "fusion mask," severely limiting its application to face-swapping forgery techniques with clear tampering boundaries and readily available mask information. It is unsuitable for broader forgery types, such as those generated entirely by Artificial Intelligence (AI), style transfer, or facial attribute editing without clear boundaries, leading to inaccurate detection results. Summary of the Invention

[0005] This application provides a deepfake detection method, apparatus, device, and readable storage medium, which solves the problem of inaccurate detection results in current deepfake detection technologies.

[0006] Firstly, a deepfake detection method is provided, including:

[0007] Obtain training data;

[0008] The training data is used to construct quadruple samples, which include: anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples; wherein the semantic similarity between the augmented positive samples, the hard negative samples, the irrelevant negative samples and the anchor samples satisfies a preset ordinal relationship.

[0009] The loss function used for model training is determined based on the quadruplet samples;

[0010] The loss function is used to optimize the feature extraction network to obtain a converged feature extraction network.

[0011] The feature extraction network is used to detect the image to be detected.

[0012] Secondly, a deepfake detection device is provided, comprising:

[0013] The data preprocessing module is used to acquire training data;

[0014] The quadruple construction module is used to construct quadruple samples using the training data. The quadruple samples include: anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples; wherein the semantic similarity between the augmented positive samples, the hard negative samples, the irrelevant negative samples, and the anchor samples satisfies a preset ordinal relationship.

[0015] The first processing module is used to determine the loss function for model training based on the quadruple samples;

[0016] The second processing module is used to optimize the feature extraction network using the loss function to obtain a converged feature extraction network.

[0017] The detection module is used to detect the image to be detected using the feature extraction network.

[0018] Thirdly, an electronic device is provided, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the deepfake detection method described above.

[0019] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of the deepfake detection method described above.

[0020] The beneficial effects of this application are:

[0021] In the embodiments of this application, quadruple samples are constructed based on training data. The semantic similarity between the enhanced positive samples, hard negative samples, irrelevant negative samples, and anchor samples in these quadruple samples satisfies a preset ordinal relationship, thus transforming forgery detection into a refined list sorting problem. The feature extraction network is iteratively optimized through a loss function, enabling the feature extraction network to learn a structured feature space that can measure "authenticity," thereby achieving accurate and robust detection of deepfakes. Attached Figure Description

[0022] Figure 1 This is one of the flowcharts illustrating the deepfake detection method of this application;

[0023] Figure 2 This is the second flowchart illustrating the deepfake detection method of this application;

[0024] Figure 3 This is a schematic diagram of the system architecture of this application;

[0025] Figure 4 This is a schematic diagram of the deepfake detection device of this application;

[0026] Figure 5 This is a schematic diagram of the deepfake detection device of this application;

[0027] Figure 6 This is a schematic diagram of the electronic device of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specified order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, a first object can be one or more. Furthermore, in the specification and claims, "and" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] It is worth noting that the technologies described in this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), and other systems. The terms "system" and "network" in this application are often used interchangeably, and the described technologies can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies. However, the following description describes New Radio (NR) systems for illustrative purposes, and NR terminology is used in most of the following description. These technologies can also be applied to applications beyond NR systems, such as 6th Generation (6G) communication systems.

[0031] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0032] In the various embodiments of this application, it should be understood that the sequence number of each process described below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0033] In addition, the terms "system" and "network" are often used interchangeably in this article.

[0034] In the embodiments provided in this application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A, but can also be determined based on A and / or other information.

[0035] like Figure 1 As shown in the figure, this application provides a deepfake detection method, including:

[0036] Step 101: Obtain training data;

[0037] Step 102: Construct quadruple samples using the training data. The quadruple samples include: anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples; wherein the semantic similarity between the augmented positive samples, the hard negative samples, the irrelevant negative samples, and the anchor samples satisfies a preset ordinal relationship.

[0038] Step 103: Determine the loss function for model training based on the quadruplet samples;

[0039] Step 104: Optimize the feature extraction network using the loss function to obtain a converged feature extraction network;

[0040] Step 105: Use the feature extraction network to detect the image to be detected.

[0041] In this embodiment, the training data is obtained from raw data acquired from publicly available, industry-recognized deepfake video datasets and then processed. Before model training, one or more quadruples of samples with clear ordinal relationships are constructed using the training data. Anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples are denoted as (A, B, C, D) respectively. Each element in this quadruple sample has a specific semantic meaning. Anchor sample A is the benchmark, representing an "absolutely true" semantic anchor. Augmented positive sample B is a sample generated after augmenting anchor sample A. Hard negative sample C is an image sample with the same identity as anchor sample A but which has undergone deepfake processing. Irrelevant negative sample D is a sample completely unrelated to the identity and content of anchor sample A.

[0042] After constructing the quadruplet samples, a loss function can be determined based on these samples for model training. The feature extraction network is then trained using this loss function, and convergence is achieved through multiple iterations. The converged feature extraction network is then used to perform deepfake detection on images.

[0043] In the embodiments of this application, quadruple samples are constructed based on training data. The semantic similarity between the enhanced positive samples, hard negative samples, irrelevant negative samples, and anchor samples in these quadruple samples satisfies a preset ordinal relationship, thus transforming forgery detection into a refined list sorting problem. The feature extraction network is iteratively optimized through a loss function, enabling the feature extraction network to learn a structured feature space that can measure "authenticity," thereby achieving accurate and robust detection of deepfakes.

[0044] In some embodiments, obtaining training data includes:

[0045] Collect raw video data, including both fake and real video data; convert the raw video data into image frames and extract facial images; crop and normalize the facial images to obtain training data.

[0046] In this embodiment, before entering the core training process, a series of preparation and preprocessing operations are required on the raw data to obtain standardized input suitable for model training. This process is the foundation for ensuring the effective implementation of subsequent technical solutions. Specifically, data acquisition is performed first, obtaining raw data from publicly available, industry-recognized deepfake video datasets. For example, face recognition datasets (such as FaceForensics++, Celeb-DF, DFFD, etc.) can be used. These datasets contain a large number of fake videos generated by different forgery techniques and their corresponding real videos, covering various scenes and personal identities, providing a solid data foundation for model training and validation. Then, frame extraction and face detection are performed on the acquired video data. Since this embodiment is based on image detection, the video data needs to be converted into image frames. Optionally, video processing tools (such as the FFmpeg library) can be used to decode the video stream into independent image frame sequences. Subsequently, face detection is performed on each frame to accurately locate facial regions. This step can use traditional face detection networks, such as Multi-Task Cascaded Convolutional Networks (MTCNN). MTCNN is a classic and efficient face detection algorithm. It uses three cascaded sub-networks (P-Net, R-Net, O-Net) to progressively refine candidate box selection and bounding box regression, enabling it to accurately locate faces in complex backgrounds.

[0047] The extracted face images undergo image cropping and normalization. Based on the facial region bounding boxes output from the face detection step, the images are cropped. To preserve some contextual information (such as hairstyle and neck) while cropping the complete face, the detected bounding boxes are typically enlarged by a fixed ratio, for example, by a factor of 1.3 in both length and width, before cropping. Finally, to meet the input size requirements of deep learning models, all cropped face images are uniformly scaled to a fixed size, such as 256×256 pixels or 384×384 pixels, and pixel values ​​are normalized, for example, scaled to the [0, 1] or [-1, 1] range. After these steps, we obtain a standardized image dataset that can be directly used for subsequent quadruple construction.

[0048] In some embodiments, constructing quadruplet samples using the training data includes:

[0049] Select one original real face image from the training data as the anchor sample;

[0050] Data augmentation is performed on the anchor point samples to generate augmented positive samples;

[0051] Among all the forged images corresponding to the anchor sample, the forged image with the closest feature distance to the anchor sample is selected as the hard-to-bear sample;

[0052] Select a real image from the training data that is different from the anchor sample in both identity and content as an irrelevant negative sample.

[0053] In this embodiment, at each iteration of model training, one or more quadruplet samples (A, B, C, D) with explicit ordinal relationships need to be dynamically constructed. Each element in the quadruplet sample has a specific semantic meaning, collectively forming an ordinal chain from "most similar" to "least similar". Anchor sample A is a randomly selected original real face image from the preprocessed dataset. It serves as a benchmark, representing an "absolutely real" semantic anchor.

[0054] Augmented Positive B is a sample B generated by applying a set of preset data augmentation operations to anchor sample A in real time. These augmentation operations are not arbitrary but are specifically designed to simulate common, non-malicious image quality degradation or content perturbation in the real world. Optionally, the data augmentation operations may include one or more of the following: 1) Random Joint Photographic Experts Group (JPEG) compression, with the compression quality factor randomly selected in the range of [50, 95] to simulate image quality loss during network transmission; 2) Gaussian blur, with the blur kernel size randomly selected between 3x3 and 7x7 to simulate out-of-focus or low-resolution scenes; 3) Color jitter, brightness, and contrast adjustment, with adjustment factors randomly selected in the range of [0.8, 1.2] to simulate changes caused by different lighting and shooting equipment. Augmented Positive B maintains consistency with anchor sample A in identity and content but introduces benign perturbations. Semantically, it should be highly similar to anchor sample A but slightly distorted.

[0055] The hard negative sample C is a deepfake image that shares the same identity as the anchor sample A. To improve the model's discriminative ability, the hard negative sample C is not randomly selected, but dynamically chosen through an online hard negative mining strategy. Optionally, before training begins, a pool of candidate fake images for all identities in the dataset is pre-generated using various mainstream deepfake algorithms (such as FaceSwap, DeepFaceLab, Face2Face, etc.). In each training iteration, once the anchor sample A is selected, the entire pool of candidate fake images for the identity corresponding to anchor sample A is input into the current state's feature extraction network, and the distance (e.g., Euclidean distance or cosine distance) between each fake image in the pool and anchor sample A is calculated in the feature space. Then, the fake image with the closest feature distance to anchor sample A is selected as the hard negative sample C. This strategy ensures that the model focuses on learning the most convincing "hard boundary" samples at each stage of training, thus forcing the model to learn more refined and discriminative fake traces.

[0056] An irrelevant negative sample D is a real image randomly selected from the preprocessed dataset, whose identity is completely different from that of the anchor sample A. Irrelevant negative sample D is unrelated to the anchor sample A in both content and identity; therefore, semantically, it should be the furthest from the anchor sample A.

[0057] Through the above construction process, a quadruple sample (A, B, C, D) can be obtained. In this quadruple sample, the semantic similarity between the enhanced positive sample, the hard negative sample, the irrelevant negative sample, and the anchor sample satisfies a preset ordinal relationship.

[0058] In some embodiments, the preset ordinal relationship includes: a first similarity between the enhanced positive sample and the anchor sample is greater than a second similarity between the difficult negative sample and the anchor sample; the second similarity is greater than a third similarity between the irrelevant negative sample and the anchor sample.

[0059] For example, the first similarity between the enhanced positive sample and the anchor sample is denoted as Sim(A, B), the second similarity between the hard negative sample and the anchor sample is denoted as Sim(A, C), and the third similarity between the irrelevant negative sample and the anchor sample is denoted as Sim(A, D). The semantic similarity of the anchor sample A itself can be denoted as Sim(A, A). Then, the preset ordinal relationship between the similarity between each element in the quadruple sample and the anchor sample A is: Sim(A, A) > Sim(A, B) > Sim(A, C) > Sim(A, D).

[0060] In some embodiments, determining the loss function for model training based on the quadruplet samples includes:

[0061] Based on the feature extraction network, feature extraction is performed on the quadruple sample to obtain the feature vector corresponding to each element;

[0062] Based on the distance between the feature vectors, a chain-order boundary loss function is determined; the chain-order boundary loss function and the classification cross-entropy loss function are weighted and summed to obtain the loss function.

[0063] In this embodiment, a feature extraction network is used to extract features from the four samples in the quadruple sample. The feature extraction network used can be a deep neural network structure with shared weights, which can be understood as a Siamese network containing four branches. This means that samples A, B, C, and D all pass through the same network for forward propagation, obtaining their respective feature vectors f(A), f(B), f(C), and f(D). Weight sharing is crucial because it ensures that all samples are mapped to the same unified metric space, making distance comparisons between them meaningful.

[0064] The feature extraction network in this application comprises a backbone network and a projection head. The backbone network is responsible for extracting deep visual features from the input image. Considering that deepfake detection tasks need to pay attention to both local high-frequency artifacts and understand global semantic inconsistencies, powerful convolutional neural networks (such as XceptionNet, which performs well in benchmarks such as FaceForensics++) or vision transformers (ViT) can be used. The latter, due to its self-attention mechanism, can better capture long-range dependencies. This application is not limited to a specific backbone network.

[0065] The projection head, connected to one or more fully connected layers above the backbone network, maps the high-dimensional features extracted by the backbone network into a lower-dimensional embedding space. For example, a 2048-dimensional feature vector can be projected to 128 or 256 dimensions. Therefore, the feature extraction network of this application ultimately outputs 128-dimensional or 256-dimensional feature vectors, such as f(A), f(B), f(C), and f(D). Ordinal constraints can be imposed in this embedding space.

[0066] After obtaining the feature vectors of each element in the quadruple, the distances between the feature vectors can be determined. For example, there is a first distance d(f(A), f(B)) between the feature vector of the enhanced positive sample and the feature vector of the anchor sample, a second distance d(f(A), f(C)) between the feature vector of the hard negative sample and the feature vector of the anchor sample, and a third distance d(f(A), f(D)) between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample. Based on the distances between these feature vectors, a chain-like ordinal boundary loss function is constructed. To enforce a preset ordinal distance relationship in the feature space, this application designs a novel chain-like ordinal boundary loss function. The chain-like ordinal boundary loss function consists of two parts, a first part loss and a second part loss, which are used to constrain adjacent pairs in the ordinal chain.

[0067] In some embodiments, the first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample is smaller than the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample; the second distance is smaller than the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample.

[0068] For example: the first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample is denoted as d(f(A), f(B)), the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample is denoted as d(f(A), f(C)), and the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample is denoted as d(f(A), f(D)). The distance between the feature vector of the anchor sample A itself can be denoted as d(f(A), f(A)). Then the distance between the feature vector of each element in the quadruple sample and the feature vector of the anchor sample A satisfies: d(f(A), f(A)) < d(f(A), f(B)) < d(f(A), f(C)) < d(f(A), f(D)), where f(·) represents the feature extraction vector and d(·, ·) represents the distance metric function.

[0069] Optionally, determining the chain-like ordinal boundary loss function based on the distance between the feature vectors includes:

[0070] A first part of the loss is determined based on a first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample, and a second distance between the feature vector of the hard-to-negative sample and the feature vector of the anchor sample; a second part of the loss is determined based on a second distance between the feature vector of the hard-to-negative sample and the feature vector of the anchor sample, and a third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample; the chain-like ordinal boundary loss function is the sum of the first part of the loss and the second part of the loss.

[0071] In this embodiment, the chained ordinal boundary loss function is composed of a first part loss and a second part loss concatenated. The first part loss constrains the relationship between the first distance d(f(A), f(B)) and the second distance d(f(A), f(C)), ensuring that the distance between the hard negative sample C and the anchor sample A is significantly greater than the distance between the enhanced positive sample B and the anchor sample A. This first part loss is expressed as:

[0072] ;

[0073] Where d(⋅, ⋅) can be the L2 norm (squared Euclidean distance). This is the first pre-defined margin. The physical meaning of this loss term is: only when the distance between the hard negative sample C and the anchor sample A is greater than the distance between the enhanced positive sample B and the anchor sample A by at least [percentage missing] Only when the time is right will the loss be zero. Otherwise, a positive penalty term will be generated, driving the model parameter update to increase d(f(A), f(C)) or decrease d(f(A), f(B)). The value can be set to, for example, 0.5, which controls the width of the discrimination boundary between "benign perturbation" and "malicious forgery".

[0074] The second part of the loss is used to constrain the relationship between the second distance d(f(A), f(C)) and the third distance d(f(A), f(D)), ensuring that the distance between the irrelevant negative sample D and the anchor sample A is significantly greater than the distance between the hard negative sample C and the anchor sample A. This second part of the loss is expressed as:

[0075] ;

[0076] in, This is the second pre-defined boundary. This loss term ensures that the model not only distinguishes between fake and real data, but also has the ability to recognize identities, pushing samples with different identities further apart. The value can be set to, for example, 1.0, and is usually greater than 1.0. This is to reflect a greater range of semantic differences.

[0077] The chain-like ordinal boundary loss function is the sum of the first part loss and the second part loss. Therefore, the resulting chain-like ordinal boundary loss function is:

[0078] .

[0079] To further enhance the stability of training and the model's classification ability, a chain-order boundary loss function is combined with a standard classification cross-entropy loss. The weighted combination is used to form the final total loss function. :

[0080] ;

[0081] in, This can be achieved by adding a classifier (e.g., a simple linear layer) after the projection head, which performs a binary classification of the input samples as true / false. This is a weighting coefficient used to balance the contributions of the two losses. Its value can range from [0.1, 1.0] and can be adjusted through experimental verification. This hybrid loss design allows the model to learn the structured metric space while retaining a clear classification objective.

[0082] In some embodiments, optimizing the feature extraction network using the loss function to obtain a converged feature extraction network includes:

[0083] Based on the loss function, the feature extraction network is optimized using a gradient descent optimizer until the first condition is met, thus obtaining a converged feature extraction network.

[0084] The first condition includes: the loss obtained using the feature extraction network no longer decreases over N consecutive cycles; or, the preset evaluation index of the feature extraction network remains stable within a preset range.

[0085] In this embodiment, the model optimization process employs the standard backpropagation algorithm. Based on the calculated total loss function... The parameters of the shared-weight feature extraction network are updated using a gradient descent optimizer (such as Adam or SGD with momentum). The steps of quadruple construction, feature extraction, loss calculation, and parameter update are repeated until the model's performance on the validation set converges. The criteria for model convergence can be: the loss on the validation set no longer decreases for several consecutive (e.g., N) epochs, or the model's key evaluation metrics (such as the area under the curve, AUC) become stable. "Becoming stable" can be understood as the value of the evaluation metric (such as AUC) falling within a preset range.

[0086] In some embodiments, the detection of the image to be detected using the feature extraction network includes:

[0087] The image to be detected is input into the feature extraction network to obtain the feature vector of the image to be detected;

[0088] The authenticity of the image to be detected is determined based on the feature vector.

[0089] Optionally, determining the authenticity of the image to be detected based on the feature vector includes:

[0090] The feature vector is input into the classifier of the feature extraction network to obtain the authenticity judgment result of the image to be detected;

[0091] or,

[0092] The authenticity of the image to be detected is determined based on the distance between the feature vector and the true category center; the true category center is calculated based on the feature vector of the image frame corresponding to the real video data in the training data.

[0093] In this embodiment, after the model training is complete, for an input image to be detected, it is input into the trained feature extraction network (backbone network + projector) to obtain the feature vector of the image. Then, the authenticity of the image can be determined by one of the following two methods:

[0094] Method 1, Classifier-based approach: If a mixture loss was used during training, i.e., the loss function includes the classification cross-entropy loss function. Then, the probability of it being a "forgery" can be directly output by the classifier at the end of the network. For example, the classifier can output a probability value between 0 and 1.

[0095] Method 2, based on distance metrics: A "true class center" can be pre-calculated, for example, by averaging the feature vectors of a large number of true samples in the dataset. Then, the distance between the feature vector of the image to be detected and this true class center is calculated. A threshold can be set; if the distance is less than the threshold, it is judged as true; otherwise, it is judged as fake. This method makes better use of the metric space learned during model training.

[0096] In the embodiments of this application, forgery detection is modeled as a more refined listwise ranking problem. Different image samples, even those belonging to the broad categories of "real" or "forged," have different semantic distances from "absolutely real," exhibiting a natural ordinal relationship. For example, the similarity between an original real image, a compressed real image, a high-quality forged image, and an image completely unrelated to the content and the original real image decreases progressively. This application constructs a quadruple sample that reflects this semantic hierarchy and designs a novel chain-like ordinal boundary loss function to guide the deep neural network to learn a structured feature space that can measure "realism," thereby achieving accurate and robust detection of deep forgeries.

[0097] The following illustration, with reference to the accompanying drawings, provides an example of the deepfake detection method of this application.

[0098] The execution process of the deepfake detection method is as follows: Figure 2 As shown, the system architecture for executing this process is as follows: Figure 3 As shown, it includes a data processing module, a quadruple construction module, a feature extraction module, and a training module.

[0099] The deepfake detection method includes:

[0100] Step 1: Data preparation and preprocessing: The data processing module processes the input raw video images to obtain training data.

[0101] Step 2, Ordinal Quadruple Construction Process: The quadruple construction module constructs quadruple samples based on the obtained training data.

[0102] Step 2 includes: Step 201, selecting an original real face image from the training data as anchor sample A, and selecting a real image from the training data that is different from the anchor sample in both identity and content as an irrelevant negative sample D.

[0103] Step 202: Perform data augmentation on the anchor point sample to generate augmented positive sample B;

[0104] Step 203: Among all the forged images corresponding to the anchor point sample, select the forged image with the closest feature distance to the anchor point sample as the hard negative sample C.

[0105] Specifically, before training begins, various mainstream deepfake algorithms (such as FaceSwap, DeepFaceLab, Face2Face, etc.) are used to pre-generate a corresponding pool of candidate fake images for all identities in the dataset. In each training iteration, once anchor sample A is selected, the entire pool of candidate fake images for the identity corresponding to anchor sample A is input into the current state's feature extraction network. The distance (e.g., Euclidean distance or cosine distance) between each fake image in the pool and anchor sample A in the feature space is calculated. Then, the fake image with the closest feature distance to anchor sample A is selected as the hard negative sample C.

[0106] Step 3: The feature extraction module extracts features from the quadruple samples based on the feature extraction network to obtain the feature vectors corresponding to each element. The feature extraction network is a Siamese network with shared weights.

[0107] Step 4: The loss calculation unit of the training module performs loss calculation. Based on the distance between the feature vectors corresponding to the quadruplet samples, a chain-order boundary loss function is determined, which includes a first part loss and a second part loss. The chain-order boundary loss function and the classification cross-entropy loss function are weighted and summed to obtain the final loss function.

[0108] Step 5: The parameter optimization unit of the training module uses the loss function to optimize the feature extraction network model and determines whether the feature extraction network model has converged. If it has converged, the process ends and the final feature extraction network is obtained. If it has not converged, the above steps of constructing the quadruplet, feature extraction, loss calculation and parameter update are repeated until the model performance converges.

[0109] After obtaining a converged feature extraction network, it can be used to detect the image to be detected. Specifically, the image to be detected is input into the trained feature extraction network (backbone network + projector) to obtain the feature vector of the image, and then the feature vector is used to further determine the authenticity of the image. The method for determining the authenticity of an image can be based on a classifier or a distance metric, which will not be elaborated here.

[0110] In this embodiment, by constructing an ordinal quadruple containing anchor sample A, enhanced positive sample B, hard negative sample C, and irrelevant negative sample D, and whose semantic similarity to the anchor sample follows a preset ordinal relationship, the forgery detection problem is transformed into a list sorting problem. A chain-like ordinal boundary loss function containing at least two coupled loss components, which can constrain the relative distance relationship of d(f(A), f(B)) < d(f(A), f(C)) and d(f(A), f(C)) < d(f(A), f(D)) in the feature space respectively, is used to optimize the detection model, thereby constructing a structured feature space that can measure "authenticity" and achieving accurate detection of deepfakes.

[0111] The proposed solution has stronger applicability and generalization capabilities, does not rely on any specific type of forgery traces or metadata, and is free from dependence on specific forgery information such as fusion masks. By constructing a semantic sample chain, this application is applicable to various forgery techniques, including face swapping, facial replay, attribute editing, and even Artificial Intelligence Generated Content (AIGC), thus having a wider range of applications.

[0112] Furthermore, this application elevates the detection problem to a listwise ranking problem. By optimizing a complete ordinal chain A > B > C > D, this application provides the model with richer and more comprehensive supervision signals than pairwise comparisons, which has been proven in the field of "learning to rank" to lead to a more stable and efficient training process.

[0113] This application explicitly trains the model to learn invariance to common image distortions (such as compression and blurring) by introducing enhanced positive samples B. The model is taught that "a compressed real image (B) is still closer to the original real image (A) than a fake image (C)", which gives this method a natural robustness advantage when dealing with low-quality images spread through social media.

[0114] This application selects a hard-negative sample C through an online hard-negative sample mining strategy and places it in a key position in the ordinal chain (between B and D), forcing the model to learn a fine boundary that distinguishes between "benign perturbations" and "malicious forgeries." By applying ordinal relationships to the hard samples more precisely, a more discriminative feature space can be learned.

[0115] This application constructs a unified framework capable of simultaneously modeling "invariance to benign perturbations" and "discrimination against malicious forgeries," thereby significantly improving the model's generalization ability and robustness in real-world complex scenarios. With the proliferation of deepfake technology, the financial, security, social media, and news sectors urgently require efficient and reliable detection technologies. The method proposed in this application has significant advantages in generalization and robustness, and can be applied to key scenarios such as identity authentication (facial recognition), content moderation, and tracing the source of fake news, possessing broad market application prospects and significant commercial value.

[0116] like Figure 4 As shown in the figure, this application embodiment also provides a deepfake detection device 400, including:

[0117] Data preprocessing module 410 is used to acquire training data;

[0118] The quadruple construction module 420 is used to construct quadruple samples using the training data. The quadruple samples include: anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples; wherein the semantic similarity between the augmented positive samples, the hard negative samples, the irrelevant negative samples, and the anchor samples satisfies a preset ordinal relationship.

[0119] The first processing module 430 is used to determine the loss function for model training based on the quadruplet samples;

[0120] The second processing module 440 is used to optimize the feature extraction network using the loss function to obtain a converged feature extraction network.

[0121] The detection module 450 is used to detect the image to be detected using the feature extraction network.

[0122] In some embodiments, the data preprocessing module is specifically used for:

[0123] Collect raw video data, which includes both fake video data and real video data;

[0124] The raw video data is converted into image frames, and facial images are extracted.

[0125] The face images are cropped and normalized to obtain training data.

[0126] In some embodiments, the quadruple-building module is specifically used for:

[0127] Select one original real face image from the training data as the anchor sample;

[0128] Data augmentation is performed on the anchor point samples to generate augmented positive samples;

[0129] Among all the forged images corresponding to the anchor sample, the forged image with the closest feature distance to the anchor sample is selected as the hard-to-bear sample;

[0130] Select a real image from the training data that is different from the anchor sample in both identity and content as an irrelevant negative sample.

[0131] In some embodiments, the preset ordinal relationship includes:

[0132] The first similarity between the enhanced positive sample and the anchor sample is greater than the second similarity between the hard negative sample and the anchor sample; the second similarity is greater than the third similarity between the irrelevant negative sample and the anchor sample.

[0133] In some embodiments, the first processing module includes:

[0134] The feature extraction module is used to extract features from the quadruple samples based on the feature extraction network to obtain the feature vectors corresponding to each element.

[0135] The loss calculation unit is used to determine the chain-order boundary loss function based on the distance between the feature vectors; and to obtain the loss function by weighted summation of the chain-order boundary loss function and the classification cross-entropy loss function.

[0136] In some embodiments, determining the chain-order boundary loss function based on the distance between the feature vectors includes:

[0137] The first part of the loss is determined based on the first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample, and the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample.

[0138] The second part of the loss is determined based on the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample, and the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample.

[0139] The chain-order boundary loss function is the sum of the first part loss and the second part loss.

[0140] In some embodiments, the first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample is smaller than the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample; the second distance is smaller than the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample.

[0141] In some embodiments, the second processing module includes:

[0142] The parameter optimization unit is used to optimize the feature extraction network using a gradient descent optimizer according to the loss function until the first condition is met, thereby obtaining a converged feature extraction network.

[0143] The first condition includes: the loss obtained using the feature extraction network no longer decreases over N consecutive cycles; or, the preset evaluation index of the feature extraction network remains stable within a preset range.

[0144] In some embodiments, the detection module is specifically used for:

[0145] The image to be detected is input into the feature extraction network to obtain the feature vector of the image to be detected;

[0146] The authenticity of the image to be detected is determined based on the feature vector.

[0147] In some embodiments, determining the authenticity of the image to be detected based on the feature vector includes:

[0148] The feature vector is input into the classifier of the feature extraction network to obtain the authenticity judgment result of the image to be detected;

[0149] or,

[0150] The authenticity of the image to be detected is determined based on the distance between the feature vector and the true category center; the true category center is calculated based on the feature vector of the image frame corresponding to the real video data in the training data.

[0151] It should be noted that the apparatus provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0152] like Figure 5 As shown, this application embodiment provides a deepfake detection device 500, including a processor 510 and a transceiver 520, wherein,

[0153] The processor 510 is configured to: acquire training data; construct quadruple samples using the training data, the quadruple samples including: anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples; wherein the semantic similarity between the augmented positive samples, the hard negative samples, the irrelevant negative samples, and the anchor samples satisfies a preset ordinal relationship; determine a loss function for model training based on the quadruple samples; optimize the feature extraction network using the loss function to obtain a converged feature extraction network; and use the feature extraction network to detect the image to be detected.

[0154] In some embodiments, the processor acquires training data, including:

[0155] Collect raw video data, which includes both fake video data and real video data;

[0156] The raw video data is converted into image frames, and facial images are extracted.

[0157] The face images are cropped and normalized to obtain training data.

[0158] In some embodiments, the processor constructs quadruplet samples using the training data, including:

[0159] Select one original real face image from the training data as the anchor sample;

[0160] Data augmentation is performed on the anchor point samples to generate augmented positive samples;

[0161] Among all the forged images corresponding to the anchor sample, the forged image with the closest feature distance to the anchor sample is selected as the hard-to-bear sample;

[0162] Select a real image from the training data that is different from the anchor sample in both identity and content as an irrelevant negative sample.

[0163] In some embodiments, the preset ordinal relationship includes:

[0164] The first similarity between the enhanced positive sample and the anchor sample is greater than the second similarity between the hard negative sample and the anchor sample; the second similarity is greater than the third similarity between the irrelevant negative sample and the anchor sample.

[0165] In some embodiments, the processor determines a loss function for model training based on the quadruplet samples, including:

[0166] Based on the feature extraction network, feature extraction is performed on the quadruple sample to obtain the feature vector corresponding to each element;

[0167] The chain-order boundary loss function is determined based on the distance between the feature vectors.

[0168] The loss function is obtained by weighted summation of the chain-like ordinal boundary loss function and the classification cross-entropy loss function.

[0169] In some embodiments, the processor determines a chain-order boundary loss function based on the distance between the feature vectors, including:

[0170] The first part of the loss is determined based on the first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample, and the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample.

[0171] The second part of the loss is determined based on the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample, and the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample.

[0172] The chain-order boundary loss function is the sum of the first part loss and the second part loss.

[0173] In some embodiments, the first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample is smaller than the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample; the second distance is smaller than the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample.

[0174] In some embodiments, the processor uses the loss function to optimize the feature extraction network to obtain a converged feature extraction network, including:

[0175] Based on the loss function, the feature extraction network is optimized using a gradient descent optimizer until the first condition is met, thus obtaining a converged feature extraction network.

[0176] The first condition includes: the loss obtained using the feature extraction network no longer decreases over N consecutive cycles; or, the preset evaluation index of the feature extraction network remains stable within a preset range.

[0177] In some embodiments, the processor utilizes the feature extraction network to detect the image to be detected, including:

[0178] The image to be detected is input into the feature extraction network to obtain the feature vector of the image to be detected;

[0179] The authenticity of the image to be detected is determined based on the feature vector.

[0180] In some embodiments, the processor determines the authenticity of the image to be detected based on the feature vector, including:

[0181] The feature vector is input into the classifier of the feature extraction network to obtain the authenticity judgment result of the image to be detected;

[0182] or,

[0183] The authenticity of the image to be detected is determined based on the distance between the feature vector and the true category center; the true category center is calculated based on the feature vector of the image frame corresponding to the real video data in the training data.

[0184] It should be noted that the deepfake detection device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0185] Another embodiment of the electronic device of this application, such as Figure 6 As shown, it includes a transceiver 610, a processor 600, a memory 620, and a program or instructions stored in the memory 620 and executable on the processor 600; when the processor 600 executes the program or instructions, it implements the above-mentioned deepfake detection method.

[0186] The transceiver 610 is used to receive and send data under the control of the processor 600.

[0187] Among them, Figure 6 In this context, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 600) and memory (memory 620). The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 610 may be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over a transmission medium. The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 during operation.

[0188] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in the deepfake detection method described above and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0189] The processor is the processor in the communication device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0190] This application also provides a computer program product that stores a program or instructions, including computer instructions. When these computer instructions are executed by a processor, they implement the steps in the deepfake detection method described above and achieve the same technical effect. To avoid repetition, further details are omitted here.

[0191] In this embodiment, the module can be implemented in software so that it can be executed by various types of processors. For example, an identified executable code module may include one or more physical or logical blocks of computer instructions, which may be constructed as objects, procedures, or functions. Nevertheless, the executable code of the identified module does not need to be physically located together, but may include different instructions stored in different locations, which, when logically combined, constitute the module and achieve the module's intended purpose.

[0192] In practice, an executable code module can be a single instruction or many instructions, and can even be distributed across multiple different code segments, different programs, and across multiple memory devices. Similarly, operational data can be identified within the module and can be implemented in any suitable form and organized within any suitable data structure. This operational data can be collected as a single dataset or distributed across different locations (including different storage devices), and can exist, at least in part, solely as electronic signals within the system or network.

[0193] When a module can be implemented using software, considering the current level of hardware technology, modules that can be implemented in software can be implemented using hardware circuits by those skilled in the art to achieve the corresponding functions, without considering cost. These hardware circuits include conventional very-large-scale integrated circuits (VLSI) or gate arrays, as well as existing semiconductors such as logic chips and transistors, or other discrete components. Modules can also be implemented using programmable hardware devices, such as field-programmable gate arrays, programmable array logic, and programmable logic devices.

[0194] The exemplary embodiments described above are with reference to the accompanying drawings. Many different forms and embodiments are feasible without departing from the spirit and teachings of this application. Therefore, this application should not be construed as limiting the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided to make this application complete and convey the scope of this application to those skilled in the art. In these drawings, component dimensions and relative dimensions may be exaggerated for clarity. The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. As used herein, unless clearly indicated otherwise, the singular forms “a,” “an,” and “the” are intended to include all such forms. It will be further understood that the terms “comprising” and / or “including”, when used in this specification, indicate the presence of the stated features, integers, steps, operations, components, and / or elements, but do not exclude the presence or addition of one or more other features, integers, steps, operations, components, and / or groups thereof. Unless otherwise indicated, when stated, a range of values ​​includes the upper and lower limits of the range and any subranges in between.

[0195] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A deepfake detection method, characterized in that, include: Obtain training data; The training data is used to construct quadruple samples, which include: anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples; wherein the semantic similarity between the augmented positive samples, the hard negative samples, the irrelevant negative samples and the anchor samples satisfies a preset ordinal relationship. The loss function used for model training is determined based on the quadruplet samples; The loss function is used to optimize the feature extraction network to obtain a converged feature extraction network. The feature extraction network is used to detect the image to be detected.

2. The method according to claim 1, characterized in that, The acquisition of training data includes: Collect raw video data, which includes both fake video data and real video data; The raw video data is converted into image frames, and facial images are extracted. The face images are cropped and normalized to obtain training data.

3. The method according to claim 1, characterized in that, The construction of quadruplet samples using the training data includes: Select one original real face image from the training data as the anchor sample; Data augmentation is performed on the anchor point samples to generate augmented positive samples; Among all the forged images corresponding to the anchor sample, the forged image with the closest feature distance to the anchor sample is selected as the hard-to-bear sample; Select a real image from the training data that is different from the anchor sample in both identity and content as an irrelevant negative sample.

4. The method according to claim 1, characterized in that, The preset ordinal relationship includes: The first similarity between the enhanced positive sample and the anchor sample is greater than the second similarity between the hard negative sample and the anchor sample; the second similarity is greater than the third similarity between the irrelevant negative sample and the anchor sample.

5. The method according to claim 1, characterized in that, The step of determining the loss function for model training based on the quadruplet samples includes: Based on the feature extraction network, feature extraction is performed on the quadruple sample to obtain the feature vector corresponding to each element; The chain-order boundary loss function is determined based on the distance between the feature vectors. The loss function is obtained by weighted summation of the chain-like ordinal boundary loss function and the classification cross-entropy loss function.

6. The method according to claim 5, characterized in that, The step of determining the chain-like ordinal boundary loss function based on the distance between the feature vectors includes: The first part of the loss is determined based on the first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample, and the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample. The second part of the loss is determined based on the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample, and the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample. The chain-order boundary loss function is the sum of the first part loss and the second part loss.

7. The method according to claim 5 or 6, characterized in that, The first distance between the feature vector of the enhanced positive sample and the feature vector of the anchor sample is smaller than the second distance between the feature vector of the hard negative sample and the feature vector of the anchor sample; the second distance is smaller than the third distance between the feature vector of the irrelevant negative sample and the feature vector of the anchor sample.

8. The method according to claim 1, characterized in that, The step of optimizing the feature extraction network using the loss function to obtain a converged feature extraction network includes: Based on the loss function, the feature extraction network is optimized using a gradient descent optimizer until the first condition is met, thus obtaining a converged feature extraction network. The first condition includes: the loss obtained using the feature extraction network no longer decreases over N consecutive cycles; or, the preset evaluation index of the feature extraction network remains stable within a preset range.

9. The method according to claim 1, characterized in that, The process of using the feature extraction network to detect the image to be detected includes: The image to be detected is input into the feature extraction network to obtain the feature vector of the image to be detected; The authenticity of the image to be detected is determined based on the feature vector.

10. The method according to claim 9, characterized in that, The step of determining the authenticity of the image to be detected based on the feature vector includes: The feature vector is input into the classifier of the feature extraction network to obtain the authenticity judgment result of the image to be detected; or, The authenticity of the image to be detected is determined based on the distance between the feature vector and the true category center; the true category center is calculated based on the feature vector of the image frame corresponding to the real video data in the training data.

11. A deepfake detection device, characterized in that, include: The data preprocessing module is used to acquire training data; The quadruple construction module is used to construct quadruple samples using the training data. The quadruple samples include: anchor samples, augmented positive samples, hard negative samples, and irrelevant negative samples; wherein the semantic similarity between the augmented positive samples, the hard negative samples, the irrelevant negative samples, and the anchor samples satisfies a preset ordinal relationship. The first processing module is used to determine the loss function for model training based on the quadruple samples; The second processing module is used to optimize the feature extraction network using the loss function to obtain a converged feature extraction network. The detection module is used to detect the image to be detected using the feature extraction network.

12. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 10.

Images