A face forgery detection method, system, device and medium

Abstract

The present application relates to the technical field of face detection, and discloses a face forgery detection method, system, device and medium. The method comprises: according to initial visual features of a to-be-detected face image, a plurality of real face mode prompts and a plurality of fake face mode prompts, attributing the initial visual features to each real face mode and each fake face mode to update each mode prompt to obtain corresponding fine-grained mode prompts; according to the fine-grained mode prompts, determining probabilities that the to-be-detected face image belongs to each real face mode and each fake face mode, and correspondingly weighting and fusing the mode prompts updated by the plurality of real face modes and the plurality of fake face modes; according to the weighted and fused mode prompts, integrating global context information of the initial visual features to obtain coarse-grained integrated visual features of the to-be-detected face image; and combining the coarse-grained integrated visual features of the to-be-detected face image and all the fine-grained mode prompts to detect whether the to-be-detected face image is a fake face.

Description

Technical Field

[0001] This invention relates to the field of face detection technology, and in particular to a method, system, device and medium for detecting face forgery. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, especially the widespread application of techniques such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Diffusion Models (DMs), the realism and production barriers of deepfake facial images have reached unprecedented levels. This has created an urgent need to maintain the authenticity of information and protect personal privacy and social security.

[0003] Currently, face forgery detection technology has made some progress. The task is typically simplified to a direct binary classification problem of face authenticity, where the trained model directly distinguishes between real and fake input images. However, while samples generated by different forgery methods may appear visually similar, their underlying patterns differ significantly. Traditional binary classification methods struggle to effectively guide the model in uncovering discriminative, universal features, leading to overfitting to specific forgery patterns in the training set. This results in severely insufficient generalization ability when faced with unknown forgery methods or different face domains. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, device and medium for detecting face forgery, which can solve the problem of insufficient generalization ability of detection methods.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a method for detecting face forgery, comprising the following steps: Acquire the face image of the subject and extract the initial visual features of the face image; Based on the initial visual features and the pattern cues of multiple real face patterns and multiple fake face patterns, the initial visual features are attributed to each real face pattern and each fake face pattern to update each pattern cue and obtain the corresponding fine-grained pattern cues; wherein, each pattern cue is a face feature generated by a real face or a fake face. Based on the fine-grained pattern hints, the probability of the face image under test belonging to each real face pattern and each fake face pattern is determined. The fine-grained pattern hints of multiple real face patterns and multiple fake face patterns are then weighted and fused to obtain the real coarse-grained hints and fake coarse-grained hints of the face image under test. The latent integrator integrates the global contextual information of the initial visual features based on real and fake coarse-grained cues to obtain the coarse-grained integrated visual features of the face image under test. The latent integrator is trained by using several real faces and several fake faces, and constrained to minimize the feature distance between the output features of the latent integrator and the corresponding real faces, and maximize the feature distance between the output features of the latent integrator and the corresponding fake faces. By combining coarse-grained integrated visual features of the face image under test with all fine-grained pattern cues, it is possible to detect whether the face image under test is a fake face.

[0006] Further, the step of attributing the initial visual features to each real face pattern and each fake face pattern based on the initial visual features and the preset pattern cues of multiple real face patterns and multiple fake face patterns, in order to update each pattern cue to obtain the corresponding fine-grained pattern cues, includes: Based on the cross-attention mechanism, each real face pattern and each fake face pattern is guided to search for features related to the corresponding pattern cues from the initial visual features, so as to update each pattern cues and obtain the corresponding fine-grained pattern cues.

[0007] Further, the step of determining the probability of the test face image belonging to each real face pattern and each fake face pattern based on fine-grained pattern cues, and correspondingly weighting and fusing the fine-grained pattern cues of multiple real face patterns and multiple fake face patterns to obtain real coarse-grained cues and fake coarse-grained cues for the test face image, includes: Each fine-grained pattern cue is input into the corresponding linear classifier to determine the probability that the face image under test belongs to each real face pattern and each fake face pattern. The probability of a test face image belonging to each real face pattern and each fake face pattern is mapped to a weight representing confidence by an activation function. Based on the weight of each face image to be tested belonging to the corresponding real face pattern, the fine-grained pattern prompts of multiple real face patterns are weighted and fused to obtain the real coarse-grained prompts. Based on the weight of each test face image belonging to the corresponding fake face mode, the fine-grained mode prompts of multiple fake face modes are weighted and fused to obtain the fake coarse-grained prompt.

[0008] Furthermore, the potential integrator employs a visual Transformer architecture; The process involves using a latent integrator to integrate global contextual information of initial visual features based on genuine and fabricated coarse-grained cues, resulting in coarse-grained integrated visual features of the face image under test. This includes: The visual Transformer integrates global contextual information representing real face patterns from the initial visual features based on real coarse-grained cues, and integrates global contextual information representing fake face patterns from the initial visual features based on fake coarse-grained cues, to obtain coarse-grained integrated visual features.

[0009] Furthermore, the test face image is input into a trained progressive chain learning model to detect whether the test face image is a fake face; the loss function used during the training of the progressive chain learning model includes: fine-grained loss. Consistency loss and differentiated integration loss .

[0010] Furthermore, the fine-grained loss The consistency loss is the mean of the asymmetric loss of the test face image with respect to each real face pattern and each fake face pattern. The differential ensemble loss is the mean difference between the feature vectors of the face image under each mode cue and the feature vector centers under all mode cues. This constraint is used to minimize the feature distance between the output features of the potential integrator and the corresponding real face, and maximize the feature distance between the output features and the corresponding fake face.

[0011] Furthermore, the pattern cues for the multiple real face patterns are obtained by: using a pre-trained face recognition model to extract face features on a real face dataset, and dividing it into multiple clusters using the K-means clustering algorithm, with each cluster representing a pattern cue for a real face pattern. The method for obtaining the pattern hints of the multiple fake face patterns is as follows: different fake faces are generated through multiple different fake algorithms, and the facial features of each fake face represent a pattern hint of a fake face pattern.

[0012] Embodiments of the present invention also provide a face forgery detection system, comprising the following modules: The feature extraction module is used to acquire the face image to be tested and extract the initial visual features of the face image to be tested; The prompt processing module is used to attribute the initial visual features to each real face pattern and each fake face pattern based on the initial visual features and the preset pattern prompts of multiple real face patterns and multiple fake face patterns, so as to update each pattern prompt to obtain the corresponding fine-grained pattern prompt; wherein, each pattern prompt is a facial feature generated by real face or fake face. And based on the fine-grained pattern cues, determine the probability that the face image under test belongs to each real face pattern and each fake face pattern, so as to weight and fuse the fine-grained pattern cues of multiple real face patterns and multiple fake face patterns to obtain the real coarse-grained cues and fake coarse-grained cues of the face image under test. The feature integration module is used to integrate the global contextual information of the initial visual features based on real coarse-grained cues and fake coarse-grained cues through a latent integrator to obtain the coarse-grained integrated visual features of the face image to be tested. The latent integrator is trained by using several real faces and several fake faces, and constraining the output features of the latent integrator to minimize the feature distance with the corresponding real face and maximize the feature distance with the corresponding fake face. The face detection module combines coarse-grained integrated visual features of the face image under test with all fine-grained pattern cues to detect whether the face image under test is a fake face.

[0013] Embodiments of the present invention also provide a computer device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described face forgery detection method.

[0014] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described face forgery detection method.

[0015] The face forgery detection method provided by this invention has at least the following beneficial effects: First, pattern cues for multiple real face patterns and multiple pattern cues for multiple fake face patterns are introduced. The existing binary classification problem of face authenticity is decomposed into multiple fine-grained pattern attribution tasks to determine the probability of the test face image belonging to each real face pattern and each fake face pattern. Based on this, a weighted fusion is performed to obtain the real concept and fake concept generated based on the test face image (real coarse-grained cues and fake coarse-grained cues). Therefore, the real coarse-grained cues and fake coarse-grained cues can be used as cognitive instructions to guide the feature integration of the corresponding real and fake patterns in the test face image features, thereby mining deep features used to distinguish whether the test face image is real or fake, and achieving high-precision face forgery detection.

[0016] Among them, the true coarse-grained prompts and fake coarse-grained prompts used to guide the generation of the final face detection features are formed based on the probability that the face image to be tested belongs to each mode. Therefore, the true coarse-grained prompts and fake coarse-grained prompts generated by different face images to be tested are different. Thus, the basis for image detection is a dynamically generated exclusive evaluation benchmark based on which type of real or fake image the image to be detected is more like, which is applicable to the detection of any fake method or face domain and has stronger generalization ability.

[0017] Furthermore, due to the constraint during training that "the output features of the latent integrator minimize the feature distance between the real face and the corresponding real face, and maximize the feature distance between the real face and the corresponding fake face," the features of the real face and the fake face can be clearly separated in the feature space, which is more conducive to subsequent face image detection. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of a face forgery detection method provided by the present invention; Figure 2 This is a schematic diagram of a system architecture for face forgery detection provided by the present invention; Figure 3 A visual schematic diagram illustrating the effect of differentiated integration loss provided by the present invention; Figure 4 This is a schematic diagram illustrating feature interaction and final classification provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0020] Currently, face forgery detection is simplified into a direct binary classification problem of real and fake faces. This "end-to-end" training approach has fundamental flaws: First, although samples generated by different forgery methods may appear visually similar, their underlying patterns differ significantly, requiring the model to simultaneously identify real samples and multiple distinct forged samples, making the task excessively difficult. Second, the constraints provided by the binary classification loss function are too broad, making it difficult to effectively guide the model to uncover discriminative general features. The direct consequence is that the model is prone to overfitting to specific forgery patterns in the training set, resulting in severely insufficient generalization ability when facing unknown forgery methods or different face domains, leading to a chaotic feature distribution in the latent space. How to improve the model's generalization ability from a single visual modality remains a core challenge that urgently needs to be addressed.

[0021] This invention provides a face forgery detection method that can effectively overcome the problems of overfitting and poor generalization caused by direct binary classification in existing technologies, thus maintaining strong detection capabilities when facing emerging and unknown forgery technologies.

[0022] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] One embodiment of the present invention relates to a face forgery detection method. The implementation details of the face forgery detection method of this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0024] The specific process of the face forgery detection method in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Obtain the face image to be tested and extract the initial visual features of the face image to be tested; Step 102: Based on the initial visual features and the preset pattern prompts of multiple real face patterns and multiple fake face patterns, the initial visual features are attributed to each real face pattern and each fake face pattern to update each pattern prompt to obtain the corresponding fine-grained pattern prompt; wherein, each pattern prompt is a face feature generated by a real face or a fake face. Step 103: Determine the probability of the face image under test belonging to each real face pattern and each fake face pattern based on the fine-grained pattern hints, so as to weight and fuse the fine-grained pattern hints of multiple real face patterns and multiple fake face patterns to obtain the real coarse-grained hints and fake coarse-grained hints of the face image under test. Step 104: The latent integrator integrates the global contextual information of the initial visual features based on the real coarse-grained cues and the fake coarse-grained cues to obtain the coarse-grained integrated visual features of the face image to be tested; wherein, the latent integrator is trained by using several real faces and several fake faces, constraining the output features of the latent integrator to minimize the feature distance with the corresponding real face and maximize the feature distance with the corresponding fake face. Step 105: Combine the coarse-grained integrated visual features of the face image to be tested with all fine-grained pattern cues to detect whether the face image to be tested is a fake face.

[0025] The implementation details of the face forgery detection method in this embodiment are described below. The following content is only for the convenience of understanding and is not necessary for implementing this solution.

[0026] In step 101, the input face image (i.e. the face image to be tested) is processed by an image encoder to extract features and obtain high-dimensional visual features, which are used as initial visual features.

[0027] In steps 102 and 103, firstly, based on the cross-attention mechanism, each real face pattern and each fake face pattern are guided to search for features related to the corresponding pattern cues from the initial visual features to update each pattern cues and obtain the corresponding fine-grained pattern cues.

[0028] Then, each fine-grained pattern cue is input into the corresponding linear classifier to determine the probability that the test face image belongs to each real face pattern and each fake face pattern. An activation function then maps the probability of the test face image belonging to each real face pattern and each fake face pattern to weights representing confidence. Based on the weight of each test face image belonging to the corresponding real face pattern, the fine-grained pattern cuees of multiple real face patterns are weighted and fused to obtain a real coarse-grained cue. Similarly, based on the weight of each test face image belonging to the corresponding fake face pattern, the fine-grained pattern cuees of multiple fake face patterns are weighted and fused to obtain a fake coarse-grained cue.

[0029] Specifically, multiple learnable pattern cues are introduced and further guided feature extraction is achieved through a cross-attention mechanism. The high-dimensional visual features and pattern cues are fused and input into a linear classifier to obtain prediction scores for each pattern attribution task. Based on the confidence weights of the prediction scores, the corresponding pattern cues are weighted and summed to generate two coarse-grained cues representing "real" and "fake" respectively. The pattern cues are used to decompose the face real / fake binary classification task into multiple fine-grained pattern attribution tasks, driving the model to attribute the input image to multiple predefined real or fake face patterns and generate fine-grained features for each pattern.

[0030] The multiple fine-grained pattern attribution task specifically introduces multiple (8) learnable pattern cues for multi-modal fine-grained attribution of face images. The 8 patterns include 4 real face patterns obtained from a real face dataset using K-means clustering and 4 fake face patterns generated based on different forgery algorithms. The predicted scores of each pattern attribution task are mapped to weights representing attribution confidence using an activation function. The cues corresponding to real face patterns are grouped into one group, and the cues corresponding to fake face patterns are grouped into another. The pattern cues within each group are then weighted and summed to obtain two coarse-grained cues representing "real" and "fake" attributions.

[0031] In step 104, the global context information corresponding to the real face pattern in the initial visual features is integrated by the latent integrator, and the global context information corresponding to the fake face pattern in the initial visual features is integrated according to the coarse-grained cues to obtain coarse-grained integrated visual features.

[0032] Specifically, the high-dimensional visual features output by the image encoder and the coarse-grained cues obtained by weighted summation (including two types of coarse-grained cues representing "real" and "fake") are input together into a latent integrator. Under the guidance of the coarse-grained cues, the latent integrator performs cross-modal differential feature interaction on the high-dimensional visual features. At the same time, it generates coarse-grained integrated visual features by explicitly constraining the feature distance between fake samples and real samples and reducing the feature distance within fake samples and within real samples.

[0033] The latent integrator employs a neural network architecture with cross-modal information integration capabilities, preferably a Vision Transformer (ViT) architecture. Guided by coarse-grained cues, this architecture performs cross-modal analysis and integration of the high-dimensional visual features output by the image encoder in step one, outputting coarse-grained integrated visual features.

[0034] In step 105, fine-grained features and coarse-grained integrated visual features are fused to obtain fused features of fine-grained and coarse-grained features, and the final face real / fake binary classification is performed based on the fused features, which constitutes the entire process of progressive chain learning.

[0035] The progressive chain learning model is trained by a composite loss function that includes fine-grained loss, consistency loss, differential ensemble loss, and binary classification loss.

[0036] Fine-grained loss is used. (Fine-grained Loss) is the mean of the asymmetric losses of the test face image with respect to each real face pattern and each fake face pattern. It is used to address the imbalance of positive and negative samples in each pattern subtask, and represents a fine-grained loss. The mathematical expression is as follows: ; In the formula, Let be the probability that the face image to be tested belongs to the i-th face pattern, i.e., the probability that the i-th face pattern belongs to ... The predicted scores of a linear classifier indicate whether the face pattern is a real face pattern or a fake face pattern. The label is 0 or 1, where 0 indicates that the feature comes from a real face and 1 indicates that the feature comes from a fake face. The sum of the number of real face patterns and fake face patterns; This is an asymmetric loss function used to calculate the loss for each subtask, and its mathematical expression is as follows: ; in, Focusing parameters for real face images; To forge the focusing parameters of a face image; This refers to the probability offset hyperparameter; The features are derived from real human face images; This indicates that the features are derived from a fake face image.

[0037] Consistency loss The Consistency Loss model is the mean difference between the feature vectors of the test face image under each modal cue and the feature vector centers under all modal cuees. This ensures that the modal cuees corresponding to different forged samples from the same real face tend to converge in the feature space, thereby constructing the correlation between different modalities and promoting the learning of common features. (Consistency Loss) The mathematical expression is as follows: ; ; in, For the input face image at the 1st Each pattern prompt is a feature vector obtained after processing by an image encoder; The mean of the features suggested for all patterns; It is an L1 norm.

[0038] Differentiated feature interaction, specifically referring to the use of differentiated ensemble loss. The Differentiated Integration Loss constrains the latent integrator, thereby minimizing the distance between the high-dimensional visual features of the real samples and the coarse-grained integrated visual features (i.e., the output features of the latent integrator), and maximizing the distance between the high-dimensional visual features of the fake samples and the coarse-grained integrated visual features. The mathematical expression is as follows: ; in, The high-dimensional visual features output by the image encoder. Coarse-grained integrated visual features for the output of the potential integrator; It is an L2 norm; 2 represents the boundary of the fake face sample.

[0039] Overall loss function For binary classification loss Fine-grained loss Consistency loss and differentiated integration loss The weighted sum, the overall loss function The mathematical expression is as follows: ; in, , , These are the weighting coefficients used to balance the various losses, and the weighting coefficients... The value of is much greater than and This emphasizes the dominant role of differentiated ensemble loss in model training.

[0040] In some embodiments, the face forgery detection method of the present invention uses... Figure 2 The system architecture shown includes coarse-grained and fine-grained modules; together they constitute a general face forgery detection method based on chain-based recognition prompt learning.

[0041] The implementation process of the face forgery detection method is explained below based on this system architecture.

[0042] 1. System initialization and data preparation: 1.1 Image encoder selection: This embodiment uses a pre-trained CLIP / ViT-B model as the image encoder. This model is pre-trained on large-scale image-text pair data, possesses powerful general visual feature extraction capabilities, and can capture rich cross-modal relationships. The input image size is uniformly adjusted to... After vector embedding processing by the image encoder, 196 high-dimensional visual features with a dimension of 768 are obtained. And a category tag.

[0043] 1.2 Definition and initialization of mode prompts: In this embodiment, the face forgery detection task is decomposed into eight fine-grained pattern factorization tasks. These eight patterns include: Four real face patterns: On a subset of real faces in the FaceForensics++ (FF++) dataset, facial features were extracted using a pre-trained face recognition model and then divided into four clusters using the K-means clustering algorithm. Each cluster represents a real face pattern.

[0044] Four fake face modes: directly corresponding to the four mainstream fake face algorithms provided in the FF++ dataset, namely Deepfake (DF), FaceSwap (FS), Face2Face (FF) and NeuralTexture (NT).

[0045] Pattern prompt It is randomly initialized to a learnable parameter matrix. Each row A cue vector representing a specific pattern (real or fake).

[0046] 2. Step 1: Fine-grained pattern attribution: 2.1 Feature Extraction and Pattern Attribution: The category labels and high-dimensional visual features output by the image encoder With pattern hints The sequences are concatenated along the sequence dimension to form a new input sequence, which is then fed into the image encoder.

[0047] The image encoder internally uses a cross-attention mechanism to provide pattern cues. It aggregates information related to its own pattern from all tile markers. After processing by the encoder, it outputs an updated pattern hint. .

[0048] 2.2 Coarse-grained prompt generation: Hints for the updated 8 modes Perform a weighted summation. Specifically, the weighted sum of the first four (real face mode) is used as the coarse-grained "real" cue. The weighted sum of the last four (forged face mode) is used as the coarse-grained "forged" hint. Finally, the two coarse-grained hints are combined to obtain the coarse-grained hint. .

[0049] 2.3 Fine-grained constraints: The updated 8 modes will be prompted. Eight independent linear classifiers are input. Each classifier is responsible for predicting the probability that the input image belongs to its corresponding pattern. Since the positive and negative samples are extremely imbalanced in each pattern subtask, this embodiment uses asymmetric loss (ASL) to calculate the fine-grained loss. Fine-grained loss The mathematical expression is as follows: ; in, For the first The prediction scores of a linear classifier; The label is either 0 or 1, where 0 indicates the feature comes from a real face image; 1 indicates the feature comes from a fake face image. This is an asymmetric loss function used to calculate the loss for each subtask, and its mathematical expression is as follows: ; in, Focusing parameters for real face images; To forge the focusing parameters of a face image; 1 represents the probability offset hyperparameter; A function to find the maximum value; The features are derived from real human face images; This indicates that the features are derived from a fake face image.

[0050] 2.4 Consistency Constraints: To establish correlations between different patterns, especially by using real samples as a bridge, this embodiment introduces a consistency loss. This loss only applies to real face samples. For a real sample, the mean of its 8 pattern cues is calculated. Then minimize each pattern hint and L1 distance between Consistency loss The mathematical expression is as follows: ; ; in, For the input face image at the 1st Each pattern prompt is a feature vector obtained after processing by an image encoder; The mean of the features suggested for all patterns; It uses the L1 norm. This loss forces the model to aggregate the feature representations of the same real face under different modal cues, thereby learning more general features.

[0051] 3. Step Two: Coarse-grained Feature Integration: 3.1 Potential Integrator Processing: The high-dimensional visual features output by the image encoder in step one Coarse-grained hints obtained by weighted summation The latent integrator is input together. This embodiment preferably uses the VisionTransformer (ViT) architecture as the latent integrator. The ViT architecture consists of multiple stacked Transformer encoder layers. Each encoder layer includes a multi-head self-attention mechanism and a feedforward neural network, enabling global modeling of dependencies between any two locations in an image and effectively capturing subtle spatial inconsistencies and global structural anomalies present in face forgery. Inside ViT, coarse-grained cues are deeply fused with high-dimensional visual features through a cross-attention layer, guiding the model to adaptively weight and integrate facial features of different patterns with the support of global contextual information, ultimately outputting coarse-grained integrated visual features. .

[0052] 3.2 Differentiated Integration Constraints (see...) Figure 3 , Figure 4 ): To explicitly separate real and fake samples in the latent space, this embodiment designs a differential ensemble loss. To constrain potential integrators, differentiated integration loss The mathematical expression is as follows: ; in, The high-dimensional visual features output by the image encoder in step one. The coarse-grained integrated visual features output by the potential integrator in step two; The features are derived from real human face images; The features indicate that the image is derived from a fake face image; It is an L2 norm; 2 represents the boundary of the fake sample, which is set in this embodiment. ; The function is used to find the maximum value. This loss function forces the high-dimensional visual features of the real samples. Integrating visual features with coarse grain The samples should be as close as possible; conversely, the distance between the forged samples should be maximized, at least greater than [a certain value]. .

[0053] 4. Step 3: Feature Interaction and Final Classification: 4.1 Feature Interaction: Extract the final pattern hint from the fine-grained module. As a fine-grained feature From the coarse-grained module, an attention pooling layer is used to... Aggregates into a global coarse-grained feature. Then, and The features are concatenated along the feature dimension to obtain the final fused features. .

[0054] 4.2 Final Classification and Total Loss: Fusion features The model is trained by a final linear classifier to perform binary classification of face images as real / fake, yielding the final prediction result. The model is trained using an overall loss function. The constraints are applied, and the loss function is a binary classification loss. Fine-grained loss Consistency loss and differentiated integration loss The weighted sum, the overall loss function The mathematical expression is as follows: ; in, , , In this embodiment, weighting coefficients are set to balance the various losses. , , And weighting coefficients The value of is much greater than and This emphasizes the dominant role of differentiated ensemble loss in model training.

[0055] 5. Model Training and Inference: 5.1 Training Configuration: This embodiment selected seven public datasets for experiments, including FaceForensics++ (FF++), Celeb-DF-V2 (CDF), DF40, DeepFakeDetection (DFD), DeepFake Detection Challenge (DFDC), DeepFake Detection Challenge Preview (DFDCP), Multi-dimensional FacialForgery Image (MFFI), and WildDeepFake (WDF).

[0056] This embodiment uses the open-source library Dlib to detect faces in all videos. For all the publicly available datasets mentioned above, this embodiment extracts 8 frames as training data and 32 frames as test data from each video.

[0057] This embodiment uses a pre-trained CLIP / ViT-B as the backbone network. Therefore, in this embodiment, the size of all training and testing images mentioned above is uniformly adjusted to [size missing]. .

[0058] This embodiment uses PyTorch to implement the model framework described above, and uses the Adan optimizer to update the model framework over 10 training epochs. This embodiment uses a single NVIDIA GeForce RTX 4090 GPU to train the model framework, where the learning rate, weight decay, and batch size are set to [specific values ​​to be filled in]. , and .

[0059] 5.2 Reasoning Process: During the inference phase, an image of a face to be detected is input and sequentially undergoes three steps: fine-grained pattern attribution, coarse-grained feature integration, and feature interaction and final classification. The final output is the probability that the image is a "forgery". The entire process is end-to-end, requiring no additional preprocessing or post-processing.

[0060] 6. Implementation Results: The method described in this embodiment significantly outperforms existing technologies in tests across multiple datasets and forgery methods. In terms of average AUC, this method (86.25%) is nearly 10 percentage points higher than the current state-of-the-art method UCF (76.35%), demonstrating its superior generalization ability. Visualization results (see...) Figure 3 This also shows that the method can effectively locate subtle forgery traces and clearly separate the feature distributions of real and forged samples in the latent space.

[0061] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0062] Another embodiment of the present invention relates to a face forgery detection system. The implementation details of this face forgery detection system are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. The face forgery detection system of this embodiment includes: The feature extraction module is used to acquire the face image to be tested and extract the initial visual features of the face image to be tested; The prompt processing module is used to attribute the initial visual features to each real face pattern and each fake face pattern based on the initial visual features and the preset pattern prompts of multiple real face patterns and multiple fake face patterns, so as to update each pattern prompt to obtain the corresponding fine-grained pattern prompt; wherein, each pattern prompt is a facial feature generated by real face or fake face. And based on the fine-grained pattern cues, determine the probability that the face image under test belongs to each real face pattern and each fake face pattern, so as to weight and fuse the fine-grained pattern cues of multiple real face patterns and multiple fake face patterns to obtain the real coarse-grained cues and fake coarse-grained cues of the face image under test. The feature integration module is used to integrate the global contextual information of the initial visual features based on real coarse-grained cues and fake coarse-grained cues through a latent integrator to obtain the coarse-grained integrated visual features of the face image to be tested. The latent integrator is trained by using several real faces and several fake faces, and constraining the output features of the latent integrator to minimize the feature distance with the corresponding real face and maximize the feature distance with the corresponding fake face. The face detection module combines coarse-grained integrated visual features of the face image under test with all fine-grained pattern cues to detect whether the face image under test is a fake face.

[0063] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0064] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0065] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the face forgery detection methods described in the above embodiments.

[0066] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0067] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0068] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0069] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0070] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A face forgery detection method, characterized in that, The method includes: Acquire the face image of the subject and extract the initial visual features of the face image; Based on the initial visual features and the pattern cues of multiple real face patterns and multiple fake face patterns, the initial visual features are attributed to each real face pattern and each fake face pattern to update each pattern cue and obtain the corresponding fine-grained pattern cues; wherein, each pattern cue is a face feature generated by a real face or a fake face. Based on the fine-grained pattern hints, the probability of the face image under test belonging to each real face pattern and each fake face pattern is determined. The fine-grained pattern hints of multiple real face patterns and multiple fake face patterns are then weighted and fused to obtain the real coarse-grained hints and fake coarse-grained hints of the face image under test. The latent integrator integrates the global contextual information of the initial visual features based on real and fake coarse-grained cues to obtain the coarse-grained integrated visual features of the face image under test. The latent integrator is trained by using several real faces and several fake faces, and constrained to minimize the feature distance between the output features of the latent integrator and the corresponding real faces, and maximize the feature distance between the output features of the latent integrator and the corresponding fake faces. By combining coarse-grained integrated visual features of the face image under test with all fine-grained pattern cues, it is possible to detect whether the face image under test is a fake face. The process involves inputting the test face image into a trained progressive chain learning model to detect whether the test face image is a fake face. The loss function used during the training of the progressive chain learning model includes: fine-grained loss. Consistency loss and differentiated integration loss ; Fine-grained loss The mean of the asymmetric loss of the test face image with respect to each real face pattern and each fake face pattern is the consistency loss. The differential ensemble loss is the mean difference between the feature vector centers of the face image under each modal cue and the feature vector centers under all modal cues. This constraint is used to minimize the feature distance between the output features of the potential integrator and the corresponding real face, and maximize the feature distance between the output features and the corresponding fake face.

2. The face forgery detection method according to claim 1, characterized in that, The step of attributing the initial visual features to each real face pattern and each fake face pattern based on initial visual features and preset pattern cues for multiple real face patterns and multiple fake face patterns, in order to update each pattern cue to obtain the corresponding fine-grained pattern cues, includes: Based on the cross-attention mechanism, each real face pattern and each fake face pattern is guided to search for features related to the corresponding pattern cues from the initial visual features, so as to update each pattern cues and obtain the corresponding fine-grained pattern cues.

3. The face forgery detection method according to claim 2, characterized in that, The step of determining the probability of the test face image belonging to each real face pattern and each fake face pattern based on fine-grained pattern cues, and correspondingly weighting and fusing the fine-grained pattern cues of multiple real face patterns and multiple fake face patterns to obtain real coarse-grained cues and fake coarse-grained cues for the test face image, includes: Each fine-grained pattern cue is input into the corresponding linear classifier to determine the probability that the face image under test belongs to each real face pattern and each fake face pattern. The probability of a test face image belonging to each real face pattern and each fake face pattern is mapped to a weight representing confidence by an activation function. Based on the weight of each face image to be tested belonging to the corresponding real face pattern, the fine-grained pattern prompts of multiple real face patterns are weighted and fused to obtain the real coarse-grained prompts. Based on the weight of each test face image belonging to the corresponding fake face mode, the fine-grained mode prompts of multiple fake face modes are weighted and fused to obtain the fake coarse-grained prompt.

4. The face forgery detection method according to claim 1, characterized in that, The potential integrator employs a visual Transformer architecture; The process involves using a latent integrator to integrate global contextual information of initial visual features based on genuine and fabricated coarse-grained cues, resulting in coarse-grained integrated visual features of the face image under test. This includes: The visual Transformer integrates global contextual information representing real face patterns from the initial visual features based on real coarse-grained cues, and integrates global contextual information representing fake face patterns from the initial visual features based on fake coarse-grained cues, to obtain coarse-grained integrated visual features.

5. The face forgery detection method according to claim 1, characterized in that, The pattern cues for the multiple real face patterns are obtained by: using a pre-trained face recognition model to extract face features on a real face dataset, and dividing it into multiple clusters using the K-means clustering algorithm, with each cluster representing a pattern cue for a real face pattern. The method for obtaining the pattern hints of the multiple fake face patterns is as follows: different fake faces are generated through multiple different fake algorithms, and the facial features of each fake face represent a pattern hint of a fake face pattern.

6. A face forgery detection system, characterized in that, The system includes: The feature extraction module is used to acquire the face image to be tested and extract the initial visual features of the face image to be tested; The prompt processing module is used to attribute the initial visual features to each real face pattern and each fake face pattern based on the initial visual features and the preset pattern prompts of multiple real face patterns and multiple fake face patterns, so as to update each pattern prompt to obtain the corresponding fine-grained pattern prompt; wherein, each pattern prompt is a facial feature generated by real face or fake face. And based on the fine-grained pattern cues, determine the probability that the face image under test belongs to each real face pattern and each fake face pattern, so as to weight and fuse the fine-grained pattern cues of multiple real face patterns and multiple fake face patterns to obtain the real coarse-grained cues and fake coarse-grained cues of the face image under test. The feature integration module is used to integrate the global contextual information of the initial visual features based on real coarse-grained cues and fake coarse-grained cues through a latent integrator to obtain the coarse-grained integrated visual features of the face image to be tested. The latent integrator is trained by using several real faces and several fake faces, and constraining the output features of the latent integrator to minimize the feature distance with the corresponding real face and maximize the feature distance with the corresponding fake face. The face detection module is used to combine coarse-grained integrated visual features of the face image under test with all fine-grained pattern cues to detect whether the face image under test is a fake face; The process involves inputting the test face image into a trained progressive chain learning model to detect whether the test face image is a fake face. The loss function used during the training of the progressive chain learning model includes: fine-grained loss. Consistency loss and differentiated integration loss ; Fine-grained loss The mean of the asymmetric loss of the test face image with respect to each real face pattern and each fake face pattern is the consistency loss. The differential ensemble loss is the mean difference between the feature vector centers of the face image under each modal cue and the feature vector centers under all modal cues. This constraint is used to minimize the feature distance between the output features of the potential integrator and the corresponding real face, and maximize the feature distance between the output features and the corresponding fake face.

7. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the face forgery detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the face forgery detection method as described in any one of claims 1 to 5.

Images