Face spoofing detection system based on uncertain modeling

Abstract

The present application belongs to the technical field of computer, and particularly to a face forgery detection system based on uncertain modeling.The system comprises a probabilistic Transformer module, an image block screening module and an uncertainty-aware single classification loss function module.The present application firstly models the dependency relationship between image blocks as Gaussian random variables, extends the Transformer model in a probabilistic manner, then introduces an image block selection module to identify areas with high uncertainty information for final classification, and finally quantifies the uncertainty of the entire image, uses the designed uncertainty-aware single classification loss function to make the model focus more on samples with high uncertainty and difficult to determine, and through only enhancing the internal compactness of real faces, improves the inter-class separability of real and false classes in the embedding space.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a face spoofing detection system. Background Technology

[0002] In recent years, internet media data in the form of images and videos has become a major channel for communication. However, recent advances in deep learning, especially deep generative models, have opened the door to the low-cost production of deceptive images and videos, posing a serious threat to the credibility of digital information. This project studies the modeling of uncertainty and explores a highly generalizable algorithm for spoofed face recognition.

[0003] The basic framework of fake face detection is to model fake detection as a binary classification problem of deterministic network, that is, firstly, the backbone network is used to extract global features, and then the features are input into a binary classifier to judge whether the fake is real or fake. The methods are divided into image-level learning methods [1-4] and video-level learning methods [5-7] according to the different detection objects. Image-level learning methods process the video into frames, design different network structures, and focus on the inconsistencies within a single frame, such as focusing on the fusion boundary [2], frequency inconsistency [8] or image patch inconsistency [9]. By judging the frames, a comprehensive decision is made on the video. Thanks to the advanced neural network backbone model, it is feasible and efficient to learn the characteristics of tampered images, and the image-level method can judge the authenticity of a single frame image, and has a wide range of applications. The video-level learning method uses recurrent neural networks to learn the temporal features of the frame sequence and makes an overall judgment on a video. The video-level learning method can learn the temporal features of the video, such as the inconsistency between frames [7], the instability of the tampered area

[10] , and other defects that often appear in fake videos. At the same time, it can also detect a small amount of tampering in the video

[11] . However, videos generated by different generative models have specific characteristics, and the feature distribution of fake content generated by different tampering methods varies greatly. Post-processing operations such as compression can also make the visual defects that the model could originally capture less clear. Models using deterministic networks cannot judge the uncertainty of prediction and are prone to overfitting on existing training sets. However, they often fail for fake videos generated by unknown generative algorithms, resulting in a serious performance degradation [12-14]. Summary of the Invention

[0004] The purpose of this invention is to provide a face forgery detection system based on uncertainty modeling with high detection efficiency and high detection accuracy.

[0005] The face forgery detection system based on uncertainty modeling proposed in this invention is based on deep learning technology. It includes: modeling the dependency relationship between image patches as Gaussian random variables and extending the Transformer model in a probabilistic manner; introducing an image patch filtering module to identify regions with high uncertainty information and ignoring global information and redundant information that are irrelevant to the detection of real and fake images; quantifying the uncertainty of the entire image and using a designed uncertainty-aware single-class loss function to make the model focus more on samples with high uncertainty and difficult to judge, and improving the inter-class separability of real and fake classes in the embedding space by only enhancing the internal compactness of real faces; specifically, it includes the following three modules: (1) probabilistic Transformer module; (2) image patch filtering module; (3) uncertainty-aware single-class loss function module.

[0006] In this invention, the probabilistic Transformer module extends the original deterministic Transformer model in a probabilistic manner

[15] , models the attention score as a Gaussian random variable, thereby capturing the random dependencies and uncertainties in the input, and thus quantifying the cognitive uncertainty of the model prediction; the specific process is as follows:

[0007] (1) Divide the image into Image blocks of pixel size;

[0008] (2) Pass the image patch through a linear mapping layer and add positional encoding to obtain the image patch feature sequence;

[0009] (3) The features of each image patch are represented by a triplet vector through three different linear mappings, which is the query vector. Key vector Sum value vector ;

[0010] (4) Use Gaussian modeling to calculate the dependency between two image patches: use two multilayer perceptrons to model the image patches. query vector and image blocks key vector Perform calculations to predict the mean of the Gaussian random variable. and variance :

[0011] , (1)

[0012] , (2)

[0013] in, This represents a multilayer perceptron. Indicates trainable weights;

[0014] (5) Use reparameterization techniques to complete forward propagation:

[0015] , (3)

[0016] in, Represents image blocks For image patches Dependence, These are values ​​sampled from a standard Gaussian distribution;

[0017] (6) Constrain the distance between the modeled attention Gaussian distribution and the standard Gaussian distribution using the Kullback-Leibler (KL) divergence:

[0018] , (4).

[0019] In this invention, the image patch filtering module, based on probabilistic attention, identifies discrimination regions with high uncertainty information through a cumulative multiplication method, removing redundant information regions from interfering with the decision-making process. Specifically, it enhances the model's focus on local regions with high uncertainty information by changing the input of the last Transformer layer, while ignoring global information and redundant information unrelated to true / false detection. The specific process is as follows:

[0020] (1) Recursively perform matrix multiplication on the probabilistic attention matrices of all layers before the last Transformer to integrate these attention weights:

[0021] , (5)

[0022] in, Indicates the first Attention score of layer It is the total number of layers in the Transformer;

[0023] (2) Choose the previous The image patch corresponding to the maximum value is selected, and the selected label is concatenated with the classification label to serve as the input sequence for the last layer.

[0024] , (6)

[0025] in, Indicates having the first The index of a high-value image patch.

[0026] In this invention, the uncertainty-aware single-class loss function module uses mutual information to quantify the model's cognitive uncertainty about the entire image; it uses a designed uncertainty-aware single-class loss function to guide the model to focus more on samples with high uncertainty and difficult to determine; and it improves the inter-class separability between real face images and fake face images in the embedding space by enhancing only the internal compactness of real faces. The specific process is as follows:

[0027] (1) Calculate mutual information to capture the cognitive uncertainty of the model for the whole image:

[0028] , (7)

[0029] , (8)

[0030] in, Represents information entropy. Indicates mutual information, Expressing expectations, Indicates a label, Indicates input, Indicates the corresponding label, and These represent deterministic model parameters and probabilistic model parameters, respectively.

[0031] (2) The uncertainty is approximated by calculating the average value of the sample through sampling:

[0032] , (9)

[0033] in, It is an attention matrix. It is the number of probability forward propagations performed on the same input;

[0034] (3) Regularize the uncertainty:

[0035] , (10)

[0036] in, This is the number of training samples;

[0037] (4) Calculate the single-class loss function for uncertain perception:

[0038] , (11)

[0039] in, It is the preset radius. It is the average value of features extracted from real faces using a pre-trained model. This represents the set of real face images in the training set. This represents the set of fake face images in the training set;

[0040] (5) Calculate the weighted KL divergence, cross-entropy loss, and uncertain perception single-classification loss to calculate the total loss for backpropagation:

[0041] , (12)

[0042] in, It is cross-entropy loss. and It is a hyperparameter used to balance the three losses;

[0043] (6) When reasoning, synthesize Results of the second sampling:

[0044] , (13).

[0045] This invention addresses the uncertainty sources in face forgery by modeling the cognitive uncertainty of model predictions and researching a face forgery detection technology with strong generalization capabilities. It utilizes a Gaussian model to model the dependencies between image patches, improving the original deterministic attention mechanism to a probabilistic attention mechanism. This, combined with an image patch selection module, strengthens the model's focus on local regions with high uncertainty information while ignoring global information irrelevant to real / fake detection. Simultaneously, by quantifying the uncertainty of the entire image, the model is guided to focus more on samples with high uncertainty and difficult judgment. By enhancing only the internal compactness of real faces, the inter-class separability between real and fake classes is improved in the embedding space.

[0046] The main innovation of this invention lies in:

[0047] (1) The face forgery detection system based on uncertainty modeling extends the original Transformer model in a probabilistic manner. Combined with the image patch filtering module and uncertainty-aware single-class loss function, it captures the cognitive uncertainty of the model and uses uncertainty to guide model training, making the model more focused on samples with high uncertainty and difficult to judge. By enhancing only the internal compactness of real faces, it improves the inter-class separability of real and fake faces in the embedding space.

[0048] (2) This paper proposes for the first time to use uncertainty learning to enhance the generalization of face forgery detection algorithms. Attached Figure Description

[0049] Figure 1 This is a structural diagram of the face forgery detection system based on uncertainty modeling of the present invention. Detailed Implementation

[0050] The present invention will be further described below through specific implementation steps.

[0051] Step 1. Video Sampling Processing: Input Video ,in and These are the height and width of the video, respectively. This is the number of channels in the video clip (usually 3). This refers to the number of frames in the video. The real label is A value of 0 indicates that the video is authentic, while a value of 1 indicates that the video has been forged or altered. The video is sampled to obtain frame images. The labels of the frames in the same video are consistent with the labels of the original video. This represents a forgery detection model whose goal is to provide a judgment result indicating whether an input frame image is genuine or fake. The objective of this invention is to utilize probabilistic methods to overcome the positional bias of forgery flaws in specific forgery methods, thereby improving the effectiveness and generalization of the detection model. It is a probabilistic model.

[0052] Step 2. Extract image features: For the input frame image Image features are extracted using the ViT-B / 16 series method, which first divides the image into segments of size [missing information]. A sequence of image blocks is obtained by taking pixel-sized image blocks, and then a sequence of image features is obtained by linear mapping and adding classification flags and position codes.

[0053] Step 3. Calculate probabilistic attention: Building on the previous step, for each label in the image feature sequence, use three different linear mappings to obtain the corresponding triplet vector representation, i.e., the query vector. Key vector Sum value vector Unlike traditional methods that use dot product operations, this invention innovatively computes dependencies between different labels using Gaussian modeling. Specifically, for each label's query vector and key vector, two multilayer perceptrons are used to predict the mean and variance in a Gaussian distribution, respectively. Then, the dependencies between labels are computed using a reparameterization technique. To constrain the modeled Gaussian distribution, this invention uses KL divergence to limit its distance from the standard Gaussian distribution. For the Transformer module that uses a probabilistic attention mechanism, this invention refers to it as a probabilistic Transformer.

[0054] Step 4. Image Patch Filtering: After the above steps, each probabilistic Transformer layer produces a probabilistic image feature sequence with the same dimension as the original input image feature sequence. To allow the model to focus more on regions with discriminative information and ignore redundant information regions, this invention modifies the input of the last probabilistic Transformer layer. Specifically, the attention matrices of all layers except the last probabilistic Transformer layer are multiplied, and the M labels with the highest dependency on the classification label (i.e., the first label in the sequence) are calculated and selected. The selected labels are then concatenated with the classification label to form the input sequence for the last layer.

[0055] Step 5. Quantifying Uncertainty: This invention uses mutual information to capture the model's cognitive uncertainty about the entire image, mainly through sampling for each input. conduct The probability is calculated forward, then calculated. The difference between the average information entropy between predictions and the average information entropy is used as an approximation of the uncertainty value of the input image.

[0056] Step 6. Calculate the loss function: In addition to the KL divergence mentioned in Step 4, this invention also uses the cross-entropy loss function commonly used for classification and the uncertainty-aware single-class loss proposed in this invention to jointly constrain the model. The model training is guided by weighting these three loss functions. The uncertainty-aware single-class loss function proposed in this invention is based on the uncertainty value corresponding to each image input obtained in Step 5. First, all input uncertainty values ​​are regularized to constrain their range. Then, the regularized values ​​are used to weight the single-class loss, aiming to make the model focus more on samples with high uncertainty and unfamiliarity, thereby enhancing the model's discrimination ability.

[0057] For method evaluation, this invention uses frame-level accuracy and the area under the ROC curve as evaluation metrics, namely ACC and AUC.

[0058] Table 1 below shows the detection results on the FaceForensics++ High Definition (HQ) and Low Definition (LQ) datasets, demonstrating the effectiveness of the method of this invention.

[0059] Table 1

[0060] .

[0061] Table 2 below shows the experimental results of cross-tampering detection experiments on four tampering methods in the FaceForensics++ low-resolution dataset. Specifically, the experiment was conducted by training the detection model on one tampering method and testing its detection performance on the other three tampering methods, with AUC as the evaluation metric. This setup effectively demonstrates the generalization ability of the detection model in cross-tampering scenarios.

[0062] Table 2

[0063] .

[0064] Table 3 below shows the experimental results of cross-dataset detection experiments conducted on the FaceForensics++ low-resolution dataset (FF++ (LQ)) and the Celeb-DF and WildDeepfake datasets. The evaluation metric is AUC. This setting effectively demonstrates the generalization ability of the detection model under cross-dataset conditions, which is also a more realistic experimental setup.

[0065] Table 3

[0066] .

[0067] References

[0068] [1]Chollet, François. "Xception: Deep learning with depthwiseseparable convolutions." Proceedings of the IEEE conference on computervision and pattern recognition. 2017.

[0069] [2]Li, Lingzhi, et al. "Face x-ray for more general face forgerydetection." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020.

[0070] [3]Masi, Iacopo, et al. "Two-branch recurrent network for isolatingdeepfakes in videos." Computer Vision–ECCV 2020: 16th European Conference,Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16. SpringerInternational Publishing, 2020.

[0071] [4]Chen, Shen, et al. "Local relation learning for face forgerydetection." Proceedings of the AAAI conference on artificial intelligence.Vol. 35. No. 2. 2021.

[0072] [5]Sabir, Ekraam, et al. "Recurrent convolutional strategies for facemanipulation detection in videos." Interfaces (GUI) 3.1 (2019): 80-87.

[0073] [6]Amerini, Irene, et al. "Deepfake video detection through opticalflow based cnn." Proceedings of the IEEE / CVF international conference oncomputer vision workshops. 2019.

[0074] [7]Y. Zheng, J. Bao, D. Chen, M. Zeng, and F. Wen, “Exploringtemporal coherence for more general video face forgery detection,” in ICCV,2021.

[0075] [8]Li, Jiaming, et al. "Frequency-aware discriminative featurelearning supervised by single-center loss for face forgery detection."Proceedings of the IEEE / CVF conference on computer vision and patternrecognition. 2021.

[0076] [9]Zhang, Baogen, et al. "Patch Diffusion: a general module for facemanipulation detection." Proceedings of the AAAI Conference on ArtificialIntelligence. Vol. 36. No. 3. 2022

[0077]

[10] Z. Gu, Y. Chen, T. Yao, S. Ding, J. Li, and L. Ma, “Delving intothe local: Dynamic inconsistency learning for deepfake video detection,” inAAAI, 2022.

[0078]

[11] Li, Xiaodan, et al. "Sharp multiple instance learning fordeepfake video detection." Proceedings of the 28th ACM internationalconference on multimedia. 2020.

[0079]

[12] Sun, Ke, et al. "Domain general face forgery detection bylearning to weight." Proceedings of the AAAI conference on artificialintelligence. Vol. 35. No. 3. 2021.

[0080]

[13] Sun, Ke, et al. "Dual contrastive learning for general faceforgery detection." Proceedings of the AAAI Conference on ArtificialIntelligence. Vol. 36. No. 2. 2022.

[0081]

[14] Luo, Yuchen, et al. "Generalizing face forgery detection withhigh-frequency features." Proceedings of the IEEE / CVF conference on computervision and pattern recognition. 2021.

[0082]

[15] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words:Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929(2020)。

Claims

1. A face forgery detection system based on uncertainty modeling, characterized in that, It is based on deep learning technology, including modeling the dependency relationship between image patches as Gaussian random variables and extending the Transformer model in a probabilistic way; introducing an image patch filtering module to identify regions with high uncertainty information and ignoring global information and redundant information that are irrelevant to real and fake detection; quantifying the uncertainty of the entire image and using a designed uncertainty-aware single-class loss function to make the model focus more on samples with high uncertainty and difficult to judge, and improving the inter-class separability of real and fake classes in the embedding space by only enhancing the internal compactness of real faces; specifically including the following three modules: (1) probabilistic Transformer module; (2) image patch filtering module; (3) uncertainty-aware single-class loss function module; The probabilistic Transformer module extends the original deterministic Transformer model in a probabilistic manner, modeling the attention score as a Gaussian random variable to capture random dependencies and uncertainties in the input, thereby quantifying the cognitive uncertainty of the model's predictions; the specific process is as follows: (1) Divide the image into Image blocks of pixel size; (2) Pass the image patch through a linear mapping layer and add positional encoding to obtain the image patch feature sequence; (3) The features of each image patch are represented by a triplet vector through three different linear mappings, which is the query vector. Key vector Sum value vector ; (4) Use Gaussian modeling to calculate the dependency between two image patches: use two multilayer perceptrons to model the image patches. query vector and image blocks key vector Perform calculations to predict the mean of the Gaussian random variable. and variance : , （1） , （2） in, This represents a multilayer perceptron. Indicates trainable weights; (5) Use reparameterization techniques to complete forward propagation: , （3） in, Represents image blocks For image patches Dependence, These are values ​​sampled from a standard Gaussian distribution; (6) Use KL divergence to constrain the distance between the modeled attention Gaussian distribution and the standard Gaussian distribution: , （4）。 2. The face forgery detection system according to claim 1, characterized in that, In the image patch filtering module, based on probabilistic attention, a multiplication method is used to identify discrimination regions with high uncertainty information, thereby removing redundant information regions from interfering with decision-making. Specifically, this is achieved by changing the input method of the last Transformer layer to enhance the model's focus on local regions with high uncertainty information, while ignoring global and redundant information irrelevant to true / false detection; the specific process is as follows: (1) Recursively perform matrix multiplication on the probabilistic attention matrices of all layers before the last Transformer to integrate these attention weights: , （5） in, Indicates the first Attention score of layer It is the total number of layers in the Transformer; (2) Choose the previous The image patch corresponding to the maximum value is selected, and the selected label is concatenated with the classification label to serve as the input sequence for the last layer. , （6） in, Indicates having the first The index of a high-value image patch.

3. The face forgery detection system according to claim 2, characterized in that, The uncertainty-aware single-class loss function module uses mutual information to quantify the model's cognitive uncertainty about the entire image; it guides the model to focus more on samples with high uncertainty and difficult to determine; and by enhancing only the internal compactness of real faces, it improves the inter-class separability between real and fake face images in the embedding space; the specific process is as follows: (1) Calculate mutual information to capture the cognitive uncertainty of the model for the whole image: , （7） , （8） in, Represents information entropy. Indicates mutual information, Expressing expectations, Indicates a label, Indicates input, Indicates the corresponding label, and These represent deterministic model parameters and probabilistic model parameters, respectively. (2) The uncertainty is approximated by calculating the average value of the sample through sampling: , （9） in, It is an attention matrix. It is the number of probability forward propagations performed on the same input; (3) Regularize the uncertainty: , （10） in, This is the number of training samples; (4) Calculate the single-class loss function for uncertain perception: in, It is the preset radius. It is the average value of features extracted from real faces using a pre-trained model. It is a collection of real face images in the training set. It is a collection of fake face images in the training set; (5) Calculate the weighted KL divergence, cross-entropy loss, and uncertain perception single-classification loss to calculate the total loss for backpropagation: ， （12） in, It is cross-entropy loss. and It is a hyperparameter used to balance the three losses. (6) When reasoning, synthesize Results of the second sampling: ， （13）。

Images