Method for detecting deepfake videos based on biosignals and spatiotemporal signals

Abstract

The application discloses a deep fake video detection method based on biological signals and space-time signals, and belongs to the field of artificial intelligence security. First, face recognition is performed on a video, the video is cropped and aligned to generate a face video, video frames and rPPG signals of a green channel are extracted from the video, the two kinds of data are respectively input into a three-dimensional residual network and a two-dimensional convolutional neural network to train a base learner, and finally, a multi-modal fusion fake judgment integrated model is constructed based on a stacking strategy, and a fake judgment result is output. The three-dimensional convolution kernel in the three-dimensional residual network is used to learn video time sequence information, a pooling layer is used to learn space information, and biological signals difficult to be forged in the video are extracted to assist in fake judgment and increase the accuracy of video fake judgment. Experiments fully show that the established fusion model can better cope with low-resolution videos, has good fake judgment effect on various forging algorithms, and has good generalization performance.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence security technology, specifically relating to a method for detecting deepfake videos based on biological signals and spatiotemporal signals. Background Technology

[0002] In recent years, deep learning technology has developed rapidly, achieving significant breakthroughs in the field of computer vision and being widely applied in areas such as image classification, object recognition, and image generation. While this powerful technology has driven progress in various fields, it has also brought numerous security risks. With the growing threat of misuse of video deepfake technology, research into visual deepfake detection technology is urgently needed.

[0003] Most existing forgery detection technologies rely on differences in color and other texture features between generated and natural images to distinguish genuine videos from fake ones. However, this is not always effective. When fake videos are transmitted multiple times over different streaming media on the network, their resolution decreases, causing artifacts to disappear, and features in the video become subject to noise interference, making them difficult to distinguish. With the increasing number of forgery algorithms and their growing sophistication, there is an urgent need to develop more generalizable and robust forgery detection algorithms. Summary of the Invention

[0004] The technical problem solved by this invention is to propose a deep fake video detection method based on biological and spatiotemporal signals that makes full use of various information in the video, can better distinguish low-resolution fake videos, and has good generalization effect on different fake detection algorithms.

[0005] Purpose of the invention: To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A method for detecting deepfake videos based on biosignals and spatiotemporal signals includes: first, performing face recognition on the video to be detected and cropping and aligning it to a video containing only faces; extracting video frames and green channel rPPG signals from the cheeks and forehead; learning the temporal and spatial signals in the video frames by constructing and training a three-dimensional residual network, and learning the differences in spatial coherence and temporal consistency of rPPG in real and fake videos by constructing and training a two-dimensional convolutional network; integrating the two models, the three-dimensional residual network and the two-dimensional convolutional network, training the weights, and constructing the final ensemble model.

[0007] Preferably, the preprocessing step of performing face recognition on the video to be detected and cropping and aligning it into a video containing only faces involves: recognizing faces in the video based on the face recognition module, aligning the faces, cropping the video, and retaining the face portion in the video.

[0008] Preferably, the process of extracting video frames from the video involves: importing, cropping, aligning, and marking video segments, reading video frames in a loop, and saving the frame images.

[0009] Preferably, the process of extracting the green channel rPPG signal from the cheek and forehead from the video is as follows:

[0010] 1) Identify and locate faces, and lock onto regions of interest, such as the cheeks, forehead, and areas around the nose where rPPG signals are relatively abundant;

[0011] 2) Calculate the mean RGB pixel value of each ROI region to obtain the initial rPPG signal C. k (t);

[0012] 3) Normalize and bandpass filter the green channel signal to remove noise and obtain the final rPPG signal, thus obtaining the time-series rPPG signals of different regions of the face.

[0013] Preferably, based on the light absorption and scattering characteristics of skin tissue, a skin pixel formula related to the rPPG signal is constructed in the optical model to obtain the initial rPPG signal. :

[0014]

[0015] Let k be the pixel value of the skin at time t in the video, which is determined by the light intensity. Skin specular reflection component Skin diffuse reflectance component and noise composition.

[0016] As a preferred method, the temporal and spatial signals in video frames are learned by constructing and training a three-dimensional residual network as follows:

[0017] The 3D residual network consists of convolutional layers, 3D residual blocks, average pooling layers, and fully connected layers, with a network depth of 18. The 3D residual blocks consist of multiple convolutional layers, batch normalization layers, skip connections, and activation function layers.

[0018] The first layer of the network uses 3D convolution to capture the relationship between temporal and spatial information and fuse the two types of information together. In subsequent layers, the network gradually extracts high-level features by stacking multiple residual blocks and obtains classification results through average pooling and fully connected layers.

[0019] As a preferred approach, the method for learning the differences in spatial coherence and temporal consistency of rPPG in real and fake videos by constructing and training a two-dimensional convolutional network is as follows:

[0020] Two-dimensional convolutional networks mainly consist of convolutional layers and fully connected layers. The convolutional layers include convolutional kernels, activation functions, batch normalization, and max pooling. The activation function used in the model is ReLU. After the extracted rPPG signal is imported into the network, the features in the signal are obtained through convolutional kernels, the computational cost is reduced through pooling layers, and further classification is performed through fully connected layers.

[0021] Preferably, the process of integrating the two models, a 3D residual network and a 2D convolutional network, training the weights, and constructing the final integrated model is as follows:

[0022] Cross-validation is performed on the 2D convolutional network and the 3D residual network to obtain preliminary classification data of the two networks for fake videos, which is then used as new input to the dataset. This data is combined with the original video labels of the dataset to generate new training and test sets. The ensemble model is then trained using the new dataset.

[0023] As a preferred approach, the ensemble model is trained to output the authentication results.

[0024] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0025] 1) This invention proposes a model for fake video authentication based on the fusion of biological signals, temporal signals and spatial signals. Compared with other models, it can learn more video information and can perform better video fake video authentication.

[0026] 2) The model of this invention integrates a two-dimensional convolutional network and a three-dimensional residual network, and proposes a multimodal integrated anti-counterfeiting model that fuses training weights.

[0027] 3) Experiments show that the integrated model proposed in this invention is less affected by resolution and has a strong generalization ability to detect various forgery algorithms. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of a deepfake video detection method based on biological and spatiotemporal signals. Detailed Implementation

[0029] The present invention will be further illustrated below with reference to specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0030] like Figure 1As shown, the deepfake video detection method based on biosignals and spatiotemporal signals of this invention first performs face recognition on the video to be detected, cropping and aligning it to a video containing only faces; extracts video frames and green channel rPPG signals from the cheeks and forehead respectively; integrates a three-dimensional residual network and a two-dimensional convolutional network, inputting these two signals into the three-dimensional residual network and the two-dimensional convolutional network respectively for base learner training, and then cross-validates the two networks to generate a new dataset to train the fusion model. Specifically, it can be divided into the following 5 steps:

[0031] Step 1: Data Preprocessing

[0032] First, the data is preprocessed to identify the face portion; then, the face recognition module identifies the face in the video, aligns the face, and crops the video, retaining only the face portion.

[0033] Step 2: Signal Extraction

[0034] Frame extraction and rPPG signal extraction are performed on the video.

[0035] 2.1 Video Frame Extraction: Import the cropped, aligned, and marked video clips, read the video frames in a loop, and save the frame images.

[0036] 2.2 rPPG signal extraction

[0037] Basic principle:

[0038] Remote photoplethysmography (rPPG) signals are a type of heart rate-related biosignal that can be obtained from video. When light shines on the skin, the camera receives light components related to the heart rate reflected back from subcutaneous blood vessels, forming a regular rPPG signal in the video. This tiny but regular biosignal is difficult to forge. rPPG signals extracted from fake videos do not conform to biological patterns and can be used for forgery detection.

[0039] Based on the absorption and scattering characteristics of light by skin tissue, the formula for skin pixels related to rPPG signals in the optical model is constructed as shown in Equation (1).

[0040] (1)

[0041] Let be the pixel value of skin k in the video at time t. It is determined by the illumination intensity. Skin specular reflection component Skin diffuse reflectance component and noise Composition. Light intensity and specular reflection on the skin are mainly related to the fixed brightness of the light source, the illumination angle, and the illumination distance. The diffuse reflection component is mainly affected by skin color and heart rate pulses. Among the three RGB channels, the green channel has the largest proportion of rPPG signal. After normalization and filtering, most of the time-invariant constants and meaningless noise can be removed, at which point the green channel signal can be approximated as an rPPG signal.

[0042] The specific steps are as follows:

[0043] 2.2.1 Identify and locate the face, and lock the region of interest of the face, namely the cheeks, forehead and the area around the nose where rPPG signals are rich.

[0044] 2.2.2 Calculate the mean RGB pixel value of each ROI region to obtain the initial rPPG signal. .

[0045] 2.2.3 The green channel signal is normalized and bandpass filtered to remove noise and obtain the final rPPG signal, thus obtaining the time-series rPPG signals of different regions of the face.

[0046] Step 3: Base Learner Training

[0047] By constructing and training a 3D residual network, the temporal and spatial signals in video frames are learned. By constructing and training a 2D convolutional network, the differences in spatial coherence and temporal consistency of rPPG in real and fake videos are learned.

[0048] 3.1 Construction and Training of 3D Residual Network

[0049] The 3D residual network consists of one convolutional layer, four 3D residual blocks, one average pooling layer, and one fully connected (FC) layer, with a network depth of 18. Each 3D residual block comprises multiple convolutional layers, a batch normalization (BN) layer, skip connections, and activation function layers. The network input is a 3D tensor containing both the temporal sequence information of the video and the spatial information of each frame. The first layer's 3D convolution captures the relationship between the temporal and spatial information and fuses them together. In subsequent layers, the network progressively extracts high-level features by stacking multiple residual blocks and obtains the classification result through average pooling and fully connected layers.

[0050] 3.2 Construction and Training of Two-Dimensional Convolutional Networks

[0051] The two-dimensional convolutional network mainly consists of three convolutional layers and two fully connected layers. The convolutional layers include convolutional kernels, activation functions, batch normalization, and max pooling. The activation function used in the model is ReLU (Rectified Linear Function). After the extracted rPPG signal is imported into the network, features are obtained through convolutional kernels, the computational cost is reduced through pooling layers, and further classification is performed through fully connected layers.

[0052] Step 4: Ensemble Model Training

[0053] An ensemble model integrates a 3D residual network and a 2D convolutional network, cross-validating the two networks to obtain preliminary classification data for fake videos. This data is then used as new input to the dataset, combined with the original video labels to generate new training and test sets. The ensemble model is then trained using this new dataset.

[0054] Step 5: Output of authentication results

[0055] Output the authentication result.

[0056] Example 1

[0057] The effectiveness and efficiency of the method of the present invention were verified through the following experiments:

[0058] Datasets: The FaceForensics++ dataset contains video types with two different resolutions and compression ratios of 23 and 40 (i.e., FF++c23 and FF++c40). Each resolution includes 1000 original YouTube videos and 5000 fake videos created using Face2Face, FaceSwap, DeepFakes, and Neural-Textures forgery methods. The Celeb-DF-v2 dataset is a deepfake dataset created using improved DeepFake techniques. It contains 590 real videos and 5639 fake videos, all with a resolution of 1024. 1024.

[0059] Forgery methods:

[0060] 1. Face2Face

[0061] Face2Face is a method that uses machine learning technology to replace facial expressions in real time. It uses deep learning algorithms to generate 3D models of facial expressions, and then maps these models onto target videos to achieve real-time facial expression replacement.

[0062] 2. FaceSwap

[0063] FaceSwap is a face-swapping algorithm based on face detection, face alignment, facial feature extraction, and image fusion technologies. It processes two images of different faces, fusing the facial features of one face with those of the other, thereby achieving face swapping.

[0064] 3. DeepFakes

[0065] DeepFakes is a method of creating synthetic videos using deep learning technology. It can fuse images of different faces to generate videos and images with high realism.

[0066] 4. FaceShifter

[0067] FaceShifter is a face transformation method based on conditional generator technology. It uses deep learning algorithms to perform specific transformations on faces in target videos, such as facial expression transformation and age adjustment.

[0068] 5. Neural-Textures

[0069] Neural-Textures is a facial synthesis method based on image synthesis technology. It uses deep learning algorithms to generate high-quality images of human faces and can predict and generate facial expressions through artificial intelligence technology.

[0070] Forgery detection model:

[0071] 1. Face X-ray

[0072] Face X-ray is a deep learning algorithm proposed in 2019 by a research team at Carnegie Mellon University, based on the paper "Toward Fairness in Visual Recognition with Uncertainty-guided Learning". Its basic principle is to use convolutional neural networks to process facial images, extract key features, and perform classification and prediction. Its advantage lies in its ability to handle uncertainty, improving the algorithm's reliability and repeatability.

[0073] 2. Xception

[0074] Xception (Extreme Inception) is a deep convolutional neural network proposed by Google in 2016. Its basic principle is to use depthwise separable convolutions to replace traditional convolution operations, thereby reducing the number of parameters and computational complexity, and improving model efficiency and accuracy. Xception originates from the paper "Xception: Deep Learning with Depthwise Separable Convolutions." This paper used the ImageNet dataset for model training and testing, and the results showed that Xception outperforms the Inception v3 model of similar size in classification tasks. Xception is a high-performing and commonly used video classification model.

[0075] 3. LipForensics

[0076] LipForensics, initially proposed by researchers at the Institute of Automation, Chinese Academy of Sciences in 2019 and published at the ACM Multimedia conference, is a deep learning-based model for detecting fake lip-reading videos. The basic principle of the LipForensics model is to use deep learning algorithms to extract and analyze features from lip-reading video frames. By extracting features such as color, texture, and shape from the lip-reading images, it distinguishes between real and fake videos, demonstrating high classification capabilities.

[0077] 4. MesoNet

[0078] MesoNet is an image spoofing detection model based on deep convolutional neural networks. It was proposed by researchers at New York University in 2018. Its main principle is to use convolutional neural networks to process images, thereby distinguishing between real and synthetic images. The model's advantages lie in its high detection accuracy and strong robustness, enabling it to effectively identify and detect various types of image spoofing attacks, including different synthetic techniques and deception methods.

[0079] 5. Steg. Features + SVM

[0080] Steg. Features + SVM is an image spoofing detection model based on Support Vector Machines. Its main principle is to extract steganalytic features from images and then use a Support Vector Machine to classify images as real or synthetic. The advantages of this model are its high detection accuracy and strong robustness, enabling effective identification and detection of various types of image spoofing attacks.

[0081] This invention uses the FaceForensics++ and Celb-DF-v2 datasets as training and testing sets, respectively. Based on the original FF++ dataset, new fake videos with compression rates of 23% and 40% from real FF++ videos were generated using the FaceShiter method and added to the dataset. This invention also selected Face X-ray, Xception, LipForensics, MesoNet, and Steg. Features + SVM forgery detection model algorithms to test the performance of the proposed model on the aforementioned test sets, comparing its performance with other forgery detection model algorithms on the datasets.

[0082] Evaluation metrics: Accuracy (Acc), False Positive Rate (FPR), and Area Under the ROC Curve (AUC) were selected as evaluation criteria for model validation.

[0083] 1. Comparative Experiment

[0084] Table 1. Performance comparison of this invention with other algorithms on the FF++ and Celeb-DF-v2 datasets.

[0085]

[0086] On videos in the FaceForensics++ dataset with compression ratios of 23% and 40%, the accuracy of the model in this invention reaches 93.38% and 88.77% respectively, which is excellent and comparable to the level of current top algorithms. However, there is still a slight gap compared with advanced algorithms such as Xception and Lipforenice. This may be because the model in this invention is mainly trained on Celeb-DF-v2. It is believed that with further parameter tuning and training, it can show better performance on the FF++ dataset.

[0087] Furthermore, as can be inferred from Table 1, the detection accuracy of the model in this invention is less affected by video resolution, and it can still maintain an accuracy of over 88% on low-resolution videos (FF++40). In contrast, the spoofing detection accuracy of most algorithms drops significantly on low-resolution videos, such as Face X-ray and Steg. Features.

[0088] 2. Performance Testing

[0089] Table 2. Performance of this invention on the Celeb-DF-v2 and FF++ datasets.

[0090]

[0091] The model of this invention demonstrates good generalization on Deepfake videos forged using Deepfake, Face2face, Faceshift, Faceswap, and NeuralTextures. It exhibits high accuracy and low false positive rates for various forgery detection methods, proving its adaptability to unseen forgery algorithms and low-resolution scenarios.

[0092] This invention proposes a deepfake video detection method based on biosignals and spatiotemporal signals. rPPG signals are extracted from the video, and a 2D convolutional network is used to learn the differences in these biosignals between real and fake videos. The video is processed into video frames, and the differences in temporal and spatial signals between real and fake videos are learned using structures such as convolutional kernels and pooling layers in a 3D residual network. A fusion model is then integrated based on the 2D and 3D convolutional networks. The proposed model maintains high accuracy in detecting fake videos with low resolution and performs well across different datasets and fake methods, demonstrating robustness and generalization ability.

[0093] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting deepfake videos based on biosignals and spatiotemporal signals, characterized in that: The process includes: first, performing face recognition on the video to be detected and cropping and aligning it to a video containing only faces; extracting video frames and green channel rPPG signals from the cheeks and forehead; learning the temporal and spatial signals in the video frames by constructing and training a 3D residual network, and learning the differences in spatial coherence and temporal consistency of rPPG between real and fake videos by constructing and training a 2D convolutional network; and integrating the two models, training the weights, and constructing the final ensemble model. The 3D residual network consists of convolutional layers, 3D residual blocks, average pooling layers, and fully connected layers, with a network depth of 18. The 3D residual blocks consist of multiple convolutional layers, batch normalization layers, skip connections, and activation function layers. The first layer of the network uses 3D convolution to capture the relationship between temporal and spatial information and fuse the two types of information together. In subsequent layers, the network gradually extracts high-level features by stacking multiple residual blocks and obtains the classification result through average pooling and fully connected layers. Two-dimensional convolutional networks mainly consist of convolutional layers and fully connected layers. The convolutional layers include convolutional kernels, activation functions, batch normalization, and max pooling. The activation function used in the model is ReLU. After the extracted rPPG signal is imported into the network, the features in the signal are obtained through convolutional kernels, the computational cost is reduced through pooling layers, and the fully connected layers are used for further classification. Cross-validation is performed on the 2D convolutional network and the 3D residual network to obtain preliminary classification data of the two networks for fake videos, which is then used as new input to the dataset. This data is combined with the original video labels of the dataset to generate new training and test sets. The ensemble model is then trained using the new dataset.

2. The method for detecting deepfake videos based on biosignals and spatiotemporal signals according to claim 1, characterized in that, The preprocessing process involves performing face recognition on the video to be detected and cropping and aligning it to a video containing only faces: identifying faces in the video based on the face recognition module, aligning the faces, cropping the video, and retaining the face portion in the video.

3. The method for detecting deepfake videos based on biosignals and spatiotemporal signals according to claim 1, characterized in that, The process of extracting video frames from a video involves: importing, cropping, aligning, and marking video segments, reading video frames in a loop, and saving the frame images.

4. The method for detecting deepfake videos based on biosignals and spatiotemporal signals according to claim 1, characterized in that, The process of extracting the green channel rPPG signal from the cheeks and forehead from the video is as follows: 1) Identify and locate faces, and lock onto regions of interest, such as the cheeks, forehead, and areas around the nose where rPPG signals are relatively abundant; 2) Calculate the mean RGB pixel value of each ROI region to obtain the initial rPPG signal C. k (t); 3) Normalize and bandpass filter the green channel signal to remove noise and obtain the final rPPG signal, thus obtaining the time-series rPPG signals of different regions of the face.

5. The method for detecting deepfake videos based on biosignals and spatiotemporal signals according to claim 4, characterized in that, Based on the light absorption and scattering characteristics of skin tissue, a skin pixel formula related to the rPPG signal is constructed in the optical model to obtain the initial rPPG signal C. k (t): C k (t)=I(t)(V s (t)+V d (t))+V n (t); C k (t) represents the pixel value of skin k in the video at time t, which is composed of the illumination intensity I(t) and the skin specular reflection component V. s (t), diffuse skin reflection component V d (t) and noise V n (t) constitutes.

6. The method for detecting deepfake videos based on biosignals and spatiotemporal signals according to claim 1, characterized in that, By training the ensemble model, the fake detection results are output.

Images