Method, device and equipment for detecting spatiotemporal combination of deepfake video

Abstract

The present disclosure relates to a deepfake video spatiotemporal combination detection method, device and equipment, belonging to the technical field of deepfake. The method comprises: for each image frame of a to-be-detected video, performing face recognition processing on the image frame to obtain face feature information of the image frame; detecting the intracframe local image features based on the face feature information of each image frame to obtain the intracframe detection result of the to-be-detected video; detecting the interframe face motion pattern based on the face feature information of all image frames to obtain the interframe detection result of the to-be-detected video; and judging whether the to-be-detected video is a deepfake video based on the combination of the intracframe detection result and the interframe detection result. The present disclosure combines the image feature detection with the timing feature (face motion pattern) detection, and fuses two different detection indexes of image features and timing features, so as to realize the multi-index detection of deepfake videos and improve the detection performance of deepfake videos.

Description

Technical Field

[0001] This disclosure relates to the field of deepfake technology, and in particular to a method, apparatus and device for spatiotemporal detection of deepfake videos. Background Technology

[0002] With the development of the Internet and artificial intelligence, deep learning-based image generation technology has been continuously researched and applied. Among them, deep learning-based face-swapping technology DeepFake has had a huge impact on society.

[0003] The principle behind deepfake videos is primarily to replace the face of the target person in the original video with the face of the person in the original video using network models such as autoencoders or generative adversarial networks (GANs). Users may create deepfake videos for entertainment (such as beautification or face swapping), or they may maliciously spread fake news or create false evidence. Actions taken for the latter purpose will confuse the public, mislead public judgment, and have a negative impact on society.

[0004] Early deepfake videos exhibited noticeable flaws such as unnatural lighting and blurred facial edges. However, with the continuous advancement of deep learning technology and driven by massive amounts of data, the content synthesized through deepfakes has become increasingly realistic, resulting in high-quality deepfake videos that are difficult to distinguish from genuine ones with the naked eye. Therefore, there is an urgent need to develop a new and effective method for detecting deepfake videos to ensure the accuracy and efficiency of deepfake video identification. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a spatiotemporal combined detection method, apparatus, and device for deepfake videos, which can achieve multi-index detection of deepfake videos and improve the detection performance of deepfake videos.

[0006] According to a first aspect of the present disclosure, a spatiotemporal combination detection method for deepfake videos is provided, comprising:

[0007] For each image frame of the video to be detected, face recognition processing is performed on the image frame to obtain the face feature information of the image frame;

[0008] Based on the facial feature information of each image frame, the intra-frame local image feature is detected to obtain the intra-frame detection result of the video to be detected.

[0009] Based on the facial feature information of all image frames, the inter-frame facial motion pattern is detected to obtain the inter-frame detection result of the video to be detected.

[0010] The combination of the intra-frame detection results and the inter-frame detection results is used to determine whether the video to be detected is a deepfake video.

[0011] Optionally, the step of performing face recognition processing on each image frame of the video to be detected to obtain the face feature information of the image frame includes:

[0012] For each image frame of the video to be detected, detect whether the image frame contains a human face;

[0013] If a face is detected in the image frame, a face image is cropped from the image frame, and facial feature points are extracted from the face image; wherein, the face feature information includes the face image and the facial feature points, the face image is the smallest image in the image frame containing the facial features and facial contours, and the facial feature points are used to locate the edges of the facial features and facial contours in the face image.

[0014] Optionally, the step of detecting intra-frame local image features based on facial feature information of each image frame to obtain the intra-frame detection result of the video to be detected includes:

[0015] For each image frame, the facial feature information is subjected to image size correction and head orientation correction to obtain the face correction feature information of the image frame; wherein, the image size correction is used to adjust the face image in the facial feature information to the target size, and the head orientation correction is used to adjust the face image in the facial feature information to the target orientation;

[0016] From the face correction feature information of each image frame, multiple local face feature information of the image frame is extracted. The multiple local face feature information of the image frame comes from different regions of the face image in the face correction feature information of the image frame, and the size of the face image in the multiple local face feature information of the image frame is consistent.

[0017] Feature extraction is performed on the local facial feature information of each image frame to obtain the local detection results of the local facial feature information of each image frame;

[0018] The intra-frame detection results are obtained based on the local detection results of all facial local feature information in all image frames.

[0019] Optionally, the step of extracting features from the local facial features of each image frame to obtain the local detection results of the local facial features of each image frame includes:

[0020] For each individual facial feature, a feature set is extracted from the facial feature information;

[0021] Based on the statistical correlation between pixel neighborhoods, the channel-level subspace approximation transformation is applied to the feature set to remove redundant information in the feature set and obtain dimensionality-reduced features.

[0022] Predict the probability that each channel of the dimensionality-reduced feature comes from a deepfake video to obtain the local detection result of the facial local feature information.

[0023] Optionally, the step of detecting inter-frame facial motion patterns based on facial feature information from all image frames to obtain inter-frame detection results of the video to be detected includes:

[0024] Feature point calibration is performed on the facial feature information of each image frame to obtain the facial calibration feature information of the image frame;

[0025] A first feature vector sequence is constructed using the face calibration feature information of each image frame, where each feature vector in the first feature vector sequence indicates the face calibration feature information of an image frame;

[0026] A second feature vector sequence is constructed using the difference in face calibration feature information between adjacent image frames, where each feature vector in the second feature vector sequence indicates the difference in face calibration feature information between a pair of adjacent image frames;

[0027] The first feature vector sequence and the second feature vector sequence are learned and predicted by a two-stream recurrent neural network, and the predicted probabilities output by the two branches of the two-stream recurrent neural network are fused to obtain the inter-frame detection result.

[0028] Optionally, the step of performing feature point calibration on the facial feature information of each image frame to obtain the facial calibration feature information of the image frame includes:

[0029] The face image in the face feature information of each image frame is downsampled multiple times to obtain multiple face images of different sizes, and a pyramid representation is constructed based on the multiple face images of different sizes;

[0030] The pyramid representation is subjected to a Lucas-Kanade operation with image blocks of the same size to utilize the continuity between frames to predict the coordinates of each facial feature point in the face feature information of the image frame in the next image frame.

[0031] The facial calibration feature information of the image frame is obtained by integrating the facial feature point coordinates predicted by the Lucas-Kanade operation with the facial feature point coordinates extracted from the facial feature information using a Kalman filter.

[0032] Optionally, determining whether the video to be detected is a deepfake video based on the combination of the intra-frame detection results and the inter-frame detection results includes:

[0033] If the intra-frame detection result is within the target result range, the intra-frame detection result and the inter-frame detection result are weighted and summed according to the target weight coefficient combination to obtain the spatiotemporal combined detection score.

[0034] If the spatiotemporal combination detection score is greater than a preset threshold, the video to be detected is determined to be a deepfake video; if the spatiotemporal combination detection score is less than the preset threshold, the video to be detected is determined not to be a deepfake video.

[0035] Optionally, the process of obtaining the target weight coefficient combination includes:

[0036] The test dataset was used to test the intra-frame local image feature detection branch and the inter-frame facial motion pattern detection branch, and the intra-frame test results and inter-frame test results were obtained.

[0037] Calculate the first AUC value based on the intra-frame test results, and calculate the second AUC value based on the inter-frame test results;

[0038] Based on the first AUC value and the second AUC value, the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient in the target weight coefficient combination are determined; wherein, the sum of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to 1, and the ratio of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to the ratio of the first AUC value and the second AUC value.

[0039] According to a second aspect of the present disclosure, a spatiotemporal combination detection apparatus for deepfake videos is provided, the apparatus comprising:

[0040] The recognition module is used to perform face recognition processing on each image frame of the video to be detected, and obtain the face feature information of the image frame.

[0041] The intra-frame detection module is used to detect local image features within each image frame based on the facial feature information of each image frame, and to obtain the intra-frame detection results of the video to be detected.

[0042] The inter-frame detection module is used to detect inter-frame facial motion patterns based on facial feature information of all image frames, and to obtain the inter-frame detection results of the video to be detected.

[0043] The determination module is used to determine whether the video to be detected is a deepfake video based on the combination of the intra-frame detection results and the inter-frame detection results.

[0044] According to a third aspect of the present disclosure, a computer device is provided, the computer device comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the spatiotemporal combination detection method for deepfake videos provided in the first aspect of the present disclosure.

[0045] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer program instructions are stored thereon, and when executed by a processor, the computer program instructions implement the spatiotemporal combination detection method for deepfake videos provided in the first aspect of the present disclosure.

[0046] The technical solutions provided in this disclosure have at least the following beneficial effects:

[0047] This invention combines image feature detection with temporal feature (facial motion pattern) detection, integrating two different detection indicators: image features and temporal features. This enables multi-indicator detection of deepfake videos and improves detection performance.

[0048] This invention proposes a scheme for image feature detection that includes a series of steps such as size and orientation correction, local image cropping, local probability prediction, and ensemble classification. The local probability prediction uses PixelHop++ units, channel-level subspace approximation transformation (Saab), and XGBoost classifiers, without using any deep learning methods. Therefore, it can not only effectively resist adversarial attacks such as deepfake, but also achieve lightweight model parameters and size by eliminating the need for backpropagation, thus effectively reducing training complexity.

[0049] Furthermore, since the face moves almost no further in each image frame, the extracted facial feature points will exhibit significant jitter. This jitter noise severely interferes with temporal modeling, ultimately leading to inaccurate detection results. Therefore, before performing inter-frame facial motion pattern detection, this invention first uses a pyramid Lucas-Kanade operation for feature point calibration, and then employs a Kalman filter to integrate the facial feature point coordinates predicted by the Lucas-Kanade operation with the actual extracted facial feature point coordinates, removing the jitter noise introduced during feature point calibration, thereby improving the detection accuracy of deepfake videos.

[0050] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0052] Figure 1 This is a schematic diagram illustrating a process for identifying deepfake videos according to an exemplary embodiment.

[0053] Figure 2 This is a flowchart illustrating a spatiotemporal combination detection method for deepfake videos according to an exemplary embodiment.

[0054] Figure 3 This is a schematic diagram illustrating facial feature information according to an exemplary embodiment.

[0055] Figure 4 This is a schematic diagram illustrating the extraction of local feature information according to an exemplary embodiment.

[0056] Figure 5 This is a schematic diagram showing a comparison of feature points before and after calibration, according to an exemplary embodiment.

[0057] Figure 6 This is a schematic diagram illustrating the learning and prediction of inter-frame facial motion patterns according to an exemplary embodiment.

[0058] Figure 7 This is a schematic diagram comparing the ROC curves of the two identification methods on the test dataset Celeb-DF-v2.

[0059] Figure 8 This is a schematic diagram comparing the PRC curves of the two discrimination methods on the test dataset Celeb-DF-v2.

[0060] Figure 9 This is a block diagram of a spatiotemporal combination detection apparatus for deepfake videos, according to an exemplary embodiment.

[0061] Figure 10 This is a block diagram of a computer device 1000 according to an exemplary embodiment. Detailed Implementation

[0062] The exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0063] like Figure 1 As shown, the deepfake video identification scheme provided by the present invention combines intra-frame local image feature detection and inter-frame facial motion pattern detection. Optionally, intra-frame local image feature detection is implemented through an intra-frame detection model, and inter-frame facial motion pattern detection is implemented through an inter-frame detection model.

[0064] like Figure 2 As shown, the detection method provided by the present invention includes the following steps (steps 1 to 4).

[0065] Step 1: For each image frame of the video to be detected, perform face recognition processing on the image frame to obtain the face feature information of the image frame.

[0066] In this invention, at least two image frames of the video to be detected contain a face. When performing face recognition processing on each image frame of the video to be detected, the system first detects whether the image frame contains a face; if it does, it further extracts facial feature information. Based on this, step 1 includes: for each image frame, if a face is detected in the image frame, cropping a face image from the image frame; extracting facial feature points from the face image; the facial feature information of the image frame includes the face image and the facial feature points.

[0067] The cropped face image is the smallest image in the image frame containing the facial features and contours, and the extracted facial feature points are used to locate the edges of the facial features and contours in this face image. This invention does not limit the number or shape of the facial feature points; optionally, 68 dots are extracted from the face image as facial feature points. For example, the face image cropped from the image frame, and the facial feature points extracted from the face image, such as... Figure 3 As shown.

[0068] Furthermore, to ensure the universality of deepfake video identification, this invention does not limit the duration, frame rate, resolution, or motion of the video to be detected.

[0069] Step 2: Detect local image features within each image frame based on the facial feature information to obtain the intra-frame detection results of the video to be detected.

[0070] In one example, such as Figure 1 As shown, step 2 above includes the following sub-steps (steps 2.1 to 2.4). Optionally, step 2 above can be implemented by an intra-frame detection model, which includes a correction module, a truncation module, a local prediction module, and an ensemble model. Step 2.1 below can be implemented by the correction module, step 2.2 can be implemented by the truncation module, step 2.3 can be implemented by the local prediction module, and step 2.4 can be implemented by the ensemble module.

[0071] Step 2.1: Perform image size correction and head orientation correction on the facial feature information of each image frame to obtain the facial correction feature information of that image frame.

[0072] Image size correction adjusts the face images in the facial feature information of all image frames to a target size, such as 128×128, to ensure that the face size is consistent across all image frames. Head orientation correction adjusts the face images in the facial feature information of all image frames to a target orientation, to ensure that the head pose is consistent across all image frames. Therefore, the face correction feature information includes the corrected face image and facial feature points.

[0073] Step 2.2: Extract multiple local facial features from the face correction feature information of each image frame.

[0074] The multi-face local feature information of each image frame comes from different regions of the face image in the face correction feature information of that image frame, such as the left eye, right eye and mouth; and the face image in the multi-face local feature information of each image frame has the same size, such as 32×32.

[0075] Step 2.3: Extract features from the local facial features of each image frame to obtain the local detection results of the local facial features of each image frame.

[0076] Optionally, step 2.3 is implemented through a local prediction module, such as... Figure 4 As shown, the local prediction module includes a feature extraction unit, a spatial dimensionality reduction unit, and a classification unit. The feature extraction unit extracts a feature set from local facial feature information; the spatial dimensionality reduction unit applies a channel-level subspace approximation transformation to the feature set based on the statistical correlation between pixel neighborhoods to remove redundant information and obtain dimensionality-reduced features; the classification unit predicts the probability that each channel of the dimensionality-reduced features originates from a deepfake video to obtain local detection results for local facial feature information.

[0077] The feature extraction unit takes the local facial feature information of each image frame as input and outputs the feature set corresponding to each local facial feature information. Optionally, such as Figure 4 As shown, the feature extraction unit consists of three cascaded PixelHop++ units. This invention does not limit the block size or stride of each unit. Figure 4 The example below uses a block size of 3×3 and a stride of 1. In this example, each hop (Hop) in the cascaded PixelHop++ unit has a different feature extraction dimension, allowing features to be extracted from multiple dimensions, thus improving the accuracy and completeness of feature extraction.

[0078] The output dimension of the feature set obtained by the feature extraction unit is still not concise enough. Therefore, this invention further uses a spatial dimensionality reduction unit to remove redundant information from the extracted features. During the spatial dimensionality reduction process, the statistical correlation between pixel neighborhoods is utilized to apply the channel-level subspace approximation transform (Saab) to the feature set. First, the feature set is decomposed into local means and frequency components. Then, principal component analysis is applied to the frequency components to derive kernels. Each kernel represents a specific frequency-selective filter; kernels with larger eigenvalues ​​extract lower frequency components, while kernels with smaller eigenvalues ​​extract higher frequency components. Dimensionality reduction is achieved by discarding high-frequency components with very small eigenvalues.

[0079] After removing redundancy, Ki1 channels can be obtained from the i-th hop (i = 1, 2, 3). A binary classifier is pre-trained for each channel to learn semantics, which is then used in the subsequent output based on the spatial dimensionality reduction unit to predict the probability that each channel comes from a deepfake video. Optionally, XGBoost (eXtreme Gradient Boosting) is used as the classifier for each channel, and its maximum depth is set to 1 to prevent overfitting.

[0080] Step 2.4: Based on the local detection results of all face local feature information in all image frames, obtain the intra-frame detection results of the video to be detected.

[0081] The local detection results of each facial feature information include the probability that the facial feature information comes from a deepfake video through multiple channels. By combining the predicted probabilities of all channels (such as averaging or weighted summation), the probability that the facial feature information comes from a deepfake video can be obtained. For each image frame, by combining the predicted probabilities of all facial feature information in that image frame (such as averaging or weighted summation), the probability that the image frame comes from a deepfake video can be obtained.

[0082] For the video to be detected, the predicted probabilities of all image frames need to be combined to obtain the probability that the video to be detected is a deepfake video, i.e., the intra-frame detection result. In order to control the input dimension of each module and reduce the amount of computation, the video to be detected can be segmented (e.g., every six image frames as a segment). The predicted probabilities of all image frames in each segment are combined (e.g., by averaging or weighted summation) to obtain the probability that the segment comes from a deepfake video. Then, the predicted probabilities of all segments are combined to obtain the intra-frame detection result of the video to be detected.

[0083] It should be understood that when an image frame contains multiple faces, the probability that the face in the image frame comes from a deepfake video can be calculated first based on the predicted probability of the local feature information of all faces belonging to the same face in the image frame, and then the probability that the image frame comes from a deepfake video can be calculated based on the predicted probability of all faces in the image frame.

[0084] Step 3: Detect inter-frame facial motion patterns based on the facial feature information of all image frames, and obtain the inter-frame detection results of the video to be detected.

[0085] In one example, such as Figure 1 As shown, step 3 above includes the following sub-steps (steps 3.1 to 3.4). Optionally, step 3 above can be implemented by an inter-frame detection model, which includes a calibration module, a sequence construction module, and a prediction module. Step 3.1 below can be implemented by the calibration module, steps 3.2 and 3.3 below can be implemented by the sequence construction module, and step 3.4 below can be implemented by the prediction module.

[0086] Step 3.1: Perform feature point calibration on the facial feature information of each image frame to obtain the facial calibration feature information of that image frame.

[0087] Detecting inter-frame facial motion patterns requires continuous facial feature point coordinates, necessitating high accuracy. However, our research has revealed that even when the face moves almost no further across image frames, the extracted facial feature points exhibit significant jitter. This jitter noise severely interferes with temporal modeling, ultimately leading to inaccurate detection results. Therefore, this invention performs feature point calibration before detecting inter-frame facial motion patterns to improve the accuracy of the detection results.

[0088] Optionally, this invention employs the Lucas-Kanade optical flow algorithm to calculate the movement of facial feature points between frames, utilizing the continuity between frames to predict the position of each facial feature point in the next image frame. However, since the Lucas-Kanade operation is sensitive to the size of facial image patches, a pyramid Lucas-Kanade operation can be introduced. First, the face image is downsampled several times to half the size of the original image, resulting in multiple face images of different sizes. A pyramid representation is then constructed based on these multiple face images of different sizes. Then, the Lucas-Kanade operation is performed on the pyramid representation using image patches of the same size to utilize the continuity between frames to predict the coordinates of each facial feature point in the face feature information of the image frame in the next image frame. Furthermore, since the Lucas-Kanade operation also introduces noise, a Kalman filter is used to integrate the facial feature point coordinates predicted by the Lucas-Kanade operation with the facial feature point coordinates extracted from the face feature information to obtain the face calibration feature information of the image frame. Based on this, the face calibration feature information includes the face image and the calibrated facial feature points.

[0089] For example, such as Figure 5 As shown, Figure 5 In (1) and (2), the dots in the left face image are facial feature points in the face feature information before feature point calibration. Figure 5 In (1) and (2), the dots in the right-hand face image represent facial feature points in the face calibration feature information after feature point calibration. It can be seen that the positions of the facial feature points have changed before and after feature point calibration.

[0090] Step 3.2: Construct the first feature vector sequence using the face calibration feature information of each image frame.

[0091] In the first feature vector sequence, each feature vector indicates the face calibration feature information of an image frame. For example, suppose the video to be detected has n image frames, and the face calibration feature information of each image frame contains m facial feature points (m is a positive integer, such as 68). The k-th facial feature point is represented by Z. k =[x k y k ] T Then the face calibration feature information α of the i-th image frame i It can be represented as The first eigenvector sequence A = [α1, ..., α2] n ] T .

[0092] Step 3.3: Construct a second feature vector sequence by using the difference in face calibration feature information between adjacent image frames.

[0093] Each feature vector in the second feature vector sequence indicates the difference in face calibration feature information between a pair of adjacent image frames. For example, suppose the video to be detected has n image frames, and each image frame has m facial feature points (n is a positive integer, such as 68) in its face calibration feature information. The k-th facial feature point is denoted as Z. k =[x k y k ] T Then the face calibration feature information α of the i-th image frame i It can be represented as The difference β between the face calibration feature information of the i-th image frame and the (i+1)-th image frame i It can be represented as The second eigenvector sequence B = [β1, ..., β2] n-1 ] T .

[0094] Step 3.4: Learn and predict the first feature vector sequence and the second feature vector sequence through a two-stream recurrent neural network, and fuse the predicted probabilities output by the two branches in the two-stream recurrent neural network to obtain the inter-frame detection results.

[0095] The first feature vector sequence can indicate the facial movement pattern of a face, while the second feature vector sequence can indicate the difference pattern of facial movement. For example... Figure 6 As shown, the prediction module can use a two-stream recurrent neural network (RNN) to learn and predict the first and second feature vector sequences of the input, respectively, to obtain the probability that the video to be detected is a deepfake video. Then, the predicted probabilities of the two branches are fused (e.g., by averaging or weighted summation) to obtain the inter-frame detection results of the video to be detected. Optionally, in order to control the input dimension of each module and reduce the computational load, the video to be detected can be segmented (e.g., every six image frames as a segment), and the inter-frame detection results of each segment can be combined (e.g., by averaging or weighted summation) to obtain the inter-frame detection results of the entire video to be detected.

[0096] Step 4: Use the combination of intra-frame detection results and inter-frame detection results to determine whether the video to be detected is a deepfake video.

[0097] In one example, step 4 above includes the following sub-steps (steps 4.1 to 4.2).

[0098] Step 4.1: If the intra-frame detection result is within the target result range, perform weighted summation on the intra-frame detection result and the inter-frame detection result according to the target weight coefficient combination to obtain the spatiotemporal combined detection score.

[0099] To more effectively combine intra-frame and inter-frame detection results, weights can be allocated based on the detection performance of the intra-frame local image feature detection branch (such as the intra-frame detection model) and the inter-frame facial motion pattern detection branch (such as the inter-frame detection model) to obtain a target weight coefficient combination. Then, the intra-frame and inter-frame detection results are weighted and summed based on the target weight coefficient combination to obtain a spatiotemporal combined detection score. This spatiotemporal combined detection score indicates the probability that the video to be detected is a deepfake video from the perspective of image feature and temporal feature fusion. Finally, the spatiotemporal combined detection score is used to determine whether the video to be detected is a deepfake video.

[0100] Optionally, the process of obtaining the target weight coefficient combination includes: testing the intra-frame local image feature detection branch and the inter-frame facial motion pattern detection branch using a test dataset to obtain intra-frame test results and inter-frame test results; calculating a first AUC value based on the intra-frame test results and a second AUC value based on the inter-frame test results; determining the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient in the target weight coefficient combination based on the first AUC value and the second AUC value; wherein the sum of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to 1, and the ratio of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to the ratio of the first AUC value and the second AUC value.

[0101] Because research has shown that the intra-frame local image feature detection branch achieves an AUC value of over 90% on various test datasets, but its predicted probability has a significant potential for error when it falls within the range of 0.4 to 0.6, this invention, in order to efficiently and accurately obtain the spatiotemporal combined detection score, fuses the intra-frame and inter-frame detection results according to the target weight coefficient combination when the intra-frame detection result falls within the target result range. Conversely, when the intra-frame detection result does not fall within the target result range, the intra-frame detection result is directly used as the spatiotemporal combined detection score. The target result range can be 0.4 to 0.6, or other ranges (such as 0.3 to 0.7), and this invention does not limit this range.

[0102] Step 4.2: If the spatiotemporal combination detection score is greater than the preset threshold, the video to be detected is determined to be a deepfake video; if the spatiotemporal combination detection score is less than the preset threshold, the video to be detected is determined not to be a deepfake video.

[0103] A preset threshold of 0.5 can typically be set. When the spatiotemporal combination detection score is greater than 0.5, the video is determined to be a deepfake; when the score is less than 0.5, it is determined to be a genuine video. Of course, other preset threshold settings are also possible. For example, when strict identification requirements for deepfake videos are needed, a preset threshold of 0.7 can be set. Furthermore, in practical applications, when the spatiotemporal combination detection score equals the preset threshold, the video can be flexibly designated as either a deepfake or not.

[0104] In summary, to ensure the accuracy and efficiency of deepfake video identification, this invention proposes a spatiotemporal combined deepfake video identification method. Based on image feature detection, it combines temporal feature (facial motion pattern) detection, integrating two different detection indicators, image features and temporal features, to achieve multi-indicator detection of deepfake videos and improve detection performance.

[0105] For image feature detection, this invention proposes a scheme that includes a series of steps such as size and orientation correction, local image cropping, local probability prediction, and ensemble classification. The local probability prediction uses PixelHop++ units, channel-level subspace approximation transformation (Saab), and XGBoost classifiers, without using any deep learning methods. Therefore, it can effectively resist adversarial attacks such as deepfake. In addition, the learning mechanism of this non-deep learning method does not require backpropagation, which enables lightweight model parameters and size, effectively reducing training complexity.

[0106] This invention uses the Celeb-DF-v2 test dataset to test and compare the deepfake video identification methods of related technologies and the present invention. Specifically, the related technologies detect deepfake videos based solely on image features, while the present invention uses a fusion of image features and temporal features for deepfake video detection.

[0107] First, on the test dataset Celeb-DF-v2, the ROC curves of the two detection methods are as follows: Figure 7 As shown in the ROC curve, the AUC value was improved by 0.0086 using the identification method provided by this invention.

[0108] Secondly, on the test dataset Celeb-DF-v2, the PRC curves of the two detection methods are as follows: Figure 8 As shown in the figure, the area under the PRC curve increases by 0.0079 compared to related technologies when using the identification method provided by this invention.

[0109] Furthermore, as shown in Table 1, by calculating relevant indicators such as accuracy, it can be seen that the identification method provided by this invention has a significant improvement over related technologies, with accuracy and precision increased by 13.39% and 15.13%, respectively.

[0110] Table 1. Comparison of generalization indices between the two identification methods

[0111]

[0112] The test dataset Celeb-DF-v2 contains 59 real videos and 59 deepfake videos. As shown in Table 2, using the identification method provided by related technologies, the number of real videos incorrectly identified as deepfake videos was 7. Using the identification method provided by this invention, this number decreased to 2, and the false positive rate also decreased from 11.86% to 3.39%. Using the identification method provided by related technologies, 47 deepfake videos were correctly identified. Using the identification method provided by this invention, this number increased to 57, and the deepfake detection rate increased from 79.66% to 96.61%.

[0113] Table 2 Comparison of Accuracy Indicators of the Two Identification Methods

[0114]

[0115] Furthermore, in the temporal feature detection process, this invention discovered that even when the face moves almost no further across image frames, the extracted facial feature points exhibit significant jitter. This jitter noise severely interferes with temporal modeling, ultimately leading to inaccurate detection results. Therefore, before performing inter-frame facial motion pattern detection, this invention first employs a pyramid Lucas-Kanade operation for feature point calibration, and then uses a Kalman filter to integrate the facial feature point coordinates predicted by the Lucas-Kanade operation with the actual extracted facial feature point coordinates. This removes the jitter noise introduced during feature point calibration, thereby improving the detection accuracy of deepfake videos.

[0116] Exemplary device

[0117] Figure 9 This is an exemplary embodiment illustrating a spatiotemporal detection device for deepfake videos, with reference to... Figure 9 The device includes: an identification module 910, an intra-frame detection module 920, an inter-frame detection module 930, and a determination module 940.

[0118] The recognition module 910 is used to perform face recognition processing on each image frame of the video to be detected, and obtain the face feature information of the image frame.

[0119] The intra-frame detection module 920 is used to detect local image features within each image frame based on the facial feature information of each image frame, and to obtain the intra-frame detection result of the video to be detected.

[0120] The inter-frame detection module 930 is used to detect inter-frame facial motion patterns based on the facial feature information of all image frames, and to obtain the inter-frame detection results of the video to be detected.

[0121] The determination module 940 is used to determine whether the video to be detected is a deepfake video based on the combination of the intra-frame detection results and the inter-frame detection results.

[0122] In one embodiment of this disclosure, the recognition module 910 is further configured to: detect whether a face is contained in each image frame of the video to be detected; if a face is detected in the image frame, crop a face image from the image frame and extract facial feature points from the face image; wherein the face feature information includes the face image and the facial feature points, the face image is the smallest image containing the facial features and facial contours of the face in the image frame, and the facial feature points are used to locate the edges of the facial features and facial contours in the face image.

[0123] In one embodiment of this disclosure, the intra-frame detection module 920 is further configured to: perform image size correction and head orientation correction on the facial feature information of each image frame to obtain facial correction feature information of the image frame; wherein, the image size correction is used to adjust the facial image in the facial feature information to a target size, and the head orientation correction is used to adjust the facial image in the facial feature information to a target orientation; extract multiple local facial feature information of the image frame from the facial correction feature information of each image frame, wherein the multiple local facial feature information of the image frame originates from different regions of the facial image in the facial correction feature information of the image frame, and the size of the facial image in the multiple local facial feature information of the image frame is consistent; perform feature extraction on each local facial feature information of each image frame to obtain the local detection result of each local facial feature information of each image frame; and obtain the intra-frame detection result based on the local detection results of all local facial feature information of all image frames.

[0124] In one embodiment of this disclosure, the intra-frame detection module 920 is further configured to: extract a feature set from each local facial feature information; apply a channel-level subspace approximation transformation to the feature set based on the statistical correlation between pixel neighborhoods to remove redundant information in the feature set and obtain a dimensionality-reduced feature; and predict the probability that each channel of the dimensionality-reduced feature comes from a deepfake video to obtain a local detection result of the local facial feature information.

[0125] In one embodiment of this disclosure, the inter-frame detection module 930 is further configured to: perform feature point calibration on the facial feature information of each image frame to obtain facial calibration feature information of the image frame; construct a first feature vector sequence using the facial calibration feature information of each image frame, wherein each feature vector in the first feature vector sequence indicates the facial calibration feature information of an image frame; construct a second feature vector sequence using the difference between the facial calibration feature information of adjacent image frames, wherein each feature vector in the second feature vector sequence indicates the difference between the facial calibration feature information of a pair of adjacent image frames; learn and predict the first feature vector sequence and the second feature vector sequence through a two-stream recurrent neural network, and fuse the predicted probabilities output by the two branches in the two-stream recurrent neural network to obtain the inter-frame detection result.

[0126] In one embodiment of this disclosure, the inter-frame detection module 930 is further configured to: downsample the face image in the face feature information of each image frame multiple times to obtain multiple face images of different sizes, and construct a pyramid representation based on the multiple face images of different sizes; perform a Lucas-Kanade operation on the pyramid representation with image blocks of the same size to predict the coordinates of each facial feature point in the face feature information of the image frame in the next image frame by utilizing the continuity between frames; and integrate the facial feature point coordinates predicted by the Lucas-Kanade operation and the facial feature point coordinates extracted from the face feature information based on a Kalman filter to obtain the face calibration feature information of the image frame.

[0127] In one embodiment of this disclosure, the determination module 940 is further configured to: when the intra-frame detection result is within the target result range, perform weighted summation processing on the intra-frame detection result and the inter-frame detection result according to the target weight coefficient combination to obtain a spatiotemporal combined detection score; if the spatiotemporal combined detection score is greater than a preset threshold, then determine that the video to be detected is a deepfake video; if the spatiotemporal combined detection score is less than the preset threshold, then determine that the video to be detected is not a deepfake video.

[0128] In one embodiment of this disclosure, the determination module 940 is further configured to: test the intra-frame local image feature detection branch and the inter-frame facial motion pattern detection branch using a test dataset, respectively, and obtain intra-frame test results and inter-frame test results; calculate a first AUC value based on the intra-frame test results, and calculate a second AUC value based on the inter-frame test results; and determine the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient in the target weight coefficient combination based on the first AUC value and the second AUC value; wherein the sum of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to 1, and the ratio of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to the ratio of the first AUC value and the second AUC value.

[0129] The exemplary device is an embodiment of the device corresponding to the exemplary method described above. The specific operation of each module can be understood with reference to the description of the method embodiment, and will not be repeated here.

[0130] Exemplary electronic devices

[0131] Figure 10 This is a block diagram illustrating a computer device 1000 according to an exemplary embodiment. The computer device 1000 may be a terminal, a laptop computer, a desktop computer, a server, a computer cluster, or other types of electronic devices.

[0132] Reference Figure 10 The computer device 1000 may include at least one processor 1010 and a memory 1020. The processor 1010 can execute instructions stored in the memory 1020. The processor 1010 is communicatively connected to the memory 1020 via a data bus. In addition to the memory 1020, the processor 1010 may also be communicatively connected to an input device 1030, an output device 1040, and a communication device 1050 via the data bus.

[0133] Processor 1010 can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0134] The memory 1020 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0135] In this embodiment of the present disclosure, the memory 1020 stores executable instructions, and the processor 1010 can read the executable instructions from the memory 1020 and execute the instructions to implement all or part of the steps of the spatiotemporal combination detection method for deepfake videos in the above exemplary embodiment.

[0136] Exemplary computer-readable storage media

[0137] In addition to the methods and apparatus described above, exemplary embodiments of this disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0138] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0139] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0140] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A spatiotemporal combined detection method for deepfake videos, characterized in that, The method includes: For each image frame of the video to be detected, face recognition processing is performed on the image frame to obtain the face feature information of the image frame; Based on the facial feature information of each image frame, the intra-frame local image feature is detected to obtain the intra-frame detection result of the video to be detected. Based on the facial feature information of all image frames, the inter-frame facial motion pattern is detected to obtain the inter-frame detection result of the video to be detected. Based on the combination of the intra-frame detection results and the inter-frame detection results, it is determined whether the video to be detected is a deepfake video; The step of detecting local image features within each image frame based on facial feature information to obtain the intra-frame detection results of the video to be detected includes: For each image frame, the facial feature information is subjected to image size correction and head orientation correction to obtain the face correction feature information of the image frame; wherein, the image size correction is used to adjust the face image in the facial feature information to the target size, and the head orientation correction is used to adjust the face image in the facial feature information to the target orientation; From the face correction feature information of each image frame, multiple local face feature information of the image frame is extracted. The multiple local face feature information of the image frame comes from different regions of the face image in the face correction feature information of the image frame, and the size of the face image in the multiple local face feature information of the image frame is consistent. Feature extraction is performed on the local facial feature information of each image frame to obtain the local detection results of the local facial feature information of each image frame; The intra-frame detection result is obtained based on the local detection results of all facial local feature information in all image frames; the inter-frame facial motion pattern detection based on facial feature information in all image frames, to obtain the inter-frame detection result of the video to be detected, includes: Feature point calibration is performed on the facial feature information of each image frame to obtain the facial calibration feature information of the image frame; A first feature vector sequence is constructed using the face calibration feature information of each image frame, where each feature vector in the first feature vector sequence indicates the face calibration feature information of an image frame; A second feature vector sequence is constructed using the difference in face calibration feature information between adjacent image frames, where each feature vector in the second feature vector sequence indicates the difference in face calibration feature information between a pair of adjacent image frames; The first feature vector sequence and the second feature vector sequence are learned and predicted by a two-stream recurrent neural network, and the predicted probabilities output by the two branches of the two-stream recurrent neural network are fused to obtain the inter-frame detection result.

2. The method according to claim 1, characterized in that, The step involves performing face recognition processing on each image frame of the video to be detected to obtain the face feature information of the image frame, including: For each image frame of the video to be detected, detect whether the image frame contains a human face; If a face is detected in the image frame, a face image is cropped from the image frame, and facial feature points are extracted from the face image; wherein, the face feature information includes the face image and the facial feature points, the face image is the smallest image in the image frame containing the facial features and facial contours, and the facial feature points are used to locate the edges of the facial features and facial contours in the face image.

3. The method according to claim 1, characterized in that, The step of extracting features from the local facial features of each image frame to obtain the local detection results of the local facial features of each image frame includes: For each individual facial feature, a feature set is extracted from the facial feature information; Based on the statistical correlation between pixel neighborhoods, the channel-level subspace approximation transformation is applied to the feature set to remove redundant information in the feature set and obtain dimensionality-reduced features. Predict the probability that each channel of the dimensionality-reduced feature comes from a deepfake video to obtain the local detection result of the facial local feature information.

4. The method according to claim 1, characterized in that, The step of performing feature point calibration on the facial feature information of each image frame to obtain the facial calibration feature information of the image frame includes: The face image in the face feature information of each image frame is downsampled multiple times to obtain multiple face images of different sizes, and a pyramid representation is constructed based on the multiple face images of different sizes; The pyramid representation is subjected to a Lucas-Kanade operation with image blocks of the same size to utilize the continuity between frames to predict the coordinates of each facial feature point in the face feature information of the image frame in the next image frame. The facial calibration feature information of the image frame is obtained by integrating the facial feature point coordinates predicted by the Lucas-Kanade operation with the facial feature point coordinates extracted from the facial feature information using a Kalman filter.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining whether the video to be detected is a deepfake video by combining the intra-frame detection results and the inter-frame detection results includes: If the intra-frame detection result is within the target result range, the intra-frame detection result and the inter-frame detection result are weighted and summed according to the target weight coefficient combination to obtain the spatiotemporal combined detection score. If the spatiotemporal combination detection score is greater than a preset threshold, the video to be detected is determined to be a deepfake video; if the spatiotemporal combination detection score is less than the preset threshold, the video to be detected is determined not to be a deepfake video.

6. The method according to claim 5, characterized in that, The process of obtaining the target weight coefficient combination includes: The test dataset was used to test the intra-frame local image feature detection branch and the inter-frame facial motion pattern detection branch, and the intra-frame test results and inter-frame test results were obtained. Calculate the first AUC value based on the intra-frame test results, and calculate the second AUC value based on the inter-frame test results; Based on the first AUC value and the second AUC value, the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient in the target weight coefficient combination are determined; wherein, the sum of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to 1, and the ratio of the values ​​of the intra-frame weight coefficient and the inter-frame weight coefficient is equal to the ratio of the first AUC value and the second AUC value.

7. A spatiotemporal detection device for deepfake videos, characterized in that, The device includes: The recognition module is used to perform face recognition processing on each image frame of the video to be detected, and obtain the face feature information of the image frame. The intra-frame detection module is used to detect local image features within each image frame based on the facial feature information of each image frame, and to obtain the intra-frame detection results of the video to be detected. The inter-frame detection module is used to detect inter-frame facial motion patterns based on facial feature information of all image frames, and to obtain the inter-frame detection results of the video to be detected. The determination module is used to determine whether the video to be detected is a deepfake video based on the combination of the intra-frame detection results and the inter-frame detection results. The step of detecting local image features within each image frame based on facial feature information to obtain the intra-frame detection results of the video to be detected includes: For each image frame, the facial feature information is subjected to image size correction and head orientation correction to obtain the face correction feature information of the image frame; wherein, the image size correction is used to adjust the face image in the facial feature information to the target size, and the head orientation correction is used to adjust the face image in the facial feature information to the target orientation; From the face correction feature information of each image frame, multiple local face feature information of the image frame is extracted. The multiple local face feature information of the image frame comes from different regions of the face image in the face correction feature information of the image frame, and the size of the face image in the multiple local face feature information of the image frame is consistent. Feature extraction is performed on the local facial feature information of each image frame to obtain the local detection results of the local facial feature information of each image frame; The intra-frame detection result is obtained based on the local detection results of all facial local feature information in all image frames; the inter-frame facial motion pattern detection based on facial feature information in all image frames, to obtain the inter-frame detection result of the video to be detected, includes: Feature point calibration is performed on the facial feature information of each image frame to obtain the facial calibration feature information of the image frame; A first feature vector sequence is constructed using the face calibration feature information of each image frame, where each feature vector in the first feature vector sequence indicates the face calibration feature information of an image frame; A second feature vector sequence is constructed using the difference in face calibration feature information between adjacent image frames, where each feature vector in the second feature vector sequence indicates the difference in face calibration feature information between a pair of adjacent image frames; The first feature vector sequence and the second feature vector sequence are learned and predicted by a two-stream recurrent neural network, and the predicted probabilities output by the two branches of the two-stream recurrent neural network are fused to obtain the inter-frame detection result.

8. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the spatiotemporal combination detection method for deepfake videos as described in any one of claims 1-6.

Images