Deepfake video detection method and device based on identity and spatiotemporal inconsistency

Abstract

The application provides a kind of based on identity and space-time inconsistency's deep fake video detection method and device, comprising: obtaining video to be detected and fake video detection model;Fake video detection model includes identity comparison network, space comparison network, frame comparison network and feature fusion network;Based on identity comparison network, identity information consistency detection is carried out to video to be detected to obtain identity information feature;Based on space comparison network, spatial inconsistency detection is carried out to video to be detected to obtain spatial inconsistency feature;Based on frame comparison network, the time inconsistency feature of video to be detected is extracted;Identity information feature, spatial inconsistency feature and time inconsistency feature are fused using feature fusion network, and fake video detection result is obtained.Through the combination of identity information consistency detection, spatial inconsistency detection and time inconsistency detection, the authenticity of video can be comprehensively analyzed, and various fake methods and new fake technologies can be coped with.

Description

Technical Field

[0001] This invention relates to the field of deepfake detection technology, and in particular to a method and apparatus for detecting deepfake videos based on identity and spatiotemporal inconsistencies. Background Technology

[0002] The misuse of deepfake technology could lead to malicious activities such as disinformation and telecommunications fraud, necessitating effective detection methods to identify forged videos. In recent years, the rapid development of deep learning has provided new solutions for the field of computer vision.

[0003] Existing deepfake detection methods utilize low-level image artifacts for detection. Forged images retain some target identity features outside the face-swapping area, leading to inconsistencies between the source and target identities in deepfakes. Existing deepfake detection methods fail to capture these identity and temporal inconsistencies, resulting in low accuracy in deepfake video detection. Summary of the Invention

[0004] This invention provides a method and apparatus for detecting deepfake videos based on identity and spatiotemporal inconsistencies, addressing the shortcomings of existing technologies in capturing identity and spatiotemporal inconsistencies in deepfake videos and improving the accuracy of deepfake video detection. The technical solution proposed by this invention is as follows:

[0005] In a first aspect, the present invention provides a method for detecting deepfake videos based on identity and spatiotemporal inconsistencies, comprising:

[0006] Obtain the video to be detected and the fake video detection model; wherein, the fake video detection model includes an identity comparison network, a spatial comparison network, a frame comparison network, and a feature fusion network;

[0007] The identity information consistency detection is performed on the video to be detected based on the identity comparison network to obtain identity information features;

[0008] Spatial inconsistency detection is performed on the video to be detected based on a spatial comparison network to obtain spatial inconsistency features;

[0009] The temporal inconsistency features of the video to be detected are extracted based on a frame comparison network.

[0010] A feature fusion network is used to fuse the identity information features, the spatial inconsistency features, and the temporal inconsistency features to obtain the fake video detection result.

[0011] Optionally, the identity comparison network includes an improved residual network; the step of performing identity information consistency detection on the video to be detected based on the identity comparison network to obtain identity information features includes:

[0012] The video to be detected is segmented into frames, and the embedding representation of each frame is extracted;

[0013] The improved residual network is used to process the embedding representation of each frame to extract identity information and obtain the identity information features.

[0014] Optionally, the fake video detection model further includes a multidimensional inconsistency detection network; the method further includes:

[0015] Based on the multidimensional inconsistency detection network, multidimensional features are extracted from the video to be detected, and inconsistency detection is performed on the multidimensional features to obtain preprocessed features.

[0016] Optionally, the spatial comparison network includes a first branch network, a second branch network, and a third branch network;

[0017] The spatial inconsistency detection of the video to be detected based on the spatial comparison network, to obtain spatial inconsistency features, includes:

[0018] The preprocessed features are processed using the first branch network to extract spatial features and obtain the first spatial features;

[0019] The preprocessed features are downsampled using a second branch network, and local features are extracted. The local features are then upsampled to obtain the second spatial features.

[0020] The third branch network outputs the preprocessed features.

[0021] The first spatial feature, the second spatial feature, and the preprocessed feature are fused to obtain the spatial inconsistency feature.

[0022] Optionally, the extraction of temporal inconsistency features of the video to be detected based on the frame comparison network includes:

[0023] The preprocessed features are compressed to obtain a compressed video vector;

[0024] Frame information for each frame is extracted from the compressed video vector;

[0025] Perform a difference operation on the frame information of each frame to obtain the corresponding temporal features;

[0026] Different convolution operations are used to extract local and global information from the temporal features to obtain the temporal inconsistency features.

[0027] Optionally, the feature fusion network includes a first fusion network; the step of fusing the identity information features, the spatial inconsistency features, and the temporal inconsistency features using the feature fusion network to obtain the forged video detection result includes:

[0028] The identity information feature and the spatial inconsistency feature are fused to obtain a first fused feature;

[0029] Based on the first fusion network, the first fusion feature is processed to obtain the processed spatial information;

[0030] The temporal inconsistency feature is fused with the preprocessed feature to obtain the fused feature, and the fused feature is fused with the processed spatial information to obtain the second fused feature;

[0031] The fake video detection result is obtained by splicing the first fusion feature and the second fusion feature.

[0032] Secondly, the present invention also provides a deepfake video detection device based on identity and spatiotemporal inconsistency, comprising the following modules:

[0033] The data acquisition module is used to acquire the video to be detected and the fake video detection model; wherein, the fake video detection model includes an identity comparison network, a spatial comparison network, a frame comparison network, and a feature fusion network;

[0034] The first detection module is used to perform identity information consistency detection on the video to be detected based on an identity comparison network to obtain identity information features.

[0035] The second detection module is used to perform spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain spatial inconsistency features.

[0036] The third detection module is used to extract the temporal inconsistency features of the video to be detected based on a frame comparison network.

[0037] The feature fusion module is used to fuse the identity information features, the spatial inconsistency features, and the temporal inconsistency features using a feature fusion network to obtain the fake video detection result.

[0038] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the deepfake video detection method based on identity and spatiotemporal inconsistency as described in the first aspect above.

[0039] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deepfake video detection method based on identity and spatiotemporal inconsistency as described in the first aspect above.

[0040] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the deepfake video detection method based on identity and spatiotemporal inconsistency as described in the first aspect above.

[0041] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows:

[0042] This invention provides a method and apparatus for detecting deepfake videos based on identity and spatiotemporal inconsistencies. Instead of relying on low-level image artifacts, it detects deepfake videos by analyzing identity information and temporal inconsistency features. This method can more effectively address the challenge of high-quality forged images because even with high image quality, forged identity information and temporal inconsistencies are still difficult to completely eliminate. The method uses an identity comparison network to perform identity information consistency detection on the video to be detected, accurately identifying whether the identity information in the video matches the real identity information. This helps solve the identity inconsistency problem in face-swapping technology and improves the accuracy of deepfake video detection. A frame comparison network extracts temporal inconsistency features from the video to be detected, and a feature fusion network fuses these features with identity information features and spatial inconsistency features. This helps capture temporal inconsistency features in the video, improving the accuracy of deepfake video detection.

[0043] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0044] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1This is a schematic diagram of a forged image during the face forgery stage provided by the present invention.

[0047] Figure 2 This is a flowchart illustrating the deepfake video detection method based on identity and spatiotemporal inconsistency provided by the present invention.

[0048] Figure 3 This is a schematic diagram of the structure of the fake video detection model provided by the present invention.

[0049] Figure 4 This is a schematic diagram illustrating the detection effect of the present invention on images processed by five different forgery operations.

[0050] Figure 5 This is a schematic diagram of the deepfake video detection device based on identity and spatiotemporal inconsistency provided by the present invention.

[0051] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] Some existing methods utilize low-level image artifacts for detection. For example, they leverage RGB information, auxiliary masks, and blended boundary information; artifacts are also used in Artificial Intelligence Generated Content Detection (AIGC). However, these methods are dependent on the artifacts and the dataset. As the quality of forged images continues to improve, methods relying on low-level textures that generate artifacts struggle to maintain their effectiveness. Figure 1 As shown, the forged image retains some target identity features outside the face-swapping region (gray outline), leading to inconsistencies between the source and target identities in deepfakes—a problem currently overlooked by face-swapping technologies. It is assumed that real faces have continuous ID information, while deepfake faces have mixed and discontinuous ID information. Although deepfake methods create realistic components, the video is synthesized by forging frames, resulting in temporal inconsistencies, particularly in facial regions between frames. Figure 1As shown. Existing intra-frame detection methods cannot capture these temporal features. T frame represents the T-th frame in the video, T+1 frame represents the T+1-th frame in the video, and zoom represents the zoom-in operation.

[0054] With the rapid development of deepfake technology, the need for detecting deepfake videos has become increasingly important. Current deepfake detection methods mainly rely on specific low-level texture cues present in deepfake videos. However, with continuous improvements in generation techniques, deepfakes are becoming increasingly realistic, and low-level artifacts are becoming less frequent, making detection more challenging. To address this challenge, this invention introduces an Identity Comparison Network (ICN), which utilizes high-level semantic information to identify deepfakes by analyzing inconsistencies in facial IDs. This ID information indicates individual identity characteristics and is then combined with spatial features extracted by a Spatial Comparison Network (SCN) to reveal spatial inconsistencies. The Spatial Comparison Network (SCN) extracts traditional image artifact information to improve the detection process. Furthermore, a Frames Comparison Network (FCN) is introduced to identify inconsistencies between frames in the video, leveraging the inherent weaknesses of frame-by-frame forgery. Experimental evaluations on the FF++ and Celeb-DF datasets demonstrate the effectiveness and generalization ability of the proposed method in deepfake video detection.

[0055] The following is combined Figures 2-5 The present invention describes a method and apparatus for detecting deepfake videos based on identity and spatiotemporal inconsistencies.

[0056] Reference Figure 2 As shown, this deepfake video detection method based on identity and spatiotemporal inconsistency includes the following:

[0057] Step S110: Obtain the video to be detected and the fake video detection model.

[0058] The video to be detected is a user-provided video that requires authenticity verification. Users submit the video to the system by uploading, entering a URL, or other methods. The system receives and stores the video, preparing it for subsequent detection processes. The system loads a pre-trained fake video detection model. (Refer to...) Figure 3As shown, the model consists of four main parts: an Identity Comparison Network (ICN), a Spatial Comparison Network (SCN), a Frames Comparison Network (FCN), and a Feature Fusion Network. These network models are implemented using deep learning frameworks (such as TensorFlow, PyTorch, etc.) and stored in model files.

[0059] Step S120: Perform identity information consistency detection on the video to be detected based on the identity comparison network to obtain identity information features.

[0060] In forged video frames, the facial region contains discontinuous areas of identity information, while a genuine face contains only a single, continuous piece of identity information. Because of the difference in identity information between genuine and forged facial images, this invention detects the authenticity of faces based on identity information. An identity comparison network is used to analyze the identity information in the input video, extracting facial features of the individuals in the video to detect the consistency of the overall facial identity information, thus obtaining identity information features.

[0061] Specifically, the identity comparison network first performs face detection on each frame of the video to be tested, extracting facial regions. Deep learning algorithms (such as convolutional neural networks) are then used to extract features from the extracted facial regions, obtaining facial feature vectors. These extracted facial feature vectors are input into the identity comparison network for identity information consistency analysis. The identity comparison network compares the facial feature vectors between different frames to determine if there are any inconsistencies in identity information. If inconsistencies exist, it indicates that the video may have been forged or tampered with. The identity comparison network outputs identity information features, which reflect the degree of consistency of facial identity information of the people in the video.

[0062] Step S130: Perform spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain spatial inconsistency features.

[0063] Facial images may contain noticeable artifacts after forgery; therefore, this invention designs a spatial comparison network to extract spatial inconsistencies. Based on the designed spatial comparison network, the video to be tested is analyzed frame by frame to detect static image features in the video frames, such as facial landmarks, skin texture, and lighting inconsistencies. Convolutional neural networks and other algorithms are used to further process and analyze the extracted static image features. The spatial comparison network compares the static image features between different frames to determine whether there are spatial inconsistencies, such as abnormal facial landmark positions or inconsistent skin texture. The spatial comparison network outputs spatial inconsistency features, which reflect the degree of consistency of spatial information in the video.

[0064] Step S140: Extract the temporal inconsistency features of the video to be detected based on the frame comparison network.

[0065] A frame comparison network is used to extract temporal features from the video under test, analyzing the temporal relationships and dynamic changes between video frames, such as eye blinks, lip synchronization, and head movements. The frame comparison network captures temporal information and determines whether there are inconsistencies in time by comparing the temporal relationships between different frames. The frame comparison network outputs temporal inconsistency features, which reflect the degree of consistency of temporal information in the video.

[0066] Step S150: Use a feature fusion network to fuse the identity information features, the spatial inconsistency features, and the temporal inconsistency features to obtain the fake video detection result.

[0067] Identity information features, spatial inconsistency features, and temporal inconsistency features are input into a feature fusion network. The feature fusion network further processes and analyzes these features through different branches, extracting higher-level feature representations. The extracted high-level feature representations are then input into a classifier (e.g., a fully connected layer + Softmax) for classification. Based on the classifier's output, it determines whether the video to be detected is a fake video (Real or Fake). The fake video detection result is output, including whether the video is fake and the corresponding confidence score or probability value. (See reference...) Figure 3 As shown, the classification layer can represent a fully connected (FC) layer.

[0068] This invention provides a deepfake video detection method based on identity and spatiotemporal inconsistencies. Instead of relying on low-level image artifacts, it detects deepfake videos by analyzing identity information and temporal inconsistency features. This method can more effectively address the challenge of high-quality forged images because even with high image quality, forged identity information and temporal inconsistencies are still difficult to completely eliminate. The method uses an identity comparison network to perform identity information consistency detection on the video to be detected, accurately identifying whether the identity information in the video matches the real identity information. This helps solve the identity inconsistency problem in face-swapping technology and improves the accuracy of deepfake video detection. A frame comparison network extracts temporal inconsistency features from the video to be detected, and a feature fusion network fuses these features with identity information features and spatial inconsistency features. This helps capture temporal inconsistency features in the video, improving the accuracy of deepfake video detection. By combining identity comparison, spatial inconsistency detection, and temporal inconsistency detection, the authenticity of the video can be comprehensively analyzed. Identity information consistency detection ensures that the identity information in the video has not been tampered with; spatial inconsistency detection can detect forgery traces in video frames; and temporal inconsistency detection can capture temporal anomalies between video frames. This method can cope with various forgery techniques and new forgery technologies. By continuously updating and optimizing the forged video detection model, the model's generalization ability and adaptability can be improved.

[0069] Existing technologies such as Convolutional 3D (C3D) and Inflated 3D Convolutional Networks (I3D) are used for deepfake detection, but they are computationally intensive and not optimized for this task, resulting in poor performance. This invention provides a deepfake video detection method based on identity and spatiotemporal inconsistencies. By constructing an efficient forgery video detection model, including identity comparison networks, spatial comparison networks, frame comparison networks, and multi-branch networks, it achieves effective detection of deepfake videos. These network structures are optimized to reduce computational load and improve performance, thus offering greater practicality and efficiency in real-world applications.

[0070] In an optional embodiment, iResNet (an improved residual network) is widely used in face recognition tasks due to its powerful feature extraction capabilities. Especially in deepfake detection tasks, iResNet can effectively extract facial identity information from real and fake images. The aforementioned identity comparison network includes an improved residual network; the step S120 above, which involves performing identity information consistency detection on the video to be detected based on the identity comparison network to obtain identity information features, includes:

[0071] S1201. The video to be detected is segmented into frames, and the embedding representation of each frame is extracted.

[0072] Before performing detection on the video to be detected, a video cropping operation can be performed. This involves selecting a portion of the video by specifying a rectangular region, while removing unnecessary areas and redundant information, retaining only the parts of interest. The video to be detected consists of a series of consecutive images (frames). Next, the video is split, extracting each frame. For each segmented frame, an embedding method (such as a deep learning model) is used to extract its embedding representation. This involves converting the image into a fixed-length vector that reflects its key features and information, resulting in the embedding representation of each frame. .

[0073] S1202. The improved residual network is used to process the embedding representation of each frame to extract identity information and obtain the identity information features.

[0074] An improved residual network is used to process the embedded representation of each frame. Residual networks (ResNet) are deep convolutional neural networks that address the vanishing and exploding gradient problems in deep neural networks by introducing residual connections (i.e., skip connections). Improved residual networks can include optimizing the network structure, adding attention mechanisms, and using more advanced activation functions to improve model performance and accuracy. In this process, the residual network extracts identity information features from each frame, including facial features, expressions, and poses, which are crucial for determining the authenticity of the video. After processing with the improved residual network, the identity information features of each frame are obtained. These features are then fused or averaged to obtain the identity information features for the entire video. This identity information feature vector will be used for subsequent judgment and classification tasks.

[0075] Specifically, refer to Figure 3 As shown, the video to be detected is segmented, and each frame is extracted. Next, an embedding representation is extracted from each frame, which converts the image into a fixed-length vector that reflects its key features and information. The extracted embedding representation of each frame (…) ,in Indicates the first in the video Each frame is input into the iResNet residual network. The iResNet residual network processes an input of size H × W × C (H is the height, W is the width, and C is the number of channels) and outputs the embedding of each frame, i.e., the identity information features. (Right now Figure 3(ID embedding in the image). These identity information features contain key facial features and information, such as the shape and position of facial contours, eyes, nose, mouth, etc.

[0076] (1)

[0077] in, This represents the operation of extracting identity features, using IResNet to extract facial information.

[0078] In forged video frames, due to limitations of deepfake technology, the facial region may contain discontinuous areas of identity information. A genuine face, however, contains only a single, continuous piece of identity information. This invention detects the authenticity of a face by comparing the consistency of the entire facial ID information. iResNet can receive the entire facial ID information as input and output a score or label representing the consistency of the identity information.

[0079] After iResNet extracts identity information features, the sigmoid function can be applied to perform binary classification, which maps the extracted features to the probability interval (0,1) to determine whether the input identity information belongs to a specific category.

[0080] Compared to traditional methods, this invention directly extracts complete facial ID information, rather than segmenting the face region first and then extracting the ID information of each part separately. This avoids the ID information separation problem caused by inaccurate or inconsistent face region segmentation. Using iResNet enables more accurate extraction of identity information features, thereby improving the accuracy of video authenticity detection. By fusing identity information features from multiple frames, the impact of noise and errors can be further reduced, improving the overall detection stability. The residual connections in the iResNet residual network help solve the gradient vanishing and gradient exploding problems in deep neural networks, thus enhancing the model's robustness. By comparing the consistency of the entire facial ID information, deepfake videos can be detected more effectively. This method avoids the problems of face region segmentation and ID information separation, further improving detection stability. Simultaneously, by directly extracting complete facial ID information, unnecessary computational steps and resource consumption are reduced.

[0081] This invention proposes a multi-branch architecture specifically designed for deepfake detection, treating it as a binary classification problem based on spatial and temporal inconsistency. The method integrates a Multi-dimensional Inconsistency Detection (MID) block into the ResNet-50 framework, such as... Figure 3As shown, ICN extracts identity features from a single video frame, while SCN captures additional spatial forgery. Features from both ICN and SCN are combined. Furthermore, the FCN branch is designed to detect temporal inconsistencies caused by operations on consecutive video frames. These inconsistencies are identified by first compressing video features and then applying differentiation to the compressed features, thus revealing unique features that highlight the temporal differences typical of deepfakes. This combination of spatiotemporal feature extraction across network branches gives the model of this invention a powerful ability to identify deepfake artifacts, providing a detailed and comprehensive method for deepfake detection.

[0082] In an optional embodiment, the forged video detection model further includes a multidimensional inconsistency detection network. By introducing the multidimensional inconsistency detection network, multidimensional features are extracted from the video to be detected, and inconsistency detection is performed on these features, thereby providing preprocessed features for subsequent spatial inconsistency detection. The method further includes:

[0083] S210. Based on the multidimensional inconsistency detection network, multidimensional features are extracted from the video to be detected, and inconsistency detection is performed on the multidimensional features to obtain preprocessed features.

[0084] Multidimensional inconsistency detection networks extract multidimensional features, aiming to extract various features from the video to be detected that can comprehensively describe the video's content and attributes. These features include color, texture, shape, motion information, etc., which together constitute the video's multidimensional feature space. Feature extraction can employ deep learning techniques such as convolutional neural networks (CNNs).

[0085] After extracting multidimensional features, inconsistency detection is required. The purpose of inconsistency detection is to identify feature patterns that are infrequent in real videos but may frequently appear in fake videos. Following inconsistency detection, a series of preprocessed features are obtained. These features have been filtered and optimized, making them more likely to contain key information related to fake videos. They will be used as input for subsequent spatial inconsistency detection to further improve the accuracy and efficiency of the detection.

[0086] The aforementioned multidimensional inconsistency detection network typically follows a series of ordered and efficient steps when processing videos to obtain preprocessed features. These steps combine the features of convolutional neural networks (Conv) and multidimensional inconsistency detection (MID) blocks to extract key information from the video. The following is a detailed description of this processing procedure:

[0087] Reference Figure 3As shown, a multi-dimensional inconsistency detection network can include convolutional (Conv) layers and two or more multi-dimensional inconsistency detection (MID) blocks. The Conv layers, which can be one or more, are responsible for extracting basic spatiotemporal features from the raw video data (i.e., the video to be detected mentioned above). The MID blocks are used to further analyze and process the features extracted from the Conv layers to detect multi-dimensional inconsistencies in the video.

[0088] The video to be detected is input into a multidimensional inconsistency detection network. The video to be detected is a video sequence containing multiple frames, each of which is a two-dimensional image. The video data first passes through Conv layers. Conv layers use a series of convolutional kernels to slide across the video frames to extract local features. These features can be color, texture, edges, etc. After the convolution operation, an activation function (such as ReLU) is applied to increase the non-linearity of the network, and pooling layers (such as max pooling) are applied to reduce the resolution of the features and reduce computation. Through the stacking of multiple Conv layers, the network can extract higher-level features that contain spatiotemporal information in the video. The features output from the Conv layers are fed into MID blocks. Inside the MID blocks, the features undergo a series of processing steps, such as feature transformation, attention mechanisms, residual connections, etc., to extract features related to inconsistency. Through the stacking of multiple MID blocks, the network can progressively and deeply detect multidimensional inconsistencies in the video and extract more refined features. After processing by the Conv layers and MID blocks, the network outputs preprocessed features. These features contain key information from the video, such as the motion trajectories of objects, changes in the scene, and features related to multidimensional inconsistency.

[0089] The multidimensional inconsistency detection network, through the collaborative work of the Conv layer and the MID block, achieves effective processing of the video to be detected. The network can more accurately extract key information from the video, detect multidimensional inconsistencies, and extract preprocessed features. These features provide strong support for subsequent video analysis tasks.

[0090] This invention, by extracting multidimensional features and performing inconsistency detection, can more comprehensively capture anomalous information in forged videos. This helps reduce false positives and false negatives, improving detection accuracy. Multidimensional feature extraction enables the model to handle more diverse video data. This helps the model maintain high detection performance when facing new forgery techniques or unknown video data. The preprocessed features have been filtered and optimized, reducing redundant information. This allows subsequent spatial inconsistency detection to be performed faster, thereby improving overall detection efficiency. The preprocessed features provide strong support for subsequent spatial inconsistency detection. They can serve as important clues and evidence to help the model more accurately identify forged videos.

[0091] In an optional embodiment, a spatial comparison network is a key component in the fake video detection task, responsible for extracting spatial inconsistency features from preprocessed features. The spatial comparison network introduces multiple branches to capture spatial inconsistencies. These branches perform convolutional operations in different receptive fields to capture features at different scales. The network includes a first branch network SCN1, a second branch network SCN2, and a third branch network SCN3, which work together to achieve spatial inconsistency detection of the video. Step S130 above, which describes performing spatial inconsistency detection on the video to be detected based on the spatial comparison network to obtain spatial inconsistency features, includes:

[0092] S1301. The preprocessed features are processed using the first branch network to extract spatial features and obtain the first spatial features.

[0093] The first branch network can be a convolutional neural network (CNN) or a similar deep learning model, which performs convolution operations on the input features to extract spatial features, called first spatial features. These features reflect the spatial structure and texture information in the video.

[0094] S1302. The preprocessed features are downsampled using the second branch network, and local features are extracted. The local features are then upsampled to obtain the second spatial features.

[0095] The input to the second branch network is also the preprocessed features. The second branch network first downsamples the input features to reduce their resolution and extract local features. Downsampling can be achieved through max pooling, average pooling, or convolution operations. Based on the downsampled features, the network further extracts local features, which are typically related to video details and texture. To maintain spatial resolution consistent with the features output by the first branch network, the second branch network upsamples these local features. Upsampling can be achieved through bilinear interpolation, nearest neighbor interpolation, or transposed convolution. The final output is a second spatial feature that combines local details and global structural information.

[0096] S1303, the third branch network outputs the preprocessed features.

[0097] The third branch network directly outputs the input features without any additional processing or transformation. This preserves all the information in the original features, providing complete input for subsequent feature fusion.

[0098] S1304. The first spatial feature, the second spatial feature, and the preprocessed feature are fused to obtain the spatial inconsistency feature.

[0099] The first spatial feature, the second spatial feature, and the preprocessed feature are fused. Fusion can be achieved through simple concatenation, addition, or more complex attention mechanisms. The fused features are called spatial inconsistency features, which contain spatial inconsistencies in the video and can be used for subsequent classification or detection tasks.

[0100] Specifically, refer to Figure 3 As shown, the first branch network, scn1, uses 3×3 convolutions to extract relatively broad spatial features, namely the first spatial features mentioned above. The second branch network, scn2, performs downsampling on the input using average pooling, and then uses 3×1 and 1×3 convolutions to extract local detailed features. These two convolution operations can extract feature details along the horizontal and vertical directions, respectively, and multi-directional convolution operations are more helpful in identifying potential forgery traces. Then, interpolation upsampling is used to restore the original size, obtaining the second spatial features mentioned above. The third branch network, scn3, uses the original feature vector (i.e., the preprocessed features mentioned above) as the output. It retains the original, unchanged information and captures subtle details and nuances that may be lost during feature transformation. This helps to enhance the model's ability to learn complex patterns and improve overall performance. Finally, the outputs of the three branches are fused using the following formula:

[0101] (2)

[0102] in, It provides a comprehensive representation of spatial inconsistency features, including global and local spatial inconsistency information. The first spatial feature is the output of the first branch network scn1. The second spatial features are the output of the second branch network scn2. This is the output of the third branch network scn3. For the first Preprocessed features corresponding to frame videos It belongs to the feature set X (T×C×H×W), where T, C, H, and W represent the number of frames, the number of image channels, the image height, and the image width, respectively. It is an sigmoid function, such as... Figure 3 The sigmoid function in [the context of the text]. Indicates the number of frames in the video.

[0103] This invention extracts global and local spatial features from videos through the collaborative work of a first branch network and a second branch network. These features collectively form the basis of spatial inconsistency features, helping the model to more accurately identify abnormal regions in fake videos. The downsampling and upsampling operations in the second branch network enable the model to handle video data of different resolutions. This helps the model maintain stable detection performance when faced with videos of varying quality or size. The third branch network directly outputs the preprocessed features, avoiding unnecessary redundant computation. Simultaneously, the feature fusion process can be performed in parallel, thereby improving overall detection efficiency. By fusing features output from different branch networks, feature diversity can be increased. This helps the model better capture complex information in videos, improving detection accuracy and generalization ability.

[0104] In an optional embodiment, FCN employs a clever one-dimensional CNN architecture, providing an alternative to traditional methods. By compressing the video to reduce dimensionality and segmenting it along the temporal dimension, it captures differences between adjacent frames, thus obtaining temporal inconsistency information. By effectively utilizing the temporal dimension, the model significantly reduces the number of parameters without affecting classification accuracy. By integrating spatial and temporal inconsistency information, the model can perform binary classification to distinguish between genuine and fake videos. The proposed method aims to detect deepfake videos by exploiting inherent vulnerabilities in forgery methods and utilizing high-level sequence information from both intra-frame and inter-frame perspectives. This invention proposes a novel method that integrates the above information to detect intra-frame and inter-frame forgery. This method utilizes identity and temporal inconsistencies in deepfake videos, employing comprehensive spatial and temporal features for detection. This provides a new and effective model for deepfake detection. Incorporating facial information as high-level semantic information into spatial inconsistency detection enhances the generalization ability and interpretability of deepfake detection. FCN compresses the video to reduce dimensionality and segments it along the temporal dimension, effectively capturing inter-frame differences to obtain temporal inconsistency information. Compared to traditional 2D and 3D CNNs, it reduces the number of network parameters. The step S140 above, which involves extracting the temporal inconsistency features of the video to be detected based on a frame comparison network, includes:

[0105] S1401. The preprocessed features are compressed to obtain a compressed video vector.

[0106] The preprocessed features are compressed to obtain a compressed video vector. This step aims to reduce redundant information in the video data and improve the efficiency of subsequent processing. Compression methods can include quantization, encoding, and feature dimensionality reduction techniques, which can effectively reduce the size of video data while preserving as much key information as possible.

[0107] S1402. Extract frame information for each frame from the compressed video vector.

[0108] Frame information is extracted from the compressed video vector for each frame. This step is fundamental to video processing, enabling subsequent analysis and comparison to be performed frame by frame. Frame information may include pixel values, color distribution, texture features, and other information crucial for identifying objects, actions, and scenes in the video.

[0109] S1403. Perform differential operation on the frame information of each frame to obtain the corresponding temporal features.

[0110] Differential operations are performed on the frame information of each frame to obtain the corresponding temporal features. Differential operations can capture motion information in the video by comparing the differences between adjacent frames. In this process, the changes between adjacent frames are quantified into difference values, which constitute the temporal features, reflecting the changes in the video over time.

[0111] S1404. Use different convolution operations to extract local and global information from the temporal features to obtain the temporal inconsistency features.

[0112] Different convolutional operations are used to extract local and global information from temporal features. Convolutional operations capture local features in the input data by sliding the convolutional kernel. In this process, different kernel sizes and strides are used to extract features at different scales. Smaller kernels are better at capturing detailed information (local information), while larger kernels are better at capturing the overall structure (global information). By combining this local and global information, more comprehensive and accurate temporal inconsistency features can be obtained.

[0113] Specifically, refer to Figure 3 As shown, the frame comparison network first compresses the input feature set X to obtain the compressed embedding, i.e., the compressed video vector mentioned above. The frame comparison network reduces the dimension of X and compresses a video vector of size C × T × H × W along both the horizontal and vertical directions, resulting in a compressed video vector of size HW × C × T. Compression embedding along both horizontal and vertical directions enables the model to effectively capture spatial artifacts in video frames. An iterative method is then used to obtain the compressed video vector. Frame information in the inner temporal dimension. Obtain the difference information between each frame and subsequent frames. In this way, the frame comparison network can effectively detect temporal inconsistencies in both the horizontal and vertical directions. Specifically, it... Perform frame processing to obtain frame information for each frame. By analyzing the frame information of each frame. Differential operations are performed to obtain the differential embedding of each frame, i.e., the aforementioned temporal features. . Used to extract features through multi-branch networks.

[0114] (3)

[0115] in, The final timing characteristics are represented as HW × C. This represents a one-dimensional convolution with a kernel size of 3 and padding of 1, used to effectively extract information from adjacent frames. From compressed video vectors Video frames obtained from Frame information, It is a video frame Frame information.

[0116] Next, a multi-branch method is used to process the temporal features. This invention employs a three-branch network structure to process... Different convolution operations are used to effectively extract local and global information from feature maps:

[0117] (4)

[0118] (5)

[0119] in, Branch returns unprocessed timing features This helps preserve the original information of the image. (Regarding temporal features) , Pooling downsampling is used, followed by convolution with a kernel size of 3, and finally upsampling is performed using interpolation. This increases the receptive field of the neural network, enabling it to receive a wider range of contextual information. Using convolutions with a kernel size of 3 to analyze temporal features Process it. , , The outputs of the three branches are fused to obtain the fused features. Features after fusion via conv1 and The function processes the data, and the final output is represented as follows: , that is Figure 3 Temporal inconsistency features , This indicates the number of video frames. conv1 is a convolution operation with a kernel size of 1. It is an sigmoid function, such as... Figure 3 The sigmoid function in [the context of the text].

[0120] This invention, by extracting temporal inconsistency features, can more accurately identify forged or tampered portions of videos. This is of great significance for combating video forgery techniques such as deepfakes. Temporal inconsistency features can capture changes in a video over time, and these changes are often closely related to forgery or tampering. Therefore, utilizing these features for video detection can significantly improve detection accuracy.

[0121] In an optional embodiment, the feature fusion network includes a first fusion network; step S150 above, which involves using the feature fusion network to fuse the identity information features, the spatial inconsistency features, and the temporal inconsistency features to obtain a forged video detection result, includes:

[0122] S1501. The identity information feature and the spatial inconsistency feature are fused to obtain the first fused feature.

[0123] The first fused feature is obtained by fusing identity information features and spatial inconsistency features. Identity information features refer to the identity information of people in the video, such as facial features, while spatial inconsistency features refer to the inconsistencies in spatial attributes such as the position, shape, and size of objects or people within video frames. (Refer to...) Figure 3 As shown, by fusing these two features, we can comprehensively consider the identity information and spatial attributes of the video, providing more comprehensive information support for subsequent fake video detection.

[0124] (6)

[0125] in, The first fusion feature represents a video feature that integrates identity information and spatial inconsistency information. For the first Identity information features of frame video, This refers to the spatial inconsistency features output by the spatial comparison network.

[0126] S1502. Based on the first fusion network, the first fusion feature is processed to obtain the processed spatial information.

[0127] The first fusion network processes the first fusion feature to obtain processed spatial information. The first fusion network can be a deep learning model, such as a convolutional neural network (CNN), which can further analyze and extract the first fusion feature to obtain more accurate and useful spatial information.

[0128] (7)

[0129] in, For the processed spatial information, It consists of a 3×3 convolution and a 1×3 convolution followed by an sigmoid function (e.g.) Figure 3 A branching processing structure composed of sigmoid. Its purpose is to enhance the fusion of two types of spatial information, ultimately obtaining processed spatial information. .

[0130] S1503. The time inconsistency feature is fused with the preprocessed feature to obtain the fused feature, and the fused feature is fused with the processed spatial information to obtain the second fused feature.

[0131] The temporal inconsistency features are fused with preprocessed features to obtain the fused features. Temporal inconsistency features refer to the inconsistencies in the time series of video frames, such as changes in frame rate and discontinuities in motion trajectories. Preprocessed features are those obtained after preprocessing the original video data, such as denoising and enhancement. By fusing these two types of features, the temporal attributes of the video and the preprocessed feature information can be comprehensively considered, providing richer information for subsequent fake video detection.

[0132] The fused features are then fused with the processed spatial information to obtain the second fused feature. This step involves fusing the two previously obtained features again to comprehensively consider the spatial, temporal, and preprocessed feature information of the video.

[0133] (8)

[0134] in, This is the second fusion feature. To extract video temporal inconsistency information using a frame comparison network, i.e., the aforementioned temporal inconsistency features, For the preprocessed features, This refers to the processed spatial information.

[0135] To ensure compatibility in subsequent operations, and processed spatial information All were reconstructed to match The dimension. In and Element-wise multiplication is performed between them to emphasize temporal changes within the spatial context of each frame. Furthermore, This, combined with the result of the multiplication, forms the second fusion feature. It integrates spatial and temporal cues.

[0136] S1504. The first fusion feature and the second fusion feature are spliced ​​together to obtain the fake video detection result.

[0137] The first and second fused features are concatenated to obtain the fake video detection result. The concatenation operation can be a concatenation, fusion, or weighting of feature vectors to obtain the final detection result.

[0138] (9)

[0139] in, To forge video detection features, This is for splicing operations. The input is fed into the MID module below, and after multiple iterations, the last layer of the MID module... The input is fed into the FC layer to get either a real or fake result.

[0140] This invention improves the accuracy of forged video detection by fusing identity information features, spatial inconsistency features, and temporal inconsistency features, comprehensively considering multiple information elements within the video. Different features reflect the authenticity and integrity of a video from different perspectives; therefore, fusing them yields more comprehensive and accurate detection results. The feature fusion network can handle various feature inputs and effectively integrate and extract these features. This allows the model to maintain stable performance when facing different types of forged videos, enhancing its robustness. The feature fusion network can adapt to different types of video data, including videos of varying resolutions, formats, and scenes. This makes the method widely applicable in practical applications and suitable for various forged video detection scenarios.

[0141] This invention utilizes two datasets: Facebook fake detection (FaceForensics++, FF++) and celebrity deepfakes (Celeb-DF). The FF++ dataset is a popular deepfake detection dataset consisting of three classes of videos: raw (original quality), c23 (low-compressed), and c40 (high-compressed). This dataset employs four methods to create fake videos: deepfakes (DF), face-to-face (F2F), face-swapping (FS), and neural textures (NT). Later, the face shifter (FST) method was also added to the pool for generating fake videos. The Celeb-DF dataset contains 5639 high-quality fake videos and 590 real videos. These processed videos were generated using an improved synthesis method.

[0142] The MID block was integrated into the ResNet-50 network, replacing the original bottleneck block, with an input image size of 224 × 224 pixels. The aforementioned fake video detection model was trained using the Adam optimizer and binary cross-entropy loss. The model was trained for 30 batch epochs with a batch size of 16. The initial learning rate was set to 0.0002, decaying by a factor of 10 every 10 epochs. Training and testing were performed on the FF++c40 training and test sets, respectively, with cross-dataset testing on the Celeb-DF test set. A Multi-task Convolutional Neural Network (MTCNN) was used to detect facial regions and input them into the model. During training, 8 consecutive video frames were extracted from each video, and during testing, 16 frames were extracted. The area under the receiver operating characteristic curve (AUC) and accuracy (ACC) were used as evaluation metrics. As shown in Table 1, the results of different models (including existing C3D, I3D, Rahmouni, SPSL, MesoNet, Xception, STIL models) and the method of this invention (Ours) on the FF++ dataset are presented. The best performance is indicated in bold, and * indicates the reproduction of the method.

[0143] Table 1

[0144]

[0145] As shown in Table 1, the deepfake video detection method based on identity and spatiotemporal inconsistency provided by this invention achieves good performance on all five methods (DF, F2F, FS, NT, FST) in FF++. Furthermore, comparisons with other methods show that this invention outperforms most existing methods and represents a significant improvement. SPSL is a method specifically designed for detection using low-level textures. The superior performance of this invention compared to SPSL demonstrates the strong generalization ability of identity features (high-level semantic information). Low-level texture features are dependent on image quality. Detection methods utilizing low-level texture features often perform relatively poorly. Moreover, this invention integrates temporal information from the video and detects forgery cues in the temporal dimension. Compared to traditional video methods using 3D CNN networks (such as C3D and I3D), this invention purposefully designs a 1D CNN network in the temporal branch, achieving better results than traditional video classification methods. This invention surpasses STIL, proving the effectiveness of the spatial inconsistency information with ID cues and the designed temporal inconsistency information in this invention.

[0146] The various existing methods listed in Table 2 (including Multi-task, Capsule, Two-branch, MesoInception4, Xception, SPSL, F3_Net, M2TR, STIL) and the method of this invention (Ours) were trained on the FF++ dataset and tested on both the FF++ and Celeb-DF datasets. The final results are reported as AUC (%), and the best performance is indicated in bold.

[0147] Table 2

[0148]

[0149] To verify the generalization ability of the method of this invention, a comparative analysis was conducted on various deepfake detection methods on different datasets. As shown in Table 2, Xception is a detection method based on single-frame analysis, while STIL focuses on extracting spatiotemporal inconsistencies. The comparison showed that the method of this invention achieved the best results, indicating that the integration of identity information and the understanding of spatiotemporal inconsistencies can lead to better generalization ability.

[0150] To evaluate the effectiveness of the method of this invention, ablation experiments were conducted on the FF++ dataset to assess the effectiveness of the three components: Identity Comparison Network (ICN), Spatial Comparison Network (SCN), and Frame Comparison Network (FCN). The components (variants) were evaluated according to the experimental scheme in Table 3, and the results are expressed as AUC (%). The experimental results are shown in Table 3. Clearly, the most effective method is to combine these three components, demonstrating the necessity of combining spatiotemporal and identity information. Experiments were also conducted on the temporal extraction method. Specifically, the FCN (STIL) + ICN + SCN method was adopted, transferring the temporal inconsistency extraction method used in STIL to FCN for experimentation. Experimental results show that this method is a significant improvement over the original method. Experiments demonstrate that the absence of any ID clue, spatial information, or temporal information leads to performance degradation, further confirming the rationality of the design scheme of this invention. In Table 3, w / o indicates no inclusion, w / o ICN indicates no inclusion of the identity comparison network, w / o SCN indicates no inclusion of the spatial comparison network, and w / o FCN indicates no inclusion of the frame comparison network.

[0151] Table 3

[0152]

[0153] Table 4 shows a comparison of the parameters of the method of this invention with several video classification models. Compared with traditional detection methods based on 3D CNN and 2D CNN, the method of this invention has the fewest parameters and the best performance.

[0154] Table 4

[0155]

[0156] Reference Figure 4 As shown, the decision-making process of the method of the present invention is also visualized using the gradient-weighted class activation mapping (Grad-CAM) method. Figure 4 The first row in the image consists of forged images, and the second row displays the corresponding CAM images. It can be observed that the method of this invention mainly focuses on regions where two identities are connected in the face replacement methods DF and FS, such as the eyes, eyebrows, and forehead. This confirms that the model of this invention utilizes merged ID information to identify forged images based on discontinuous ID regions and locates forged regions in other methods. This verifies that the method has achieved its intended design goals, and the judgment based on discontinuous ID regions has high interpretability.

[0157] This invention utilizes identity information as high-level semantic information, enhancing interpretability and generalization ability. By supplementing it with other spatial information, it achieves good results. Furthermore, a customized 1D CNN network is employed to utilize the temporal differences between adjacent frames in forged videos for detection. Compared to traditional video classification models, this method reduces the number of parameters, and experimental results are satisfactory. Extensive experiments demonstrate the effectiveness of this method by detecting cues from multiple dimensions.

[0158] The following describes the deepfake video detection device based on identity and spatiotemporal inconsistency provided by the present invention. The deepfake video detection device based on identity and spatiotemporal inconsistency described below can be referred to in correspondence with the deepfake video detection method based on identity and spatiotemporal inconsistency described above.

[0159] The deepfake video detection device based on identity and spatiotemporal inconsistency provided by this invention refers to... Figure 5 As shown, it includes:

[0160] The data acquisition module 310 is used to acquire the video to be detected and the fake video detection model; wherein, the fake video detection model includes an identity comparison network, a spatial comparison network, a frame comparison network and a feature fusion network;

[0161] The first detection module 320 is used to perform identity information consistency detection on the video to be detected based on an identity comparison network to obtain identity information features.

[0162] The second detection module 330 is used to perform spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain spatial inconsistency features.

[0163] The third detection module 340 is used to extract the temporal inconsistency features of the video to be detected based on a frame comparison network.

[0164] The feature fusion module 350 is used to fuse the identity information features, the spatial inconsistency features, and the temporal inconsistency features using a feature fusion network to obtain the fake video detection result.

[0165] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can invoke logical instructions from the memory 430 to execute a deepfake video detection method based on identity and spatiotemporal inconsistencies.

[0166] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the deepfake video detection method based on identity and spatiotemporal inconsistency provided by the above methods.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the deepfake video detection method based on identity and spatiotemporal inconsistency provided by the methods described above.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting deepfake videos based on identity and spatiotemporal inconsistency, characterized in that, include: The process involves acquiring a video to be detected and a fake video detection model. The fake video detection model includes an identity comparison network, a multidimensional inconsistency detection network, a spatial comparison network, a frame comparison network, and a feature fusion network. The multidimensional inconsistency detection network performs multidimensional feature extraction, aiming to extract multiple features from the video to be detected that comprehensively describe the video content and attributes. The spatial comparison network includes a first branch network, a second branch network, and a third branch network. The identity information consistency detection is performed on the video to be detected based on the identity comparison network to obtain identity information features; Based on the multidimensional inconsistency detection network, multidimensional features are extracted from the video to be detected, and inconsistency detection is performed on the multidimensional features to obtain preprocessed features; The preprocessed features are processed using the first branch network to extract spatial features and obtain the first spatial features; The preprocessed features are downsampled using a second branch network, and local features are extracted. The local features are then upsampled to obtain the second spatial features. The third branch network outputs the preprocessed features. The first spatial feature, the second spatial feature, and the preprocessed feature are fused to obtain the spatial inconsistency feature; The temporal inconsistency features of the video to be detected are extracted based on a frame comparison network. A feature fusion network is used to fuse the identity information features, the spatial inconsistency features, and the temporal inconsistency features to obtain the fake video detection result.

2. The deepfake video detection method based on identity and spatiotemporal inconsistency according to claim 1, characterized in that, The identity comparison network includes an improved residual network; The identity information consistency detection of the video to be detected based on the identity comparison network is used to obtain identity information features, including: The video to be detected is segmented into frames, and the embedding representation of each frame is extracted; The improved residual network is used to process the embedding representation of each frame to extract identity information and obtain the identity information features.

3. The deepfake video detection method based on identity and spatiotemporal inconsistency according to claim 1, characterized in that, The extraction of temporal inconsistency features from the video to be detected based on the frame comparison network includes: The preprocessed features are compressed to obtain a compressed video vector; Frame information for each frame is extracted from the compressed video vector; Perform a difference operation on the frame information of each frame to obtain the corresponding temporal features; Different convolution operations are used to extract local and global information from the temporal features to obtain the temporal inconsistency features.

4. The deepfake video detection method based on identity and spatiotemporal inconsistency according to claim 1, characterized in that, The feature fusion network includes a first fusion network; the step of fusing the identity information features, the spatial inconsistency features, and the temporal inconsistency features using the feature fusion network to obtain the forged video detection result includes: The identity information feature and the spatial inconsistency feature are fused to obtain a first fused feature; Based on the first fusion network, the first fusion feature is processed to obtain the processed spatial information; The temporal inconsistency feature is fused with the preprocessed feature to obtain the fused feature, and the fused feature is fused with the processed spatial information to obtain the second fused feature; The fake video detection result is obtained by splicing the first fusion feature and the second fusion feature.

5. A deepfake video detection device based on identity and spatiotemporal inconsistency, characterized in that, include: The data acquisition module is used to acquire the video to be detected and the fake video detection model; wherein, the fake video detection model includes an identity comparison network, a multidimensional inconsistency detection network, a spatial comparison network, a frame comparison network, and a feature fusion network; the multidimensional inconsistency detection network performs multidimensional feature extraction, with the goal of extracting multiple features from the video to be detected that can comprehensively describe the video content and attributes; the spatial comparison network includes a first branch network, a second branch network, and a third branch network; The first detection module is used to perform identity information consistency detection on the video to be detected based on an identity comparison network to obtain identity information features. The second detection module is used to extract multidimensional features from the video to be detected based on the multidimensional inconsistency detection network, and perform inconsistency detection on the multidimensional features to obtain preprocessed features; process the preprocessed features using a first branch network to extract spatial features to obtain a first spatial feature; downsample the preprocessed features using a second branch network and extract local features, and upsample the local features to obtain a second spatial feature; the third branch network outputs the preprocessed features; and fuse the first spatial feature, the second spatial feature, and the preprocessed features to obtain spatial inconsistency features. The third detection module is used to extract the temporal inconsistency features of the video to be detected based on a frame comparison network. The feature fusion module is used to fuse the identity information features, the spatial inconsistency features, and the temporal inconsistency features using a feature fusion network to obtain the fake video detection result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the deepfake video detection method based on identity and spatiotemporal inconsistency as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the deepfake video detection method based on identity and spatiotemporal inconsistency as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the deepfake video detection method based on identity and spatiotemporal inconsistency as described in any one of claims 1 to 4.

Images