Deep counterfeit video detection method and device based on identity and space-time inconsistency

Abstract

The invention provides a deep forged video detection method and device based on identity and space-time inconsistency. The method comprises the following steps: acquiring a to-be-detected video and a forged video detection model; the forged video detection model comprises an identity comparison network, a space comparison network, a frame comparison network and a feature fusion network; performing identity information consistency detection on the to-be-detected video based on the identity comparison network to obtain identity information features; performing spatial inconsistency detection on the to-be-detected video based on the spatial comparison network to obtain spatial inconsistency features; extracting time inconsistency characteristics of the video to be detected based on the frame comparison network; and fusing the identity information feature, the space inconsistency feature and the time inconsistency feature by using a feature fusion network to obtain a forged video detection result. By combining identity information consistency detection, space inconsistency detection and time inconsistency detection, the authenticity of the video can be comprehensively analyzed, and various counterfeiting techniques and novel counterfeiting technologies can be coped with.

Description

Technical Field

[0001] The present invention relates to the field of deep fake detection technology, and in particular to a deep fake video detection method and device based on identity and spatiotemporal inconsistency. Background Art

[0002] The misuse of deep fake technology may lead to malicious activities such as false information and telecommunications fraud, and effective detection methods are needed to identify fake videos. In recent years, the rapid development of deep learning has provided new solutions in the field of computer vision.

[0003] Existing deepfake detection methods use low-level image artifacts for detection. The fake image retains some target identity features outside the face swap area, resulting in inconsistent identities between the source and target of the deepfake. Existing deepfake detection methods are unable to capture identity inconsistency features and time inconsistency features in deepfake videos, resulting in low accuracy in deepfake video detection. Summary of the invention

[0004] The present invention provides a method and device for detecting deep fake videos based on identity and spatiotemporal inconsistency, which is used to solve the defect that the prior art cannot capture the identity inconsistency features and time inconsistency features in deep fake videos, and improve the accuracy of deep fake video detection. The technical solution proposed by the present invention is as follows: In a first aspect, the present invention provides a deep fake video detection method based on identity and spatiotemporal inconsistency, comprising: Obtaining a video to be detected and a forged video detection model; wherein the forged video detection model includes an identity comparison network, a space comparison network, a frame comparison network and a feature fusion network; Perform identity information consistency detection on the video to be detected based on the identity comparison network to obtain identity information features; Performing spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain spatial inconsistency features; Extracting the temporal inconsistency features of the video to be detected based on a frame comparison network; A feature fusion network is used to fuse the identity information feature, the spatial inconsistency feature and the temporal inconsistency feature to obtain a forged video detection result.

[0005] Optionally, the identity comparison network includes an improved residual network; the identity information consistency detection of the video to be detected based on the identity comparison network to obtain the identity information features includes: Segment the video to be detected into frames, and extract the embedded representation of each frame; The improved residual network is used to process the embedded representation of each frame, extract the identity information, and obtain the identity information feature.

[0006] Optionally, the forged video detection model further includes a multi-dimensional inconsistency detection network; and the method further includes: Based on the multidimensional inconsistency detection network, multidimensional features are extracted from the video to be detected, and inconsistency detection is performed on the multidimensional features to obtain preprocessed features.

[0007] Optionally, the spatial comparison network includes a first branch network, a second branch network and a third branch network; The spatial inconsistency detection is performed on the video to be detected based on the spatial comparison network to obtain the spatial inconsistency feature, including: Using a first branch network to process the preprocessed features, extracting spatial features to obtain first spatial features; Down-sampling the preprocessed features using a second branch network, extracting local features, and up-sampling the local features to obtain second spatial features; The third branch network takes the preprocessed features as output; The first spatial feature, the second spatial feature and the preprocessed feature are fused to obtain the spatial inconsistency feature.

[0008] Optionally, the extracting the temporal inconsistency feature of the video to be detected based on a frame comparison network includes: Compressing the preprocessed features to obtain a compressed video vector; Extracting frame information of each frame from the compressed video vector; Perform differential operation on the frame information of each frame to obtain the corresponding time series features; Different convolution operations are used to extract local information and global information from the temporal features to obtain the temporal inconsistency features.

[0009] Optionally, the feature fusion network includes a first fusion network; and the step of using the feature fusion network to fuse the identity information feature, the spatial inconsistency feature, and the temporal inconsistency feature to obtain a forged video detection result includes: Fusing the identity information feature and the spatial inconsistency feature to obtain a first fused feature; Processing the first fusion feature based on the first fusion network to obtain processed spatial information; Fusing the time inconsistency feature with the preprocessed feature to obtain a fused feature, and fusing the fused feature with the processed spatial information to obtain a second fused feature; The first fusion feature and the second fusion feature are concatenated to obtain the forged video detection result.

[0010] In a second aspect, the present invention also provides a deep fake video detection device based on identity and spatiotemporal inconsistency, comprising the following modules: A data acquisition module, used to acquire a video to be detected and a forged video detection model; wherein the forged video detection model includes an identity comparison network, a space comparison network, a frame comparison network and a feature fusion network; A first detection module is used to perform identity information consistency detection on the video to be detected based on an identity comparison network to obtain identity information features; A second detection module is used to perform spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain a spatial inconsistency feature; A third detection module is used to extract the time inconsistency feature of the video to be detected based on a frame comparison network; The feature fusion module is used to fuse the identity information feature, the spatial inconsistency feature and the temporal inconsistency feature using a feature fusion network to obtain a forged video detection result.

[0011] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the deep fake video detection method based on identity and spatiotemporal inconsistency as described in the first aspect above is implemented.

[0012] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep fake video detection method based on identity and spatiotemporal inconsistency as described in the first aspect above.

[0013] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the deep fake video detection method based on identity and spatiotemporal inconsistency as described in the first aspect above.

[0014] Based on the above technical solution, the beneficial effects of the present invention compared with the prior art are as follows: The deep fake video detection method and device based on identity and spatiotemporal inconsistency provided by the present invention do not rely on low-level image artifacts, but detect deep fake videos by analyzing identity information and temporal inconsistency features. This method can more effectively cope with the challenge of high-quality fake images, because even if the image quality is very high, the fake identity information and temporal inconsistency are still difficult to completely eliminate. The method performs identity information consistency detection on the video to be detected through an identity comparison network, and can accurately identify whether the identity information in the video is consistent with the real identity information. This helps to solve the identity inconsistency problem in face swapping technology and improve the accuracy of deep fake video detection. The temporal inconsistency features of the video to be detected are extracted through a frame comparison network, and these features are fused with identity information features and spatial inconsistency features using a feature fusion network. This helps to capture the temporal inconsistency features in the video and improve the accuracy of deep fake video detection.

[0015] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a schematic diagram of a forged image in the face forging stage provided by the present invention.

[0019] Figure 2 It is a flowchart of the deep fake video detection method based on identity and spatiotemporal inconsistency provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the forged video detection model provided by the present invention.

[0021] Figure 4 It is a schematic diagram of the detection effect of the present invention on an image processed by five different fraud operations.

[0022] Figure 5It is a structural schematic diagram of a deep fake video detection device based on identity and spatiotemporal inconsistency provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Some existing methods use low-level image artifacts for detection. For example, RGB information, auxiliary masks, and mixed boundary information are used. Artifacts are also used for artificial intelligence generated content detection (AIGC). However, these methods rely on artifacts and datasets. As the quality of forged images continues to improve, methods that rely on low-level textures to generate artifacts are difficult to maintain their effectiveness. Figure 1 As shown in Figure 2, the fake image retains some target identity features outside the face swap area (grey outline), resulting in inconsistent source and target identities of the deepfake, which is a problem ignored by current face swapping techniques. It is assumed that real faces have continuous ID information, while deepfake faces have mixed and discontinuous ID information. Although the deepfake method creates realistic components, the video is synthesized by forging frames, resulting in temporal inconsistencies, especially in the facial area between frames, such as Figure 1 As shown in Figure 2. Existing intra-frame detection methods cannot capture these temporal features. T frame represents the Tth frame in the video, T +1 frame represents the T+1th frame in the video, and zoom represents the zoom operation.

[0026] With the rapid development of deepfake technology, the need for deepfake video detection has become increasingly important. Current deepfake detection methods mainly rely on specific underlying texture cues present in deepfake videos. However, with the continuous improvement of generation technology, deepfakes are becoming more and more realistic with fewer low-level artifacts, which makes detection more challenging. To address this challenge, this paper introduces an Identity Comparison Network (ICN), which uses high-level semantic information to identify deepfakes by analyzing the inconsistency of facial ID. This ID information indicates personal identity features, which are then combined with spatial features extracted by the Spatial Comparison Network to reveal spatial inconsistencies. The Spatial Comparison Network (SCN) extracts traditional image artifact information to improve the detection process. In addition, the Frames Comparison Network (FCN) is introduced to identify inconsistencies between frames in a video, exploiting the inherent weakness of frame-by-frame forgery. The effectiveness and generalization ability of the proposed method in deepfake video detection are demonstrated through experimental evaluation on the FF++ and Celeb-DF datasets.

[0027] Combine the following Figure 2-Figure 5 The present invention describes a method and device for deep fake video detection based on identity and spatiotemporal inconsistency.

[0028] Reference Figure 2 As shown, the deep fake video detection method based on identity and spatiotemporal inconsistency includes the following: Step S110: Obtain the video to be detected and the forged video detection model.

[0029] The video to be tested is a video provided by the user that needs to be verified for authenticity. The user submits the video to be verified for authenticity to the system by uploading or entering a URL. The system receives and stores the video and prepares for the subsequent detection process. The system loads a pre-trained forgery video detection model. Figure 3 As shown in Figure 1, the model consists of four main parts: Identity Comparison Network (ICN), Spatial Comparison Network (SCN), Frames Comparison Network (FCN), and Feature Fusion Network. These network models are implemented using deep learning frameworks (such as TensorFlow, PyTorch, etc.) and saved in model files.

[0030] Step S120: performing identity information consistency detection on the video to be detected based on the identity comparison network to obtain identity information features.

[0031] In the forged video frame, there are discontinuous identity information areas in the face area, while the real face only contains a single continuous identity information. Since there is a difference in identity information between the real face image and the forged face image, the present invention detects the authenticity of the face based on the identity information. The identity information in the input video is analyzed using an identity comparison network, and the facial features of the characters in the video are extracted to detect the consistency of the entire facial identity information and obtain the identity information features.

[0032] Specifically, the identity comparison network first performs facial detection on each frame in the video to be detected and extracts the facial area. A deep learning algorithm (such as a convolutional neural network) is used to extract features from the extracted facial area to obtain a facial feature vector. The extracted facial feature vector is input into the identity comparison network to analyze the consistency of identity information. The identity comparison network determines whether there is inconsistency in identity information by comparing the facial feature vectors between different frames. If there is an inconsistency, it indicates that the video may have been forged or tampered with. The identity comparison network outputs identity information features, which reflect the consistency of the facial identity information of the characters in the video.

[0033] Step S130: Perform spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain spatial inconsistency features.

[0034] Facial images may contain perceptible artifacts after being forged, so the present invention designs a spatial comparison network to extract spatial inconsistency information. Based on the designed spatial comparison network, the video to be detected is analyzed frame by frame to detect static image features in the video frame, such as facial key points, skin texture, inconsistent lighting, etc. The extracted static image features are further processed and analyzed using algorithms such as convolutional neural networks. The spatial comparison network determines whether there is a spatial inconsistency by comparing the static image features between different frames, such as abnormal position of facial key points, inconsistent skin texture, etc. The spatial comparison network outputs spatial inconsistency features, which reflect the consistency of spatial information in the video.

[0035] Step S140: extracting temporal inconsistency features of the video to be detected based on a frame comparison network.

[0036] Use the frame comparison network to extract the temporal features of the video to be detected, and analyze the temporal relationship and dynamic changes between video frames, such as eye blinking, lip synchronization, head movement, etc. Use the frame comparison network to capture the temporal information, and judge whether there is a temporal inconsistency by comparing the temporal relationship between different frames. The frame comparison network outputs temporal inconsistency features, which reflect the consistency of temporal information in the video.

[0037] Step S150: using a feature fusion network to fuse the identity information feature, the spatial inconsistency feature and the temporal inconsistency feature to obtain a forged video detection result.

[0038] The identity information features, spatial inconsistency features, and temporal inconsistency features are input into the feature fusion network. The feature fusion network further processes and analyzes these features through different branches to extract higher-level feature representations. The feature fusion network inputs the extracted high-level feature representations into the classifier (such as the fully connected layer + Softmax) for classification judgment. According to the output of the classifier, it is judged whether the video to be detected is a fake video (Real or Fake). The fake video detection results are output, including whether the video is a fake video and the corresponding confidence or probability value. Reference Figure 3 As shown, the classification layer can represent a fully connected (FC) layer.

[0039] The deep fake video detection method based on identity and spatiotemporal inconsistency provided by the present invention does not rely on low-level image artifacts, but detects deep fake videos by analyzing identity information and time inconsistency features. This method can more effectively cope with the challenge of high-quality fake images, because even if the image quality is high, the fake identity information and time inconsistency are still difficult to completely eliminate. The method performs identity information consistency detection on the video to be detected through an identity comparison network, and can accurately identify whether the identity information in the video is consistent with the real identity information. This helps to solve the identity inconsistency problem in face swapping technology and improve the accuracy of deep fake video detection. The time inconsistency features of the video to be detected are extracted through a frame comparison network, and these features are fused with identity information features and spatial inconsistency features using a feature fusion network. This helps to capture the time inconsistency features in the video and improve the accuracy of deep fake video detection. By combining identity comparison, spatial inconsistency detection and temporal inconsistency detection, the authenticity of the video can be comprehensively analyzed. Identity information consistency detection can ensure that the identity information in the video has not been tampered with; spatial inconsistency detection can find traces of forgery in video frames; and temporal inconsistency detection can capture timing anomalies between video frames. This method can deal with a variety of counterfeiting techniques and new counterfeiting technologies. By continuously updating and optimizing the counterfeit video detection model, the generalization ability and adaptability of the model can be improved.

[0040] In the prior art, models such as Convolutional 3D (C3D) and Inflated 3D Convolutional Networks (I3D) are used for deep fake detection, but they are computationally intensive and not optimized for the task, resulting in poor performance. The deep fake video detection method based on identity and spatiotemporal inconsistency provided by the present invention achieves effective detection of deep fake videos by constructing an efficient fake video detection model, including an identity comparison network, a spatial comparison network, a frame comparison network, and a multi-branch network. These network structures are optimized to reduce the amount of computation and improve performance, thereby having higher practicality and efficiency in practical applications.

[0041] In an optional embodiment, in face recognition tasks, iResNet (improved residual network) is widely used due to its powerful feature extraction capabilities. In particular, in deep fake detection tasks, iResNet can effectively extract facial identity information from real and fake images. The above-mentioned identity comparison network includes an improved residual network; the identity information consistency detection of the video to be detected based on the identity comparison network described in step S120 above to obtain identity information features includes: S1201: Segment the video to be detected into frames, and extract an embedded representation of each frame.

[0042] Before testing the video to be tested, you can perform a video cropping operation on the video to select a part of the video to retain by specifying a rectangular area, and remove the unnecessary area, remove the redundant information in the video, and only retain the part of interest. The video to be tested is composed of a series of continuous images (frames). After that, the video to be tested is segmented (split) to extract each frame of the image. For each segmented frame of the image, use an embedding method (such as a deep learning model) to extract its embedded representation, that is, convert the image into a fixed-length vector that can reflect its key features and information, and obtain the embedded representation of each frame. .

[0043] S1202: Use the improved residual network to process the embedded representation of each frame, extract the identity information, and obtain the identity information feature.

[0044] The embedded representation of each frame is processed using an improved residual network. The residual network (ResNet) is a deep convolutional neural network that solves the gradient vanishing and gradient exploding problems in deep neural networks by introducing residual connections (i.e., skip connections). The improved residual network can include optimizing the network structure, adding attention mechanisms, using more advanced activation functions, etc. to improve the performance and accuracy of the model. In this process, the residual network extracts the identity information features in each frame, which include facial features, expressions, postures, etc. These features are crucial for judging the authenticity of the video. After processing with the improved residual network, the identity information features of each frame can be obtained. Then, these features are fused or averaged to obtain the identity information features of the entire video. This identity information feature vector will be used for subsequent judgment and classification tasks.

[0045] Specifically, refer to Figure 3 As shown in the figure, the video to be detected is segmented and each frame of the image is extracted. Then, the embedded representation of each frame of the image is extracted, that is, the image is converted into a fixed-length vector that can reflect its key features and information. The embedded representation of each frame extracted ( ,in Indicates the first The iResNet residual network processes inputs of size H × W × C (H is height, W is width, and C is the number of channels) and outputs the embedding of each frame, i.e., the identity information feature (Right now Figure 3 These identity information features contain key features and information of the face, such as the shape and position of the facial contour, eyes, nose, mouth, etc.

[0046] (1) in, Represents the operation of identity feature extraction, using IResNet to extract face information.

[0047] In a fake video frame, due to the limitations of deep fake technology, there may be discontinuous identity information areas in the face area. A real face only contains a single continuous identity information. The present invention detects the authenticity of the face by comparing the consistency of the entire face ID information. iResNet can receive the entire face ID information as input and output a score or label representing the consistency of the identity information.

[0048] After iResNet extracts the identity information features, the sigmoid function can be applied to perform a binary classification operation, that is, mapping the extracted features to a probability interval of (0,1) to determine whether the input identity information belongs to a specific category.

[0049] Compared with the traditional method, the method of the present invention directly extracts the complete facial ID information, instead of first segmenting the face area and then extracting the ID information of each part separately. This can avoid the problem of ID information separation caused by inaccurate or inconsistent face area segmentation. iResNet can more accurately extract identity information features, thereby improving the accuracy of video authenticity detection. By fusing the identity information features of multiple frames, the influence of noise and error can be further reduced, and the stability of the overall detection can be improved. The residual connection in the iResNet residual network helps to solve the gradient vanishing and gradient explosion problems in deep neural networks, thereby enhancing the robustness of the model. By comparing the consistency of the entire facial ID information, deep fake videos can be detected more effectively. This method avoids the problem of face area segmentation and ID information separation, and further improves the stability of detection. At the same time, due to the direct extraction of complete facial ID information, unnecessary calculation steps and resource consumption are reduced.

[0050] This paper proposes a multi-branch architecture designed specifically for deep fake detection, treating it as a binary classification problem based on spatial and temporal inconsistencies. The method of this paper integrates the Multi-dimensional Inconsistency Detection (MID) block into the ResNet-50 framework, such as Figure 3 As shown. ICN extracts identity features from a single video frame, while SCN captures additional spatial forgeries. The features of ICN and SCN are combined. In addition, the FCN branch is designed to detect temporal inconsistencies caused by the manipulation of consecutive video frames. These inconsistencies are identified by first compressing the video features and then applying differential operations to the compressed features, revealing unique features that highlight the temporal differences typical of deep forgeries. This combination of spatiotemporal feature extraction across network branches makes the proposed model powerful in identifying deep forgery artifacts, providing a detailed and comprehensive approach to deep forgery detection.

[0051] In an optional embodiment, the forged video detection model further includes a multidimensional inconsistency detection network, by introducing the multidimensional inconsistency detection network, extracting multidimensional features from the video to be detected, and performing inconsistency detection on these features, thereby providing preprocessed features for subsequent spatial inconsistency detection. The method further includes: S210, extracting multidimensional features from the video to be detected based on the multidimensional inconsistency detection network, and performing inconsistency detection on the multidimensional features to obtain preprocessed features.

[0052] The multi-dimensional inconsistency detection network performs multi-dimensional feature extraction, aiming to extract multiple features from the video to be detected that can fully describe the video content and attributes. These features include color, texture, shape, motion information, etc., which together constitute the multi-dimensional feature space of the video. Feature extraction can use convolutional neural networks (CNNs) in deep learning, etc.

[0053] After extracting multidimensional features, inconsistency detection is required for these features. The purpose of inconsistency detection is to identify those feature patterns that do not appear often in real videos but may appear frequently in forged videos. After inconsistency detection, a series of preprocessed features will be obtained. These features have been screened and optimized and are more likely to contain key information related to forged videos. They will be used as input for subsequent spatial inconsistency detection to further improve the accuracy and efficiency of detection.

[0054] The above multi-dimensional inconsistency detection network usually follows a series of orderly and efficient steps when processing the video to be detected to obtain the pre-processed features. These steps combine the characteristics of convolutional neural network (Conv) and multi-dimensional inconsistency detection (MID) blocks to extract key information in the video. The following is a detailed description of the process: Reference Figure 3 As shown, the multi-dimensional inconsistency detection network may include a convolution (Conv) layer and two or more multi-dimensional inconsistency detection (MID) blocks. There may be one or more Conv layers, which are responsible for extracting basic spatiotemporal features from the original video data (i.e., the video to be detected above). The MID block is used to further analyze and process the features extracted from the Conv layer to detect multi-dimensional inconsistency in the video.

[0055] The video to be detected is input into the multi-dimensional inconsistency detection network. The video to be detected is a video sequence containing multiple frames, each of which is a two-dimensional image. The video data first passes through the Conv layer. The Conv layer uses a series of convolution kernels to slide over the video frames to extract local features. These features can be color, texture, edges, etc. After the convolution operation, an activation function (such as ReLU) is applied to increase the nonlinearity of the network, and a pooling layer (such as max pooling) is applied to reduce the resolution of the features and reduce the amount of computation. After stacking multiple Conv layers, the network is able to extract higher-level features that contain spatiotemporal information in the video. The features output from the Conv layer are fed into the MID block, where the features undergo a series of processing steps such as feature transformation, attention mechanism, residual connection, etc. to extract features related to inconsistency. By stacking multiple MID blocks, the network is able to gradually and deeply detect multi-dimensional inconsistencies in the video and extract more refined features. After processing by the Conv layer and the MID block, the network outputs the pre-processed features. These features contain key information in the video, such as the motion trajectory of objects, scene changes, etc., as well as features related to multi-dimensional inconsistency.

[0056] The multi-dimensional inconsistency detection network achieves effective processing of the video to be detected through the collaborative work of the Conv layer and the MID block. The network can more accurately extract key information in the video, detect multi-dimensional inconsistencies, and extract pre-processed features. These features provide strong support for subsequent video analysis tasks.

[0057] The present invention can capture abnormal information in forged videos more comprehensively by extracting multidimensional features and performing inconsistency detection. This helps to reduce false positives and false negatives and improve the accuracy of detection. Multidimensional feature extraction enables the model to process more diverse video data. This helps the model maintain high detection performance when faced with new forgery techniques or unknown video data. The preprocessed features have been screened and optimized to reduce redundant information. This enables subsequent spatial inconsistency detection to be performed faster, thereby improving the overall detection efficiency. The preprocessed features provide strong support for subsequent spatial inconsistency detection. They can serve as important clues and basis to help the model more accurately identify forged videos.

[0058] In an optional embodiment, in the forged video detection task, the spatial comparison network is a key component, which is responsible for extracting spatial inconsistency features from the preprocessed features. The spatial comparison network introduces multiple branches to capture spatial inconsistencies. These branches perform convolution operations in different receptive fields to capture features of different scales. The network includes a first branch network scn1, a second branch network scn2, and a third branch network scn3, which work together to achieve spatial inconsistency detection of the video. The spatial inconsistency detection of the video to be detected based on the spatial comparison network described in step S130 above to obtain spatial inconsistency features includes: S1301. Process the preprocessed features using a first branch network to extract spatial features to obtain first spatial features.

[0059] The first branch network can be a convolutional neural network (CNN) or a similar deep learning model, which can perform convolution operations on the input features and extract spatial features, called first spatial features. These features reflect the spatial structure and texture information in the video.

[0060] S1302: downsample the preprocessed features using a second branch network, extract local features, and upsample the local features to obtain second spatial features.

[0061] The input of the second branch network is also the preprocessed features. The second branch network first downsamples the input features to reduce the resolution of the features and extract local features. Downsampling can be achieved through maximum pooling, average pooling or convolution operations. On the downsampled features, the network will further extract local features, which are usually related to the details and texture of the video. In order to keep consistent with the features output by the first branch network in spatial resolution, the second branch network will upsample the local features. Upsampling can be achieved through bilinear interpolation, nearest neighbor interpolation or transposed convolution. Finally, the second spatial features are output, which combine local details and global structural information.

[0062] S1303: The third branch network uses the preprocessed features as output.

[0063] The third branch network directly uses the input features as output without any additional processing or transformation. This can retain all the information in the original features and provide complete input for subsequent feature fusion.

[0064] S1304: Fusing the first spatial feature, the second spatial feature, and the preprocessed feature to obtain the spatial inconsistency feature.

[0065] The first spatial feature, the second spatial feature and the preprocessed feature are fused. The fusion method can be simple splicing, addition or more complex attention mechanism. The fused features are spatial inconsistency features, which contain the spatial inconsistency information of the video and can be used for subsequent classification or detection tasks.

[0066] Specifically, refer to Figure 3 As shown, the first branch network scn1 uses 3×3 convolution to extract relatively broad spatial features, namely the first spatial features mentioned above. The second branch network scn2 uses average pooling (avgpool) to downsample the input (Down), and then uses 3 × 1 and 1 × 3 convolutions to extract local detailed features. These two convolution operations can extract feature details in the horizontal and vertical directions respectively, and multi-directional convolution operations are more helpful in identifying potential forgery traces. Then the original size is restored by interpolation upsampling (Up) to obtain the above-mentioned second spatial features. The third branch network scn3 uses the original feature vector (that is, the preprocessed features mentioned above) as output. It retains the original, unchanged information and captures subtle details and nuances that may be lost during feature conversion. This helps to enhance the model's ability to learn complex patterns and improve overall performance. Finally, the outputs of the three branch networks are fused using the following formula: (2) in, The spatial inconsistency feature contains global and local spatial inconsistency information, and provides a comprehensive representation of spatial inconsistency information. is the first spatial feature output by the first branch network scn1, is the second spatial feature output by the second branch network scn2, is the output of the third branch network scn3, For the The preprocessed features corresponding to the frame video, It belongs to the feature set X(T×C×H×W), where T, C, H, and W represent the number of frames, the number of image channels, the image height, and the image width, respectively. is a sigmoid function, such as Figure 3 The sigmoid function in . Indicates the frame number of the video frame.

[0067] The present invention can extract global and local spatial features in the video through the collaborative work of the first branch network and the second branch network. These features together form the basis of the spatial inconsistency feature, which helps the model to more accurately identify abnormal areas in the forged video. The downsampling and upsampling operations in the second branch network enable the model to process video data of different resolutions. This helps the model maintain stable detection performance when facing videos of different quality or size. The third branch network directly outputs the preprocessed features to avoid unnecessary repeated calculations. At the same time, the feature fusion process can be carried out in parallel, thereby improving the overall detection efficiency. By fusing the features output by different branch networks, the diversity of features can be increased. This helps the model better capture the complex information in the video and improve the accuracy and generalization ability of detection.

[0068] In an optional embodiment, FCN uses a clever one-dimensional CNN architecture to provide an alternative to traditional methods. By compressing the video to reduce the dimension and splitting it along the time dimension. It can capture the differences between adjacent frames to obtain temporal inconsistency information. By effectively utilizing the time dimension, the model of the present invention greatly reduces the number of parameters without affecting the classification accuracy. By integrating spatial and temporal inconsistency information, the model of the present invention can perform binary classification to distinguish the authenticity of the video. The proposed method aims to detect deep fake videos by exploiting the inherent vulnerabilities of the forgery method itself, and utilizes high-level sequence information from both intra-frame and inter-frame perspectives. The present invention proposes a new method that integrates the above information to detect intra-frame and inter-frame forgeries. The method exploits the identity and temporal inconsistencies in deep anti-counterfeiting videos and uses comprehensive spatial and temporal features for detection. This provides a new and effective model for deep fake detection. Incorporating facial information as high-level semantic information into spatial inconsistency detection can enhance the generalization ability and interpretability of deep fake detection. FCN compresses the video to reduce the dimension and splits it along the time dimension, effectively capturing the differences between frames to obtain temporal inconsistency information. Compared with the traditional two-dimensional CNN and three-dimensional CNN, it reduces the number of network parameters. The step S140 described in extracting the temporal inconsistency feature of the video to be detected based on the frame comparison network includes: S1401. Compress the preprocessed features to obtain a compressed video vector.

[0069] The preprocessed features are compressed to obtain compressed video vectors. The purpose of this step is to reduce the redundant information of the video data and improve the efficiency of subsequent processing. Compression methods can include quantization, encoding, and reducing feature dimensions, which can effectively reduce the size of video data while retaining key information as much as possible.

[0070] S1402. Extract frame information of each frame from the compressed video vector.

[0071] Extract the frame information of each frame from the compressed video vector. This step is the basis of video processing, which enables subsequent analysis and comparison to be performed on each frame of the video. Frame information may include pixel values, color distribution, texture features, etc. of the image, which are crucial for identifying objects, actions, and scenes in the video.

[0072] S1403: Perform a differential operation on the frame information of each frame to obtain corresponding time series features.

[0073] Perform differential operation on the frame information of each frame to obtain the corresponding temporal features. The differential operation can capture the motion information in the video by comparing the differences between adjacent frames. In this process, the changes between adjacent frames are quantified into difference values, which constitute the temporal features, which reflect the changes of the video in the time dimension.

[0074] S1404: Use different convolution operations to extract local information and global information from the time series features to obtain the time inconsistency features.

[0075] Use different convolution operations to extract local and global information from temporal features. The convolution operation can capture local features in the input data by sliding the convolution kernel. In this process, different convolution kernel sizes and steps are used to extract features of different scales. Smaller convolution kernels are better at capturing detailed information (local information), while larger convolution kernels are better at capturing overall structure (global information). By combining these local and global information, more comprehensive and accurate temporal inconsistency features can be obtained.

[0076] Specifically, refer to Figure 3 As shown in the figure, the frame comparison network first compresses the input feature set X to obtain the compressed embedding, that is, the compressed video vector The frame comparison network reduces the dimension of X and compresses the video vector of size C × T × H × W in the horizontal and vertical directions to obtain a compressed video vector of size HW × C × T. The embeddings are compressed along the horizontal and vertical directions, allowing the model to effectively capture spatial artifacts in the video frame. An iterative method is then used to obtain the compressed video vector The frame information along the time dimension is obtained. The difference information between each frame and the subsequent frame is obtained. In this way, the frame comparison network can effectively detect time inconsistencies in the horizontal and vertical directions. Perform frame processing to obtain frame information of each frame . By analyzing the frame information of each frame Perform differential operation to obtain the differential embedding of each frame, that is, the above time series features . Used to extract features through multi-branch networks.

[0077] (3) in, The final timing characteristics are expressed as HW ×C. Function Represents a one-dimensional convolution with a kernel size of 3 and padding of 1, which is used to effectively extract information from adjacent frames. is the vector from the compressed video The video frames obtained from Frame information, is the video frame Frame information.

[0078] Next, a multi-branch method is used to process the time series features The present invention uses a three-branch network structure to process , where different convolution operations are used to effectively extract local and global information from the feature map: (4) (5) in, Branch return unprocessed timing characteristics , which helps to preserve the original information of the image. , A pooling downsampling operation is used, followed by a convolution with a kernel size of 3, and finally upsampling is performed using interpolation. This increases the receptive field of the neural network, enabling it to receive a wide range of contextual information. Use convolution with a kernel size of 3 to perform time series features Processing. , , The outputs of the three branches are fused to obtain the fused features The fused features Through conv1 and Function processing, the final output is expressed as , that is Figure 3 Time inconsistency characteristics in , Indicates the frame number of the video frame. conv1 is a convolution operation with a kernel size of 1. is a sigmoid function, such as Figure 3 The sigmoid function in .

[0079] By extracting the time inconsistency feature, the present invention can more accurately identify the forged or tampered parts in the video. This is of great significance for combating video forgery techniques such as deep fakes. The time inconsistency feature can capture the changes in the video in the time dimension, and these changes are often closely related to forgery or tampering. Therefore, using these features for video detection can significantly improve the accuracy of detection.

[0080] In an optional embodiment, the feature fusion network includes a first fusion network; the step S150 described above uses the feature fusion network to fuse the identity information feature, the spatial inconsistency feature, and the temporal inconsistency feature to obtain a forged video detection result, including: S1501. Fuse the identity information feature and the spatial inconsistency feature to obtain a first fused feature.

[0081] The identity information feature and the spatial inconsistency feature are fused to obtain the first fused feature. The identity information feature refers to the identity information of the person in the video, such as facial feature information, while the spatial inconsistency feature refers to the inconsistency of the spatial attributes such as the position, shape, and size of the object or person in the video frame. Figure 3 As shown in the figure, by fusing these two features, the identity information and spatial attributes of the video can be comprehensively considered, providing more comprehensive information support for subsequent forged video detection.

[0082] (6) in, is the first fusion feature, which represents the video feature integrating identity information and spatial inconsistency information. For the Identity information features of frame video, Spatial inconsistency features output by the spatial comparison network.

[0083] S1502: Process the first fusion feature based on the first fusion network to obtain processed spatial information.

[0084] The first fusion feature is processed based on the first fusion network to obtain processed spatial information. The first fusion network can be a deep learning model, such as a convolutional neural network (CNN), etc., which can further analyze and extract the first fusion feature to obtain more accurate and useful spatial information.

[0085] (7) in, is the processed spatial information, It is composed of a 3×3 convolution and a 1×3 convolution followed by a sigmoid function (such as Figure 3 The branch processing structure composed of sigmoid. The role of is to enhance the fusion of two types of spatial information, and finally obtain the processed spatial information .

[0086] S1503: Fusing the time inconsistency feature with the preprocessed feature to obtain a fused feature, and fusing the fused feature with the processed spatial information to obtain a second fused feature.

[0087] The temporal inconsistency feature is fused with the preprocessed feature to obtain the fused feature. The temporal inconsistency feature refers to the inconsistency of the video frames in the time series, such as the change of frame rate, the discontinuity of motion trajectory, etc. The preprocessed feature is the feature obtained after preprocessing the original video data, such as denoising, enhancement, etc. By fusing these two features, the temporal attributes of the video and the preprocessed feature information can be comprehensively considered, providing richer information for subsequent forged video detection.

[0088] The fused features are fused with the processed spatial information to obtain the second fused features. This step is to fuse the two features obtained previously again to comprehensively consider the spatial, temporal and pre-processed feature information of the video.

[0089] (8) in, is the second fusion feature, To extract the video time inconsistency information using the frame comparison network, that is, the above-mentioned time inconsistency feature, is the preprocessed feature, is the processed spatial information.

[0090] To ensure compatibility of subsequent operations, And processed spatial information were refactored to match Dimensions. and Element-wise multiplication is performed between , to emphasize the temporal changes in the spatial context of each frame. In addition, Fused with the result of this multiplication to form the second fused feature , which integrates spatial and temporal cues.

[0091] S1504: Concatenate the first fusion feature and the second fusion feature to obtain the forged video detection result.

[0092] The first fusion feature and the second fusion feature are spliced ​​to obtain a forged video detection result. The splicing operation can be an operation such as connection, fusion or weighting of feature vectors to obtain a final detection result.

[0093] (9) in, To detect forged video features, For splicing operation. Input to the following MID module, after multiple iterations, the last layer of MID module After inputting into the FC layer, real or fake results are obtained.

[0094] The present invention can comprehensively consider various information in the video by fusing identity information features, spatial inconsistency features and temporal inconsistency features, thereby improving the accuracy of forged video detection. Different features can reflect the authenticity and integrity of the video from different angles, so fusing them can obtain more comprehensive and accurate detection results. The feature fusion network can process a variety of different feature inputs and effectively integrate and extract these features. This enables the model to maintain stable performance when facing different types of forged videos, and enhances the robustness of the model. The feature fusion network can adapt to different types of video data, including videos of different resolutions, different formats and different scenes. This makes the method widely applicable in practical applications and can be applied to various forged video detection scenarios.

[0095] The present invention uses two datasets: FaceForensics++ (FF++) and Celebrity-DeepFakes (Celeb-DF). The FF++ dataset is a popular deep fake detection dataset, which consists of three types of videos: original quality video raw, low compression video c23 and high compression video c40. The dataset uses four methods to make fake videos, namely: DeepFakes (DF), Face2Face (F2F), FaceSwap (FS) and NeuralTextures (NT). Later, the FaceShifter (FST) method was also added to the ranks for generating fake videos. The Celeb-DF dataset contains 5639 high-quality fake videos and 590 real videos. These processed videos are generated by an improved synthesis method.

[0096] The MID block is integrated into the ResNet-50 network to replace the original bottleneck block, and the input image size is 224 × 224 pixels. The Adam optimizer and binary cross entropy loss are used to train the above fake video detection model. The model is trained for 30 batches with a batch size of 16. The initial learning rate is set to 0.0002 and decays by 10 times every 10 epochs. Training and testing are performed on the FF++c40 training set and test set, respectively, and cross-dataset experimental testing is performed on the Celeb-DF test set. The multi-task convolutional neural network (MTCNN) in the existing technology is used to detect the facial region and input it into the model. During training, 8 consecutive video frames are extracted from each video, and 16 frames are extracted during testing. The area under the receiver operating characteristic curve (AUC) and accuracy (ACC) are used as evaluation indicators. As shown in Table 1, the results of different models (including existing C3D, I3D, Rahmouni, SPSL, MesoNet, Xception, STIL models) and the proposed method (Ours) on the FF++ dataset are shown in bold, and * indicates the reproduction of this method.

[0097] Table 1 As can be seen from Table 1, the deep fake video detection method based on identity and spatiotemporal inconsistency provided by the present invention has achieved good performance on all five methods (DF, F2F, FS, NT, FST) in FF++. In addition, the comparison with other methods shows that the method of the present invention outperforms most existing methods and has significant improvements. SPSL is a method specially designed for detection using low-level textures. The performance of the method of the present invention that is better than SPSL shows that identity features (high-level semantic information) show strong generalization ability. Low-level texture features depend on image quality. Detection methods using low-level texture features tend to perform relatively poorly. In addition, the method of the present invention integrates temporal information from the video and detects fake clues in the temporal dimension. Compared with traditional video methods (such as C3D and I3D) using 3D CNN networks, it can be seen that the present invention purposefully designs a 1D CNN network on the time branch, achieving better results than traditional video classification methods. The method of the present invention has surpassed STIL, proving the effectiveness of spatial inconsistency information with ID clues and designed temporal inconsistency information in the method of the present invention.

[0098] Various existing methods in Table 2 (including Multi-task, Capsule, Two-branch, MesoInception4, Xception, SPSL, F3_Net, M2TR, STIL) and the proposed method (Ours) are trained on the FF++ dataset and tested on the FF++ dataset and Celeb-DF dataset. The final results are reported as AUC (%), and the best performance is indicated in bold.

[0099] Table 2 In order to verify the generalization of the proposed method, various deep fake detection methods on different data sets were compared and analyzed. As shown in Table 2, Xception is a detection method based on single frame analysis, while STIL focuses on extracting spatiotemporal inconsistencies. After comparison, the proposed method achieved the best results, indicating that the integration of identity information and temporal and spatial inconsistencies can lead to better generalization capabilities.

[0100] In order to evaluate the effectiveness of the proposed method, an ablation experiment was conducted on the FF++ dataset by evaluating the effectiveness of the three components of the identity comparison network ICN, the spatial comparison network SCN, and the frame comparison network FCN. The components (variants) were evaluated according to the experimental scheme in Table 3, and the results were expressed as AUC (%). The experimental results are shown in Table 3. Obviously, the most effective method is to combine these three components, which shows the necessity of combining spatiotemporal information with identity information. The temporal extraction method was also experimented. Specifically, the FCN (STIL) + ICN + SCN method was used to transfer the temporal inconsistency extraction method used in STIL to FCN for experiment. The experimental results show that this method is significantly improved over the original method. The experiment proves that the lack of any ID clue, spatial information or temporal information will lead to a decrease in performance, which further confirms the rationality of the design scheme of the present invention. In Table 3, w / o means not included, w / o ICN means not including the identity comparison network, w / o SCN means not including the spatial comparison network, and w / o FCN means not including the frame comparison network.

[0101] Table 3 As shown in Table 4, the parameters of the method of the present invention are compared with several video classification models. Compared with the traditional detection methods based on 3D CNN and 2D CNN, the method of the present invention has the least parameters and the best performance.

[0102] Table 4 Reference Figure 4 As shown, the decision-making process of the method of the present invention is visualized by using the gradient-weighted class activation mapping (Grad-CAM) method. Figure 4 The first row in consists of forged images, and the second row shows the corresponding CAM images. It can be observed that the proposed method mainly focuses on the areas where two identities are connected in the face replacement methods DF and FS, such as eyes, eyebrows, and forehead. This confirms that the proposed model uses the merged ID information to identify forged images based on discontinuous ID regions and locates forged regions in other methods. It verifies that the proposed method achieves the expected design goals, and the judgment based on discontinuous ID regions has high interpretability.

[0103] The proposed method uses identity information as high-level semantic information to enhance interpretability and generalization. By supplementing other spatial information, better results are achieved. In addition, a customized 1D CNN network is used to detect forged videos using the temporal differences between adjacent frames. Compared with traditional video classification models, this method reduces the number of parameters and the experimental results are satisfactory. A large number of experiments have proven the effectiveness of the method by detecting clues from multiple dimensions.

[0104] The deep fake video detection device based on identity and time-space inconsistency provided by the present invention is described below. The deep fake video detection device based on identity and time-space inconsistency described below and the deep fake video detection method based on identity and time-space inconsistency described above can be referenced to each other.

[0105] The deep fake video detection device based on identity and spatiotemporal inconsistency provided by the present invention refers to Figure 5 As shown, including: The data acquisition module 310 is used to acquire the video to be detected and the forged video detection model; wherein the forged video detection model includes an identity comparison network, a space comparison network, a frame comparison network and a feature fusion network; A first detection module 320 is used to perform identity information consistency detection on the video to be detected based on an identity comparison network to obtain identity information features; A second detection module 330 is used to perform spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain a spatial inconsistency feature; A third detection module 340 is used to extract the time inconsistency feature of the video to be detected based on a frame comparison network; The feature fusion module 350 is used to fuse the identity information feature, the spatial inconsistency feature and the temporal inconsistency feature using a feature fusion network to obtain a forged video detection result.

[0106] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the deep fake video detection method based on identity and spatiotemporal inconsistency.

[0107] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the deep fake video detection method based on identity and temporal and spatial inconsistency provided by the above methods.

[0109] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the deep fake video detection method based on identity and spatiotemporal inconsistency provided by the above-mentioned methods.

[0110] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep fake video detection method based on identity and spatiotemporal inconsistency, characterized in that: include: Obtaining a video to be detected and a forged video detection model; wherein the forged video detection model includes an identity comparison network, a space comparison network, a frame comparison network and a feature fusion network; Perform identity information consistency detection on the video to be detected based on the identity comparison network to obtain identity information features; Performing spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain spatial inconsistency features; Extracting the temporal inconsistency features of the video to be detected based on a frame comparison network; A feature fusion network is used to fuse the identity information feature, the spatial inconsistency feature and the temporal inconsistency feature to obtain a forged video detection result.

2. The deep fake video detection method based on identity and spatiotemporal inconsistency according to claim 1 is characterized in that: The identity comparison network includes an improved residual network; The identity information consistency detection of the video to be detected is performed based on the identity comparison network to obtain identity information features, including: Segment the video to be detected into frames, and extract the embedded representation of each frame; The improved residual network is used to process the embedded representation of each frame, extract the identity information, and obtain the identity information feature.

3. The deep fake video detection method based on identity and spatiotemporal inconsistency according to claim 1 is characterized in that: The forged video detection model also includes a multi-dimensional inconsistency detection network; the method also includes: Based on the multidimensional inconsistency detection network, multidimensional features are extracted from the video to be detected, and inconsistency detection is performed on the multidimensional features to obtain preprocessed features.

4. The deep fake video detection method based on identity and spatiotemporal inconsistency according to claim 3 is characterized in that: The spatial comparison network includes a first branch network, a second branch network and a third branch network; The spatial inconsistency detection is performed on the video to be detected based on the spatial comparison network to obtain the spatial inconsistency feature, including: Using a first branch network to process the preprocessed features, extracting spatial features to obtain first spatial features; Down-sampling the preprocessed features using a second branch network, extracting local features, and up-sampling the local features to obtain second spatial features; The third branch network takes the preprocessed features as output; The first spatial feature, the second spatial feature and the preprocessed feature are fused to obtain the spatial inconsistency feature.

5. The deep fake video detection method based on identity and spatiotemporal inconsistency according to claim 3 is characterized in that: The extracting the time inconsistency feature of the video to be detected based on the frame comparison network includes: Compressing the preprocessed features to obtain a compressed video vector; Extracting frame information of each frame from the compressed video vector; Perform differential operation on the frame information of each frame to obtain the corresponding time series features; Different convolution operations are used to extract local information and global information from the temporal features to obtain the temporal inconsistency features.

6. The deep fake video detection method based on identity and spatiotemporal inconsistency according to claim 3 is characterized in that: The feature fusion network includes a first fusion network; the feature fusion network is used to fuse the identity information feature, the spatial inconsistency feature and the temporal inconsistency feature to obtain a forged video detection result, including: Fusing the identity information feature and the spatial inconsistency feature to obtain a first fused feature; Processing the first fusion feature based on the first fusion network to obtain processed spatial information; Fusing the time inconsistency feature with the preprocessed feature to obtain a fused feature, and fusing the fused feature with the processed spatial information to obtain a second fused feature; The first fusion feature and the second fusion feature are concatenated to obtain the forged video detection result.

7. A deep fake video detection device based on identity and spatiotemporal inconsistency, characterized in that: include: A data acquisition module, used to acquire a video to be detected and a forged video detection model; wherein the forged video detection model includes an identity comparison network, a space comparison network, a frame comparison network and a feature fusion network; A first detection module is used to perform identity information consistency detection on the video to be detected based on an identity comparison network to obtain identity information features; A second detection module is used to perform spatial inconsistency detection on the video to be detected based on a spatial comparison network to obtain a spatial inconsistency feature; A third detection module is used to extract the time inconsistency feature of the video to be detected based on a frame comparison network; The feature fusion module is used to fuse the identity information feature, the spatial inconsistency feature and the temporal inconsistency feature using a feature fusion network to obtain a forged video detection result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the deep fake video detection method based on identity and spatiotemporal inconsistency as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the deep fake video detection method based on identity and spatiotemporal inconsistency as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the deep fake video detection method based on identity and spatiotemporal inconsistency as described in any one of claims 1 to 6 is implemented.

Images