Deepfake video detection method, apparatus, device and medium

Abstract

This application relates to the field of artificial intelligence technology, and discloses a method, apparatus, device, and medium for deepfake video detection. The method includes: acquiring a video to be detected; extracting facial features from video frames of the video to obtain a facial feature map; extracting fine-grained attention features from the facial feature map to obtain a fine-grained attention feature map; extracting illumination features from the facial feature map to obtain an illumination feature map; performing cross-attention processing on the fine-grained attention feature map and the illumination feature map to obtain a cross-fusion feature map; performing decompositional spatiotemporal attention feature extraction on the cross-fusion feature map to extract the spatiotemporal dependencies between video frames, obtaining spatiotemporal attention features; and determining the deepfake detection result of the video to be detected based on the spatiotemporal attention features. The embodiments of this application can improve the accuracy and real-time performance of deepfake detection methods.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for detecting deepfake videos. Background Technology

[0002] The combination of artificial intelligence and multimedia synthesis technologies has enabled deepfakes to develop rapidly in both video and audio fields. Deepfake videos are becoming increasingly realistic, posing a significant security risk to cybersecurity and even social stability.

[0003] To address the potential threat posed by deepfake media to cyberspace and public opinion, numerous detection methods targeting single-modal media files have been proposed and have achieved good performance. However, existing deepfake detection methods are often affected in terms of accuracy and real-time performance when dealing with multimodal deepfake techniques. Furthermore, existing deepfake detection methods struggle to capture discontinuities or forgery traces between modalities, and fail to fully utilize the tampering traces left in forged videos by multimodal deepfake techniques. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, device, and medium for detecting deepfake videos, which can improve the accuracy and real-time performance of deepfake detection methods.

[0005] This application provides a method for detecting deepfake videos, including:

[0006] Obtain the video to be tested;

[0007] Facial features are extracted from the video frames of the video to be detected to obtain a facial feature map;

[0008] Fine-grained attention feature extraction is performed on the facial feature map to obtain a fine-grained attention feature map;

[0009] Illumination features are extracted from the facial feature map to obtain an illumination feature map;

[0010] Cross-attention processing is performed on the fine-grained attention feature map and the illumination feature map to obtain a cross-fused feature map;

[0011] The cross-fusion feature map is subjected to decompositional spatiotemporal attention feature extraction to extract the spatiotemporal dependencies between the video frames and obtain spatiotemporal attention features;

[0012] Based on the spatiotemporal attention features, the depth forgery detection result of the video to be detected is determined.

[0013] In some embodiments, the fine-grained attention feature extraction of the facial feature map includes:

[0014] Shallow feature extraction is performed on the face feature map to obtain a shallow feature map;

[0015] Texture features are extracted from the shallow feature map to obtain a texture feature map;

[0016] The shallow feature map is processed by an attention mechanism to obtain a face attention feature map;

[0017] Random features are extracted from both the texture feature map and the face attention feature map, and the extracted features are fitted to obtain the fine-grained attention feature map.

[0018] In some embodiments, the step of extracting illumination features from the facial feature map includes:

[0019] The facial feature map is converted into a corresponding grayscale facial feature map;

[0020] Key contour features are extracted from the grayscale feature map of the face to obtain a contour feature map;

[0021] Calculate the edge pixel density and average pixel density of multiple local regions in the contour feature map;

[0022] Based on the edge pixel density and the average pixel density, the local illumination direction of the local region is solved by least squares and weighted fitting to obtain the illumination feature map.

[0023] In some embodiments, the cross-attention processing of the fine-grained attention feature map and the illumination feature map includes:

[0024] The self-attention feature is determined as a cross-attention query vector;

[0025] The illumination feature map is transformed into a cross-attention key vector and a cross-attention value vector;

[0026] Cross-attention processing is performed based on the cross-attention query vector, the cross-attention key vector, and the cross-attention value vector to obtain the cross-fusion feature map.

[0027] In some embodiments, the spatial attention encoding of the cross-fused feature map includes:

[0028] Spatial feature embedding is performed on the cross-fusion feature map to obtain spatial embedding features;

[0029] The spatial embedding features are subjected to matrix transformation to obtain a spatial self-attention query vector, a spatial self-attention key vector, and a spatial self-attention value vector;

[0030] The spatial attention features are obtained by performing self-attention mechanism processing based on the spatial self-attention query vector, the spatial self-attention key vector, and the spatial self-attention value vector.

[0031] And / or the temporal attention encoding of the spatial attention features includes:

[0032] Temporal feature embedding is performed on the spatial attention features to obtain temporal embedding features;

[0033] The temporal embedding features are subjected to matrix transformation to obtain a temporal self-attention query vector, a temporal self-attention key vector, and a temporal self-attention value vector;

[0034] The spatiotemporal attention features are obtained by performing self-attention mechanism processing based on the temporal self-attention query vector, the temporal self-attention key vector, and the temporal self-attention value vector.

[0035] In some embodiments, the decompositional spatiotemporal attention feature extraction of the cross-fused feature map includes:

[0036] Spatial attention encoding is performed on the cross-fused feature map to extract spatial features from each video frame, thus obtaining spatial attention features;

[0037] Temporal attention encoding is performed on the spatial attention features to extract the spatiotemporal dependencies between the video frames, thus obtaining the spatiotemporal attention features.

[0038] In some embodiments, determining the deepfake detection result of the video to be detected based on the spatiotemporal attention features includes:

[0039] The spatiotemporal features contained in the spatiotemporal attention features are nonlinearly combined and mapped to obtain the original scores corresponding to each preset category;

[0040] Based on the original scores, a predicted probability distribution of the spatiotemporal attention features belonging to each of the preset categories is generated;

[0041] Based on the predicted probability distribution, the depth forgery detection result of the video to be detected is determined.

[0042] This application also provides a deepfake video detection device, including:

[0043] The first module is used to acquire the video to be detected;

[0044] The second module is used to extract facial features from the video frames of the video to be detected, and obtain a facial feature map;

[0045] The third module is used to extract fine-grained attention features from the face feature map to obtain a fine-grained attention feature map.

[0046] The fourth module is used to extract illumination features from the face feature map to obtain an illumination feature map;

[0047] The fifth module is used to perform cross-attention processing on the fine-grained attention feature map and the illumination feature map to obtain a cross-fused feature map;

[0048] The sixth module is used to perform decompositional spatiotemporal attention feature extraction on the cross-fusion feature map to extract the spatiotemporal dependencies between the video frames and obtain spatiotemporal attention features.

[0049] The seventh module is used to determine the depth forgery detection result of the video to be detected based on the spatiotemporal attention features.

[0050] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described deepfake video detection method.

[0051] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described deepfake video detection method.

[0052] The beneficial effects of this application are as follows: By introducing fine-grained attention feature extraction and illumination feature extraction, and further fusing them through cross-attention processing, decompositional spatiotemporal attention feature extraction is performed on the cross-fused feature map to extract the spatiotemporal dependencies between video frames. Finally, the deepfake detection result of the video to be detected is determined based on the spatiotemporal attention features. Therefore, when there is an inconsistency between the subtle texture of the face and the illumination pattern in the video, cross-attention processing can sensitively capture this inconsistency between modalities, enabling a deeper exploration of tampering traces left in deepfake videos, thereby improving the robustness of detection. Furthermore, by employing decompositional spatiotemporal attention feature extraction to extract the spatiotemporal dependencies between video frames, when deepfake videos exhibit unnatural motion, facial expression changes, or illumination transitions between frames, decompositional spatiotemporal attention can capture these anomalies from both spatial and temporal dimensions. This allows for a more comprehensive and refined utilization of the spatiotemporal dynamic information of the video, significantly improving the detection capability for complex deepfake videos and enhancing the accuracy and real-time performance of deepfake detection methods. Attached Figure Description

[0053] Figure 1 This diagram illustrates the application environment of the deepfake video detection method provided in the embodiments of this application.

[0054] Figure 2 This is a flowchart of the deepfake video detection method provided in the embodiments of this application.

[0055] Figure 3 This is a flowchart of a method for fine-grained attention feature extraction from a face feature map, provided in an embodiment of this application.

[0056] Figure 4 This is a flowchart of a method for extracting illumination features from a face feature map, as provided in an embodiment of this application.

[0057] Figure 5 This is a flowchart of a method for cross-attention processing of fine-grained attention feature maps and illumination feature maps provided in an embodiment of this application.

[0058] Figure 6 This is a schematic diagram of the deepfake video detection device provided in the embodiments of this application.

[0059] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0063] The deepfake video detection method provided in this application can be executed by a computer device, which can be a terminal device or a server. The terminal device includes, but is not limited to, mobile phones, computers, smart home appliances, vehicle terminals, and aircraft. The server can be a standalone physical server, a server cluster consisting of multiple physical servers, a distributed system, or a cloud server. Furthermore, the information, data, and signals involved in this application's embodiments are all authorized by the relevant parties or have received full authorization from all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0064] In the field of deepfake video detection technology, existing single-modality detection methods have significant limitations when handling multimodal forgery scenarios. Specifically, they cannot effectively capture the spatiotemporal dependencies and feature discontinuities between video frames caused by forgery operations, resulting in insufficient sensitivity to forgery traces and consequently affecting the accuracy of detection results and system processing efficiency. In particular, logical contradictions between facial regions and ambient lighting conditions in multimodal forged videos are often ignored by existing feature extraction mechanisms, and insufficient fusion of cross-modal features further weakens the ability to identify subtle tampering traces, thus reducing the reliability of the detection system in complex forgery scenarios. For example, in the content security review scenario of social media platforms, when the video to be detected contains facial animations synthesized using deepfake technology and altered background lighting information, existing detection models lack the ability to jointly analyze fine-grained facial features and local lighting directions. They cannot identify the deviation between the edge pixel density distribution in the forged region and the actual physical lighting patterns, causing the system to misclassify the forged video as genuine content. Furthermore, the spatiotemporal correlation features of the video frame sequence in this scenario were not fully modeled, making it difficult for the model to distinguish between natural facial movements and fake, non-continuous actions, ultimately causing the review process to fail.

[0065] If the aforementioned technical issues are not resolved, deepfake content will more easily circumvent existing detection mechanisms' limitations on its spread in cyberspace. The resulting proliferation of misinformation will directly threaten the security of user identity authentication systems and exacerbate the uncontrollable risks in the social environment. At the same time, the detection system's lack of adaptability to multimodal forgery samples will continue to restrict its practical application value in real-time security protection scenarios.

[0066] Based on this, embodiments of this application provide a method, apparatus, device, and medium for detecting deepfake videos. By introducing fine-grained attention feature extraction and illumination feature extraction, and combining decomposed spatiotemporal attention feature extraction technology to decouple feature processing in the spatial and temporal dimensions, the accuracy and real-time performance of the deepfake detection method are improved.

[0067] Figure 1This diagram illustrates the application environment of the deepfake video detection method provided in this embodiment. (See attached diagram.) Figure 1 This method is applied to a deepfake video detection system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. The terminal 110 sends the video to be detected to the server 120. The server 120 acquires the video, extracts facial features from the video frames to obtain a facial feature map, extracts fine-grained attention features from the facial feature map to obtain a fine-grained attention feature map, extracts illumination features from the facial feature map to obtain an illumination feature map, performs cross-attention processing on the fine-grained attention feature map and the illumination feature map to obtain a cross-fusion feature map, performs decompositional spatiotemporal attention feature extraction on the cross-fusion feature map to extract the spatiotemporal dependencies between video frames to obtain spatiotemporal attention features, and determines the deepfake detection result of the video based on the spatiotemporal attention features.

[0068] It should be understood that Figure 1 The application scenarios shown are merely examples. In practical applications, the deepfake video detection method provided in this application embodiment can also be applied to other scenarios. For example, the above-described deepfake video detection method can be directly applied to terminal 110. Terminal 110 is used to acquire the video to be detected, extract facial features from the video frames of the video to be detected to obtain a facial feature map, extract fine-grained attention features from the facial feature map to obtain a fine-grained attention feature map, extract illumination features from the fine-grained attention feature map to obtain an illumination feature map, perform cross-attention processing on the fine-grained attention feature map and the illumination feature map to obtain a cross-fusion feature map, perform decompositional spatiotemporal attention feature extraction on the cross-fusion feature map to extract the spatiotemporal dependencies between each video frame to obtain spatiotemporal attention features, and determine the deepfake detection result of the video to be detected based on the spatiotemporal attention features.

[0069] See Figure 2 In one embodiment, a deepfake video detection method is provided. The execution subject of the method can be a terminal or a server, including but not limited to steps S201 to S205.

[0070] Step S201: Obtain the video to be tested.

[0071] Acquiring the video to be tested refers to the process of obtaining video data from an external source. This can be achieved by directly reading video files from storage media or by receiving real-time video streams. For example, it can be done by calling interface functions of standard video processing libraries or by using network protocols to transmit video data.

[0072] Step S202: Extract facial features from video frames of the video to be detected to obtain facial feature maps.

[0073] Facial feature extraction from video frames of the video to be tested refers to identifying facial regions from the video frames and generating feature representations.

[0074] In practical applications, facial feature extraction can be achieved using deep learning-based feature extraction models, such as end-to-end feature embedding using pre-trained convolutional neural networks, or by combining traditional computer vision methods such as Haar feature detectors with local binary pattern descriptors. The main purpose is to capture the structured features of the facial region to support subsequent forgery analysis.

[0075] After the video to be detected is acquired, the video frames undergo facial feature extraction processing to obtain facial feature maps. This facial feature extraction focuses on key areas such as the eyes, eyebrows, and mouth in the video frames. These areas, as carriers of biomechanical features, can effectively capture subtle tampering traces left by deepfake technology in local areas. Specifically, the executing entity identifies and extracts key facial regions through structured feature representation, thereby enhancing the perception of forgery-sensitive areas and avoiding the omission of local artifacts caused by global feature analysis.

[0076] Step S203: Fine-grained attention feature extraction is performed on the face feature map to obtain a fine-grained attention feature map.

[0077] Fine-grained attention feature extraction refers to the process of further focusing on minute, subtle details in the facial region based on the facial feature map, and enhancing them using attention mechanisms. This process aims to capture subtle traces of forgery that may exist in the facial region. A fine-grained attention feature map is a data representation of the enhanced minute details in the facial region after fine-grained attention feature extraction. This feature map can highlight local information that is important for deepfake detection.

[0078] Fine-grained attention feature extraction from the facial feature map can be achieved by applying a series of standard convolutional layers to extract its basic visual features, followed by a global average pooling layer to generate a coarse attention weight map. This weight map indicates the commonly concerned areas in the facial region, thus forming a fine-grained attention feature map. Alternatively, a simple encoder-decoder structure can be used to process the facial feature map, where the latent representation output by the encoder is considered the fine-grained attention feature map.

[0079] Step S204: Extract illumination features from the face feature map to obtain the illumination feature map.

[0080] Illumination feature extraction refers to the process of analyzing and extracting visual attributes related to lighting conditions from a facial feature map. This process aims to identify information such as the direction, intensity, and shadow distribution of light in facial regions of a video, information that may be inconsistent in forged videos. An illumination feature map is a set of lighting-related features represented in a data structure after illumination feature extraction. This feature map reflects the visual appearance of a facial region under different lighting conditions.

[0081] Illumination feature extraction from a facial feature map can be achieved by calculating the average pixel intensity of predefined local regions within the map. These average intensity values ​​can serve as features characterizing the local illumination strength, thus forming the illumination feature map. Alternatively, an edge detection filter can be applied to the facial feature map, and the edge intensity in different directions can be statistically analyzed to roughly estimate the dominant illumination direction, thereby generating the illumination feature map.

[0082] Step S205: Perform cross-attention processing on the fine-grained attention feature map and the illumination feature map to obtain a cross-fused feature map.

[0083] Cross-attention processing is a mechanism that allows information exchange and fusion between features from different modalities or sources. In this process, one feature acts as the query, and another as the key and value, thereby achieving mutual enhancement and fusion of features. A cross-fused feature map is a comprehensive feature representation that combines information from different feature modalities after cross-attention processing. This feature map integrates the advantages of fine-grained attention features and illumination features to provide more comprehensive information.

[0084] Cross-attention processing of the fine-grained attention feature map and the illumination feature map can be performed by simply concatenating them along the feature dimension to form a joint feature map containing information from both. Alternatively, after adjusting both to the same size, element-wise addition or multiplication can be performed to achieve preliminary feature fusion.

[0085] Step S206: Perform decompositional spatiotemporal attention feature extraction on the cross-fusion feature map to extract the spatiotemporal dependencies between video frames and obtain spatiotemporal attention features.

[0086] Decompositional spatiotemporal attention feature extraction refers to a method that decomposes the spatiotemporal attention mechanism into two independent parts: spatial attention and temporal attention. This method aims to capture the spatial features within a video frame and the temporal dependencies between video frame sequences, respectively. In practical applications, decompositional spatiotemporal attention feature extraction can employ a sequential processing approach, first encoding spatial features and then aggregating temporal features, or it can use separate convolutional operations to process spatial and temporal information separately. For example, it can process the spatial features of a single frame using a two-dimensional convolutional layer and then use a recurrent neural network to process the temporal series. The main purpose is to reduce computational complexity and effectively capture the dynamic relationships between video frames.

[0087] Spatiotemporal dependencies refer to the interrelationships between different video frames and between different regions within the same video frame. In deepfake videos, these dependencies may be disrupted or exhibit anomalous patterns. Spatiotemporal attention features, after decompositional spatiotemporal attention feature extraction, are a comprehensive feature representation that includes both the spatial features within a video frame and the temporal dependencies between video frame sequences. This feature can comprehensively reflect the spatiotemporal dynamics of the video.

[0088] When performing decompositional spatiotemporal attention feature extraction on the cross-fused feature map, spatial attention encoding is first performed to extract spatial features within a single frame, followed by temporal attention encoding to capture the temporal dependencies between video frames. Spatial attention encoding can generate query vectors, key vectors, and value vectors through matrix transformations and calculate spatial features based on a self-attention mechanism. For example, in specific implementations, spatial attention encoding can employ a multi-head self-attention mechanism to model the spatial-dimensional dependencies of the feature map. Furthermore, temporal attention encoding performs temporal-dimensional feature embedding and transformation based on the spatial attention features, thereby extracting dynamic change information between frames. This decompositional processing avoids the high computational complexity caused by joint spatiotemporal modeling and significantly improves processing efficiency.

[0089] Step S207: Based on spatiotemporal attention features, determine the depth forgery detection result of the video to be detected.

[0090] In practical applications, the results of deepfake detection can be determined by using a fully connected neural network for feature mapping, or by using a support vector machine classifier to output the category probability distribution. For example, the features can be input into a softmax layer to generate the predicted probabilities of real and fake categories. The main purpose is to integrate the spatiotemporal dependencies to achieve forgery determination.

[0091] Based on the obtained spatiotemporal attention features, the original scores of the preset categories are generated through nonlinear combination and mapping, and then further converted into predicted probability distributions to finally determine the deepfake detection results.

[0092] The following example will provide a more detailed explanation of the above technical solution:

[0093] Suppose there is a video to be detected that is suspected of being a deepfake video, in which the texture details and lighting patterns of the face area may show subtle signs of tampering, and these signs may appear to have unnatural jumps between video frames.

[0094] First, the executing entity acquires the video to be analyzed. The video is then loaded into the processing module, ready for analysis.

[0095] Subsequently, the execution unit extracts facial features from each frame of the video to be detected. Specifically, a pre-trained face detection model accurately locates and crops out the facial regions in each video frame. These cropped facial images are further processed, for example, by using a convolutional neural network to extract facial feature maps containing basic information such as facial structure and expression. For instance, in a particular frame, the features of key facial features such as the eyes, nose, and mouth are encoded into numerical vectors.

[0096] Next, fine-grained attention feature extraction is performed on these extracted facial feature maps. In this step, an attention network is applied to the facial feature maps to identify and enhance minute texture details in the facial regions that may contain traces of forgery, such as skin pores, fine wrinkles, or facial blemishes. In this way, the agent is able to generate a fine-grained attention feature map that highlights the areas on the facial surface most likely to be tampered with; for example, in deepfake videos, skin textures may be too smooth or unnaturally blurred.

[0097] Simultaneously, illumination features are extracted from the facial feature map. In this step, the subject analyzes the brightness distribution and gradient changes of pixels in the facial feature map to infer the local illumination direction and intensity of the facial region. For example, by calculating the edge pixel density and average pixel density of different regions of the face, the local illumination direction can be solved using least squares and weighted fitting to obtain the illumination feature map. In real videos, the illumination pattern of a face is usually continuous and natural, while in fake videos, the illumination may be inconsistent with the background or show inconsistent changes between different frames.

[0098] Then, the fine-grained attention feature map and the illumination feature map are subjected to cross-attention processing. In this process, the fine-grained attention feature map serves as the query vector, and the illumination feature map is converted into a key vector and a value vector. By calculating the similarity between the query and the key, the executing agent can selectively fuse illumination information into the fine-grained texture features, thereby generating a cross-fused feature map. This fusion allows the executing agent to simultaneously consider the microscopic texture details of the face and its surrounding lighting environment, capturing potential inconsistencies between modalities. For example, a fake face might have realistic textures, but its shadow direction might not match the light source direction in the video.

[0099] Furthermore, decompositional spatiotemporal attention feature extraction is performed on the cross-fused feature map. This step first performs spatial attention encoding on the cross-fused feature map of each frame to capture intra-frame spatial features, such as the relationship between local regions of a face and its overall structure. Subsequently, these spatial attention feature sequences are temporally attention encoded to analyze the temporal dependencies between different video frames. For example, in real videos, facial movements, expression changes, and lighting transitions are smooth and continuous, while in deepfake videos, these temporal coherences may be disrupted, manifesting as unnatural jumps or flickering between frames. Through decompositional spatiotemporal attention, the executing agent can effectively capture these spatial and temporal anomalies.

[0100] Finally, based on the spatiotemporal attention feature, the deepfake detection result of the video to be detected is determined. The executing agent inputs the extracted features containing rich spatiotemporal information into a classifier. The classifier performs nonlinear combination and mapping on these features to generate raw scores corresponding to two preset categories, "real" and "fake," and further generates predicted probability distributions for each category. For example, if the predicted probability shows that the probability of the video being fake is much higher than the probability of it being real, the executing agent determines that the video to be detected is a deepfake video. In this way, the method can comprehensively utilize information from multiple aspects such as the subtle texture of the face, illumination consistency, and spatiotemporal coherence, thereby improving the detection accuracy of deepfake videos.

[0101] Based on the above examples, the deepfake video detection method proposed in this embodiment demonstrates significant technical contributions.

[0102] Existing technologies often struggle to effectively fuse information from different modalities when processing multimodal deepfake videos, limiting detection accuracy and real-time performance, and making it difficult to capture discontinuities or forgery traces between modalities. For example, some existing methods may focus only on facial texture while ignoring lighting conditions, or only on temporal sequences while neglecting intra-frame details.

[0103] This embodiment effectively solves the aforementioned problems by introducing fine-grained attention feature extraction and illumination feature extraction, and further fusing them through cross-attention processing. In the example above, when there is an inconsistency between the subtle texture of the face (represented by the fine-grained attention feature map) and the illumination pattern in the video (represented by the illumination feature map), for example, the direction of the face's shadow does not match the direction of the background light source, the cross-attention processing can sensitively capture this inconsistency between modalities. Compared to existing methods that simply stitch together or independently analyze different features, this can more deeply uncover the tampering traces left in deepfake videos, thereby improving the robustness of detection.

[0104] Furthermore, this embodiment employs decompositional spatiotemporal attention feature extraction to extract the spatiotemporal dependencies between video frames. In the example above, when the deepfake video exhibits unnatural motion, facial expression changes, or lighting transitions between frames, decompositional spatiotemporal attention can capture these anomalies from both spatial and temporal dimensions. This allows the executing agent to not only identify forgery traces within a single frame but also detect more subtle temporal inconsistencies across frames. Compared to existing methods that only use 3D convolution or independently process spatial and temporal information, this embodiment can utilize the spatiotemporal dynamics of the video more comprehensively and precisely, thereby significantly improving the detection capability for complex deepfake videos.

[0105] In summary, this embodiment constructs a more comprehensive and robust deepfake video detection framework through refined facial feature extraction, cross-fusion of multimodal features, and capture of decomposed spatiotemporal dependencies. This framework effectively addresses the challenges faced by existing technologies in multimodal deepfake detection, significantly improving detection accuracy and the ability to identify complex forgery techniques.

[0106] See Figure 3 In one embodiment, the method for fine-grained attention feature extraction of facial feature maps includes, but is not limited to, steps S301 to S304.

[0107] Step S301: Perform shallow feature extraction on the face feature map to obtain a shallow feature map.

[0108] Step S302: Extract texture features from the shallow feature map to obtain a texture feature map.

[0109] Step S303: Perform attention mechanism processing on the shallow feature map to obtain the face attention feature map.

[0110] Step S304: Randomly extract features from both the texture feature map and the face attention feature map, and fit the extracted features to obtain a fine-grained attention feature map.

[0111] Shallow feature extraction of facial feature maps aims to extract basic, low-level visual information from the original facial feature map. These shallow features typically include edges, corners, color gradients, etc., which are fundamental elements that constitute more complex textures and structures. This step can be implemented using one or more convolutional layers in a convolutional neural network. For example, the facial feature map can be scanned using 3x3 or 5x5 convolutional kernels, and non-linearity can be introduced through activation functions such as ReLU. Alternatively, these basic features can be extracted by applying a series of predefined filter banks (such as Gaussian filters or Sobel filters). Another approach is to input the facial feature map into a pre-defined attention operation network, where the input facial feature map is sequentially convolved, normalized, attention weighted, softmax weighted, and masked to obtain the corresponding shallow feature map.

[0112] Texture feature extraction from shallow feature maps aims to quantify and describe recurring patterns or structures in an image. This texture information is crucial for identifying forgery artifacts in deepfake videos, such as unnatural skin textures, blurred local areas, or excessive smoothing. This step can utilize traditional image processing techniques such as Local Binary Patterns (LBP), Gabor wavelet transform, or Gray-Level Co-occurrence Matrix (GLCM) to extract texture descriptors. Alternatively, specially designed convolutional layers can be used to learn and extract texture features, capturing texture information at different scales and orientations.

[0113] Applying an attention mechanism to shallow feature maps aims to enable the model to dynamically focus on the more important parts of the face region for deepfake detection, while suppressing irrelevant background or noise information. Introducing an attention mechanism enhances the model's ability to perceive key forgery regions. This step can employ channel attention mechanisms (such as the Squeeze-and-Excitation network module), which weights the channels of the feature map to highlight important channels; or spatial attention mechanisms (such as the spatial attention module in the Convolutional Block Attention Module), which generates a spatial weight map to allow the model to focus on key spatial locations within the feature map.

[0114] Random feature extraction from both the texture feature map and the face attention feature map, followed by feature fitting, aims to effectively fuse complementary information from different dimensions (texture and attention) to generate a more comprehensive and discriminative fine-grained feature representation. Random feature extraction helps reduce feature redundancy and improves the model's generalization ability to different feature combinations. Feature fitting aims to learn an optimal mapping relationship, integrating these extracted features into a unified fine-grained attention feature map. Random feature extraction can be performed by randomly selecting a subset of feature vectors, while feature fitting can reduce dimensionality and find the optimal feature combination through principal component analysis (PCA), or learn a non-linear fusion mapping through a small fully connected neural network (multilayer perceptron).

[0115] This application's solution effectively addresses the problem of accurately identifying subtle forgery traces by relying solely on general facial features through multi-stage, multi-dimensional fine-grained feature extraction of facial feature maps. First, shallow feature extraction provides the foundational visual elements for subsequent texture and attention analysis. Building upon this, texture feature extraction specifically captures texture anomalies commonly found in deepfake videos, while the attention mechanism dynamically focuses on crucial local information within the facial region. By randomly extracting and fitting these two complementary feature maps (texture feature map and facial attention feature map), this method comprehensively utilizes the saliency information of texture details and key regions to generate a fine-grained attention feature map that contains rich texture details while highlighting important areas. This fusion approach enables the model to more comprehensively and accurately capture subtle, imperceptible forgery traces in deepfake videos, significantly improving detection accuracy and robustness.

[0116] The following is a concrete example. As a specific implementation, shallow feature extraction from a face feature map can be achieved using a convolutional block containing two convolutional layers (e.g., each using a 3x3 kernel and outputting 32 and 64 feature channels respectively), followed by a batch normalization layer and a ReLU activation function. Texture feature extraction from the shallow feature map can utilize a series of Gabor filter banks containing filters with different orientations (e.g., 0°, 45°, 90°, 135°) and frequencies to comprehensively capture texture information, followed by downsampling through a max-pooling layer. Attention processing of the shallow feature map can employ a Squeeze-and-Excitation (SE) module. This module first performs global average pooling on the shallow feature map, then generates channel attention weights through two fully connected layers (one for dimensionality reduction and one for dimensionality increase using a sigmoid activation function), and finally multiplies these weights back onto the shallow feature map to obtain the face attention feature map. For random feature extraction and fitting, a fixed number (e.g., 1024 each) of feature vectors can be randomly extracted from the texture feature map and the face attention feature map, concatenated, and then input into a neural network consisting of two fully connected layers (e.g., with 512 and 256 neurons respectively, using the ReLU activation function) for fitting, and finally outputting a fine-grained attention feature map.

[0117] By employing the aforementioned technical solution, after extracting facial features from the video frames of the video to be detected, further shallow feature extraction, texture feature extraction, attention mechanism processing, and random feature extraction and fitting of the facial feature map can effectively capture subtle texture anomalies and key local features from the facial region. This multi-level, multi-angle feature extraction method, especially the combination of texture information and attention mechanism, enables the model to more accurately focus on common, hard-to-detect forgery traces in deepfake videos, such as unnatural skin texture, inconsistent local lighting, or stiff facial expressions. Random feature extraction and fitting further enhance the robustness and discriminative power of the features, avoiding the limitations of single features, thereby significantly improving the detection accuracy and generalization ability of deepfake videos, effectively solving the technical problem that relying solely on general facial features makes it difficult to accurately identify subtle forgery traces.

[0118] See Figure 4 In one embodiment, the method for extracting illumination features from a face feature map includes, but is not limited to, steps S401 to S404.

[0119] Step S401: Convert the face feature map into the corresponding face grayscale feature map.

[0120] Step S402: Extract key contour features from the grayscale feature map of the face to obtain a contour feature map.

[0121] Step S403: Calculate the edge pixel density and average pixel density of multiple local regions in the contour feature map.

[0122] Step S404: Based on the edge pixel density and average pixel density, perform least squares solution and weighted fitting on the local illumination direction of the local area to obtain the illumination feature map.

[0123] Converting a facial feature map into a corresponding grayscale facial feature map serves to simplify the feature data by transforming a multi-channel facial feature map into a single-channel grayscale representation. This reduces the computational complexity of subsequent processing and allows the analysis to focus more on the brightness or intensity information of the feature map, rather than the complex combination of color or multiple channels. For example, a grayscale image can be generated by weighted averaging of the various channels of the facial feature map, or by extracting its principal intensity components as the grayscale representation through principal component analysis (PCA).

[0124] Key contour feature extraction from grayscale facial feature maps aims to identify boundary information with significant changes in brightness or intensity. These contours typically correspond to geometric variations, material differences, or lighting and shadow boundaries on an object's surface, and are crucial for analyzing lighting conditions. For example, edge detection algorithms such as the Canny operator, Sobel operator, or Prewitt operator can be used to identify and extract key contours from the image.

[0125] To calculate the edge pixel density and average pixel density of multiple local regions in a contour feature map, the contour feature map can be divided into several local regions of equal size. Then, the ratio of the number of non-zero pixels (edge ​​pixels) in each region to the total number of pixels can be used as the edge pixel density, and the ratio of the sum of all pixel values ​​to the total number of pixels can be used as the average pixel density.

[0126] Based on edge pixel density and average pixel density, the local illumination direction of a local region is solved using least squares and weighted fitting. This can be achieved by employing least squares optimization techniques to infer the local illumination direction that best matches these observations from information such as edge pixel density and average pixel density. Then, through weighted fitting, different data points (e.g., different local regions) are assigned different weights to reflect their reliability or importance in the illumination direction estimation; for example, regions with richer edge information may be given higher weights. In this way, by comprehensively utilizing the texture and brightness information of the local region, the illumination direction of each local region is robustly estimated, ultimately forming an illumination feature map representing the overall illumination distribution.

[0127] This application's solution employs a series of refined processing steps to accurately extract illumination features from facial feature maps, overcoming the shortcomings of traditional methods in distinguishing between real and fake illumination. First, the facial feature map is processed into grayscale, simplifying complex feature information into more easily analyzable intensity representations. This removes potential interference from color or multi-channel features, allowing subsequent illumination analysis to focus more on brightness changes themselves. Based on this, key contour feature extraction identifies the edge information in the facial grayscale feature map that is most sensitive to illumination changes. These contours directly reflect changes in illumination direction and intensity, effectively capturing subtle light-dark boundaries and shadow distributions on the facial surface. Subsequently, by calculating the edge pixel density and average pixel density of multiple local regions in these contour feature maps, the texture complexity and overall brightness information of the local regions are quantified. These density values ​​provide statistical clues about the local illumination environment. Finally, based on this local statistical information, a least-squares solution and weighted fitting method are used to accurately estimate the local illumination direction of each local region. Least squares optimization ensures the mathematical optimality of the illumination direction estimation, while weighted fitting allows adjustments based on the reliability of local region information (e.g., regions with high edge density may provide more reliable illumination cues), thereby improving the robustness and accuracy of the illumination direction estimation. Through this progressive, coarse-to-fine analysis, this scheme can extract high-quality illumination feature maps from fine-grained attention feature maps. These illumination feature maps more accurately reflect the real illumination conditions of the face region in the video frame, providing a crucial and reliable basis for subsequent deepfake detection.

[0128] The following is a concrete example. When processing a face feature map in grayscale, a weighted average method can be used. For example, if the face feature map has multiple channels, the pixel values ​​of each channel can be weighted and summed according to preset weights (e.g., simulating the human eye's sensitivity to different colors) to obtain a single-channel face grayscale feature map. Next, when extracting key contour features from the face grayscale feature map, the Canny edge detection algorithm can be used. This algorithm first performs Gaussian smoothing on the face grayscale feature map to remove noise, then calculates the image gradient intensity and direction, followed by non-maximum suppression to refine the edges, and finally determines the final contour feature map through double thresholding and edge tracking. The lower threshold can be set to 50, and the higher threshold to 150. When calculating the edge pixel density and average pixel density of multiple local regions in the contour feature map, the contour feature map can be divided into several 16x16 pixel local regions. For each local region, the number of non-zero pixels (i.e., edge pixels detected by the Canny algorithm) is counted, and this number is divided by the total number of pixels in that region (16x16=256) to obtain the edge pixel density. Simultaneously, the average value of all pixel values ​​within the local region is calculated as the average pixel density. Finally, based on these edge pixel densities and the average pixel density, the local illumination direction of the local region is solved using least squares and weighted fitting. For example, a simplified illumination model can be constructed, assuming that the brightness variation in the local region is related to the illumination direction and surface normal, and then the least squares method is used to solve for the local illumination direction vector that best matches the observed density value. During the weighted fitting process, different weights can be assigned according to the edge pixel density of the local region; for example, regions with higher edge pixel density have a greater weight in their illumination direction estimation results, thus giving more emphasis to this information during the fitting process, ultimately generating the illumination feature map.

[0129] Through the above technical solution, this application can extract illumination features more accurately and robustly from facial feature maps. The combination of grayscale processing and key contour feature extraction enables illumination analysis to effectively avoid interference from non-illumination factors such as texture and color, focusing on the light and dark boundaries and intensity changes formed by the light itself in the image. By calculating the edge pixel density and average pixel density of local regions, this solution can quantify the subtle features of the local illumination environment. Furthermore, based on these local statistical information, least squares solving and weighted fitting can accurately estimate the local illumination direction, thereby generating a high-quality illumination feature map. This illumination feature map can more realistically reflect the lighting conditions of the face region in the video frame, effectively capturing common lighting inconsistencies or unnatural phenomena in deepfake videos. Compared with direct analysis relying solely on fine-grained attention feature maps, the illumination features extracted by this solution have higher discriminative power, significantly enhancing the accuracy and robustness of deepfake video detection, especially providing a more reliable basis for discrimination when facing complex lighting environments or high-quality forged videos.

[0130] See Figure 5 In one embodiment, the method for cross-attention processing of fine-grained attention feature maps and illumination feature maps includes, but is not limited to, steps S501 to S503.

[0131] Step S501: Determine the self-attention features as cross-attention query vectors.

[0132] Step S502: Transform the illumination feature map into a cross-attention key vector and a cross-attention value vector.

[0133] Step S503: Perform cross-attention processing based on the cross-attention query vector, cross-attention key vector, and cross-attention value vector to obtain the cross-fusion feature map.

[0134] This application's solution achieves deep fusion of fine-grained attention feature maps and illumination feature maps by introducing cross-attention processing. Specifically, the fine-grained attention feature map is processed to form a self-attention feature, which is used as a query vector. This query vector represents detailed information such as fine texture and expression of the facial region, actively "asking" for relevant illumination information in the illumination feature map. Simultaneously, the illumination feature map is converted into a key vector and a value vector, where the key vector responds to the query, and the value vector provides the actual illumination content. During the cross-attention processing, the query vector is matched with the key vector to calculate attention weights. These weights indicate which parts of the fine-grained attention feature map should focus on which areas of the illumination feature map. Subsequently, these weights are applied to the value vector, selectively extracting and fusing the most relevant information from the illumination feature map to the fine-grained attention feature map. This mechanism enables the model to intelligently associate facial details with local illumination conditions. For example, when facial textures display a certain illumination pattern, but the illumination feature map shows inconsistent lighting direction or intensity, this mismatch can be effectively captured. In this way, the cross-fusion feature map not only contains independent information of the two features, but more importantly, it encodes the interdependence and potential inconsistencies between them, providing a more discriminative input for subsequent decompositional spatiotemporal attention feature extraction, thereby enhancing the ability to detect lighting forgery traces in deepfake videos.

[0135] As a specific implementation, when performing cross-attention processing, the fine-grained attention feature map can first be passed through a convolutional layer and a self-attention module to generate self-attention features, which are then determined as the cross-attention query vector. Simultaneously, the illumination feature map can be passed through two independent 1x1 convolutional layers to generate cross-attention key vectors and cross-attention value vectors, respectively. For example, for a fine-grained attention feature map of size H x W x C, it can first be flattened into a (H*W) x C sequence, then input into a multi-head self-attention layer, with the output features serving as the query vector. A similar flattening operation can be performed on the illumination feature map, followed by two different linear transformation layers to generate key and value vectors. Next, cross-attention processing is performed using the standard scaled dot product attention mechanism: the dot product of the query vector and the key vector is calculated, scaled, and normalized using the Softmax function to obtain attention weights. These weights are then multiplied by the value vectors and summed to finally obtain the cross-fused feature map. For example, this can be achieved using the cross-attention layer in the Transformer architecture, where the query Q comes from the self-attention feature, and the key K and value V come from the illumination feature map.

[0136] Through the above technical solution, this application can effectively fuse fine-grained attention feature maps and illumination feature maps, solving the problem of difficulty in deep interaction between different feature information in traditional methods. This cross-attention processing mechanism enables the model to actively associate facial details with illumination information, thereby more sensitively capturing forgery traces common in deepfake videos where facial textures do not match the lighting environment. For example, when a facial region in a forged video exhibits certain illumination features in its texture details, but its corresponding illumination feature map shows a lighting direction or intensity inconsistent with the real environment, this inconsistency can be clearly reflected by the cross-fused feature map. This significantly improves the accuracy and robustness of deepfake video detection, especially in scenarios with complex lighting conditions or sophisticated forgery techniques, providing a more reliable basis for judgment.

[0137] In some embodiments, decompositional spatiotemporal attention feature extraction of the cross-fused feature map includes: spatial attention encoding of the cross-fused feature map to extract spatial features in each video frame to obtain spatial attention features; and temporal attention encoding of the spatial attention features to extract the spatiotemporal dependencies between each video frame to obtain spatiotemporal attention features.

[0138] Spatial attention coding is a mechanism for extracting spatial features within a single frame. It can be implemented using self-attention mechanisms or convolutional neural networks. Its purpose is to focus on key regions in a video frame to identify anomalous features in the spatial dimension.

[0139] Temporal attention coding is a mechanism for modeling inter-frame temporal dependencies. It can be implemented using recurrent neural networks or temporal self-attention mechanisms. Its purpose is to capture the dynamic changes in video sequences and avoid the high computational overhead of joint modeling.

[0140] The proposed solution first employs spatial attention encoding based on cross-fused feature maps. Since these feature maps incorporate fine-grained texture information from key facial regions, they guide the model to precisely focus on forgery-sensitive areas such as the eyes and lips, which are prone to artifacts. This effectively identifies spatial anomalies within a single frame. Subsequently, temporal attention encoding is applied to the generated spatial attention features. Because the spatial features have extracted the core spatial information for each frame, the scale of the temporal input data is significantly reduced. This allows the calculation of temporal dependencies to be performed only on the simplified feature sequence, avoiding the problem of computational complexity increasing quadratically with the number of frames in the joint spatiotemporal attention mechanism. Simultaneously, it ensures that the analysis of the coherence of dynamic changes between frames is not interfered with by redundant spatial information, ultimately achieving high sensitivity in capturing spatiotemporal inconsistencies in forged videos.

[0141] As a specific implementation method, the scheme of this application is implemented as follows: In the spatial attention encoding step, a multi-head self-attention mechanism is adopted, wherein the query, key and value vectors are obtained by linear transformation of the cross-fusion feature map; in the temporal attention encoding step, the spatial attention features are serialized and input into the temporal self-attention module, and the inter-frame dependency features are extracted by calculating the similarity relationship between each time point in the sequence.

[0142] Through the above scheme, this application effectively reduces the computational complexity in the deepfake video detection process, improves detection efficiency and real-time performance, and strengthens the ability to separate and identify spatial local anomalies and temporal discontinuities in fake videos, thereby capturing subtle forgery traces more accurately.

[0143] In some embodiments, spatial attention encoding of the cross-fused feature map includes: spatial feature embedding of the cross-fused feature map to obtain spatial embedded features; matrix transformation processing of the spatial embedded features to obtain spatial self-attention query vector, spatial self-attention key vector, and spatial self-attention value vector; and self-attention mechanism processing based on the spatial self-attention query vector, spatial self-attention key vector, and spatial self-attention value vector to obtain spatial attention features.

[0144] Spatial feature embedding refers to the process of converting cross-fused feature maps into structured feature representations. It can be implemented using linear projection layers or convolutional neural network layers. The purpose is to standardize feature dimensions and reduce noise interference, providing a clear input basis for subsequent processing.

[0145] Matrix transformation processing can be understood as the operation of linearly transforming spatially embedded features to generate query, key, and value vectors. It can be implemented using fully connected layers or matrix multiplication operations. The purpose is to accurately extract the semantic relationships between features and ensure that the vectors can reflect spatial positional relationships.

[0146] Self-attention mechanism processing refers to the process of dynamically calculating attention weights based on query, key, and value vectors. It can be implemented using dot product attention mechanism or scaled dot product attention mechanism. The purpose is to adaptively focus on key regions in video frames and enhance the ability to respond to local anomalies.

[0147] This application's solution forms a complete spatial feature extraction process by sequentially performing spatial feature embedding, matrix transformation, and self-attention mechanism processing on the cross-fused feature map. First, spatial feature embedding transforms the original features into a structured representation, preserving the high-attention characteristics of forgery-sensitive regions. Second, matrix transformation generates query, key, and value vectors from the embedded features; these vectors accurately capture the dynamic consistency between spatial locations. Finally, the self-attention mechanism calculates attention weights using vector similarity, enabling the model to focus on high-probability forgery areas such as around facial keypoints. This step-by-step processing mechanism effectively reduces computational complexity while ensuring efficient capture of local forgery traces, thereby improving the accuracy and real-time performance of spatial feature extraction.

[0148] As a specific implementation method, the solution of this application is implemented as follows: In the spatial feature embedding step, a 1×1 convolutional layer can be used to perform feature transformation on the cross-fused feature map. In the matrix transformation processing step, three independent fully connected layers can be used to generate spatial self-attention query vector, spatial self-attention key vector, and spatial self-attention value vector, respectively. In the self-attention mechanism processing step, attention weights can be calculated based on the generated vectors by scaling the dot product, and the spatial attention features can be obtained by weighted summation of the value vectors.

[0149] Through the above scheme, this application can effectively reduce the computational resource consumption of spatial attention encoding, while accurately focusing on the forgery-sensitive areas around facial key points, so that local forgery traces can be captured efficiently, thereby significantly improving the accuracy and real-time performance of deepfake video detection.

[0150] In some embodiments, temporal attention encoding of spatial attention features includes: embedding spatial attention features into temporal features to obtain temporal embedded features; performing matrix transformation processing on the temporal embedded features to obtain a temporal self-attention query vector, a temporal self-attention key vector, and a temporal self-attention value vector; and performing self-attention mechanism processing based on the temporal self-attention query vector, the temporal self-attention key vector, and the temporal self-attention value vector to obtain spatiotemporal attention features.

[0151] Temporal feature embedding refers to the technique of transforming features extracted from the spatial dimension into a time series representation. It can be implemented using positional encoding or recurrent neural networks. The purpose is to retain spatial structure information while introducing the temporal dimension, laying the foundation for efficiently capturing dynamic changes between frames.

[0152] The proposed solution first embeds spatial attention features into temporal features, transforming the features extracted from the spatial dimension into a time-series representation. This allows the features to naturally adapt to subsequent time analysis, avoiding redundant computation caused by directly processing the raw data. Based on this, matrix transformation is performed on the embedded temporal features to accurately generate query, key, and value vectors reflecting temporal relationships, reducing computational complexity while maintaining the robustness of feature extraction. Finally, a self-attention mechanism is applied to these specific vectors, dynamically weighting time information to focus on key temporal patterns and generating highly discriminative spatiotemporal attention features, providing reliable temporal evidence for deepfake detection.

[0153] As a specific implementation method, the scheme of this application is implemented as follows: Temporal feature embedding can be specifically implemented through sinusoidal positional encoding, adding spatial attention features to the positional encoding vector to introduce the temporal dimension. Matrix transformation processing can specifically employ three independent fully connected layers to generate a temporal self-attention query vector, a temporal self-attention key vector, and a temporal self-attention value vector, respectively. The self-attention mechanism processing can specifically be implemented by calculating the dot product of the query vector and the key vector, normalizing it with softmax, and then multiplying it with the value vector, thereby dynamically weighting temporal information and generating spatiotemporal attention features.

[0154] The above technical solution effectively optimizes the extraction process of time dependencies between video frames, reduces computational resource consumption while ensuring the accuracy of deepfake detection, and enables the execution entity to efficiently process dynamic change features in video frame sequences, thereby improving the real-time performance and reliability of detection.

[0155] In some embodiments described above, this application proposes a method for determining the deepfake detection result of a video under test based on spatiotemporal attention features. However, existing deepfake video detection methods, when determining the detection result based on spatiotemporal attention features, may not effectively handle the nonlinear relationships of features by directly using simple classification, leading to unstable detection results or low accuracy, especially challenging the identification of subtle forgery traces in deepfake videos. Since spatiotemporal features contain complex high-dimensional patterns, without targeted nonlinear transformation and probabilistic modeling mechanisms, it is difficult to reliably map features into interpretable classification criteria, thus affecting the robustness and adaptability of the detection system. Therefore, this application further proposes a method for determining the deepfake detection result of a video under test.

[0156] In some embodiments, determining the deepfake detection result of the video to be detected based on spatiotemporal attention features includes: performing nonlinear combination and mapping on the spatiotemporal features contained in the spatiotemporal attention features to obtain the original scores corresponding to each preset category; generating a predicted probability distribution of the spatiotemporal attention features belonging to each preset category based on the original scores; and determining the deepfake detection result of the video to be detected based on the predicted probability distribution.

[0157] After obtaining the spatiotemporal attention features extracted through decompositional spatiotemporal attention feature extraction, the proposed method transforms these high-dimensional and complex features into interpretable deepfake detection results through a series of processing steps. First, the spatiotemporal features contained within the spatiotemporal attention features are nonlinearly combined and mapped. This process aims to capture deeper nonlinear relationships between features and project them into a latent space associated with preset categories (e.g., "real" and "fake"), thereby obtaining raw scores corresponding to each preset category. These raw scores represent the model's initial judgment of the input video belonging to each category, and their magnitude reflects the model's bias towards that category. Subsequently, based on these raw scores, a predicted probability distribution of the spatiotemporal attention features belonging to each preset category is generated through probability normalization (e.g., using the Softmax function). This step converts the raw scores into statistically significant probability values, making the probability of each category intuitively representable, and ensuring that the sum of the probabilities of all categories is 1. Finally, based on this predicted probability distribution, the deepfake detection result of the video to be detected is determined by selecting the category with the highest probability or comparing it with a preset threshold. The entire process effectively transforms complex spatiotemporal features into clear classification judgments, thereby solving the problem of how to efficiently and accurately convert features into reliable detection results.

[0158] As a specific implementation method, the above-mentioned technical means can be implemented with reference to the following example. After obtaining the spatiotemporal attention features, they can be input into a multilayer perceptron (MLP) network. This MLP network contains multiple fully connected layers, each followed by a non-linear activation function, such as ReLU. Through the non-linear combination and mapping of these layers, a vector equal to the preset number of categories is finally output, and each element of this vector is the original score of the corresponding category. For example, for a binary classification problem (real / fake), two original scores are output. Subsequently, these two original scores are input into a softmax layer, which converts the original scores into two probability values, representing the probability that the video is real and fake, respectively. Finally, the magnitudes of these two probability values ​​are compared, and the category with the highest probability value is selected as the final detection result. For example, if the probability of the fake category is 0.8 and the probability of the real category is 0.2, then the video is determined to be a deepfake video.

[0159] Through the above technical solution, this application can effectively transform complex spatiotemporal attention features into predictive probability distributions with clear physical meaning, thereby achieving accurate classification of deepfake videos. Nonlinear combination and mapping can fully exploit the deep discriminative information contained in spatiotemporal features, improving the model's ability to identify subtle differences between real and fake videos. Generating predictive probability distributions makes the detection results more interpretable and provides a quantitative basis for subsequent decision-making. Finally, the detection results are determined based on the probability distribution, ensuring the robustness and accuracy of the detection process and significantly improving the overall performance of deepfake video detection.

[0160] See Figure 6 This application also provides a deepfake video detection device that can implement the above-described deepfake video detection method. The device includes:

[0161] The first module 601 is used to acquire the video to be detected;

[0162] The second module 602 is used to extract facial features from video frames of the video to be detected, and obtain facial feature maps.

[0163] The third module 603 is used to extract fine-grained attention features from the face feature map to obtain a fine-grained attention feature map.

[0164] The fourth module 604 is used to extract illumination features from the face feature map to obtain the illumination feature map;

[0165] The fifth module 605 is used to perform cross-attention processing on the fine-grained attention feature map and the illumination feature map to obtain a cross-fused feature map;

[0166] The sixth module 606 is used to perform decompositional spatiotemporal attention feature extraction on the cross-fused feature map in order to extract the spatiotemporal dependencies between video frames and obtain spatiotemporal attention features.

[0167] Module 7, 607, is used to determine the depth forgery detection result of the video to be detected based on spatiotemporal attention features.

[0168] The specific implementation of this deepfake video detection device is basically the same as the specific implementation of the deepfake video detection method described above, and will not be repeated here.

[0169] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0170] The following reference Figure 7 To describe an electronic device 700 according to such an embodiment of the present disclosure. Figure 7The electronic device 700 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0171] like Figure 7 As shown, the electronic device 700 is presented in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including storage unit 720 and processing unit 710), a display unit 740, etc.

[0172] The storage unit stores program code, which can be executed by the processing unit 710, causing the processing unit 710 to perform the steps described in the above-described section of the deepfake video detection method according to various exemplary embodiments of this disclosure.

[0173] Storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 7201 and / or a cache memory unit 7202, and may further include a read-only memory unit (ROM) 7203.

[0174] The storage unit 720 may also include a program / utility 7204 having a set (at least one) program module 7205, such program module 7205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0175] Bus 730 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0176] Electronic device 700 can also communicate with one or more external devices 700' (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 700, and / or with any device that enables electronic device 700 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 750. Furthermore, electronic device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 760. Network adapter 760 can communicate with other modules of electronic device 700 via bus 730. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0177] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0178] The deepfake video detection method, apparatus, device, and medium provided in this application introduce fine-grained attention feature extraction and illumination feature extraction, and further fuse them deeply through cross-attention processing. Then, decompositional spatiotemporal attention feature extraction is performed on the cross-fused feature map to extract the spatiotemporal dependencies between video frames. Finally, the deepfake detection result of the video to be detected is determined based on the spatiotemporal attention features. Therefore, when there is inconsistency between the subtle textures of a face and the illumination pattern in the video, cross-attention processing can sensitively capture this intermodal inconsistency, enabling a deeper exploration of tampering traces left in the deepfake video, thereby improving the robustness of detection. Furthermore, by employing decompositional spatiotemporal attention feature extraction to extract the spatiotemporal dependencies between video frames, when the deepfake video exhibits unnatural motion, facial expression changes, or illumination transitions between frames, decompositional spatiotemporal attention can capture these anomalies from both spatial and temporal dimensions. This allows for a more comprehensive and refined utilization of the video's spatiotemporal dynamic information, significantly improving the detection capability for complex deepfake videos and enhancing the accuracy and real-time performance of the deepfake detection method.

[0179] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this disclosure.

[0180] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0181] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0182] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0183] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A method for detecting deepfake videos, characterized in that, include: Obtain the video to be tested; Facial features are extracted from the video frames of the video to be detected to obtain a facial feature map; Fine-grained attention feature extraction is performed on the facial feature map to obtain a fine-grained attention feature map; Illumination features are extracted from the facial feature map to obtain an illumination feature map; Cross-attention processing is performed on the fine-grained attention feature map and the illumination feature map to obtain a cross-fused feature map; The cross-fusion feature map is subjected to decompositional spatiotemporal attention feature extraction to extract the spatiotemporal dependencies between the video frames and obtain spatiotemporal attention features; Based on the spatiotemporal attention features, the deepfake detection result of the video to be detected is determined; The step of extracting illumination features from the facial feature map includes: The facial feature map is converted into a corresponding grayscale facial feature map; Key contour features are extracted from the grayscale feature map of the face to obtain a contour feature map; Calculate the edge pixel density and average pixel density of multiple local regions in the contour feature map; the edge pixel density is the ratio of the number of non-zero pixels in each local region to the total number of pixels when the contour feature map is divided into several equally sized local regions; the average pixel density is the ratio of the sum of all pixel values ​​in the local region to the total number of pixels. Based on the edge pixel density and the average pixel density, the local illumination direction of the local region is solved by least squares and weighted fitting to obtain the illumination feature map. The decompositional spatiotemporal attention feature extraction of the cross-fused feature map includes: Spatial attention encoding is performed on the cross-fused feature map to extract spatial features from each video frame, thus obtaining spatial attention features; Temporal attention encoding is performed on the spatial attention features to extract the spatiotemporal dependencies between the video frames, thus obtaining the spatiotemporal attention features.

2. The deepfake video detection method according to claim 1, characterized in that, The fine-grained attention feature extraction of the facial feature map includes: Shallow feature extraction is performed on the face feature map to obtain a shallow feature map; Texture features are extracted from the shallow feature map to obtain a texture feature map; The shallow feature map is processed by an attention mechanism to obtain a face attention feature map; Random feature extraction is performed on both the texture feature map and the face attention feature map to randomly extract a fixed number of features from the texture feature map and the face attention feature map. The extracted features are then fitted to obtain the fine-grained attention feature map. The fitting is performed by reducing dimensionality and finding the optimal feature combination through principal component analysis or by nonlinear fusion mapping through a fully connected neural network.

3. The deepfake video detection method according to claim 1, characterized in that, The cross-attention processing of the fine-grained attention feature map and the illumination feature map includes: The self-attention feature is determined as the cross-attention query vector; the self-attention feature is obtained by processing the fine-grained attention feature map; The illumination feature map is transformed into a cross-attention key vector and a cross-attention value vector; Cross-attention processing is performed based on the cross-attention query vector, the cross-attention key vector, and the cross-attention value vector to obtain the cross-fusion feature map.

4. The deepfake video detection method according to claim 1, characterized in that, The spatial attention encoding of the cross-fused feature map includes: Spatial feature embedding is performed on the cross-fusion feature map to obtain spatial embedding features; The spatial embedding features are subjected to matrix transformation processing to obtain spatial self-attention query vector, spatial self-attention key vector, and spatial self-attention value vector; The spatial attention features are obtained by performing self-attention mechanism processing based on the spatial self-attention query vector, the spatial self-attention key vector, and the spatial self-attention value vector. And / or the temporal attention encoding of the spatial attention features includes: Temporal feature embedding is performed on the spatial attention features to obtain temporal embedding features; The temporal embedding features are subjected to matrix transformation to obtain a temporal self-attention query vector, a temporal self-attention key vector, and a temporal self-attention value vector; The spatiotemporal attention features are obtained by performing self-attention mechanism processing based on the temporal self-attention query vector, the temporal self-attention key vector, and the temporal self-attention value vector.

5. The deepfake video detection method according to claim 1, characterized in that, The step of determining the deepfake detection result of the video to be detected based on the spatiotemporal attention features includes: The spatiotemporal features contained in the spatiotemporal attention features are nonlinearly combined and mapped to obtain the original scores corresponding to each preset category; Based on the original scores, a predicted probability distribution of the spatiotemporal attention features belonging to each of the preset categories is generated; Based on the predicted probability distribution, the depth forgery detection result of the video to be detected is determined.

6. A deepfake video detection device, characterized in that, include: The first module is used to acquire the video to be detected; The second module is used to extract facial features from the video frames of the video to be detected, and obtain a facial feature map; The third module is used to extract fine-grained attention features from the face feature map to obtain a fine-grained attention feature map. The fourth module is used to extract illumination features from the face feature map to obtain an illumination feature map; The fifth module is used to perform cross-attention processing on the fine-grained attention feature map and the illumination feature map to obtain a cross-fused feature map; The sixth module is used to perform decompositional spatiotemporal attention feature extraction on the cross-fusion feature map to extract the spatiotemporal dependencies between the video frames and obtain spatiotemporal attention features. The seventh module is used to determine the depth forgery detection result of the video to be detected based on the spatiotemporal attention features; The step of extracting illumination features from the facial feature map includes: The facial feature map is converted into a corresponding grayscale facial feature map; Key contour features are extracted from the grayscale feature map of the face to obtain a contour feature map; Calculate the edge pixel density and average pixel density of multiple local regions in the contour feature map; the edge pixel density is the ratio of the number of non-zero pixels in each local region to the total number of pixels when the contour feature map is divided into several equally sized local regions; the average pixel density is the ratio of the sum of all pixel values ​​in the local region to the total number of pixels. Based on the edge pixel density and the average pixel density, the local illumination direction of the local region is solved by least squares and weighted fitting to obtain the illumination feature map. The decompositional spatiotemporal attention feature extraction of the cross-fused feature map includes: Spatial attention encoding is performed on the cross-fused feature map to extract spatial features from each video frame, thus obtaining spatial attention features; Temporal attention encoding is performed on the spatial attention features to extract the spatiotemporal dependencies between the video frames, thus obtaining the spatiotemporal attention features.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the deepfake video detection method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the deepfake video detection method according to any one of claims 1 to 5.

Images