Deep counterfeit video detection system based on combination of space domain and frequency domain and detection method thereof

Abstract

The invention discloses a deep counterfeit video detection system based on combination of a space domain and a frequency domain and a detection method thereof, and belongs to the technical field of computer vision and deep learning. The data pre-processing module is used for performing pre-processing and feature enhancement on the video with the face in the data set to obtain a training and testing data set; the double-flow network module comprises a spatial domain flow module and a frequency domain flow module, the two flows work in parallel, and specific characteristics of a spatial domain and a frequency domain are mined respectively; the feature fusion module is used for realizing effective fusion of spatial domain features and frequency domain features; and the classifier module receives the final fusion feature and outputs a detection result of the deep forged video. The method is used for solving the problems that in an existing double-flow network deep counterfeiting detection method, specific features of a space domain and specific features of a frequency domain are not fully mined, and in the prior art, the feature fusion effect of the two domains is poor.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning technology, specifically relating to a deep fake video detection system and method based on the combination of spatial and frequency domains. Background Technology

[0002] With the rapid development of deep learning technology, deepfake technology can generate highly realistic fake videos, posing a serious threat to social security, personal privacy and information authenticity. Therefore, deepfake detection technology has become a research hotspot in the fields of computer vision and deep learning.

[0003] Current deepfake detection methods are mainly divided into two categories: single-stream network-based methods and two-stream network-based methods. Single-stream network-based methods focus solely on extracting features from a single dimension in either the spatial or frequency domain for deepfake detection. Spatial domain correlation methods primarily focus on spatial features such as local details, edges, and textures, while frequency domain correlation methods emphasize frequency features such as global structure, illumination, and color distribution, mining artifact information in the frequency domain for detection. Two-stream network-based methods extract spatial and frequency domain features through two parallel network branches: a spatial stream and a frequency stream. These two types of features are then fused, utilizing the feature information from both domains for deepfake detection. Compared to single-stream network methods, they can capture artifact features more comprehensively, resulting in relatively better detection performance.

[0004] However, existing technologies have significant drawbacks and limitations. Single-stream network-based methods, focusing only on features in the spatial or frequency domains, cannot comprehensively capture artifact features in different domains of deepfake videos, resulting in poor detection accuracy and robustness, making them ill-suited to complex and diverse deepfake techniques. While dual-stream network-based methods extract and fuse features from the spatial and frequency domains separately, they fail to fully exploit the unique features of each domain and lack specific designs for the artifact patterns specific to each domain, leading to insufficient feature extraction. Furthermore, the lack of effective fusion strategies during the feature fusion stage prevents the full utilization of complementary information from the two domains, further impacting detection accuracy and robustness. Summary of the Invention

[0005] This invention provides a deepfake video detection system and method based on spatial-frequency domain combination. Firstly, it addresses the problem of insufficient feature extraction in the spatial and frequency domains in existing dual-stream network deepfake detection methods, achieving full extraction of unique artifact features in both domains. Secondly, it provides an effective feature fusion strategy to solve the problem of poor feature fusion between the two domains in existing technologies, achieving complementary fusion and mutual enhancement of spatial and frequency domain features, and fully utilizing the complementary information of the two domains. Thirdly, it improves the accuracy and robustness of deepfake video detection, enhancing the method's generalization ability to different types and levels of deepfake videos.

[0006] This invention is achieved through the following technical solution: A deepfake video detection system based on spatial-frequency domain combination, the detection system includes a data preprocessing module, a dual-stream network module, a feature fusion module, and a classifier module; The data preprocessing module is used to preprocess and enhance the features of videos with faces in the dataset to obtain training and testing datasets. The dual-stream network module includes a spatial domain stream module and a frequency domain stream module. The two streams work in parallel to mine the unique features of the spatial domain and the frequency domain, respectively. The feature fusion module adopts a two-level fusion strategy, including an adaptive complementary fusion module and a multi-scale fusion module, to achieve effective fusion of spatial domain features and frequency domain features. The classifier module receives the final fused features and outputs the detection results of the deepfake video.

[0007] Furthermore, the spatial domain flow module adopts a phased design, starting from the input RGB face image and sequentially going through five stages: initial feature extraction, basic feature enhancement, multi-scale feature extraction, artifact perception multi-scale attention processing, and deep semantic extraction, to fully explore the unique artifact features in the spatial domain. Furthermore, the initial feature extraction includes three convolutional layers, a batch normalization layer, and a ReLU activation function; The basic feature enhancement includes two residual blocks, each containing two convolutional layers, a batch normalization layer, and a ReLU activation function; The multi-scale feature extraction uses a convolutional layer with a stride of 2 for downsampling; The artifact-aware multi-scale attention processing is used to adaptively focus on the regions and scales in the image where forgery traces exist, and to enhance the features. The deep semantic extraction includes two convolutional layers, a batch normalization layer, and a ReLU activation function.

[0008] Furthermore, the spatial domain flow module sequentially employs an artifact detection network module, a multi-scale feature extraction sub-module, and a triple attention mechanism sub-module to perform artifact perception multi-scale attention processing; The artifact detection network module generates an "artifact probability map" to guide the subsequent attention mechanism; The multi-scale feature extraction submodule processes the input feature map using four convolutional layers with different kernel sizes in parallel. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The triple attention mechanism submodule weights features from three dimensions: space, channel, and scale, highlighting important information.

[0009] Furthermore, the frequency domain flow module also adopts a phased design, similar to the spatial domain flow structure, and sequentially goes through four stages: initial feature extraction, adaptive frequency decomposition, frequency domain feature extraction, and frequency domain deep semantic extraction. The initial feature extraction includes three convolutional layers, a batch normalization layer, and a ReLU activation function; The adaptive frequency decomposition transforms spatial domain features into the frequency domain and performs adaptive decomposition. The frequency domain feature extraction includes three convolutional layers, a batch normalization layer, and a ReLU activation function; The frequency domain deep semantic extraction includes two convolutional layers, a batch normalization layer, a ReLU activation function, and Fourier units. The Fourier units are interspersed between the convolutional layers and perform convolution operations in the frequency domain.

[0010] Furthermore, the frequency domain flow module sequentially employs frequency domain transformation, adaptive frequency decomposition, frequency domain artifact detection and enhancement, cross-band interaction and adaptive fusion, and inverse frequency domain transformation to transform spatial domain features into the frequency domain and perform adaptive decomposition. The frequency domain transformation uses a two-dimensional fast Fourier transform to convert the input spatial domain feature map into a frequency domain representation; The adaptive frequency decomposition generates an adaptive decomposition threshold through a lightweight frequency analysis network, which consists of convolutional layers, batch normalization layers, ReLU activation functions, and fully connected layers. The frequency domain artifact detection and enhancement processes the decomposed frequency bands using dedicated convolutional networks. The cross-band interaction and adaptive fusion stitches together the enhanced feature maps at the boundary along the channel dimension. The inverse frequency domain transformation uses the inverse fast Fourier transform to convert the fused frequency domain features back to the spatial domain, resulting in an enhanced frequency domain feature map.

[0011] A method for detecting deepfake videos based on spatial-frequency domain combination is disclosed. The detection method utilizes the aforementioned deepfake video detection system based on spatial-frequency domain combination, and includes the following steps: Step 1: Preprocess and feature-enhance the videos with faces in the dataset to obtain the training dataset and the test dataset; Step 2: Extract the unique features of the spatial domain and frequency domain from the training dataset of Step 1 using a two-stream network module; Step 3: Effectively fuse the unique features of the spatial domain and frequency domain from Step 2 using the feature fusion module; Step 4: Input the data effectively fused in Step 3 into the classifier module to output the detection result as "real" or "fake".

[0012] Furthermore, in step 2, the spatial domain flow module sequentially employs an artifact detection network module, a multi-scale feature extraction sub-module, and a triple attention mechanism sub-module to perform artifact perception multi-scale attention processing. The artifact detection network module consists of a lightweight convolutional neural network. It receives the feature map output from the previous stage as input and outputs a single-channel probability map P(x,y) with the same size as the input feature map. The value of each pixel in the map is between 0 and 1, representing the probability that there is a forgery trace at that spatial location (x,y). The multi-scale feature extraction submodule processes the input feature map using four convolutional layers with different kernel sizes in parallel. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, and the number of output channels is 2048. By extracting spatial domain features at different scales through multi-scale convolution, we can capture the artifact patterns unique to the spatial domain. The aforementioned triple attention mechanism submodule weights features from three dimensions: space, channel, and scale. Spatial attention : Artifact probability map As weights, with the input feature map Element-wise multiplication is performed, followed by a 3×3 convolutional layer and a sigmoid activation function to obtain the spatial attention weights, as shown in the following formula: This represents the sigmoid activation function. This represents a spatial attention convolutional layer;

[0013] Channel attention : For the input feature map Global average pooling is performed to obtain the channel feature vectors. Then, the channel weights are learned through a fully connected network and a sigmoid activation function, as shown in the following formula:

[0014] Scale attention : The four scale feature maps output by the multi-scale feature extraction submodule The values ​​[1×1, 3×3, 5×5, 7×7] are concatenated, and then scale weights are generated using the softmax function, as shown in the following formula:

[0015] Finally, spatial attention, channel attention, and scale attention are multiplied by their respective feature maps to obtain the enhanced spatial domain features.

[0016] Furthermore, in step 2, the frequency domain flow module sequentially employs frequency domain transformation, adaptive frequency decomposition, frequency domain artifact detection and enhancement, cross-band interaction and adaptive fusion, and inverse frequency domain transformation to transform spatial domain features into the frequency domain and perform adaptive decomposition. The frequency domain transformation uses a two-dimensional fast Fourier transform to convert the input spatial domain feature map f(x,y) into a frequency domain representation F(u,v); The adaptive frequency decomposition takes a frequency domain feature map as input. The amplitude graph outputs three coefficients. Then, based on the mean of the frequency domain features Calculate three frequency thresholds: , , Using these three thresholds to extract frequency domain features It is divided into three non-overlapping frequency bands: low frequency, mid frequency, and high frequency.

[0017]

[0018]

[0019] The frequency domain artifact detection and enhancement process uses dedicated convolutional networks for each of the three frequency bands. The low-frequency band convolutional network consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The mid-frequency band convolutional network consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The high-frequency band convolutional network consists of two 1×1 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. Different convolutional networks are used to detect and enhance artifact features within each frequency band. The cross-band interaction and adaptive fusion concatenates the enhanced feature maps of the three frequency bands in the channel dimension, learns the dependencies between features of different frequency bands through a channel attention network (composed of a global average pooling layer, two fully connected layers and a sigmoid activation function), generates frequency band weights, and then performs weighted fusion of the three frequency band features according to the weights to obtain the fused frequency domain features. The inverse frequency domain transformation uses the inverse fast Fourier transform to convert the fused frequency domain features back to the spatial domain, resulting in an enhanced frequency domain feature map.

[0020] Furthermore, step 3 specifically involves first fusing spatial domain features and frequency domain features at a deeper level of the network, and then finally fusing the fused features at different scales.

[0021] The beneficial effects of this invention are: This invention fully leverages the unique features of the spatial and frequency domains. Through an artifact-aware multi-scale attention module, it can accurately capture unique artifact features such as local details, edges, and textures in the spatial domain; through an adaptive frequency decomposition module, it can comprehensively mine unique artifact features such as global structure, illumination, and color distribution in the frequency domain, achieving comprehensive capture of artifact information from deepfake videos across different domains.

[0022] This invention employs an adaptive complementary fusion module, which fully utilizes the complementary information of the two domains, effectively improving the accuracy of deepfake video detection. In tests on the same dataset, the highest AUC reached 96.53%.

[0023] This invention demonstrates stable detection performance in cross-dataset testing, exhibiting good robustness and generalization ability. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the architecture of the deepfake video detection system of the present invention.

[0025] Figure 2 This is a schematic diagram of the architecture of the dual-stream network module of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0029] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0031] Implementation Method 1 This embodiment provides a deepfake video detection system based on spatial-frequency domain combination. The overall architecture of the deepfake video detection system of this invention is as follows: Figure 1 As shown, the system mainly includes a data preprocessing module, a dual-stream network module (spatial domain stream module and frequency domain stream module), a feature fusion module (adaptive complementary fusion module and multi-scale fusion module), and a classifier module. The system receives a preprocessed face image as input, extracts unique features in the spatial domain and frequency domain through the dual-stream network module, and after the feature fusion module effectively fuses the features, it inputs the results into the classifier module to output the detection result of "real" or "fake".

[0032] The detailed architecture of the dual-stream network module is as follows: Figure 2 As shown, the spatial domain stream and the frequency domain stream start from the RGB input, and each goes through multiple processing stages to mine the unique features of its respective domain. Then they are fused through the feature fusion module and finally input into the classifier to complete the detection.

[0033] The data preprocessing module is used to preprocess and enhance the features of videos containing faces in the dataset, resulting in training and testing datasets. First, video frames are extracted from the original videos at a fixed frame rate to ensure that the extracted frames cover the key content of the video. Next, a face detection algorithm is used to detect faces in the extracted video frames, locate face regions, and crop them to remove irrelevant background information. Then, the cropped face images are aligned to eliminate the influence of differences in face pose, and the aligned face images are normalized to a fixed size for model input. Finally, to improve the model's generalization ability and prevent overfitting, various data augmentation techniques are used to enhance the preprocessed images, including random horizontal flipping, random rotation, random blurring, brightness and contrast adjustment, and JPEG compression.

[0034] The publicly available deepfake video datasets used in this invention include FaceForensics++ (FF++), Celeb-DF (v1 & v2), DeepfakeDetectionChallenge (DFDC), DeepFakeDetection, UADFV, DFDCP, etc. These datasets cover different generation techniques, scenarios, and forgery levels, and can effectively verify the effectiveness and generalization ability of the model.

[0035] The dual-stream network module includes a spatial domain stream module and a frequency domain stream module. The two streams work in parallel to mine the unique features of the spatial domain and the frequency domain, respectively.

[0036] The spatial domain flow module adopts a phased design, starting from the input RGB face image and sequentially going through five stages: initial feature extraction, basic feature enhancement, multi-scale feature extraction, artifact-aware multi-scale attention processing, and deep semantic extraction. This fully exploits the unique artifact features in the spatial domain, as detailed below: 1. Initial Feature Extraction: This stage consists of three convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernels are all 3×3 with a stride of 1, and the output channels are 64, 128, and 256 respectively. This stage performs preliminary feature extraction on the input image, capturing low-level features such as color, basic texture, and edges.

[0037] 2. Basic Feature Enhancement: Composed of two residual blocks, each containing two convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernel size is 3×3, and the stride is 1. Feature representation is enhanced through residual connections, enabling the learning of more complex local patterns.

[0038] 3. Multi-scale feature extraction: Downsampling is performed using convolutional layers with a stride of 2, with a total of 3 downsampling operations. The resolution of the feature map is successively reduced to 1 / 2, 1 / 4, and 1 / 8 of the original resolution, while the number of output channels is increased to 512, 1024, and 2048. Downsampling expands the receptive field, allowing for the extraction of larger-scale and more abstract features.

[0039] 4. Artifact-Aware Multi-Scale Attention Processing: This module is the core of spatial domain flow, used to adaptively focus on regions and scales in the image where forgery artifacts exist, and to enhance features. It specifically includes the following three sub-modules: Artifact Detection Network: This submodule generates an "artifact probability map" to guide the subsequent attention mechanism. It consists of a lightweight convolutional neural network that takes the feature map from the previous stage as input and outputs a single-channel probability map P(x,y) of the same size as the input feature map. Each pixel in the map has a value between 0 and 1, representing the probability that a forgery exists at that spatial location (x,y). This allows the network to allocate more computational resources to regions with higher probabilities.

[0040] The multi-scale feature extraction submodule uses four convolutional layers with different kernel sizes (1×1, 3×3, 5×5, and 7×7) in parallel to process the input feature map. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, and the output channel count is 2048. Multi-scale convolution extracts spatial domain features at different scales, capturing spatial domain-specific artifact patterns such as local details, edges, and textures.

[0041] The triple attention mechanism submodule weights features from three dimensions: space, channel, and scale, highlighting important information; Spatial attention : Artifact probability map As weights, with the input feature map Element-wise multiplication is performed, followed by a 3×3 convolutional layer and a sigmoid activation function to obtain the spatial attention weights, as shown in the following formula: This represents the sigmoid activation function. This represents a spatial attention convolutional layer.

[0042]

[0043] Channel attention : For the input feature map Perform global average pooling ( The channel feature vectors are obtained, and then the channel weights are learned through a fully connected network (MLP) and a sigmoid activation function, as shown in the following formula:

[0044] Scale attention : The four scale feature maps output by the multi-scale feature extraction submodule The values ​​[1×1, 3×3, 5×5, 7×7] are concatenated, and then scale weights are generated using the softmax function, as shown in the following formula:

[0045] Finally, spatial attention, channel attention, and scale attention are multiplied by their respective feature maps to obtain the enhanced spatial domain features.

[0046] 5. Deep Semantic Extraction: This stage consists of two convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernel size is 3×3, the stride is 1, and the number of output channels is 2048. This stage extracts global, highly abstract semantic features and outputs a multi-scale spatial domain feature map for subsequent fusion.

[0047] The frequency domain flow module also adopts a phased design, similar to the spatial domain flow structure, but the key stages are replaced by modules specifically for frequency analysis. It sequentially goes through four stages: initial feature extraction, adaptive frequency decomposition, frequency domain feature extraction, and deep frequency domain semantic extraction, as detailed below: 1. Initial Feature Extraction: The structure is exactly the same as the initial feature extraction stage of the spatial domain flow. Preliminary feature extraction is performed on the input image to obtain a feature map with the same initial feature dimension as the spatial domain flow.

[0048] 2. Adaptive Frequency Decomposition: This module transforms spatial domain features to the frequency domain and performs adaptive decomposition. The specific steps are as follows: Frequency domain transformation: The input spatial domain feature map f(x,y) is transformed into a frequency domain representation F(u,v) using a two-dimensional fast Fourier transform (2D-FFT).

[0049] Adaptive Frequency Decomposition: An adaptive decomposition threshold is generated through a lightweight frequency analysis network. The frequency analysis network consists of convolutional layers, batch normalization layers, ReLU activation functions, and fully connected layers. The input is a frequency domain feature map. The amplitude graph outputs three coefficients. Then, based on the mean of the frequency domain features Calculate three frequency thresholds: , , Using these three thresholds to extract frequency domain features It is divided into three non-overlapping frequency bands: low frequency, mid frequency, and high frequency.

[0050]

[0051]

[0052]

[0053] Frequency Domain Artifact Detection and Enhancement: Dedicated convolutional networks are used to process the three frequency bands. The low-frequency band convolutional network consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The mid-frequency band convolutional network also consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The high-frequency band convolutional network consists of two 1×1 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. Different convolutional networks are used to detect and enhance artifact features within each frequency band. The low-frequency band captures abnormal patterns such as global structure, illumination, and color distribution; the mid-frequency band captures abnormal patterns such as structural information and texture patterns; and the high-frequency band captures abnormal patterns such as edge artifacts, detail artifacts, and compression artifacts.

[0054] Cross-band interaction and adaptive fusion: The feature maps of the three frequency bands are concatenated in the channel dimension. The dependencies between the features of different frequency bands are learned through a channel attention network (composed of a global average pooling layer, two fully connected layers and a sigmoid activation function) to generate frequency band weights. Then, the features of the three frequency bands are weighted and fused according to the weights to obtain the fused frequency domain features.

[0055] Inverse frequency domain transformation: The inverse fast Fourier transform (IFFT) is used to transform the fused frequency domain features back to the spatial domain, resulting in an enhanced frequency domain feature map.

[0056] 3. Frequency Domain Feature Extraction: This consists of three convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernel size is 3×3, the stride is 1, and the number of output channels is 512, 1024, and 2048, respectively. Further feature extraction and enhancement are performed on the feature map after inverse frequency domain transformation.

[0057] 4. Deep Semantic Extraction in the Frequency Domain: This layer consists of two convolutional layers, a batch normalization layer, a ReLU activation function, and Fourier units. The convolutional kernel size is 3×3, the stride is 1, and the number of output channels is 2048. The Fourier units are interspersed between the convolutional layers, performing convolution operations in the frequency domain to more directly learn the relationships between frequency domain features, ultimately outputting a multi-scale frequency domain feature map for fusion.

[0058] The feature fusion module adopts a two-level fusion strategy, including an adaptive complementary fusion module and a multi-scale fusion module, to achieve effective fusion of spatial domain features and frequency domain features.

[0059] The adaptive complementary fusion module fuses spatial domain features and frequency domain features at a deeper level in the network. The specific steps are as follows: 1. Confidence assessment: Feature map of spatial domain flow output Feature map of frequency domain stream output The corresponding confidence maps are generated through two independent lightweight confidence networks. and The confidence network consists of convolutional layers, batch normalization layers, ReLU activation functions, and sigmoid activation functions. The convolutional kernel size is 3×3, and the stride is 1. The formula is as follows, where... and These represent confidence convolutional layers in the spatial and frequency domains, respectively.

[0060]

[0061]

[0062] 2. Cross-domain guided enhancement: A cross-domain guided mechanism is used to mutually enhance features from two domains. For spatial domain features, frequency domain features are enhanced... Guiding weights are generated using a 1×1 convolutional layer and a sigmoid activation function. Then, with spatial domain features Multiplication yields enhanced spatial domain features The formula is as follows. Similarly, a similar operation is performed on frequency domain features.

[0063]

[0064]

[0065] 3. Complementary weight generation: This involves generating the confidence graph. and The data is concatenated along the channel dimension and normalized using the softmax function to obtain a complementary weight map. and ,satisfy + =1, the formula is as follows.

[0066]

[0067] 4. Feature Interaction: The two weighted feature maps are concatenated along the channel dimension and processed through a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function to generate feature interaction terms. The final fusion features This is the sum of the weighted summation result and the feature interaction terms.

[0068]

[0069]

[0070] The multi-scale fusion module is used to perform the final fusion of features from different scales. The specific steps are as follows: 1. Feature Upsampling: Multiple fused feature maps of different scales obtained from spatial domain flow and frequency domain flow at different deep stages (such as after multi-scale feature extraction stage, after deep semantic extraction stage) through an adaptive complementary fusion module are upsampled to a uniform size (i.e., the size of the deepest feature map). To avoid information loss caused by large-scale upsampling, the maximum upsampling factor is limited to 2 times, and bilinear interpolation is used for upsampling.

[0071] 2. Adaptive weighted fusion: All fused feature maps with uniform size are concatenated along the channel dimension, and a feature vector is obtained through a global average pooling layer. This vector is then input into a small multilayer perceptron. It consists of two fully connected layers and a ReLU activation function, and outputs the fusion weights corresponding to each scale feature. Finally, the feature maps at different scales are weighted and summed to obtain the final fused features. The formula is as follows, where F1, F2, ..., Fn represent fused feature maps at different scales.

[0072]

[0073]

[0074] The classifier module receives the final fused features. The output shows the detection results of the deepfake video, with the following structure: 1. Global Average Pooling (GAP) layer: This layer fuses the final feature maps. Average pooling is performed in the spatial dimension to convert it into a fixed-length feature vector, the length of which is equal to the number of channels in the feature map (2048).

[0075] 2. Fully connected layer: It contains two fully connected layers. The first fully connected layer has an input dimension of 2048 and an output dimension of 1024. The second fully connected layer has an input dimension of 1024 and an output dimension of 512. Each fully connected layer is followed by a ReLU activation function and a dropout layer (dropout probability of 0.5) to increase the non-linear expressive power of the model and prevent overfitting.

[0076] 3. Output Layer: This is a fully connected layer with two output units. It is normalized using the softmax function and outputs a two-dimensional probability vector [real, fake], representing the probability that the input image is "real" or "fake," respectively. The final decision is made based on the probability values, and the category with the higher probability value is the detection result. 1. Experimental Environment Hardware environment: CPU is a 16v Intel(R) Xeon(R) Platinum 8352V CPU @ 2.10GHz, GPU is an RTX4090 (24GB)*1, and memory is 120GB.

[0077] Software environment: Operating system is Ubuntu 20.04, deep learning framework is PyTorch 1.10.0, Python version is 3.8.10, and CUDA version is 11.3.

[0078] 2. Training parameters Batch size: 32.

[0079] Learning rate: The initial learning rate is 0.001. A cosine annealing learning rate scheduling strategy is adopted, and the learning rate decays once every 5 epochs with a decay coefficient of 0.9.

[0080] Training epochs: 30.

[0081] Loss function: Cross-entropy loss function.

[0082] Optimizer: Adam optimizer, with a weight decay factor of 0.0001.

[0083] 3. Experimental Results This embodiment was experimentally verified on multiple public datasets, and AUC (area under the ROC curve) was used as the performance evaluation index. The closer the AUC is to 1, the better the model classification effect.

[0084] Test results on the same dataset: On the FaceForensics++ (FF++) dataset, video data generated using four face-swapping techniques (DeepFakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT)) were used for training and testing. The results are shown in the table below:

[0085] As shown in the table, the model of this invention has a high AUC on the same dataset, demonstrating good classification performance.

[0086] Cross-dataset test results: The model trained on the FF++ dataset was tested on datasets such as Celeb-DF (v1 and v2), DFDC, DFD, UADFV, and DFDCP to verify the model's generalization performance and robustness. The results are shown in the table below:

[0087] As can be seen from the table, the model achieved stable AUC performance on test datasets with different distribution characteristics, verifying that the method of this invention has good cross-dataset robustness.

[0088] Implementation Method 2 This implementation provides A method for detecting deepfake videos based on spatial-frequency domain combination is disclosed. The method utilizes the deepfake video detection system based on spatial-frequency domain combination as described in Embodiment 1, and includes the following steps: Step 1: Preprocess and feature-enhance the videos with faces in the dataset to obtain the training dataset and the test dataset; Step 2: Extract the unique features of the spatial domain and frequency domain from the training dataset of Step 1 using a two-stream network module; Step 3: Effectively fuse the unique features of the spatial domain and frequency domain from Step 2 using the feature fusion module; Step 4: Input the data effectively fused in Step 3 into the classifier module to output the detection result as "real" or "fake".

[0089] Furthermore, step 1 specifically involves: firstly, extracting video frames from the original video at a fixed frame rate to ensure that the extracted frames cover the key content of the video; then, using a face detection algorithm to detect faces in the extracted video frames, locating face regions and cropping them to remove irrelevant background information; next, aligning the cropped face images to eliminate the influence of differences in face poses, and then normalizing the aligned face images to a fixed size for easy model input; finally, to improve the model's generalization ability and prevent overfitting, using various data augmentation techniques to enhance the preprocessed image, including random horizontal flipping, random rotation, random blurring, brightness and contrast adjustment, and JPEG compression.

[0090] The publicly available deepfake video datasets used in this invention include FaceForensics++ (FF++), Celeb-DF (v1 & v2), DeepfakeDetectionChallenge (DFDC), DeepFakeDetection, UADFV, DFDCP, etc. These datasets cover different generation techniques, scenarios, and forgery levels, and can effectively verify the effectiveness and generalization ability of the model.

[0091] Furthermore, in step 2, the spatial domain flow module sequentially employs an artifact detection network module, a multi-scale feature extraction submodule, and a triple attention mechanism submodule to perform artifact-aware multi-scale attention processing, as detailed below: Step K2.1. Initial Feature Extraction: This step consists of three convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernels are all 3×3 with a stride of 1, and the output channels are 64, 128, and 256 respectively. This stage performs preliminary feature extraction on the input image, capturing low-level features such as color, basic texture, and edges.

[0092] Step K2.2. Basic Feature Enhancement: This step consists of two residual blocks, each containing two convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernel size is 3×3, and the stride is 1. Feature representation is enhanced through residual connections to learn more complex local patterns.

[0093] Step K2.3. Multi-scale feature extraction: Downsampling is performed using convolutional layers with a stride of 2, with a total of 3 downsampling operations. The resolution of the feature map is successively reduced to 1 / 2, 1 / 4, and 1 / 8 of the original resolution, while the number of output channels is increased to 512, 1024, and 2048. Downsampling expands the receptive field, allowing for the extraction of larger-scale and more abstract features.

[0094] Step K2.4. Artifact-Aware Multi-Scale Attention Processing: This module is the core of spatial domain flow, used to adaptively focus on regions and scales in the image where forgery traces exist, and to enhance features. It specifically includes the following three sub-modules: The artifact detection network submodule generates an "artifact probability map" to guide the subsequent attention mechanism. It consists of a lightweight convolutional neural network that takes the feature map from the previous stage as input and outputs a single-channel probability map P(x,y) of the same size as the input feature map. Each pixel in the map has a value between 0 and 1, representing the probability that a forgery exists at that spatial location (x,y). This allows the network to allocate more computational resources to regions with higher probabilities.

[0095] The multi-scale feature extraction submodule uses four convolutional layers with different kernel sizes (1×1, 3×3, 5×5, and 7×7) in parallel to process the input feature map. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, and the output channel count is 2048. Multi-scale convolution extracts spatial domain features at different scales, capturing spatial domain-specific artifact patterns such as local details, edges, and textures.

[0096] The triple attention mechanism submodule weights features from three dimensions: space, channel, and scale, highlighting important information; Spatial attention : Artifact probability map As weights, with the input feature map Element-wise multiplication is performed, followed by a 3×3 convolutional layer and a sigmoid activation function to obtain the spatial attention weights, as shown in the following formula: This represents the sigmoid activation function. This represents a spatial attention convolutional layer.

[0097]

[0098] Channel attention : For the input feature map Perform global average pooling ( The channel feature vectors are obtained, and then the channel weights are learned through a fully connected network (MLP) and a sigmoid activation function, as shown in the following formula:

[0099] Scale attention : The four scale feature maps output by the multi-scale feature extraction submodule The values ​​[1×1, 3×3, 5×5, 7×7] are concatenated, and then scale weights are generated using the softmax function, as shown in the following formula:

[0100] Finally, spatial attention, channel attention, and scale attention are multiplied by their respective feature maps to obtain the enhanced spatial domain features.

[0101] Step K2.5. Deep Semantic Extraction: This step consists of two convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernel size is 3×3, the stride is 1, and the number of output channels is 2048. This stage extracts global, highly abstract semantic features and outputs a multi-scale spatial domain feature map for subsequent fusion.

[0102] Furthermore, in step 2, the frequency domain flow module sequentially employs frequency domain transformation, adaptive frequency decomposition, frequency domain artifact detection and enhancement, cross-band interaction and adaptive fusion, and inverse frequency domain transformation to convert spatial domain features to the frequency domain and perform adaptive decomposition, as detailed below: Step P2.1. Initial Feature Extraction: The structure is exactly the same as the initial feature extraction stage of the spatial domain flow. Preliminary feature extraction is performed on the input image to obtain a feature map with the same dimension as the initial feature of the spatial domain flow.

[0103] Step P2.2. Adaptive Frequency Decomposition: This module transforms the spatial domain features to the frequency domain and performs adaptive decomposition. The specific steps are as follows: Step P2.2.1. Frequency domain transformation: Use two-dimensional fast Fourier transform (2D-FFT) to transform the input spatial domain feature map f(x,y) into the frequency domain representation F(u,v).

[0104] Step P2.2.2. Adaptive Frequency Decomposition: An adaptive decomposition threshold is generated using a lightweight frequency analysis network. The frequency analysis network consists of convolutional layers, batch normalization layers, ReLU activation functions, and fully connected layers. The input is the frequency domain feature map. The amplitude graph outputs three coefficients. Then, based on the mean of the frequency domain features Calculate three frequency thresholds: , , Using these three thresholds to extract frequency domain features It is divided into three non-overlapping frequency bands: low frequency, mid frequency, and high frequency.

[0105]

[0106]

[0107]

[0108] Step P2.2.3. Frequency Domain Artifact Detection and Enhancement: Dedicated convolutional networks are used to process the three frequency bands. The low-frequency band convolutional network consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The mid-frequency band convolutional network consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The high-frequency band convolutional network consists of two 1×1 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. Different convolutional networks are used to detect and enhance artifact features within each frequency band. The low-frequency band captures abnormal patterns such as global structure, illumination, and color distribution; the mid-frequency band captures abnormal patterns such as structural information and texture patterns; and the high-frequency band captures abnormal patterns such as edge, detail, and compression artifacts.

[0109] Step P2.2.4. Cross-band interaction and adaptive fusion: The feature maps of the three frequency bands are concatenated in the channel dimension. The dependencies between the features of different frequency bands are learned through a channel attention network (composed of a global average pooling layer, two fully connected layers and a sigmoid activation function) to generate frequency band weights. Then, the features of the three frequency bands are weighted and fused according to the weights to obtain the fused frequency domain features.

[0110] Step P2.2.5. Inverse Frequency Domain Transformation: Use Inverse Fast Fourier Transform (IFFT) to transform the fused frequency domain features back to the spatial domain to obtain the enhanced frequency domain feature map.

[0111] Step P2.3. Frequency Domain Feature Extraction: This step consists of three convolutional layers, a batch normalization layer, and a ReLU activation function. The convolutional kernel size is 3×3, the stride is 1, and the number of output channels are 512, 1024, and 2048, respectively. Further feature extraction and enhancement are then performed on the feature map after inverse frequency domain transformation.

[0112] Step P2.4. Deep semantic extraction in the frequency domain: This step consists of two convolutional layers, a batch normalization layer, a ReLU activation function, and Fourier units. The convolutional kernel size is 3×3, the stride is 1, and the number of output channels is 2048. The Fourier units are interspersed between the convolutional layers, performing convolution operations in the frequency domain to more directly learn the relationships between frequency domain features, and finally outputting a multi-scale frequency domain feature map for fusion.

[0113] The feature fusion module in step 3 includes an adaptive complementary fusion module and a multi-scale fusion module. The adaptive complementary fusion module is specifically as follows: Confidence assessment: Feature map of spatial domain flow output Feature map of frequency domain stream output The corresponding confidence maps are generated through two independent lightweight confidence networks. and The confidence network consists of convolutional layers, batch normalization, ReLU activation function, and sigmoid activation function. The convolutional kernel size is 3×3, and the stride is 1. The formula is as follows, where... and These represent confidence convolutional layers in the spatial and frequency domains, respectively.

[0114]

[0115]

[0116] Cross-domain guided enhancement: A cross-domain guided mechanism is used to mutually enhance features from two domains. For spatial domain features, frequency domain features are enhanced. Guiding weights are generated using a 1×1 convolutional layer and a sigmoid activation function. Then, with spatial domain features Multiplication yields enhanced spatial domain features The formula is as follows. Similarly, a similar operation is performed on frequency domain features.

[0117]

[0118]

[0119] Complementary weight generation: Confidence map and The data is concatenated along the channel dimension and normalized using the softmax function to obtain a complementary weight map. and ,satisfy + =1, the formula is as follows.

[0120]

[0121] Feature interaction: The two weighted feature maps are concatenated along the channel dimension and processed through a 1×1 convolutional layer, a batch normalization layer, and a ReLU activation function to generate feature interaction terms. The final fusion features This is the sum of the weighted summation result and the feature interaction terms.

[0122]

[0123] .

[0124] The multi-scale fusion module is specifically: Feature upsampling: Multiple fused feature maps of different scales obtained from spatial and frequency domain flows at different deep stages (such as after multi-scale feature extraction or deep semantic extraction) through an adaptive complementary fusion module are upsampled to a uniform size (i.e., the size of the deepest feature map). To avoid information loss caused by large upsampling factors, the maximum upsampling factor is limited to 2 times, and bilinear interpolation is used for upsampling.

[0125] Adaptive weighted fusion: All fused feature maps of uniform size are concatenated along the channel dimension, and a feature vector is obtained through a global average pooling layer. Then, it is input into a small multilayer perceptron. It consists of two fully connected layers and a ReLU activation function, and outputs the fusion weights corresponding to each scale feature. Finally, the feature maps at different scales are weighted and summed to obtain the final fused features. The formula is as follows, where F1, F2, ..., Fn represent fused feature maps at different scales.

[0126]

[0127] .

[0128] Furthermore, step 4 specifically involves the classifier module receiving the final fused features. The output shows the detection results of the deepfake video, with the following structure: 1. Global Average Pooling (GAP) layer: This layer fuses the final feature maps. Average pooling is performed in the spatial dimension to convert it into a fixed-length feature vector, the length of which is equal to the number of channels in the feature map (2048).

[0129] 2. Fully connected layer: It contains two fully connected layers. The first fully connected layer has an input dimension of 2048 and an output dimension of 1024. The second fully connected layer has an input dimension of 1024 and an output dimension of 512. Each fully connected layer is followed by a ReLU activation function and a dropout layer (dropout probability of 0.5) to increase the non-linear expressive power of the model and prevent overfitting.

[0130] 3. Output Layer: This is a fully connected layer with two output units. It is normalized using the softmax function and outputs a two-dimensional probability vector [real, fake], representing the probability that the input image is "real" or "fake," respectively. The final decision is made based on the probability values, and the category with the higher probability value is the detection result.

[0131] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A deepfake video detection system based on spatial-frequency domain combination, characterized in that, The detection system includes a data preprocessing module, a two-stream network module, a feature fusion module, and a classifier module; The data preprocessing module is used to preprocess and enhance the features of videos with faces in the dataset to obtain training and testing datasets. The dual-stream network module includes a spatial domain stream module and a frequency domain stream module. The two streams work in parallel to mine the unique features of the spatial domain and the frequency domain, respectively. The feature fusion module adopts a two-level fusion strategy, including an adaptive complementary fusion module and a multi-scale fusion module, to achieve effective fusion of spatial domain features and frequency domain features. The classifier module receives the final fused features and outputs the detection results of the deepfake video.

2. The deepfake video detection system according to claim 1, characterized in that, The spatial domain flow module adopts a phased design, starting from the input RGB face image and sequentially going through five stages: initial feature extraction, basic feature enhancement, multi-scale feature extraction, artifact perception multi-scale attention processing, and deep semantic extraction, so as to fully explore the unique artifact features in the spatial domain.

3. The deepfake video detection system according to claim 1, characterized in that, The initial feature extraction includes three convolutional layers, a batch normalization layer, and a ReLU activation function; The basic feature enhancement includes two residual blocks, each containing two convolutional layers, a batch normalization layer, and a ReLU activation function; The multi-scale feature extraction uses a convolutional layer with a stride of 2 for downsampling; The artifact-aware multi-scale attention processing is used to adaptively focus on the regions and scales in the image where forgery traces exist, and to enhance the features. The deep semantic extraction includes two convolutional layers, a batch normalization layer, and a ReLU activation function.

4. The deepfake video detection system according to claim 1, characterized in that, The spatial domain flow module sequentially employs an artifact detection network module, a multi-scale feature extraction sub-module, and a triple attention mechanism sub-module to perform artifact perception multi-scale attention processing. The artifact detection network module generates an "artifact probability graph" to guide the subsequent attention mechanism; The multi-scale feature extraction submodule processes the input feature map using four convolutional layers with different kernel sizes in parallel. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. The triple attention mechanism submodule weights features from three dimensions: space, channel, and scale, highlighting important information.

5. The deepfake video detection system according to claim 1, characterized in that, The frequency domain flow module also adopts a phased design, similar to the spatial domain flow structure, and goes through four stages in sequence: initial feature extraction, adaptive frequency decomposition, frequency domain feature extraction, and frequency domain deep semantic extraction. The initial feature extraction includes three convolutional layers, a batch normalization layer, and a ReLU activation function; The adaptive frequency decomposition transforms spatial domain features into the frequency domain and performs adaptive decomposition. The frequency domain feature extraction includes three convolutional layers, a batch normalization layer, and a ReLU activation function; The frequency domain deep semantic extraction includes two convolutional layers, a batch normalization layer, a ReLU activation function, and Fourier units. The Fourier units are interspersed between the convolutional layers and perform convolution operations in the frequency domain.

6. The deepfake video detection system according to claim 1, characterized in that, The frequency domain flow module sequentially employs frequency domain transformation, adaptive frequency decomposition, frequency domain artifact detection and enhancement, cross-band interaction and adaptive fusion, and inverse frequency domain transformation to transform spatial domain features into the frequency domain and perform adaptive decomposition. The frequency domain transformation uses a two-dimensional fast Fourier transform to convert the input spatial domain feature map into a frequency domain representation; The adaptive frequency decomposition generates an adaptive decomposition threshold through a lightweight frequency analysis network, which consists of convolutional layers, batch normalization layers, ReLU activation functions, and fully connected layers. The frequency domain artifact detection and enhancement processes the decomposed frequency bands using dedicated convolutional networks. The cross-band interaction and adaptive fusion stitches together the enhanced feature maps at the boundary along the channel dimension. The inverse frequency domain transformation uses the inverse fast Fourier transform to convert the fused frequency domain features back to the spatial domain, resulting in an enhanced frequency domain feature map.

7. A method for detecting deepfake videos based on spatial-frequency domain combination, characterized in that, The detection method uses the deepfake video detection system based on spatial frequency domain combination as described in any one of claims 1-6, and the detection method includes the following steps: Step 1: Preprocess and feature-enhance the videos with faces in the dataset to obtain the training dataset and the test dataset; Step 2: Extract the unique features of the spatial domain and frequency domain from the training dataset of Step 1 using a two-stream network module; Step 3: Effectively fuse the unique features of the spatial domain and frequency domain from Step 2 using the feature fusion module; Step 4: Input the data effectively fused in Step 3 into the classifier module to output the detection result as "real" or "fake".

8. The deepfake video detection system according to claim 7, characterized in that, In step 2, the spatial domain flow module sequentially employs an artifact detection network module, a multi-scale feature extraction sub-module, and a triple attention mechanism sub-module to perform artifact perception multi-scale attention processing. The artifact detection network module consists of a lightweight convolutional neural network. It receives the feature map output from the previous stage as input and outputs a single-channel probability map P(x,y) with the same size as the input feature map. The value of each pixel in the map is between 0 and 1, representing the probability that there is a forgery trace at that spatial location (x,y). The multi-scale feature extraction submodule processes the input feature map using four convolutional layers with different kernel sizes in parallel. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, and the number of output channels is 2048. By extracting spatial domain features at different scales through multi-scale convolution, we can capture the artifact patterns unique to the spatial domain. The aforementioned triple attention mechanism submodule weights features from three dimensions: space, channel, and scale. Spatial attention : Artifact probability map As weights, with the input feature map Element-wise multiplication is performed, followed by a 3×3 convolutional layer and a sigmoid activation function to obtain the spatial attention weights, as shown in the following formula: This represents the sigmoid activation function. This represents a spatial attention convolutional layer; Channel attention : For the input feature map Global average pooling is performed to obtain the channel feature vectors. Then, the channel weights are learned through a fully connected network and a sigmoid activation function, as shown in the following formula: Scale attention : The four scale feature maps output by the multi-scale feature extraction submodule The values ​​[1×1, 3×3, 5×5, 7×7] are concatenated, and then scale weights are generated using the softmax function, as shown in the following formula: Finally, spatial attention, channel attention, and scale attention are multiplied by their respective feature maps to obtain the enhanced spatial domain features.

9. The deepfake video detection system according to claim 7, characterized in that, In step 2, the frequency domain flow module sequentially performs frequency domain transformation, adaptive frequency decomposition, frequency domain artifact detection and enhancement, cross-band interaction and adaptive fusion, and inverse frequency domain transformation to transform spatial domain features into the frequency domain and perform adaptive decomposition. The frequency domain transformation uses a two-dimensional fast Fourier transform to convert the input spatial domain feature map f(x,y) into a frequency domain representation F(u,v); The adaptive frequency decomposition takes a frequency domain feature map as input. The amplitude graph outputs three coefficients. Then, based on the mean of the frequency domain features Calculate three frequency thresholds: , , Using these three thresholds to extract frequency domain features It is divided into three non-overlapping frequency bands: low frequency, mid frequency, and high frequency. The frequency domain artifact detection and enhancement process uses dedicated convolutional networks for each of the three frequency bands. The low-frequency band convolutional network consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The mid-frequency band convolutional network consists of two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. The high-frequency band convolutional network consists of two 1×1 convolutional layers, a batch normalization layer, and a ReLU activation function, with 1024 output channels. Different convolutional networks are used to detect and enhance artifact features within each frequency band. The cross-band interaction and adaptive fusion concatenates the enhanced feature maps of the three frequency bands in the channel dimension, learns the dependencies between features of different frequency bands through a channel attention network, generates frequency band weights, and then performs weighted fusion of the three frequency band features according to the weights to obtain the fused frequency domain features. The inverse frequency domain transformation uses the inverse fast Fourier transform to convert the fused frequency domain features back to the spatial domain, resulting in an enhanced frequency domain feature map.

10. The deepfake video detection system according to claim 7, characterized in that, Step 3 specifically involves first fusing spatial domain features and frequency domain features at a deeper layer of the network, and then finally fusing the fused features at different scales.

Images