A deep fake video detection method based on multi-dimensional feature collaborative modeling

Abstract

The application discloses a kind of deep fake video detection methods based on multidimensional feature collaborative modeling, belong to computer vision field, including the following steps: using face recognition technology to the complete face image is preprocessed, obtain the image of face local area;Respectively extract the frequency domain feature and spatial domain feature of face local area;Adopt position perception dual-domain fusion module to the extracted spatial domain feature and frequency domain feature are adaptively weighted modulation and nonlinear fusion;Adjust the contribution degree of local area in local-global feature fusion process, obtain the final global fusion feature;Automatic weighting balances the loss of each task, realizes the prediction classification to fake video.The application has adopted the multidimensional feature fusion architecture of local and global, spatial domain and frequency domain, effectively improves the detection precision of deep fake video.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a deepfake video detection method based on multi-dimensional feature collaborative modeling. Background Technology

[0002] With the rapid development of deep learning, deepfake technology, which uses generative adversarial networks (GANs) and variational autoencoders (VAEs) to continuously train and generate images through mutual competition, has made significant progress in the field of image and video generation. This rapid development has not only brought many interesting applications, such as everyday entertainment, special effects production, and portrait reconstruction, but has also generated serious security problems, such as fake news. To reduce the security problems caused by the misuse of deepfake technology, more and more researchers are investing in the detection of fake videos.

[0003] Currently, the development of deepfake technology is becoming more diversified and complex, making the content of fake videos more sophisticated and realistic. Existing detection methods have low detection accuracy and poor generalization when faced with unseen forgery techniques, making it difficult to distinguish between real and fake videos. The existing mainstream methods have the following key problems: (1) Weak cross-scene generalization: Existing methods are prone to overfitting artifacts after training on a single dataset, and their detection performance drops significantly when faced with unknown forgery methods or cross-domain data. (2) Low utilization of high-frequency features: Traditional discrete wavelet transform (DWT) retains limited information in high-frequency subbands during decomposition and lacks targeted enhancement, resulting in the neglect of tiny forgery traces in high-frequency edge textures. (3) Static feature interaction: Existing space-frequency dual-stream networks mostly adopt simple splicing or weighted addition fusion methods, without establishing dynamic associations of cross-domain features, making it difficult to capture nonlinear distribution forgery patterns. Summary of the Invention

[0004] To address the shortcomings of existing methods in compressed forged video detection, frequency domain feature representation, and feature interaction mechanisms, this invention proposes a deep forged video detection method based on multi-dimensional feature collaborative modeling. This method integrates three core technical modules: multi-scale wavelet feature extraction, dual-domain dynamic attention fusion, and multi-level, multi-task collaborative supervision, overcoming the limitations of traditional detection frameworks. Specifically, it employs a local-global multi-dimensional spatial and frequency domain fusion architecture. A multi-scale wavelet transform method is designed to extract frequency domain information from local regions (eyes, mouth). A spatial-frequency dynamic attention mechanism is used to select and fuse important spatial and frequency domain features from important local regions (eyes, mouth). Finally, multi-level supervised optimization is used to collaboratively optimize the multi-feature fusion. The multi-dimensional, multi-information research method proposed in this invention improves detection accuracy. Local (eyes, mouth) features can capture detailed information from specific regions within a video frame, while global features provide overall information about the entire video frame and help identify the overall structure and logical coherence of the video.

[0005] The technical solution of the present invention is as follows:

[0006] A deepfake video detection method based on multi-dimensional feature collaborative modeling includes the following steps:

[0007] Step 1: Obtain a complete face image and use face recognition technology to preprocess the complete face image to obtain an image of a local area of ​​the face;

[0008] Step 2: Extract frequency domain features of local facial regions using a multi-scale wavelet feature processing module;

[0009] Step 3: Extract spatial features of local facial regions based on the first six layers of the pre-trained deep convolutional neural network ResNet34.

[0010] Step 4: Adaptive weight modulation and nonlinear fusion of the extracted spatial and frequency domain features are performed using a position-aware dual-domain dynamic attention fusion module.

[0011] Step 5: Use a dual-gated dynamic alignment fusion module to adjust the contribution of local regions in the local-global feature fusion process to obtain the final global fused features;

[0012] Step 6: The loss of each task is automatically weighted and balanced based on the category-aware multi-task loss module to achieve prediction and classification of fake videos.

[0013] Furthermore, the specific process of step 1 is as follows:

[0014] Step 1.1: Use a video capture device to acquire video frames, and use multi-task cascaded convolutional neural network face recognition technology to preprocess the video frames. During preprocessing, crop the images to a size of 224*224 to obtain complete face images.

[0015] Step 1.2: Use dlib facial landmark detection technology to crop out local areas of the eyes and mouth from the complete face image according to the preset size.

[0016] Furthermore, the specific process of step 2 is as follows:

[0017] Step 2.1: Divide the preprocessed eye and mouth region images into batches;

[0018] Step 2.2: Extract the cropped individual face region images from the batch, and use the cross-platform computer vision library OpenCV to convert the RGB images into YCrCb images; the image conversion formula is as follows:

[0019]

[0020] Where l represents a local area on the face; when l = eye, it represents the eye; when l = mouth, it represents the mouth; Y (l) It is the brightness channel at position l; Cr (l) and Cb (l) These are the red and blue chroma channels at position l; R, G, and B are the red, green, and blue color channels, respectively. The image of YCrCb at position l after conversion;

[0021] Step 2.3: Using the Dobessi wavelet basis, the brightness channel is decomposed into components of different scales, and the output includes the low-frequency component and the high-frequency component at position l, as shown in the following formula:

[0022]

[0023] Where S is the number of decomposition levels; cA (l) It is the low-frequency component at point l; cV s (l) , For the high-frequency components of scale s at point l in the horizontal, vertical, and diagonal directions; This represents a multi-scale wavelet transform operation;

[0024] Step 2.4: Extract the high-frequency component at position l. cV s (l) , The components are stitched along the channel dimension, and the high-frequency components after stitching are then subjected to high-frequency enhancement processing. The specific process is as follows: First, the high-frequency component at position l is... cV s (l) , The input channel dimension is concatenated into a high-frequency concatenated feature map, which is then expanded from 3 to 16 through a 3×3 convolutional layer. Next, the ReLU activation function is applied, and finally, another 3×3 convolutional layer restores the input channel dimension, outputting the high-frequency feature map at position l.

[0025]

[0026] in, Represents a high-frequency enhancement function; ⊕ represents channel dimension concatenation; Conv 3×3 This represents a 3×3 convolution operation; Represents the ReLU activation function;

[0027] At the same time, the low-frequency component cA at point l (l) Low-frequency enhancement processing: First, the low-frequency component cA at point l (l)After passing through a convolutional layer with a 3×3 kernel, a ReLU activation function is applied to introduce non-linearity. Finally, adaptive average pooling is used to downsample the feature map to 7×13, outputting the low-frequency feature map at position l.

[0028]

[0029] in, Represents a low-frequency enhancement function; Avg (H,W) For adaptive average pooling in the spatial dimension; H and W are the height and width of the image, respectively; Conv 3×3 This represents a 3×3 convolution operation;

[0030] Step 2.5, Cr (l) and Cb (l) The channel is adjusted to match the low-frequency feature map using bilinear interpolation. Same size:

[0031]

[0032] Among them, Cr′ (l) 、Cb′ (l) These represent the low-frequency enhancement features of the red and blue chromaticity channels at position l, respectively. It is a bilinear interpolation function; The target size is defined by Shape(·); Shape(·) represents the size information for extracting input features.

[0033] Step 2.6, Cr′ (l) 、Cb′ (l) and By splicing along the channel dimension, the final frequency domain feature at point l is formed.

[0034] Concat(·) means concatenating along the channel dimension.

[0035] Furthermore, the formula for step 3 is:

[0036]

[0037] in, The spatial characteristics at point l; φ ResNet34_pre (·) represents the feature extraction function of the first six layers of the deep convolutional neural network ResNet34; This is the RGB image at position l.

[0038] Furthermore, the specific process of step 4 is as follows:

[0039] Step 4.1: Extract the frequency domain features at position l. An adaptive filter is applied for processing, as shown in the following formula:

[0040]

[0041] in, The frequency domain characteristics at position l are obtained after passing through the adaptive filter; DWConv 3×3 (·) represents depthwise separable convolution; Conv 1×1 This represents a 1×1 convolution operation;

[0042] Step 4.2: Extract the spatial features at position l. and frequency domain characteristics after adaptive filter processing Each feature channel is weighted using channel attention, as shown in the formula:

[0043]

[0044] Where MaxPool(·) is the max pooling operation; σ(·) represents the sigmoid normalization function; represent the spatial attention weights and frequency attention weights generated at position l, respectively; ⊙ represents the element-wise multiplication operation. These are the spatial and frequency domain features at position l after weighting, respectively; Conv 1×1 This represents a 1×1 convolution operation;

[0045] Step 4.3: Combine frequency domain and pixel-level coordinate encoding to create a fusion model, and perform dual-domain fusion of spatial and frequency domain features of the local region.

[0046] Furthermore, the specific process of step 4.3 is as follows:

[0047] Step 4.3.1: Perform pixel-level spatial perception fusion modeling; first, generate a normalized spatial coordinate grid on the two-dimensional image plane:

[0048]

[0049] Where u is a normalized coordinate sequence in the horizontal direction; The width of the spatial feature map; v is the index of the point in the horizontal direction; v is the normalized coordinate sequence in the vertical direction. is the length of the spatial feature map; j is the index of the point in the vertical direction;

[0050] Subsequently, a multi-frequency activation-driven spatial coordinate encoding function is designed:

[0051]

[0052] Where Φ(u,v) is the spatial coordinate encoding; tanh is the hyperbolic tangent function; W1, W2, and W3 represent different weight matrices; and b1, b2, and b3 represent different bias vectors. This is the matrix transpose operation; γ is a hyperparameter.

[0053] Step 4.3.2: Analyze the frequency domain characteristics at point l obtained after the adaptive filter. Three different scales of convolution kernels are applied to extract multi-scale frequency domain responses, which are then fused to obtain a unified multi-scale feature vector G(F″) at position l. fre ) (l) :

[0054]

[0055] Among them, Conv k×k This represents a k×k convolution operation, where k is the kernel size.

[0056] Simultaneously, higher-order statistics of frequency domain features are introduced as supplementary information to jointly construct a structure-statistical fusion vector.

[0057] Where AvgPool(·) represents global average pooling; σ 2 (·) represents the unbiased variance in the spatial dimension; MaxPool(·) represents the max pooling operation; ∈ is a stable constant;

[0058] Finally, modulation coding is generated through a multilayer perceptron to produce the frequency domain modulation vector Ψ(F″) at position l. fre ) (l) :

[0059]

[0060] Among them, f mlp (·) denotes a multilayer perceptron function; Flatten(·) denotes a feature flattening operation;

[0061] Step 4.3.3: Combine the spatial coordinate code Φ(u,v) generated in step 4.3.1 with the frequency domain modulation vector Ψ(F″) obtained in step 4.3.2. fre ) (l) A concatenation operation is performed along the feature dimension to construct the cross-domain fusion feature matrix Z at point l. (l) :

[0062]

[0063] in, Indicates a broadcast extended copy operation;

[0064] Step 4.3.4, Z (l) The input is mapped into a multilayer perceptron structure, and the explicit fusion weight W at l is generated by the sigmoid normalization function. (l) :

[0065] W (l) =σ(f mlp (Z (l) ));

[0066] Step 4.3.5: Use this weight to perform pixel-level weighted fusion of spatial and frequency domain features to obtain the final spatial-frequency fused features:

[0067]

[0068] in, This represents the spatial-frequency fusion feature at point l.

[0069] Furthermore, the specific process of step 5 is as follows:

[0070] Step 5.1: Use a deep convolutional neural network ResNet34 to extract the complete face image X. global global features F global :

[0071] F global =φ ResNet34 (X global );

[0072] Where, φ ResNet34 (·) is the feature extraction function of the deep convolutional neural network ResNet34;

[0073] Step 5.2: Introduce an adaptive global attention mechanism, employing a structure combining bidirectional channel attention and lightweight spatial attention, ultimately enhancing feature representation through residual connections; the specific process is as follows:

[0074] Step 5.2.1: Extract channels using global average pooling and global max pooling respectively, and obtain the channel attention output F. channel :

[0075] F channel =σ(Conv 1×1 (ReLU(Conv1(AvgPool(F global ))))+Conv2(ReLU(Conv 1×1 (MaxPool(F global )))))⊙F global ;

[0076] Among them, Conv1×1 This represents a 1×1 convolution operation;

[0077] Step 5.2.2: Calculate the average and maximum values ​​for all channels at each spatial location using lightweight spatial attention to obtain the average spatial feature map S. avg and the maximum spatial feature map S max The two data points are concatenated along the channel dimension and then fed into a 7×7 convolutional layer to generate a spatial attention weight map M. spatial The formula is:

[0078]

[0079] M spatial =σ(Conv 7×7 (Concat(S avg ,S max )));

[0080] in, This represents the number of channels for the input feature; Represents the channel index; Concat(·) represents concatenation along the channel dimension; Conv 7×7 This represents a 7×7 convolution operation; For channel The corresponding channel attention output;

[0081] Step 5.2.3: Transfer the spatial attention weight map M spatial With channel attention output F channel Element-wise multiplication yields the attention feature map F. spatial :

[0082] F spatial =M spatial ⊙F channel ;

[0083] Step 5.2.4: Process the global features F of the input. global The residual information F is obtained by using 1×1 convolution, batch normalization, and ReLU activation. residual :

[0084]

[0085] Where BN represents batch normalization; Conv 1×1 This represents a 1×1 convolution operation;

[0086] Step 5.2.5: Add the attention feature map to the residual information to obtain the global attention feature F after global attention adjustment. global_att :

[0087] F global_att =Fspatial +f residual ;

[0088] Step 5.3: Enhance channel attention to enable precise focus on detailed information in the forged region using local features; first, fuse the eye spatial frequency features. Mouth spatial frequency fusion characteristics Channel attention is used for fusion:

[0089]

[0090] Among them, A eye and A mouth These are the attention weights for local features of the eyes and mouth, respectively; F local This is a local fusion feature;

[0091] Step 5.4: Employ an adaptive fusion weighting mechanism to fuse the local features F. local With global attention feature F global_att Dynamic fusion is performed to obtain the global fusion feature F. global_fused :

[0092] G=σ(Conv 1×1 Concat(F local ,F global_att ));

[0093] F global_fused =G⊙F global_att +(1-G)⊙F local ;

[0094] Where Concat(·) means concatenating along the channel dimension; G is the fusion weight.

[0095] Furthermore, in step 6, the category-aware multi-task loss module employs a multi-task learning strategy, combining local and global classification, information alignment, and category prototype optimization across multiple tasks, automatically weighting and balancing the losses of each task; the specific process is as follows:

[0096] Step 6.1: Analyze the global fusion feature F global_fused Perform standard cross-entropy loss calculation to obtain the main cross-entropy loss.

[0097]

[0098] in, Let y be the i-th class of the unique hot label of the b-th sample; C is the number of classes; B is the total number of samples; i For the i-th type of unique hot tags; Let be the fusion feature of the b-th sample; p(·) is the predicted probability of the b-th sample in category i;

[0099] Step 6.2: Calculate the global fusion feature F using KL divergence. global_fused With global feature F global Mutual information; design global cross-entropy loss and global mutual information loss as follows:

[0100]

[0101] Where T is the temperature parameter; D K (·) represents the function for calculating KL divergence; These are the global fusion features and the space-frequency fusion features of the b-th sample, respectively. y is the softmax normalization function; y is the class position of the real and fake categories in the prediction distribution;

[0102] Step 6.3: The local information loss part uses KL divergence to calculate the eye spatial-frequency fusion features. Mouth spatial frequency fusion characteristics Mutual information; designing local mutual information loss as follows:

[0103]

[0104] Where B represents the total number of samples; P (b) This is the one-hot label distribution of the b-th sample; and This is the scaled predicted distribution of the eye and mouth temperatures of the b-th sample;

[0105] A local feature linear layer is used to calculate the unnormalized predicted values ​​for the eyes and mouth of each sample, and a temperature parameter T is applied to the unnormalized predicted values ​​to smooth the predicted distribution Q.

[0106]

[0107] in, W represents the unnormalized predicted values ​​for the eyes and mouth of the b-th sample; local b represents the weight matrix of the local feature linear layer; local This represents the bias vector of the local feature linear layer;

[0108] Step 6.4: Introduce the category prototype alignment loss to align each local region separately; first, define the category prototype center:

[0109]

[0110] in, Indicates the first Feature representation of each cluster center; For belonging to the first Samples from one cluster; This represents the local features of the b-th sample;

[0111] Then, calculate the L2 distance between the local features and the class prototype:

[0112]

[0113] in, The category prototype alignment loss for the eye; The class prototype alignment loss for the mouth; This results in localized information loss. For the yth (b) Feature representation of each cluster center;

[0114] Step 6.5: Employ an automatic weighting mechanism to balance the importance of different loss terms:

[0115]

[0116] in, This is the total loss function; Indicates the first Adaptive learning weights for each loss term, These correspond to global cross-entropy loss, local information loss, and principal cross-entropy loss, respectively. The first The, the Each loss term is a learnable hyperparameter.

[0117] The beneficial technical effects of this invention are as follows: (1) It realizes the extraction of frequency domain information in areas of subtle tampering. Multi-scale wavelet transform is used to extract frequency domain features in areas (eyes, mouth) where there is a lot of forgery in deepfake videos, revealing subtle anomalies in the forged content and more accurately identifying the features of deepfake videos. (2) It realizes the fusion and optimization of multi-dimensional features. By using dynamic attention fusion of dual-domain features and dynamic fusion of local and global features, it can adapt to different types of forgery methods in deepfake detection tasks, accurately fuse useful information, and improve the accuracy and generalization of detection. (3) It adopts a multi-task learning strategy, combining multiple tasks such as global-local classification, information alignment, and category prototype optimization. By automatically weighting and balancing the loss of each task, the model becomes more robust in deepfake detection tasks and improves its adaptability to different forgery modes. Attached Figure Description

[0118] Figure 1 This is a process diagram of the deepfake detection research method based on multi-dimensional feature collaborative modeling of the present invention.

[0119] Figure 2 This is the ROC curve of the present invention on the FaceForensics++ dataset HQ version.

[0120] Figure 3 This is the ROC curve of the cross-dataset generalization detection of this invention on the Celeb-DF(v2) dataset. Detailed Implementation

[0121] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0122] like Figure 1 As shown, this invention proposes a deepfake video detection method based on multi-dimensional feature collaborative modeling. This method fuses multi-dimensional features from the local and global domains, as well as the spatial and frequency domains, to collaboratively construct a fake video prediction model. This model mainly includes a multi-scale wavelet feature processing module, a position-aware dual-domain dynamic attention fusion module, a dual-gated dynamic alignment fusion module, and a category-aware multi-task loss module. The specific steps of this invention are as follows:

[0123] Step 1: Obtain a complete face image and preprocess it using face recognition technology to obtain images of local face regions. The specific process is as follows:

[0124] Step 1.1: Acquire video frames using a video capture device, and preprocess the video frames using MTCNN (Multi-Task Cascaded Convolutional Neural Network) face recognition technology. During preprocessing, the images are cropped to a size of 224*224 to obtain complete face images. in For batch size, Here, H represents the number of channels, and H and W represent the height and width of the image, respectively.

[0125] Step 1.2: Use dlib facial landmark detection technology to locate the face, and X-ray the complete face image. global Cropped images of the eyes and mouth at preset sizes are extracted, and X is defined. eye X mouth These are partial images of the eyes and mouth, respectively.

[0126] Step 2: Extract frequency domain features of local facial regions using a multi-scale wavelet feature processing module. This module employs a multi-scale wavelet feature processing mechanism, which inputs the local facial regions individually into a multi-scale wavelet transform and converts them into YCrCb-processed frequency domain features. The specific process is as follows:

[0127] Step 2.1: To efficiently process multiple images and regions, the preprocessed eye and mouth region images are first divided into batches. Each batch is processed separately to ensure that the network can perform efficient parallel computation within local regions.

[0128] Step 2.2: Extract the cropped individual face region images from the batch, and use the cross-platform computer vision library OpenCV to convert the RGB images into YCrCb images. YCrCb images allow for separate analysis of the Y channel (luminance channel), Cr channel, and Cb channel (chrominance channel). Given an RGB image of a local region l on a person's face as input, the converted YCrCb image is... The image conversion formula is as follows:

[0129]

[0130] Where l represents a local area on the face; when l = eye, it represents the eye; when l = mouth, it represents the mouth; Y (l) It is the brightness channel at position l, which contains structural information of the image; while Cr (l) and Cb (l) These are the red and blue chromaticity channels at position l, capturing color information; R, G, and B are the red, green, and blue color channels, respectively.

[0131] Step 2.3: Deepfake videos often introduce discontinuities or abnormal changes in brightness in the tampered areas. This invention focuses on extracting frequency domain features in the Y channel. The number of wavelet decomposition layers is selected, and the Daubechies3 wavelet basis is chosen based on the focus and computational complexity of deepfake video detection. The Y channel (brightness channel) is decomposed into components of different scales, outputting both low-frequency and high-frequency components at position l. The decomposition formula is as follows:

[0132]

[0133] Where S is the number of decomposition levels; cA (l) It is the low-frequency component at point l. The low-frequency component is an approximation coefficient, which usually contains low-frequency information. cV s (l) , For the high-frequency components of scale s at point l in the horizontal, vertical, and diagonal directions, the high-frequency components are a type of detail coefficient that typically contains high-frequency information; This represents the multi-scale wavelet transform operation.

[0134] Step 2.4: Extract the high-frequency component at position l. cV s (l) , The components are stitched along the channel dimension, and the high-frequency components after stitching are enhanced to highlight details. Specifically, the high-frequency components at module input l are... cV s (l) , First, high-frequency concatenated features are constructed along the channel dimension. Then, a 3×3 convolutional layer expands the channel dimension from 3 to 16 to extract more channel feature information. Next, the ReLU activation function is applied to make edge and texture information more prominent. Finally, a 3×3 convolutional layer restores the input channel dimension, outputting the high-frequency feature map at position l.

[0135]

[0136] in, Represents a high-frequency enhancement function; ⊕ represents channel dimension concatenation; Conv 3×3 This represents a 3×3 convolution operation; This represents the ReLU activation function.

[0137] At the same time, the low-frequency component cA at point l (l) Low-frequency enhancement processing is performed to enhance the global structural features of the image. The low-frequency component cA at input l... (l) First, the feature map is passed through a convolutional layer with a 3×3 kernel. Then, a ReLU activation function is applied to introduce non-linearity. To facilitate subsequent fusion, adaptive average pooling is used to downsample the feature map to 7×13, outputting the low-frequency feature map at position l.

[0138]

[0139] in, Represents a low-frequency enhancement function; Avg (H,W) This indicates adaptive average pooling in the spatial dimensions (height H and width W); Conv 3×3 This represents a 3×3 convolution operation; This represents the ReLU activation function.

[0140] Step 2.5: To achieve joint modeling of luminance structural features and color information, Cr... (l) and Cb (l) The channel is adjusted to match the low-frequency feature map using bilinear interpolation. Same size:

[0141]

[0142] Among them, Cr′ (l) 、Cb′ (l)These represent the low-frequency enhancement features of the red and blue chromaticity channels at position l, respectively. This is a bilinear interpolation function used to adjust the size of features; The target size is derived from the shape of the low-frequency enhanced features; Shape(·) extracts the size information of the input features.

[0143] Step 2.6, Cr′ (l) 、Cb′ (l) and By stitching along the channel dimension, the final frequency domain features of the local location 'l' on the face are formed.

[0144]

[0145] Concat(·) means concatenating along the channel dimension.

[0146] The processed features are stacked along the batch dimension to form a complete batch output, yielding the frequency domain features of the eyes and mouth.

[0147] Step 3: Based on the first six layers of the pre-trained deep convolutional neural network ResNet34, extract the spatial features of local facial regions (eyes, mouth). The first six layers of ResNet34 form a shallow network that can effectively extract basic image features, retain useful details, and avoid introducing too much noise into complex features, thus affecting detection accuracy. The formula is:

[0148]

[0149] in, The spatial characteristics at point l; φ ResNet34_pre (·) represents the feature extraction function of the first six layers of the deep convolutional neural network ResNet34.

[0150] Step 4: A position-aware dual-domain dynamic attention fusion module is used to perform adaptive weight modulation and nonlinear fusion based on position coding on the extracted spatial and frequency domain features to obtain the final spatial-frequency fused features. The specific process is as follows:

[0151] Step 4.1: Extract the frequency domain features at position l. An adaptive filter is applied, dynamically adjusting its parameters based on input features to further enhance the representation capability of frequency domain features. The adaptive filter utilizes group convolutions of depthwise separable convolutions, allowing each frequency sub-band to be filtered independently, avoiding interference from cross-channel information. The threshold transition characteristics of the GELU activation function prevent the loss of high-frequency information. The filter formula is as follows:

[0152]

[0153] in, The frequency domain feature at position l is obtained after passing through the adaptive filter; ⊙ represents the element-wise multiplication operation;

[0154] DEConv 3×3 (·) represents depthwise separable convolution; Conv 1×1 This represents a 1×1 convolution operation; This is the GELU activation function.

[0155] Step 4.2: Extract the spatial features at position l. and frequency domain characteristics after adaptive filter processing Each feature channel is weighted using channel attention, allowing the network to focus on the most discriminative channel. The weighting of spatial and frequency domain features can be expressed by the following formula:

[0156]

[0157] Where MaxPool(·) is the max pooling operation; σ(·) represents the sigmoid normalization function; These represent the spatial attention weights and frequency attention weights generated at position l, respectively; ⊙ represents the element-wise multiplication operation; Conv 1×1 This represents a 1×1 convolution operation; These are the spatial and frequency domain features at position l after weighting, respectively;

[0158] Step 4.3: Fuse frequency domain and pixel-level coordinate encoding into a model to realize the spatial features of local regions. and frequency domain features The dual-domain fusion. The specific process is as follows:

[0159] Step 4.3.1: To achieve pixel-level spatial perception fusion modeling, a normalized spatial coordinate grid is first generated on the two-dimensional image plane:

[0160]

[0161] Where u is a normalized coordinate sequence in the horizontal direction, ranging from -1 to 1, with a total of One point, The width of the spatial feature map. is the index of the point in the horizontal direction; v is the normalized coordinate sequence in the vertical direction, ranging from -1 to 1, with a total of One point, is the length of the spatial feature map, and j is the index of the point in the vertical direction.

[0162] Subsequently, a multi-frequency activation-driven spatial coordinate encoding function is designed. Through dual sine mapping and cosine perturbation, a nested sine-cosine combination transformation is performed on the coordinates (u,v) of each pixel position. A hyperparameter γ is introduced to control the amplitude of the nonlinear term, thus constructing a nonlinear periodic spatial representation.

[0163]

[0164] Where Φ(u,v) is the spatial coordinate encoding; tanh is the hyperbolic tangent function, used to limit the output to the range [-1,1] to enhance boundary smoothness; W1, W2, W3 represent different weight matrices; b1, b2, b3 represent different bias vectors; This is a matrix transpose operation.

[0165] Step 4.3.2: Analyze the frequency domain characteristics at point l obtained after the adaptive filter. Three convolutional kernels of different scales (1×1, 3×3, and 5×5) are applied to extract multi-scale frequency domain responses, which are then fused to obtain a unified multi-scale feature vector G(F″) at position l. fre ) (l) :

[0166]

[0167] Among them, Conv k×k This represents a k×k convolution operation, where k is the kernel size.

[0168] Simultaneously, higher-order statistics of frequency domain features are introduced as supplementary information to jointly construct a structure-statistical fusion vector.

[0169] Where AvgPool(·) represents global average pooling; σ 2 (·) represents the unbiased variance in the spatial dimension; MaxPool(·) represents the max pooling operation; ∈ is a stability constant to prevent numerical instability.

[0170] Finally, the fused vector is modulated and coded using a multilayer perceptron to generate a frequency domain modulation vector Ψ(F″) at position l. fre ) (l) This is used to guide the subsequent space-frequency fusion process, enabling more discriminative cross-domain feature enhancement:

[0171]

[0172] Among them, f mlp (·) denotes a multilayer perceptron function; Flatten(·) denotes a feature flattening operation.

[0173] Step 4.3.3: In order to generate a weight map for nonlinear adaptive fusion, the spatial coordinate code Φ(u,v) generated in step 4.3.1 and the frequency domain modulation vector Ψ(F″) obtained in step 4.3.2 are combined. fre ) (l) A concatenation operation is performed along the feature dimension to construct the cross-domain fusion feature matrix Z at point l. (l) :

[0174]

[0175] in, This indicates a broadcast extended copy operation; d represents the dimension after coordinate encoding and frequency encoding.

[0176] Step 4.3.4, Z (l) The input is mapped into a multilayer perceptron structure, and the explicit fusion weight W at l is generated by the sigmoid normalization function. (l) :

[0177]

[0178] Step 4.3.5: Finally, the spatial and frequency domain features are weighted and fused at the pixel level using this weight to obtain the final spatial-frequency fused features:

[0179]

[0180] in, This represents the spatial-frequency fusion feature at point l.

[0181] Step 5: Employ a dual-gated dynamic alignment and fusion module to adjust the contribution of local regions in the local-global feature fusion process, achieving efficient feature fusion and obtaining the final global fused features. The specific process is as follows:

[0182] Step 5.1: Use a deep convolutional neural network ResNet34 to extract the complete face X. global global features F global It captures the global context information of the image.

[0183] F global =φ ResNet34 (X global );

[0184] Where, φ ResNet34 (·) is the feature extraction function of the deep convolutional neural network ResNet34.

[0185] Step 5.2: To further enhance the discriminative power of global features, an adaptive global attention mechanism is introduced, employing a structure combining bidirectional channel attention and lightweight spatial attention. Finally, residual connections are used to enhance feature representation. The specific process is as follows:

[0186] Step 5.2.1: First, extract channels using global average pooling and global max pooling respectively to obtain the channel attention output F. channel :

[0187] F channel =σ(Conv 1×1 (ReLU(Conv1(AvgPool(F global ))))+Conv2(ReLU(Conv 1×1 (MaxPool(F global )))))⊙F global ;

[0188] Where AvgPool(·) represents global average pooling; MaxPool(·) represents max pooling; ⊙ represents element-wise multiplication; Conv 1×1 σ(·) represents a 1×1 convolution operation; σ(·) represents the sigmoid normalization function, which normalizes the attention weights to the range [0,1]. This is the ReLU activation function.

[0189] Step 5.2.2: Subsequently, using lightweight spatial attention, the average and maximum values ​​of all channels at each spatial location are calculated to obtain the average spatial feature map S. avg and the maximum spatial feature map S max The two data points are concatenated along the channel dimension and then fed into a 7×7 convolutional layer to generate a spatial attention weight map M. spatial The formula is:

[0190]

[0191] M spatial =σ(Conv 7×7 (Concat(S avg ,s max )));

[0192] in, This represents the number of channels for the input feature; Represents the channel index; Concat(·) concatenates along the channel dimension; σ(·) represents the sigmoid normalization function, which normalizes the attention weights to the range [0,1]; Conv 7×7 This represents a 7×7 convolution operation; For channel The corresponding channel attention output.

[0193] Step 5.2.3: Finally, the spatial attention weight map M... spatial With channel attention output F channel Element-wise multiplication enhances features in the spatial dimension, resulting in the attention feature map F. spatial :

[0194] F spatial =M spatial ⊙F channel ;

[0195] Step 5.2.4: Process the global features F of the input. global The residual information F is obtained by using 1×1 convolution, BatchNorm (batch normalization), and ReLU activation. residual :

[0196]

[0197] in, ReLU activation function; BN represents batch normalization; Conv 1×1 This represents a 1×1 convolution operation.

[0198] Step 5.2.5: Finally, add the attention feature map to the residual information to obtain the global attention feature F after global attention adjustment. global_att :

[0199] F global_att =F spatial +F residual ;

[0200] Step 5.3: To enhance the representational power of local features and allow the model to dynamically adjust the importance of key regions, channel attention enhancement is used to make local features more accurately focus on the details of the fake region. First, the eye spatial-frequency fusion features are... Mouth spatial frequency fusion characteristics Channel attention is used for fusion:

[0201]

[0202] Among them, Conv 1×1 It is a 1×1 convolution operation; σ(·) is the sigmoid normalization function, which normalizes the attention weights to the range [0,1]; A eye and A mouth These are the attention weights for local features of the eyes and mouth, respectively; F local This is a local fusion feature.

[0203] Step 5.4, Local Fusion Feature Flocal With global attention feature F global_att Dynamic fusion is required to obtain the global fusion feature F. global_fused To ensure flexibility in adjusting across different forgery modes, an adaptive fusion weighting mechanism is employed.

[0204] G=σ(Conv 1×1 Concat(F local ,F global_att ));

[0205] F global_fused =G⊙F global_att +(1-G)⊙F local ;

[0206] Among them, Conv 1×1 The first line represents a 1×1 convolution operation, and the second line represents concatenation along the channel dimension. This is used to calculate the importance of local and global features. The fusion weights G enable the network to adaptively select the contribution of global or local features, thereby improving the generalization ability of the detection.

[0207] Step 6: The category-aware multi-task loss module employs a multi-task learning strategy, combining multiple tasks such as local and global classification, information alignment, and category prototype optimization. By automatically weighting and balancing the losses of each task, this module achieves predictive classification of forged videos through multi-level, multi-task collaborative supervised classification. The specific process is as follows:

[0208] Step 6.1, Global Fusion Feature F global_fused Perform standard cross-entropy loss calculation to obtain the main cross-entropy loss. Provides a strong monitoring signal:

[0209]

[0210] in, Let y be the i-th class of the one-hot label of the b-th sample; C is the number of classes; B is the total number of samples; i For the i-th type of unique hot tags; Let be the fusion feature of the b-th sample; p(·) is the predicted probability of the b-th sample in category i.

[0211] Step 6.2: Calculate the global fusion feature F using KL divergence (also known as relative entropy or cross-entropy). global_fused With global feature F global Mutual information is used to ensure that global features contribute to the classification task. The global cross-entropy loss is calculated. and global mutual information loss

[0212]

[0213] Where T is the temperature parameter, which makes the distribution smoother and improves gradient stability; It is the softmax normalization function. D K (·) represents the function for calculating KL divergence; y represents the global fusion feature and spatial-frequency fusion feature of the b-th sample, respectively; y is the category position of the real and fake categories in the prediction distribution.

[0214] Step 6.3: The local information loss part uses KL divergence to calculate the eye spatial-frequency fusion features. Mouth spatial frequency fusion characteristics Mutual information is used to ensure that local features can work together and prevent information conflicts. Local mutual information loss. The calculation is as follows:

[0215]

[0216] Where B represents the total number of samples; and This is the scaled predicted distribution of the eye and mouth temperatures for the b-th sample; P (b) Let be the one-hot label distribution of the b-th sample. Using the one-hot labels as labels for mutual information alignment, the following condition must be met:

[0217]

[0218] When the category index i matches the true category label y of the current b-th sample (b) If they are equal, the value is 1; otherwise, the value is 0.

[0219] A local feature linear layer is used to calculate the unnormalized predicted values ​​of the eyes and mouth for each sample, thereby achieving the prediction and classification of forged images. A temperature parameter T is then applied to the unnormalized predicted values ​​(logits) to smooth the prediction distribution Q.

[0220]

[0221] in, W represents the unnormalized predicted values ​​(logits) for the eyes and mouth of the b-th sample; local b represents the weight matrix of the local feature linear layer; local The bias vector represents the local feature linear layer; y is the class position of the real and fake classes in the prediction distribution, and the predicted value is... Each dimension corresponds to a position in the category y, which is mapped to the predicted probability of each category in the probability space via softmax. Therefore, y is the position of each category in the softmax output, and These are the original predicted scores for these categories, which correspond one-to-one; T is the temperature parameter, which makes the distribution smoother and improves gradient stability.

[0222] Step 6.4: To further optimize local features, a class prototype alignment loss is introduced to align each local region separately, enhancing the representational ability of different forged regions, ensuring that local features are close to the correct class center, improving feature discriminative power, and achieving multi-region information integration. First, define the class prototype center:

[0223]

[0224] in, Indicates the first Feature representation of each cluster center; For belonging to the first Samples from one cluster; Indicates belonging to the first Take the reciprocal of the number of samples in each cluster and calculate the average. This represents the local features of the b-th sample.

[0225] Then, calculate the L2 distance between the local features and the class prototype:

[0226]

[0227] in, The category prototype alignment loss for the eye; The class prototype alignment loss for the mouth; This results in localized information loss. For the yth (b) The feature representation of each cluster center.

[0228] Step 6.5: To balance the importance of different loss terms, this invention employs an automatic weighting mechanism to construct the total loss function:

[0229]

[0230] in, This is the total loss function; Indicates the first Adaptive learning weights for each loss term, These correspond to global cross-entropy loss, local information loss, and main cross-entropy loss, respectively. The first The, the Each loss term is a learnable hyperparameter that is automatically optimized during model training; the summation index in the denominator. This means that all loss terms are normalized to ensure that the impact of different loss terms can be dynamically adjusted, and to prevent any one loss term from dominating the optimization process.

[0231] This invention provides a deepfake detection method based on multi-dimensional feature collaborative modeling. By designing a multi-scale wavelet feature processing mechanism, dual-domain dynamic attention fusion, and a multi-level, multi-task collaborative classification supervision mechanism, it improves the accuracy of deepfake detection across datasets. The traditional method of spatial phase shallow learning (SPSL) for spatial-frequency feature fusion is used as a baseline for comparison of experimental results. Evaluation metrics include: accuracy (ACC), receptivity curve (ROC curve), and area under the receptivity curve (AUC).

[0232] ACC measures the proportion of samples correctly classified by the model on the test set, and is the most intuitive evaluation metric. The specific calculation formula is as follows:

[0233]

[0234] Here, ACC represents the accuracy rate.

[0235] The ROC curve (also known as the receptivity curve) uses the True Positive Rate (TPR) on the vertical axis and the False Positive Rate (FPR) on the horizontal axis. It can intuitively evaluate the classification performance of a compressed face image discrimination classifier. The specific calculation formula is as follows:

[0236]

[0237] Wherein, TPR is the true positive rate; FPR is the false positive rate; TP represents true positive, which is the number of samples whose actual label is positive and whose prediction result is also positive; FP represents false positive, which is the number of samples whose actual label is negative and whose prediction result is also positive; TN represents true negative, which is the number of samples whose actual label is negative and whose prediction result is also negative; and FN represents false negative, which is the number of samples whose actual label is positive and whose prediction result is also negative.

[0238] AUC stands for Area Under the ROC Curve, and is used numerically to evaluate classification ability.

[0239] This invention evaluates its performance on the FaceForensics++ and Celeb-DF(v2) datasets. The FaceForensics++ dataset contains 1000 real videos and 4000 deepfake videos. The real videos are original videos from YouTube, and the deepfake videos are generated using four deepfake techniques based on the original videos. The dataset is divided into high-quality (HQ) and low-quality (LQ) versions based on different compression levels. The Celeb-DF(v2) dataset has two versions, v1 and v2, with v2 being an extension of v1. v1 contains 408 original videos and 795 deepfake videos. v2 contains 590 original videos and 5639 deepfake videos. This invention trains the model using the FF++(HQ) dataset and tests it on both the FF++(HQ) and Celeb-DF(v2) datasets. The detection performance within the same dataset is tested on the FF++(HQ) dataset, while the detection performance across datasets is tested on the Celeb-DF(v2) dataset.

[0240] The comparison results of the accuracy (ACC) and area under the sensitivity curve (AUC) of the method of the present invention with those of SPSL are shown in Table 1 below:

[0241] Table 1. Accuracy comparison between within and across datasets.

[0242]

[0243] As shown in Table 1, the present invention improves upon the baseline in both monitoring accuracy within the dataset and generalization across datasets. Within the FF++ (HQ) dataset, ACC and AUC are improved by 2.62% and 3.38%, respectively. Significant improvements are observed on the Celeb-DF (v2) dataset, with ACC and AUC increasing by 13.01% and 13.53%, respectively, compared to the baseline method.

[0244] Figure 2 This is the ROC curve of the present invention on the FaceForensics++ dataset HQ version. Figure 3 This is the ROC curve of the generalization detection across datasets using the Celeb-DF(v2) dataset, based on the present invention. From... Figure 2 and Figure 3 As can be seen, the present invention has good detection capabilities on the FaceForensics++ dataset and also demonstrates strong generalization ability on the Celeb-DF(v2) dataset.

[0245] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A deepfake video detection method based on multi-dimensional feature collaborative modeling, characterized in that, Includes the following steps: Step 1: Obtain a complete face image and use face recognition technology to preprocess the complete face image to obtain an image of a local area of ​​the face; Step 2: Extract frequency domain features of local facial regions using a multi-scale wavelet feature processing module; Step 3: Extract spatial features of local facial regions based on the first six layers of the pre-trained deep convolutional neural network ResNet34. Step 4: Adaptive weight modulation and nonlinear fusion are performed on the extracted spatial and frequency domain features using a position-aware dual-domain dynamic attention fusion module; the specific process is as follows: Step 4.1: For the already extracted... Frequency domain characteristics at An adaptive filter is applied for processing, as shown in the following formula: ； in, The result obtained after passing through the adaptive filter Frequency domain characteristics at that location; For depthwise separable convolution; This represents a 1×1 convolution operation; This indicates a batch normalization operation; This represents the GELU activation function; Step 4.2: For the already extracted... Spatial characteristics at the location and the result after processing with an adaptive filter Frequency domain characteristics at Each feature channel is weighted using channel attention, as shown in the formula: ； ； ； ； in, This is a max pooling operation; This represents the sigmoid normalization function; , They represent Spatial domain attention weights and frequency domain attention weights generated at the specified location; This represents the element-wise multiplication operation; , They are weighted as follows Spatial and frequency domain characteristics at the location; Represents the ReLU activation function; Step 4.3: Fuse the frequency domain and pixel-level coordinate encoding into a model, and perform dual-domain fusion of spatial and frequency domain features of the local region; Step 5: Use a dual-gated dynamic alignment fusion module to adjust the contribution of local regions in the local-global feature fusion process to obtain the final global fused features; Step 6: The loss of each task is automatically weighted and balanced based on the category-aware multi-task loss module to achieve prediction and classification of fake videos.

2. The deepfake video detection method based on multi-dimensional feature collaborative modeling according to claim 1, characterized in that, The specific process of step 1 is as follows: Step 1.1: Use a video capture device to acquire video frames, and use multi-task cascaded convolutional neural network face recognition technology to preprocess the video frames. During preprocessing, crop the images to a size of 224*224 to obtain complete face images. Step 1.2: Use dlib facial landmark detection technology to crop out local areas of the eyes and mouth from the complete face image according to the preset size.

3. The deepfake video detection method based on multi-dimensional feature collaborative modeling according to claim 2, characterized in that, The specific process of step 2 is as follows: Step 2.1: Divide the preprocessed eye and mouth region images into batches; Step 2.2: Extract the cropped local facial region images from the batch, and use the cross-platform computer vision library OpenCV to convert the RGB images into YCrCb images; the image conversion formula is as follows: ； ； Among them, is the position of a local area on the human face, when it represents the eyes, when it represents the mouth; is the luminance channel at the position; and are respectively the chrominance channels of red and blue at the position; , , are respectively the three color channels of red, green, and blue; is the YCrCb image at the position after conversion ; Step 2.3: Using the Dobessi wavelet basis, the brightness channel is decomposed into components of different scales, and the output includes... The formulas for the low-frequency and high-frequency components at a given point are as follows: ； in, It represents the number of levels in the decomposition. yes Low-frequency components at the location; , , for scale at the location High-frequency components in the horizontal, vertical, and diagonal directions; This represents a multi-scale wavelet transform operation; Step 2.4, extract the... High-frequency components at the location , The components are stitched along the channel dimension, and the high-frequency components after stitching are then subjected to high-frequency enhancement processing; the specific process is as follows: First, the high-frequency components are stitched together along the channel dimension. High-frequency components at the location , The input channel dimension is concatenated into high-frequency concatenated features, then expanded from 3 to 16 through a 3×3 convolutional layer. Next, the ReLU activation function is applied, and finally, another 3×3 convolutional layer restores the input channel dimension, resulting in the output. High-frequency feature map at the location : ； in, Represents a high-frequency enhancement function; Indicates concatenation of channel dimensions; This represents a 3×3 convolution operation; At the same time, low-frequency components Perform low-frequency enhancement processing: First low-frequency components After passing through a convolutional layer with a 3×3 kernel, a ReLU activation function is applied to introduce non-linearity. Finally, adaptive average pooling is used to downsample the feature map to 7×13, and the output is... Low-frequency feature map at the location : ； in, This represents a low-frequency enhancement function; To achieve adaptive average pooling in the spatial dimension; and These are the height and width of the image, respectively; This represents a 3×3 convolution operation; Step 2.5, and The channel is adjusted to match the low-frequency feature map using bilinear interpolation. Same size: ； ； in, , They are respectively Low-frequency enhancement features in the red and blue chroma channels; It is a bilinear interpolation function; Target size; To extract the size information of the input features; Step 2.6, , , and By splicing along the channel dimension, the final shape is formed. Frequency domain characteristics at : ； in, This indicates splicing along the channel dimension.

4. The deepfake video detection method based on multi-dimensional feature collaborative modeling according to claim 3, characterized in that, The formula for step 3 is: ； in, for Spatial characteristics at that location; The feature extraction function for the first six layers of the deep convolutional neural network ResNet34; for The RGB image at that location.

5. The deepfake video detection method based on multi-dimensional feature collaborative modeling according to claim 4, characterized in that, The specific process of step 4.3 is as follows: Step 4.3.1: Perform pixel-level spatial perception fusion modeling; first, generate a normalized spatial coordinate grid on the two-dimensional image plane: ； ； in, It is a normalized coordinate sequence in the horizontal direction; The width of the spatial feature map; The index is the serial number of the point in the horizontal direction; It is a normalized coordinate sequence in the vertical direction; The length of the spatial feature map; The index of the point in the vertical direction; Subsequently, a multi-frequency activation-driven spatial coordinate encoding function is designed: ； in, Encode spatial coordinates; It is the hyperbolic tangent function; , , Representing different weight matrices; , , Representing different bias vectors; This is a matrix transpose operation; For hyperparameters; Step 4.3.2: The result obtained after the adaptive filter... Frequency domain characteristics at Three different scales of convolutional kernels are applied to extract multi-scale frequency domain responses, which are then fused to obtain... Unified multi-scale feature vectors at the location : ； in, express Convolution operation, The kernel size; Simultaneously, higher-order statistics of frequency domain features are introduced as supplementary information to jointly construct a structure-statistical fusion vector. : ； in, Indicates global average pooling; This represents the unbiased variance in the spatial dimension; This represents the max pooling operation; It is a stability constant; Finally, modulation codes are generated through a multilayer perceptron to generate... Frequency domain modulation vector at : ； in, Represents the function of a multilayer perceptron; Indicates the feature flattening operation; Step 4.3.3: Encode the spatial coordinates generated in step 4.3.

1. The frequency domain modulation vector obtained in step 4.3.2 Perform cascaded operations along the feature dimension to construct... Cross-domain fusion feature matrix : ； in, Indicates a broadcast extended copy operation; Step 4.3.4, will The input is mapped into a multilayer perceptron structure and generated by a sigmoid normalization function. Explicit fusion weights at the location : ； Step 4.3.5: Use this weight to perform pixel-level weighted fusion of spatial and frequency domain features to obtain the final spatial-frequency fused features: ； in, for Spatial-frequency fusion characteristics at the location.

6. The deepfake video detection method based on multi-dimensional feature collaborative modeling according to claim 5, characterized in that, The specific process of step 5 is as follows: Step 5.1: Extract the complete face image using a deep convolutional neural network ResNet34. global features : ； in, Extracting feature functions for the deep convolutional neural network ResNet34; Step 5.2: Introduce an adaptive global attention mechanism, employing a structure combining bidirectional channel attention and lightweight spatial attention, ultimately enhancing feature representation through residual connections; the specific process is as follows: Step 5.2.1: Extract channels using global average pooling and global max pooling respectively to obtain the channel attention output. : ； in, This represents a 1×1 convolution operation; Step 5.2.2: Calculate the average and maximum values ​​for all channels at each spatial location using lightweight spatial attention to obtain the average spatial feature map. and maximum spatial feature map The two data points are concatenated along the channel dimension and then fed into a 7×7 convolutional layer to generate a spatial attention weight map. The formula is: ； ； ； in, This represents the number of channels for the input feature; Indicates the channel index; This indicates splicing along the channel dimension; This represents a 7×7 convolution operation; For channel The corresponding channel attention output; Step 5.2.3: Calculate the spatial attention weight map. Output of channel attention Element-wise multiplication yields the attention feature map. : ； Step 5.2.4: Global features of the input Residual information is obtained by using 1×1 convolution, batch normalization, and ReLU activation. : ； in, This represents a 1×1 convolution operation; Step 5.2.5: Add the attention feature map to the residual information to obtain the global attention feature after global attention adjustment. : ； Step 5.3: Enhance channel attention to enable precise focus on detailed information in the forged region using local features; first, fuse the eye spatial frequency features. Mouth spatial frequency fusion characteristics Channel attention is used for fusion: ； ； ； in, and These are the attention weights for local features of the eyes and mouth, respectively. This is a local fusion feature; Step 5.4: Employ an adaptive fusion weight mechanism to fuse local features. With global attention features Dynamic fusion is performed to obtain global fusion features. : ； ； in, This indicates splicing along the channel dimension; For weight fusion.

7. The deepfake video detection method based on multi-dimensional feature collaborative modeling according to claim 6, characterized in that, In step 6, the category-aware multi-task loss module employs a multi-task learning strategy, combining local and global classification, information alignment, and category prototype optimization across multiple tasks, automatically weighting and balancing the losses of each task; the specific process is as follows: Step 6.1: Global fusion features Perform standard cross-entropy loss calculation to obtain the main cross-entropy loss. : ； in, For the first The unique hot label of the sample kind; It is the number of categories; The total number of samples; The first of the hot tags kind; For the first The fusion features of each sample; For the first Each sample in category The predicted probability; Step 6.2: Calculate global fusion features using KL divergence. With global features Mutual information; design global cross-entropy loss and global mutual information loss as follows: ； ； in, It is a temperature parameter; The function representing the calculation of KL divergence; , The first Global fusion features and spatial-frequency fusion features of individual samples; It is the softmax normalization function; It predicts the class positions of the real and fake categories in the distribution; Step 6.3: The local information loss part uses KL divergence to calculate the eye spatial-frequency fusion features. Mouth spatial frequency fusion characteristics Mutual information; designing local mutual information loss as follows: ； in, Indicates the total number of samples; It is the first Distribution of unique hot labels for each sample; and It is the first Predicted distribution of scaled eye and mouth temperatures for each sample; A local feature linear layer is used to calculate the unnormalized predicted values ​​for the eyes and mouth of each sample, and a temperature parameter is applied to the unnormalized predicted values. Smooth the prediction distribution : ； ； in, For the first Unnormalized predicted values ​​for the eyes and mouth of each sample; This represents the weight matrix of the local feature linear layer; This represents the bias vector of the local feature linear layer; Step 6.4: Introduce the category prototype alignment loss to align each local region separately; first, define the category prototype center: ； in, Indicates the first Feature representation of each cluster center; For belonging to the first Samples from one cluster; Indicates the first Local features of a sample; Then, calculate the L2 distance between the local features and the class prototype: ； ； ； in, The category prototype alignment loss for the eye; The class prototype alignment loss for the mouth; This results in localized information loss. For the first Feature representation of each cluster center; Step 6.5: Employ an automatic weighting mechanism to balance the importance of different loss terms: ； ； in, This is the total loss function; Indicates the first Adaptive learning weights for each loss term, These correspond to global cross-entropy loss, local information loss, and primary cross-entropy loss, respectively. , The first The, the Each loss term is a learnable hyperparameter.

Images