Video-level deepfake detection method based on adaptive fusion of two-path features

Abstract

This invention discloses a video-level deep forgery detection method based on dual-path feature adaptive fusion. The specific steps are as follows: (1) The input video is fragmented in the time dimension, and the original video is divided into multiple video segments with fixed time lengths; (2) Appearance information based on RGB features is extracted for each video segment, and local energy changes based on wavelet features are extracted at the same time; (3) Based on the global statistical information of RGB features and wavelet features in step (2), a fusion weight for the modulation feature fusion process is generated; and a global learnable constraint factor is additionally set in the feature fusion modulation process to adaptively learn the injection intensity of the fusion weight under different forgery types; (4) The discrimination results of each video segment are aggregated at the video level, and the final detection result of the corresponding input video is output.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and deep learning technology, specifically relating to a video-level deep forgery detection method based on adaptive fusion of spatial domain and wavelet domain features. It is used to jointly model features from different feature spaces and use scene indication information to modulate the feature fusion process for video-level deep forgery detection. Background Technology

[0002] With the development of deep learning generative modeling technology, video forgery detection methods have gradually shifted from single-feature modeling to multi-branch feature joint modeling. However, in multi-branch fusion structures, different feature branches often exhibit different convergence characteristics in the early stages of training. This instability is further amplified when faced with mixed training settings involving multiple forgery generation mechanisms, as the differences in spatial distribution, temporal consistency, and wavelet domain statistical properties among different forgery types exacerbate these instabilities. Under these conditions, directly adopting a fixed feature fusion strategy can easily interfere with the learning of the main features, leading to slow training convergence.

[0003] Chinese patent publication CN 116665089A discloses a method for detecting deepfake videos based on a three-dimensional spatiotemporal network. By employing a three-dimensional spatiotemporal network, it can extract spatiotemporal features from deepfake videos, improving the detection accuracy. It adds optical flow acoustic feature extraction and noise feature extraction modules to the RGB domain, enabling the detection of inconsistencies in micro-expression changes and noise domain anomalies in fake videos, respectively. However, this patented technical solution does not solve the aforementioned problem of slow training convergence. Summary of the Invention

[0004] To address the aforementioned problems with existing deepfake detection technologies, this invention proposes a video-level deepfake detection method based on dual-path feature adaptive fusion.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The video-level deep forgery detection method based on dual-path feature adaptive fusion mainly includes the video input and segment generation stage, the dual-path feature extraction stage, the feature fusion modulation stage, and the video-level judgment and output stage. The specific details of each stage are as follows:

[0007] (1) Video input and segment generation: The input video is fragmented in the time dimension, and the original video is divided into multiple video segments with fixed time lengths in order to construct a video representation suitable for subsequent multi-instance learning;

[0008] (2) Dual-path feature extraction: Extract appearance information based on RGB features and local energy changes based on wavelet features for each video segment respectively;

[0009] (3) Feature fusion modulation: Based on the global statistical information of RGB features and wavelet features, fusion weights are generated for the modulation feature fusion process. An additional globally learnable constraint factor is set during the feature fusion modulation process to adaptively learn the injection strength of the fusion weights under different forgery types.

[0010] (4) Video-level judgment and output: The judgment results of each video segment are aggregated at the video level, and the final detection result of the corresponding input video is output, thereby realizing stable and reliable video-level deep forgery detection.

[0011] This invention uses a single forgery type training setting as a controlled experimental condition to reduce the interference of differences in forgery generation mechanisms on the feature fusion process. Based on this, adaptive and learnable fusion modulation parameters are introduced, enabling the model to dynamically adjust the injection intensity of wavelet features based on global statistical information during training. This suppresses the interference of the wavelet branch on the RGB branch in the early stages of training, improving the stability of feature fusion learning. Simultaneously, by introducing globally learnable parameters during the fusion modulation process, the modulation mechanism possesses the ability to adaptively adjust with data distribution, thereby avoiding excessive model dependence on a specific forgery type and providing a more robust feature fusion foundation for subsequent cross-forgery type applications.

[0012] Preferably, in step (1), the video segment generation step is as follows:

[0013] The dataset contains both real and fake videos in MP4 format. The dataset is partitioned according to the type of video forgery to generate training, validation, and test sets. These sets are then divided in a preset ratio (8:1:1) for model training, parameter tuning, and final performance evaluation.

[0014] The same segment generation process is performed on each video in the training set, validation set, and test set to ensure that the input format is consistent at different stages and to avoid data leakage.

[0015] During the data loading process, face detection and alignment are performed using a multi-task convolutional neural network (MTCNN) or dlib tool, and then the data is uniformly scaled to a resolution of 128x128 pixels.

[0016] The input video is segmented into multiple temporally continuous video segments using a sliding window of a preset time window length M. Each video segment contains a fixed number of consecutive frames to cover facial motion changes within a local time range. Adjacent video segments are sampled using an L-frame overlap method.

[0017] Preferably, in step (2), the multi-path feature extraction step is divided into an RGB feature extraction sub-step and a wavelet feature extraction sub-step. Dual-path feature extraction is performed on each video segment generated from the training set, validation set, and test set; wherein, the training set participates in parameter learning, while the validation set and test set only perform forward computation for model selection and performance evaluation.

[0018] Further optimized, the RGB feature extraction sub-step is as follows:

[0019] The preprocessed face video clips are used as input, and their input tensor dimensions are represented as follows: Where B represents the batch size, T is the number of frames contained in a sliding window, 3 represents the three RGB color channels, and H and W represent the height and width of the input image, respectively. First, the video clip is unfolded along the temporal dimension, and each frame is independently input into a parameter-shared convolutional neural network feature encoder (ResNet) to extract its spatial appearance features. The convolutional neural network feature encoder is constructed using a multi-layer convolutional structure to model the overall appearance structure, local texture distribution, and spatial semantic information of the face.

[0020] After processing by the feature encoder, each frame of the image is mapped to a high-dimensional feature representation, and its output feature dimension can be expressed as: Where C represents the number of feature channels output by the RGB spatial feature encoder (ResNet), This represents the spatial resolution of the feature map. Subsequently, global pooling is performed on the spatial feature map of each frame to obtain the frame-by-frame feature representation. Then, temporal average pooling is used to integrate the frame-by-frame features into a unified fragment-level feature representation. Through the above processing, the RGB feature extraction submodule maps the input video segment into a segment-level RGB feature representation for subsequent feature fusion and video-level determination.

[0021] The wavelet feature extraction sub-steps are further optimized as follows:

[0022] The preprocessed face video clip is used as input, and its input tensor dimension is... First, a discrete wavelet transform is independently performed on each frame of the face image in the video clip at full spatial resolution. The discrete wavelet transform applies a wavelet transform to each of the three RGB color channels of the image. For any channel, the wavelet transform decomposes the image into four sub-bands by performing low-pass / high-pass filtering in the row and column directions: a low-frequency sub-band LL and three high-frequency sub-bands LH, HL, and HH, used to represent the overall structural information and local detail changes of the image, respectively. The eye and mouth regions are then cropped from the four sub-bands and pre-constructed with all-zero 24 channels, with a size of [missing information]. The feature tensor of the eye and mouth regions is interpolated and filled into the 24-channel feature tensor according to the region coordinates. After the above processing, each frame of the image is mapped to a 24-channel wavelet domain feature representation, and its output dimension can be expressed as... ,in, This represents the number of channels in the channel dimension of the wavelet features composed of 4 sub-band features after wavelet transform of the eye and mouth regions; therefore, the value is 24. This indicates the spatial resolution of the corresponding sub-band features, consistent with the original face image. Subsequently, the wavelet domain features are input into a feature encoding network to model the wavelet features and output the number of feature channels C. The wavelet domain feature encoding network is constructed using a ResNet18 convolutional neural network to learn the correlation between low-frequency structural information and high-frequency texture changes. After processing by the ResNet18 encoder, the corresponding wavelet domain feature vector is obtained through global average pooling. Subsequently, temporal average pooling is used to integrate the frame-by-frame wavelet domain features into a unified fragment-level feature representation. The wavelet feature extraction sub-step maps the input video segment into a segment-level wavelet domain feature representation for subsequent feature fusion and video-level determination.

[0023] Furthermore, in obtaining the wavelet domain feature vector Subsequently, the corresponding energy characteristics were calculated for the low-frequency subband of the wavelet. Specifically, the process is as follows: First, the low-frequency subbands corresponding to the three RGB color channels are averaged along the channel dimension to obtain the low-frequency wavelet representation of a single channel. Then, the absolute value of the difference between adjacent frames is calculated along the time dimension, and averaged over the spatial and temporal dimensions to obtain the energy features. . in, The energy characteristics Broadcasting is performed along the channel dimension, thereby incorporating wavelet features. Addition. The energy modulation process does not introduce new feature channels, and the dimension of the wavelet domain feature vector remains consistent before and after energy injection.

[0024] Preferably, step (3), the feature fusion modulation process, is as follows:

[0025] Firstly, because All data are represented by fragment-level channel vectors, and the global statistical mapping uses the identity mapping. ,Right now .

[0026] Then Global statistical vector and wavelet global statistical vector Concatenation is performed at the channel dimension to obtain the joint context representation. The input is then fed into the fusion weight generation network. The fusion weight generation network uses a multilayer perceptron. Its network structure includes a first fully connected layer, a non-linear activation function, and a second fully connected layer. The function. The first fully connected layer is used to reduce the input dimension from... Mapping to hidden dimensions Nonlinear activation function The first layer is used to introduce non-linear expressive power; the second fully connected layer is used to output the fusion weight parameters. The function is used to constrain the output weights to... Interval. Obtain the fusion weight parameters: .

[0027] The fusion weights are used to characterize the wavelet features relative to... The degree to which a feature contributes under the current input conditions.

[0028] During the training phase, to further suppress the influence of wavelet features on the model's discriminative ability before it is fully established... Interference caused by feature learning is addressed by introducing a fusion strength constraint mechanism after the fusion weights are generated, thus constraining the fusion weights. Apply a preset fusion strength constraint factor The modulated fusion weights are obtained. Specifically, a globally learnable factor is set in the feature fusion module. It consists of trainable parameters through The mapping is obtained, making .

[0029] The validation and test sets no longer update the constraint factors and network parameters. Instead, they use the fusion weight parameters learned from the training set and the globally learnable factors for forward fusion to ensure the consistency and reproducibility of the evaluation.

[0030] Subsequently, after obtaining the modulated fusion weights, the wavelet features are weighted and injected into the system as residuals. Forming fused feature representations from features .in, This represents element-wise multiplication. By employing a residual fusion structure, it enables... Features remain dominant in the fused features, while controlled wavelet feature information is introduced to improve model stability in the early stages of training.

[0031] Preferably, step (4) includes three stages: segment-level forward prediction, multi-instance aggregation mechanism, and video-level output and loss calculation. Specifically:

[0032] After the feature fusion modulation step, the resulting fused feature representation is denoted as: .

[0033] First, fuse the features at the segment level for each video in the same batch. The video segments are grouped according to their video indexes to form a set of video-level segment features, which is then input into a segment-level prediction network for forward prediction. The segment-level prediction network preferably employs a multilayer perceptron structure, consisting of two fully connected layers, used to independently discriminate and model the fused features of each video segment. The first fully connected layer is used to integrate the segment-level fused features. Mapped to the intermediate discriminative feature space, its output dimension is The second fully connected layer maps intermediate features to the category space, and its output dimension is... These correspond to the unnormalized discrimination scores for the real video and fake video categories, respectively. The segment-level prediction network shares parameters across all video segments and independently performs forward prediction on the fusion features of each video segment in the current batch. The discrimination scores corresponding to each segment are stacked in segment order to obtain the segment-level discrimination score matrix, the output of which is expressed as: The two-dimensional outputs correspond to the discrimination scores for the two categories of "real video" and "fake video".

[0034] Subsequently, based on the correspondence between video segments and their respective videos, the segment-level discrimination scores are regrouped into video-level discrimination sets. For each video, multiple segment-level discrimination scores are aggregated at the video level using a multi-instance learning (MIL) mechanism to generate video-level discrimination representations.

[0035] The multi-instance aggregation mechanism is implemented using a Top-K aggregation method based on discrimination strength ranking. For multiple segment-level discrimination scores corresponding to the same video, they are sorted in descending order according to the probability of being fake, and the top K video segments are selected. Subsequently, the arithmetic mean of the selected K segment-level discrimination scores is calculated on their respective category output dimensions (corresponding to the two-dimensional discrimination scores of the real category and the fake category) to generate a video-level discrimination score representation.

[0036] After aggregation of multiple instances, the video-level discrimination score is represented as follows: , where N is the number of video sources in a batch.

[0037] Finally, the video-level discrimination score is input into the Softmax layer to obtain the corresponding class probability output, which represents the confidence level that the input video is a real video or a fake video. For the training set, the classification loss is calculated by combining the video-level discrimination score with the corresponding video-level ground truth label. The classification loss uses the cross-entropy loss function, which is defined as follows: in, This represents the true category label corresponding to the i-th video level.

[0038] For the validation and test sets, the model parameters trained on the training set are used to perform segment-level predictions on all video segments of the input video, and a video-level judgment result is generated through a multi-instance aggregation mechanism, thereby outputting the final video-level deep forgery detection result.

[0039] Compared with existing technologies, this invention introduces a feature fusion modulation mechanism in the feature fusion stage, enabling wavelet features to be injected into the backbone RGB features in a controlled residual manner, thereby improving the stability and accuracy of video-level judgment. This invention achieves excellent detection performance under various forgery types and compression conditions, and its model parameter count is smaller than many current mainstream methods, demonstrating significant technical advantages. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be introduced below.

[0041] Figure 1 The flowchart below shows a preferred embodiment of a video-level deep forgery detection method based on dual-path feature adaptive fusion. Detailed Implementation

[0042] The invention will now be described in detail with reference to preferred embodiments and accompanying drawings, so that those skilled in the art can better understand the invention.

[0043] like Figure 1 As shown in the figure, this embodiment discloses a video-level deep forgery detection method based on dual-path feature adaptive fusion, which includes four main steps: video input and segment generation, dual-path feature extraction, feature fusion modulation, and video-level judgment and output. The specific details of each step are as follows:

[0044] 1) Video Input and Segment Generation:

[0045] This step collects videos containing real faces and various forgery types from publicly available deepfake detection datasets. Regarding dataset partitioning, firstly, real videos and videos of various forgery types are obtained from publicly available deepfake detection datasets to construct a video-level dataset with complete videos as the basic sample unit. The video-level data is then randomly divided into training, validation, and test sets according to a preset ratio of 8:1:1, ensuring that segments of the same video do not cross sets.

[0046] The same segment generation process is performed on each video in the training, validation, and test sets to ensure consistent input format across different stages and avoid data leakage. Then, a multi-task convolutional neural network (MTCNN) or dlib tool is used for face detection and alignment, and the results are uniformly scaled to... The resolution of pixels. Next, the input video is fragmented along the temporal dimension. A sliding window is used to divide the video into multiple temporally continuous video clips, each consisting of a fixed number of consecutive frames. Adjacent clips overlap by a preset step size of L frames, where L is a fixed value of 6. This clip generation structure is used to construct the multi-segment single-video input format required for subsequent Multiple Instance Learning (MIL).

[0047] In a preferred embodiment, when the video duration is long and the number of clips is excessive, candidate clips can be filtered based on the degree of anomaly within each clip: several clips with a high degree of anomaly are retained first, while some normal clips are randomly retained, in order to maintain the diversity of video content while controlling the computational load. After the above processing, each video corresponds to multiple clips, providing an input basis for subsequent multi-path feature extraction and video-level judgment.

[0048] 2) Dual-path feature extraction: This includes RGB feature extraction and wavelet feature extraction sub-steps. Dual-path feature extraction is performed on each video segment generated from the training, validation, and test sets. The training set is used for parameter learning, while the validation and test sets are only used for forward computation for model selection and performance evaluation. Details are as follows:

[0049] 2.1 RGB Feature Extraction: The preprocessed face video clip is used as input, and its input tensor dimension is represented as... Where B represents the batch size, set to 4; T represents the number of frames contained in a sliding window, set to 12; 3 represents the three RGB color channels; and H and W represent the height and width of the input image, respectively, set to 112. First, the video clip is unfolded along the temporal dimension, and each frame is independently input into a parameter-shared convolutional neural network feature encoder (ResNet) to extract its spatial appearance features. The convolutional neural network feature encoder is constructed using a multi-layer convolutional structure to model the overall appearance structure, local texture distribution, and spatial semantic information of the face.

[0050] After processing by the feature encoder, each frame of the image is mapped to a high-dimensional feature representation, and its output feature dimension can be expressed as: Where C represents the number of feature channels output by the RGB spatial feature encoder (ResNet), specifically 512. This represents the spatial resolution of the feature map. All values ​​are 4. Subsequently, global pooling is performed on the spatial feature map of each frame to obtain frame-by-frame feature representations. Then, temporal average pooling is used to integrate the frame-by-frame features into a unified fragment-level feature representation. Through the above processing, the RGB feature extraction submodule maps the input video segment into a segment-level RGB feature representation for subsequent feature fusion and video-level determination.

[0051] 2.2 Wavelet Feature Extraction: The preprocessed face video segment is used as input, and its input tensor dimension is... First, a discrete wavelet transform is independently performed on each frame of the face image in the video clip at full spatial resolution. The discrete wavelet transform applies a wavelet transform to each of the three RGB color channels of the image. For any channel, the wavelet transform decomposes the image into four sub-bands by performing low-pass / high-pass filtering in the row and column directions: a low-frequency sub-band LL and three high-frequency sub-bands LH, HL, and HH, used to represent the overall structural information and local detail changes of the image, respectively. The eye and mouth regions are then cropped from the four sub-bands and pre-constructed with all-zero 24 channels, with a size of [missing information]. The feature tensor of the eye and mouth regions is interpolated and filled into the 24-channel feature tensor according to the region coordinates. After the above processing, each frame of the image is mapped to a 24-channel wavelet domain feature representation, and its output dimension can be expressed as... ,in, This represents the number of channels in the channel dimension of the wavelet features composed of 4 sub-band features after wavelet transform of the eye and mouth regions; therefore, the value is 24. This indicates the spatial resolution of the corresponding sub-band features, consistent with the original face image. Subsequently, the wavelet domain features are input into a feature encoding network to model the wavelet features and output the number of feature channels C. The wavelet domain feature encoding network is constructed using a ResNet18 convolutional neural network to learn the direct correlation between low-frequency structural information and high-frequency texture changes. After processing by the ResNet18 encoder, the corresponding wavelet domain feature vector is obtained through global average pooling. Subsequently, temporal average pooling is used to integrate the frame-by-frame wavelet domain features into a unified fragment-level feature representation. Furthermore, in obtaining the wavelet domain feature vector... Subsequently, the corresponding energy characteristics were calculated for the low-frequency subband of the wavelet. Specifically, the process is as follows: First, the low-frequency subbands corresponding to the three RGB color channels are averaged along the channel dimension to obtain the low-frequency wavelet representation of a single channel. Then, the absolute value of the difference between adjacent frames is calculated along the time dimension, and averaged over the spatial and temporal dimensions to obtain the energy features. The energy features are incorporated into the wavelet domain features as residuals, i.e., the wavelet domain is modulated by element-wise addition to obtain wavelet features containing energy information. The energy characteristics mentioned above Broadcasting is performed along the channel dimension, thereby incorporating wavelet features. Addition. The energy modulation process does not introduce new feature channels, and the dimension of the wavelet domain feature vector remains consistent before and after energy injection.

[0052] 3) Feature Fusion Modulation: Receives fragment-level feature representations from the dual-path feature extraction unit as input. Let the fragment-level feature representation output by the RGB feature extraction unit be... The fragment-level features output by the wavelet feature extraction unit are represented as follows: .

[0053] Firstly, because All are represented by fragment-level channel vectors, and the global statistical mapping uses the identity mapping, i.e. The global average pooling operation is used to smooth local noise and occasional abnormal responses, so that the resulting global statistical vector can characterize the overall activation intensity and distribution of the corresponding features in the current video segment, providing a stable statistical basis for the generation of subsequent fusion weights.

[0054] Subsequently, the global statistical vector corresponding to the RGB features is... Global statistical vector corresponding to wavelet features The data is concatenated at the channel dimension to form a joint context representation. The joint context representation is input into the fusion weight generation network to generate fusion weight parameters that control the strength of feature fusion. The fusion weight generation network employs a multilayer perceptron (MLP), and its network components include a first fully connected layer, a ReLU nonlinear activation function, a second fully connected layer, and a sigmoid activation function. The first fully connected layer is used to transform the input feature dimension from... Mapped to a preset hidden dimension The nonlinear activation function is used to introduce nonlinear expressive power; the second fully connected layer is used to output the fusion weight parameters; the sigmoid activation function is used to constrain the fusion weights to... Within the interval. The fusion weight parameters are obtained through the fusion weight generation network: The fusion weights are used to characterize the contribution of wavelet features to RGB features in the feature fusion process under the current input conditions. During the training phase, to further suppress the interference of wavelet features on RGB feature learning before the model's discriminative ability is fully established, this embodiment introduces a fusion strength constraint mechanism after the fusion weights are generated. Specifically, based on the forgery type information used in the training settings, a preset fusion strength constraint factor s is applied to the fusion weights α to obtain the modulated fusion weights. s is a globally learnable constraint factor. Specifically, the globally learnable constraint factor s is set in the feature fusion module, and it is determined by the trainable parameters. Obtained through Sigmoid mapping, making .

[0055] The validation and test sets no longer update the constraint factors and network parameters. Instead, they use the fusion weight parameters learned from the training set and the globally learnable factors for forward fusion to ensure the consistency and reproducibility of the evaluation.

[0056] After obtaining the modulation fusion weights, the wavelet features are weighted and injected into the RGB features in the form of residuals to form a fusion feature representation. This represents element-wise multiplication. By employing a residual fusion structure, RGB features maintain their dominant role in the fused features, while controlled wavelet feature information is introduced, thereby improving model stability in the early stages of training.

[0057] 4) Video-level decision-making and output: The video-level decision-making process includes three stages: segment-level forward prediction, multi-instance aggregation mechanism, and video-level output and loss calculation. Details are as follows:

[0058] After the aforementioned feature fusion modulation steps, the resulting fused feature representation is denoted as: First, the features of multiple segments corresponding to each video in the same batch are fused together. The video segments are grouped according to their video indexes to form a set of video-level segment features, which is then input into a segment-level prediction network for forward prediction. The segment-level prediction network preferably employs a multilayer perceptron structure, consisting of two fully connected layers, used to independently discriminate and model the fused features of each video segment. The first fully connected layer is used to integrate the segment-level fused features. Mapped to the intermediate discriminative feature space, its output dimension is The second fully connected layer maps intermediate features to the category space, and its output dimension is... These correspond to the unnormalized discrimination scores for the real video and fake video categories, respectively. The segment-level prediction network shares parameters across all video segments and independently performs forward prediction on the fusion features of each video segment in the current batch. The discrimination scores corresponding to each segment are stacked in segment order to obtain the segment-level discrimination score matrix, the output of which is expressed as: The two-dimensional outputs correspond to the discrimination scores for the two categories of "real video" and "fake video".

[0059] Subsequently, based on the correspondence between video segments and their respective videos, the segment-level discrimination scores are regrouped into video-level discrimination sets. For each video, multiple segment-level discrimination scores are aggregated at the video level using a multi-instance learning (MIL) mechanism to generate video-level discrimination representations.

[0060] The multi-instance aggregation mechanism employs a Top-K aggregation method based on discrimination strength ranking. For multiple segment-level discrimination scores corresponding to the same video, they are sorted in descending order of forgery class probability, and the top K video segments are selected. Subsequently, the arithmetic mean of the selected K segment-level discrimination scores is calculated on their respective category output dimensions (corresponding to the two-dimensional discrimination scores of the real and forgery categories) to generate a video-level discrimination score representation. The segment-level score uses the forgery class probability after binary classification Softmax. Indicate, and according to Sort the data from largest to smallest, and take the top K segments for aggregation. r is the Top-K scaling parameter (default 0.2). This is the number of segments in the video (the default number of segments is 8). This indicates rounding up. The average of the logits for the selected segment is used as the video-level logits.

[0061] After aggregation of multiple instances, the video-level discrimination score is represented as follows: , where N is the number of video sources in a batch.

[0062] Finally, the discrimination score is input into the Softmax layer to obtain the corresponding class probability output, which represents the confidence level that the input video is a real video or a fake video. For the training set, the classification loss is calculated by combining the video-level discrimination score with the corresponding video-level ground truth label. The classification loss preferably uses the cross-entropy loss function, which is defined as follows:

[0063]

[0064] in, This represents the actual category label corresponding to the i-th video.

[0065] For the validation and test sets, the model parameters trained on the training set are used to perform segment-level predictions on all video segments of the input video, and a video-level judgment result is generated through a multi-instance aggregation mechanism, thereby outputting the final video-level deep forgery detection result.

[0066] To verify the effectiveness of the video deepfake detection method proposed in this invention, this embodiment is based on a publicly available face forgery dataset. The method was experimentally verified.

[0067] The dataset includes various mainstream video forgery methods, comprehensively reflecting the distribution of abnormal video features under different forgery mechanisms. This embodiment selects... The dataset was used to validate four typical forgery types, including Deepfakes, Face2Face, FaceSwap, and NeuralTextures. Experiments were conducted under two different compression quality conditions (c23 and c40) to evaluate the discrimination performance and stability of the proposed method under different video quality conditions.

[0068] In the experiment, the model was first trained on the training set according to the specific implementation method described above. Then, the trained model parameters were used on the validation set for inference and performance metrics were calculated for parameter tuning and optimal model selection. Finally, inference was performed on the test set and performance metrics were calculated as the final evaluation result. The evaluation uses video-level decision results as the final output, and classification performance metrics are calculated based on video-level ground truth labels. To comprehensively reflect the model's video-level detection performance, this embodiment uses video-level classification accuracy (ACC) and parameter count as performance metrics.

[0069] Under high video quality conditions (c23), the method of the present invention achieved high classification accuracy for all four types of forgery, and can effectively distinguish between real videos and forged videos.

[0070] Under more challenging high compression conditions (c40), despite a significant decrease in video quality, the method of the present invention can still maintain relatively stable detection performance. The classification accuracy, F1 score and recall rate do not show significant degradation, indicating that the method has good robustness to noise interference caused by video compression.

[0071] The method of this invention can effectively suppress the misidentification of real videos as fake videos under various forgery types, while maintaining a high recognition rate for fake videos, verifying the effectiveness of the multi-path feature fusion and stable modulation mechanism in video-level deep forgery detection tasks.

[0072] Table 1 lists the methods of the present invention in... The dataset includes four types of forgery and video-level classification performance metrics under two compression conditions.

[0073] Table 1 Classification accuracy of datasets under different forgery types and compression conditions

[0074]

[0075] Table 2 shows the video-level classification performance metrics of the method of the present invention on the CelebDF dataset.

[0076] Table 2 Classification accuracy on the CelebDF dataset

[0077]

[0078] Table 3 presents the video-level classification performance metrics of the method of this invention on the UADFV dataset.

[0079] Table 3 Classification accuracy on the UADFV dataset

[0080]

[0081] Table 4 provides a comparison of the parameter quantities of the present invention with those of other methods.

[0082] Table 4 Comparison of Parameter Quantities

[0083]

[0084] The experimental results of the above embodiments demonstrate that the video-level deepfake detection method proposed in this invention can achieve stable and reliable detection performance under various forgery types and different video quality conditions. These results verify the effectiveness of the dual-path feature extraction and stable feature fusion modulation mechanism of this invention in practical application scenarios, providing a feasible and robust technical solution for video-level deepfake detection.

[0085] In summary, this invention discloses a video-level deep forgery detection method based on dual-path feature adaptive fusion. The method performs face detection alignment and sliding window fragmentation on the input video, extracting fragment-level RGB features and fragment-level wavelet features from each fragment. RGB features represent global texture features, while wavelet features express local energy changes in the eyes and mouth. Low-frequency subband energy is used to further enhance the feature representation capability of the wavelet features. Furthermore, fusion parameters are generated using RGB and wavelet features to modulate the feature fusion process, adjusting the fusion ratio of the dual-path features. Finally, Top-K multi-instance aggregation is used from multiple fragment-level features generated from a video to achieve video-level judgment output. This invention achieves excellent detection performance under various forgery types and compression conditions, with a low model parameter count.

[0086] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A video-level deepfake detection method based on dual-path feature adaptive fusion, characterized by: The specific steps are as follows: (1) Video input and segment generation: The input video is fragmented in the time dimension, and the original video is divided into multiple video segments with fixed time lengths; (2) Dual-path feature extraction: Extract appearance information based on RGB features for each video segment, and extract local energy changes based on wavelet features at the same time; (3) Feature fusion modulation: Based on the global statistical information of the RGB features and wavelet features in step (2), generate fusion weights for the modulation feature fusion process; Furthermore, a globally learnable constraint factor is set during the feature fusion modulation process to adaptively learn the injection strength of the fusion weights under different forgery types. (4) Video-level judgment and output: The judgment results of each video segment are aggregated at the video level, and the final detection result of the corresponding input video is output.

2. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 1, characterized in that, In step (1), the video clip generation steps are as follows: First, collect real and fake videos from public datasets to build a dataset. Divide the dataset into training, validation and test sets. Then perform face detection and alignment, and uniformly scale it to a set pixel resolution. Finally, the input video is divided into multiple time-continuous video segments by sliding window according to the preset time window length M; each video segment contains a fixed number of consecutive frames to cover the changes in facial movement within a local time range; adjacent video segments are sampled by L-frame overlap.

3. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 1 or 2, characterized in that, In step (2), the dual-path feature extraction step is divided into an RGB feature extraction sub-step and a wavelet feature extraction sub-step. Dual-path feature extraction is performed on each video segment of the training set, validation set and test set.

4. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 3, characterized in that, The specific sub-steps for RGB feature extraction are as follows: Taking the face video clip from step (1) as input, its input tensor dimension is represented as follows: Where B represents the batch size, T is the number of frames contained in a sliding window, 3 represents the three color channels RGB, and H and W represent the height and width of the input image, respectively. First, the video clip is expanded in the time dimension, and each frame image is independently input into the parameter-shared convolutional neural network feature encoder ResNet18 to extract spatial appearance features. After processing by the ResNet18 convolutional neural network feature encoder, each frame of the image is mapped to a high-dimensional feature representation, and the output feature dimension is represented as follows: Where C represents the number of feature channels output by the convolutional neural network feature encoder. The spatial resolution of the feature map is represented; then global pooling is performed on the spatial feature map of each frame to obtain the frame-by-frame feature representation. Then, temporal average pooling is used to integrate the frame-by-frame features into a unified segment-level feature representation. .

5. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 4, characterized in that, The specific steps of wavelet feature extraction are as follows: Taking the face video clip from step (1) as input, its input tensor dimension is... First, a discrete wavelet transform is independently performed on each frame of the face image in the video clip at full spatial resolution. The discrete wavelet transform is then applied to the three RGB color channels of each frame. For any given channel, the discrete wavelet transform decomposes the image into four sub-bands by performing low-pass / high-pass filtering in the row and column directions: a low-frequency sub-band LL and three high-frequency sub-bands LH, HL, and HH, used to represent the overall structural information and local detail changes of the image, respectively. The eye and mouth regions are then cropped from these four sub-bands and pre-constructed with all-zero 24 channels of a size of [size missing]. The feature tensor of the eye and mouth regions is interpolated and filled into the 24-channel feature tensor according to the region coordinates. This results in a 24-channel wavelet domain feature representation for each frame of the image, with the output dimension represented as... ,in, This represents the number of channels in the channel dimension of the wavelet features composed of four sub-band features after wavelet transform of the eye and mouth regions. This represents the spatial resolution of the corresponding sub-band features; then, the wavelet domain features are input into the feature coding network to model the wavelet features and output the number of feature channels C.

6. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 5, characterized in that the wavelet domain feature encoding network is constructed using a convolutional neural network ResNet18, and after processing by ResNet18, the corresponding wavelet domain feature vector is obtained through global average pooling. Subsequently, time-average pooling was used to integrate the frame-by-frame wavelet domain features into a unified segment-level feature representation. .

7. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 6, characterized in that, Obtain the wavelet domain feature vector Subsequently, the corresponding energy characteristics were calculated for the low-frequency subband of the wavelet. Specifically, the process is as follows: First, the low-frequency subbands corresponding to the three RGB color channels are averaged along the channel dimension to obtain the low-frequency wavelet representation of a single channel; then, the absolute value of the difference between adjacent frames is calculated along the time dimension, and averaged along the spatial and temporal dimensions to obtain the energy features. The energy features are incorporated into the wavelet domain features as residuals to obtain wavelet features containing energy information. ,in, Energy characteristics Broadcasting is performed along the channel dimension, thereby incorporating wavelet features. Add them together.

8. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 7, characterized in that, Step (3), the feature fusion modulation step, is as follows: Firstly, because All representations are segment-level channel vectors, and the global statistical mapping uses the identity mapping: ; Subsequently, the RGB global statistical vector and wavelet global statistical vector Concatenation is performed at the channel dimension to obtain the joint context representation. The input is then fed into the fusion weight generation network. The fusion weight generation network uses a multilayer perceptron (MLP), which includes a first fully connected layer, a non-linear activation function, a second fully connected layer, and a sigmoid function. The first fully connected layer is used to map the input dimension from 2C to the hidden dimension D. h The ReLU activation function is used to introduce non-linear expressiveness; a second fully connected layer is used to output fused weight parameters; and the Sigmoid function is used to constrain the output weights to... Interval; obtain the fusion weight parameters: . After the fusion weights are generated, a fusion strength constraint mechanism is introduced, applying a preset fusion strength constraint factor s to the fusion weights α, thus obtaining the modulated fusion weights. Specifically, a globally learnable constraint factor s is set during the feature fusion process, which is determined by the trainable parameters. Obtained through Sigmoid mapping, making This represents the Sigmoid function; After obtaining the modulation fusion weights, the wavelet features are weighted and injected into the RGB features in the form of residuals to form a fusion feature representation. This indicates element-wise multiplication.

9. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 8, characterized in that, Step (4) The video-level determination step includes three sub-steps: segment-level forward prediction, multi-instance aggregation mechanism, and video-level output and loss calculation.

10. The video-level deepfake detection method based on dual-path feature adaptive fusion as described in claim 9, characterized in that, Step (4), the video-level determination steps are as follows: The fusion feature representation obtained after step (3) is denoted as ; First, fuse the features at the segment level for each video in the same batch. The video segments are grouped according to their video indices to form a set of video-level segment features, which are then input into a segment-level prediction network for forward prediction. The segment-level prediction network adopts a multilayer perceptron structure, consisting of two fully connected layers, used to independently discriminate and model the fused features of each video segment. The first fully connected layer is used to integrate the segment-level fused features. Mapped to the intermediate discriminative feature space, the output dimension is The second fully connected layer maps intermediate features to the category space, with an output dimension of [missing value]. These correspond to the unnormalized discrimination scores for real videos and fake videos, respectively. The segment-level prediction network shares parameters across all video segments and independently performs forward prediction on the fusion features of each video segment in the current batch. The discrimination scores corresponding to each segment are stacked in segment order to obtain the segment-level discrimination score matrix, and the output form is as follows: The two-dimensional outputs correspond to the discrimination scores for the two categories of "real video" and "fake video" respectively; Then, based on the correspondence between video segments and their respective videos, the segment-level discrimination scores are regrouped into video-level discrimination sets; for the multiple segment-level discrimination scores corresponding to each video, a multi-instance learning mechanism (MIL) is introduced to perform video-level aggregation to generate video-level discrimination representations; After aggregation of multiple instances, the video-level discrimination score is represented as follows: Where N is the number of video sources in a batch; Finally, Softmax normalization is applied to the video-level discrimination scores to obtain the class probabilities, resulting in the corresponding class probability output, which represents the confidence level that the input video is a real video or a fake video. For the training set, the classification loss is calculated by combining the video-level discrimination scores with the corresponding video-level ground truth labels. The classification loss uses the cross-entropy loss function, defined as follows: in, This represents the real category label corresponding to the i-th video level; For the validation and test sets, the model parameters trained on the training set are used to perform segment-level predictions on all video segments of the input video, and a video-level judgment result is generated through a multi-instance aggregation mechanism, outputting the final video-level deep forgery detection result.

Images