Video time sequence forgery positioning method based on cross-modal spatial-temporal feature decoupling

Abstract

The invention provides a video time sequence forgery positioning method based on cross-modal spatial-temporal feature decoupling. According to the method, through multi-modal information fusion and forgery feature decoupling, the bottleneck that in the prior art, the time sequence forgery positioning precision is insufficient, and the robustness of a model to unknown forgery is limited is broken through, and the method mainly comprises the following steps that (1) a ViT and ResNet-based cross-modal space-time unified encoder is provided, global semantic and local texture features are fused, and the positioning precision of the time sequence forgery is improved; meanwhile, through combination of Transform and dynamic graph convolution modeling complex space-time dependence, the capability of capturing fine forgery traces in the video is enhanced; (2) a multi-scale counterfeiting-style feature decoupling module is constructed, counterfeiting features and style features are effectively separated through multi-scale convolution and orthogonal constraint, the situation that the model depends on style fingerprints of a counterfeiting method is avoided, and the generalization performance is remarkably improved; and (3) designing a domain adaptive multi-task learner, introducing a dynamic adversarial training and joint loss optimization strategy, and further enhancing the robustness of the model in an unknown counterfeit scene. Compared with an existing method, the method has the advantages that higher-precision forgery positioning is achieved, generalization ability and robustness of cross-forgery types are remarkably improved, and the method is suitable for complex and changeable real application scenes.

Description

Technical Field

[0001] This invention belongs to the technical field of multimedia digital forensics, specifically relating to a video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling. Background Technology

[0002] With the rapid development of deep learning and AI generation technologies, generating realistic forged images and videos has become increasingly easy, significantly increasing the difficulty of distinguishing tampered media content from genuine content. This phenomenon poses a significant threat to media security and public opinion security. Therefore, detection technologies for video tampering, especially deepfake video detection technologies, have gradually become an important research area. Currently, deepfake detection technologies for videos have made great progress, especially binary classification models, which can determine whether a video has been tampered with at a coarse-grained level, achieving high levels of accuracy and robustness. However, these methods still have certain limitations. In practical applications, only a few frames in a video may be tampered with. These minor tamperings can lead to significant reversals in key semantics or emotions within the video, misleading viewers. Traditional coarse-grained tampering detection methods often fail to accurately identify these localized tamperings, easily resulting in false positives and false negatives. Therefore, developing a fine-grained temporal tampering localization technology capable of accurately locating tampered frames is particularly important, as it is crucial for more effectively protecting multimedia content security.

[0003] In the field of temporal forgery localization in videos, high-quality benchmark datasets are crucial for driving algorithmic innovation. Among them, the paper "He,Y.,Gan,B.,Chen,S.,Zhou,Y.,Yin,G.,Song,L.,Sheng,L.,Shao,J.,&Liu,Z.(2021).ForgeryNet: Aversatile benchmark for comprehensive forgery analysis.2021 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)." first provided researchers with a comprehensive and large-scale benchmark dataset, ForgeryNet. Its massive scale, diverse forgery methods, and detailed annotations provide an important foundation for research in video temporal forgery localization and even deepfake detection.

[0004] Subsequently, the paper "Cai Z, Ghosh S, Dhall A, et al. Glitchinthematrix: A large-scale benchmark for content-driven audio–visual forgery detection and localization[J]. Computer Vision and Image Understanding, 2023, 236:103818." proposed the landmark LAV-DF dataset, which for the first time focused research on the more challenging identification of content-driven audio-visual semantic inconsistencies, where tampering traces are more subtle and detection is more difficult. Meanwhile, the authors proposed the boundary-aware temporal forgery detection model BA-TFD and its improved version BA-TFD+ as benchmark methods. Based on 3D-CNN and a multi-scale visual converter respectively, and combining boundary awareness and contrastive learning loss functions, they can simultaneously perform forgery detection and tampering boundary localization. However, this method still mainly relies on the local modeling capabilities of convolution operators and visual converters in feature representation, making it difficult to fully capture high-level semantic features; at the same time, it also has shortcomings in characterizing long-range dependencies between video frames, thus limiting the effective modeling of subtle tampering traces.

[0005] To overcome the limitations of long-range modeling, the paper "Zhang R, Wang H, Du M, et al. Ummaformer: Universal multimodal-adaptive transformer framework for temporal forgery localization[C] / / Proceedings of the 31st ACM International Conference on Multimedia. 2023:8749-8759." innovatively introduces...

[0006] The Transformer architecture is designed as a general multimodal adaptive framework. It proposes a temporal feature anomaly attention module and a parallel cross-attention feature pyramid network, effectively capturing temporal anomalies by modeling feature distribution biases, significantly improving detection capabilities for complex scenarios such as short-term tampering and video inpainting. However, because the Transformer directly models global relationships, it is prone to introducing redundancy and noise, resulting in room for improvement in the accuracy and stability of its temporal modeling in complex dynamic scenes.

[0007] Furthermore, these advanced methods share a common challenge: limited generalization ability. Most models are highly dependent on the distribution of training data, tending to learn known traces left by specific forgery techniques rather than general physical or semantic flaws underlying the tampering. This learning strategy results in models that, while performing well on specific test sets, show significant declines in performance and robustness when facing unknown attack types or varied real-world scenarios, making them ill-suited for forgery localization tasks in complex environments.

[0008] Despite the significant progress made by the aforementioned methods, shortcomings remain in the sufficiency of feature representation, the flexibility of temporal relationship modeling, and robustness in complex environments. Therefore, this paper proposes a video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling. Summary of the Invention

[0009] This invention proposes a video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling, aiming to address the increasingly complex deepfake techniques, improve the robustness and stability of the detection model in real-world scenarios, and solve the problems of insufficient feature representation and model susceptibility to forgery styles. The main contents include the following:

[0010] (1) A cross-modal spatiotemporal unified encoder based on ViT and ResNet is proposed.

[0011] (2) A multi-scale forgery-style feature decoupling module is proposed.

[0012] (3) A domain-adaptive multi-task learner was designed.

[0013] The details are as follows:

[0014] (1) A cross-modal spatiotemporal unified encoder based on ViT and ResNet is proposed: First, the global semantic features extracted by the Vision Transformer (ViT) are fused with the local texture features extracted by ResNet to obtain a more comprehensive and complementary video frame representation. On this basis, a spatiotemporal unified coding module is introduced, which replaces the traditional static graph convolution with dynamic graph convolution and combines it with the temporal Transformer architecture to more effectively capture the complex spatiotemporal dependencies in video sequences.

[0015] In the cross-modal feature fusion module, the input ViT frame features are first... (B is the batch size, T is the sequence length, D) ViT (for feature dimensions) and ResNet frame features (D ResNet Projecting the feature dimension onto a uniform hidden dimension D hiddenProjected features (F') ViT and F' ResNet Iterative interaction and fusion are achieved through stacked cross-modal attention blocks. Each attention block allows features from one modality to query features from another modality, thereby achieving mutual enhancement and synergy of information. Finally, the fused features are concatenated and projected again to obtain video frame features that combine global semantics and local details.

[0016] The spatiotemporal unified coding module first processes the fused multimodal features F fused Add learned positional encoding Get F pos =F fused +P is injected to inject temporal information. These features are then fed into a Transformer network containing multiple layers of residual attention blocks for processing, utilizing its self-attention mechanism to model long-range inter-frame dependencies. To further explore more complex inter-frame connection patterns, this invention innovatively introduces dynamic graph convolution instead of static graph convolution. Subsequently, a dynamic adjacency matrix (content graph) is constructed based on content similarity, and an adjacency matrix (structure graph) is established based on inter-frame distance to effectively capture local temporal dependencies. Then, the input features are subjected to dynamic graph convolution operations on these two graph structures respectively. The specific process can be represented as follows:

[0017] For the dynamic graph convolution output F of the content graph gc_sim :

[0018]

[0019] For the dynamic graph convolution output F of the structure graph gc_dist :

[0020]

[0021] Among them, GC i (i = 1, 2, 3, 4) all represent dynamic graph convolutional blocks, which can adaptively adjust the aggregation weights according to the input features and the adjacency matrix; For content images, For structural diagrams; F pos The features, after being infused with temporal information, are used as the input to this module; GELU is the activation function. The aforementioned dynamic graph convolutional blocks are applied to two different graph structures to fuse multi-source topological information and enhance feature representation. Finally, the outputs F of the two graph convolutions are... gc_sim and F gc_dist By concatenating the data and passing it through a linear layer, a feature representation containing rich spatiotemporal information is generated. It significantly enhances the model's ability to capture subtle forgery traces in videos.

[0022] (2) A multi-scale forgery-style feature decoupling module is proposed: Existing deep forgery detection methods often face the problem that the model easily learns the "style" artifacts specific to the forgery method, rather than the real forgery traces, resulting in poor generalization ability. This invention introduces a multi-scale feature decoupling module to solve this problem.

[0023] This module receives data from F. final Video global features obtained by temporal average aggregation By using convolution kernels with kernel sizes of 1, 3, and 8 in D hidden Multi-scale 1D convolution operations are performed along the dimension to capture features of different granularities. These multi-scale features are concatenated and passed through two independent linear projection layers W. f and W s Mapped to fake features respectively and style characteristics To ensure that these two feature spaces are independent, an orthogonal regularization loss L is introduced. orth It encourages F f and F s They are orthogonal in the vector space, meaning their inner product is close to zero. This decoupling allows subsequent classifiers to focus more on intrinsic traces related to forgery, thereby improving the model's robustness to different forgery styles and unknown forgeries.

[0024] (3) A domain-adaptive multi-task learner was designed to further improve the model's generalization performance in diverse deepfake scenarios. The core of this module lies in dynamic adversarial training, which uses a dynamic adversarial classifier as a discriminator to force the feature encoder to generate fake features that are difficult for the discriminator to distinguish. This adversarial training makes the fake representations learned by the model more general and can effectively resist attacks from unknown fake methods.

[0025] Meanwhile, this learner employs a multi-task joint loss optimization strategy, combining multiple loss functions to guide the model's learning. The total loss function L... total It not only includes frame-level loss L for precise frame-level spoofing localization frame Video-level loss L for determining video-level authenticity video It also includes the orthogonal constraint loss L for feature decoupling. orth and adversarial loss L for domain adaptation adv_cls .

[0026] L total =L frame +λ video .L video +λ adv .L adv_cls +λ orth .L orth(5)

[0027] Where, λ video , λ adv and λ orth The weights are assigned to the video-level loss, adversarial loss, and orthogonal constraint loss, respectively. This joint optimization ensures that the model achieves a balance among multiple objectives, resulting in excellent accuracy, robustness, and interpretability.

[0028] The dynamic adversarial head, serving as the discriminator in adversarial training, is centered around a gradient inversion layer. It receives forged features from the feature decoupling module. As input, it attempts to predict the "domain" (i.e., forgery type or source) to which these features belong. During forward propagation, the gradient inversion layer simply passes F... f However, during backpropagation, it multiplies the gradient by a negative value α. By dynamically generating domain weights and weightedly combining the outputs of multiple base classifiers, this module can flexibly adapt to different domains. The model contains N domains (Number of forgery methods) basic linear classifiers, each classifier C i Output a domain prediction score. (Adversarial classification score) This is a weighted sum of the outputs of these base classifiers. During training, it is calculated using the adversarial classification score L. adv The resulting adversarial loss L adv_cls This will prompt the feature encoder to generate fake features that are difficult for the discriminator to distinguish, thereby making the fake representation learned by the model more general and effectively resisting attacks from unknown fake methods.

[0029] Compared with other methods, the present invention has the following significant advantages:

[0030] (1) Superior generalization ability and robustness: Through a multi-pronged strategy, the detection capability against unknown or novel deepfake techniques is significantly improved. The cross-modal feature fusion module effectively integrates the global semantic features of ViT and the local texture features of ResNet, providing a more comprehensive view of video content. At the same time, the multi-scale feature decoupling module can decouple video features into forgery-related features F. f Style-related features F s And supplemented by orthogonal constraint loss L orth This ensures that the model learns more fundamental forgery traces that are independent of video style. Furthermore, the dynamic adversarial head, combined with the gradient inversion layer, prompts the feature encoder to generate domain-indistinguishable forgery features, freeing the model from relying on the "fingerprint" of specific forgery methods and thus maintaining high detection performance even when facing unknown forgery techniques.

[0031] (2) More refined spatiotemporal analysis capabilities: This method not only determines the authenticity of videos but also scores them at the frame level, thanks to its in-depth modeling of spatiotemporal relationships. The Transformer in the model can effectively capture long-distance temporal dependencies between video frames, thus providing richer temporal context information. Furthermore, the introduction of dynamic graph convolution enables the model to dynamically construct and utilize complex relationships between frames, more accurately capturing local inconsistencies and nonlinear spatiotemporal anomalies that may be caused by forgery, achieving more fine-grained and accurate forgery analysis.

[0032] (3) Clearer Distinction Between Forgery and Style: Addressing the issue of existing methods easily confusing forgery traces with video style, this method provides a clear solution. The multi-scale forgery-style feature decoupling module is the core of this advantage. By using forgery features F... f and style characteristics F s With clear separation, the model can focus on key information related to the forgery. This means that regardless of the video's scene, lighting conditions, or content style, the model can more purely identify traces introduced by deepfake operations, greatly improving the accuracy and interpretability of detection and avoiding misjudgments of videos that "look unnatural."

[0033] (4) More concise and efficient structure: Compared to some methods that require complex post-processing or boundary prediction modules, this method is more intuitive and efficient. This method is based directly on frame-level prediction, meaning the model output is no longer a general video-level binary classification result, but rather the forgery probability for each frame of the video. This design eliminates the complex boundary prediction and boundary loss that may occur in traditional methods, making the overall structure more intuitive and easier to understand. At the same time, this frame-level prediction naturally brings more fine-grained forgery localization capabilities, enabling researchers and users to more accurately understand which specific time periods or frames in the video have been tampered with, thus providing valuable information for subsequent analysis and source tracing. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the timing forgery positioning framework of the present invention;

[0035] Figure 2 This is a schematic diagram of the multimodal feature fusion module of the present invention;

[0036] Figure 3 This is a schematic diagram of the spatiotemporal dynamic encoder structure of the present invention;

[0037] Figure 4 This is a schematic diagram of the multi-scale feature decoupling module structure of the present invention. Detailed Implementation

[0038] This invention is a video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling. The following uses the ForgeryNet large-scale dataset as an example to illustrate specific embodiments of the invention, aiming to provide further explanation of this application. Those skilled in the art should understand that these embodiments are merely for explaining the technical principles of the invention and are not intended to limit the scope of application of the invention.

[0039] This embodiment can be implemented according to the following steps. In this example, the Python programming language is used as an example to build the model on the PyTorch deep learning framework. The specific steps are as follows:

[0040] Step 1: Dataset Preparation. We selected the ForgeryNet dataset as our dataset. In terms of data scale, ForgeryNet fake videos include 15 deepfake face operations, incorporate over 36 perturbations to simulate real-world challenges, and contain rich data annotations for researchers to explore. We used 59,994 videos as the training set, 14,019 videos as the validation set, and 14,019 videos as the test set. Furthermore, the original annotation files needed to be converted to a more user-friendly JSON file format for easier subsequent operations.

[0041] Step 2: Feature Extraction. First, extract ViT and ResNet features frame-by-frame for each video in the dataset to obtain the ViT feature F. ViT (num_frames, 1, 1024) and ResNet features F ResNet (num_frames, 1, 512). Because the number of frames (numframes) varies for each video, to facilitate subsequent processing, the numframes dimension needs to be padded to a fixed value. Considering that the videos in the dataset are relatively short, we choose to padded all features to 512 and use a mask to store the actual effective frame count for easier subsequent loss calculation. During dataset loading, frame-level labels, video-level labels, frame masks, and forgery method IDs are provided simultaneously.

[0042] Step 3: Constructing the Model Framework. According to the video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling described in this invention, the model is built on the PyTorch deep learning framework and achieves accurate localization and recognition of video forgeries through multimodal input and multi-task learning. It mainly includes the following core modules: a cross-modal spatiotemporal unified encoder, a multi-scale forgery-style feature decoupling module, a dynamic adversarial classifier, a frame-level forgery classifier, and a video-level forgery classifier. The output includes four items: frame-level forgery score `frame_logits`, used for frame-level forgery localization; video-level forgery score `video_logits`, used for overall video authenticity judgment; adversarial classification score `adv_logits`, used for adversarial training of forgery method recognition; and orthogonal constraint loss term `orth_loss`: used for regularization of feature decoupling.

[0043] Step 4: Model Training. Training is performed on the model framework established in Step 2, employing a multi-task joint optimization strategy, and stabilizing the training process through multi-stage learning rate adjustment and gradient clipping. Parameter settings: The video-level loss weight λ in formula (5) video Adversarial loss weight λ adv and orthogonal loss weights λ orth The values ​​were set to 0.3, 0.005, and 1 respectively. Optimizer: The AdamW optimizer was used with an initial learning rate of 2 × 10⁻⁶. -5 The weight decay coefficient is 0.01. For the first 3000 iterations of the linear warm-up phase, a linear learning rate warm-up strategy is used, with the learning rate increasing linearly from near 0 to 2 × 10⁻⁶. -5 Cosine annealing phase: After warm-up, the learning rate follows the cosine annealing learning rate scheduler. The length of the cosine annealing cycle, T_max, is set to 20 cycles (corresponding to the total number of training steps), and the lower bound of the learning rate, eta_min, is set to 1×10. -6 This allows for a smooth decay of the learning rate. The batch size is 64 (B as mentioned above), and the training lasts for 15 epochs. It's worth noting that to prevent gradient explosion during training, the gradients of the model parameters are clipped, with max_norm set to 1.0.

[0044] Step 5: Frame-by-frame score to temporal segment conversion. The model's forward propagation outputs the fake scores for each frame of the video (values ​​ranging from negative infinity to positive infinity). To convert these into structured fake segment proposals, the following post-processing procedure is performed:

[0045] First, probabilistic mapping and binary determination are performed: the original scores are mapped to forgery probabilities between 0 and 1 using an activation function. Then, a preset threshold of 0.5 is used for binary determination, marking frames with probabilities higher than this threshold as "forgery frames." Next, connected component analysis is performed: consecutive forgery frames in the binary sequence are labeled as regions, identifying all consecutive forgery intervals. Finally, time-series proposals are generated: for each consecutive forgery segment, its start and end frame positions are recorded, and the timestamps are converted to start and end timestamps in seconds based on the video frame rate (fps). The maximum probability of all frames within the segment is used as its confidence level. The final output is a list of forgery segment proposals in the form of [confidence level, start time, end time].

[0046] Step Six: Time Positioning Performance Evaluation.

[0047] After generating forged fragment proposals, the average precision (AP) is used as a quantitative metric for model performance. This evaluation method borrows the concept of Intersection over Union (IoU) from the image domain and extends it to the temporal dimension, called Temporal Intersection over Union (tIoU). tIoU is calculated as the ratio of the overlap between the predicted and actual forged periods to their combined duration. This metric measures the temporal alignment between the predicted fragments and the ground truth annotations; a higher tIoU value indicates a high degree of consistency between the predicted results and the ground truth.

[0048] For each video, all fake proposals sorted by confidence are matched against real fake intervals. If a proposal's tIoU with a real interval exceeds a preset IoU threshold, and that real interval has not yet been matched by a proposal with higher confidence, then the proposal is considered a true positive.

[0049] Based on the number of true positives and false positives, we can calculate the precision and recall at different confidence levels, and then plot the precision-recall (PR) curve. The area under the PR curve is the AP value of a single video; the higher the AP, the better the model performs in identifying fake segments, in terms of both recall and precision.

[0050] To provide a more comprehensive view of performance, this method calculates the AP values ​​at multiple IoU thresholds (0.55, 0.65, 0.75, 0.85, 0.95) and averages them to obtain mAP (mean Average Precision). mAP is a commonly used comprehensive evaluation metric in the field of video event detection, which can comprehensively reflect the average performance of the model under different matching strictness.

[0051] In summary, to address the increasingly serious challenge of deepfake videos, this invention proposes a video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling. This method demonstrates excellent accuracy and robustness when facing various complex and varied deepfake types, meeting the requirements for high-precision temporal localization of forged content in real-world complex scenarios. Furthermore, by introducing forgery-style feature decoupling and dynamic adversarial training, the model can focus on the inherent traces of the forgery itself, rather than being confused by the surface style of the video, resulting in more accurate detection results with good interpretability.

[0052] Those skilled in the art will understand that the scope of protection of this invention is not limited to the specific embodiments described. Without departing from the principles of this invention, those skilled in the art can make equivalent modifications or substitutions to the relevant technical features. It should be noted that the modified or substituted technical solutions will all fall within the scope of protection of this invention.

Claims

1. A video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling, characterized in that, The method constitutes a cross-modal multi-task training framework based on orthogonal decoupling and domain adaptation, including the following steps: A cross-modal spatiotemporal unified encoder is constructed to extract global semantic features and local texture features from video frames. A fused spatiotemporal feature representation is obtained through dynamic graph convolution and temporal Transformer modeling. A multi-scale forgery-style feature decoupling module maps these spatiotemporal features to forgery features and style features, and an orthogonal constraint loss is introduced to maintain their independence in the vector space. The forgery features are input into a domain-adaptive multi-task learner, and adversarial training is performed using a gradient reversal layer and a dynamic adversarial classifier to obtain a domain-invariant forgery representation. The model parameters are updated using a multi-task joint optimization approach. The loss function includes frame-level forgery localization loss, video-level real / fake classification loss, orthogonal constraint loss, and domain adversarial loss.

2. The video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling according to claim 1, characterized in that, The cross-modal spatiotemporal unified encoder specifically includes: The ViT frame features and ResNet frame features of the input video are projected to a unified dimension and iteratively interacted through stacked cross-modal attention blocks. Each modality feature can extract key information from the other modality feature, achieving dynamic fusion. The fused features are then concatenated and projected to form a unified video frame representation, and learnable positional encoding is added to inject temporal information. This representation is then input into a Transformer network containing multiple layers of residual attention blocks, and long-range inter-frame dependencies are modeled through a self-attention mechanism. Simultaneously, dynamic graph convolution is introduced to construct a dynamic adjacency matrix (content graph) based on content similarity and an adjacency matrix (structure graph) based on inter-frame distance, and convolution operations are performed on the two types of graph structures respectively. Finally, the two convolution results are concatenated and linearly mapped to obtain a frame-level feature representation containing rich spatiotemporal relationships.

3. The video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling according to claim 1, characterized in that, The multi-scale forgery-style feature decoupling module specifically includes: The system receives global video features after temporal average aggregation; extracts features of different granularities by using multi-scale one-dimensional convolution with kernel sizes of 1, 3, and 8; concatenates the obtained features and inputs them into two independent linear projection layers to map them as forgery features and style features; and introduces orthogonal regularization constraints during training to keep the forgery features and style features approximately orthogonal in the vector space.

4. The video temporal forgery localization method based on cross-modal spatiotemporal feature decoupling according to claim 1, characterized in that, The domain-adaptive multi-task learner specifically includes: A dynamic adversarial classifier based on a gradient inversion layer is used to receive forged features and dynamically generate domain weights and weightedly combine the outputs of multiple base classifiers to adapt to different forgery sources; the adversarial training forces the feature encoder to learn a domain-invariant forgery representation; and through multi-task joint optimization, frame-level detection loss, video-level detection loss, adversarial loss and orthogonal constraint loss are weighted and combined to achieve multi-objective collaborative training.

Images