Generated video detection method based on cross fusion of texture, optical flow and depth features
By employing a detection method that integrates texture, optical flow, and depth features, this approach addresses the shortcomings in the accuracy and generalization capabilities of existing technologies for recognizing high-fidelity AI-generated videos, enabling precise detection of logical conflicts and physical laws.
Patent Information
- Application Number
- CN202511882525.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-03-03
AI Technical Summary
Existing generated video detection technologies struggle to identify logical flaws in physical laws when faced with high-fidelity AI-generated videos. They also lack generalization ability and deep cross-detection of intermodal interactions, resulting in low detection accuracy.
A detection method based on the cross-fusion of texture, optical flow and depth features is adopted. Through a global optical flow model, monocular depth estimation and cross-attention mechanism, combined with a dynamic dropout strategy, multimodal feature fusion and robust training are carried out.
It improves the accuracy of recognizing high-fidelity AI-generated videos, enhances the model's generalization ability, and can identify complex logical conflicts and violations of physical laws, adapting to fake videos with different generation architectures.
Smart Images
Figure CN121600446A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for detecting generated videos, specifically a method for detecting generated videos based on the cross-fusion of texture, optical flow, and depth features. Background Technology
[0002] With the rapid development of deep learning technology, advanced video generation models such as Sora, Gen-2, Runway, and Pika have emerged. The videos generated by these models have become nearly realistic in terms of texture detail, lighting variations, scene construction, and motion smoothness, even reaching a level that is difficult to distinguish with the naked eye. These highly realistic "hard samples" pose unprecedented and severe challenges to existing digital content security, public opinion governance, and judicial evidence collection.
[0003] Traditional deepfake technology has primarily focused on specific areas such as face replacement and facial expression manipulation. However, the new generation of AIGC videos has expanded to general scenarios, covering a wide range of content including natural scenery, urban street scenes, complex object movements, and non-human-faced creatures. This makes the spread of misinformation more covert.
[0004] However, existing detection technologies face significant bottlenecks. Most existing detection models are trained on specific datasets, and they often make judgments by learning the "fingerprints" left by specific generators. When faced with unknown, heterogeneous, and highly high-quality novel generative models, these models are prone to failure due to overfitting to the specific distribution of the training data, resulting in a precipitous drop in detection accuracy and insufficient generalization ability.
[0005] Furthermore, while AI-generated videos are becoming increasingly realistic in terms of visual perception, their generated content often contains deep logical flaws at the level of physical laws due to the limitations of the generation models' understanding of the physical world. Summary of the Invention
[0006] The purpose of this invention is to overcome the above-mentioned technical deficiencies and provide a method for detecting generated videos based on the cross-fusion of texture, optical flow and depth features, thereby solving the technical problem of difficulty in detecting the authenticity of AI-generated videos in the prior art.
[0007] To achieve the above-mentioned technical objectives, in a first aspect, the present invention provides a method for generating video detection based on the cross-fusion of texture, optical flow, and depth features, comprising the following steps:
[0008] The original video is acquired, and then subjected to uniform frame sampling, image normalization, and enhancement.
[0009] The original video is input into the global optical flow motion branch, and optical flow features are obtained through feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding.
[0010] The original video is input into the geometric depth branch to perform depth map transformation, and depth features are extracted by a convolutional encoder.
[0011] The original video is input into the RGB texture branch to extract spatial texture details and preliminary temporal context features of the video;
[0012] The final fused feature is obtained by fusing the optical flow features, the depth features, and the temporal context features.
[0013] A dynamic mask is generated and applied to the fused features. The resulting data is then fed into a multilayer perceptron to perform a classification operation and obtain the video authenticity classification result.
[0014] Compared with the prior art, the beneficial effects of the present invention include:
[0015] 1. Introduce an optical flow model based on global feature matching: to replace the traditional iterative optical flow, it utilizes the global receptive field characteristics to accurately capture global large displacement motion anomalies that violate physical laws in the generated video, while improving inference efficiency.
[0016] 2. Introducing a large model based on monocular depth estimation: Creatively introducing a large model based on monocular depth estimation, constructing an RGB-Flow-Depth three-flow architecture, adding physical consistency verification of geometric perspective dimension, and identifying forgeries that are "similar in form but not in spirit".
[0017] 3. Design a Cross-Attention conflict perception fusion mechanism: Abandon simple splicing and design a cross-attention mechanism to allow different modalities to "question" each other and actively discover semantic conflicts and logical loopholes between modalities.
[0018] 4. Introduce Dynamic Dropout hard sample mining strategy: randomly mask features and force learning of more robust discrimination clues to improve the model's ability to detect forgery traces and its cross-architecture generalization.
[0019] According to some embodiments of the present invention, uniform frame sampling, image normalization, and enhancement are performed on the original video, including the following steps:
[0020] Uniform frame sampling: In order to strike a balance between capturing long-term temporal dependencies and reducing computational redundancy, a uniform sampling strategy is adopted for the input video to obtain multiple sampled frames.
[0021] Image normalization and enhancement: All sampled frames are adjusted to a uniform resolution and normalized using the statistical mean and variance of the ImageNet dataset to eliminate differences in illumination and contrast; random flipping and random cropping data augmentation operations are introduced during the training phase to improve the robustness of the model.
[0022] According to some embodiments of the present invention, the original video is input into a global optical flow motion branch, and optical flow features are obtained through feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding, including the following steps:
[0023] Feature extraction: For two adjacent frames, the first feature map and the second feature map are extracted using an encoder with shared weights;
[0024] Global correlation calculation: Calculate the correlation between each pixel in the first feature map and all pixels (m,n) in the second feature map, and construct a global correlation matrix;
[0025] Optical flow regression: The matching probability distribution is calculated based on the global correlation matrix through the Softmax operation, and the expected optical flow displacement is obtained by regression in one step using this distribution. The expected optical flow displacement is a "pixel-level motion displacement map", which is further encoded into high-dimensional optical flow features.
[0026] According to some embodiments of the present invention, the original video input geometric depth branch performs depth map transformation, and depth features are extracted by a convolutional encoder, including the following steps:
[0027] The sequence of sampled frames in RGB format is converted into a sequence of depth maps, and high-dimensional depth features are extracted from them using a convolutional encoder.
[0028] According to some embodiments of the present invention, the original video is input into an RGB texture branch to extract spatial texture details and preliminary temporal context features of the video, including the following steps:
[0029] The spatial texture details and preliminary temporal context features of the original video are extracted using 3D Swin Transformer. These temporal context features are used to capture edge artifacts, blurring, or noise residues generated during image synthesis.
[0030] According to some embodiments of the present invention, feature fusion based on the optical flow features, the depth features, and the temporal context features is performed to obtain the final fused features, including the following steps:
[0031] The temporal context feature with the richest information is used as Query(Q), and the optical flow feature and the depth feature are concatenated as Key(K) and Value(V);
[0032] The consistency score matrix between temporal context features, optical flow features, and depth features is calculated using a scaled dot product attention mechanism.
[0033] The temporal context features are fused with the consistency score matrix using residual connections and layer normalization to generate the final fused features.
[0034] According to some embodiments of the present invention, generating a dynamic mask and applying the dynamic mask to the fused feature includes the following steps:
[0035] During training, a mask vector following a Bernoulli distribution is generated. The retention probability p is not fixed, but dynamically adjusted with each training round to obtain the dynamic mask.
[0036] The dynamic mask is applied to the fused features to randomly mask one or two modal features.
[0037] Secondly, the present invention provides a video detection system based on the cross-fusion of texture, optical flow, and depth features, comprising:
[0038] The data preprocessing module acquires the original video and performs uniform frame sampling, image normalization, and enhancement on the original video.
[0039] The optical flow feature extraction module inputs the original video into the global optical flow motion branch, and obtains optical flow features through feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding.
[0040] The depth feature extraction module inputs the original video into the geometric depth branch to perform depth map transformation, and extracts depth features through a convolutional encoder.
[0041] The temporal context feature extraction module inputs the original video into the RGB texture branch to extract the spatial texture details and preliminary temporal context features of the video;
[0042] The feature fusion module performs feature fusion based on the optical flow features, the depth features, and the temporal context features to obtain the final fused features;
[0043] The output module generates a dynamic mask, applies the dynamic mask to the fused features, and then sends it to a multilayer perceptron to perform a classification operation to obtain the video authenticity classification result.
[0044] Thirdly, the present invention provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a video generation detection method based on the cross-fusion of texture, optical flow and depth features as described in any one of the first aspects.
[0045] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0046] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein the abstract drawings are to be completely consistent with one of the drawings in the specification:
[0047] Figure 1 This is a schematic diagram of the overall framework of a video generation detection method provided in one embodiment of the present invention;
[0048] Figure 2 This is a schematic diagram of a global matching optical flow model framework based on Transformer provided in one embodiment of the present invention. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0050] It should be noted that although functional modules are divided in the system diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0051] Disadvantages of existing technologies: As mentioned above, existing video detection solutions, whether single-modal analysis or simple multimodal stitching, suffer from one or more of the following disadvantages when dealing with novel high-fidelity videos such as Sora:
[0052] 1. Optical flow calculation is inefficient and struggles to capture large displacement anomalies.
[0053] In traditional detection schemes, optical flow extraction typically relies on iterative algorithms (such as RAFT and PWC-Net). These algorithms require multiple iterative updates during inference, resulting in slow inference speed and high computational overhead, making it difficult to meet the real-time requirements of large-scale video detection. More seriously, traditional optical flow algorithms are based on the assumption of local correlation, which easily leads to smoothing errors or cumulative errors when dealing with common "non-rigid large displacements," "instantaneous object movement," or "illusory motion" in generated videos, failing to accurately reflect the unique motion logic defects of generated videos.
[0054] 2. Lack of physical and geometric laws (lack of in-depth verification)
[0055] Most existing technologies are limited to the analysis of "2D texture (RGB)" and "2D motion (optical flow)," ignoring the geometric consistency of video content in 3D space. AI-generated videos often exhibit phenomena that violate physical laws, such as "perspective errors," "disorganized object depth relationships," "objects embedded in the background," or "reversed foreground and background occlusion relationships." Due to the lack of depth information, existing solutions cannot identify such forgeries at the 3D geometric level, resulting in an extremely high false negative rate for high-fidelity videos.
[0056] 3. Fragmentation of intermodal interactions (inability to detect logical conflicts)
[0057] Current mainstream solutions employ a "simple feature concatenation" approach (e.g., the formula `ffusion = [frgb; fflow]`), implicitly assuming that the features of each modality are complementary and consistent. However, the core feature of fake videos is often the conflict between modalities. For example, a video might have very realistic textures (RGB modality tends to classify it as true), but its motion trajectory might violate the laws of gravity (optical flow modality tends to classify it as false), or the object's depth might blend seamlessly with the background depth (depth modality anomaly). Simple concatenation often leads to the dominant modality (usually the most informative RGB) masking the anomalous signals of the weaker modalities, and the model lacks deep interaction and challenge mechanisms to discover this inherent contradiction.
[0058] 4. Poor generalization ability, specific generator features
[0059] Existing fake video detection methods often rely on fixed noise fingerprints or artifact patterns generated by specific generative architectures (such as GAN series models) in the training set. When test samples are generated by new generative paradigms (such as Sora videos based on DiffusionTransformer), the models cannot extract common fake features shared across architectures due to the lack of robust modeling for missing or perturbated features, resulting in significant performance degradation in cross-dataset (OOD) tests.
[0060] Reference Figures 1 to 2 , Figure 1 This is a schematic diagram of the overall framework of a video generation detection method provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of a global matching optical flow model framework based on Transformer provided in one embodiment of the present invention.
[0061] In one embodiment, a video detection method based on the cross-fusion of texture, optical flow, and depth features includes the following steps: acquiring the original video; performing uniform frame sampling, image normalization, and enhancement on the original video; inputting the original video into a global optical flow motion branch, performing feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding to obtain optical flow features; inputting the original video into a geometric depth branch to perform depth map transformation, and extracting depth features through a convolutional encoder; inputting the original video into an RGB texture branch to extract spatial texture details and preliminary temporal context features of the video; performing feature fusion based on optical flow features, depth features, and temporal context features to obtain the final fused features; generating a dynamic mask, applying the dynamic mask to the fused features, and then feeding it into a multilayer perceptron to perform a classification operation to obtain the video authenticity classification result.
[0062] 1. Overall Overview of the Technical Solution
[0063] This invention provides an end-to-end deepfake video detection framework that includes "three-stream feature extraction + cross-conflict perception + random masking features".
[0064] The system first preprocesses the input video; then it constructs a three-in-one feature extraction branch of RGB (texture), GMFlow (optical flow), and DepthAnything (depth) to encode the video from the three dimensions of vision, motion, and geometry, respectively; subsequently, it uses the Cross-Attention mechanism to mine inconsistencies between modalities (such as texture and depth mismatch, texture and motion conflict); finally, in the classification stage, it uses the Dynamic Dropout strategy to randomly mask features, forcing the learning of more robust discrimination cues, thereby improving the detection generalization of OOD data.
[0065] 2. Specific Implementation Steps
[0066] Step 1: Data Reception and Preprocessing
[0067] The purpose of this step is to standardize the original videos, which vary in format and length, into sequences suitable for model input, thus laying the foundation for subsequent accurate analysis.
[0068] 1. Uniform Frame Sampling: To strike a balance between capturing long-term temporal dependencies and reducing computational redundancy, a uniform sampling strategy is employed on the input video, ensuring that the selected frames are evenly distributed across the entire video's temporal domain. Let the input video be V, and the extracted frame sequence be X:
[0069] X = {I1, I2, ..., I} T},I t ∈R H×W×3 (2)
[0070] Where T is the number of sampling frames, and H and W are the image height and width. This uniform sampling method ensures that the sampling frames cover the beginning, middle, and end segments of the video, which is beneficial for obtaining the overall temporal variation features of the video, thereby improving the effectiveness of subsequent feature extraction.
[0071] 2. Image Normalization and Enhancement: All frames are adjusted to a uniform resolution (e.g., 224×224), and standardized using the statistical mean and variance of the ImageNet dataset to eliminate differences in lighting and contrast. Simultaneously, data augmentation operations such as random flipping and random cropping are introduced during the training phase to improve the model's robustness.
[0072] Step Two: Three-Stream Parallel Feature Extraction and Modeling [Core Module]
[0073] This invention designs three parallel feature extraction branches to capture forgery clues from different dimensions. These three branches share a Transformer-based backbone network architecture, but are specifically optimized for different modalities.
[0074] 2.1 Global optical flow motion branch, technical issues: Traditional optical flow algorithms (such as RAFT) rely on iterative loops, resulting in slow inference speed. Furthermore, based on local correlation, they are prone to failure when faced with "instantaneous large displacements that violate physical laws" commonly found in AI-generated videos.
[0075] Solution: This invention introduces a Transformer-based global matching optical flow model to extract motion features, such as GMFlow (Global Matching Flow).
[0076] Core principles and formula derivation: GMFlow abandons multi-scale pyramids and iterative refinement, and directly calculates global correlations on full-resolution feature maps.
[0077] • Feature extraction: Extracting features from two adjacent frames of images I t ,I t+ 1. Extract feature maps using a weighted encoder.
[0078] F1,F2∈R H×W×D
[0079] • Global correlation calculation: Calculate the correlation between each pixel (i,j) in F1 and all pixels (m,n) in F2, and construct the global correlation matrix C:
[0080]
[0081] Where D is the feature dimension.
[0082] • Optical flow regression: The matching probability distribution P is calculated through a softmax operation, and the desired optical flow displacement Vflow is obtained by a one-time regression using this distribution.
[0083] P(m,n|i,j)=Softmax(C(i,j,·,·))(m,n) (4)
[0084]
[0085] The original optical flow Vflow is a "pixel-level motion displacement map" that needs to be further encoded into a high-dimensional feature vector Fflow.
[0086] Technical effects: This method not only improves inference speed, but also, by utilizing the global receptive field, can keenly capture discontinuous and abrupt large displacement anomalies in the generated video, which is an important basis for judging AI-generated videos.
[0087] 2.2 Geometric Depth Branch, Technical Issues: Existing technologies only analyze two-dimensional images and cannot identify logical flaws in the three-dimensional geometric level, such as "perspective errors" and "objects embedded in the background." Solution: This invention creatively introduces a pre-trained monocular depth estimation large model, such as Depth Anything.
[0088] Core principle: The Depth Anything model is trained on large-scale unlabeled data and has extremely strong zero-shot generalization ability. This invention utilizes it to transform an RGB frame sequence X into a depth map sequence D. seq And extract high-dimensional geometric features F through a convolutional encoder. depth :
[0089] D seq =DepthAnything(X),D seq ∈R T×H×W×1 (6)
[0090] F depth =ConvEncoder(D seq (7)
[0091] To eliminate the scale uncertainty in depth estimation, we perform affine invariant normalization on the depth map before feature extraction.
[0092] Technical effect: It adds a completely new "geometric dimension" to the detection model. It can reveal fake content that is "similar in form but not in spirit" but has extremely realistic textures but chaotic depth layering (e.g., a background that is clearly in the distance, but the depth map shows it on the same plane as the foreground object).
[0093] 2.3 RGB texture branch: Utilize 3D Swing Transformer to extract spatial texture details and preliminary temporal context features F from the video. rgb It is mainly used to capture edge artifacts, blurring, or noise residue generated in image synthesis.
[0094] Step 3: Problem with the Conflict-Aware Fusion Module Based on Multimodal Cross-Attention: Simple feature concatenation assumes modal complementarity and cannot detect inconsistencies between modalities (e.g., RGB displays a running dog, but optical flow displays a translated piece of paper). Solution: This invention introduces a multimodal cross-attention fusion module to replace the traditional concatenation operation.
[0095] Core principles and formula derivation: The core idea of this module is to use one modality as a query to "question" the information of other modalities, thereby calculating the consistency between modalities.
[0096] 1. QKV Design: We use the most informative RGB texture features (temporal context features) as the Query (Q), and concatenate the optical flow features and depth features as the Key (K) and Value (V).
[0097] Q = F rgb W q (8)
[0098] K = Concat(F) flow ,F depth W k (9)
[0099] V = Concat(F) flow ,F depth W v (10)
[0100] Among them W q W k W v It is a learnable projection matrix.
[0101] 2. Conflict Calculation (Attention): The model calculates the "consistency score" matrix between the texture and auxiliary modalities (motion, geometry) through a scaled dot product attention mechanism.
[0102]
[0103] Attention(Q,K,V)=Softmax(Score)V (12)
[0104] 3. Feature fusion output: Residual connections and layer normalization (LayerNorm) are used to generate the feature output.
[0105] The final fusion feature Ff used:
[0106] Ffused=LayerNorm(Frgb+Attention(Q,K,V))(13)
[0107] Technical effect: It achieves a leap from "information overlay" to "logical questioning". If the content displayed by Q (texture) does not semantically match the content displayed by K (depth / optical flow), the Attention score distribution will show anomalies, thereby generating a strong feature response (conflict signal) and accurately identifying fine-grained forgeries.
[0108] Step 4: Dynamic Dropout Mechanism
[0109] Technical Issue: Existing forgery detection models are prone to feature co-adaptation, meaning the classifier relies solely on texture artifacts or noise patterns from specific generators in the training set. When the source of the forged samples changes (e.g., a previously unseen Sora video), these specific artifacts no longer exist, leading to a decrease in the model's generalization ability.
[0110] Solution: This invention introduces a Dynamic Modality Dropout mechanism into the classification head. During training, this mechanism randomly masks some fused feature dimensions with a probability that dynamically changes with each epoch, ensuring the model can still perform the discrimination task even with incomplete feature input, thereby suppressing over-reliance on single modalities or local artifacts.
[0111] Core principles and formula derivation:
[0112] 1. Dynamic Mask Generation: During training, a mask vector M∈{0,1}D following a Bernoulli distribution is generated. Its retention probability p is not fixed, but dynamically adjusted with each training epoch (e.g., gradually decreasing p as training progresses to increase difficulty).
[0113]
[0114] M~Bernoulli(pepoch) (15)
[0115] 2. Feature Masking (Hard Sample Simulation): Applying a mask to the fused features:
[0116] F masked =F fused ⊙M (16)
[0117] This operation randomly masks features of one or two modalities (e.g., forcing the model to make judgments when there is only optical flow and no texture).
[0118] 3. Final classification: The masked features are fed into a multilayer perceptron (MLP) and a softmax layer.
[0119] ypred=Softmax(MLP(Fmasked)) (17)
[0120] Technical Results: Dynamic feature Dropout proactively constructs training conditions with missing information, essentially introducing "perturbation samples" of varying degrees. This perturbation forces the model to learn more robust motion, structural, and generative consistency features, rather than relying solely on texture artifacts that are easily smoothed out by strong generators. Experimental results show that this mechanism improves the model's detection accuracy on out-of-domain (OOD) datasets and enhances its generalization ability to forgery traces from different generative architectures.
[0121] The "Generalization-based AI-generated video detection method based on multimodal hard sample learning" provided by this invention has the following beneficial effects:
[0122] 1. Resolved the contradiction between optical flow calculation efficiency and large displacement capture:
[0123] Existing technologies mostly employ iterative optical flow algorithms, which are slow and prone to large errors. This invention replaces the traditional iterative optical flow algorithm with an optical flow model based on global feature matching. By utilizing the global receptive field of the Transformer, it not only improves inference speed (meeting the requirements of real-time detection) but also effectively captures long-distance abnormal motions that violate physical laws in the generated video (such as instantaneous movement of objects or discontinuous jumps).
[0124] 2. Achieved accurate identification of sophisticated forged videos that are "similar in form but not in spirit":
[0125] Existing technologies lack 3D geometric verification. This invention fills the gap in 3D geometric consistency detection by introducing a large model based on monocular depth estimation. It can effectively identify videos generated by models such as Sora that have "realistic lighting and shadows but incorrect perspective," thus compensating for spatial structural anomalies that cannot be detected by RGB and optical flow alone.
[0126] 3. It overcomes the limitations of simple feature splicing and achieves deep logical conflict detection:
[0127] Existing technologies often employ simple concat stitching, which fails to detect modal conflicts. This invention utilizes a cross-attention mechanism to replace traditional stitching, enabling the model to possess "logical reasoning" capabilities. The system can proactively seek out cross-modal semantic conflicts (e.g., a static background in an image, but optical flow shows large-scale noise motion), improving detection accuracy.
[0128] 4. Enhance generalization ability across generative models (OOD Robustness):
[0129] Existing forgery detection methods typically rely on fixed artifact patterns generated by a specific type of generator in the training set, which easily leads to significant performance degradation in cross-architecture tests. This invention introduces a dynamic feature dropout mechanism that dynamically masks certain feature dimensions during training, constructing scenarios with varying degrees of information loss. This suppresses the model's over-reliance on specific artifacts or modal features, prompting the classifier to learn more stable, consistent generation features. Experimental results show that this invention maintains high detection accuracy even when facing unseen generation paradigms (such as videos based on Diffusion Transformer), and its out-of-domain (OOD) robustness is superior to existing technologies. Specific Implementation Example 1
[0131] To make the objectives, technical solutions, and advantages of this invention clearer, a specific scenario will be used as an example for explanation below.
[0132] Scene Description
[0133] The video to be tested is a video about a coffee cup generated by a certain diffusion model.
[0134] • Video footage: The coffee cup's texture is clear, but no obvious artifacts are visible; it's difficult to distinguish between real and fake based solely on the RGB lighting.
[0135] • Physical anomaly: When the coffee cup is moved, its reflection on the table does not move accordingly (inconsistent optical flow), and the depth of the bottom of the cup blends with the depth of the table (geometric error).
[0136] Processing flow of the present invention
[0137] 1. Preprocessing: The system receives the video, extracts keyframes evenly, adjusts them to a uniform 224×224 resolution, and normalizes them.
[0138] 2. Feature extraction:
[0139] • RGB branch: Extracted cup images with clear texture features, without obvious artifacts.
[0140] • GMFlow branch: Using global matching calculation, it detects that the pixels of the cup are moving horizontally, but the shadow pixels at the bottom of the cup do not move with it, generating abnormal motion field features.
[0141] • Depth branch: Extracting the depth map reveals that the depth boundary between the cup and the table merges during the movement, making the cup appear to be "embedded" inside the table, resulting in the extraction of incorrect geometric features.
[0142] 3. Fusion Inference: The Cross-Attention module uses RGB features as the query and depth and optical flow features as the key. Calculations revealed extremely low consistency between the two, and an abnormal distribution of attention weights. The model captured a strong semantic conflict between RGB (the cup's outline is intact) and depth (the cup is embedded in the table).
[0143] 4. Final judgment: Although the RGB features tend to be "true", due to severe cross-modal conflicts, the classifier trained by DynamicDropout made a comprehensive judgment and finally determined that the video was "AI generated".
[0144] The process of uniformly sampling, normalizing, and enhancing the original video includes the following steps: Uniform frame sampling: To achieve a balance between capturing long-term temporal dependencies and reducing computational redundancy, a uniform sampling strategy is adopted for the input video to obtain multiple sampled frames; Image normalization and enhancement: All sampled frames are adjusted to a uniform resolution and normalized using the statistical mean and variance of the ImageNet dataset to eliminate differences in illumination and contrast; Random flipping and random cropping data enhancement operations are introduced during the training phase to improve the robustness of the model.
[0145] The process involves inputting the original video into the global optical flow motion branch, and obtaining optical flow features through feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding. The steps include: Feature extraction: Extracting a first feature map and a second feature map from two adjacent frames using a weighted encoder; Global correlation calculation: Calculating the correlation between each pixel in the first feature map and all pixels in the second feature map to construct a global correlation matrix; Optical flow regression: Calculating the matching probability distribution based on the global correlation matrix using a Softmax operation, and then using this distribution to regress the desired optical flow displacement in one step. The desired optical flow displacement is a "pixel-level motion displacement map," which is further encoded into high-dimensional optical flow features.
[0146] The original video is input into the geometric depth branch to perform depth map transformation, and depth features are extracted by a convolutional encoder. The steps include: converting a sequence of sampled frames in RGB format into a sequence of depth maps, and using a convolutional encoder to extract high-dimensional depth features.
[0147] The original video is input into the RGB texture branch to extract spatial texture details and preliminary temporal context features of the video. The steps include: using 3D Swin Transformer to extract spatial texture details and preliminary temporal context features of the original video. The temporal context features are used to capture edge artifacts, blur or noise residue generated in image synthesis.
[0148] The final fused feature is obtained by fusing optical flow features, depth features, and temporal context features. The steps include: taking the temporal context feature with the richest information as Query(Q), and concatenating the optical flow features and depth features as Key(K) and Value(V); calculating the consistency score matrix between the temporal context feature, optical flow feature, and depth feature through a scaled dot product attention mechanism; and fusing the temporal context feature with the consistency score matrix using residual connections and layer normalization to generate the final fused feature.
[0149] The process of generating a dynamic mask and applying it to the fusion features includes the following steps: During training, a mask vector following a Bernoulli distribution is generated, where the retention probability p is not fixed but dynamically adjusted with each training round to obtain the dynamic mask; the dynamic mask is then applied to the fusion features, randomly masking one or two modalities.
[0150] In one embodiment, a video detection system based on the cross-fusion of texture, optical flow, and depth features includes: a data preprocessing module for acquiring the original video and performing uniform frame sampling, image normalization, and enhancement on the original video; an optical flow feature extraction module for inputting the original video into a global optical flow motion branch, performing feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding to obtain optical flow features; a depth feature extraction module for inputting the original video into a geometric depth branch to perform depth map transformation and extracting depth features through a convolutional encoder; a temporal context feature extraction module for inputting the original video into an RGB texture branch to extract spatial texture details and preliminary temporal context features of the video; a feature fusion module for performing feature fusion based on optical flow features, depth features, and temporal context features to obtain the final fused features; and a result output module for generating a dynamic mask, applying the dynamic mask to the fused features, and then sending it to a multilayer perceptron to perform a classification operation to obtain the video authenticity classification result.
[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0152] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described terminal embodiment, such that the processor performs the video generation detection method based on the cross-fusion of texture, optical flow, and depth features described in the above-described embodiment.
[0153] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0154] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.
[0155] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A method for generating video detection based on the cross-fusion of texture, optical flow, and depth features, characterized in that, Including the following steps: The original video is acquired, and then subjected to uniform frame sampling, image normalization, and enhancement. The original video is input into the global optical flow motion branch, and optical flow features are obtained through feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding. The original video is input into the geometric depth branch to perform depth map transformation, and depth features are extracted by a convolutional encoder. The original video is input into the RGB texture branch to extract spatial texture details and preliminary temporal context features of the video; The final fused feature is obtained by fusing the optical flow features, the depth features, and the temporal context features. A dynamic mask is generated and applied to the fused features. The resulting data is then fed into a multilayer perceptron to perform a classification operation and obtain the video authenticity classification result.
2. The video generation detection method based on the cross-fusion of texture, optical flow, and depth features according to claim 1, characterized in that, The original video undergoes uniform frame sampling, image normalization, and enhancement, including the following steps: Uniform frame sampling: In order to strike a balance between capturing long-term temporal dependency information and reducing computational redundancy, a uniform sampling strategy is adopted for the input video to obtain multiple sampled frames. Image normalization and enhancement: All sampled frames are adjusted to a uniform resolution and normalized using the statistical mean and variance of the ImageNet dataset to eliminate differences in illumination and contrast. During the training phase, random flipping and random cropping of data augmentation operations are introduced to improve the robustness of the model.
3. The video generation detection method based on the cross-fusion of texture, optical flow, and depth features according to claim 1, characterized in that, The original video is input into the global optical flow motion branch, and optical flow features are obtained through feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding, including the following steps: Feature extraction: For two adjacent frames, the first feature map and the second feature map are extracted using an encoder with shared weights; Global correlation calculation: Calculate the correlation between each pixel in the first feature map and all pixels in the second feature map, and construct a global correlation matrix; Optical flow regression: The matching probability distribution is calculated based on the global correlation matrix through the Softmax operation, and the expected optical flow displacement is obtained by regression in one step using this distribution. The expected optical flow displacement is further encoded into high-dimensional optical flow features.
4. The video generation detection method based on the cross-fusion of texture, optical flow, and depth features according to claim 2, characterized in that, The original video is input into the geometric depth branch to perform depth map transformation, and depth features are extracted through a convolutional encoder, including the following steps: The sequence of sampled frames in RGB format is converted into a sequence of depth maps, and high-dimensional depth features are extracted from them using a convolutional encoder.
5. The video generation detection method based on the cross-fusion of texture, optical flow, and depth features according to claim 1, characterized in that, The original video is input into the RGB texture branch to extract spatial texture details and preliminary temporal context features, including the following steps: The spatial texture details and preliminary temporal context features of the original video are extracted using 3D Swin Transformer. These temporal context features are used to capture edge artifacts, blurring, or noise residues generated during image synthesis.
6. The video generation detection method based on the cross-fusion of texture, optical flow, and depth features according to claim 1, characterized in that, The final fused feature is obtained by fusing the optical flow features, the depth features, and the temporal context features, including the following steps: The temporal context feature with the richest information is used as Query(Q), and the optical flow feature and the depth feature are concatenated as Key(K) and Value(V); The consistency score matrix between temporal context features, optical flow features, and depth features is calculated using a scaled dot product attention mechanism. The temporal context features are fused with the consistency score matrix using residual connections and layer normalization to generate the final fused features.
7. The video generation detection method based on the cross-fusion of texture, optical flow, and depth features according to claim 1, characterized in that, Generating a dynamic mask and applying the dynamic mask to the fused feature includes the following steps: During training, a mask vector following a Bernoulli distribution is generated. The retention probability p is not fixed, but dynamically adjusted with each training round to obtain the dynamic mask. The dynamic mask is applied to the fused features to randomly mask one or two modal features.
8. A generative video detection system based on the cross-fusion of texture, optical flow, and depth features, characterized in that, include: The data preprocessing module acquires the original video and performs uniform frame sampling, image normalization, and enhancement on the original video. The optical flow feature extraction module inputs the original video into the global optical flow motion branch, and obtains optical flow features through feature extraction, global correlation calculation, optical flow regression, and high-dimensional encoding. The depth feature extraction module inputs the original video into the geometric depth branch to perform depth map transformation, and extracts depth features through a convolutional encoder. The temporal context feature extraction module inputs the original video into the RGB texture branch to extract the spatial texture details and preliminary temporal context features of the video; The feature fusion module performs feature fusion based on the optical flow features, the depth features, and the temporal context features to obtain the final fused features; The output module generates a dynamic mask, applies the dynamic mask to the fused features, and then sends it to a multilayer perceptron to perform a classification operation to obtain the video authenticity classification result.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the video generation detection method based on the cross-fusion of texture, optical flow and depth features as described in any one of claims 1 to 7.