AI synthetic video detection method based on multi-level feature fusion

Abstract

The invention discloses an AI synthetic video detection method based on multi-level feature fusion, and relates to the technical field of artificial intelligence security and multimedia content authentication. The method comprises the following steps: 1) inputting a to-be-detected video V, preprocessing each key frame, and expanding the key frame; 2) extracting low-layer visual texture features to output low-layer detection confidence, performing modeling of middle-layer time sequence consistency, and performing high-layer semantic cross-modal analysis; (3) carrying out Transform-Mama fusion and adaptive weighted decision making, wherein a Transform-Mama fusion module and an adaptive weighted decision making module are included; and 4) outputting a detection result and judging a threshold value: comparing a set detection threshold value to judge whether an AI synthetic video exists or not, and realizing efficient identification of various AI synthetic videos by capturing visual, time sequence and semantic features of the video in a layered manner and dynamically optimizing a fusion strategy while considering the detection precision, the real-time performance and the robustness.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security and multimedia content authentication technology, and in particular to an AI-synthesized video detection method based on multi-level feature fusion. It is applicable to scenarios such as video platform content review, social media false information identification, and judicial evidence authenticity verification, and can efficiently identify AI-generated or tampered video content. Background Technology

[0002] With the rapid development of generative AI technologies (such as GAN, Diffusion models, and Transformer), the visual realism and temporal coherence of deepfake videos have been continuously improved, and they have been widely used in digital content creation, but at the same time they have been abused for illegal activities.

[0003] Existing deepfake detection methods have significant limitations: First, they suffer from a prominent problem of single-feature dependency. Methods based on 2D / 3D convolution only focus on spatial features and ignore temporal consistency; while Transformer-based methods can model long-range dependencies, they have high computational complexity. First, the real-time performance is poor and local details are not captured adequately. Second, the fusion strategy is rigid, mostly using fixed weight splicing or simple weighting, and cannot dynamically adjust the importance of features according to the video content. Third, the generalization ability is insufficient, and the detection accuracy drops significantly when facing unseen generative models (such as Sora and Stable VideoDiffusion) or adversarial attacks. Fourth, multimodal information is not fully utilized, relying only on visual features and failing to effectively combine cross-modal semantic consistency analysis such as audio and text.

[0004] To address the aforementioned issues, there is an urgent need to design a multi-level feature fusion detection system that balances detection accuracy, real-time performance, and robustness. This system should capture the visual, temporal, and semantic features of videos in layers and dynamically optimize the fusion strategy to achieve efficient identification of various deepfake videos. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing deepfake detection methods and provide an AI-synthesized video detection method based on multi-level feature fusion. By constructing a three-level detection architecture of low-level vision, mid-level temporal sequence, and high-level semantics, and combining the Transformer-Mamba hybrid mechanism with an adaptive weighted fusion strategy, this invention solves the problems of limitations of single features, rigid static fusion, high computational complexity, and insufficient generalization ability.

[0006] The technical solution of this invention is: an AI-synthesized video detection method based on multi-level feature fusion, including an offline model training stage and an online detection stage, the specific steps of which are as follows: The online detection phase includes steps 1) to 4). Step 1) Video preprocessing and key frame extraction: Input the video V to be detected, preprocess each key frame, and use the RG13 data augmentation strategy to augment the key frames.

[0007] The specific method in step 1) is as follows: Step 1.1) Input the video V to be detected, and extract K key frames using a uniform sampling strategy, denoted as... The sampling interval is dynamically adjusted according to the video duration. When the duration is ≤10s, K=16, and when the duration is >10s, K=32, thereby covering the complete temporal information of the video. Step 1.2) Preprocess each keyframe: Normalize the image size to 224×224 using OpenCV, convert it to RGB color space, perform mean subtraction and standard deviation normalization to eliminate the influence of lighting and resolution differences on feature extraction; where mean = [123.68, 116.779, 103.939], standard deviation = [58.393, 57.12, 57.375]; Step 1.3) Use the RG13 data augmentation strategy to augment the keyframes, including random horizontal flipping, brightness adjustment, contrast adjustment, and Gaussian blur.

[0008] Step 2) Three-level feature extraction: Extract low-level visual texture features to output low-level detection confidence, then perform mid-level temporal consistency modeling, and finally perform high-level semantic cross-modal analysis.

[0009] The specific method in step 2) is as follows: 2.1) Low-level visual texture feature extraction Using a pre-trained ResNet-50 as the backbone network, and taking pre-processed keyframes as input, 2048-dimensional depth visual features are extracted from each frame, denoted as... ,in Analysis using a local texture discriminator It detects blurred edges, unnatural texture transitions, and abnormal skin color distribution, and outputs low-level detection confidence scores. The mathematical expression is:

[0010]

[0011] in The average pooling result of K-frame features is used, the Dropout probability is set to 0.5, and Linear is a fully connected layer; 2.2) Mid-level timing consistency modeling Build a Transformer-Mamba hybrid architecture to handle the Transformer module and the Mamba module separately. To capture temporal features: The Transformer module employs a 4-layer encoder, with each layer containing a multi-head self-attention layer and a feedforward network, to capture long-range inter-frame dependencies and output features. ; The Mamba module employs a selective state-space architecture, setting the dimension D=2048, the expansion factor E=2, and the convolution kernel size=4, achieving linear complexity. Model local temporal patterns and output features ; through dynamic weights , right and Weighted fusion is performed, in which, , ; Obtain the mid-level temporal features Output the confidence level of the middle layer detection. The mathematical expression is:

[0012]

[0013]

[0014]

[0015] GRU stands for Gated Recurrent Unit, used for temporal feature aggregation.

[0016] 2.3) High-level semantic cross-modal analysis

[0017] Integrating visual, audio, and textual multimodal information, a cross-modal consistency detection mechanism is constructed: Visual-Text Alignment: A pre-trained XCLIP model is used. Inputting keyframes and video title / description text T, the cosine similarity between visual and textual features is calculated. ; Audio-video synchronization: Using the AVHubert model, the input video audio track A and the lip movement region of keyframes are analyzed to determine the temporal synchronization between lip movements and audio, and the similarity score is output. ; Semantic consistency: Using the CLIP model, input keyframes and text T to calculate cross-modal semantic matching degree. The three similarities are concatenated into cross-modal features. Output high-level detection confidence through semantic classifier The mathematical expression is:

[0018]

[0019]

[0020]

[0021] Step 3) Transformer-Mamba Fusion and Adaptive Weighted Decision: This includes the Transformer-Mamba fusion module and the adaptive weighted decision module.

[0022] 3.1) Transformer-Mamba Fusion Module

[0023] For the mid-level temporal features obtained in step 2) Further optimization: The feature is split into local windows, and a Transformer self-attention model is used to capture local dependencies within each window. Features are transferred between windows using a Mamba state-space model, and the output is a fused temporal feature. The time complexity is optimized to ; 3.2) Adaptive Weighted Decision Module Construct a learnable weight vector w=[ , , ],in , , The weights corresponding to low-level, mid-level, and high-level features respectively satisfy the following conditions: + + =1; The detection confidence scores of the three levels are fused using self-weighted logistic regression to calculate the final detection score. The mathematical expression is:

[0024] For trainable parameters,

[0025] ) Among them, the weight w is adaptively adjusted during the training process: when visual forgery traces exist in the video... When the time sequence increases, it becomes abnormal. Increase, semantic inconsistency Increase; Step 4) Detection result output and threshold judgment: Compare with the set detection threshold τ to determine whether AI-synthesized video exists.

[0026] The specific method in step 4) is as follows: 4.1) Set the detection threshold ; 4.2) If ≥ Determine if the video is a deepfake, and output "deepfake" result and confidence level. ;like < Determine if the video is real, output "Real" result and confidence level 1- ; 4.3) Output auxiliary detection information: including forged regions detected at lower layers, indexes of temporal anomalies at middle layers, and cross-modal inconsistency types at higher layers.

[0027] The offline model training phase includes step 5). Step 5) Model Training and Optimization: Train and optimize the model parameters and adaptive weight vector from Steps 2-3 using the labeled dataset. The specific method is as follows: 5.1) Constructing the training dataset: Integrating the UCF101 real video dataset, the Chinese Academy of Sciences AI-generated video dataset, and the FF++ deepfake video dataset, a total of 26,020 video samples were divided into training and validation sets in an 8:2 ratio; 5.2) The AdamW optimizer was used to train the model with an initial learning rate of 1e-3, which decayed to 1 / 10 of the original rate every 5 rounds. The weight decay coefficient was 1e-4, the batch size was 8, and the number of training rounds was 30. 5.3) The cross-entropy loss function Loss = -y × log ( )-(1-y) × log (1- ), where y=1 represents a fake video and y=0 represents a real video. Combined with an early stopping mechanism, i.e., if the validation set loss does not decrease for 3 consecutive rounds, training is stopped to prevent the model from overfitting; Through training in step 5, the optimized model parameters are obtained, including the ResNet-50 fine-tuning parameters, the Transformer-Mamba fusion module parameters, and the initial trainable parameters of the adaptive weight vector w, which are then applied to the online detection stage.

[0028] The beneficial effects of this invention are as follows: 1. Multi-level feature coverage: Through a three-level architecture of low-level vision, mid-level temporal sequence, and high-level semantics, it comprehensively captures pixel-level traces, temporal anomalies, and cross-modal contradictions in deepfake videos. The detection accuracy is improved by an average of 5.1% compared with existing methods, and the accuracy reaches 94.2% on the UCF101 and Chinese Academy of Sciences AI-generated video datasets. 2. High-efficiency computing architecture: The Transformer-Mamba hybrid mechanism reduces the complexity of time-series modeling from... Down to The 16-frame video detection time is only 0.23 seconds, which meets the real-time detection requirements, i.e., frame rate ≥ 4fps; 3. Adaptive fusion strategy: The learnable weight vector dynamically adjusts the importance of each level according to the video content, maintaining high robustness against different types of forgery, such as face swapping, facial expression manipulation, and scene replacement, with a 12% improvement in generalization ability; 4. Multimodal collaboration: Combining visual, audio, and text cross-modal analysis, it effectively identifies advanced forgery features such as audio-visual asynchrony and semantic contradictions, and improves resistance to adversarial attacks such as adding Gaussian noise and compression distortion by 8%. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the detection framework of the present invention.

[0030] Figure 2 This is a flowchart of the method of the present invention.

[0031] Specific implementation method

[0032] The present invention will be further described in detail below with reference to specific embodiments: A method for detecting AI-synthesized videos based on multi-level feature fusion, comprising the following steps: Step 1) Video preprocessing and keyframe extraction 1.1) Input the video V to be detected, and extract K key frames using a uniform sampling strategy, denoted as... The sampling interval is dynamically adjusted according to the video duration (K=16 when the duration is ≤10s, K=32 when the duration is >10s) to ensure that the complete temporal information of the video is covered; 1.2) Preprocess each keyframe: Normalize the image size to 224×224 using OpenCV, convert it to RGB color space, and perform mean subtraction (mean = [123.68, 116.779, 103.939]) and standard deviation normalization (standard deviation = [58.393, 57.12, 57.375]) to eliminate the influence of lighting and resolution differences on feature extraction; 1.3) The RG13 data augmentation strategy is used to augment the keyframes, including random horizontal flipping (probability 0.5), brightness adjustment (±10%), contrast adjustment (±15%), and Gaussian blur (kernel size=3×3, probability 0.2), to improve the model's generalization ability.

[0033] Step 2) Three-level feature extraction

[0034] 2.1) Low-level visual texture feature extraction

[0035] Using a pre-trained ResNet-50 as the backbone network, and taking pre-processed keyframes as input, 2048-dimensional depth visual features are extracted from each frame, denoted as... ,in Analysis using a local texture discriminator It focuses on detecting blurred edges, unnatural texture transitions, and abnormal skin color distribution, and outputs low-level detection confidence scores. The mathematical expression is:

[0036]

[0037] in The average pooling result is the feature of K frames, the Dropout probability is set to 0.5, and Linear is a fully connected layer.

[0038] 2.2) Mid-level timing consistency modeling

[0039] Build a Transformer-Mamba hybrid architecture to handle different tasks. To capture temporal features: The Transformer module employs a 4-layer encoder, each layer containing one multi-head self-attention layer (8 heads, 256 attention dimensions) and one feedforward network (1024 hidden layer dimensions) to capture long-range inter-frame dependencies and output features. ; The Mamba module employs a selective state-space architecture, setting the dimension D=2048, the expansion factor E=2, and the convolution kernel size=4, achieving linear complexity. Model local temporal patterns and output features ; through dynamic weights , ( , )right and Weighted fusion is performed to obtain the mid-level time series features. Output the confidence level of the middle layer detection. The mathematical expression is:

[0040]

[0041]

[0042]

[0043] GRU stands for Gated Recurrent Unit, used for temporal feature aggregation.

[0044] 2.3) High-level semantic cross-modal analysis, integrating visual, audio, and textual multimodal information to construct a cross-modal consistency detection mechanism: Visual-Text Alignment: A pre-trained XCLIP model is used. Inputting keyframes and video title / description text T, the cosine similarity between visual and textual features is calculated. ; Audio-video synchronization: Using the AVHubert model, the input video audio track A and the lip movement region of keyframes are analyzed to determine the temporal synchronization between lip movements and audio, and the similarity score is output. ; Semantic consistency: Using the CLIP model, input keyframes and text T to calculate cross-modal semantic matching degree. The three similarities are concatenated into cross-modal features. Output high-level detection confidence through semantic classifier The mathematical expression is:

[0045]

[0046]

[0047]

[0048] Step 3) Transformer-Mamba Fusion and Adaptive Weighted Decision

[0049] 3.1) Transformer-Mamba Fusion Module

[0050] Mid-level time series characteristics Further optimization: The model is split into local windows (window size = 4). Within each window, Transformer self-attention is used to capture local dependencies. Features are transferred between windows using the Mamba state-space model, achieving "refined local modeling + efficient global transfer", and outputting fused temporal features. The time complexity is optimized to ; 3.2) Adaptive Weighted Decision Module Construct a learnable weight vector w=[ , , ],in , , The weights corresponding to low-level, mid-level, and high-level features respectively satisfy the following conditions: + + =1; The detection confidence scores of the three levels are fused using self-weighted logistic regression to calculate the final detection score. The mathematical expression is:

[0051] For trainable parameters,

[0052] ) Among them, the weight w is adaptively adjusted during the training process: when there are obvious visual forgeries in the video... When the time sequence increases, it becomes abnormal. Increase, semantic inconsistency Increase.

[0053] Step 4) Output of detection results and threshold determination

[0054] 4.1) Set the detection threshold (default It can be adjusted according to the application scenario: judicial scenario Regular content review ); 4.2) If ≥ Determine if the video is a deepfake, and output "deepfake" result and confidence level. ;like < Determine if the video is real, output "Real" result and confidence level 1- ; 4.3) Output auxiliary detection information: including forged regions detected at the lower level (such as blurred edge coordinates), intermediate-level temporal anomaly frame indexes, and higher-level cross-modal inconsistency types (audio-visual asynchrony / semantic contradictions), to improve the interpretability of detection results.

[0055] Step 5) Model Training and Optimization

[0056] 5.1) Constructing the training dataset: Integrating the UCF101 real video dataset (13,320 videos), the Chinese Academy of Sciences AI-generated video dataset (8,500 AI-generated videos), and the FF++ deepfake dataset (4,200 fake videos), a total of 26,020 video samples were collected and divided into training and validation sets in an 8:2 ratio; 5.2) The AdamW optimizer was used to train the model. The initial learning rate was 1e-3, which decayed to 1 / 10 of the original rate every 5 rounds. The weight decay coefficient was 1e-4, the batch size was 8, and the number of training rounds was 30. 5.3) Employing the cross-entropy loss function (where y=1 represents a fake video and y=0 represents a real video), combined with an early stopping mechanism (if the validation set loss does not decrease for 3 consecutive rounds, training is stopped) to prevent the model from overfitting.

[0057] Example 1: Content review scenario on a short video platform

[0058] Input: A 15-second video uploaded by a user on a short video platform (MP4 format, 1080p resolution).

[0059] 1) Extract 32 keyframes, preprocess them, and then perform RG13 data augmentation; 2) Low layer: ResNet-50 extracts 2048-dimensional features and detects blurred facial edges in the video. =0.72); Mid-layer: After Transformer-Mamba fusion, a temporal discontinuity in facial expressions was found in frames 8-12. =0.81); High-level: AVHubert detected audio and lip movement asynchrony ( =0.35), CLIP found a semantic contradiction between the video content and the title "Real Life Record" ( ). =0.42), =0.78; 3) Adaptive weights w=[0.32, 0.35, 0.33], ultimately... =0.77; 4) Settings =0.5, ≥ The video was identified as a "fake video" and output a confidence level of 0.77 and abnormal information (blurred edge area, abnormal time sequence frame index, audio and video desynchronization).

[0060] Example 2: Video Verification Scenario for Judicial Evidence

[0061] Input: 5 minutes of surveillance video from a legal case (AVI format, 720p resolution)

[0062] 1) Extract 64 keyframes and disable data augmentation during preprocessing (to ensure the authenticity of the evidence).

[0063] 2) Lower layers: ResNet-50 did not detect any obvious visual abnormalities ( =0.21)

[0064] Middle layer: Transformer-Mamba modeling reveals continuous inter-frame illumination changes ( =0.18); Senior management: The XCLIP and CLIP verification video content is consistent with the case description. =0.92, =0.89), AVHubert detects audio and video synchronization ( =0.95), =0.15; 3) Adaptive weights w=[0.28, 0.29, 0.43], ultimately... =0.17; 4) Settings =0.8, < It was determined to be a "real video" with a confidence level of 0.83, and was used as the basis for judicial evidence acceptance.

Claims

1. An AI-synthesized video detection method based on multi-level feature fusion, characterized in that, The process includes an offline model training phase and an online detection phase, with the following specific steps: The online detection phase includes steps 1) to 4). Step 1) Video preprocessing and key frame extraction: Input the video V to be detected, preprocess each key frame, and use the RG13 data augmentation strategy to augment the key frames; Step 2) Three-level feature extraction: Extract low-level visual texture features and output low-level detection confidence, then perform mid-level temporal consistency modeling, and finally perform high-level semantic cross-modal analysis; Step 3) Transformer-Mamba Fusion and Adaptive Weighted Decision: This includes the Transformer-Mamba fusion module and the adaptive weighted decision module; Step 3.1) Transformer-Mamba Fusion Module For the mid-level temporal features obtained in step 2) Further optimization: The feature is split into local windows, and a Transformer self-attention model is used to capture local dependencies within each window. Features are transferred between windows using a Mamba state-space model, and the output is a fused temporal feature. The time complexity is optimized to ; Step 3.2) Adaptive Weighted Decision Module Construct a learnable weight vector w=[ , , ],in ,、 , The weights corresponding to low-level, mid-level, and high-level features respectively satisfy the following conditions: + + =1; The detection confidence scores of the three levels are fused using self-weighted logistic regression to calculate the final detection score. The mathematical expression is: For trainable parameters, ) Among them, the weight w is adaptively adjusted during the training process: when visual forgery traces exist in the video... When the time sequence increases, it becomes abnormal. Increase, semantic inconsistency Increase; Step 4) Detection result output and threshold judgment: Compare the results with the set detection threshold τ to determine whether AI-synthesized video exists; The offline model training phase includes step 5). Step 5) Model Training and Optimization: Train and optimize the model parameters and adaptive weight vector from Steps 2-3 using the labeled dataset. The specific method is as follows: Step 5.1) Construct the training dataset: Integrate the UCF101 real video dataset, the Chinese Academy of Sciences AI-generated video dataset, and the FF++ deepfake video dataset, totaling 26,020 video samples, and divide them into training and validation sets in an 8:2 ratio; Step 5.2) The AdamW optimizer is used to train the model with an initial learning rate of 1e-3, which decays to 1 / 10 of the original rate every 5 rounds. The weight decay coefficient is 1e-4, the batch size is 8, and the number of training rounds is 30. Step 5.3) Use the cross-entropy loss function Loss = -y × log ( )-(1-y)×log (1- ), where y=1 represents a fake video and y=0 represents a real video. Combined with an early stopping mechanism, i.e., if the validation set loss does not decrease for 3 consecutive rounds, training is stopped to prevent the model from overfitting; Through training in step 5, the optimized model parameters are obtained, including the ResNet-50 fine-tuning parameters, the Transformer-Mamba fusion module parameters, and the initial trainable parameters of the adaptive weight vector w, which are then applied to the online detection stage.

2. The AI-synthesized video detection method based on multi-level feature fusion according to claim 1, characterized in that, The specific method in step 1) is as follows: Step 1.1) Input the video V to be detected, and extract K key frames using a uniform sampling strategy, denoted as... The sampling interval is dynamically adjusted according to the video duration. When the duration is ≤10s, K=16, and when the duration is >10s, K=32, thereby covering the complete temporal information of the video. Step 1.2) Preprocess each keyframe: Normalize the image size to 224×224 using OpenCV, convert it to RGB color space, perform mean subtraction and standard deviation normalization to eliminate the influence of lighting and resolution differences on feature extraction; where mean = [123.68, 116.779, 103.939], standard deviation = [58.393, 57.12, 57.375]; Step 1.3) Use the RG13 data augmentation strategy to augment the keyframes, including random horizontal flipping, brightness adjustment, contrast adjustment, and Gaussian blur.

3. The AI-synthesized video detection method based on multi-level feature fusion according to claim 1, characterized in that, The specific method in step 2) is as follows: Step 2.1) Low-level visual texture feature extraction Using a pre-trained ResNet-50 as the backbone network, and taking pre-processed keyframes as input, 2048-dimensional depth visual features are extracted from each frame, denoted as... ,in Analysis using a local texture discriminator It detects blurred edges, unnatural texture transitions, and abnormal skin color distribution, and outputs low-level detection confidence scores. The mathematical expression is: in The average pooling result of K-frame features is used, the Dropout probability is set to 0.5, and Linear is a fully connected layer; Step 2.2) Mid-level timing consistency modeling Build a Transformer-Mamba hybrid architecture to handle the Transformer module and the Mamba module separately. To capture temporal features: The Transformer module employs a 4-layer encoder, with each layer containing a multi-head self-attention layer and a feedforward network, to capture long-range inter-frame dependencies and output features. ; The Mamba module employs a selective state-space architecture, setting the dimension D=2048, the expansion factor E=2, and the convolution kernel size=4, achieving linear complexity. Model local temporal patterns and output features ; through dynamic weights , right and Weighted fusion is performed, in which, , ; Obtain the mid-level temporal features Output the confidence level of the middle layer detection. The mathematical expression is: GRU stands for Gated Recurrent Unit, used for temporal feature aggregation; Step 2.3) High-level semantic cross-modal analysis Integrating visual, audio, and textual multimodal information, a cross-modal consistency detection mechanism is constructed: Visual-Text Alignment: A pre-trained XCLIP model is used. Inputting keyframes and video title / description text T, the cosine similarity between visual and textual features is calculated. ; Audio-video synchronization: Using the AVHubert model, the input video audio track A and the lip movement region of keyframes are analyzed to determine the temporal synchronization between lip movements and audio, and the similarity score is output. ; Semantic consistency: Using the CLIP model, input keyframes and text T to calculate cross-modal semantic matching degree. The three similarities are concatenated into cross-modal features. Output high-level detection confidence through semantic classifier The mathematical expression is: 。 4. The AI-synthesized video detection method based on multi-level feature fusion according to claim 1, characterized in that, The specific method in step 4) is as follows: Step 4.1) Set the detection threshold ; Step 4.2) If ≥ Determine if the video is a deepfake, and output "deepfake" result and confidence level. ;like < Determine if the video is real, output "Real" result and confidence level 1- ; Step 4.3) Output auxiliary detection information: including forged regions detected at lower layers, indexes of temporal anomalies at middle layers, and cross-modal inconsistency types at higher layers.

Images