Spatial-temporal feature fusion AI generation video detection method based on ViT-ResNet double-branch collaborative modeling

Abstract

The invention relates to a spatiotemporal feature fusion AI generated video detection method based on ViT-ResNet double-branch collaborative modeling, and aims to realize effective detection of synthesized video contents of different generation models. According to the method, a double-flow neural network architecture is constructed, spatial features of video frame images and time sequence features of optical flow images are extracted respectively, collaborative modeling of spatial and temporal features is realized through a feature fusion module, and the fused spatial and temporal features are used for training a classifier to judge the authenticity of video content. The method mainly comprises the following steps: (1) providing an AI generated video detection method fusing spatio-temporal information; (2) a ViT-ResNet double-branch feature extraction and collaborative modeling mechanism is provided; and (3) providing a high-robustness classification decision method based on double-flow feature fusion.

Description

Technical Field

[0001] The present invention relates to the interdisciplinary field of artificial intelligence and digital multimedia security, and more particularly to an AI-generated video detection method based on spatiotemporal feature fusion of ViT-ResNet dual-branch collaborative modeling. Background Art

[0002] With the rapid development of generative artificial intelligence (AIGC) technology, the video synthesis capabilities of deep generative algorithms such as diffusion models and generative adversarial networks (GANs) have continued to improve. In recent years, advanced video generation models such as DynamicCrafter, SVD, and OpenSora have been able to generate highly realistic synthetic videos that are almost impossible for the human eye to distinguish. While such technologies have broad application prospects in film and television production, virtual reality, human-computer interaction, and other fields, they can also be abused by criminals to create false information, conduct online fraud, or violate personal privacy, posing a serious threat to public information security and personal privacy protection. Therefore, accurately detecting AI-generated video content has become a key research direction in the field of AI security.

[0003] For AI-generated video detection, the current mainstream detection method is mainly based on Convolutional Neural Network (CNN) to extract spatial features from static RGB frame images and train deep learning models for binary classification. Some studies have introduced temporal modeling modules (such as LSTM and 3D-CNN) to capture the temporal consistency characteristics between video frames. Although existing methods have achieved good detection results on specific datasets, the adaptability and detection stability of the models remain challenging when faced with synthetic videos from different sources and styles, especially when the performance is less than ideal when training samples are limited. In addition, some methods also suffer from high computational costs and low detection efficiency. For example, the paper "Chen, Haoxing, et al." Demamba: Ai-generated video detection on million-scale genvideo benchmark." arXiv preprint arXiv:2405.19707(2024)" proposes a million-scale AI-generated video dataset and uses it to train a detection model, which places a heavy burden on computing power. The paper "Liu, Qingyuan, et al." Turns Out I'm Not Real: Towards Robust Detection of AI-Generated Videos." arXiv preprint arXiv:2406.09601(2024)" proposes a detection method that uses RGB frames and diffusion reconstruction error (DIRE value) with a CNN+LSTM architecture for detection. This requires additional image reconstruction, which is not only time-consuming but also places higher demands on computing power, reducing its practicality. The paper "Ma, Long, et al." DeCoF: Generated Video Detection via Frame Consistency." arXiv e-prints (2024): arXiv-2402" proposes a method based on inter-frame consistency modeling. Although it can effectively capture temporal artifacts, it ignores the joint modeling with spatial texture and local structure, making it difficult to fully explore the multi-dimensional features of AI forgeries. Therefore, there is an urgent need for an AI-generated video recognition method that can integrate spatiotemporal information and enhance cross-domain detection performance.

[0004] To address these issues, this paper proposes a method for detecting AI-generated videos using spatiotemporal features, based on dual-branch collaborative modeling using ViT-ResNet. This method leverages the spatial local texture features of video frames with the temporal dynamics represented by inter-frame optical flow maps to detect AI-generated videos. This integration of spatiotemporal features improves the model's detection accuracy for different types of generated videos, demonstrating promising application prospects and potential for widespread adoption. Summary of the Invention

[0005] This paper proposes a method for detecting AI-generated videos based on the spatiotemporal feature fusion of ViT-ResNet dual-branch collaborative modeling, which is used to determine whether a given video is AI-generated. The method mainly includes the following contents:

[0006] (1) An AI-generated video detection method that integrates spatiotemporal information is proposed.

[0007] (2) A ViT-ResNet dual-branch feature extraction and collaborative modeling mechanism is proposed.

[0008] (3) A highly robust classification decision method based on dual-stream feature fusion is proposed.

[0009] The specific contents are as follows:

[0010] (1) A method for detecting AI-generated videos that integrates spatiotemporal information is proposed: The present invention proposes a method for detecting AI-generated videos that integrates spatial features (static texture information within a frame) and temporal features (dynamic change information between frames). A two-stream network structure is used to jointly model the spatial texture information in the video frame and the inter-frame motion change features, thereby achieving effective detection of AI-forged videos. The two-branch network is composed of two independent feature extraction networks, which process spatial domain and temporal domain information respectively, thereby deeply mining potential generation traces in the video. Subsequently, the extracted spatial features and temporal features will be fused in the feature dimension to form a unified spatiotemporal feature representation, and finally the classification module will use the fused spatiotemporal features to realize the authenticity judgment of the video. The method as a whole is composed of a two-branch feature extraction module, a feature fusion module and a classification module. The detection framework is as follows: Figure 1 shown.

[0011] (2) A ViT-ResNet dual-branch feature extraction and collaborative modeling mechanism is proposed: The present invention designs a dual-branch feature extraction and collaborative modeling mechanism, which uses ViT-B / 16 and ResNet50 to extract spatial features and temporal features from the original video frames and optical flow images respectively. Specifically, the spatial features are extracted from the original RGB frame images by the visual transformer ViT-B / 16, and a 768-dimensional spatial domain feature vector is generated for each RGB frame image. The temporal domain features are extracted from the inter-frame optical flow images, which are calculated by the RAFT (Recurrent All-Pairs Field Transforms) optical flow estimation algorithm. Specifically, an optical flow image is obtained by the RAFT algorithm for every two RGB frame images. If N RGB frame images are extracted from the original video, N-1 optical flow maps can be generated. The optical flow map can accurately capture the temporal change characteristics between frames. Subsequently, the optical flow map is input into the pre-trained ResNet50 model to extract features, and a 2048-dimensional temporal feature representation corresponding to each optical flow map is obtained. Then, the two features are directly spliced ​​in the feature dimension to form a 2816-dimensional joint feature vector. Compared with the traditional weighted fusion method, the present invention adopts a direct splicing method to fully preserve the original and complementary information of spatial features and temporal features, avoiding the loss of key forgery clues. The feature fusion process is expressed as:

[0012] F fused =Concat(F RGB ,F flow ) (1)

[0013] Among them, F RGB represents the spatial features extracted from the RGB frame image, F flow represents the temporal features extracted from the optical flow image, F fused This method effectively integrates the spatial features and temporal information in the video, improving the model's comprehensive perception of forgery clues.

[0014] (3) A highly robust classification decision method based on dual-stream feature fusion is proposed: After completing the fusion of spatial features and temporal features, the present invention uses a multilayer perceptron (MLP) as a classifier to distinguish the authenticity of videos. First, the 2816-dimensional joint feature vector is reduced to 512 dimensions through a fully connected layer (FullyConnectedLayer) to compress redundant information and extract more discriminative feature representations; then the ReLU activation function is introduced to enhance the nonlinear expression ability; then the Dropout regularization technology is used to suppress the risk of model overfitting, and then the predicted category (real / fake) is output through the final linear layer. The classification decision process can be expressed as:

[0015] P(real)=Softmax(W2●ReLU(W1●F fused +b1)+b2) (2)

[0016] Among them, W1, W2 are the weight matrices of the fully connected layer, b1, b2 are bias terms, and F fused To further enhance the model's cross-domain generalization capabilities, this paper introduces feature normalization and a dropout strategy into the classifier, enabling the model to adapt to forged videos from different generative models and scenarios.

[0017] Compared with existing technologies, the above solution has at least the following significant advantages:

[0018] 1. The present invention provides a method for detecting AI-generated videos by fusing spatiotemporal features based on ViT-ResNet dual-branch collaborative modeling. By combining the temporal and spatial features of video frames, it can effectively capture subtle traces of forgery in AI-generated videos.

[0019] 2. This paper adopts ViT-B / 16 and ResNet50 networks to complement each other to enhance feature representation capabilities. Through direct feature splicing and fusion, it retains more original feature information and improves detection accuracy.

[0020] 3. The method of the present invention can achieve detection results that are on par with or even better than existing technologies, even when the amount of training data is much smaller than that of existing work (only 5,000 samples per category of true and false videos), reflecting higher data utilization efficiency and model generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the overall detection framework diagram of the present invention's "AI-generated video detection method based on spatiotemporal feature fusion and ViT-ResNet dual-branch collaborative modeling." DETAILED DESCRIPTION

[0022] In order to more clearly demonstrate the features and advantages of this patent, a detailed description of the embodiments is provided below. It should be understood that the following detailed description is intended to provide further explanation of this application by way of example only. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by those skilled in the art to which this application belongs.

[0023] The present invention relates to a method for detecting AI-generated videos by fusing spatiotemporal features based on ViT-ResNet dual-branch collaborative modeling. The method can effectively improve the generalized detection of different types of AI-generated videos, specifically, it can effectively detect video content generated by ZeroScope, I2VGen-XL, SVD, VideoCrafter, Pika, DynamiCrafter, SD, SEINE, Latte, OpenSora, Crafter, Gen2, Lavie, MoonValley, MorphStudio and Show_1 models, and has good application prospects.

[0024] Experimental environment and dataset preparation:

[0025] Development language: Python

[0026] Operating system environment: Windows 10

[0027] Main dependent libraries: PyTorch, OpenCV, RAFT optical flow extraction model, etc.

[0028] Dataset source: Genvideo large-scale generated video dataset (Chen, Haoxing, et al., "Demamba: AI-generated video detection on million-scale GenVideo benchmark." arXiv preprint arXiv:2405.19707(2024))

[0029] The specific steps are as follows:

[0030] Step 1: Data division method:

[0031] Training and validation sets: 5,000 samples of generated videos and 5,000 samples of real videos were randomly selected from the Genvideo dataset. The generated video samples came from models such as DynamicCrafter, I2VGEN, Latte, and OpenSora (500 samples per category). The real video samples totaled 5,000. The training and validation sets were split in an 8:2 ratio.

[0032] The test set is selected from the Genvideo-val subset and includes videos generated by 10 types of generation tools: Crafter, HotShot, Gen2, MorphStudio, ModelScope, Sora, MoonValley, Lavie, Show-1, and WildScrape. Gen-2, ModelScope, MoonValley, Sora, LaVie, and WildScrape generate videos based on the diffusion model or a diffusion model combined with the Transformer architecture. Show-1, MorphStudio, HotShot, and Crafter primarily use the diffusion model, with some versions incorporating GANs for local detail optimization. From these generated videos, 100 videos were selected from each category, for a total of 1,000 fake video samples. The number of real video samples is 1,000.

[0033] Step 2: Data Preprocessing. During the data preprocessing stage, RGB frame images and corresponding optical flow maps must be extracted from each video to form frame-flow pairs. To extract RGB frame images, the OpenCV library is used to read and process the video frame by frame. For generated videos, all frames are extracted. However, for real videos, given their typically long duration, a strategy of extracting every fifth frame is adopted to reduce data redundancy and computational burden. For optical flow images, optical flow calculation is first performed. The pretrained RAFT model is used to process the input RGB frame pairs. If N RGB frames are extracted from the original video, N-1 optical flow maps can be generated, with one optical flow image generated for every two RGB frames. The extracted optical flow maps are saved in JPG format to ensure the integrity of spatial and temporal information during the subsequent feature extraction process. After extraction, all optical flow images are stored in directories based on video source and label, and paired with the corresponding RGB frame images to create a unified data loading structure, facilitating efficient reading and training of the ViT-ResNet two-stream model.

[0034] Step 3: Feature Extraction. During the feature extraction phase, feature modeling is performed for both the RGB frame image and the optical flow image. The ViT-B / 16 network model is used to extract features from the RGB frame image, generating a 768-dimensional spatial domain feature vector for each RGB frame image. The ResNet50 residual network structure is used to extract features from the optical flow image, obtaining a 2048-dimensional temporal feature representation for each optical flow image. This step yields a spatial representation of the RGB frame image and a temporal motion representation of the optical flow image, which serve as input to the subsequent feature fusion module.

[0035] Step 4: Feature fusion and network training. During the feature fusion and training phase, the 768-dimensional spatial features extracted by ViT-B / 16 are fused with the 2048-dimensional temporal features extracted by ResNet50 to form a 2816-dimensional joint feature vector. The multi-layer perceptron (MLP) is then used to further learn the interaction between the fused features to enhance the model's comprehensive discrimination capabilities. During network training, cross-entropy loss is used as the optimization objective, and the Adam optimizer is used for parameter updates. The initial learning rate (lr) is set to 0.0001. The batch size (Batch Size) of training is set to 16, and the number of training rounds (Epochs) is set to 20. Through this series of steps, the model can effectively integrate spatiotemporal information and achieve accurate classification of real and fake videos.

[0036] Step 5: During the model testing phase, the model weights that perform best on the validation set are selected for inference testing. The test data is preprocessed according to the methods of steps 1 and 2 to extract the features of the RGB frame image and the optical flow image, forming a frame flow-optical flow pair input fusion model, and outputting the true and false classification results. During the evaluation process, the accuracy (ACC), area under the curve (AUC), and F1 score (F1-score) are used as performance indicators for the accuracy of the detection model. These indicators can comprehensively evaluate the model's generalization detection ability on unknown generative models and verify the effectiveness and reliability of this method in practical applications.

[0037] In summary, this paper presents a method for detecting AI-generated videos based on spatiotemporal feature fusion using ViT-ResNet dual-branch collaborative modeling. This method effectively extracts and fuses spatial and temporal features from videos, enhancing the model's ability to detect forgery cues and achieving accurate recognition of AI-generated videos. Using only 5,000 real videos and 5,000 fake videos for training, this method achieves stable and accurate cross-domain detection on the Genvideo-val subset for videos generated by mainstream generation tools including Crafter, Gen2, MorphStudio, Show-1, Lavie, and MoonValley, with an average detection accuracy of 97%. This result fully demonstrates the method's efficient learning and good cross-domain generalization performance under limited training data conditions. Furthermore, the model's lightweight design and low inference overhead make it suitable for applications with limited computing resources. Compared to traditional detection methods, this method offers significant advantages in detection accuracy, cross-domain adaptability, and resource efficiency, and has broad practical application prospects. It can serve multiple key areas, including social platform content review, news media authenticity verification, digital media copyright protection, and network security.

[0038] Those skilled in the art will understand that the scope of protection of the present invention is not limited to the specific embodiments described. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features. It should be noted that the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A video detection method based on spatiotemporal feature fusion AI generated by ViT-ResNet dual-branch collaborative modeling, characterized by Extracting the spatial features of video frame images and the temporal motion features of optical flow images, and detecting AI-generated videos by fusing spatiotemporal features, includes the following steps: (1) Data preprocessing: Extract RGB frame images and corresponding optical flow images from the original video, and construct input pairs of frame images and optical flow images; (2) Image feature extraction: Use the ViT-B / 16 network to extract the spatial features of the frame image, and use the ResNet50 network to extract the temporal features of the optical flow image; (3) Feature fusion and model training: Fuse the spatial features with the temporal features to generate a spatiotemporal feature vector, and train the classification model based on the fused features; (4) Model inference: Use the model that performs best on the validation set for inference, and input the trained model after extracting the spatiotemporal features of the video to be detected through the above process, and output a true or false recognition result.

2. The method for generating video detection based on AI using spatiotemporal feature fusion and ViT-ResNet dual-branch collaborative modeling according to claim 1 is characterized in that: During the data preprocessing and feature extraction steps, frame image extraction is implemented using the cv2.VideoCapture module of the OpenCV library. All frames are extracted from the generated video. For real-world videos, only one frame is extracted every five frames, or an adaptive adjustment is made. Optical flow image extraction uses the RAFT algorithm to estimate the optical flow of the extracted frames, generating an optical flow image for every two frames. The frame images and optical flow images are cropped or scaled to a uniform size of 224×224 pixels to ensure consistent input dimensions. The spatial feature extraction network uses ViT-B / 16, and the temporal feature extraction network uses ResNet50.

3. The video detection method generated by AI based on spatiotemporal feature fusion and ViT-ResNet dual-branch collaborative modeling according to claim 1 is characterized in that: The feature fusion method is: splicing the spatial feature vector and the temporal feature vector in the feature dimension, and performing weighted adjustment through the channel attention module to strengthen the key feature channel.

4. The method for generating video detection based on spatiotemporal feature fusion AI generated by ViT-ResNet dual-branch collaborative modeling according to claim 1 is characterized in that: The classifier includes at least one fully connected layer and one Softmax layer, wherein the output dimension of the fully connected layer is 2, corresponding to the real category and the forged category, and the Softmax layer outputs a normalized probability distribution.

5. The method for generating video detection based on spatiotemporal feature fusion AI generated by ViT-ResNet dual-branch collaborative modeling according to claim 1 is characterized in that: The model training and inference are both based on a deep learning framework that supports GPU acceleration, including but not limited to the PyTorch framework, and batch processing is used in the inference stage to accelerate the detection speed. It is not limited to a specific hardware or computing platform.

Images