AI generated video detection method based on cross-modal space-time artifacts and related device
By employing a cross-modal spatiotemporal artifact detection method, this approach utilizes the visual and textual features of video frames to capture the spatiotemporal artifact differences between AI-generated videos and real videos in video-text cross-modal alignment. This solves the problem of decreased detection performance in existing methods and achieves highly accurate and robust AI-generated video detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-18
- Publication Date
- 2026-04-03
AI Technical Summary
Existing AI-generated video detection methods mainly rely on temporal artifacts within the visual modality, making it difficult to capture subtle artifacts in high-fidelity generated videos. Furthermore, their detection performance declines rapidly with advancements in generation technology, especially in videos with high visual coherence.
A cross-modal spatiotemporal artifact detection method is adopted. By acquiring the visual and text features of video frames, the method performs temporal modeling and feature encoding using cross-modal feature sequences and similarity sequences. The detection results are then combined with the output of the fusion classification layer to explore the spatiotemporal artifact differences in video-text cross-modal alignment.
It significantly improves the accuracy and robustness of AI-generated video detection, maintains excellent detection performance when facing unknown generative models, and demonstrates high average precision and accuracy on large-scale benchmark datasets.
Smart Images
Figure CN121786569A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of video detection and relates to an AI-generated video detection method and related apparatus based on cross-modal spatiotemporal artifacts. Background Technology
[0002] In recent years, generative artificial intelligence (AI) technology has made groundbreaking progress, particularly text-to-video and image-to-video generation models such as Sora, Gen-2, and Pika. These models are now capable of synthesizing visually highly realistic video content. While these AI-generated videos have shown great potential in film and television production and content creation, they have also been maliciously used to create and spread misinformation, posing a threat to social trust and public order. Therefore, the technology for accurately detecting AI-generated videos has become crucial.
[0003] Existing AI-generated video detection methods primarily rely on analyzing the spatiotemporal inconsistencies within the video content itself. For example, some methods reveal temporal discontinuities in synthesized videos by analyzing inter-frame continuity; others employ a dual-branch architecture, utilizing both RGB appearance information and optical flow dynamics for spatiotemporal discrimination; still others introduce a two-stream architecture to simultaneously model spatial texture and motion cues to capture spatiotemporal anomalies. Recently, frameworks have incorporated structured state-space models to learn local spatiotemporal dependencies.
[0004] However, these existing detection methods remain confined to the visual modality, focusing only on temporal artifacts within the modality and solely on mining temporal or spatial artifacts within the video data itself (RGB pixel sequences and derived optical flow). With the rapid iteration of video generation model technology, the visual realism and spatiotemporal coherence of generated videos are improving dramatically, making artifacts within the visual modality increasingly subtle and difficult to detect. Relying solely on visual analysis results in a significant performance ceiling and makes these methods highly susceptible to becoming obsolete as generation technologies advance. Furthermore, for AI-generated videos that have achieved high visual coherence without significant spatial distortions or temporal jumps, the detection performance of existing AI-generated video detection methods is significantly reduced. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide an AI-generated video detection method and related apparatus based on cross-modal spatiotemporal artifacts.
[0006] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, this invention provides an AI-generated video detection method based on cross-modal spatiotemporal artifacts, comprising: acquiring several consecutively sampled video frames from a video to be detected; acquiring text descriptions of each video frame, and extracting visual features and textual features of the text descriptions of each video frame to obtain visual feature vectors and textual feature vectors of each video frame; concatenating the visual feature vectors and textual feature vectors of each video frame to obtain a cross-modal feature sequence, and obtaining the similarity between the visual feature vectors and textual feature vectors of each video frame to obtain a cross-modal similarity sequence; and performing detection based on the cross-modal feature sequence and the cross-modal similarity sequence. The system uses a similarity sequence to call a pre-trained AI-generated video detection model to obtain the detection results of the video to be detected. The AI-generated video detection model includes a temporal modeling layer, a feature encoding layer, and a fusion classification layer. The temporal modeling layer is used to perform temporal modeling on the cross-modal similarity sequence to obtain a cross-modal temporal similarity representation. The feature encoding layer is used to perform feature encoding on the cross-modal feature sequence to obtain a cross-modal spatiotemporal representation. The fusion classification layer is used to concatenate the cross-modal temporal similarity representation and the cross-modal spatiotemporal representation to obtain a fusion feature, and output the detection results of the video to be detected based on the fusion feature.
[0007] Optionally, obtaining the text description of each video frame includes: inputting each video frame into a preset image description generation model to obtain the text description of each video frame; wherein, the image description generation model is a guided language-image pre-trained BLIP model.
[0008] Optionally, the extraction of visual features and textual features of text descriptions for each video frame includes: extracting visual features of each video frame using a visual encoder employing a contrastive language-image pre-trained CLIP model; and extracting textual features of text descriptions for each video frame using a text encoder employing a contrastive language-image pre-trained CLIP model.
[0009] Optionally, obtaining the similarity between the visual feature vectors and text feature vectors of each video frame includes obtaining the cosine similarity between the visual feature vectors and text feature vectors of each video frame.
[0010] Optionally, the temporal modeling layer is constructed using a single-layer gated recurrent unit (GRU) network; the feature encoding layer is constructed using a two-layer visual Transformer encoder; and the fusion classification layer includes a feature splicing layer, a fully connected layer, a normalization layer, and an output layer connected in sequence.
[0011] Optionally, the pre-trained AI-generated video detection model is pre-trained using a binary cross-entropy loss function, and the ratio of real videos to AI-generated videos in the training and test data during the pre-training process is 1:1.
[0012] Optionally, obtaining several consecutively sampled video frames in the video to be detected includes: randomly sampling several consecutively sampled video frames in the video to be detected, and uniformly adjusting the resolution of each video frame to a preset resolution and performing normalization processing.
[0013] In a second aspect, this invention provides an AI-generated video detection system based on cross-modal spatiotemporal artifacts, comprising: a video frame sampling module for acquiring several consecutively sampled video frames from a video to be detected; a feature extraction module for acquiring text descriptions of each video frame and extracting visual features and text features of the text descriptions of each video frame to obtain visual feature vectors and text feature vectors of each video frame; and a sequence construction module for concatenating the visual feature vectors and text feature vectors of each video frame to obtain a cross-modal feature sequence, and acquiring the similarity between the visual feature vectors and text feature vectors of each video frame to obtain a cross-modal similarity sequence; video The detection module is used to call a pre-trained AI-generated video detection model based on cross-modal feature sequences and cross-modal similarity sequences to obtain the detection results of the video to be detected. The AI-generated video detection model includes a temporal modeling layer, a feature encoding layer, and a fusion classification layer. The temporal modeling layer is used to perform temporal modeling on the cross-modal similarity sequences to obtain a cross-modal temporal similarity representation. The feature encoding layer is used to perform feature encoding on the cross-modal feature sequences to obtain a cross-modal spatiotemporal representation. The fusion classification layer is used to concatenate the cross-modal temporal similarity representation and the cross-modal spatiotemporal representation to obtain a fused feature, and output the detection results of the video to be detected based on the fused feature.
[0014] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the AI-generated video detection method based on cross-modal spatiotemporal artifacts described above.
[0015] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the AI-generated video detection method based on cross-modal spatiotemporal artifacts described above.
[0016] Compared with the prior art, the present invention has the following beneficial effects: This invention presents an AI-generated video detection method based on cross-modal spatiotemporal artifacts. It generates cross-modal feature sequences and cross-modal similarity sequences. A temporal modeling layer is used to perform temporal modeling on the cross-modal feature sequences to obtain a cross-modal temporal similarity representation. A feature encoding layer is then used to encode the cross-modal feature sequences to obtain a cross-modal spatiotemporal representation. Finally, a fusion classification layer concatenates the cross-modal temporal similarity representation and the cross-modal spatiotemporal representation to obtain a fused feature. The detection result of the video to be detected is output based on the fused feature. This method systematically mines and utilizes, for the first time, the inherent artifact differences between AI-generated videos and real videos in the spatiotemporal dimension of video-text cross-modal alignment. It overcomes the limitations of traditional methods that rely solely on single-modal (visual) internal features, effectively improving the accuracy of AI-generated video detection. Furthermore, by comprehensively modeling temporal stability and fine-grained spatial inconsistencies, it can capture deep-seated generation traces common to different AI video generation models, maintaining excellent detection performance even when facing unknown generation models and exhibiting strong generalization ability. Furthermore, experiments on large-scale benchmark datasets demonstrate that the method of this invention significantly outperforms existing methods in both average precision and accuracy, and exhibits good robustness to complex variations in video content. Attached Figure Description
[0017] Figure 1 This is a flowchart of an AI-generated video detection method based on cross-modal spatiotemporal artifacts according to an embodiment of the present invention.
[0018] Figure 2 This is a schematic diagram of the AI-generated video detection method based on cross-modal spatiotemporal artifacts in an embodiment of the present invention.
[0019] Figure 3 This is a block diagram of the AI-generated video detection system based on cross-modal spatiotemporal artifacts according to an embodiment of the present invention. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0022] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, an AI-generated video detection method based on cross-modal spatiotemporal artifacts is provided, which helps content platforms or regulatory agencies to effectively identify and filter AI-generated videos.
[0023] Specifically, the AI-generated video detection method based on cross-modal spatiotemporal artifacts of this invention includes the following steps: S1: Obtain several consecutively sampled video frames from the video to be detected.
[0024] S2: Obtain the text description of each video frame, and extract the visual features and text features of the text description of each video frame to obtain the visual feature vector and text feature vector of each video frame.
[0025] S3: Concatenate the visual feature vectors and text feature vectors of each video frame to obtain a cross-modal feature sequence, and obtain the similarity between the visual feature vectors and text feature vectors of each video frame to obtain a cross-modal similarity sequence.
[0026] S4: Based on the cross-modal feature sequence and cross-modal similarity sequence, call the pre-trained AI-generated video detection model to obtain the detection results of the video to be detected.
[0027] The AI-generated video detection model includes a temporal modeling layer, a feature encoding layer, and a fusion classification layer. The temporal modeling layer is used to perform temporal modeling on cross-modal similarity sequences to obtain cross-modal temporal similarity representations. The feature encoding layer is used to perform feature encoding on cross-modal feature sequences to obtain cross-modal spatiotemporal representations. The fusion classification layer is used to concatenate the cross-modal temporal similarity representations and cross-modal spatiotemporal representations to obtain fusion features, and output the detection results of the video to be detected based on the fusion features.
[0028] As described in the background section, existing AI-generated video detection methods are limited to the visual modality and only focus on temporal artifacts within that modality. However, the inventors discovered a crucial clue: cross-modal inconsistencies exist in AI-generated videos. Specifically, real videos, due to their complex and dynamically changing scenes, often exhibit significant fluctuations in the cross-modal correlation between video frames and automatically generated text descriptions. These cross-modal alignment differences and temporal fluctuation artifacts, caused by inherent differences in the generation process, have not been explicitly modeled and utilized by existing detection methods. This limits the robustness and generalization ability of current detection methods when faced with high-fidelity generated videos.
[0029] In contrast, this invention presents an AI-generated video detection method based on cross-modal spatiotemporal artifacts. It generates cross-modal feature sequences and cross-modal similarity sequences, performs temporal modeling on the cross-modal feature sequences using a temporal modeling layer to obtain a cross-modal temporal similarity representation, and then encodes the cross-modal feature sequences using a feature encoding layer to obtain a cross-modal spatiotemporal representation. Finally, a fusion classification layer concatenates the cross-modal temporal similarity representation and the cross-modal spatiotemporal representation to obtain a fused feature, and outputs the detection result of the video to be detected based on the fused feature. This method systematically mines and utilizes, for the first time, the inherent artifact differences between AI-generated videos and real videos in the spatiotemporal dimension of video-text cross-modal alignment. It overcomes the limitations of traditional methods that rely solely on single-modal (visual) internal features, effectively improving the accuracy of AI-generated video detection. Furthermore, by comprehensively modeling temporal stability and fine-grained spatial inconsistencies, it can capture deep-seated generation traces common to different AI video generation models, maintaining excellent detection performance even when facing unknown generation models, and exhibiting strong generalization ability. Furthermore, experiments on large-scale benchmark datasets demonstrate that the method of this invention significantly outperforms existing methods in both average precision and accuracy, and exhibits good robustness to complex variations in video content.
[0030] In one possible implementation, obtaining the text description of each video frame includes: inputting each video frame into a preset image description generation model to obtain the text description of each video frame; wherein, the image description generation model is a Guided Language-Image Pre-trained BLIP model.
[0031] Interpretive, the BLIP (Guided Language-Image Pre-training) model is an advanced vision-language model that pre-trains on large-scale noisy image-text pairs through a bootstrapping learning mechanism, enabling simultaneous understanding and generative tasks. In this implementation, the BLIP model generates high-quality, semantically accurate text descriptions for each video frame, reliably converting visual content into text features suitable for alignment analysis by the CLIP model. This provides an accurate semantic foundation for subsequent cross-modal consistency detection, significantly improving the system's understanding depth and detection reliability of video content.
[0032] For example, the BLIP model is specifically Salesforce / blip-image-captioning-base.
[0033] In one possible implementation, the extraction of visual features and textual features of text descriptions for each video frame includes: extracting visual features of each video frame using a visual encoder employing a contrastive language-image pre-trained CLIP model; and extracting textual features of text descriptions for each video frame using a text encoder employing a contrastive language-image pre-trained CLIP model.
[0034] Interpretive, the CLIP (Contrastive Language-Image Pre-trained) model maps visual and linguistic information to the same semantic space through contrastive learning training on massive image-text pairs. In this implementation, CLIP's visual encoder extracts visual features from each video frame, while the text encoder extracts text features from the corresponding text descriptions. Since both features are naturally aligned to the same semantic space, cross-modal similarity can be directly calculated, thus providing a reliable and interpretable cross-modal alignment benchmark for subsequent temporal modeling and spatiotemporal analysis.
[0035] For example, the CLIP model is specifically microsoft / clip-vit-base-patch32.
[0036] In one possible implementation, obtaining the similarity between the visual feature vectors and text feature vectors of each video frame includes obtaining the cosine similarity between the visual feature vectors and text feature vectors of each video frame.
[0037] Interpretive, cosine similarity measures directional consistency by calculating the cosine of the angle between two feature vectors in space. In this implementation, cosine similarity is calculated between the visual feature vectors and text feature vectors of each video frame, effectively quantifying the semantic matching degree between visual content and text description. Since the CLIP model maps features from both modalities to the same semantic space, cosine similarity can stably and interpretably reflect the cross-modal alignment quality of each frame, and the resulting temporal sequence can be directly used to capture potential cross-modal temporal inconsistencies in AI-generated videos.
[0038] In one possible implementation, the temporal modeling layer is constructed using a single-layer gated recurrent unit (GRU) network; the feature encoding layer is constructed using a two-layer visual Transformer encoder; and the fusion classification layer includes a feature splicing layer, a fully connected layer, a normalization layer, and an output layer connected in sequence.
[0039] The interpretive temporal modeling layer is constructed using a single-layer gated recurrent unit (GRU) network. Its gating mechanism effectively captures temporal dependencies and dynamic evolution patterns in cross-modal similarity sequences, learning the periodic fluctuations or anomalous mutations that may exist in AI-generated videos. The feature encoding layer is constructed using a two-layer visual Transformer (ViT) encoder. Through its multi-head self-attention mechanism, it simultaneously models the fine-grained alignment relationship between intra-frame visual and textual features, as well as the spatiotemporal dependencies of inter-frame cross-modal features, thereby extracting discriminative features representing local inconsistencies and global discontinuities. The fusion classification layer is a cascaded structure containing a feature concatenation layer, a fully connected layer, a normalization layer, and an output layer. First, the temporal representation output by GRU and the spatiotemporal representation output by ViT are concatenated to form a fusion feature. Then, a nonlinear transformation and feature compression are performed through a fully connected layer. After the numerical distribution is stabilized by a normalization layer, the output layer generates binary classification probabilities to complete the judgment of AI-generated videos.
[0040] In one possible implementation, the pre-trained AI-generated video detection model is pre-trained using a binary cross-entropy loss function, and the ratio of real videos to AI-generated videos in the training and test data during the pre-training process is 1:1.
[0041] Interpretatively, a binary cross-entropy loss function is used for optimization during the model pre-training phase. This loss function provides a clear direction for optimization by comparing the difference between the model's predicted probabilities and the true labels, ensuring that the model effectively learns the decision boundary that distinguishes between real videos and AI-generated videos. Simultaneously, during pre-training, the ratio of real videos to AI-generated videos in the training and test sets is controlled at 1:1. This balancing strategy aims to avoid model bias caused by skewed data distribution, forcing the model to learn the features of both classes of samples equally, thereby improving the detection's generalization ability and robustness, and preventing the model from overfitting to a particular class.
[0042] In one possible implementation, obtaining several consecutively sampled video frames in the video to be detected includes: randomly sampling several consecutively sampled video frames in the video to be detected, and uniformly adjusting the resolution of each video frame to a preset resolution and performing normalization processing.
[0043] For example, for each input video, eight consecutive video frames are randomly sampled, and the resolution of each video frame is uniformly adjusted to 224×224 pixels and normalized. This sampling strategy ensures both the integrity of temporal information and computational efficiency.
[0044] In one possible implementation, see Figure 2The AI-generated video detection method based on cross-modal spatiotemporal artifacts of this invention includes the following detailed steps: randomly sampling T consecutive video frames from the video to be detected to form a video frame sequence. The BLIP model is used for each video frame. Generate corresponding text descriptions ,in t represents the frame number; the visual encoder using the CLIP model extracts the visual features of each video frame. ,in The text encoder using the CLIP model extracts text features for each text description. , ; Calculate the cross-modal similarity of each video frame The calculation formula is: ; The cross-modal similarity sequence { , , ..., Inputting the data into a GRU network yields cross-modal temporal similarity representations. , The visual and textual features of each video frame are concatenated to obtain cross-modal features. , ; cross-modal feature sequences Input the Vision Transformer encoder to obtain the cross-modal spatiotemporal representation. Representing cross-modal temporal similarity Cross-modal spatiotemporal representation By splicing the data, the fused features are obtained. , The fused features are input into the fully connected layer, and the final classification result is obtained through the Softmax function. The calculation formula is as follows: =Softmax(W· +b), where W and b are learnable parameters. This indicates the probability that the video to be detected is an AI-generated video.
[0045] For example, the dimensions of visual and textual features are set to 512; the hidden layer dimension of the GRU network is set to 256, and the dropout rate is set to 0.1 to capture the temporal evolution patterns of cross-modal alignment; the visual Transformer encoder contains 2 encoding layers and 4 attention heads, with a hidden dimension of 256 and a feedforward network dimension of 512; training is performed using a binary cross-entropy loss function, defined as: =-[y·log( )+(1-y)·log(1- ]], where y=1 represents AI-generated video and y=0 represents real video; the optimizer uses Adam, and the initial learning rate is set to 1×10. -4 .
[0046] In one possible implementation, experiments on typical datasets GenVideo, EvalCrafter, VideoPhy, and VidProM demonstrate that the AI-generated video detection method based on cross-modal spatiotemporal artifacts is applicable to video detection tasks with different generation models and in different scenarios. Comparative experiments with various single-modal detection methods and traditional spatiotemporal detection methods show that the present invention can identify AI-generated videos with higher accuracy and exhibits excellent generalization performance when facing unknown generation models. Furthermore, ablation experiments further verify the effectiveness and complementarity of the two branches: cross-modal temporal analysis and cross-modal spatiotemporal analysis.
[0047] In summary, this invention, based on cross-modal spatiotemporal artifact detection of AI-generated videos, essentially takes a cross-modal analysis perspective. By uncovering the fundamental differences between AI-generated videos and real videos in terms of spatiotemporal stability of video-text alignment, it achieves accurate and robust detection. Specifically, by analyzing the fluctuation patterns of cross-modal temporal similarity, it can effectively capture the abnormal stability in AI-generated videos caused by the controlled generation process; by modeling cross-modal spatiotemporal representations, it can reveal fine-grained inconsistencies within and between video frames. Finally, based on the complementary fusion of the two representations, this invention can comprehensively capture the deep features of AI-generated videos and exhibits excellent generalization ability when facing various advanced generation models.
[0048] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.
[0049] See Figure 3 In another embodiment of the present invention, an AI-generated video detection system based on cross-modal spatiotemporal artifacts is provided, which can be used to implement the above-mentioned AI-generated video detection method based on cross-modal spatiotemporal artifacts. Specifically, the AI-generated video detection system based on cross-modal spatiotemporal artifacts includes a video frame sampling module, a feature extraction module, a sequence construction module, and a video detection module.
[0050] The system comprises the following modules: a video frame sampling module for acquiring several consecutively sampled video frames from the video to be detected; a feature extraction module for acquiring the text description of each video frame and extracting the visual features and text features of the text description of each video frame to obtain the visual feature vector and text feature vector of each video frame; a sequence construction module for concatenating the visual feature vector and text feature vector of each video frame to obtain a cross-modal feature sequence, and for obtaining the similarity between the visual feature vector and text feature vector of each video frame to obtain a cross-modal similarity sequence; and a video detection module for detecting video frames based on the cross-modal feature sequence and cross-modal similarity. The system uses a similarity sequence to call a pre-trained AI-generated video detection model to obtain the detection results of the video to be detected. The AI-generated video detection model includes a temporal modeling layer, a feature encoding layer, and a fusion classification layer. The temporal modeling layer is used to perform temporal modeling on the cross-modal similarity sequence to obtain a cross-modal temporal similarity representation. The feature encoding layer is used to perform feature encoding on the cross-modal feature sequence to obtain a cross-modal spatiotemporal representation. The fusion classification layer is used to concatenate the cross-modal temporal similarity representation and the cross-modal spatiotemporal representation to obtain a fusion feature, and output the detection results of the video to be detected based on the fusion feature.
[0051] In one possible implementation, obtaining the text description of each video frame includes: inputting each video frame into a preset image description generation model to obtain the text description of each video frame; wherein, the image description generation model is a Guided Language-Image Pre-trained BLIP model.
[0052] In one possible implementation, the extraction of visual features and textual features of text descriptions for each video frame includes: extracting visual features of each video frame using a visual encoder employing a contrastive language-image pre-trained CLIP model; and extracting textual features of text descriptions for each video frame using a text encoder employing a contrastive language-image pre-trained CLIP model.
[0053] In one possible implementation, obtaining the similarity between the visual feature vectors and text feature vectors of each video frame includes obtaining the cosine similarity between the visual feature vectors and text feature vectors of each video frame.
[0054] In one possible implementation, the temporal modeling layer is constructed using a single-layer gated recurrent unit (GRU) network; the feature encoding layer is constructed using a two-layer visual Transformer encoder; and the fusion classification layer includes a feature splicing layer, a fully connected layer, a normalization layer, and an output layer connected in sequence.
[0055] In one possible implementation, the pre-trained AI-generated video detection model is pre-trained using a binary cross-entropy loss function, and the ratio of real videos to AI-generated videos in the training and test data during the pre-training process is 1:1.
[0056] In one possible implementation, obtaining several consecutively sampled video frames in the video to be detected includes: randomly sampling several consecutively sampled video frames in the video to be detected, and uniformly adjusting the resolution of each video frame to a preset resolution and performing normalization processing.
[0057] All relevant content of each step involved in the aforementioned embodiments of the AI-generated video detection method based on cross-modal spatiotemporal artifacts can be referenced to the functional description of the corresponding functional module of the AI-generated video detection system based on cross-modal spatiotemporal artifacts in the embodiments of the present invention, and will not be repeated here.
[0058] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0059] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of an AI-generated video detection method based on cross-modal spatiotemporal artifacts.
[0060] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the AI-generated video detection method based on cross-modal spatiotemporal artifacts in the above embodiments.
[0061] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0062] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0063] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0064] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A method for detecting AI-generated videos based on cross-modal spatiotemporal artifacts, characterized in that, include: Acquire several consecutively sampled video frames from the video to be detected; Obtain the text description of each video frame, and extract the visual features and text features of the text description of each video frame to obtain the visual feature vector and text feature vector of each video frame. The visual feature vectors and text feature vectors of each video frame are concatenated to obtain a cross-modal feature sequence, and the similarity between the visual feature vectors and text feature vectors of each video frame is obtained to obtain a cross-modal similarity sequence. Based on cross-modal feature sequences and cross-modal similarity sequences, a pre-trained AI-generated video detection model is invoked to obtain the detection results of the video to be detected; The AI-generated video detection model includes a temporal modeling layer, a feature encoding layer, and a fusion classification layer. The temporal modeling layer is used to perform temporal modeling on cross-modal similarity sequences to obtain cross-modal temporal similarity representations, and the feature encoding layer is used to perform feature encoding on cross-modal feature sequences to obtain cross-modal spatiotemporal representations. The fusion classification layer is used to concatenate cross-modal temporal similarity representation and cross-modal spatiotemporal representation to obtain fusion features, and output the detection results of the video to be detected based on the fusion features.
2. The AI-generated video detection method based on cross-modal spatiotemporal artifacts according to claim 1, characterized in that, The text descriptions of each video frame obtained include: Each video frame is input into a preset image description generation model to obtain a text description of each video frame; the image description generation model is a guided language-image pre-trained BLIP model.
3. The AI-generated video detection method based on cross-modal spatiotemporal artifacts according to claim 1, characterized in that, The extraction of visual features and textual features from each video frame includes: A visual encoder using a contrastive language-image pre-trained CLIP model extracts visual features from each video frame; and a text encoder using a contrastive language-image pre-trained CLIP model extracts text features from the text descriptions of each video frame.
4. The AI-generated video detection method based on cross-modal spatiotemporal artifacts according to claim 1, characterized in that, The process of obtaining the similarity between the visual feature vectors and text feature vectors of each video frame includes: Obtain the cosine similarity between the visual feature vector and the text feature vector of each video frame.
5. The AI-generated video detection method based on cross-modal spatiotemporal artifacts according to claim 1, characterized in that, The temporal modeling layer is constructed using a single-layer gated recurrent unit (GRU) network. The feature encoding layer is constructed using a two-layer visual Transformer encoder. The fusion classification layer includes a feature splicing layer, a fully connected layer, a normalization layer, and an output layer connected in sequence.
6. The AI-generated video detection method based on cross-modal spatiotemporal artifacts according to claim 1, characterized in that, The pre-trained AI-generated video detection model during pre-training: The binary cross-entropy loss function is used for pre-training, and the ratio of real videos to AI-generated videos in the training and testing data during the pre-training process is 1:
1.
7. The AI-generated video detection method based on cross-modal spatiotemporal artifacts according to claim 1, characterized in that, The process of acquiring several consecutively sampled video frames from the video to be detected includes: Randomly sample several consecutive video frames from the video to be detected, and adjust the resolution of each video frame to a preset resolution and perform normalization processing.
8. An AI-generated video detection system based on cross-modal spatiotemporal artifacts, characterized in that, include: The video frame sampling module is used to acquire several consecutively sampled video frames from the video to be detected; The feature extraction module is used to obtain the text description of each video frame, and extract the visual features and text features of the text description of each video frame to obtain the visual feature vector and text feature vector of each video frame. The sequence construction module is used to concatenate the visual feature vectors and text feature vectors of each video frame to obtain a cross-modal feature sequence, and to obtain the similarity between the visual feature vectors and text feature vectors of each video frame to obtain a cross-modal similarity sequence. The video detection module is used to call a pre-trained AI-generated video detection model based on cross-modal feature sequences and cross-modal similarity sequences to obtain the detection results of the video to be detected; The AI-generated video detection model includes a temporal modeling layer, a feature encoding layer, and a fusion classification layer. The temporal modeling layer is used to perform temporal modeling on cross-modal similarity sequences to obtain cross-modal temporal similarity representations, and the feature encoding layer is used to perform feature encoding on cross-modal feature sequences to obtain cross-modal spatiotemporal representations. The fusion classification layer is used to concatenate cross-modal temporal similarity representation and cross-modal spatiotemporal representation to obtain fusion features, and output the detection results of the video to be detected based on the fusion features.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the AI-generated video detection method based on cross-modal spatiotemporal artifacts as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the AI-generated video detection method based on cross-modal spatiotemporal artifacts as described in any one of claims 1 to 7.