Artificial intelligence generated video detection method and device based on space-time anomaly mining

Abstract

The invention provides an artificial intelligence generated video detection method and device based on space-time anomaly mining, and the method comprises the steps: inputting a training video into a Transform-based encoder, carrying out the feature extraction of the training video through the encoder, restoring the obtained video features through a decoder, obtaining a restored video, constructing a loss function according to the restored video and the training video, and carrying out the detection of the generated video through the loss function. Fine tuning the encoder and the decoder, and reserving the encoder after fine tuning as an intermediate encoder; a detector is added behind an intermediate encoder, the detector obtains a detection result of whether video detection belongs to an artificial intelligence generated video according to video features output by the intermediate encoder, a loss function is constructed according to the detection result and a generation source label of a training video, and the intermediate encoder and the detector are finely adjusted. Taking the fine-tuned intermediate encoder and detector as a video detection model; and inputting a to-be-detected video into the video detection model to obtain a video detection result whether the to-be-detected video belongs to the artificial intelligence generated video or not.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence security and image classification technology, and in particular relates to an artificial intelligence-generated video detection method, device, electronic device, computer-readable storage medium and computer program product based on spatiotemporal anomaly mining. Background Art

[0002] Currently, AI-generated video technology has evolved from early approaches based on image splicing and generative adversarial networks (GANs) to high-quality generation based on diffusion models. For example, models like OpenAI's Sora and Kuaishou's Keling employ a "stepwise noise optimization" mechanism, significantly improving the coherence and realism of generated videos, making them almost indistinguishable from the real thing and difficult for humans to accurately identify with the naked eye. Today, ordinary people can use these models to generate a wide variety of videos by simply inputting text, significantly lowering the barrier to entry. Coupled with the rapid spread of information online, using model-generated videos as fodder has become a key means of fabricating online disinformation and conducting illegal activities such as telecommunications fraud. This has accelerated information pollution on the internet and seriously threatened public safety. To combat these threats, there is an urgent need to develop detection technologies for AI-generated videos.

[0003] Previous detection technologies for generated videos were mainly divided into the following two categories: 1. Directly transferring early generation image detection technology, extracting certain frames from the video to be detected as images to detect the authenticity of the video. Traditional image generation technology focuses more on facial forgery, so this type of generated video detection technology is often more suitable for detecting generated videos containing faces, and its scope of application is relatively limited. 2. Detection models specially trained for GAN-generated videos, which mainly make judgments by identifying features such as abnormal pixel arrangements and discontinuous inter-frame movements. This type of method is effective when the quality of early generated videos is low. However, with the rapid advancement of generation technology and the transformation of generation mechanisms from adversarial networks to diffusion models, the differences in generation mechanisms have become more significant, making it difficult for detection methods targeting GANs to adapt to the detection of diffusion model-generated videos. Due to the above limitations, these methods have difficulty capturing video traces of advanced AI video generation models such as Sora.

[0004] Existing detection technologies only consider spatial rationality (spatial artifacts) and ignore the rationality of the video temporal continuity logic (temporal artifacts), resulting in low detection accuracy. Summary of the Invention

[0005] This invention aims to address the technical challenges of existing detection technologies, such as their limited applicability and low accuracy, in capturing artifacts in advanced AI-generated videos. By proposing a method for detecting AI-generated videos based on spatiotemporal anomaly mining, this method extracts artifact features from both the temporal and spatial domains of the video and fuses these features to model temporal plausibility information, thereby enabling the detection of more artifacts in the video.

[0006] Specifically, in view of the shortcomings of existing technologies, such as Figure 3 As shown, the present invention proposes an artificial intelligence-generated video detection method based on spatiotemporal anomaly mining, which includes:

[0007] In the initial step, multiple training videos with labeled generation source labels are obtained, where the generation source labels include AI-generated videos and non-AI-generated videos;

[0008] In the first fine-tuning step, the training video is input into a Transformer-based encoder. The encoder extracts features from the training video, and the decoder restores the obtained video features to obtain a restored video. A loss function is constructed based on the restored video and the training video. The encoder and decoder are fine-tuned, and the fine-tuned encoder is retained as an intermediate encoder.

[0009] In the second fine-tuning step, a detector is added after the intermediate encoder. The detector detects whether the video is AI-generated based on the video features output by the intermediate encoder. Based on the detection result and the generation source label of the training video, a loss function is constructed, and the intermediate encoder and the detector are fine-tuned. The fine-tuned intermediate encoder and the detector are used as the video detection model.

[0010] In the detection step, the video to be detected is input into the video detection model to obtain a video detection result of whether it is an artificial intelligence generated video.

[0011] The artificial intelligence-generated video detection method based on spatiotemporal anomaly mining, wherein the first fine-tuning step includes:

[0012] The input training video extracts multiple video frames according to the set time interval, and the time dimension initial features E0 of the multiple video frames are extracted through three-dimensional convolution as the input of the encoder; the encoder is composed of multiple spatiotemporal inconsistency learning modules in series, and the spatiotemporal inconsistency learning module includes a time feature mining module, a spatial feature mining module and a fusion module; in the spatiotemporal inconsistency learning module, the features are first obtained by TubeletEmbedding in the time feature mining module to obtain spatiotemporal tokens sorted by time, and each token is added with a position code embedding, and then the time feature E of the current t step is obtained through the Video Transformer model in the module. time,t :

[0013] E time,t =VideoTransformer(TubeletEmbedding(E t-1 )),

[0014] Among them, E t-1 is the output feature of the t-1th spatiotemporal inconsistency learning module;

[0015] In the spatial feature mining module, each frame of the video frames is divided into blocks to obtain the spatial token embedding of the blocks. The position encoding embedding Position Embedding is superimposed on each token embedding, and the superimposed embedding is embedded into the input video Transformer model and combined with E time，t Interactively obtain the t-step spatial feature E space,t :

[0016] E space,t =VisionTransformer(PatchEmbedding(E t-1 ))+E time,t .

[0017] Among them, PatchEmbedding() is a mapping embedding module used to obtain the superposition embedding representation of the spatial token embedding and position code embedding of each image; VisionTransformer() is a video Transformer model;

[0018] E space With E time Splicing, and then with the input E of this module t-1 Superposition, get the output E of step t t The video feature as the final output of the decoder is: t =Concat(E time，t , E space,t )+Et-1 .

[0019] The artificial intelligence-generated video detection method based on spatiotemporal anomaly mining, wherein the second fine-tuning step includes: adding a bypass branch to the attention layer of the encoder, and only performing gradient updates on the parameters of the bypass branch during fine-tuning.

[0020] like Figure 4 As shown, the present invention also proposes an artificial intelligence-generated video detection device based on spatiotemporal anomaly mining, which includes:

[0021] The initial module obtains multiple training videos that have been annotated with generation source labels, where the generation source labels include AI-generated videos and non-AI-generated videos;

[0022] The first fine-tuning module inputs the training video into a Transformer-based encoder. The encoder extracts features from the training video, and the decoder restores the obtained video features to obtain a restored video. A loss function is constructed based on the restored video and the training video. The encoder and decoder are fine-tuned, and the fine-tuned encoder is retained as an intermediate encoder.

[0023] The second fine-tuning module adds a detector after the intermediate encoder. The detector obtains a detection result of whether the video is AI-generated based on the video features output by the intermediate encoder. Based on the detection result and the generation source label of the training video, a loss function is constructed, and the intermediate encoder and the detector are fine-tuned. The fine-tuned intermediate encoder and the detector are used as the video detection model;

[0024] The detection module inputs the video to be detected into the video detection model to obtain a video detection result of whether it is an artificial intelligence generated video.

[0025] The artificial intelligence-generated video detection device based on spatiotemporal anomaly mining, wherein the first fine-tuning module includes:

[0026] The input training video extracts multiple video frames according to the set time interval, and the time dimension initial features E0 of the multiple video frames are extracted through three-dimensional convolution as the input of the encoder; the encoder is composed of multiple spatiotemporal inconsistency learning modules in series, and the spatiotemporal inconsistency learning module includes a time feature mining module, a spatial feature mining module and a fusion module; in the spatiotemporal inconsistency learning module, the features are first obtained by TubeletEmbedding in the time feature mining module to obtain spatiotemporal tokens sorted by time, and each token is added with a position code embedding, and then the time feature E of the current t step is obtained through the Video Transformer model in the module. time,t :

[0027] E time,t =VideoTransformer(TubeletEmbedding(E t-1 )),

[0028] Among them, E t-1 is the output feature of the t-1th spatiotemporal inconsistency learning module;

[0029] In the spatial feature mining module, each frame of the video frames is divided into blocks to obtain the spatial token embedding of the blocks. The position encoding embedding Position Embedding is superimposed on each token embedding, and the superimposed embedding is embedded into the input video Transformer model and combined with E time，t Interactively obtain the t-step spatial feature E space,t :

[0030] E space,t =VisionTransformer(PatchEmbedding(E t-1 ))+E time,t .

[0031] Among them, PatchEmbedding() is a mapping embedding module used to obtain the superposition embedding representation of the spatial token embedding and position code embedding of each image; VisionTransformer() is a video Transformer model;

[0032] E space With E time Splicing, and then with the input E of this module t-1 Superposition, get the output E of step t t The video feature as the final output of the decoder is: t =Concat(E time,t , E space,t )+E t-1 .

[0033] The artificial intelligence-generated video detection device based on spatiotemporal anomaly mining, wherein the second fine-tuning module includes: adding a bypass branch to the attention layer of the encoder, and only performing gradient updates on the parameters of the bypass branch during fine-tuning.

[0034] The present invention also proposes a client for the artificial intelligence-generated video detection device based on spatiotemporal anomaly mining.

[0035] The present invention also proposes an electronic device, which includes the aforementioned artificial intelligence-generated video detection device based on spatiotemporal anomaly mining. The electronic device may be connected to an information display device, which is used to display the video detection results using display parameters and attributes set by the user or through an artificial intelligence model.

[0036] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the artificial intelligence-generated video detection method based on spatiotemporal anomaly mining are implemented.

[0037] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the artificial intelligence-generated video detection method based on spatiotemporal anomaly mining are implemented.

[0038] From the above solutions, it can be seen that the core technical points of the present invention are:

[0039] Key Point 1: A spatiotemporal inconsistency learning method based on spatiotemporal anomaly mining. This method extracts forgery features from the temporal domain through a temporal feature mining module and from the spatial domain of the video through a spatial feature mining module, simultaneously capturing spatial artifacts and temporally continuous rationality information in the video.

[0040] Key point 2: A large, high-parameter detection base model. This method uses more model parameters than traditional models, resulting in stronger learning capabilities. Through two stages of training, pre-training on fake videos and fine-tuning on downstream tasks, the model is better able to detect currently advanced AI-generated videos.

[0041] Therefore, the overall technical effect of the present invention is:

[0042] Key Point 1's spatiotemporal inconsistency learning method based on spatiotemporal anomaly mining enables the model to capture the logical rationality of video temporal continuity, which is overlooked by some traditional methods, thereby improving video detection accuracy. Key Point 2's large detection base model, with its larger number of parameters, can capture more complex video information. The two-stage model (pre-training and fine-tuning) also makes subsequent iterations and applications of the model more efficient. Compared to existing AI-generated video detection methods, this method demonstrates stronger detection capabilities on videos generated by current mainstream models. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 Schematic diagram of pre-training of the large model for detection base;

[0044] Figure 2 Schematic diagram of fine-tuning for generative video detection;

[0045] Figure 3Flow chart of the method of the present invention;

[0046] Figure 4 This is a module diagram of the device of the present invention;

[0047] Figure 5 This is a schematic structural diagram of a first electronic device of the present invention;

[0048] Figure 6 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0049] Figure 7 This is a schematic structural diagram of a second electronic device according to the present invention.

[0050] Reference numerals:

[0051] A-First electronic device;

[0052] B-AI-generated video detection device based on spatiotemporal anomaly mining;

[0053] C-data acquisition equipment;

[0054] D-information display device;

[0055] 1000- second electronic device;

[0056] Ⅰ-computing unit;

[0057] II-ROM;

[0058] III-RAM;

[0059] IV-bus;

[0060] V-interface;

[0061] VI-input unit;

[0062] VII-output unit;

[0063] VIII-Storage medium;

[0064] IX-Communication unit. DETAILED DESCRIPTION

[0065] It should be noted that, in this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0066] Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0067] The processor described in the present invention is the control center of an electronic device and can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0068] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0069] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smartphones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0070] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0071] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0072] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0073] It should also be understood that the term "and / or" in this document simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " in this document generally indicates an "or" relationship between the related objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0074] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0075] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0076] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0077] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0078] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0079] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0080] To illustrate the above-mentioned features and effects of the present invention more clearly and easily, the following embodiments are specifically described below with reference to the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are for illustrative purposes only. The scope of protection of the present invention is not limited to the disclosed embodiments; the present invention is defined by the appended claims.

[0081] Self-supervised pre-training of a large detection base model based on spatiotemporal anomaly mining:

[0082] With the introduction of Vision Transformer (ViT), masked autoencoder models have become applicable to the pre-training of large visual models. For visual information, it is highly redundant. Even if an image block is missing, it can be restored based on adjacent image blocks without having too much advanced understanding of other parts, objects, and scenes. In order to promote the learning of useful features, this model refers to the Transformer-based bidirectional encoder (Bidirectional Encoder Representations from Transformers, BERT), which masks random image blocks in the video, such as all pixel values ​​in a random 5% of all image blocks in an image) are set to 0. Design self-supervised training tasks to complete the pre-training of large models. The pre-training process is as follows: Figure 1 shown.

[0083] During the preprocessing stage, n frames of the input video are extracted at a set time interval, and random image blocks within each frame are masked. Subsequently, three-dimensional convolution is performed to extract initial temporal features E0 from multiple consecutive frames as input to the encoder. The input video includes both real-world videos and forged videos generated by the video generation model, ensuring that the model learns the basic representations of the forged videos. The encoder consists of multiple spatiotemporal inconsistency learning modules, each of which includes a temporal feature mining module, a spatial feature mining module, and a fusion module. The temporal and spatial feature mining modules extract temporal and spatial features, respectively. They are initialized with different random numbers, and the content updated during gradient backpropagation is also different. This design benefits from considering the temporal characteristics between different video frames. If temporal order is not considered, the neural network would treat "I love to eat apples" and "Apples love to eat, I" as identical content, as they both consist of three words. The encoder is composed of multiple stacked "temporal feature mining modules + spatial feature mining modules" (i.e., spatiotemporal inconsistency learning modules). The output of the first spatiotemporal inconsistency learning module will serve as the input of the second spatiotemporal inconsistency module, and so on.

[0084] In each spatiotemporal inconsistency learning module, features are first processed by Tubelet Embedding in the temporal feature mining module to obtain time-ordered spatiotemporal tokens. Videos can be viewed as three-dimensional data (space H × W + time T). The spatiotemporal information needs to be integrated into a unified embedding representation. Tubelet Embedding is a segmentation strategy designed for this purpose. This module simultaneously segments the video along the temporal and spatial dimensions, producing a spatiotemporal block (Tubelet) of size T × H × W. If the video is viewed as a "space-time cube," Tubelet is equivalent to carving out a thin spatiotemporal tube from it, hence the name "Tubelet."

[0085] Each spatiotemporal token is embedded with a positional encoding, and then the temporal feature E of the current step t is obtained through the Video Transformer model in the module. time,t :

[0086] E time,t =VideoTransformer(TubeletEmbedding(E t-1 )),

[0087] Among them, E t-1 is the module output obtained in step t-1. t-1 is the output of the t-1th spatiotemporal inconsistency learning module, and serves as the input of the tth spatiotemporal inconsistency learning module to generate the output E of the tth spatiotemporal inconsistency learning module. t .

[0088] Secondly, in the spatial feature mining module, each frame image is first divided into blocks, the visual semantics of the blocks are extracted, and the spatial token embedding is obtained. Then, the position encoding embedding is superimposed on each token embedding. The position encoding embedding is only related to the position of the image block, but has nothing to do with the semantics expressed by the image block itself. The superimposed embedding is input into the Vision Transformer model and compared with E time,t Interactively obtain the t-step spatial feature E space,t :

[0089] E space,t =VisionTransformer(PatchEmbedding(E t-1 ))+E time,t .

[0090] Among them, PatchEmbedding() is a mapping embedding module, which is used to obtain the embedded representation of the superposition of spatial token embedding and position encoding embedding of each image; VisionTransformer() refers to the Vision Transformer model, such as ViT, Swin Transformer model, etc., which is used to obtain the final features.

[0091] Finally, E space With E time Splicing, and then with the input E of this module t-1 Superposition, we get the output of step t:

[0092] E t =Concat(E time,t , E space,t )+E t-1 .

[0093] After processing multiple spatiotemporal inconsistency learning modules, the model is able to learn enough details to distinguish videos.

[0094] Fine-tuning large models for generative video detection:

[0095] The pre-trained model only has the ability to extract features and cannot be directly applied to AI-generated detection tasks. Next, the pre-trained model is fine-tuned on the AI-generated video detection task. During fine-tuning, the decoder part is removed, the encoder is retained, and a detector is added after the encoder, such as a linear classification neural network layer. The fine-tuning process is as follows: Figure 2 Fine-tuning is only performed on the pre-trained encoder and detector, and the training data uses a constructed labeled training set containing real and fake videos.

[0096] To improve model fine-tuning efficiency and ultimate generalization, a bypass branch is added to the attention layer of the detection model during fine-tuning. This freezes the original model parameters and only performs gradient updates on the bypass branch. This freezes the original encoder parameters and trains only the fine-tuned parameter matrices A and B (A and B here are the so-called bypass branch). It's important to note that the parameters trained during fine-tuning include the fine-tuning parameter matrices A and B, as well as the detector (linear classification neural network layer) mentioned above.

[0097] This method, when faced with videos from unseen domains, only requires dozens or hundreds of small sample training data to achieve model generalization to unseen domain data. Since the original weight parameter W0 is frozen, the model avoids catastrophic forgetting of existing forgery methods. When learning with small samples, the only fine-tuning parameters are A and B. The updated model weight W is:

[0098] W=W0+ΔW,

[0099] ΔW=BA.

[0100] By fusing spatial and temporal features, the model can more comprehensively understand video content, thereby improving the accuracy of forgery detection. The fine-tuning matrix A is initialized with random numbers from a small-variance Gaussian distribution, and the fine-tuning matrix B is initialized with all zeros. Both A and B will be updated during fine-tuning.

[0101] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0102] like Figure 4 As shown, the present invention also proposes an artificial intelligence-generated video detection device based on spatiotemporal anomaly mining, which includes:

[0103] The initial module obtains multiple training videos that have been annotated with generation source labels, where the generation source labels include AI-generated videos and non-AI-generated videos;

[0104] The first fine-tuning module inputs the training video into a Transformer-based encoder. The encoder extracts features from the training video, and the decoder restores the obtained video features to obtain a restored video. A loss function is constructed based on the restored video and the training video. The encoder and decoder are fine-tuned, and the fine-tuned encoder is retained as an intermediate encoder.

[0105] The second fine-tuning module adds a detector after the intermediate encoder. The detector obtains a detection result of whether the video is AI-generated based on the video features output by the intermediate encoder. Based on the detection result and the generation source label of the training video, a loss function is constructed, and the intermediate encoder and the detector are fine-tuned. The fine-tuned intermediate encoder and the detector are used as the video detection model;

[0106] The detection module inputs the video to be detected into the video detection model to obtain a video detection result of whether it is an artificial intelligence generated video.

[0107] The artificial intelligence-generated video detection device based on spatiotemporal anomaly mining, wherein the first fine-tuning module includes:

[0108] The input training video extracts multiple video frames according to the set time interval, and the time dimension initial features E0 of the multiple video frames are extracted through three-dimensional convolution as the input of the encoder; the encoder is composed of multiple spatiotemporal inconsistency learning modules in series, and the spatiotemporal inconsistency learning module includes a time feature mining module, a spatial feature mining module and a fusion module; in the spatiotemporal inconsistency learning module, the features are first obtained by TubeletEmbedding in the time feature mining module to obtain spatiotemporal tokens sorted by time, and each token is added with a position code embedding, and then the time feature E of the current t step is obtained through the Video Transformer model in the module. time,t :

[0109] E time,t =VideoTransformer(TubeletEmbedding(E t-1 )),

[0110] Among them, E t-1 is the output feature of the t-1th spatiotemporal inconsistency learning module;

[0111] In the spatial feature mining module, each frame of the video frames is divided into blocks to obtain the spatial token embedding of the blocks. The position encoding embedding Position Embedding is superimposed on each token embedding, and the superimposed embedding is embedded into the input video Transformer model and combined with E time，t Interactively obtain the t-step spatial feature E space,t :

[0112] E space,t =VisionTransformer(PatchEmbedding(E t-1 ))+E time,t ·

[0113] Among them, PatchEmbedding() is a mapping embedding module used to obtain the superposition embedding representation of the spatial token embedding and position code embedding of each image; VisionTransformer() is a video Transformer model;

[0114] E space With E time Splicing, and then with the input E of this module t-1 Superposition, get the output E of step t t The video feature as the final output of the decoder is: t =Concat(E time,t , E space,t )+Et-1 .

[0115] The artificial intelligence-generated video detection device based on spatiotemporal anomaly mining, wherein the second fine-tuning module includes: adding a bypass branch to the attention layer of the encoder, and only performing gradient updates on the parameters of the bypass branch during fine-tuning.

[0116] The present invention also proposes a client for the artificial intelligence-generated video detection device based on spatiotemporal anomaly mining.

[0117] like Figure 5 As shown, the present invention further proposes a first electronic device A in another embodiment, which includes the aforementioned artificial intelligence-generated video detection device based on spatiotemporal anomaly mining.

[0118] like Figure 6 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to collect and obtain the video to be detected, such as videos posted on social platforms or videos to be uploaded to social platforms and need to be reviewed. The information display device D is used to display the detection results obtained by the analysis of the present invention.

[0119] The information display device D can organize and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll, etc. The user is presented with the key information specified by the user, and the user can understand this information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present the user with rich and necessary key information.

[0120] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the artificial intelligence-generated video detection method based on spatiotemporal anomaly mining provided by the above-mentioned methods.

[0121] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the artificial intelligence-generated video detection method based on spatiotemporal anomaly mining. It should be understood that the storage medium in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM).

[0122] Figure 7 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0123] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. Various programs and data required for the operation of the device 1000 can also be stored in the RAM III. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0124] Multiple components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard and mouse; an output unit VII, such as various types of displays and speakers; a storage medium VIII, such as a magnetic disk and optical disk; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0125] Computing unit I can be various general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. Computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the method described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0126] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. An artificial intelligence generated video detection method based on spatiotemporal anomaly mining, characterized by: include: In the initial step, multiple training videos with labeled generation source labels are obtained, where the generation source labels include AI-generated videos and non-AI-generated videos; In the first fine-tuning step, the training video is input into a Transformer-based encoder. The encoder extracts features from the training video, and the decoder restores the obtained video features to obtain a restored video. A loss function is constructed based on the restored video and the training video. The encoder and decoder are fine-tuned, and the fine-tuned encoder is retained as an intermediate encoder. In the second fine-tuning step, a detector is added after the intermediate encoder. The detector detects whether the video is AI-generated based on the video features output by the intermediate encoder. Based on the detection result and the generation source label of the training video, a loss function is constructed, and the intermediate encoder and the detector are fine-tuned. The fine-tuned intermediate encoder and the detector are used as the video detection model. In the detection step, the video to be detected is input into the video detection model to obtain a video detection result of whether it is an artificial intelligence generated video.

2. The artificial intelligence generated video detection method based on spatiotemporal anomaly mining according to claim 1, characterized in that: This first fine-tuning step includes: The input training video extracts multiple video frames according to the set time interval, and the time dimension initial features E0 of the multiple video frames are extracted through three-dimensional convolution as the input of the encoder; the encoder is composed of multiple spatiotemporal inconsistency learning modules in series, and the spatiotemporal inconsistency learning module includes a time feature mining module, a spatial feature mining module and a fusion module; in the spatiotemporal inconsistency learning module, the features are first obtained by TubeletEmbedding in the time feature mining module to obtain spatiotemporal tokens sorted by time, and each token is added with a position code embedding, and then the time feature E of the current step t is obtained through the VideoTransformer model in the module. time,t : e time,t =VideoTransformer(TubeletEmbedding(E t-1 )), Among them, E t-1 is the output feature of the t-1th spatiotemporal inconsistency learning module; In the spatial feature mining module, each frame of the multiple video frames is divided into blocks to obtain the spatial token embedding of the blocks. The position code embedding PositionEmbedding is superimposed on each token embedding, and the superimposed embedding is embedded into the input video Transformer model and compared with E time,t Interactively obtain the t-step spatial feature E space,t : E space,t =VisionTransformer(PatchEmbedding(E t-1 ))+E time,t . Among them, PatchEmbedding() is a mapping embedding module used to obtain the superposition embedding representation of the spatial token embedding and position code embedding of each image; VisionTransformer() is a video Transformer model; E space With E time Splicing, and then with the input E of this module t-1 Superposition, get the output E of step t t The video feature as the final output of the decoder is: t =Concat(E time,t ,E space,t )+E t-1 .

3. The artificial intelligence generated video detection method based on spatiotemporal anomaly mining according to claim 1 is characterized in that: The second fine-tuning step includes: adding a bypass branch to the attention layer of the encoder, and performing gradient updates only on the parameters of the bypass branch during fine-tuning.

4. An artificial intelligence generated video detection device based on spatiotemporal anomaly mining, characterized in that: include: The initial module obtains multiple training videos that have been annotated with generation source labels, where the generation source labels include AI-generated videos and non-AI-generated videos; The first fine-tuning module inputs the training video into a Transformer-based encoder. The encoder extracts features from the training video, and the decoder restores the obtained video features to obtain a restored video. A loss function is constructed based on the restored video and the training video. The encoder and decoder are fine-tuned, and the fine-tuned encoder is retained as an intermediate encoder. The second fine-tuning module adds a detector after the intermediate encoder. The detector obtains a detection result of whether the video is AI-generated based on the video features output by the intermediate encoder. Based on the detection result and the generation source label of the training video, a loss function is constructed, and the intermediate encoder and the detector are fine-tuned. The fine-tuned intermediate encoder and the detector are used as the video detection model; The detection module inputs the video to be detected into the video detection model to obtain a video detection result of whether it is an artificial intelligence generated video.

5. The artificial intelligence generated video detection device based on spatiotemporal anomaly mining according to claim 1, characterized in that: The first fine-tuning module includes: The input training video extracts multiple video frames according to the set time interval, and the time dimension initial features E0 of the multiple video frames are extracted through three-dimensional convolution as the input of the encoder; the encoder is composed of multiple spatiotemporal inconsistency learning modules in series, and the spatiotemporal inconsistency learning module includes a time feature mining module, a spatial feature mining module and a fusion module; in the spatiotemporal inconsistency learning module, the features are first obtained by TubeletEmbedding in the time feature mining module to obtain spatiotemporal tokens sorted by time, and each token is added with a position code embedding, and then the time feature E of the current step t is obtained through the VideoTransformer model in the module. time,t : E time,t =VideoTransformer(TubeletEmbedding(E t-1 )), Among them, E t-1 is the output feature of the t-1th spatiotemporal inconsistency learning module; In the spatial feature mining module, each frame of the multiple video frames is divided into blocks to obtain the spatial token embedding of the blocks. The position code embedding PositionEmbedding is superimposed on each token embedding, and the superimposed embedding is embedded into the input video Transformer model and compared with E time,t Interactively obtain the t-step spatial feature E space,t : E space,t =VisionTransformer(PatchEmbedding(E t-1 ))+E time,t . Among them, PatchEmbedding() is a mapping embedding module used to obtain the superposition embedding representation of the spatial token embedding and position code embedding of each image; VisionTransformer() is a video Transformer model; E space With E time Splicing, and then with the input E of this module t-1 Superposition, get the output E of step t t The video feature as the final output of the decoder is: t =Concat(E time,t ,E space,t )+E t-1 .

6. The artificial intelligence generated video detection device based on spatiotemporal anomaly mining according to claim 1, characterized in that: The second fine-tuning module includes: adding a bypass branch to the attention layer of the encoder, and only performing gradient updates on the parameters of the bypass branch during fine-tuning.

7. A client for use in any one of the artificial intelligence-generated video detection devices based on spatiotemporal anomaly mining as described in claims 4-6.

8. An electronic device, characterized in that: It includes an artificial intelligence-generated video detection device based on spatiotemporal anomaly mining as described in claims 4-6, and the electronic device is connected to an information display device, which is used to display the video detection results with display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the artificial intelligence-generated video detection method based on spatiotemporal anomaly mining as described in any one of claims 1 to 3.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the artificial intelligence-generated video detection method based on spatiotemporal anomaly mining described in any one of claims 1 to 3 are implemented.

Images