Video detection method and system for natural scene generation based on spectral volume

Abstract

This application provides a method and system for detecting naturally generated videos based on spectral volume, belonging to the field of video detection technology. The method includes: acquiring a sample video set, which includes multiple sample videos with preset true / false labels; preprocessing the multiple sample videos, extracting features using a two-stream network, and training a convolutional neural network to obtain a video detection model; acquiring the target video information to be detected and preprocessing it using an extraction strategy; extracting features using the video detection model, designing a classifier for analysis, obtaining the probability representing the category of the target video information, and determining whether the target video information is a generated video. In the model training stage, this application focuses on both pixel features and motion features, using a spectral volume motion representation method to further represent generation cues, reducing computational overhead and addressing the shortcomings of generated video detection in natural scenes.

Description

Technical Field

[0001] This application relates to the field of video detection technology, specifically to a method and system for detecting natural scene generation videos based on spectral volume. Background Technology

[0002] Generative artificial intelligence technology has developed rapidly since the advent of ChatGPT, enabling the generation of content media such as text, voice, images, and videos using AI. For example, OpenAI's Sora has demonstrated extremely high video generation capabilities. Users can generate high-quality videos using text, images, and motion skeletons as conditions. However, the application of such video generation models is very simple, especially text-based video generation. Users only need to use text descriptions to generate a video. While the creativity of video generation models can provide entertainment and commercial value, their ease of use also makes them prone to producing videos with false content, endangering digital media security. Therefore, it is necessary to conduct detection work on AI-generated videos.

[0003] Current video generation models have achieved near-realistic results in creating videos of natural scenes, with relatively few motion inconsistencies. This can be attributed to the fact that most motion patterns in natural scenes are periodic or oscillatory, and these regular motion patterns are easier for neural networks to capture and learn. Therefore, generative models perform better when processing natural scenes.

[0004] However, existing video detection methods often lack sufficient specificity when dealing with generated videos of natural scenes. These methods primarily rely on pixel information in the video to find detection cues, or supplement them with optical flow information to extract motion features from the video. Although optical flow information can provide useful cues about pixel motion, its computational cost is high and directly related to the video duration, which limits its application in large-scale video detection. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application provides a method and system for detecting natural scene generated videos based on spectral volume, which solves the problems of insufficient accuracy and reliability of existing generated video detection technologies in natural scene detection.

[0006] To achieve the above objectives, this application provides the following technical solution:

[0007] In a first aspect, embodiments of this application provide a method for detecting naturally generated videos based on spectral volume. This method includes: acquiring a sample video set, which includes multiple sample videos with preset true / false labels; preprocessing the multiple sample videos based on a preset extraction strategy to extract a first spectral volume feature; extracting features from the sample videos and the first spectral volume feature using an acquired two-stream network to obtain a first pixel feature and a first motion feature; training a video detection model using the first pixel feature and the first motion feature as samples based on a convolutional neural network; acquiring target video information to be detected, and preprocessing the target video information using the extraction strategy to obtain a second spectral volume feature; extracting features from the target video information and the second spectral volume feature using the video detection model to obtain a second pixel feature and a second motion feature; designing a classifier to analyze the second pixel feature and the second motion feature to obtain the probability representing the category of the target video information, and determining whether the target video information is a generated video.

[0008] According to a first aspect of the embodiments of this application, the sample video set includes real videos and multiple generated videos, with each generated video corresponding to a different video generation algorithm, and each generated video is determined by generating text or images based on the corresponding video generation algorithm.

[0009] According to a first aspect of the embodiments of this application, the aforementioned preprocessing of multiple sample videos based on a preset extraction strategy to extract a first spectral volume feature may specifically include the following steps: extracting the sample videos into optical flow to preliminarily determine motion texture; performing a Fourier transform on the optical flow in the time dimension to obtain a frequency domain representation to obtain the first spectral volume feature; wherein, the first spectral volume feature is the time Fourier transform of the trajectory of each pixel extracted from the sample videos.

[0010] According to a first aspect of the embodiments of this application, the dual-stream structure network includes a pixel feature extraction network and a motion feature extraction network, wherein the pixel feature extraction network is a pre-trained network and is one of 3DResNet and Video ViT.

[0011] According to a first aspect of the embodiments of this application, a motion feature extraction network is trained on a preset basic dataset and for an action recognition task, so that the motion feature extraction network has the ability to extract motion features and learn the general motion information of spectral volume in different videos.

[0012] According to a first aspect of the embodiments of this application, the basic dataset includes at least one of the Kinetics-400 dataset and the UCF101 dataset.

[0013] According to a first aspect of the embodiments of this application, the aforementioned feature extraction from the sample video and the first spectral volume feature through the acquired dual-stream structure network to obtain the first pixel feature and the first motion feature may specifically include the following steps: extracting features from the sample video through a pixel feature extraction network to obtain the first pixel feature; extracting features from the first spectral volume feature through a motion feature extraction network to obtain the first motion feature; wherein the first pixel feature and the first motion feature are both one-dimensional feature vectors, the first pixel feature includes color, illumination and texture information, and the first motion feature includes motion information of the camera and objects in the video.

[0014] According to a first aspect of the embodiments of this application, the aforementioned method of using first pixel features and first motion features as samples and training based on a convolutional neural network to obtain a video detection model may specifically include the following steps: using a multi-head cross-attention mechanism to fuse the first pixel features and the first motion features to generate fused features to characterize the common information of the first pixel features and the first motion features; based on the fused features, determining the detection result corresponding to each sample video in the sample video set; and based on the detection result of each sample video and its corresponding label, iteratively training based on a convolutional neural network to obtain a video detection model.

[0015] According to a first aspect of the embodiments of this application, the aforementioned video detection model is obtained by iteratively training a convolutional neural network based on the detection results and corresponding labels of each sample video. Specifically, it may include the following steps: during the training process to obtain an initial detection model, the detection loss corresponding to the initial detection model is calculated; based on the detection loss, the initial detection model is iteratively trained until a predetermined training termination condition is met to obtain an optimized video detection model.

[0016] Secondly, embodiments of this application provide a natural scene generation video detection system based on spectral volume. This system includes: an acquisition module, a first preprocessing module, a first feature extraction module, a training module, a second preprocessing module, a second feature extraction module, and a judgment module. The acquisition module acquires a set of sample videos, which includes multiple sample videos with preset true / false labels. The first preprocessing module preprocesses the multiple sample videos based on a preset extraction strategy to extract first spectral volume features. The first feature extraction module extracts features from the sample videos and the first spectral volume features using the acquired two-stream network. The system consists of three modules: a first pixel feature and a first motion feature; a training module, which uses the first pixel feature and the first motion feature as samples to train a video detection model based on a convolutional neural network; a second preprocessing module, which acquires the target video information to be detected and preprocesses the target video information using an extraction strategy to obtain a second spectral volume feature; a second feature extraction module, which extracts features from the target video information and the second spectral volume feature using the video detection model to obtain the second pixel feature and the second motion feature; and a judgment module, which designs a classifier to analyze the second pixel feature and the second motion feature, obtains the probability representing the category of the target video information, and determines whether the target video information is a generated video.

[0017] Thirdly, embodiments of this application provide an electronic device, which includes: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the spectral volume-based natural scene generation video detection method described in the first aspect above.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the spectral volume-based natural scene generation video detection method described in the first aspect above.

[0019] This application provides a method and system for detecting natural scene generation videos based on spectral volume. Compared with existing technologies, it has the following advantages:

[0020] This application trains a video detection model based on a sample video set with pre-defined true and false labels. A first spectral volume feature is extracted from each sample video. A two-stream network is then used to process both the sample video and the first spectral volume feature to obtain first pixel features and first motion features, which are then used to train the video detection model. This collaborative work of the two-stream network enables comprehensive analysis of video content, providing rich feature information for video detection. During model training, this application focuses on both pixel and motion features, using a motion representation method of spectral volume to further represent generation cues, reducing computational overhead and addressing the shortcomings of generated video detection in natural scenes. During video detection, a second spectral volume feature is extracted based on the same extraction strategy. The obtained video detection model is then used for feature extraction to further classify and determine the generated video. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a natural scene generation video detection method based on spectral volume provided in an embodiment of this application;

[0023] Figure 2 yes Figure 1 An exemplary process diagram of S140;

[0024] Figure 3 This is a schematic diagram of the structure of a natural scene generation video detection system based on spectral volume provided in an embodiment of this application;

[0025] Figure 4 This is a structural diagram of an electronic device provided in an embodiment of the present application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0028] This application provides a method and system for detecting generated videos of natural scenes based on spectral volume, which solves the problems of insufficient accuracy and reliability of existing generated video detection technologies in the face of natural scenes.

[0029] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0030] With the rapid advancement of deep learning technology, video generation technology has achieved a qualitative leap, creating increasingly realistic video content. This has greatly enriched people's entertainment lives and stimulated creativity, but it has also inevitably brought security risks, especially the proliferation of misinformation. Therefore, conducting detection work on generated videos is of great significance.

[0031] Currently, the field of generated video detection mainly employs techniques such as 3D convolutional neural networks or visual Transformer (ViT) models to directly extract features from the video for detection. These methods perform well when processing data in the same domain as the training data, but they suffer from limitations in generalization, especially when faced with unknown generation methods. This is because existing methods primarily focus on pixel features in the video, failing to adequately capture motion variation features in the generated video. Since current video generation models still need improvement in motion consistency, by capturing motion variations and extracting generation cues, detection models can discover generation cues that were not previously found in pixel features. Therefore, researchers have begun to try using optical flow maps to represent motion information in videos, extracting motion features from optical flow maps to enhance the generalization ability of detection models. Optical flow maps provide motion information of pixels between consecutive frames, which is crucial for understanding motion variations in videos. However, optical flow maps have high computational and storage costs and require normalization processing for videos of different durations, limiting their application in large-scale training.

[0032] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0033] The following section first introduces a method for natural scene generation and video detection based on spectral volume provided in the embodiments of this application.

[0034] This application provides a flowchart illustrating a natural scene generation video detection method based on spectral volume, as shown in the embodiments below. Figure 1 As shown, the natural scene generation video detection method may include the following steps S110-S170.

[0035] S110. Obtain a sample video set, which includes multiple sample videos with preset authenticity labels.

[0036] S120. Based on a preset extraction strategy, preprocess multiple sample videos to extract the first spectral volume features. It is understandable that, in order to better obtain motion features, a motion representation of object vibration, namely the first spectral volume features, is obtained from the sample videos. Compared to using optical flow as a motion feature, the first spectral volume features select K output frequencies, thus eliminating the need to use data of the same duration as the entire video as input, thereby reducing overhead.

[0037] S130. The first pixel feature and the first motion feature are obtained by extracting features from the sample video and the first spectral volume feature through the acquired dual-stream structure network.

[0038] S140. Using the first pixel feature and the first motion feature as samples, a video detection model is obtained by training based on a convolutional neural network.

[0039] S150. Obtain the target video information to be detected, and preprocess the target video information through an extraction strategy to obtain the second spectral volume feature.

[0040] S160. The target video information and the second spectral volume feature are extracted by the video detection model to obtain the second pixel feature and the second motion feature.

[0041] S170. Design a classifier to analyze the second pixel features and the second motion features to obtain the probability representing the category of the target video information, and determine whether the target video information is a generated video. It can be understood that the extracted second pixel features and second motion features contain the differences between real and generated videos. The classifier classifies the one-dimensional features to determine the detection result of the target video information to be detected.

[0042] The above is a specific implementation of a natural scene generation video detection method based on spectral volume provided in this application. It can be understood that this application is trained based on a sample video set with preset true and false labels. The first spectral volume feature is extracted from each sample video. The sample video and the first spectral volume feature are processed by a two-stream structure network to obtain the first pixel feature and the first motion feature to train the video detection model. Through the collaborative work of this two-stream structure network, the video content can be comprehensively analyzed, providing rich feature information for video detection.

[0043] It should be noted that this application focuses on both pixel features and motion features during the model training phase. A motion representation method using spectral volume is employed to further represent the generation cues, reducing computational overhead and addressing the shortcomings of generated video detection in natural scenes. During video detection, the corresponding second spectral volume features are extracted based on the same extraction strategy. These features are then used by the obtained video detection model for further feature extraction and classification to determine the generated video.

[0044] In some embodiments, the sample video set includes real videos and multiple generated videos, with each generated video corresponding to a different video generation algorithm. Each generated video is determined using text or images based on the corresponding video generation algorithm.

[0045] In the embodiments of this application, it is understood that real videos and videos generated by various video generation algorithms are used as samples to train the initial generated video detection model. This process enables the model to identify and learn specific features of generated videos and the differences between real videos and generated videos.

[0046] These generated videos correspond to various video generation algorithms, each capable of directly generating videos using specific prompts. For example, text-driven and image-driven video generation techniques can generate videos based on text descriptions or image content. In practical applications, the video generation algorithms involved may include Modelscope, Zeroscope, Stable video diffusion, and I2VGen-XL, each representing different video generation techniques and methods. Videos generated by these algorithms can provide rich training samples for video detection models, improving their ability to detect generated videos.

[0047] In one example, the aforementioned preprocessing of multiple sample videos based on a preset extraction strategy to extract the first spectral volume feature, specifically S120 may include the following steps:

[0048] S210. Extract optical flow from the sample video to preliminarily determine motion texture;

[0049] S220. Perform a Fourier transform on the optical flow in the time dimension to represent it in the frequency domain, and obtain the first spectral volume feature; wherein, the first spectral volume feature is the time Fourier transform of the trajectory of each pixel extracted from the sample video.

[0050] In the embodiments of this application, it is understood that the motion texture is a series of 2D displacement maps that change over time. To better extract the first spectral volume features, optical flow is used to extract the video motion texture, which describes the pattern of pixel movement on the surface of an object in an image sequence. It focuses on the displacement description in the video, discarding information such as color and lighting in the video, which is beneficial for extracting motion signals from the video.

[0051] It's important to note that in videos generated from natural scenes, much of the natural motion can be described as a superposition of a small number of harmonic oscillations with different frequencies, amplitudes, and phases. Because these underlying motions are quasi-periodic, it's natural to model them in the frequency domain. The spectral volume, extracted from the time-fourth-transform of the per-pixel trajectory in the video, can represent the quasi-periodic motion present in natural scenes.

[0052] In some embodiments, the aforementioned dual-stream structure network extracts features from the sample video and the first spectral volume feature respectively to obtain the first pixel feature and the first motion feature. That is, the aforementioned S130 may specifically include the following steps:

[0053] S310. Extract features from the sample video using a pixel feature extraction network to obtain the first pixel feature;

[0054] S320. The first spectral volume feature is extracted using a motion feature extraction network to obtain the first motion feature;

[0055] Among them, the first pixel feature and the first motion feature are both one-dimensional feature vectors. The first pixel feature includes color, lighting and texture information, and the first motion feature includes motion information of the camera and objects in the video.

[0056] In the embodiments of this application, it is understood that this application first extracts first pixel features from the sample video. These features can reveal the spatial characteristics of the sample video, including color, lighting, and texture. Motion features in the sample video are extracted by analyzing the first spectral volume. These features can capture the regularity of object motion in the sample video.

[0057] This application employs two dedicated networks to extract pixel and motion features. The pixel feature extraction network is responsible for acquiring visual appearance features such as color, lighting, and texture from the video. The motion feature extraction network, on the other hand, processes spectral volume data to capture dynamic features such as object motion and camera movement within the video. Through the collaborative operation of this dual-stream network structure, comprehensive video content analysis is achieved, providing rich feature information for video detection.

[0058] It should be noted that, in practical applications, any suitable network can be used for feature extraction. Optionally, since the 3D ResNet network can directly process video frames or time-series data, capturing pixel-level spatial features and inter-frame temporal features, the pixel feature extraction network and the motion feature extraction network can use the same 3D ResNet structure to effectively extract pixel and motion features from the video.

[0059] In some embodiments, the dual-stream network includes a pixel feature extraction network and a motion feature extraction network, wherein the pixel feature extraction network is a pre-trained network and is one of 3D ResNet and Video ViT.

[0060] The motion feature extraction network is trained on a pre-defined base dataset for action recognition tasks, enabling it to extract motion features and learn general motion information of spectral volume in different videos. The base dataset includes at least one of the Kinetics-400 dataset and the UCF101 dataset.

[0061] In this embodiment, it is understood that the two-stream network includes two feature extraction networks, both of which are pre-trained to enhance their feature extraction performance. For the pixel feature extraction network, existing pre-trained networks, such as 3D ResNet or Video ViT, can be used; these networks have been applied in a wide range of video tasks. For the spectral volume network, due to the lack of readily available pre-trained models, this embodiment performs dedicated pre-training. Pre-training on action recognition tasks can significantly improve its performance in extracting action features. For this purpose, datasets such as Kinetics-400 and UCF101 can be selected for pre-training. These datasets contain rich action information, which helps the network learn more discriminative action features. This pre-training strategy ensures the efficiency and accuracy of both networks in feature extraction.

[0062] In some embodiments, such as Figure 2 As shown, the aforementioned video detection model is trained using the first pixel feature and the first motion feature as samples, based on a convolutional neural network. Specifically, S140 may include the following steps:

[0063] S410. Use a multi-head cross-attention mechanism to fuse the first pixel feature and the first motion feature to generate a fused feature to represent the common information of the first pixel feature and the first motion feature;

[0064] S420. Based on the fusion features, determine the detection result corresponding to each sample video in the sample video set;

[0065] S430. Based on the detection results and corresponding labels of each sample video, a video detection model is obtained through iterative training using a convolutional neural network.

[0066] In this embodiment, the convolutional neural network uses a common residual neural network to extract pixel-level features and motion features from video frames. Pixel features include information such as color, lighting, and texture, while motion features may include the trajectory of objects and vibration frequencies. Real videos and generated videos have potential differences in visual appearance and motion. Elements such as color, lighting, and texture constitute the visual appearance of the video, while the motion of objects and the movement of the camera constitute the motion of the video.

[0067] Understandably, multi-head cross-attention mechanisms consist of multiple attention layers, aiming to capture the correlation between the first pixel feature and the first motion feature, thereby extracting common information between the two modalities over a wider range. This application, by combining the first pixel feature and the first motion feature, helps avoid the video detection model's over-reliance on a single feature, thus reducing the risk of overfitting; it also improves the accuracy of video detection and enhances the model's generalization ability to unknown video content.

[0068] It's important to note that the multi-head cross-attention mechanism allows for parallel attention to multiple locations within a sequence when processing sequential data, effectively integrating information from both the pixel and motion levels. This approach can more accurately identify forgery signs in generated videos. Compared to simply concatenating two features, feature fusion techniques make the model more robust when processing video data. This is because different features capture different aspects of the video, and when one feature is insufficient to provide adequate information, others can supplement it. This complementarity enhances the model's robustness and reduces overfitting to single-modality features, further improving the model's performance and generalization ability in video detection tasks.

[0069] In one example, the aforementioned video detection model is obtained by iteratively training a convolutional neural network based on the detection results and corresponding labels of each sample video. Specifically, S430 may include: calculating the detection loss corresponding to the initial detection model during the training process; iteratively training the initial detection model based on the detection loss until the predetermined training termination condition is met, thereby obtaining the optimized video detection model.

[0070] In the specific implementation of model training, the initial detection model is first used to extract pixel features from the video, which reflect the visual details of the video. Next, by converting the video into spectral volume as a representation of motion information, the model can extract action features from the samples based on the spectral volume. Furthermore, the model is continuously iterated and trained using the detection results and actual labels from the sample video set to optimize the initial detection model. This process not only makes the model adept at capturing pixel features of the video but also equips it with the skills to analyze the dynamic characteristics of the video.

[0071] Specifically, by utilizing the detection results and true / false labels of each sample video in the sample video set, combined with a pre-defined loss function, the detection loss of the initial video detection model can be calculated. Subsequently, based on this detection loss, the initial generated video detection model is iteratively trained until a predetermined training stopping condition is met, ultimately yielding the optimized generated video detection model.

[0072] For example, in the embodiments of this application, three preset loss functions are defined. The first loss function measures the performance loss of the model when generating video detection based on pixel features; the second loss function evaluates the performance loss of the model when detecting based on motion features; and the third loss function addresses the performance loss of the model when detecting based on fused features. Finally, by weighted summing these three preset losses, the total loss of the initial generated video detection model is obtained, which serves as the optimization objective for model training. This method helps to balance the contributions of different features in the detection process, ensuring the model's comprehensive performance across various features.

[0073] In some embodiments, this application provides a natural scene generation video detection system 500 based on spectral volume, such as... Figure 3 As shown, the natural scene generation video detection system 500 may include the following modules:

[0074] The acquisition module 510 is used to acquire a sample video set, which includes multiple sample videos with preset true / false labels;

[0075] The first preprocessing module 520 is used to preprocess multiple sample videos based on a preset extraction strategy to extract the first spectral volume features.

[0076] The first feature extraction module 530 is used to extract features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network to obtain the first pixel feature and the first motion feature;

[0077] Training module 540 is used to train a video detection model based on a convolutional neural network using the first pixel feature and the first motion feature as samples.

[0078] The second preprocessing module 550 is used to acquire the target video information to be detected, and to preprocess the target video information through an extraction strategy to obtain the second spectral volume feature.

[0079] The second feature extraction module 560 is used to extract features from the target video information and the second spectral volume feature through the video detection model to obtain the second pixel feature and the second motion feature.

[0080] The judgment module 570 is used to design a classifier to analyze the second pixel features and the second motion features, obtain the probability of representing the category of the target video information, and determine whether the target video information is a generated video.

[0081] According to embodiments of this application, any and multiple modules among the acquisition module 510, the first preprocessing module 520, the first feature extraction module 530, the training module 540, the second preprocessing module 550, the second feature extraction module 560, and the judgment module 570 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module.

[0082] In some embodiments, the first preprocessing module 520 may specifically be used for:

[0083] The sample video was extracted as optical flow to preliminarily determine the motion texture;

[0084] The optical flow is represented in the frequency domain by performing a Fourier transform in the time dimension to obtain the first spectral volume feature; wherein, the first spectral volume feature is the time Fourier transform of the trajectory of each pixel extracted from the sample video.

[0085] In some embodiments, the first feature extraction module 530 may specifically be used for:

[0086] The first pixel feature is obtained by extracting features from the sample video using a pixel feature extraction network.

[0087] The first motion feature is obtained by extracting features from the first spectral volume feature using a motion feature extraction network;

[0088] Among them, the first pixel feature and the first motion feature are both one-dimensional feature vectors. The first pixel feature includes color, lighting and texture information, and the first motion feature includes motion information of the camera and objects in the video.

[0089] In some embodiments, the training module 540 may specifically be used for:

[0090] A multi-head cross-attention mechanism is used to fuse the first pixel feature and the first motion feature to generate a fused feature that represents the common information of the first pixel feature and the first motion feature.

[0091] Based on the fusion features, the detection result corresponding to each sample video in the sample video set is determined;

[0092] Based on the detection results and corresponding labels of each sample video, a video detection model is obtained through iterative training using a convolutional neural network.

[0093] Figure 3 Each module in the system shown has the function of implementing each step in the aforementioned method for generating natural scenes based on spectral volume and can achieve the corresponding technical effect. For the sake of brevity, it will not be elaborated here.

[0094] In some embodiments, this application provides an electronic device, the structural schematic of which is shown below. Figure 4 As shown.

[0095] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0096] Specifically, the processor 610 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0097] Memory 620 may include mass storage for data or instructions. For example, and not limitingly, memory 620 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 620 may include removable or non-removable (or fixed) media. Where appropriate, memory 620 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 620 is non-volatile solid-state memory.

[0098] Memory 620 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the spectral volume-based natural scene generation video detection methods in the above embodiments.

[0099] The processor 610 reads and executes computer program instructions stored in the memory 620 to implement any of the spectral volume-based natural scene generation video detection methods in the above embodiments.

[0100] In one example, the electronic device may also include a communication interface 630 and a bus 600. Wherein, such as Figure 4 As shown, the processor 610, memory 620, and communication interface 630 are connected via bus 600 and communicate with each other.

[0101] The communication interface 630 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0102] Bus 600 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 600 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0103] Furthermore, in conjunction with the spectral volume-based natural scene generation video detection method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the spectral volume-based natural scene generation video detection methods in the above embodiments.

[0104] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0105] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0106] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0107] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0108] In summary, compared with the prior art, this application has the following beneficial effects:

[0109] 1. This application is trained based on a sample video set with preset true and false labels. The first spectral volume feature is extracted from each sample video. The sample video and the first spectral volume feature are processed by a two-stream structure network to obtain the first pixel feature and the first motion feature to train the video detection model. Through the collaborative work of this two-stream structure network, the video content can be comprehensively analyzed, providing rich feature information for video detection.

[0110] 2. In the model training stage, this application focuses on two aspects: pixel features and motion features. It uses the motion representation method of spectral volume to further represent the generated cues and adopts a multi-head cross-attention mechanism to process the sequence data, and performs parallel attention on multiple positions in the sequence, thereby effectively integrating the information of the video at the pixel level and the motion level.

[0111] 3. In the specific implementation of model training in this application, an initial detection model is used to extract pixel features from the video. These features reflect the details of the video's appearance. Then, by converting the video into spectral volume as a representation of motion information, the model is able to extract motion features from the samples based on the spectral volume. Through continuous iteration, the model is trained using the detection results and actual labels in the sample video set. This process not only makes the model adept at capturing pixel features of the video but also enables it to master the skills of analyzing the dynamic characteristics of the video.

[0112] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting natural scene generation videos based on spectral volume, characterized in that, include: Obtain a sample video set, which includes multiple sample videos with preset authenticity labels; Based on a preset extraction strategy, multiple sample videos are preprocessed to extract the first spectral volume features; The first pixel feature and the first motion feature are obtained by extracting features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network; Using the first pixel feature and the first motion feature as samples, a video detection model is obtained by training a convolutional neural network. The target video information to be detected is obtained, and the target video information is preprocessed using the extraction strategy to obtain the second spectral volume feature; The video detection model is used to extract features from the target video information and the second spectral volume feature to obtain the second pixel feature and the second motion feature. A classifier is designed to analyze the second pixel features and the second motion features to obtain the probability characterizing the category of the target video information, and to determine whether the target video information is a generated video. The method based on a preset extraction strategy preprocesses multiple sample videos to extract first spectral volume features, including: The sample video was extracted as optical flow to preliminarily determine motion texture; The optical flow is transformed into a frequency domain representation by performing a Fourier transform in the time dimension to obtain a first spectral volume feature; wherein, the first spectral volume feature is the time Fourier transform of the trajectory of each pixel extracted from the sample video.

2. The method for detecting natural scene generation videos based on spectral volume as described in claim 1, characterized in that, The sample video set includes real videos and various generated videos. Each generated video corresponds to a different video generation algorithm, and each generated video is determined using text or images based on the corresponding video generation algorithm.

3. The method for detecting natural scene generation videos based on spectral volume as described in claim 1, characterized in that, The dual-stream network includes a pixel feature extraction network and a motion feature extraction network. The pixel feature extraction network is a pre-trained network and is either 3D ResNet or Video ViT. The motion feature extraction network is trained on a preset basic dataset and for action recognition tasks, so that the motion feature extraction network has the ability to extract motion features and learn the general motion information of spectral volume in different videos. The underlying datasets include at least one of the Kinetics-400 dataset and the UCF101 dataset.

4. The natural scene generation video detection method based on spectral volume as described in claim 3, characterized in that, The step of extracting features from the sample video and the first spectral volume feature using the acquired dual-stream structure network to obtain the first pixel feature and the first motion feature includes: The first pixel feature is obtained by extracting features from the sample video using the pixel feature extraction network. The first motion feature is obtained by extracting features from the first spectral volume feature using the motion feature extraction network. Wherein, the first pixel feature and the first motion feature are both one-dimensional feature vectors. The first pixel feature includes color, illumination and texture information, and the first motion feature includes motion information of the camera and objects in the video.

5. The method for detecting natural scene generation videos based on spectral volume as described in claim 1, characterized in that, The step of using the first pixel feature and the first motion feature as samples, and training based on a convolutional neural network to obtain a video detection model includes: The first pixel feature and the first motion feature are fused using a multi-head cross-attention mechanism to generate fused features that represent the common information of the first pixel feature and the first motion feature. Based on the fusion features, the detection result corresponding to each sample video in the sample video set is determined; Based on the detection results and corresponding labels of each sample video, a video detection model is obtained through iterative training using a convolutional neural network.

6. The method for detecting natural scene generation videos based on spectral volume as described in claim 5, characterized in that, The process of obtaining a video detection model based on the detection results and corresponding labels of each sample video, using a convolutional neural network for iterative training, includes: During the training process to obtain the initial detection model, the detection loss corresponding to the initial detection model is calculated; Based on the detection loss, the initial detection model is iteratively trained until a predetermined training termination condition is met, resulting in an optimized video detection model.

7. A natural scene generation video detection system based on spectral volume, characterized in that, include: The acquisition module is used to acquire a sample video set, which includes multiple sample videos with preset authenticity labels; The first preprocessing module is used to preprocess multiple sample videos based on a preset extraction strategy to extract the first spectral volume feature. The first feature extraction module is used to extract features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network to obtain the first pixel feature and the first motion feature; The training module is used to train a video detection model based on a convolutional neural network using the first pixel feature and the first motion feature as samples. The second preprocessing module is used to acquire the target video information to be detected, and to preprocess the target video information through the extraction strategy to obtain the second spectral volume feature; The second feature extraction module is used to extract features from the target video information and the second spectral volume feature through the video detection model to obtain the second pixel feature and the second motion feature; The judgment module is used to design a classifier to analyze the second pixel features and the second motion features, obtain the probability characterizing the category of the target video information, and determine whether the target video information is a generated video. The method based on a preset extraction strategy preprocesses multiple sample videos to extract first spectral volume features, including: The sample video was extracted as optical flow to preliminarily determine motion texture; The optical flow is transformed into a frequency domain representation by performing a Fourier transform in the time dimension to obtain a first spectral volume feature; wherein, the first spectral volume feature is the time Fourier transform of the trajectory of each pixel extracted from the sample video.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the natural scene generation video detection method based on spectral volume as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the natural scene generation video detection method based on spectral volume as described in any one of claims 1 to 6.

Images