Natural scene generation video detection method and system based on spectral volume

Abstract

The invention provides a natural scene generation video detection method and system based on spectral volume, and relates to the technical field of video detection, the method comprises the following steps: obtaining a sample video set, the sample video set comprising a plurality of sample videos with preset true and false labels; preprocessing the plurality of sample videos, performing feature extraction through a double-flow structure network, and performing training based on a convolutional neural network to obtain a video detection model; obtaining to-be-detected target video information, and performing preprocessing through the extraction strategy; and feature extraction is carried out through the video detection model, a classifier is designed for analysis, the probability representing the category of the target video information is obtained, and whether the target video information is a generated video is judged. According to the method, in the model training stage, pixel features and motion features are concerned, the motion representation method of the spectral volume is used for further representing the generation clues, the calculation overhead is reduced, and the defects of generated video detection in the aspect of natural scenes are overcome.

Description

Technical Field

[0001] The present application relates to the field of video detection technology, and in particular to a method and system for generating video detection of natural scenes based on spectral volumes. Background Art

[0002] Generative AI technology has developed rapidly since the advent of ChatGPT. Content media such as text, voice, pictures and videos can all be generated using AI. For example, OpenAI's sora has demonstrated extremely high video generation capabilities. Users can use text, pictures and action skeletons as conditions to generate high-quality videos. However, the application of such video generation models is very simple, especially text-generated videos. Users only need to use text descriptions to generate a video. Although the creativity of video generation models can provide entertainment and commercial value, their simplicity and ease of use can also easily generate videos with false content, endangering the security of digital media. Therefore, it is necessary to detect AI-generated videos.

[0003] Current video generation models have achieved results close to real videos in producing videos of natural scenes, and there are relatively few motion inconsistencies. The reason for this phenomenon can be attributed to the fact that most motion patterns in natural scenes are periodic or vibrating, and these regular motion patterns are easier to capture and learn by neural networks. Therefore, the performance of the generation model is better when processing natural scenes.

[0004] However, existing video detection methods often lack sufficient pertinence when faced with generated videos of natural scenes. These methods mainly rely on the pixel information of the video to find detection clues, or use optical flow information to extract motion features in the video. Although optical flow information can provide useful clues about the motion of pixels, its computational cost is high and is directly related to the length of the video, which limits its application in large-scale video detection. Summary of the invention

[0005] In view of the deficiencies of the prior art, the present application provides a natural scene generated video detection method and system based on spectral volume, which solves the deficiencies of the existing generated video detection technology in terms of natural scenes and the problems of unsatisfactory detection accuracy and reliability.

[0006] To achieve the above objectives, this application is implemented through the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a natural scene generated video detection method based on spectral volume, and the natural scene generated video detection method includes: obtaining a sample video set, the sample video set including multiple sample videos with preset authenticity labels; based on a preset extraction strategy, preprocessing the multiple sample videos to extract a first spectral volume feature; performing feature extraction from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network to obtain a first pixel feature and a first motion feature; using the first pixel feature and the first motion feature as samples, training based on a convolutional neural network to obtain a video detection model; obtaining target video information to be detected, preprocessing the target video information through the extraction strategy to obtain a second spectral volume feature; performing feature extraction on the target video information and the second spectral volume feature through the video detection model to obtain a second pixel feature and a second motion feature; designing a classifier to analyze the second pixel feature and the second motion feature to obtain a probability characterizing the category of the target video information, and judging whether the target video information is a generated video.

[0008] According to a first aspect of an embodiment of the present application, a sample video set includes real videos and multiple generated videos, the multiple generated videos correspond one-to-one to multiple video generation algorithms, and each generated video is generated and determined based on a corresponding video generation algorithm using text or pictures.

[0009] According to the first aspect of the embodiment of the present application, the aforementioned preprocessing of multiple sample videos based on a preset extraction strategy to extract a first spectral volume feature may specifically include the following steps: extracting the sample video as an optical flow to preliminarily determine the motion texture; performing Fourier transform on the optical flow in the time dimension into a frequency domain representation to obtain a first spectral volume feature; wherein the first spectral volume feature is a time Fourier transform of the per-pixel trajectory extracted from the sample video.

[0010] According to a first aspect of an embodiment of the present application, the dual-stream structure network includes a pixel feature extraction network and a motion feature extraction network. The pixel feature extraction network is a pre-trained network and is one of 3DResNet and Video ViT.

[0011] According to a first aspect of an embodiment of the present application, a motion feature extraction network is based on a preset basic data set and is trained for an action recognition task, so that the motion feature extraction network has the ability to extract motion features and learn the general motion information of spectral volumes in different videos.

[0012] According to a first aspect of an embodiment of the present application, the basic data set includes at least one of a Kinetics-400 data set and a UCF101 data set.

[0013] According to the first aspect of the embodiment of the present application, the aforementioned dual-stream structure network obtained by respectively extracting features from the sample video and the first spectral volume feature to obtain a first pixel feature and a first motion feature can specifically include the following steps: performing feature extraction on the sample video through a pixel feature extraction network to obtain a first pixel feature; performing feature extraction on the first spectral volume feature through a motion feature extraction network to obtain a first motion feature; wherein the first pixel feature and the first motion feature are both one-dimensional feature vectors, the first pixel feature includes color, lighting and texture information, and the first motion feature includes motion information of the lens and objects in the video.

[0014] According to the first aspect of the embodiment of the present application, the aforementioned method of using the first pixel feature and the first motion feature as samples and training based on a convolutional neural network to obtain a video detection model can specifically include the following steps: using a multi-head cross-attention mechanism to fuse the first pixel feature and the first motion feature, and generating a fused feature to characterize the common information of the first pixel feature and the first motion feature; based on the fused feature, determining the detection result corresponding to each sample video in the sample video set; based on the detection result of each sample video and the corresponding label, iterative training is performed based on the convolutional neural network to obtain a video detection model.

[0015] According to the first aspect of the embodiments of the present application, the aforementioned detection results based on each sample video and the corresponding labels are iteratively trained based on a convolutional neural network to obtain a video detection model, which may specifically include the following steps: in the process of training to obtain the initial detection model, calculating the detection loss corresponding to the initial detection model; based on the detection loss, iteratively training the initial detection model until a predetermined training termination condition is met to obtain an optimized video detection model.

[0016] In a second aspect, an embodiment of the present application provides a natural scene generation video detection system based on spectral volume, and the natural scene generation video detection system includes: an acquisition module, a first preprocessing module, a first feature extraction module, a training module, a second preprocessing module, a second feature extraction module and a judgment module; wherein the acquisition module is used to acquire a sample video set, and the sample video set includes a plurality of sample videos with preset authenticity labels; the first preprocessing module is used to preprocess the plurality of sample videos based on a preset extraction strategy to extract a first spectral volume feature; the first feature extraction module is used to extract features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network, and obtain to the first pixel feature and the first motion feature; the training module is used to use the first pixel feature and the first motion feature as samples, and perform training based on the convolutional neural network to obtain a video detection model; the second preprocessing module is used to obtain the target video information to be detected, and preprocess the target video information through the extraction strategy to obtain the second spectral volume feature; the second feature extraction module is used to extract the target video information and the second spectral volume feature through the video detection model to obtain the second pixel feature and the second motion feature; the judgment module is used to design a classifier to analyze the second pixel feature and the second motion feature, obtain the probability of characterizing the category of the target video information, and judge whether the target video information is a generated video.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for generating video detection for natural scenes based on spectral volume in the aforementioned first aspect is implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a program or instruction. When the program or instruction is executed by a processor, the method for generating video detection of natural scenes based on spectral volume in the aforementioned first aspect is implemented.

[0019] This application provides a natural scene generation video detection method and system based on spectral volume. Compared with the prior art, it has the following beneficial effects:

[0020] The present application is trained based on a set of sample videos with preset authenticity labels, extracts the first spectral volume feature from each sample video, processes the sample video and the first spectral volume feature respectively through a dual-stream structure network, obtains the first pixel feature and the first motion feature to train a video detection model, and through the collaborative work of this dual-stream structure network, it is possible to comprehensively analyze the video content and provide rich feature information for video detection. In the model training stage, the present application focuses on two aspects: pixel features and motion features, and uses the motion representation method of the spectral volume to further represent the generation clues, reducing the computational overhead and solving the shortcomings of the generated video detection in natural scenes. In the process of video detection, the corresponding second spectral volume feature is also extracted based on the extraction strategy, and feature extraction is performed through the obtained video detection model to further detect and judge the generated video through the classifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flowchart of a natural scene generation video detection method based on spectral volume provided in an embodiment of the present application;

[0023] Figure 2 yes Figure 1 An exemplary flow chart of S140;

[0024] Figure 3 It is a structural schematic diagram of a natural scene generation video detection system based on spectral volume provided in an embodiment of the present application;

[0025] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0028] The embodiments of the present application provide a method and system for natural scene generated video detection based on spectral volume, thereby solving the problems of the shortcomings of existing generated video detection technology in terms of natural scenes and the unsatisfactory detection accuracy and reliability.

[0029] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0030] With the rapid development of deep learning technology, video generation technology has achieved a qualitative leap, and the video content created has become more realistic. This has greatly enriched people's entertainment life and stimulated creativity, but it has also inevitably brought security risks, especially the proliferation of false information. Therefore, it is of great significance to carry out the detection of generated videos.

[0031] At present, the field of generative video detection mainly uses technologies such as 3D convolutional neural networks or visual transformer (ViT) models to directly extract features from videos for detection. These methods are effective when processing data in the same domain as the training data, but they are insufficient in generalization, especially when facing unknown generation methods. This is because existing methods mainly focus on pixel features in the video, but do not capture enough motion change features in the generated video. Since the current video generation model still needs to be improved in terms of motion consistency, by capturing motion changes and extracting generation clues, the detection model can discover generation clues that were not originally found in pixel features. Therefore, researchers began to try to use optical flow maps to represent the motion information of the video, and extract motion features from the optical flow maps to enhance the generalization ability of the detection model. Optical flow maps can provide motion information of pixels between consecutive frames, which is crucial for understanding motion changes in videos. However, the computational and storage overhead of optical flow maps is large, and videos of different lengths need to be normalized, which limits their application in large-scale training.

[0032] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0033] The following first introduces a natural scene generation video detection method based on spectral volume provided in an embodiment of the present application.

[0034] The present application embodiment provides a flow chart of a natural scene generation video detection method based on spectral volume, such as Figure 1 As shown, the natural scene generated video detection method may include the following steps S110-S170.

[0035] S110: Obtain a sample video set, where the sample video set includes a plurality of sample videos with preset authenticity labels.

[0036] S120, based on a preset extraction strategy, preprocess the plurality of sample videos to extract a first spectral volume feature. It is understandable that, in order to better obtain motion features, a motion representation representing the vibration of an object, i.e., the first spectral volume feature, is obtained from the sample video. Compared with using optical flow as a motion feature, the first spectral volume feature selects K output frequencies, so that data with the same length as the entire video does not need to be used as input, thereby reducing overhead.

[0037] S130 , extracting features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network to obtain a first pixel feature and a first motion feature.

[0038] S140: Using the first pixel feature and the first motion feature as samples, training is performed based on a convolutional neural network to obtain a video detection model.

[0039] S150 , obtaining target video information to be detected, and preprocessing the target video information through an extraction strategy to obtain a second spectral volume feature.

[0040] S160 , extracting features from the target video information and the second spectral volume features through a video detection model to obtain second pixel features and second motion features.

[0041] S170, designing a classifier to analyze the second pixel feature and the second motion feature, obtain the probability of characterizing the category of the target video information, and determine whether the target video information is a generated video. It can be understood that the extracted second pixel feature and the second motion feature contain the difference between the real video and the generated video, and the classifier classifies the one-dimensional feature to determine the detection result of the target video information to be detected.

[0042] The above is a specific implementation method of a natural scene generation video detection method based on spectral volume provided in an embodiment of the present application. It can be understood that the present application is trained based on a set of sample videos with preset authenticity labels, extracts the first spectral volume feature from each sample video, and processes the sample video and the first spectral volume feature respectively through a dual-stream structure network to obtain the first pixel feature and the first motion feature to train a video detection model. Through the collaborative work of this dual-stream structure network, the video content can be comprehensively analyzed to provide rich feature information for video detection.

[0043] It should be noted that, in the model training stage, the present application focuses on two aspects, pixel features and motion features, and uses the motion representation method of the spectral volume to further represent the generation clues, reducing the computational overhead and solving the shortcomings of the generated video detection in natural scenes. In the process of video detection, the corresponding second spectral volume features are also extracted based on the extraction strategy, and feature extraction is performed through the obtained video detection model to further detect and judge the generated video through the classifier.

[0044] In some embodiments, the sample video set includes real videos and multiple generated videos, the multiple generated videos correspond one-to-one to multiple video generation algorithms, and each generated video is determined based on the corresponding video generation algorithm using text or pictures.

[0045] In the embodiments of the present application, it is understood that real videos and videos generated by various video generation algorithms are used as samples to train the initial generated video detection model. This process enables the model to recognize and learn the specific features of the generated video and the differences between the real video and the generated video.

[0046] These generated videos correspond to a variety of video generation algorithms, each of which can directly generate videos through specific prompting methods. For example, text-driven and image-driven video generation techniques can generate videos based on text descriptions or image content. In practical applications, the video generation algorithms involved may include Modelscope algorithm, Zeroscope algorithm, Stable video diffusion algorithm, and I2VGen-XL algorithm, each of which represents a different video generation technology and method. The videos generated by these algorithms can provide rich training samples for the video detection model to improve its ability to detect generated videos.

[0047] In one example, the aforementioned preprocessing of the plurality of sample videos based on a preset extraction strategy to extract the first spectral volume feature, that is, the aforementioned S120 may specifically include the following steps:

[0048] S210, extracting the sample video as an optical flow to preliminarily determine the motion texture;

[0049] S220, performing Fourier transform on the optical flow in the time dimension into a frequency domain representation to obtain a first spectral volume feature; wherein the first spectral volume feature is a time Fourier transform of a per-pixel trajectory extracted from the sample video.

[0050] In the embodiment of the present application, it can be understood that the motion texture is a series of 2D displacement maps that change over time. In order to better extract the first spectral volume feature, the optical flow method is used to extract the video motion texture, which describes the pattern of pixel movement on the surface of the object in the image sequence. It focuses on the displacement description in the video, discards the color, lighting and other information in the video, which is conducive to extracting the motion signal in the video.

[0051] It should be noted that videos are generated for natural scenes, where many natural motions can be described as the superposition of a small number of harmonic oscillations represented by different frequencies, amplitudes, and phases. Because these underlying motions are quasi-periodic, it is natural to model them in the frequency domain. The spectral volume is the temporal Fourier transform of the per-pixel trajectory extracted from the video, which can represent the quasi-periodic motion present in natural scenes.

[0052] In some embodiments, the aforementioned two-stream structure network is used to extract features from the sample video and the first spectral volume feature respectively to obtain the first pixel feature and the first motion feature, that is, the aforementioned S130 may specifically include the following steps:

[0053] S310, extracting features from the sample video through a pixel feature extraction network to obtain a first pixel feature;

[0054] S320, extracting the first spectral volume feature through a motion feature extraction network to obtain a first motion feature;

[0055] The first pixel feature and the first motion feature are both one-dimensional feature vectors. The first pixel feature includes color, illumination and texture information, and the first motion feature includes motion information of the lens and the object in the video.

[0056] In the embodiment of the present application, it can be understood that the present application first extracts first pixel features from the sample video, and these features can reveal the characteristics of the sample video in the spatial dimension, including color, illumination, and texture, etc. The motion features in the sample video are extracted by analyzing the first spectral volume, and these features can capture the regularity of the motion of the object in the sample video.

[0057] This application uses two specialized networks to extract pixel features and motion features respectively. The pixel feature extraction network is responsible for obtaining visual appearance features such as color, lighting, and texture from the video. The motion feature extraction network processes spectral volume data to capture dynamic features such as object motion and lens motion in the video. Through the collaborative work of this dual-stream structure network, the video content can be fully analyzed to provide rich feature information for video detection.

[0058] It should be noted that in practical applications, the feature extraction network can use any appropriate network. Optionally, since the 3D ResNet network can directly process video frames or time series data, capture pixel-level spatial features and inter-frame temporal features, the pixel feature extraction network and the motion feature extraction network can use the same 3D ResNet structure, thereby effectively extracting pixel features and motion features in the video.

[0059] In some embodiments, the dual-stream structure network includes a pixel feature extraction network and a motion feature extraction network, the pixel feature extraction network is a pre-trained network and is one of 3D ResNet and Video ViT;

[0060] The motion feature extraction network is based on a preset basic data set and is trained for action recognition tasks so that the motion feature extraction network has the ability to extract motion features and learn the general motion information of spectral volumes in different videos; the basic data set includes at least one of the Kinetics-400 data set and the UCF101 data set.

[0061] In the embodiment of the present application, it can be understood that the dual-stream structure network includes two feature extraction networks, and both feature extraction networks enhance their feature extraction performance through pre-training. For the pixel feature extraction network, an existing pre-trained network, such as 3D ResNet or Video ViT, can be used. These networks have been applied in a wide range of video tasks. For the spectral volume network, due to the lack of a ready-made pre-training model, this embodiment performs special pre-training. Through pre-training on action recognition tasks, its performance in extracting action features can be significantly improved. To this end, data sets such as Kinetics-400 and UCF101 can be selected for pre-training. These data sets contain rich action information, which helps the network learn more discriminative action features. Through such a pre-training strategy, the efficiency and accuracy of the two networks in feature extraction can be ensured.

[0062] In some embodiments, Figure 2 As shown, the first pixel feature and the first motion feature are used as samples, and training is performed based on a convolutional neural network to obtain a video detection model, that is, the aforementioned S140 may specifically include the following steps:

[0063] S410, using a multi-head cross attention mechanism to fuse the first pixel feature and the first motion feature to generate a fused feature to represent common information of the first pixel feature and the first motion feature;

[0064] S420, determining a detection result corresponding to each sample video in the sample video set based on the fusion feature;

[0065] S430: Based on the detection result and corresponding label of each sample video, iterative training is performed based on a convolutional neural network to obtain a video detection model.

[0066] In an embodiment of the present application, a convolutional neural network uses a common residual neural network to extract pixel-level features and motion features from video frames. Pixel features include information such as color, lighting, and texture, while motion features may include the object's motion trajectory and vibration frequency. There are potential differences between real videos and generated videos in both visual appearance and action. Elements such as color, lighting, and texture of the video constitute the visual appearance of the video, while the movement of the object and the movement of the lens constitute the action of the video.

[0067] It is understandable that the multi-head cross attention mechanism is composed of multiple attention layers, which aims to capture the correlation between the first pixel feature and the first motion feature, so as to extract the common information between the two modalities in a wider range. By combining the first pixel feature and the first motion feature, this application helps to avoid the excessive reliance of the video detection model on a single feature, thereby reducing the risk of overfitting; it is conducive to improving the accuracy of video detection and enhancing the model's generalization ability for unknown video content.

[0068] It should be noted that the multi-head cross-attention mechanism can pay attention to multiple positions in the sequence in parallel when processing sequence data, thereby effectively integrating the information of the video at the pixel level and the motion level. This method can more accurately identify signs of forgery in generated videos. Compared with the method of simply splicing two features, feature fusion technology makes the model more robust when processing video data. This is because different features can capture different aspects of the video, and when a feature is not enough to provide sufficient information, other features can provide supplements. This complementarity enhances the robustness of the model and reduces the overfitting phenomenon of single modality features, further improving the performance and generalization ability of the model in video detection tasks.

[0069] In an example, the detection results and corresponding labels of each sample video are iteratively trained based on a convolutional neural network to obtain a video detection model, that is, the aforementioned S430 may specifically include: in the process of training to obtain the initial detection model, calculating the detection loss corresponding to the initial detection model; based on the detection loss, iteratively training the initial detection model until a predetermined training termination condition is met to obtain an optimized video detection model.

[0070] In the specific implementation of model training, the initial detection model is first used to extract pixel features from the video, which reflect the details of the video appearance. Then, by converting the video into a spectral volume as a representation of motion information, the model can extract motion features from the sample based on the spectral volume. Furthermore, the model is continuously iterated to train using the detection results and actual labels in the sample video collection to optimize the initial detection model. This process not only makes the model proficient in capturing the pixel features of the video, but also allows it to master the skills of parsing the dynamic characteristics of the video.

[0071] Specifically, by using the detection results and true and false labels of each sample video in the sample video set, combined with the preset loss function, the detection loss of the initial video detection model can be calculated. Subsequently, based on this detection loss, the initial generated video detection model is iteratively trained until the established training stop condition is met, and finally the optimized generated video detection model is obtained.

[0072] Exemplarily, in the embodiments of the present application, three preset loss functions are defined. The first loss function is used to measure the performance loss of the model when generating video detection based on pixel features; the second loss function is used to evaluate the performance loss of the model when detecting based on motion features; the third loss function is for the performance loss of the model when detecting based on fusion features. Finally, by taking the weighted sum of these three preset losses, the total loss of the initial generated video detection model is obtained, which is used as the optimization target for model training. This method helps to balance the contribution of different features in the detection process and ensure the comprehensive performance of the model on various features.

[0073] In some embodiments, the present application provides a natural scene generation video detection system 500 based on spectral volume, such as Figure 3 As shown, the natural scene generation video detection system 500 may include the following modules:

[0074] An acquisition module 510 is used to acquire a sample video set, where the sample video set includes a plurality of sample videos with preset authenticity labels;

[0075] A first preprocessing module 520, configured to preprocess the plurality of sample videos based on a preset extraction strategy to extract a first spectral volume feature;

[0076] A first feature extraction module 530, configured to extract features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network to obtain a first pixel feature and a first motion feature;

[0077] A training module 540 is used to perform training based on a convolutional neural network using the first pixel feature and the first motion feature as samples to obtain a video detection model;

[0078] A second preprocessing module 550 is used to obtain target video information to be detected, and preprocess the target video information through an extraction strategy to obtain a second spectral volume feature;

[0079] A second feature extraction module 560 is used to extract the target video information and the second spectral volume feature through a video detection model to obtain a second pixel feature and a second motion feature;

[0080] The judgment module 570 is used to design a classifier to analyze the second pixel feature and the second motion feature, obtain the probability of characterizing the category of the target video information, and judge whether the target video information is a generated video.

[0081] According to an embodiment of the present application, any multiple modules among the acquisition module 510, the first preprocessing module 520, the first feature extraction module 530, the training module 540, the second preprocessing module 550, the second feature extraction module 560 and the judgment module 570 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0082] In some embodiments, the first pre-processing module 520 may be specifically used to:

[0083] Extract the sample video as optical flow to preliminarily determine the motion texture;

[0084] The optical flow is Fourier transformed into a frequency domain representation in the time dimension to obtain a first spectral volume feature; wherein the first spectral volume feature is a time Fourier transform of a per-pixel trajectory extracted from the sample video.

[0085] In some embodiments, the first feature extraction module 530 may be specifically used for:

[0086] Extract features from the sample video using a pixel feature extraction network to obtain a first pixel feature;

[0087] Extracting the first spectral volume feature through a motion feature extraction network to obtain a first motion feature;

[0088] The first pixel feature and the first motion feature are both one-dimensional feature vectors. The first pixel feature includes color, illumination and texture information, and the first motion feature includes motion information of the lens and the object in the video.

[0089] In some embodiments, the training module 540 may be specifically used to:

[0090] A multi-head cross attention mechanism is used to fuse the first pixel feature and the first motion feature to generate a fused feature to represent the common information of the first pixel feature and the first motion feature;

[0091] Based on the fusion features, determine the detection result corresponding to each sample video in the sample video set;

[0092] Based on the detection results and corresponding labels of each sample video, iterative training is performed based on the convolutional neural network to obtain a video detection model.

[0093] Figure 3 Each module in the system shown has the function of implementing each step in the aforementioned natural scene generation video detection method based on spectral volume, and can achieve its corresponding technical effect. For the sake of concise description, it will not be repeated here.

[0094] In some embodiments, the present application provides an electronic device, the structure diagram of the electronic device is as follows Figure 4 shown.

[0095] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0096] Specifically, the processor 610 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0097] The memory 620 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 620 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 620 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 620 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 620 is a non-volatile solid-state memory.

[0098] The memory 620 may include a read-only memory (ROM), a random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the above-mentioned embodiments of the natural scene generation video detection method based on spectral volume.

[0099] The processor 610 implements any one of the natural scene generation video detection methods based on spectral volume in the above embodiments by reading and executing computer program instructions stored in the memory 620 .

[0100] In one example, the electronic device may further include a communication interface 630 and a bus 600. Figure 4 As shown, the processor 610, the memory 620, and the communication interface 630 are connected via a bus 600 and communicate with each other.

[0101] The communication interface 630 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0102] Bus 600 includes hardware, software or both, and the parts of online data flow billing equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front-end bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 600 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0103] In addition, in combination with the natural scene generated video detection method based on spectral volume in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the natural scene generated video detection methods based on spectral volume in the above embodiment is implemented.

[0104] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0105] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0106] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0107] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0108] In summary, compared with the prior art, this application has the following beneficial effects:

[0109] 1. This application is based on training on a set of sample videos with preset authenticity labels, extracts a first spectral volume feature from each sample video, processes the sample video and the first spectral volume feature respectively through a dual-stream structure network, obtains a first pixel feature and a first motion feature to train a video detection model, and through the collaborative work of this dual-stream structure network, it is possible to comprehensively analyze the video content and provide rich feature information for video detection.

[0110] 2. During the model training stage, this application focuses on two aspects: pixel features and motion features. It uses the motion representation method of spectral volume to further represent the generation clues, adopts a multi-head cross-attention mechanism to process sequence data, and pays parallel attention to multiple positions in the sequence, thereby effectively integrating the video information at the pixel level and the motion level.

[0111] 3. In the specific implementation of model training, the application uses an initial detection model to extract pixel features from the video, which reflect the details of the video appearance. Then, by converting the video into a spectral volume as a representation of motion information, the model can extract action features from the sample based on the spectral volume. The model is trained by continuous iteration using the detection results and actual labels in the sample video collection. This process not only makes the model proficient in capturing the pixel features of the video, but also allows it to master the skills of parsing the dynamic characteristics of the video.

[0112] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A natural scene generation video detection method based on spectral volume, characterized in that: include: Acquire a sample video set, wherein the sample video set includes a plurality of sample videos with preset authenticity labels; Based on a preset extraction strategy, preprocessing the plurality of sample videos to extract a first spectral volume feature; Extracting features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network to obtain a first pixel feature and a first motion feature; Using the first pixel feature and the first motion feature as samples, training is performed based on a convolutional neural network to obtain a video detection model; Acquire target video information to be detected, and preprocess the target video information using the extraction strategy to obtain a second spectral volume feature; Extracting features from the target video information and the second spectral volume features by using the video detection model to obtain second pixel features and second motion features; A classifier is designed to analyze the second pixel feature and the second motion feature to obtain a probability representing a category of the target video information, and to determine whether the target video information is a generated video.

2. The natural scene generation video detection method based on spectral volume according to claim 1, characterized in that: The sample video set includes real videos and multiple generated videos, wherein the multiple generated videos correspond one-to-one to multiple video generation algorithms, and each of the generated videos is generated and determined based on the corresponding video generation algorithm using text or pictures.

3. The natural scene generation video detection method based on spectral volume according to claim 1, characterized in that: The preprocessing of the plurality of sample videos based on a preset extraction strategy to extract a first spectral volume feature includes: Extracting the sample video as optical flow to preliminarily determine motion texture; Performing Fourier transform on the optical flow in the time dimension into a frequency domain representation to obtain a first spectral volume feature; wherein the first spectral volume feature is a time Fourier transform of a per-pixel trajectory extracted from the sample video.

4. The natural scene generation video detection method based on spectral volume according to claim 1, characterized in that: The dual-stream structure network includes a pixel feature extraction network and a motion feature extraction network, wherein the pixel feature extraction network is a pre-trained network and is one of 3D ResNet and Video ViT; The motion feature extraction network is based on a preset basic data set and trained for action recognition tasks, so that the motion feature extraction network has the ability to extract motion features and learn the general motion information of spectral volumes in different videos; The basic dataset includes at least one of a Kinetics-400 dataset and a UCF101 dataset.

5. The natural scene generation video detection method based on spectral volume as claimed in claim 4, characterized in that: The method of extracting features from the sample video and the first spectral volume feature respectively through the obtained dual-stream structure network to obtain a first pixel feature and a first motion feature includes: Extracting features from the sample video using the pixel feature extraction network to obtain a first pixel feature; Extracting the first spectral volume feature through the motion feature extraction network to obtain a first motion feature; The first pixel feature and the first motion feature are both one-dimensional feature vectors, the first pixel feature includes color, lighting and texture information, and the first motion feature includes motion information of the lens and the object in the video.

6. The natural scene generation video detection method based on spectral volume according to claim 1, characterized in that: The method of using the first pixel feature and the first motion feature as samples and performing training based on a convolutional neural network to obtain a video detection model includes: Using a multi-head cross attention mechanism to fuse the first pixel feature and the first motion feature, generating a fused feature to represent common information of the first pixel feature and the first motion feature; Based on the fusion feature, determining a detection result corresponding to each of the sample videos in the sample video set; Based on the detection result and corresponding label of each sample video, iterative training is performed based on a convolutional neural network to obtain a video detection model.

7. The natural scene generation video detection method based on spectral volume according to claim 6, characterized in that: The detection result and corresponding label of each sample video are used to perform iterative training based on a convolutional neural network to obtain a video detection model, including: In the process of training the initial detection model, calculating the detection loss corresponding to the initial detection model; Based on the detection loss, the initial detection model is iteratively trained until a predetermined training termination condition is met, thereby obtaining an optimized video detection model.

8. A natural scene generation video detection system based on spectral volume, characterized in that: include: An acquisition module, used to acquire a sample video set, wherein the sample video set includes a plurality of sample videos with preset authenticity labels; A first preprocessing module, configured to preprocess the plurality of sample videos based on a preset extraction strategy to extract a first spectral volume feature; A first feature extraction module, used to extract features from the sample video and the first spectral volume feature respectively through the acquired dual-stream structure network to obtain a first pixel feature and a first motion feature; A training module, configured to use the first pixel feature and the first motion feature as samples, perform training based on a convolutional neural network, and obtain a video detection model; A second preprocessing module is used to obtain target video information to be detected, and preprocess the target video information through the extraction strategy to obtain a second spectral volume feature; A second feature extraction module, used for performing feature extraction on the target video information and the second spectral volume feature through the video detection model to obtain a second pixel feature and a second motion feature; A judgment module is used to design a classifier to analyze the second pixel feature and the second motion feature, obtain a probability representing the category of the target video information, and judge whether the target video information is a generated video.

9. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the method for generating video detection of natural scene based on spectral volume as claimed in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the natural scene generation video detection method based on spectral volume is implemented as described in any one of claims 1 to 7.

Images