Generated video detection method and device, electronic equipment and storage medium

By performing discrete cosine transformation processing and time-frequency domain fusion on the video image, and combining with the Transformers model for video detection, the detection generalization problem in the prior art is solved, and the rapid and accurate detection of videos of different generation methods is achieved.

CN119011905APending Publication Date: 2024-11-22BEIJING YUANJIAN INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411047978.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Currently, CNN-trained detectors are difficult to effectively detect videos generated by generative adversarial networks (GANs) and diffusion models (DMs), resulting in generalization of video detection under different generation methods.

Method used

By performing discrete cosine transformation processing on video images, processing on time domain and frequency domain branches, integrating time domain and frequency domain information, using the video detection model obtained by iterative training of Transformers model for classification prediction, and determining whether the video is a generated video .

Benefits of technology

The accuracy of video detection under different generation methods is achieved, and videos generated by unseen generation methods can be better processed, improving the rapid accuracy of detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011905A_ABST
    Figure CN119011905A_ABST
Patent Text Reader

Abstract

The invention provides a generated video detection method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the discrete cosine transform processing of each frame of video image in a to-be-detected video, and determining each frame of video image after the discrete cosine transform processing; synchronously inputting the first feature of each frame of video image after discrete cosine transform processing and the first feature of each frame of video image into a detection network layer, and performing time domain branch processing on the first feature of each frame of video image, performing frequency domain branch processing on the first feature of each frame of video image after discrete cosine transform processing, fusing the time domain feature of each frame and the frequency domain feature of each frame, and outputting a fusion vector of each frame; and performing classification prediction processing on the fusion vector of each frame, and outputting whether the to-be-detected video is a generated video or not. Whether the to-be-detected video is the generated video or not can be quickly and accurately determined, and the video which is not generated by the generation method can be better processed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of video authenticity detection, and in particular to a detection method, device, electronic device and storage medium for generating a video. Background Art

[0002] With the rapid development of AI technology, image generation technology has become more and more mature, even to the point where it is difficult for the human eye to distinguish true from false. If these generated images are used maliciously, they may cause a variety of security issues, which has also led to the increasing demand for generated image detection technology. The current mainstream generation methods include generative adversarial networks (GAN) and diffusion models (DM). Due to the different generation methods of these two methods, it is difficult for current CNN-trained detectors to solve the generalization problem of detecting images generated by these two methods. Therefore, how to accurately detect generated videos under different generation methods has become a technical problem that cannot be underestimated. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a detection method, device, electronic device and storage medium for generated videos, by utilizing a video detection model to process the video image processed by discrete cosine transform and the RGB video image in the time domain branch and the frequency domain branch, and finally weightedly fuse the time domain and frequency domain information to obtain a fusion vector, and then classify the fusion vector to quickly and accurately determine whether the video to be detected is a generated video, and can better process videos generated by unseen generation methods.

[0004] The present application embodiment provides a detection method for generating a video, the detection method comprising:

[0005] Acquire a video to be detected, perform discrete cosine transform processing on each frame of video image in the video to be detected, and determine the video image after discrete cosine transform processing of each frame;

[0006] The first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are synchronously input into the detection network layer of the pre-trained video detection model, the first feature of each frame of the video image is processed by time domain branching, the first feature of each frame of the video image processed by discrete cosine transform is processed by frequency domain branching, and the time domain feature of each frame and the frequency domain feature of each frame are fused to output the fusion vector of each frame; the video detection model is obtained by iterative training of the Transformers model;

[0007] The fusion vector of each frame is input into the classification network layer of the video detection model, and the fusion vector of each frame is classified and predicted, and whether the video to be detected is a generated video is output, and if so, the generation interval of the video to be detected is output.

[0008] In a possible implementation manner, the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are synchronously input into the detection network layer of the pre-trained video detection model, the first feature of each frame of the video image is processed by time domain branching, the first feature of each frame of the video image processed by discrete cosine transform is processed by frequency domain branching, and the time domain feature of each frame and the frequency domain feature of each frame are fused to output the first feature of the fusion vector of each frame, including:

[0009] Normalizing and multi-head attention processing are respectively performed on the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image;

[0010] Inputting the plurality of first target features into the RGB image feedforward network block of the detection network layer for time domain branch processing to determine the time domain features of each frame;

[0011] Inputting the plurality of first target features into the DCT image feedforward network block of the detection network layer for frequency domain branch processing to determine the frequency domain features of each frame;

[0012] The time domain features of each frame and the frequency domain features of each frame are fused, and the fused vector of each frame is output.

[0013] In a possible implementation manner, the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are respectively normalized and multi-headed attention processed to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image, including:

[0014] Normalizing the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image, respectively, to determine the first feature of the video image processed by discrete cosine transform after normalization and the first feature of the video image after normalization;

[0015] Performing multi-head attention processing on the first feature of the video image after the normalization processing and the first feature of the video image after the normalization processing, respectively, to determine the second feature of the video image after the discrete cosine transform processing and the first multi-head attention matrix, the second feature of the video image and the second multi-head attention matrix;

[0016] Normalizing the second feature of the video image processed by discrete cosine transform and the first multi-head attention matrix to obtain the first target feature of each frame of the video image processed by discrete cosine transform;

[0017] The second feature of the video image and the second multi-head attention matrix are normalized to obtain the second target feature of each frame of the video image.

[0018] In a possible implementation manner, the fusion vector of each frame is input into the classification network layer of the video detection model, the fusion vector of each frame is classified and predicted, and whether the video to be detected is a generated video is output, and if so, the generation interval of the video to be detected is output, including:

[0019] Performing classification prediction processing on the fusion vector of each frame to determine a detection result sequence of the video to be detected; wherein 0 in the detection result sequence represents that the detection result of the frame video image is true, and 1 represents that the detection result of the frame video image is false;

[0020] Taking multiple consecutive frames of false video images in the detection result sequence as the starting position of the generation interval, and taking multiple consecutive frames of true video images as the ending position of the generation interval;

[0021] If the detection result sequence has at least one generated interval, the video to be detected is the generated video.

[0022] In a possible implementation manner, the video detection model is trained by the following steps:

[0023] The network parameters of the image branch of the BeiT3 model are used as the network parameters of the initial RGB image feed-forward network block in the Transformers model, and the network parameters of the initial RGB image feed-forward network block are frozen;

[0024] The initial network parameters of the initial DCT image feed-forward network block in the Transformers model are set to be the same as the network parameters of the initial RGB image feed-forward network block;

[0025] Synchronously inputting the sample video image processed by discrete cosine transform and the sample video image into the Transformers model for processing, and outputting the predicted label of the sample video image;

[0026] Determining a loss value of the Transformers model based on the predicted label of the sample video image and the true label of the sample video image;

[0027] The Transformers model is iteratively trained based on the loss value and the sample video image training set to determine the video detection model.

[0028] In a possible implementation manner, the iterative training of the Transformers model based on the loss value and the sample video image training set to determine the video detection model includes:

[0029] Detecting whether the loss value is greater than or equal to a loss threshold;

[0030] If yes, then the initial network parameters of the initial DCT image feedforward network block and the network parameters of the initial classification network layer of the Transformers model are changed, and the modified Transformers model is iteratively trained based on the sample video image training set;

[0031] If not, the Transformers model is used as the video detection model.

[0032] The present application also provides a detection device for generating a video, the detection device comprising:

[0033] An image conversion module is used to obtain a video to be detected, perform discrete cosine transform processing on each frame of the video image in the video to be detected, and determine the video image after discrete cosine transform processing of each frame;

[0034] The first processing module is used to synchronously input the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image into the detection network layer of the pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image processed by discrete cosine transform, fuse the time domain feature of each frame and the frequency domain feature of each frame to output a fusion vector of each frame; the video detection model is obtained by iterative training of the Transformers model;

[0035] The second processing module is used to input the fusion vector of each frame into the classification network layer of the video detection model, perform classification prediction processing on the fusion vector of each frame, and output whether the video to be detected is a generated video. If so, it also outputs the generation interval of the video to be detected.

[0036] In a possible implementation manner, when the first processing module is used to synchronously input the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image into the detection network layer of the pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image processed by discrete cosine transform, fuse the time domain feature of each frame and the frequency domain feature of each frame to output the first feature of the fusion vector of each frame, the first processing module is used to:

[0037] Normalizing and multi-head attention processing are respectively performed on the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image;

[0038] Inputting the plurality of first target features into the RGB image feedforward network block of the detection network layer for time domain branch processing to determine the time domain features of each frame;

[0039] Inputting the plurality of first target features into the DCT image feedforward network block of the detection network layer for frequency domain branch processing to determine the frequency domain features of each frame;

[0040] The time domain features of each frame and the frequency domain features of each frame are fused, and the fused vector of each frame is output.

[0041] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the detection method for generating a video as described above are performed.

[0042] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the detection method for generating a video as described above are executed.

[0043] The embodiments of the present application provide a detection method, device, electronic device and storage medium for generating a video, which obtain a video to be detected, perform discrete cosine transform processing on each frame of the video image to be detected, and determine the video image after the discrete cosine transform processing on each frame; synchronously input the first feature of each frame of the video image after the discrete cosine transform processing and the first feature of each frame of the video image into the detection network layer of a pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image after the discrete cosine transform processing, fuse the time domain features of each frame and the frequency domain features of each frame to output a fusion vector of each frame; the video detection model is obtained by iteratively training the Transformers model; the fusion vector of each frame is input into the classification network layer of the video detection model, the fusion vector of each frame is classified and predicted, and whether the video to be detected is a generated video is output, and if so, the generation interval of the video to be detected is output. By utilizing the video detection model to process the discrete cosine transform video image and the RGB video image in the time domain branch and the frequency domain branch, the time domain and frequency domain information are finally weightedly fused to obtain a fusion vector, which is then classified to quickly and accurately determine whether the video to be detected is a generated video, and can better process videos generated by unseen generation methods.

[0044] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a detection method for generating a video provided in an embodiment of the present application;

[0047] Figure 2 One of the structural schematic diagrams of a detection device for generating a video provided in an embodiment of the present application;

[0048] Figure 3 A second structural schematic diagram of a detection device for generating a video provided in an embodiment of the present application;

[0049] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0051] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied to the field of video authenticity detection technology.

[0052] Research has found that with the rapid development of AI technology, image generation technology has become more and more mature, even to the point where it is difficult for the human eye to distinguish true from false. If these generated images are used maliciously, they may cause a variety of security issues, which has also led to the increasing demand for generated image detection technology. The current mainstream generation methods include generative adversarial networks (GAN) and diffusion models (DM). Due to the different generation methods of these two methods, it is difficult for current CNN-trained detectors to solve the generalization problem of detecting images generated by these two methods. Therefore, how to accurately detect generated videos under different generation methods has become a technical problem that cannot be underestimated.

[0053] Based on this, an embodiment of the present application provides a detection method for generated videos, by utilizing a video detection model to process the video image processed by discrete cosine transform and the RGB video image in the time domain branch and the frequency domain branch, and finally weightedly fuse the time domain and frequency domain information to obtain a fusion vector, and then classify the fusion vector to quickly and accurately determine whether the video to be detected is a generated video, and can better process videos generated by unseen generation methods.

[0054] See also Figure 1 , Figure 1 The following is a flow chart of a detection method for generating a video provided in an embodiment of the present application. Figure 1 As shown in, the detection method provided in the embodiment of the present application includes:

[0055] S101: Acquire a video to be detected, perform discrete cosine transform processing on each frame of video image in the video to be detected, and determine each frame of video image after discrete cosine transform processing.

[0056] In this step, a video to be detected is obtained, and discrete cosine transform processing is performed on each frame of video image in the video to be detected, so as to determine each frame of video image after discrete cosine transform processing.

[0057] S102: Synchronously input the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image into the detection network layer of the pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image processed by discrete cosine transform, fuse the time domain features of each frame and the frequency domain features of each frame to output a fusion vector of each frame.

[0058] In this step, the first feature of each frame of video image processed by discrete cosine transform and the first feature of each frame of video image are synchronously input into the detection network layer of the video detection model, the first feature of each frame of video image is processed by time domain branch, the first feature of each frame of video image processed by discrete cosine transform is processed by frequency domain branch, the time domain features of each frame and the frequency domain features of each frame are fused to output the fusion vector of each frame.

[0059] Here, the video image is an RGB image, and the video image after discrete cosine transform processing is a DCT image. The RGB image and the DCT image are input into the detection network layer. The RGB image passes through the time domain branch, and the DCT image passes through the frequency domain branch to finally merge the results of the two branches.

[0060] In a possible implementation manner, the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are synchronously input into the detection network layer of the pre-trained video detection model, the first feature of each frame of the video image is processed by time domain branching, the first feature of each frame of the video image processed by discrete cosine transform is processed by frequency domain branching, and the time domain feature of each frame and the frequency domain feature of each frame are fused to output the first feature of the fusion vector of each frame, including:

[0061] A: The first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are respectively normalized and multi-headed attention processed to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image.

[0062] Here, the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are normalized and multi-head attention processed respectively to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image.

[0063] In a possible implementation manner, the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are respectively normalized and multi-headed attention processed to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image, including:

[0064] a: respectively normalize the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image to determine the first feature of the video image processed by discrete cosine transform after normalization and the first feature of the video image after normalization.

[0065] Here, the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are normalized to obtain the first feature of the video image processed by discrete cosine transform after normalization and the first feature of the video image after normalization.

[0066] b: Perform multi-head attention processing on the first feature of the video image after normalization and discrete cosine transform processing and the first feature of the video image after normalization, and determine the second feature of the video image after discrete cosine transform processing and the first multi-head attention matrix, the second feature of the video image and the second multi-head attention matrix.

[0067] Here, multi-head attention processing is performed on the first feature of the video image after normalization and discrete cosine transform processing and the first feature of the video image after normalization to determine the second feature of the video image after discrete cosine transform processing and the first multi-head attention matrix of the video image after discrete cosine transform processing, the second feature of the video image and the second multi-head attention matrix of the video image.

[0068] c: Normalize the second feature of the video image processed by discrete cosine transform and the first multi-head attention matrix to obtain the first target feature of each frame of the video image processed by discrete cosine transform. Normalize the second feature of the video image and the second multi-head attention matrix to obtain the second target feature of each frame of the video image.

[0069] Here, the second feature of the video image processed by discrete cosine transform and the first multi-head attention matrix are normalized to obtain the first target feature of each frame of the video image processed by discrete cosine transform. The second feature of the video image and the second multi-head attention matrix are normalized to obtain the second target feature of each frame of the video image.

[0070] Among them, the first target feature and the second target feature obtained after normalization and multi-head attention processing contain more important feature information.

[0071] B: Input multiple of the first target features into the RGB image feedforward network block of the detection network layer for time domain branch processing to determine the time domain features of each frame.

[0072] Here, multiple first target features are input into the RGB image feed-forward network block of the detection network layer for time domain branch processing to determine the time domain features of each frame.

[0073] Among them, the time domain branch processing is to process the features of the first target feature in the time domain dimension. The process of time domain branch processing can be implemented by any time domain processing technology in the existing technology, and this part does not make specific limitations.

[0074] C: Input multiple of the first target features into the DCT image feedforward network block of the detection network layer for frequency domain branch processing to determine the frequency domain features of each frame.

[0075] Here, multiple first target features are input into the DCT image feedforward network block of the detection network layer for frequency domain branch processing to determine the frequency domain features of each frame.

[0076] Among them, the frequency domain branch processing is to process the features of the first target feature in the time domain dimension. The process of frequency domain branch processing can be implemented by any frequency domain processing technology in the existing technology, and this part does not make specific limitations.

[0077] D: Fuse the time domain features of each frame and the frequency domain features of each frame, and output the fusion vector of each frame.

[0078] Here, the method of fusing the time domain and frequency domain branches is as follows: use a weight vector W with the same latitude as the time domain feature S and the frequency domain feature F, the W parameter can be learned, and the fusion vector C = W*S+(1-W)*F, thereby realizing element-by-element fusion of the vectors.

[0079] S103: Input the fusion vector of each frame into the classification network layer of the video detection model, perform classification prediction processing on the fusion vector of each frame, and output whether the video to be detected is a generated video. If so, output the generation interval of the video to be detected.

[0080] In this step, the fusion vector of each frame is input into the classification network layer of the video detection model, and the fusion vector of each frame is classified and predicted to output whether the video to be detected is a generated video. If so, the generation interval of the video to be detected is output.

[0081] The generated interval is the forged interval corresponding to the video to be detected.

[0082] In a possible implementation manner, the fusion vector of each frame is input into the classification network layer of the video detection model, the fusion vector of each frame is classified and predicted, and whether the video to be detected is a generated video is output, and if so, the generation interval of the video to be detected is output, including:

[0083] (1): Classify and predict the fusion vector of each frame to determine the detection result sequence of the video to be detected; wherein 0 in the detection result sequence represents that the detection result of the frame video image is true, and 1 represents that the detection result of the frame video image is false.

[0084] Here, classification prediction processing is performed on each frame fusion vector to determine the detection result sequence of the video to be detected.

[0085] Among them, the classification network layer performs classification prediction processing on each frame fusion vector to obtain a prediction score. If the prediction score is greater than or equal to 0.5, the frame of video image is true, otherwise it is false.

[0086] (2): Multiple consecutive frames of false video images in the detection result sequence are used as the starting position of the generation interval, and multiple consecutive frames of true video images are used as the ending position of the generation interval.

[0087] Here, the generation interval is determined according to the detection result sequence L: n consecutive false frames are considered to be the interval starting position Ls, and n consecutive true frames are considered to be the interval ending position Le, then the generated interval is [Ls, Le], where n is a configuration parameter. The larger n is, the higher the accuracy, but the lower the recall rate; the smaller n is, the higher the recall rate, but the lower the accuracy rate. The default value of n is 3; example: [0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0], the sequence length is L = 20, the starting position is Ls = 3, the ending position is Le = 16, and the forged interval is [3, 16].

[0088] (3): If there is at least one generated interval in the detection result sequence, the video to be detected is the generated video.

[0089] Here, if there is at least one generation interval in the detection result sequence, the video to be detected is the generated video.

[0090] In a possible implementation manner, the video detection model is trained by the following steps:

[0091] I: The network parameters of the image branch of the BeiT3 model are used as the network parameters of the initial RGB image feedforward network block in the Transformers model, and the network parameters of the initial RGB image feedforward network block are frozen.

[0092] Here, the network parameters of the image branch of the BeiT3 model are used as the network parameters of the initial RGB image feedforward network block in the Transformers model, and the network parameters of the initial RGB image feedforward network block are frozen.

[0093] II: Set the initial network parameters of the initial DCT image feed-forward network block in the Transformers model to be the same as the network parameters of the initial RGB image feed-forward network block.

[0094] Here, the initial network parameters of the initial DCT image feed-forward network block in the Transformers model are set the same as the network parameters of the initial RGB image feed-forward network block.

[0095] Among them, the BeiT-3 model is loaded to read the training parameters of its image branch, and the time domain branch of this method is initialized with this parameter, and the training parameters are frozen to ensure that they are no longer updated in the subsequent training process; the frequency domain branch has the same network structure as the time domain branch, so its initialization parameters are consistent with those of the time domain branch. The difference is that the frequency domain branch does not freeze the training parameters and participates in the training and updating parameters normally;

[0096] III: The sample video image processed by discrete cosine transform and the sample video image are synchronously input into the Transformers model for processing, and the predicted label of the sample video image is output.

[0097] Here, the sample video image after discrete cosine transform processing and the sample video image are synchronously input into the Transformers model for processing, and the predicted label of the sample video image is output.

[0098] Among them, the processing process of predicting labels is consistent with the processing process of the first feature of each frame of the video image after discrete cosine transform processing and the first feature of each frame of the video image by the above-mentioned video detection model, and will not be repeated in this part.

[0099] IV: Determine the loss value of the Transformers model based on the predicted label of the sample video image and the true label of the sample video image.

[0100] Here, the loss value of the Transformers model is determined based on the predicted label of the sample video image and the true label of the sample video image using the binary cross entropy loss function.

[0101] V: Iteratively train the Transformers model based on the loss value and the sample video image training set to determine the video detection model.

[0102] Here, the Transformers model is iteratively trained based on the loss value and the sample video image training set to determine the video detection model.

[0103] In a possible implementation manner, the iterative training of the Transformers model based on the loss value and the sample video image training set to determine the video detection model includes:

[0104] Detect whether the loss value is greater than or equal to a loss threshold; if so, change the initial network parameters of the initial DCT image feedforward network block and the network parameters of the initial classification network layer of the Transformers model, and continue to iteratively train the changed Transformers model based on the sample video image training set; if not, use the Transformers model as the video detection model.

[0105] An embodiment of the present application provides a detection method for generating a video, the detection method comprising: obtaining a video to be detected, performing discrete cosine transform processing on each frame of the video image in the video to be detected, and determining each frame of the video image after discrete cosine transform processing; synchronously inputting the first feature of each frame of the video image after discrete cosine transform processing and the first feature of each frame of the video image into the detection network layer of a pre-trained video detection model, performing time domain branch processing on the first feature of each frame of the video image, performing frequency domain branch processing on the first feature of each frame of the video image after discrete cosine transform processing, fusing the time domain features of each frame and the frequency domain features of each frame to output a fusion vector of each frame; the video detection model is obtained by iterative training of the Transformers model; inputting the fusion vector of each frame into the classification network layer of the video detection model, performing classification prediction processing on the fusion vector of each frame, and outputting whether the video to be detected is a generated video, and if so, outputting the generation interval of the video to be detected. By utilizing the video detection model to process the discrete cosine transform video image and the RGB video image in the time domain branch and the frequency domain branch, the time domain and frequency domain information are finally weightedly fused to obtain a fusion vector, which is then classified to quickly and accurately determine whether the video to be detected is a generated video, and can better process videos generated by unseen generation methods.

[0106] See also Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a detection device for generating a video provided in an embodiment of the present application; Figure 3 This is a second structural diagram of a detection device for generating a video provided in an embodiment of the present application. Figure 2 As shown in , the detection device 200 includes:

[0107] The image conversion module 210 is used to obtain a video to be detected, perform discrete cosine transform processing on each frame of the video image in the video to be detected, and determine the video image after the discrete cosine transform processing of each frame;

[0108] The first processing module 220 is used to synchronously input the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image into the detection network layer of the pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image processed by discrete cosine transform, fuse the time domain feature of each frame and the frequency domain feature of each frame to output a fusion vector of each frame; the video detection model is obtained by iterative training of the Transformers model;

[0109] The second processing module 230 is used to input the fusion vector of each frame into the classification network layer of the video detection model, perform classification prediction processing on the fusion vector of each frame, and output whether the video to be detected is a generated video. If so, it also outputs the generation interval of the video to be detected.

[0110] Further, when the first processing module 220 is used to synchronously input the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image into the detection network layer of the pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image processed by discrete cosine transform, fuse the time domain feature of each frame and the frequency domain feature of each frame to output the first feature of the fusion vector of each frame, the first processing module 220 is used to:

[0111] Normalizing and multi-head attention processing are respectively performed on the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image;

[0112] Inputting the plurality of first target features into the RGB image feedforward network block of the detection network layer for time domain branch processing to determine the time domain features of each frame;

[0113] Inputting the plurality of first target features into the DCT image feedforward network block of the detection network layer for frequency domain branch processing to determine the frequency domain features of each frame;

[0114] The time domain features of each frame and the frequency domain features of each frame are fused, and the fused vector of each frame is output.

[0115] Further, when the first processing module 220 is used to perform normalization processing and multi-head attention processing on the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image, respectively, to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image, the first processing module 220 is specifically used to:

[0116] Normalizing the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image, respectively, to determine the first feature of the video image processed by discrete cosine transform after normalization and the first feature of the video image after normalization;

[0117] Performing multi-head attention processing on the first feature of the video image after the normalization processing and the first feature of the video image after the normalization processing, respectively, to determine the second feature of the video image after the discrete cosine transform processing and the first multi-head attention matrix, the second feature of the video image and the second multi-head attention matrix;

[0118] Normalizing the second feature of the video image processed by discrete cosine transform and the first multi-head attention matrix to obtain the first target feature of each frame of the video image processed by discrete cosine transform;

[0119] The second feature of the video image and the second multi-head attention matrix are normalized to obtain the second target feature of each frame of the video image.

[0120] Furthermore, the second processing module 230 inputs the fusion vector of each frame into the classification network layer of the video detection model, performs classification prediction processing on the fusion vector of each frame, outputs whether the video to be detected is a generated video, and if so, outputs the generation interval of the video to be detected. The second processing module 230 is specifically used for:

[0121] Performing classification prediction processing on the fusion vector of each frame to determine a detection result sequence of the video to be detected; wherein 0 in the detection result sequence represents that the detection result of the frame video image is true, and 1 represents that the detection result of the frame video image is false;

[0122] Taking multiple consecutive frames of false video images in the detection result sequence as the starting position of the generation interval, and taking multiple consecutive frames of true video images as the ending position of the generation interval;

[0123] If the detection result sequence has at least one generated interval, the video to be detected is the generated video.

[0124] Further, such as Figure 3 As shown, the detection device 200 further includes a model training module 240, and the model training module 240 trains the video detection model through the following steps:

[0125] The network parameters of the image branch of the BeiT3 model are used as the network parameters of the initial RGB image feed-forward network block in the Transformers model, and the network parameters of the initial RGB image feed-forward network block are frozen;

[0126] The initial network parameters of the initial DCT image feed-forward network block in the Transformers model are set to be the same as the network parameters of the initial RGB image feed-forward network block;

[0127] Synchronously inputting the sample video image processed by discrete cosine transform and the sample video image into the Transformers model for processing, and outputting the predicted label of the sample video image;

[0128] Determining a loss value of the Transformers model based on the predicted label of the sample video image and the true label of the sample video image;

[0129] The Transformers model is iteratively trained based on the loss value and the sample video image training set to determine the video detection model.

[0130] Further, when the model training module 240 is used to iteratively train the Transformers model based on the loss value and the sample video image training set to determine the video detection model, the model training module 240 is specifically used to:

[0131] Detecting whether the loss value is greater than or equal to a loss threshold;

[0132] If yes, then the initial network parameters of the initial DCT image feedforward network block and the network parameters of the initial classification network layer of the Transformers model are changed, and the modified Transformers model is iteratively trained based on the sample video image training set;

[0133] If not, the Transformers model is used as the video detection model.

[0134] An embodiment of the present application provides a detection device for generating a video, the detection device comprising: an image conversion module, used to obtain a video to be detected, perform discrete cosine transform processing on each frame of the video image to be detected, and determine each frame of the video image after discrete cosine transform processing; a first processing module, used to synchronously input the first feature of each frame of the video image after discrete cosine transform processing and the first feature of each frame of the video image into the detection network layer of a pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image after discrete cosine transform processing, fuse the time domain features of each frame and the frequency domain features of each frame to output a fusion vector of each frame; the video detection model is obtained by iterative training of the Transformers model; a second processing module, used to input the fusion vector of each frame into the classification network layer of the video detection model, perform classification prediction processing on the fusion vector of each frame, and output whether the video to be detected is a generated video, and if so, output the generation interval of the video to be detected. By utilizing the video detection model to process the discrete cosine transform video image and the RGB video image in the time domain branch and the frequency domain branch, the time domain and frequency domain information are finally weightedly fused to obtain a fusion vector, which is then classified to quickly and accurately determine whether the video to be detected is a generated video, and can better process videos generated by unseen generation methods.

[0135] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0136] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the detection method for generating a video in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0137] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the detection method for generating a video in the method embodiment shown, the specific implementation method can be found in the method embodiment, and will not be repeated here.

[0138] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0139] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0140] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0141] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0142] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0143] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A detection method for generating a video, characterized in that: The detection method comprises: Acquire a video to be detected, perform discrete cosine transform processing on each frame of video image in the video to be detected, and determine the video image after discrete cosine transform processing of each frame; The first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are synchronously input into the detection network layer of the pre-trained video detection model, the first feature of each frame of the video image is processed by time domain branching, the first feature of each frame of the video image processed by discrete cosine transform is processed by frequency domain branching, and the time domain feature of each frame and the frequency domain feature of each frame are fused to output the fusion vector of each frame; the video detection model is obtained by iterative training of the Transformers model; The fusion vector of each frame is input into the classification network layer of the video detection model, and the fusion vector of each frame is classified and predicted, and whether the video to be detected is a generated video is output, and if so, the generation interval of the video to be detected is output.

2. The detection method according to claim 1, characterized in that: The first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are synchronously input into the detection network layer of the pre-trained video detection model, the first feature of each frame of the video image is processed by time domain branching, the first feature of each frame of the video image processed by discrete cosine transform is processed by frequency domain branching, and the time domain feature of each frame and the frequency domain feature of each frame are fused to output the first feature of the fusion vector of each frame, including: Normalizing and multi-head attention processing are respectively performed on the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image; Inputting the plurality of first target features into the RGB image feedforward network block of the detection network layer for time domain branch processing to determine the time domain features of each frame; Inputting the plurality of first target features into the DCT image feedforward network block of the detection network layer for frequency domain branch processing to determine the frequency domain features of each frame; The time domain features of each frame and the frequency domain features of each frame are fused, and the fused vector of each frame is output.

3. The detection method according to claim 2, characterized in that: The first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image are respectively normalized and multi-headed attention processed to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image, including: Normalizing the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image, respectively, to determine the first feature of the video image processed by discrete cosine transform after normalization and the first feature of the video image after normalization; Performing multi-head attention processing on the first feature of the video image after the normalization processing and the first feature of the video image after the normalization processing, respectively, to determine the second feature of the video image after the discrete cosine transform processing and the first multi-head attention matrix, the second feature of the video image and the second multi-head attention matrix; Normalizing the second feature of the video image processed by discrete cosine transform and the first multi-head attention matrix to obtain the first target feature of each frame of the video image processed by discrete cosine transform; The second feature of the video image and the second multi-head attention matrix are normalized to obtain the second target feature of each frame of the video image.

4. The detection method according to claim 1, characterized in that: The fusion vector of each frame is input into the classification network layer of the video detection model, the fusion vector of each frame is classified and predicted, and whether the video to be detected is a generated video is output, and if so, the generation interval of the video to be detected is output, including: Performing classification prediction processing on the fusion vector of each frame to determine a detection result sequence of the video to be detected; wherein 0 in the detection result sequence represents that the detection result of the frame video image is true, and 1 represents that the detection result of the frame video image is false; Taking multiple consecutive frames of false video images in the detection result sequence as the starting position of the generation interval, and taking multiple consecutive frames of true video images as the ending position of the generation interval; If the detection result sequence has at least one generated interval, the video to be detected is the generated video.

5. The detection method according to claim 1, characterized in that: The video detection model is trained by the following steps: The network parameters of the image branch of the BeiT3 model are used as the network parameters of the initial RGB image feed-forward network block in the Transformers model, and the network parameters of the initial RGB image feed-forward network block are frozen; The initial network parameters of the initial DCT image feed-forward network block in the Transformers model are set to be the same as the network parameters of the initial RGB image feed-forward network block; Synchronously inputting the sample video image processed by discrete cosine transform and the sample video image into the Transformers model for processing, and outputting the predicted label of the sample video image; Determining a loss value of the Transformers model based on the predicted label of the sample video image and the true label of the sample video image; The Transformers model is iteratively trained based on the loss value and the sample video image training set to determine the video detection model.

6. The detection method according to claim 5, characterized in that: The iterative training of the Transformers model based on the loss value and the sample video image training set to determine the video detection model includes: Detecting whether the loss value is greater than or equal to a loss threshold; If yes, then the initial network parameters of the initial DCT image feedforward network block and the network parameters of the initial classification network layer of the Transformers model are changed, and the modified Transformers model is iteratively trained based on the sample video image training set; If not, the Transformers model is used as the video detection model.

7. A detection device for generating a video, characterized in that: The detection device comprises: An image conversion module is used to obtain a video to be detected, perform discrete cosine transform processing on each frame of the video image in the video to be detected, and determine the video image after discrete cosine transform processing of each frame; The first processing module is used to synchronously input the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image into the detection network layer of the pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image processed by discrete cosine transform, fuse the time domain feature of each frame and the frequency domain feature of each frame to output a fusion vector of each frame; the video detection model is obtained by iterative training of the Transformers model; The second processing module is used to input the fusion vector of each frame into the classification network layer of the video detection model, perform classification prediction processing on the fusion vector of each frame, and output whether the video to be detected is a generated video. If so, it also outputs the generation interval of the video to be detected.

8. The detection device according to claim 7, characterized in that: When the first processing module is used to synchronously input the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image into the detection network layer of the pre-trained video detection model, perform time domain branch processing on the first feature of each frame of the video image, perform frequency domain branch processing on the first feature of each frame of the video image processed by discrete cosine transform, fuse the time domain feature of each frame and the frequency domain feature of each frame to output the first feature of the fusion vector of each frame, the first processing module is used to: Normalizing and multi-head attention processing are respectively performed on the first feature of each frame of the video image processed by discrete cosine transform and the first feature of each frame of the video image to obtain the first target feature of each frame of the video image processed by discrete cosine transform and the second target feature of each frame of the video image; Inputting the plurality of first target features into the RGB image feedforward network block of the detection network layer for time domain branch processing to determine the time domain features of each frame; Inputting the plurality of first target features into the DCT image feedforward network block of the detection network layer for frequency domain branch processing to determine the frequency domain features of each frame; The time domain features of each frame and the frequency domain features of each frame are fused, and the fused vector of each frame is output.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the detection method for generating a video as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the detection method for generating a video as described in any one of claims 1 to 6 are executed.