Deep counterfeit video detection method based on double-flow network
Through the deep fake video detection method based on dual-stream network, the RGB and noise feature streams in the video frame are extracted and combined and calculated, which solves the problem of deep fake video detection in the prior art, and achieves higher detection accuracy and robustness .
Patent Information
- Application Number
- CN202411009344.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-10
AI Technical Summary
The existing technology is difficult to effectively detect and deal with in-depth fake videos, and cannot distinguish between real and false content well, leading to the spread of false information.
Using a deep fake video detection method based on dual-stream network, the video to be detected is decomposed into a sequence of picture frames, the RGB feature stream and noise feature stream are extracted through a convolutional neural network, and input them into the MobileNet model for feature merge, and calculate each picture frame The probability of forgery is to finally determine whether the video is a deep forgery video.
Through the design of the dual-stream network framework, the model's robustness and detection accuracy for video compression are improved, and more comprehensive feature learning and higher accuracy of fake video detection are achieved.
Smart Images
Figure CN119107580A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video detection technology, and in particular to a deep fake video detection method based on a dual-stream network. Background Art
[0002] With the rapid development of deep learning technology, deep fake videos (Deepfakes) have begun to emerge in large numbers on the Internet. These videos can easily replace faces or tamper with expression information through generative adversarial networks and autoencoder technology, and then create false content, spread rumors, and spread fake news, causing great negative impact on society. However, the detection technology for deep fake videos is still in its early stages, and the existing technology has limitations and cannot effectively cope with the detection challenges of deep fake videos. Summary of the invention
[0003] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a deep fake video detection method based on a dual-stream network.
[0004] In order to solve the above technical problems, the technical solution of the present invention is as follows: A deep fake video detection method based on a dual-stream network, comprising: Decompose the video to be detected into a sequence of picture frames; A convolutional neural network is used to extract the RGB feature stream and the noise feature stream in each of the picture frames; The RGB feature stream and the noise feature stream are respectively input into a MobileNet model, and the fully connected layers of the two MobileNet models are shared, so that the RGB feature stream and the noise feature stream are merged in the fully connected layer, and the forgery probability of each of the picture frames is calculated; The forgery probability of the video to be detected is calculated based on the forgery probability of each of the image frames, and whether the video to be detected is a deep forgery video is determined based on the forgery probability.
[0005] As a preferred solution of the deep fake video detection method based on the dual-stream network of the present invention, wherein: decomposing the video to be detected into a picture frame sequence includes: Use OpenCV library or ffmpeg library to decompose the video to be detected into a sequence of picture frames.
[0006] As a preferred solution of the deep fake video detection method based on dual-stream network described in the present invention, the picture frames in the picture frame sequence are arranged according to their playback order in the video to be detected.
[0007] As a preferred solution of the deep fake video detection method based on dual-stream network described in the present invention, the decomposition frequency of decomposing the video to be detected into a picture frame sequence is 24 frames per second or 30 frames per second.
[0008] As a preferred solution of the deep fake video detection method based on the dual-stream network of the present invention, wherein: the use of a convolutional neural network to extract the RGB feature stream and the noise feature stream in each of the picture frames includes: Inputting the picture frame sequence into a convolutional neural network model to generate an RGB feature stream; The picture frame sequence is input into the convolutional neural network model again and passes through a noise selector to generate a noise feature stream.
[0009] As an optimal solution of the deep fake video detection method based on dual-stream network described in the present invention, wherein: the convolutional neural network model includes a convolution layer, a ReLU function layer, a convolution layer, a ReLU function layer, and a pooling layer in sequence.
[0010] As a preferred solution of the deep fake video detection method based on the dual-stream network of the present invention, the noise selector is defined by the following equation: ,in, Indicates The characteristic noise, Indicates The neighborhood of a feature, express predictions, Indicates Features, Represents the dimension of the feature.
[0011] As a preferred solution of the deep forgery video detection method based on the dual-stream network of the present invention, wherein: the RGB feature stream and the noise feature stream are merged in the fully connected layer, and the forgery probability of each picture frame is calculated, which includes: The synthesis weight of the RGB feature stream and the noise feature stream during synthesis is set to: ,in, is the input picture frame, is the output of the noise characteristic stream, is the output of the RGB feature stream, is a variable factor, It is the output after the RGB feature stream and the noise feature stream are synthesized, which is expressed in the form of probability; The forgery probability of each of the picture frames is a probability output value synthesized from the RGB feature stream and the noise feature stream of the corresponding frame.
[0012] As a preferred solution of the deep fake video detection method based on the dual-stream network of the present invention, the method of calculating the fake probability of the video to be detected based on the fake probability of each picture frame, and judging whether the video to be detected is a deep fake video based on the fake probability includes: Calculate the average of the forgery probabilities of all image frames and use it as the forgery probability of the video to be detected; It is determined whether the forgery probability of the video to be detected is greater than or equal to 0.5. If so, the video to be detected is determined to be a forged video. If not, the video to be detected is determined to be a non-forged video.
[0013] As a preferred solution of the deep fake video detection method based on the dual-stream network of the present invention, wherein: It is defined by the following equation: , It is defined by the following equation: .
[0014] The beneficial effects of the present invention are: The present invention uses the MobileNet network as the backbone to construct a dual-stream network framework, in which one stream is dedicated to processing image noise features to improve the model's robustness to video compression, and the other stream is responsible for processing RGB image information to learn the feature distribution of real and fake faces. The two streams are fused at the model output layer to achieve more comprehensive feature learning and higher detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0016] Figure 1 A schematic diagram of the process of the deep fake video detection method based on the dual-stream network provided by the present invention; Figure 2 Another schematic diagram of the flow chart of the deep fake video detection method based on the dual-stream network provided by the present invention. DETAILED DESCRIPTION
[0017] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific implementation modes and in combination with the accompanying drawings.
[0018] Figure 1 A schematic diagram of a deep fake video detection method based on a dual-stream network provided in an embodiment of the present application. The method comprises the following steps: Step S101: Decompose the video to be detected into a sequence of picture frames.
[0019] Specifically, the video to be detected is decomposed into a sequence of frames at the input end. The specific implementation method is: using the OpenCV library or the ffmpeg library to decompose the video to be detected into a sequence of picture frames.
[0020] When the OpenCV library is used to decompose the video to be detected into a sequence of picture frames, it is defined by the following Python function: reading frames is cv2.VideoCapture(video_path).read(), writing frames is cv2.imwrite(), and counting frames is cv2.VideoCapture(video_path).get(cv2.CAP_PROP_FRAME_COUNT). When the ffmpeg library is used to decompose the video to be detected into a sequence of picture frames, it is defined by the following Python function: ffmpeg.input(video_path).output().run().
[0021] It should be noted that, in the picture frame sequence formed by decomposition, the picture frames are arranged according to the playback order of the video to be detected, and the decomposition frequency of decomposing the video to be detected into the picture frame sequence is 24 frames per second or 30 frames per second.
[0022] Step S102: using a convolutional neural network to extract the RGB feature stream and the noise feature stream in each of the picture frames.
[0023] Specifically, the convolutional neural network model consists of a convolutional layer, a ReLU function layer, a convolutional layer, a ReLU function layer, and a pooling layer. The convolutional neural network model is used twice to generate two branches. One branch directly generates an RGB feature stream, and the other branch passes through a noise selector to generate a noise feature stream.
[0024] It can be understood that noise refers to the incoherent pixel distribution, unnatural facial movement patterns, or lighting inconsistencies generated during the synthesis of deep fake videos. The noise selector is a convolutional neural network that extracts the features of these noises and filters out those that are not noise to form a noise stream.
[0025] In this embodiment, the noise selector is defined by the following equation: ,in, Indicates The characteristic noise, Indicates The neighborhood of a feature, express predictions, Indicates Features, Represents the dimension of the feature.
[0026] Step S103: input the RGB feature stream and the noise feature stream into a MobileNet model respectively, and share the fully connected layers of the two MobileNet models, so that the RGB feature stream and the noise feature stream are merged in the fully connected layer, and the forgery probability of each of the picture frames is calculated.
[0027] Specifically, the RGB feature stream and the noise feature stream obtained in step S102 are respectively input into a MobileNet model. The parameters and number of layers of the two MobileNets are the same. The feature streams passing through the two MobileNet models are synthesized into a feature stream at the output layer of the MobileNet model. The specific implementation method of the synthesis is to share the last fully connected layer of the two branches of MobileNet, that is, to expand the last fully connected layer of the traditional MobileNet to twice, link it to the two branches of MobileNet, and then perform a total output.
[0028] It should be noted that the RGB feature stream and the noise feature stream are synthesized according to certain weights. The synthesis weights are set as follows: .
[0029] in, is the input picture frame, is the output of the noise characteristic stream, is the output of the RGB feature stream, is a variable factor, It is the output after the RGB feature stream and the noise feature stream are synthesized, which is expressed in the form of probability.
[0030] It is defined by the following equation: , It is defined by the following equation: .
[0031] It can be understood that the forgery probability of each picture frame is the probability output value after the RGB feature stream and the noise feature stream of the corresponding frame are synthesized.
[0032] Step S104: Calculate the forgery probability of the video to be detected based on the forgery probability of each of the image frames, and determine whether the video to be detected is a deep fake video based on the forgery probability.
[0033] Specifically, each picture frame will generate a , the probability output value of the entire video is the average value of the forgery probability of all picture frames. After that, it is determined whether the forgery probability of the video to be detected is greater than or equal to 0.5. If so, the video to be detected is determined to be a forged video. If not, the video to be detected is determined to be a non-forged video (genuine video).
[0034] Therefore, the technical solution of this application uses the MobileNet network as the backbone to build a dual-stream network framework, in which one stream is dedicated to processing image noise features to improve the model's robustness to video compression, and the other stream is responsible for processing RGB image information for learning the feature distribution of real and fake faces. The two streams are fused at the model output layer to achieve more comprehensive feature learning and higher detection accuracy.
[0035] In addition to the above embodiments, the present invention may also have other implementation modes; any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.
Claims
1. A deep fake video detection method based on a dual-stream network, characterized by: include: Decompose the video to be detected into a sequence of picture frames; A convolutional neural network is used to extract the RGB feature stream and the noise feature stream in each of the picture frames; The RGB feature stream and the noise feature stream are respectively input into a MobileNet model, and the fully connected layers of the two MobileNet models are shared, so that the RGB feature stream and the noise feature stream are merged in the fully connected layer, and the forgery probability of each of the picture frames is calculated; The forgery probability of the video to be detected is calculated based on the forgery probability of each of the image frames, and whether the video to be detected is a deep forgery video is determined based on the forgery probability.
2. The deep fake video detection method based on dual-stream network according to claim 1 is characterized in that: Decomposing the video to be detected into a sequence of picture frames comprises: Use OpenCV library or ffmpeg library to decompose the video to be detected into a sequence of picture frames.
3. The deep fake video detection method based on dual-stream network according to claim 1 is characterized in that: The picture frames in the picture frame sequence are arranged according to their playback order in the video to be detected.
4. The deep fake video detection method based on dual-stream network according to claim 1 is characterized in that: The decomposition frequency of decomposing the to-be-detected video into a picture frame sequence is 24 frames per second or 30 frames per second.
5. The deep fake video detection method based on dual-stream network according to claim 1 is characterized in that: The method of extracting the RGB feature stream and the noise feature stream in each picture frame by using a convolutional neural network includes: Inputting the picture frame sequence into a convolutional neural network model to generate an RGB feature stream; The picture frame sequence is input into the convolutional neural network model again and passes through a noise selector to generate a noise feature stream.
6. The deep fake video detection method based on dual-stream network according to claim 5 is characterized in that: The convolutional neural network model includes a convolution layer, a ReLU function layer, a convolution layer, a ReLU function layer, and a pooling layer in sequence.
7. The deep fake video detection method based on dual-stream network according to claim 5 is characterized in that: The noise selector is defined by the following equation: ,in, Indicates The characteristic noise, Indicates The neighborhood of a feature, express predictions, Indicates Features, Represents the dimension of the feature.
8. The deep fake video detection method based on dual-stream network according to claim 1 is characterized in that: The RGB feature stream and the noise feature stream are merged in the fully connected layer, and the forgery probability of each picture frame is calculated, including: The synthesis weight of the RGB feature stream and the noise feature stream during synthesis is set to: ,in, is the input picture frame, is the output of the noise characteristic stream, is the output of the RGB feature stream, is a variable factor, It is the output after the RGB feature stream and the noise feature stream are synthesized, which is expressed in the form of probability; The forgery probability of each of the picture frames is a probability output value synthesized from the RGB feature stream and the noise feature stream of the corresponding frame.
9. The deep fake video detection method based on dual-stream network according to claim 8 is characterized in that: The calculating the forgery probability of the video to be detected based on the forgery probability of each picture frame, and judging whether the video to be detected is a deep forgery video based on the forgery probability includes: Calculate the average of the forgery probabilities of all image frames and use it as the forgery probability of the video to be detected; It is determined whether the forgery probability of the video to be detected is greater than or equal to 0.
5. If so, the video to be detected is determined to be a forged video. If not, the video to be detected is determined to be a non-forged video.
10. The deep fake video detection method based on dual-stream network according to claim 8, characterized in that: Said It is defined by the following equation: , It is defined by the following equation: .
Citation Information
Patent Citations
Image tampering detection method, device and equipment and computer readable storage medium
CN114519689A