A deep fake video detection method, device, equipment and medium

Abstract

The application provides a deepfake video detection method, device, equipment and medium, the method comprises: video feature analysis is carried out on the video to be detected, and the audio-visual features of the video to be detected are obtained; the audio-visual features of the video to be detected are input into a preset multi-layer perception machine classification model, and a detection result is obtained; wherein the audio-visual features include visual features and audio features, the preset multi-layer perception machine classification model is trained with the audio-visual features of deepfake videos and the audio-visual features of real videos as samples, and labels corresponding to the audio-visual features of deepfake videos and the audio-visual features of real videos respectively. The purpose of the application is to solve the defect that the existing technology cannot detect the fake audio and video in the deepfake video separately, resulting in the problem of insufficient accuracy of detecting the deepfake video.

Description

Technical Field

[0001] This invention relates to the field of video detection technology, and in particular to a method, apparatus, equipment and medium for detecting deepfake videos. Background Technology

[0002] Video media serves as a vital carrier of information and plays a crucial role in information retrieval. For example, deepfake videos generated using deep learning technology have proliferated online in recent years. As the quality of these fake videos continues to improve, the malicious application of deepfake technology poses a significant threat.

[0003] Currently, deepfake video detection technologies can be mainly divided into two categories: detection methods based on specific artifacts and detection methods based on deep learning. Detection methods based on specific artifacts focus on specific visual artifacts generated during the video forgery process. These artifacts may be obvious or subtle to humans, but machine learning and forensic analysis techniques can detect them. Meanwhile, detection methods based on deep learning treat deepfake video detection as a regular image or video classification task, training carefully designed deep neural networks to automatically extract useful features to distinguish between real and forged videos.

[0004] However, the existing detection methods mentioned above are all based on a single modality, i.e., images or videos, while existing deepfake videos are often accompanied by forged audio. Therefore, how to effectively utilize the visual and auditory information in the video remains an unsolved problem. Although some audiovisual-based deepfake video detection algorithms detect forgeries based on the consistency between the visual and auditory modalities (such as the facial emotions of people in the video and the emotions in the speech content), they ignore the forgery features in a single modality. Furthermore, current detection algorithms ignore the relationship between the visual and auditory modalities and do not effectively fuse audiovisual features. This leads to insufficient accuracy in the detection of deepfake videos using existing audiovisual-based methods. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for detecting deepfake videos, which solves the problem that existing technologies cannot separately detect the fake audio and video in deepfake videos, resulting in insufficient accuracy in detecting deepfake videos.

[0006] This invention provides a method for detecting deepfake videos, comprising:

[0007] Video feature analysis is performed on the video to be tested to obtain the audio and video features of the video to be tested;

[0008] The audio and video features of the video to be detected are input into a preset multilayer perceptron classification model to obtain the detection result;

[0009] The audio-visual features include visual features and audio features. The preset multilayer perceptron classification model is trained using the audio-visual features of deepfake videos and real videos as samples, and the labels corresponding to the audio-visual features of deepfake videos and real videos, respectively.

[0010] According to the present invention, a deepfake video detection method is provided, wherein the step of performing video feature analysis on the video to be detected to obtain the audio and video features of the video to be detected includes:

[0011] Perform preliminary video feature analysis on the video to be tested to obtain preliminary audio and video features;

[0012] By using a composite attention module, the preliminary audio-visual features are subjected to visual and auditory information interaction to obtain audio-visual features;

[0013] The preliminary audio-visual features include preliminary visual features with video frames and preliminary audio features with Mel-frequency cepstral coefficients.

[0014] The composite attention module is obtained by replacing the intermediate convolutional layer in the residual neural network with a composite attention layer, using the residual module as the skeleton.

[0015] According to the present invention, a deepfake video detection method is provided, wherein the preliminary feature analysis of the video to be detected to obtain preliminary audio and video features includes:

[0016] Extract video frames and Mel-frequency cepstral coefficients of the face region in the video to be detected;

[0017] Based on the video frames and Mel-frequency cepstral coefficients, the preliminary visual features and preliminary audio features are derived using a residual neural network.

[0018] According to the present invention, a deepfake video detection method is provided, wherein the step of extracting video frames of the face region in the video to be detected includes:

[0019] Using a multi-task convolutional neural network model, facial region images are extracted from each frame of the video frame to be detected, thus obtaining the video frame.

[0020] According to the present invention, a deepfake video detection method is provided, wherein the step of extracting Mel-frequency cepstral coefficients from the video to be detected includes:

[0021] The Mel-spectral coefficients are obtained by using audio analysis tools to perform frame segmentation and windowing processing on the audio in the video to be detected.

[0022] According to the present invention, a deepfake video detection method is provided. The method utilizes a composite attention module to perform visual and auditory information interaction on the preliminary audio-visual features to obtain audio-visual features, including:

[0023] The preliminary visual features and preliminary audio features are respectively input into the convolutional layer in the composite attention module for convolution;

[0024] Along the channel dimension of the composite attention module, the convolutional visual preliminary features and audio preliminary features are each divided into self-attention features and cross-modal attention features;

[0025] The self-attention features and cross-modal attention features are input into the composite attention layer in the composite attention module to obtain the self-attention weights of the self-attention features and the cross-modal attention weights of the cross-modal attention features;

[0026] The self-attention features are multiplied by their self-attention weights, and the cross-modal attention features are multiplied by their cross-modal attention weights, and then concatenated to obtain the audio-visual features.

[0027] According to the present invention, a deepfake video detection method is provided. The specific correspondence between the labels corresponding to the audio-visual features of the deepfake video and the audio-visual features of the real video and the audio-visual features of the real video includes:

[0028] The audio and video feature tags of deepfake videos correspond to the following audio and video features: real visual features and fake audio features, fake video features and real audio features, and fake video features and fake audio features.

[0029] The audio-visual feature tags of real videos correspond to the audio-visual features of real videos, including real visual features and real audio features.

[0030] The present invention also provides a deepfake video detection device, comprising:

[0031] Analysis module: Used to perform video feature analysis on the video to be detected, and obtain the audio and video features of the video to be detected;

[0032] Detection module: Inputs the audio and video features of the video to be detected into a preset multilayer perceptron classification model to obtain the detection result;

[0033] The audio-visual features include visual features and audio features. The preset multilayer perceptron classification model is trained using the audio-visual features of deepfake videos and real videos as samples, and the labels corresponding to the audio-visual features of deepfake videos and real videos, respectively.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the deepfake video detection method as described above.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deepfake video detection method as described above.

[0036] This invention provides a method, apparatus, device, and medium for detecting deepfake videos. The method analyzes the video features of the video to be detected to obtain audio-visual features including visual and audio features. The visual and audio features are detected separately using a preset multilayer perceptron classification model. By distinguishing between real and fake video features and audio features, the detection of deepfake videos can be achieved more effectively and accurately. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0038] Figure 1 This is one of the flowcharts of a deepfake video detection method provided by the present invention;

[0039] Figure 2 This is the second flowchart of a deepfake video detection method provided by the present invention;

[0040] Figure 3 This is a flowchart illustrating the composite attention module in a deepfake video detection method provided by the present invention.

[0041] Figure 4 This is a schematic diagram of the structure of a deepfake video detection method and apparatus provided by the present invention;

[0042] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] The following is combined Figures 1-3 A deepfake video detection method according to the present invention includes:

[0045] Step S1: Perform video feature analysis on the video to be detected to obtain the audio and video features of the video to be detected.

[0046] Step S2: Input the audio and video features of the video to be detected into the preset multilayer perceptron classification model to obtain the detection results.

[0047] In step S1, the audio-visual features include visual features and audio features. In step S2, the pre-set multilayer perceptron classification model is trained using the audio-visual features of deepfake videos and real videos as samples, along with the corresponding labels for each of the audio-visual features of deepfake videos and real videos.

[0048] Specifically, the labels corresponding to the audio-visual features of deepfake videos and real videos, and their corresponding relationships, include:

[0049] The audio-visual feature tags corresponding to deepfake videos include: real visual features and fake audio features, fake video features and real audio features, and fake video features and fake audio features.

[0050] The audio-visual feature tags of real videos correspond to the audio-visual features of real videos, including real visual features and real audio features.

[0051] This invention obtains audio-visual features, including visual and audio features, by performing video feature analysis on the video to be detected. The visual and audio features are then detected separately using a pre-set multilayer perceptron classification model. By distinguishing between real and fake video features and audio features, this invention achieves more effective and accurate detection of deepfake videos.

[0052] Please refer to the following: Figure 2 and Figure 3 Specifically, video feature analysis is performed on the video to be detected to obtain its audio and video features, including:

[0053] Preliminary video feature analysis is performed on the video to be tested to obtain preliminary audio and video features.

[0054] By using a composite attention module, visual and auditory information interaction is performed on the preliminary audio and video features to obtain audio and video features.

[0055] The preliminary audio and video features include preliminary visual features with video frames and preliminary audio features with Mel-frequency cepstral coefficients.

[0056] The composite attention module is derived by replacing the intermediate convolutional layers in the residual neural network with composite attention layers. For example, replacing the intermediate convolutional layers in the residual module of ResNet-50 with composite attention layers yields the composite attention module. This composite attention layer consists of self-attention and cross-modal attention mechanisms, with the weights for both derived from visual and auditory features through convolutional layers and a sigmoid activation function. Furthermore, multiple composite attention modules can be stacked, with each layer consisting of a composite attention layer obtained by replacing the intermediate convolutional layers in the residual module of the residual neural network. The composite attention model, constructed from multiple stacked composite attention modules, enhances the feature representation of each modality and facilitates information interaction between visual and audio features, thus improving the detection of each modality.

[0057] To better perform preliminary feature analysis on the video to be tested, this embodiment performs preliminary feature analysis on the video to be tested to obtain preliminary audio and video features, including:

[0058] The video to be detected is processed to extract video frames of the face region and their corresponding Mel-frequency cepstral coefficients. In other words, audio features are extracted at the corresponding timestamps of the extracted video frames.

[0059] Based on video frames and Mel-frequency cepstral coefficients, preliminary visual features and preliminary audio features are obtained using a residual neural network.

[0060] This invention uses a Residual Neural Network (ResNet) as the backbone network. The backbone network extracts preliminary visual and audio features based on video frames and Mel-frequency cepstral coefficients, respectively. The feature extraction network can initially extract features relevant to the video to be detected and reduce the size of the feature map, thereby facilitating effective interaction of information between visual and audio features in the composite attention model.

[0061] The step of extracting and processing the video to be detected to obtain video frames of the face region in the video includes:

[0062] Using a multi-task convolutional neural network model, facial region images are extracted from each frame of the video to be detected, thus obtaining the video frames.

[0063] This invention uses a multi-task convolutional neural network (MTCNN) model to extract face region images from each frame as input to a neural network (residual neural network). This improves the accuracy of face region detection in video frames. The extraction time can be selected as the first 3 seconds of the video to be detected.

[0064] The steps for extracting and processing the video to be tested to obtain Mel-frequency cepstral coefficients include:

[0065] The Mel-spectral coefficients are obtained by performing frame segmentation and windowing on the audio in the video to be tested using an audio analysis tool. An example audio analysis tool is Librosa.

[0066] In this embodiment, the Librosa tool is first used to segment, window, and extract Mel-frequency cepstral coefficients (MFCCs) from the audio in the video to be detected. The MFCCs of each frame of audio are then used as input to a neural network (residual neural network). The extraction time can be selected from the first 3 seconds of the video to be detected. By using the same time for both video and audio, it is possible to detect deepfake videos where the video and audio are aligned.

[0067] To better achieve information interaction between visual and audio features, in this embodiment, the composite attention module is used to perform visual and auditory information interaction on the preliminary audio-visual features to obtain audio-visual features, including:

[0068] The preliminary visual features and preliminary audio features (auditory features) are respectively input into the convolutional layer in the composite attention module for convolution.

[0069] By utilizing the channel dimension of the composite attention module, the convolutional visual preliminary features and audio preliminary features are each divided into self-attention features and cross-modal attention features.

[0070] The self-attention features and cross-modal attention features are input into the composite attention layer in the composite attention module to obtain the self-attention weights of the self-attention features and the cross-modal attention weights of the cross-modal attention features.

[0071] The self-attention features are multiplied by their self-attention weights, and the cross-modal attention features are multiplied by their cross-modal attention weights, and then concatenated to obtain the audio-visual features.

[0072] In this embodiment, visual features are taken as an example and represented as x∈R T×C×H×W Where T represents the number of extracted video frames, C represents the number of feature channels, and H and W represent the height and width of the feature map. First, the preliminary visual features are divided into two parts along the channel dimension: x1∈R. T×C×H×W and x2∈R T×C×H×W (x1 is the attention feature, x2 is the cross-modal attention feature). x1 is input into a convolutional layer to obtain the self-attention weights for the visual features, while x2 is similarly input into a convolutional layer to obtain the cross-modal attention weights for the audio features. The self-attention weights of the visual features and the cross-modal attention weights from the audio features are multiplied by the two parts of the visual features (x1 and x2), respectively, and then concatenated along the channel dimension to obtain the final visual features. The self-attention mechanism allows the neural network to focus more on regions of interest, while the cross-modal attention mechanism enables information interaction between visual and audio features, allowing the model to focus more on easily detectable video and audio frames, thus achieving more effective and accurate detection of deepfake videos.

[0073] The deepfake video detection device provided by the present invention is described below. The deepfake video detection device described below can be referred to in correspondence with the deepfake video detection method described above.

[0074] Please see Figure 4 The present invention also provides a deepfake video detection device, including an analysis module 410 and a detection module 420.

[0075] The analysis module 410 is used to perform video feature analysis on the video to be detected, and obtain the audio and video features of the video to be detected.

[0076] The detection module 420 inputs the audio and video features of the video to be detected into a preset multilayer perceptron classification model to obtain the detection result.

[0077] The audio-visual features include visual features and audio features. The preset multilayer perceptron classification model is trained using the audio-visual features of deepfake videos and real videos as samples, as well as the labels corresponding to the audio-visual features of deepfake videos and real videos, respectively.

[0078] This invention uses analysis module 410 to perform video feature analysis on the video to be detected, obtaining the audio and video features of the video. Then, detection module 420 inputs these audio and video features into a preset multilayer perceptron classification model to obtain the detection result. This achieves more effective and accurate detection of deepfake videos.

[0079] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the deepfake video detection method provided above, which includes:

[0080] S1. Perform video feature analysis on the video to be detected to obtain the audio and video features of the video to be detected.

[0081] S2. Input the audio and video features of the video to be detected into a preset multilayer perceptron classification model to obtain the detection result.

[0082] The audio-visual features include visual features and audio features. The preset multilayer perceptron classification model is trained using the audio-visual features of deepfake videos and real videos as samples, and the labels corresponding to the audio-visual features of deepfake videos and real videos, respectively.

[0083] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aforementioned deepfake video detection method, the method comprising:

[0085] S1. Perform video feature analysis on the video to be detected to obtain the audio and video features of the video to be detected.

[0086] S2. Input the audio and video features of the video to be detected into a preset multilayer perceptron classification model to obtain the detection result.

[0087] The audio-visual features include visual features and audio features. The preset multilayer perceptron classification model is trained using the audio-visual features of deepfake videos and real videos as samples, and the labels corresponding to the audio-visual features of deepfake videos and real videos, respectively.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for deepfake video detection, the method comprising: The method comprises the following steps: video feature analysis is performed on the to-be-detected video to obtain audio-visual features of the to-be-detected video; the audio-visual features of the to-be-detected video are input into a preset multi-layer perception machine classification model to obtain a detection result; wherein the audio-visual features include visual features and audio features, and the preset multi-layer perception machine classification model is trained by using audio-visual features of deepfake videos and audio-visual features of real videos as samples and labels corresponding to the audio-visual features of the deepfake videos and the real videos respectively; the video feature analysis on the to-be-detected video to obtain the audio-visual features of the to-be-detected video comprises the following steps: video preliminary feature analysis is performed on the to-be-detected video to obtain audio-visual preliminary features; a composite attention module is used to perform information interaction of vision and hearing on the audio-visual preliminary features to obtain audio-visual features; wherein the audio-visual preliminary features include visual preliminary features with video frames and audio preliminary features with mel-frequency cepstrum coefficients; the composite attention module is obtained by replacing an intermediate convolution layer in a residual module of a residual neural network with a composite attention layer; the specific corresponding relationship between the labels corresponding to the audio-visual features of the deepfake videos and the real videos and the audio-visual features of the deepfake videos and the real videos comprises: the audio-visual feature labels of the deepfake videos correspond to audio-visual features of the deepfake videos, which include real visual features and fake audio features, fake video features and real audio features, and fake video features and fake audio features; the audio-visual feature labels of the real videos correspond to audio-visual features of the real videos, which include real visual features and real audio features.

2. The deepfake video detection method of claim 1, wherein, the video preliminary feature analysis on the to-be-detected video to obtain the audio-visual preliminary features comprises the following steps: video frames and mel-frequency cepstrum coefficients of a face region in the to-be-detected video are extracted; the residual neural network is used to obtain the visual preliminary features and the audio preliminary features respectively according to the video frames and the mel-frequency cepstrum coefficients.

3. The deepfake video detection method according to claim 2, wherein the step of extracting the video frames of the face region in the to-be-detected video comprises: a multi-task convolutional neural network model is used to extract face region pictures in each frame of the video frames of the to-be-detected video to obtain the video frames.

4. The deepfake video detection method according to claim 2, wherein the step of extracting the mel-frequency cepstrum coefficients in the to-be-detected video comprises: after the audio analysis tool is used to perform frame division and windowing processing on the audio in the to-be-detected video, the mel-frequency cepstrum coefficients are obtained.

5. The deepfake video detection method according to any one of claims 1 to 4, characterized in that, the step of using the composite attention module to perform information interaction of vision and hearing on the audio-visual preliminary features to obtain audio-visual features comprises: the visual preliminary features and the audio preliminary features are respectively input into convolution layers in the composite attention module for convolution; the visual preliminary features and the audio preliminary features after the convolution are respectively divided into self-attention features and cross-modal attention features along a channel dimension of the composite attention module; input the self-attention feature and the cross-modal attention feature into a composite attention layer in the composite attention module to obtain a self-attention weight of the self-attention feature and a cross-modal attention weight of the cross-modal attention feature; multiply the self-attention feature by the self-attention weight and multiply the cross-modal attention feature by the cross-modal attention weight, and then splice them to obtain the audio-visual feature.

6. A deepfake video detection apparatus, comprising: Comprise: an analysis module configured to perform video feature analysis on a to-be-detected video to obtain an audio-visual feature of the to-be-detected video; a detection module configured to input the audio-visual feature of the to-be-detected video into a preset multi-layer perception classification model to obtain a detection result; wherein the audio-visual feature comprises a visual feature and an audio feature, the preset multi-layer perception classification model is trained using, as samples, audio-visual features of deepfake videos and audio-visual features of real videos, and labels corresponding to the audio-visual features of the deepfake videos and the audio-visual features of the real videos, respectively; the video feature analysis on the to-be-detected video to obtain the audio-visual feature of the to-be-detected video comprises: performing preliminary video feature analysis on the to-be-detected video to obtain an audio-visual preliminary feature; using a composite attention module to perform information interaction between vision and hearing on the audio-visual preliminary feature to obtain the audio-visual feature; wherein the audio-visual preliminary feature comprises a visual preliminary feature having video frames and an audio preliminary feature having mel-frequency cepstral coefficients; the composite attention module is obtained by replacing an intermediate convolution layer in a residual module of a residual neural network with a composite attention layer; the specific corresponding relationship between the labels corresponding to the audio-visual features of the deepfake videos and the audio-visual features of the real videos and the audio-visual features of the deepfake videos and the audio-visual features of the real videos comprises: the audio-visual feature label of the deepfake video corresponds to an audio-visual feature of the deepfake video, which comprises real visual features and fake audio features, fake video features and real audio features, and fake video features and fake audio features; the audio-visual feature label of the real video corresponds to an audio-visual feature of the real video, which comprises real visual features and real audio features.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the deepfake video detection method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the deepfake video detection method of any one of claims 1-5.

Images