Deep counterfeit video detection method and device fusing multi-modal information

Abstract

The invention relates to a deep counterfeit video detection method and device fusing multi-modal information, and is applied to the technical field of video detection. According to the invention, a multi-head self-attention mechanism is adopted to process a video frame feature vector Xv of a video to be detected and an audio feature vector Xa of an audio MEL spectrogram respectively, and then weighted fusion is carried out on two modal features of the video frame feature vector Xv of the video to be detected and the audio feature vector Xa of the audio MEL spectrogram. Therefore, the accuracy and robustness of the deep counterfeit video detection method are improved. The multi-modal feature fusion can make full use of the complementarity of different modal information to ensure that more comprehensive features are captured, thereby improving the accuracy and robustness of deep counterfeit video detection.

Description

Technical Field

[0001] The present invention belongs to the field of video detection technology, and specifically relates to a deep fake video detection method and device that integrates multimodal information. Background Art

[0002] Deep fake technology is a technology based on artificial intelligence and deep learning. It can generate highly realistic fake images, videos and voices by simulating and forging human facial features, voices and behaviors. At present, video face-changing and voice-changing technologies are not only widely used in film and television production, but also abused to create fake news and malicious videos, posing serious security threats to personal privacy, social stability and national security.

[0003] At present, common deep fake detection technologies mainly include: detection technology based on traditional digital forensics technology and detection technology based on deep learning models.

[0004] (1) Based on traditional digital forensics technology, for the audio part, the main feature analysis is performed on the audio, waveform, spectrum and other sound signals; for the video and graphic parts, the metadata, color distribution, pixel arrangement and compression traces of the video and image are usually used to determine whether the image and video have been edited and forged. However, this technology mainly relies on static analysis of the video or image content. However, since the number of frames will be severely degraded after the video is compressed, and the timing characteristics between video frames have certain changes, most deep fake image detection methods based on static features cannot be directly used for deep fake video detection.

[0005] (2) Detection technology based on deep learning models, which automatically detects deep fake content using trained detection models. However, with the advancement of deep fake technology, single-modality detection methods have great limitations when dealing with complex and highly realistic deep fake videos, especially after the fake content has been compressed, transmitted or edited, the accuracy of single-modality detection methods has dropped significantly. In addition, multimodal information in deep fake videos that have been compressed, transmitted or edited often has inconsistencies, and single-modality detection methods cannot fully capture these cross-modal inconsistencies, resulting in low detection accuracy.

[0006] In view of this, the present invention is proposed. Summary of the invention

[0007] The present invention proposes a deep fake video detection method and device that integrates multimodal information, so as to solve the problem that the existing deep fake video detection technology relies on a single modality detection method, resulting in low detection accuracy for complex and highly realistic deep fake videos.

[0008] In a first aspect, the present invention proposes a deep fake video detection method integrating multimodal information, comprising: S1. Preprocessing the video to be detected to obtain multiple video frames; the video to be detected includes audio, and each video frame includes the audio; S2. Obtain the audio file in the video to be detected, and convert the audio file into a MEL spectrogram; S3. For each video frame of the multiple video frames, extract the feature vector of each video frame, and fuse the feature vector of each video frame to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a ; S4. The video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a Perform linear transformations respectively and map them into query (Query, Q), key (Key, K) and value (Value, V) spaces to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The characteristic dimension of the attention mechanism d k 、 The video frame feature vector of the video to be detected X v The query vector Q v , key vector K v , value vector V v and the audio feature vector of the audio MEL spectrogram X a The query vector Q a , key vector K a , value vector V a , specifically: ; in, Wq v ,Wk v ,Wv v is the video frame feature vector of the video to be detectedX v The weight matrix for linear transformation, Wq a ,Wk a ,Wv a is the audio feature vector of the audio MEL spectrogram X a The weight matrix for the transformation; S5. Use a multi-head self-attention mechanism to process the video frame feature vectors of the video to be detected respectively X v and the audio feature vector of the audio MEL spectrogram X a , calculate the video frame feature vector of the video to be detected X v The attention weight matrix Attention v and the audio feature vector of the audio MEL spectrogram X a The attention weight matrix Attention a ， Specifically: ; in, d k To use for Q * K T The product result is normalized; S6. Introduce an inter-modal interactive attention mechanism to compare the video frame feature vectors of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The similarities between them are used to enhance the interaction between modalities, specifically: ; S7. The video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The weighted fusion of the two modal features is used to obtain the final multimodal fusion feature F , specifically: ; in, is a hyperparameter; S8. The multimodal fusion features FInput is a multi-layer fully connected neural network consisting of a batch normalization layer, a fully connected layer, an activation function, and an output layer. D Get the multimodal fusion features after dimensionality reduction F’ , specifically: ; in, W is the weight, F is the multimodal fusion feature, b It is a deviation; S9. The multimodal fusion features after dimension reduction F’ Input the classifier to get the classification result , obtain the probability that the video to be detected is a real category and a forged category, and determine whether the video to be detected is a forged video by comparing the probability values ​​of the real category and the forged category of the video to be detected, specifically: ; S10. When the probability that the video to be detected is a forged video is greater than the probability that the video to be detected is a real video, the video to be detected is determined to be a forged video; otherwise, the video to be detected is determined to be a forged video.

[0009] Furthermore, the video to be detected is preprocessed as described in S1 to obtain multiple video frames, including: S101. Use the VideoCapture function in the computer vision library OpenCV to read the video to be detected and determine the total number of frames of the video to be detected; S102. Calculate the interval of the video frames to be extracted according to the total number of frames of the video to be detected, and evenly extract multiple video frames from the video to be detected; S103. Perform data normalization processing on the multiple video frames to convert them into floating-point data.

[0010] Furthermore, in S2, an audio file in the video to be detected is obtained, and the audio file is converted into a MEL spectrogram, including: S201. Read the video file to be detected by calling the VideoFileClip function in the video processing library MoviePy, obtain the audio file in the video file to be detected, and save the audio file in WAV format; S202. Load the WAV audio file by calling the audio processing library Librosa library, and store the audio data in the WAV audio file in the form of a time series; S203. By calling the melspectrogram function in the audio processing library Librosa library, the audio data in the WAV audio file is converted into a MEL spectrogram, and the floating-point data of the MEL spectrogram is normalized into integer value data; S204. Resize the normalized MEL spectrum graph and scale the normalized MEL spectrum graph to a specified target size.

[0011] Furthermore, in S3, a video frame feature vector of the detected video is obtained. X v ,include: S301. Divide each of the plurality of video frames input into the Video Swin Transforme model into a plurality of small image blocks, and convert the plurality of small image blocks into high-dimensional feature vectors through a linear embedding layer in the Video Swin Transforme model; S302. Introduce temporal coding, embed the time sequence information between the multiple video frames into the high-dimensional feature vector, and obtain the video frame feature vector of the detection video after being processed by the Video Swin Transforme model. X v .

[0012] Further, in S3, the audio feature vector of the audio MEL spectrum graph is obtained. X a ， include: S303. Input the audio MEL spectrogram into the pre-trained ResNet18 model, and perform deep feature extraction on the audio MEL spectrogram through multiple convolutional layers and residual modules of the ResNet18 model; S304. Through the global average pooling layer of the ResNet18 model, the audio MEL spectrogram is compressed into an audio feature vector X a , the audio feature vector X a Contains the time-frequency information of the audio signal.

[0013] Furthermore, the multimodal fusion features after dimensionality reduction obtained in S8 are F’ ,include: S801. The multimodal fusion feature F Input the batch normalization layer to obtain multimodal fusion features F1 , ensuring that the multimodal fusion features F1 has zero mean and unit variance; S802. The multimodal fusion feature F1 Input to the fully connected layer to fusion the multimodal features F1 Perform dimensionality reduction to obtain multimodal fusion features F2 ; S803. The multimodal fusion feature F2 Input the batch normalization layer to obtain multimodal fusion features F3； S804. The multimodal fusion feature F3 Through the Tanh activation function, nonlinear transformation is performed to obtain multimodal fusion features. F4， in: ; S805. The multimodal fusion feature F4 Input to the output layer, the multimodal fusion features F4 Compression into multimodal fusion features F’， The multimodal fusion feature F’ is a two-dimensional vector 。

[0014] In a second aspect, the present invention proposes a deep fake video detection device that integrates multimodal information, which adopts any deep fake video detection method that integrates multimodal information as described in the first aspect, including: Data processing module 210: pre-processing the video to be detected, obtaining multiple video frames of the video to be detected and an audio file in the video to be detected, and converting the audio file into a MEL spectrogram; Feature vector extraction module 220: converting the multiple video frames of the video to be detected and the audio file into a MEL spectrum to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a ; Multimodal attention fusion module 230: according to the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a , the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a Perform multimodal cross-attention feature fusion to obtain multimodal fusion features F ; Video forgery detection module 240: The multimodal fusion featureF Input multi-layer fully connected neural network D Get the multimodal fusion features after dimensionality reduction F’， The multimodal fusion features after dimension reduction F’ Input the classifier to get the classification result , determine whether the video to be detected is a forged video.

[0015] In a third aspect, the present invention further provides an electronic device, comprising: a memory and a processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes any deep fake video detection method that integrates multimodal information as described in the first aspect of the present invention.

[0016] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to enable the processor to execute any deep fake video detection method integrating multimodal information as described in the first aspect of the present invention.

[0017] The beneficial effect of the present invention is that the deep fake video detection method and device that integrates multimodal information proposed in the technical solution can solve the problem of inconsistencies in the fake content after compression, transmission or editing, by combining the visual information, audio information and the cross-modal relationship in the video, thereby improving the accuracy and robustness of fake video detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0019] Figure 1 It is a schematic diagram of the video forgery detection method integrating multimodal information of the present invention.

[0020] Figure 2 It is a schematic diagram of a video forgery detection device integrating multimodal information according to the present invention.

[0021] Explanation of reference numerals: data processing module 210; feature vector extraction module 220; multimodal attention fusion module 230; video forgery detection module 240. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0024] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0025] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0026] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0027] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0028] In the first aspect, as shown in FIG1 , the present invention proposes a deep fake video detection method integrating multimodal information, comprising: S1. Preprocessing the video to be detected to obtain multiple video frames; the video to be detected includes audio, and each video frame includes the audio; S2. Obtain the audio file in the video to be detected, and convert the audio file into a MEL spectrogram; S3. For each video frame of the multiple video frames, extract the feature vector of each video frame, and fuse the feature vector of each video frame to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a ; S4. The video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a Perform linear transformations respectively and map them into query (Query, Q), key (Key, K) and value (Value, V) spaces to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The characteristic dimension of the attention mechanism d k 、 The video frame feature vector of the video to be detected X v The query vector Q v , key vector K v , value vector V v and the audio feature vector of the audio MEL spectrogram X a The query vector Q a , key vector K a , value vector V a , specifically: ; in, Wq v ,Wk v ,Wv v is the video frame feature vector of the video to be detected X v The weight matrix for linear transformation, Wq a ,Wk a ,Wva is the audio feature vector of the audio MEL spectrogram X a The weight matrix for the transformation; S5. Use a multi-head self-attention mechanism to process the video frame feature vectors of the video to be detected respectively X v and the audio feature vector of the audio MEL spectrogram X a , calculate the video frame feature vector of the video to be detected X v The attention weight matrix Attention v and the audio feature vector of the audio MEL spectrogram X a The attention weight matrix Attention a ， Specifically: ; in, d k To use for Q * K T The product result is normalized; S6. Introduce an inter-modal interactive attention mechanism to compare the video frame feature vectors of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The similarities between them are used to enhance the interaction between modalities, specifically: ; S7. The video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The weighted fusion of the two modal features is used to obtain the final multimodal fusion feature F , specifically: ; in, is a hyperparameter; S8. The multimodal fusion features F Input is a multi-layer fully connected neural network consisting of a batch normalization layer, a fully connected layer, an activation function, and an output layer. D Get the multimodal fusion features after dimensionality reduction F’ , specifically: ; in, W is the weight, F is the multimodal fusion feature, b It is a deviation; S9. The multimodal fusion features after dimension reduction F’ Input the classifier to get the classification result , obtain the probability that the video to be detected is a real category and a forged category, and determine whether the video to be detected is a forged video by comparing the probability values ​​of the real category and the forged category of the video to be detected, specifically: ; S10. When the probability that the video to be detected is a forged video is greater than the probability that the video to be detected is a real video, the video to be detected is determined to be a forged video; otherwise, the video to be detected is determined to be a forged video.

[0029] Among them, in S5, the attention weight matrix is ​​generated by calculating the dot product of the query vector and the key vector; softmax The function comes with the attention mechanism formula. Its function is to normalize the attention weights and convert them into probability distribution. After obtaining the multi-head attention output, residual connection and layer normalization are used to improve the stability of training and enhance the retention of information.

[0030] In S6, the inter-modal interactive attention mechanism is introduced to help capture the video frame feature vector of the video to be detected. X v and the audio feature vector of the audio MEL spectrogram X a The potential correlation between them is obtained by using the self-attention mechanism and the interactive attention mechanism. X v and the audio feature vector of the audio MEL spectrogram X a After processing and generating new feature representations F v and F a Through the multimodal self-attention mechanism and inter-modal interactive attention, potential consistency and anomalies between video frames and audio, such as lip movement and speech synchronization, can be captured.

[0031] In S7, the multimodal fusion feature F Representing the deep association of video and audio information can provide rich semantic information for deep fake video detection.

[0032] In S8, the multi-layer fully connected neural network D is composed of a batch normalization layer, a fully connected layer, an activation function and an output layer, which can effectively compress the feature space and enhance the nonlinear expression ability of the model, thereby accurately predicting whether the video to be detected is forged.

[0033] The present invention adopts a multi-head self-attention mechanism to process the video frame feature vectors of the video to be detected respectively. X v and the audio feature vector of the audio MEL spectrogram X a , and then the video frame feature vector of the video to be detected is X v and the audio feature vector of the audio MEL spectrogram X a The two modal features are weightedly fused to improve the accuracy and robustness of the deep fake video detection method. The fusion of multimodal features can make full use of the complementarity of different modal information to ensure that more comprehensive features are captured, thereby improving the accuracy and robustness of deep fake video detection.

[0034] The present invention requires the video to be detected to be: (1) The frame rate and resolution of the video to be detected can remain clear and stable. If the video to be detected includes a face, the audio in the video to be detected is the voice of the person corresponding to the face; (2) The video to be detected includes audio, and the audio part requires time synchronization between the audio and video content.

[0035] It should be noted that the deep fake video detection method that integrates multimodal information provided by the present invention is suitable for detecting whether a video has been forged, including tampering with a clip. The executor of the method can be a forged video detection device, such as an electronic device, or a control module in the forged video detection device for executing the deep fake video detection method that integrates multimodal information.

[0036] In addition, in terms of video processing, the present invention uses the Video Swin Transformer model to extract visual spatiotemporal features, paying special attention to the detailed information and dynamic changes of video frames, so as to better capture the subtle anomalies generated during the deep fake process. In audio processing, the present invention uses the ResNet18 pre-trained model to process the MEL spectrogram to capture subtle changes during audio forgery. In contrast, although the paper "Joint Audio-Visual Deepfake Detection" also uses video frame information, its processing method focuses more on local inter-frame correlations, while the present invention focuses on global dynamics and complex spatiotemporal change patterns. The paper "Joint Audio-Visual Deepfake Detection" tends to rely on simple pattern matching of voice content in audio detection, and does not deeply analyze the frequency and time domain characteristics of audio signals.

[0037] In addition, the present invention adopts a multimodal feature fusion method based on the attention mechanism, using self-attention and inter-modal interactive attention to establish a more detailed association between visual and audio features, ensuring more accurate feature alignment and fusion. However, the fusion process in the paper "Joint Audio-Visual Deepfake Detection" is relatively simple and does not fully capture the complex dynamic relationship between audio and video, resulting in a decrease in detection accuracy when facing advanced counterfeiting techniques.

[0038] Furthermore, the video to be detected is preprocessed as described in S1 to obtain multiple video frames, including: S101. Use the VideoCapture function in the computer vision library OpenCV to read the video to be detected and determine the total number of frames of the video to be detected; S102. Calculate the interval of the video frames to be extracted according to the total number of frames of the video to be detected, and evenly extract multiple video frames from the video to be detected; S103. Perform data normalization processing on the multiple video frames to convert them into floating-point data.

[0039] Wherein, the reading of the video to be detected in step S101 is performed through the OpenCV library, so the total number of frames of the video to be detected can be easily obtained.

[0040] The purpose of evenly extracting video frames from the video to be detected in step S102 is to ensure that the multiple video frames extracted from the video to be detected are key video frames and are representative.

[0041] The data preprocessing of the multiple video frames in step S103 is to ensure that the video frame features are suitable for input into the deep learning model so as to be compatible with the deep learning model. The preprocessing process of the multiple video frames is as follows: first, resize each of the multiple video frames; then adjust the pixel value range of the multiple video frames, and normalize the pixel value from the integer range of 0-255 to a floating point number between 0-1. Convert the processed multiple video frames to the torch.tensor data type, and specify the data type as a floating point number to be compatible with the deep learning model.

[0042] Furthermore, in S2, an audio file in the video to be detected is obtained, and the audio file is converted into a MEL spectrogram, including: S201. Read the video file to be detected by calling the VideoFileClip function in the video processing library MoviePy, obtain the audio file in the video file to be detected, and save the audio file in WAV format; S202. Load the WAV audio file by calling the audio processing library Librosa library, and store the audio data in the WAV audio file in the form of a time series; S203. By calling the melspectrogram function in the audio processing library Librosa library, the audio data in the WAV audio file is converted into a MEL spectrogram, and the floating-point data of the MEL spectrogram is normalized into integer value data; S204. Resize the normalized MEL spectrum graph and scale the normalized MEL spectrum graph to a specified target size.

[0043] In step S204, the scaled MEL spectrum graph may be used for subsequent model input or other processing steps to adapt to different input size requirements, and finally return the processed MEL spectrum graph.

[0044] Furthermore, in S3, a video frame feature vector of the detected video is obtained. X v ,include: S301. Divide each of the plurality of video frames input into the Video Swin Transforme model into a plurality of small image blocks, and convert the plurality of small image blocks into high-dimensional feature vectors through a linear embedding layer in the Video Swin Transforme model; S302. Introduce temporal coding, embed the time sequence information between the multiple video frames into the high-dimensional feature vector, and obtain the video frame feature vector of the detection video after being processed by the Video Swin Transforme model. X v .

[0045] In order to efficiently extract the spatiotemporal features of video frames, the present invention adopts the Video Swin Transforme pre-trained model, which is a self-attention mechanism model based on the Swin Transformer architecture. It is particularly suitable for processing complex spatiotemporal information in videos. Its multi-scale convolution method can capture the global features in the video frames while maintaining the local features of the video frames, thereby effectively extracting dynamically changing details.

[0046] In S301, the Video Swin Transformer model captures the temporal and spatial associations between the multiple video frames through its multi-layer self-attention mechanism. The Video Swin Transformer model can extract local features from the video frames and establish long-distance dependencies between the video frames, thereby making full use of the temporal and spatial information of the video. Using the self-attention mechanism of Swin Transformer, the Video Swin Transformer model can gradually fuse local and global temporal and spatial information to extract deep semantic features.

[0047] In S302, the temporal coding is the position coding in the Transformer model, the purpose of which is to embed the position information into the feature, so that the Transformer retains the position information of the feature, which can improve the model's ability to understand the sequence. After being processed by the Video Swin Transforme model, the video frame feature vector obtained by fusion of the feature vectors of the multiple video frames is obtained, that is, the video frame feature vector of the detection video. X v , for alignment with the audio features.

[0048] Further, in S3, the audio feature vector of the audio MEL spectrum graph is obtained. X a ， include: S303. Input the audio MEL spectrogram into the pre-trained ResNet18 model, and perform deep feature extraction on the audio MEL spectrogram through multiple convolutional layers and residual modules of the ResNet18 model; S304. Through the global average pooling layer of the ResNet18 model, the audio MEL spectrogram is compressed into an audio feature vectorX a , the audio feature vector X a Contains the time-frequency information of the audio signal.

[0049] Among them, in S303, the ResNet18 model can effectively extract multi-scale features in the audio spectrum and identify key patterns and frequency changes in the audio signal through hierarchical convolution and residual connection. The initial convolution layer is used to extract the basic features of the audio spectrum. Through residual connection, the ResNet18 model can more effectively capture complex patterns in the audio and avoid the gradient vanishing problem of the deep network.

[0050] There is only one audio feature of the video to be detected. In S304, the ResNet18 model extracts the features of the audio MEL spectrogram into a vector of a fixed length, which represents the time-frequency pattern and its feature distribution in the audio, as the output of the audio feature.

[0051] Furthermore, the multimodal fusion features after dimensionality reduction obtained in S8 are F’ ,include: S801. The multimodal fusion feature F Input the batch normalization layer to obtain multimodal fusion features F1 , ensuring that the multimodal fusion features F1 has zero mean and unit variance; S802. The multimodal fusion feature F1 Input to the fully connected layer to fusion the multimodal features F1 Perform dimensionality reduction to obtain multimodal fusion features F2 ; S803. The multimodal fusion feature F2 Input the batch normalization layer to obtain multimodal fusion features F3； S804. The multimodal fusion feature F3 Through the Tanh activation function, nonlinear transformation is performed to obtain multimodal fusion features. F4， in: ; S805. The multimodal fusion feature F4 Input to the output layer, the multimodal fusion features F4 Compression into multimodal fusion features F’， The multimodal fusion feature F’ is a two-dimensional vector 。

[0052] Among them, in S802, the multimodal fusion feature F1The purpose of dimensionality reduction is to extract more representative multimodal fusion features, while reducing feature dimensions and computational complexity.

[0053] The multimodal fusion feature in S805 F’ Used to evaluate the video to be detected for a binary classification task.

[0054] In a second aspect, the present invention proposes a deep fake video detection device that integrates multimodal information, which adopts any deep fake video detection method that integrates multimodal information as described in the first aspect, including: Data processing module 210: pre-processing the video to be detected, obtaining multiple video frames of the video to be detected and an audio file in the video to be detected, and converting the audio file into a MEL spectrogram; Feature vector extraction module 220: converting the multiple video frames of the video to be detected and the audio file into a MEL spectrum to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a ; Multimodal attention fusion module 230: according to the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a , the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a Perform multimodal cross-attention feature fusion to obtain multimodal fusion features F ; Video forgery detection module 240: The multimodal fusion feature F Input multi-layer fully connected neural network D Get the multimodal fusion features after dimensionality reduction F’， The multimodal fusion features after dimension reduction F’ Input the classifier to get the classification result , determine whether the video to be detected is a forged video.

[0055] In a third aspect, the present invention further provides an electronic device, comprising: a memory and a processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes any deep fake video detection method that integrates multimodal information as described in the first aspect of the present invention.

[0056] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to enable the processor to execute any deep fake video detection method integrating multimodal information as described in the first aspect of the present invention.

[0057] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0058] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0059] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0060] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0061] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep fake video detection method integrating multimodal information, characterized in that: include: S1. Preprocessing the video to be detected to obtain multiple video frames; the video to be detected includes audio, and each video frame includes the audio; S2. Obtain the audio file in the video to be detected, and convert the audio file into a MEL spectrogram; S3. For each video frame of the multiple video frames, extract the feature vector of each video frame, and fuse the feature vector of each video frame to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a ; S4. The video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a Perform linear transformations respectively and map them into query (Query, Q), key (Key, K) and value (Value, V) spaces to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The characteristic dimension of the attention mechanism d k 、 The video frame feature vector of the video to be detected X v The query vector Q v , key vector K v , value vector V v and the audio feature vector of the audio MEL spectrogram X a The query vector Q a , key vector K a , value vector V a , specifically: ； in, Wlq v , Wk v , Wv v is the video frame feature vector of the video to be detected X v The weight matrix for linear transformation, Wlq a , Wk a , Wv a is the audio feature vector of the audio MEL spectrogram X a The weight matrix for the transformation; S5. Use a multi-head self-attention mechanism to process the video frame feature vectors of the video to be detected respectively X v and the audio feature vector of the audio MEL spectrogram X a , calculate the video frame feature vector of the video to be detected X v The attention weight matrix Attention v and the audio feature vector of the audio MEL spectrogram X a The attention weight matrix Attention a ， Specifically: ； in, d k To use for Q * K T The product result is normalized; S6. Introduce an inter-modal interactive attention mechanism to compare the video frame feature vectors of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The similarities between them are used to enhance the interaction between modalities, specifically: ； S7. The video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a The weighted fusion of the two modal features is used to obtain the final multimodal fusion feature F , specifically: ； in, is a hyperparameter; S8. The multimodal fusion features F Input is a multi-layer fully connected neural network consisting of a batch normalization layer, a fully connected layer, an activation function, and an output layer. D Get the multimodal fusion features after dimensionality reduction F’ , specifically: ； in, W is the weight, F is the multimodal fusion feature, b It is a deviation; S9. The multimodal fusion features after dimension reduction F’ Input the classifier to get the classification result , obtain the probability that the video to be detected is a real category and a forged category, and determine whether the video to be detected is a forged video by comparing the probability values ​​of the real category and the forged category of the video to be detected, specifically: ； S10. When the probability that the video to be detected is a forged video is greater than the probability that the video to be detected is a real video, the video to be detected is determined to be a forged video; otherwise, the video to be detected is determined to be a forged video.

2. A deep fake video detection method integrating multimodal information as claimed in claim 1, characterized in that: As described in S1, the video to be detected is preprocessed to obtain multiple video frames, including: S101. Use the VideoCapture function in the computer vision library OpenCV to read the video to be detected and determine the total number of frames of the video to be detected; S102. Calculate the interval of the video frames to be extracted according to the total number of frames of the video to be detected, and evenly extract multiple video frames from the video to be detected; S103. Perform data normalization processing on the multiple video frames to convert them into floating-point data.

3. A deep fake video detection method integrating multimodal information as claimed in claim 1, characterized in that: S2 obtains the audio file in the video to be detected and converts the audio file into a MEL spectrogram, including: S201. Read the video file to be detected by calling the VideoFileClip function in the video processing library MoviePy, obtain the audio file in the video file to be detected, and save the audio file in WAV format; S202. Load the WAV audio file by calling the audio processing library Librosa library, and store the audio data in the WAV audio file in the form of a time series; S203. By calling the melspectrogram function in the audio processing library Librosa library, the audio data in the WAV audio file is converted into a MEL spectrogram, and the floating-point data of the MEL spectrogram is normalized into integer value data; S204. Resize the normalized MEL spectrum graph and scale the normalized MEL spectrum graph to a specified target size.

4. A deep fake video detection method integrating multimodal information as claimed in claim 1, characterized in that: S3 obtains the video frame feature vector of the detected video X v ,include: S301. Divide each of the plurality of video frames input into the Video Swin Transforme model into a plurality of small image blocks, and convert the plurality of small image blocks into high-dimensional feature vectors through a linear embedding layer in the Video Swin Transforme model; S302. Introduce temporal coding, embed the time sequence information between the multiple video frames into the high-dimensional feature vector, and obtain the video frame feature vector of the detection video after being processed by the Video Swin Transforme model. X v .

5. A deep fake video detection method integrating multimodal information as claimed in claim 1, characterized in that: S3 obtains the audio feature vector of the audio MEL spectrogram X a ， include: S303. Input the audio MEL spectrogram into the pre-trained ResNet18 model, and perform deep feature extraction on the audio MEL spectrogram through multiple convolutional layers and residual modules of the ResNet18 model; S304. Through the global average pooling layer of the ResNet18 model, the audio MEL spectrogram is compressed into an audio feature vector X a , the audio feature vector X a Contains the time-frequency information of the audio signal.

6. A deep fake video detection method integrating multimodal information as claimed in claim 1, characterized in that: The multimodal fusion features after dimensionality reduction obtained in S8 F’ ,include: S801. The multimodal fusion feature F Input the batch normalization layer to obtain multimodal fusion features F1 , ensuring that the multimodal fusion features F1 has zero mean and unit variance; S802. The multimodal fusion feature F1 Input to the fully connected layer to fusion the multimodal features F1 Perform dimensionality reduction to obtain multimodal fusion features F2 ; S803. The multimodal fusion feature F2 Input the batch normalization layer to obtain multimodal fusion features F3； S804. The multimodal fusion feature F3 Through the Tanh activation function, nonlinear transformation is performed to obtain multimodal fusion features. F4， in: ； S805. The multimodal fusion feature F4 Input to the output layer, the multimodal fusion features F4 Compression into multimodal fusion features F’， The multimodal fusion feature F’ is a two-dimensional vector 。 7. A deep fake video detection device integrating multimodal information, adopting the deep fake video detection method integrating multimodal information as claimed in any one of claims 1 to 6, comprising: Data processing module 210: pre-processing the video to be detected, obtaining multiple video frames of the video to be detected and an audio file in the video to be detected, and converting the audio file into a MEL spectrogram; Feature vector extraction module 220: converting the multiple video frames of the video to be detected and the audio file into a MEL spectrum to obtain the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a ; Multimodal attention fusion module 230: according to the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a , the video frame feature vector of the video to be detected X v and the audio feature vector of the audio MEL spectrogram X a Perform multimodal cross-attention feature fusion to obtain multimodal fusion features F ; Video forgery detection module 240: The multimodal fusion feature F Input multi-layer fully connected neural network D Get the multimodal fusion features after dimensionality reduction F’， The multimodal fusion features after dimension reduction F’ Input the classifier to get the classification result , determine whether the video to be detected is a forged video.

8. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the deep fake video detection method that integrates multimodal information as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: A plurality of instructions are stored therein, and the instructions are loaded by a processor, so that the processor executes a deep fake video detection method integrating multimodal information as claimed in any one of claims 1 to 6.

Images