A deep fake video detection method and device fusing multi-modal information

Abstract

The application relates to a deep fake video detection method and device fusing multi-modal information, and is applied to the technical field of video detection. X v and an audio feature vector of an audio MEL spectrum diagram X a Then, two modal features of the video frame feature vector of the to-be-detected video X v and the audio feature vector of the audio MEL spectrum diagram X a are weighted and fused, so that the accuracy and robustness of the deep fake video detection method are improved. The fusion of multi-modal features can fully utilize the complementarity of different modal information, ensure that more comprehensive features are captured, and thus the accuracy and robustness of the deep fake video detection are improved.

Description

Technical Field

[0001] The present invention belongs to the field of video detection technology, and specifically relates to a deep fake video detection method and device that integrates multimodal information. Background Art

[0002] Deepfake technology, based on artificial intelligence and deep learning, can generate highly realistic fake images, videos, and audio by simulating and forging human facial features, voices, and behaviors. Currently, technologies such as face-swapping and voice-swapping in videos are widely used in film and television production.

[0003] At present, common deep fake detection technologies mainly include: detection technology based on traditional digital forensics technology and detection technology based on deep learning models.

[0004] (1) Based on traditional digital forensics technology, for the audio part, the main focus is on feature analysis of sound signals such as audio, waveform, and spectrum; for the video and graphic parts, features such as metadata, color distribution, pixel arrangement, and compression traces of the video and image are usually used to determine whether the image and video have been edited and forged. However, this technology mainly relies on static analysis of the video or image content. However, since the number of frames will be severely degraded after the video is compressed, and there are certain changes in the temporal characteristics between video frames, most deep fake image detection methods based on static features cannot be directly used for the detection of deep fake videos.

[0005] (2) Detection technology based on deep learning models, which automatically detects deep fake content using trained detection models. However, with the advancement of deep fake technology, single-modality detection methods have significant limitations when dealing with complex, highly realistic deep fake videos. In particular, the accuracy of single-modality detection methods decreases significantly after the fake content is compressed, transmitted, or edited. In addition, multimodal information in deep fake videos that have been compressed, transmitted, or edited often has inconsistencies. Single-modality detection methods cannot fully capture these cross-modal inconsistencies, resulting in low detection accuracy.

[0006] In view of this, the present invention is proposed. Summary of the Invention

[0007] The present invention proposes a deep fake video detection method and device that integrates multimodal information to solve the problem that existing deep fake video detection technology relies on a single modality detection method, resulting in low detection accuracy for complex and highly realistic deep fake videos.

[0008] In a first aspect, the present invention proposes a deep fake video detection method that integrates multimodal information, comprising:

[0009] S1. The video to be detected is preprocessed to obtain multiple video frames; the video to be detected includes audio, and each video frame includes the audio;

[0010] S2 obtains the audio file in the video to be detected and converts the audio file into a MEL spectrogram;

[0011] S3. For each video frame of the plurality of video frames, extract the feature vector of each video frame, and fuse the feature vectors of each video frame to obtain the video frame feature vector X of the video to be detected. v and the audio feature vector X of the audio MEL spectrogram a ;

[0012] S4. The video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a Perform linear transformations respectively and map them to the query Q, key K and value V spaces to obtain the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The feature dimension d of the attention mechanism k , the video frame feature vector X of the video to be detected v The query vector Q v , key vector K v , value vector V v And the audio feature vector X of the audio MEL spectrogram a The query vector Q a , key vector K a , value vector V a , specifically:

[0013] Q v =Wq v *X v , K v =Wk v *X v , V v =Wv v *X v ;

[0014] Q a =Wq a *X a , K a =Wk a *X a , V a =Wv a *X a ;

[0015] Among them, Wqv , Wk v , Wv v are respectively the video frame feature vectors X of the video to be detected v The weight matrix for linear transformation, Wq a , Wk a , Wv a is the audio feature vector X of the audio MEL spectrogram a The weight matrix for the transformation;

[0016] S5. Use a multi-head self-attention mechanism to process the video frame feature vector X of the video to be detected respectively. v and the audio feature vector X of the audio MEL spectrogram a , calculate the video frame feature vector X of the video to be detected v Attention weight matrix v and the audio feature vector X of the audio MEL spectrogram a Attention weight matrix a , specifically:

[0017]

[0018] Among them, d k To Q*K T The product result is normalized;

[0019] S6. Introduce the inter-modal interactive attention mechanism to compare the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The similarities between them are used to enhance the interaction between modalities, specifically:

[0020]

[0021] S7. The video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The weighted fusion of the two modal features of is used to obtain the final multimodal fusion feature F, which is specifically:

[0022] F=αF v +(1-α)F a ;

[0023] Here, α is a hyperparameter.

[0024] S8. Input the multimodal fusion feature F into a multi-layer fully connected neural network D consisting of a batch normalization layer, a fully connected layer, an activation function, and an output layer to obtain a multimodal fusion feature F' after dimensionality reduction, specifically:

[0025] F'=W*F+b;

[0026] Where W is the weight, F is the multimodal fusion feature, and b is the bias;

[0027] S9. Input the multimodal fusion feature F' after dimensionality reduction into the classifier to obtain the classification result Obtain the probabilities of the video to be detected being a real category and a forged category respectively, and determine whether the video to be detected is a forged video by comparing the probability values ​​of the real category and the forged category of the video to be detected, specifically:

[0028]

[0029] S10. When the probability that the video to be detected is a forged category is greater than the probability that it is a real category, determine that the video to be detected is a forged video; otherwise, determine that the video to be detected is a forged video.

[0030] Furthermore, the video to be detected is pre-processed as described in S1 to obtain multiple video frames, including:

[0031] S101 uses the VideoCapture function in the computer vision library OpenCV to read the video to be detected and determine the total number of frames of the video to be detected;

[0032] S102. Calculate the interval between video frames to be extracted based on the total number of frames of the video to be detected, and evenly extract multiple video frames from the video to be detected;

[0033] S103. Perform data normalization processing on the multiple video frames to convert them into floating-point data.

[0034] Furthermore, in S2, an audio file in the video to be detected is obtained and the audio file is converted into a MEL spectrogram, including:

[0035] S201. Read the video file to be detected by calling the VideoFileClip function in the video processing library MoviePy, obtain the audio file in the video file to be detected, and save the audio file in WAV format;

[0036] S202. Load the WAV audio file by calling the audio processing library Librosa library and store the audio data in the WAV audio file in the form of a time series;

[0037] S203. By calling the melspectrogram function in the audio processing library Librosa library, the audio data in the WAV audio file is converted into a MEL spectrogram, and the floating-point data of the MEL spectrogram is normalized into integer-valued data;

[0038] S204 . Resize the normalized MEL spectrum graph, and scale the normalized MEL spectrum graph to a specified target size.

[0039] Furthermore, in S3, the video frame feature vector X of the detection video is obtained. v ,include:

[0040] S301. Divide each of the plurality of video frames input into the Video Swin Transforme model into a plurality of small image blocks, and convert the plurality of small image blocks into high-dimensional feature vectors through a linear embedding layer in the Video Swin Transforme model;

[0041] S302. Introduce temporal coding to embed the temporal sequence information between the multiple video frames into the high-dimensional feature vector, and obtain the video frame feature vector X of the detection video after processing by the Video SwinTransforme model. v .

[0042] Furthermore, in S3, the audio feature vector X of the audio MEL spectrogram is obtained. a ,include:

[0043] S303. Input the audio MEL spectrogram into the pre-trained ResNet18 model, and perform deep feature extraction on the audio MEL spectrogram through multiple convolutional layers and residual modules of the ResNet18 model;

[0044] S304. Through the global average pooling layer of the ResNet18 model, the audio MEL spectrogram is compressed into an audio feature vector X a , the audio feature vector X a Contains the time-frequency information of the audio signal.

[0045] Furthermore, the multimodal fusion feature F' obtained after dimensionality reduction in S8 includes:

[0046] S801. Input the multimodal fusion feature F into the batch normalization layer to obtain a multimodal fusion feature F1, ensuring that the multimodal fusion feature F1 has zero mean and unit variance;

[0047] S802. Input the multimodal fusion feature F1 into the fully connected layer, and perform dimensionality reduction processing on the multimodal fusion feature F1 to obtain the multimodal fusion feature F2;

[0048] S803. Input the multimodal fusion feature F2 into the batch normalization layer to obtain a multimodal fusion feature F3;

[0049] S804. The multimodal fusion feature F3 is nonlinearly transformed by the Tanh activation function to obtain a multimodal fusion feature F4, where:

[0050]

[0051] S805. Input the multimodal fusion feature F4 to the output layer, and compress the multimodal fusion feature F4 into a multimodal fusion feature F', wherein the multimodal fusion feature F' is a two-dimensional vector.

[0052] In a second aspect, the present invention provides a deep fake video detection device that integrates multimodal information, which adopts any of the deep fake video detection methods that integrate multimodal information as described in the first aspect, including:

[0053] Data processing module 210: pre-processes the video to be detected, obtains multiple video frames of the video to be detected and an audio file in the video to be detected, and converts the audio file into a MEL spectrogram;

[0054] Feature vector extraction module 220: converting the multiple video frames of the video to be detected and the audio file into a MEL spectrum to obtain the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a ;

[0055] Multimodal attention fusion module 230: according to the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a , the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a Perform multimodal cross-attention feature fusion to obtain multimodal fusion feature F;

[0056] Video forgery detection module 240: Input the multimodal fusion feature F into the multi-layer fully connected neural network D to obtain the multimodal fusion feature F' after dimensionality reduction, and input the multimodal fusion feature F' after dimensionality reduction into the classifier to obtain the classification result Determine whether the video to be detected is a forged video.

[0057] In a third aspect, the present invention further provides an electronic device, comprising: a memory and a processor;

[0058] wherein the memory stores computer-executable instructions;

[0059] The processor executes the computer-executable instructions stored in the memory, so that the processor executes any deep fake video detection method that integrates multimodal information as described in the first aspect of the present invention.

[0060] In a fourth aspect, the present invention also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to enable the processor to execute any deep fake video detection method that integrates multimodal information as described in the first aspect of the invention.

[0061] The beneficial effect of the present invention is that the deep fake video detection method and device that integrates multimodal information proposed in this technical solution can solve the problem of inconsistencies in fake content after compression, transmission or editing by combining the visual information, audio information and cross-modal relationship in the video, thereby improving the accuracy and robustness of fake video detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and it is clear that those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0063] Figure 1 Schematic diagram of the video forgery detection method integrating multimodal information according to the present invention.

[0064] Figure 2 Schematic diagram of a video forgery detection device integrating multimodal information according to the present invention.

[0065] Explanation of reference numerals: data processing module 210 ; feature vector extraction module 220 ; multimodal attention fusion module 230 ; video forgery detection module 240 . DETAILED DESCRIPTION

[0066] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0067] The terms used in the embodiments of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The singular forms "a," "an," "the," and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, and unless the context clearly indicates otherwise, "a plurality" generally includes at least two.

[0068] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0069] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0070] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or device comprising the element.

[0071] The optional embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0072] In the first aspect, as shown in FIG1 , the present invention proposes a method for detecting deep fake videos by integrating multimodal information, comprising:

[0073] S1. The video to be detected is preprocessed to obtain multiple video frames; the video to be detected includes audio, and each video frame includes the audio;

[0074] S2 obtains the audio file in the video to be detected and converts the audio file into a MEL spectrogram;

[0075] S3. For each video frame of the plurality of video frames, extract the feature vector of each video frame, and fuse the feature vectors of each video frame to obtain the video frame feature vector X of the video to be detected. vand the audio feature vector X of the audio MEL spectrogram a ;

[0076] S4. The video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a Perform linear transformations respectively and map them to the query Q, key K and value V spaces to obtain the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The feature dimension d of the attention mechanism k , the video frame feature vector X of the video to be detected v The query vector Q v , key vector K v , value vector V v And the audio feature vector X of the audio MEL spectrogram a The query vector Q a , key vector K a , value vector V a , specifically:

[0077] Q v =Wq v *X v , K v =Wk v *X v , V v =Wv v *X v ;

[0078] Q a =Wq a *X a , K a =Wk a *X a , V a =Wv a *X a ;

[0079] Among them, Wq v , Wk v , Wv v are respectively the video frame feature vectors X of the video to be detected v The weight matrix for linear transformation, Wq a , Wk a , Wv a is the audio feature vector X of the audio MEL spectrogram a The weight matrix for the transformation;

[0080] S5. Use a multi-head self-attention mechanism to process the video frame feature vector X of the video to be detected respectively. v and the audio feature vector X of the audio MEL spectrogram a , calculate the video frame feature vector X of the video to be detected v Attention weight matrix v and the audio feature vector X of the audio MEL spectrogram a Attention weight matrix a , specifically:

[0081]

[0082] Among them, d k To Q*K T The product result is normalized;

[0083] S6. Introduce the inter-modal interactive attention mechanism to compare the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The similarities between them are used to enhance the interaction between modalities, specifically:

[0084]

[0085] S7. The video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The weighted fusion of the two modal features of is used to obtain the final multimodal fusion feature F, which is specifically:

[0086] F=αF v +(1-α)F a ;

[0087] Here, α is a hyperparameter.

[0088] S8. Input the multimodal fusion feature F into a multi-layer fully connected neural network D consisting of a batch normalization layer, a fully connected layer, an activation function, and an output layer to obtain a multimodal fusion feature F' after dimensionality reduction, specifically:

[0089] F'=W*F+b;

[0090] Where W is the weight, F is the multimodal fusion feature, and b is the bias;

[0091] S9. Input the multimodal fusion feature F' after dimensionality reduction into the classifier to obtain the classification result Obtain the probabilities of the video to be detected being a real category and a forged category respectively, and determine whether the video to be detected is a forged video by comparing the probability values ​​of the real category and the forged category of the video to be detected, specifically:

[0092]

[0093] S10. When the probability that the video to be detected is a forged category is greater than the probability that it is a real category, determine that the video to be detected is a forged video; otherwise, determine that the video to be detected is a forged video.

[0094] In S5, the attention weight matrix is ​​generated by calculating the dot product of the query vector and the key vector. The softmax function is built into the attention mechanism formula and its function is to normalize the attention weights and convert them into probability distributions. After obtaining the multi-head attention output, residual connections and layer normalization are used to improve the stability of training and enhance information retention.

[0095] In S6, the inter-modal interactive attention mechanism is introduced to help capture the video frame feature vector X of the video to be detected. v and the audio feature vector X of the audio MEL spectrogram a The potential correlation between the video frame feature vector X of the video to be detected is obtained through the self-attention mechanism and the interactive attention mechanism. v and the audio feature vector X of the audio MEL spectrogram a After processing, new feature representation F is generated. v and F a Through the multimodal self-attention mechanism and inter-modal interactive attention, we can capture potential consistency and anomalies between video frames and audio, such as lip movement and speech synchronization.

[0096] In S7, the multimodal fusion feature F represents the deep correlation between video and audio information, and can provide rich semantic information for deep fake video detection.

[0097] In S8, the multi-layer fully connected neural network D is composed of a batch normalization layer, a fully connected layer, an activation function and an output layer, which can effectively compress the feature space, enhance the nonlinear expression ability of the model, and thus accurately predict whether the video to be detected is forged.

[0098] The present invention adopts a multi-head self-attention mechanism to process the video frame feature vector X of the video to be detected respectively v and the audio feature vector X of the audio MEL spectrogram a , and then the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrograma The weighted fusion of the two modal features improves the accuracy and robustness of deepfake video detection. The fusion of multimodal features can fully utilize the complementarity of different modal information to ensure that more comprehensive features are captured, thereby improving the accuracy and robustness of deepfake video detection.

[0099] The present invention requires the video to be detected to be:

[0100] 1) The frame rate and resolution of the video to be detected can be kept clear and stable. If the video to be detected includes a face, the audio in the video to be detected is the voice of the person corresponding to the face;

[0101] 2) The video to be detected includes audio, and the audio portion requires time synchronization between the audio and video content.

[0102] It should be noted that the deep fake video detection method that integrates multimodal information provided by the present invention is suitable for detecting whether a video has been forged, including tampering with the clips. The executor of this method can be a forged video detection device, such as an electronic device, or a control module in the forged video detection device for executing the deep fake video detection method that integrates multimodal information.

[0103] In addition, in terms of video processing, the present invention uses the Video SwinTransformer model to extract visual spatiotemporal features, paying special attention to the detailed information and dynamic changes of video frames, so as to better capture the subtle anomalies generated during the deep fake process. In audio processing, the present invention uses the ResNet18 pre-trained model to process the MEL spectrogram to capture subtle changes during audio forgery. In contrast, although the paper "JointAudio-Visual Deepfake Detection" also uses video frame information, its processing method focuses more on local inter-frame correlations, while the present invention focuses on global dynamics and complex spatiotemporal change patterns. The paper "JointAudio-Visual Deepfake Detection" tends to rely on simple pattern matching of voice content in audio detection, and does not deeply analyze the frequency and time domain characteristics of the audio signal.

[0104] This paper also employs a multimodal feature fusion method based on an attention mechanism, leveraging self-attention and inter-modal interactive attention to establish more detailed associations between visual and audio features, ensuring more accurate feature alignment and fusion. The fusion process in the paper "JointAudio-Visual Deepfake Detection" is relatively simple and fails to fully capture the complex dynamic relationship between audio and video, resulting in reduced detection accuracy when facing advanced forgery techniques.

[0105] Further, the pre-processing of the to-be-detected video in S1 to obtain a plurality of video frames includes:

[0106] S101. Read the to-be-detected video using the VideoCapture function in the computer vision library OpenCV to determine the total number of frames of the to-be-detected video;

[0107] S102. According to the total number of frames of the to-be-detected video, calculate the interval of video frames to be extracted, and uniformly extract a plurality of video frames from the to-be-detected video;

[0108] S103. Data normalization processing is performed on the plurality of video frames to become floating point data.

[0109] In step S101, the reading of the to-be-detected video is performed through the OpenCV library, so that the total number of frames of the to-be-detected video can be conveniently obtained.

[0110] In step S102, the video frames are uniformly extracted from the to-be-detected video, so as to ensure that the plurality of video frames extracted from the to-be-detected video are key video frames and are representative.

[0111] In step S103, the plurality of video frames are pre-processed to ensure that the video frame features are suitable for input into the deep learning model to be compatible with the deep learning model. The pre-processing process of the plurality of video frames is as follows: first, the size of each video frame of the plurality of video frames is adjusted; then the pixel value range of the plurality of video frames is adjusted, and the pixel value is normalized from the integer range of 0-255 to the floating point number between 0-1. The processed plurality of video frames are converted to torch.tensor data type, and the data type is specified as floating point number to be compatible with the deep learning model.

[0112] Further, in S2, the audio file in the to-be-detected video is obtained, and the audio file is converted into a MEL spectrum, including:

[0113] S201. Read the to-be-detected video file by calling the VideoFileClip function in the video processing library MoviePy, obtain the audio file in the to-be-detected video file, and save the audio file in WAV format;

[0114] S202. Load the WAV audio file by calling the audio processing library Librosa library to store the audio data in the WAV audio file in the form of time series;

[0115] S203. By calling the melspectrogram function in the audio processing library Librosa library, the audio data in the WAV audio file is converted into a MEL spectrogram, and the floating-point data of the MEL spectrogram is normalized into integer-valued data;

[0116] S204 . Resize the normalized MEL spectrum graph, and scale the normalized MEL spectrum graph to a specified target size.

[0117] In step S204 , the scaled MEL spectrum graph may be used for subsequent model input or other processing steps to adapt to different input size requirements, and the processed MEL spectrum graph is finally returned.

[0118] Furthermore, in S3, the video frame feature vector X of the detection video is obtained. v ,include:

[0119] S301. Divide each of the plurality of video frames input into the Video Swin Transforme model into a plurality of small image blocks, and convert the plurality of small image blocks into high-dimensional feature vectors through a linear embedding layer in the Video Swin Transforme model;

[0120] S302. Introduce temporal coding to embed the temporal sequence information between the multiple video frames into the high-dimensional feature vector, and obtain the video frame feature vector X of the detection video after processing by the Video SwinTransforme model. v .

[0121] In order to efficiently extract the spatiotemporal features of video frames, the present invention adopts the Video Swin Transforme pre-trained model, which is a self-attention mechanism model based on the Swin Transformer architecture. It is particularly suitable for processing complex spatiotemporal information in videos. Its multi-scale convolution method can capture the global features in the video frame while maintaining the local features of the video frame, thereby effectively extracting dynamically changing details.

[0122] In S301, the Video SwinTransformer model captures the temporal and spatial correlations between the multiple video frames through its multi-layer self-attention mechanism. The Video SwinTransformer model can extract local features from the video frames while establishing long-range dependencies between frames, thereby fully utilizing the spatiotemporal information of the video. Leveraging the SwinTransformer self-attention mechanism, the Video SwinTransformer model can gradually fuse local and global spatiotemporal information to extract deep semantic features.

[0123] In S302, the temporal encoding is the position encoding in the Transformer model. The purpose is to embed the position information into the feature, so that the Transformer retains the position information of the feature, which can improve the model's ability to understand the sequence. After being processed by the Video SwinTransforme model, the video frame feature vector obtained by fusion of the feature vectors of the multiple video frames is obtained, that is, the video frame feature vector X of the detection video. v , for alignment with the audio features.

[0124] Furthermore, in S3, the audio feature vector X of the audio MEL spectrogram is obtained. a ,include:

[0125] S303. Input the audio MEL spectrogram into the pre-trained ResNet18 model, and perform deep feature extraction on the audio MEL spectrogram through multiple convolutional layers and residual modules of the ResNet18 model;

[0126] S304. Through the global average pooling layer of the ResNet18 model, the audio MEL spectrogram is compressed into an audio feature vector X a , the audio feature vector X a Contains the time-frequency information of the audio signal.

[0127] In S303, the ResNet18 model, through hierarchical convolution and residual connections, effectively extracts multi-scale features from the audio spectrum and identifies key patterns and frequency variations in the audio signal. The initial convolutional layers extract the basic features of the audio spectrum. Through residual connections, the ResNet18 model can more effectively capture complex patterns in audio, avoiding the vanishing gradient problem in deep networks.

[0128] There is only one audio feature of the video to be detected. In S304, the ResNet18 model extracts the features of the audio MEL spectrogram into a vector of fixed length, which represents the time-frequency pattern and feature distribution in the audio as the output of the audio feature.

[0129] Furthermore, the multimodal fusion feature F' obtained after dimensionality reduction in S8 includes:

[0130] S801. Input the multimodal fusion feature F into the batch normalization layer to obtain a multimodal fusion feature F1, ensuring that the multimodal fusion feature F1 has zero mean and unit variance;

[0131] S802. Input the multimodal fusion feature F1 into the fully connected layer, and perform dimensionality reduction processing on the multimodal fusion feature F1 to obtain the multimodal fusion feature F2;

[0132] S803. Input the multimodal fusion feature F2 into the batch normalization layer to obtain a multimodal fusion feature F3;

[0133] S804. The multimodal fusion feature F3 is nonlinearly transformed by the Tanh activation function to obtain a multimodal fusion feature F4, where:

[0134]

[0135] S805. Input the multimodal fusion feature F4 to the output layer, and compress the multimodal fusion feature F4 into a multimodal fusion feature F', wherein the multimodal fusion feature F' is a two-dimensional vector.

[0136] The dimensionality reduction processing of the multimodal fusion feature F1 in S802 is for extracting more representative multimodal fusion features, while reducing the feature dimension and lowering the computational complexity.

[0137] The multimodal fusion feature F' in S805 is used to evaluate the video to be detected and perform a binary classification task.

[0138] In a second aspect, as shown in FIG2 , the present invention proposes a deep fake video detection device that integrates multimodal information, which adopts any of the deep fake video detection methods that integrate multimodal information as described in the first aspect, including:

[0139] Data processing module 210: pre-processes the video to be detected, obtains multiple video frames of the video to be detected and an audio file in the video to be detected, and converts the audio file into a MEL spectrogram;

[0140] Feature vector extraction module 220: converting the multiple video frames of the video to be detected and the audio file into a MEL spectrum to obtain the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a ;

[0141] Multimodal attention fusion module 230: according to the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a , the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a Perform multimodal cross-attention feature fusion to obtain multimodal fusion feature F;

[0142] Video forgery detection module 240: Input the multimodal fusion feature F into the multi-layer fully connected neural network D to obtain the multimodal fusion feature F' after dimensionality reduction, and input the multimodal fusion feature F' after dimensionality reduction into the classifier to obtain the classification result Determine whether the video to be detected is a forged video.

[0143] In a third aspect, the present invention further provides an electronic device, comprising: a memory and a processor;

[0144] wherein the memory stores computer-executable instructions;

[0145] The processor executes the computer-executable instructions stored in the memory, so that the processor executes any deep fake video detection method that integrates multimodal information as described in the first aspect of the present invention.

[0146] In a fourth aspect, the present invention also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to enable the processor to execute any deep fake video detection method that integrates multimodal information as described in the first aspect of the invention.

[0147] Specifically, a system or device equipped with a storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program codes stored in the storage medium.

[0148] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute part of the present invention.

[0149] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RYMs, DVD-RWs, DVD+RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer via a communications network.

[0150] In addition, it should be clear that the functions of any of the above embodiments can be achieved not only by executing the program code read by the computer, but also by enabling the operating system operating on the computer to complete part or all of the actual operations based on the instructions of the program code.

[0151] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU installed on the expansion board or expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep fake video detection method integrating multimodal information, characterized in that: include: S1. The video to be detected is preprocessed to obtain multiple video frames; the video to be detected includes audio, and each video frame includes the audio; S2 obtains the audio file in the video to be detected and converts the audio file into a MEL spectrogram; S3. For each video frame of the plurality of video frames, extract the feature vector of each video frame, and fuse the feature vectors of each video frame to obtain the video frame feature vector X of the video to be detected. v and the audio feature vector X of the audio MEL spectrogram a ; S4. The video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a Perform linear transformations respectively and map them to the query Q, key K and value V spaces to obtain the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The feature dimension d of the attention mechanism k , the video frame feature vector X of the video to be detected v The query vector Q v , key vector K v , value vector V v And the audio feature vector X of the audio MEL spectrogram a The query vector Q a , key vector K a , value vector V a , specifically: Q v =Wq v *X v ;K v =Wk v *X v ;V v =Wv v *X v Q a =Wq a *X a ;K a =Wk a *X a ;V a =Wv a *X a Among them, Wq v , Wk v , Wv v are respectively the video frame feature vectors X of the video to be detected v The weight matrix for linear transformation, Wq a , Wk a , Wv a is the audio feature vector X of the audio MEL spectrogram a The weight matrix for the transformation; S5. Use a multi-head self-attention mechanism to process the video frame feature vector X of the video to be detected respectively. v and the audio feature vector X of the audio MEL spectrogram a , calculate the video frame feature vector X of the video to be detected v Attention weight matrix v and the audio feature vector X of the audio MEL spectrogram a Attention weight matrix a , specifically: Among them, d k To Q*K T The product result is normalized; S6. Introduce the inter-modal interactive attention mechanism to compare the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The similarities between them are used to enhance the interaction between modalities, specifically: S7. The video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a The weighted fusion of the two modal features of is used to obtain the final multimodal fusion feature F, which is specifically: F=αF v +(1-α)F a Among them, α is a hyperparameter; S8. Input the multimodal fusion feature F into a multi-layer fully connected neural network D consisting of a batch normalization layer, a fully connected layer, an activation function, and an output layer to obtain a multimodal fusion feature F' after dimensionality reduction, specifically: F'=W*F+b Where W is the weight, F is the multimodal fusion feature, and b is the bias; S9. Input the multimodal fusion feature F' after dimensionality reduction into the classifier to obtain the classification result Obtain the probabilities of the video to be detected being a real category and a forged category respectively, and determine whether the video to be detected is a forged video by comparing the probability values ​​of the real category and the forged category of the video to be detected, specifically: S10. When the probability that the video to be detected is a forged category is greater than the probability that it is a real category, determine that the video to be detected is a forged video; otherwise, determine that the video to be detected is a forged video.

2. A deep fake video detection method integrating multimodal information according to claim 1, characterized in that: As described in S1, the video to be detected is preprocessed to obtain multiple video frames, including: S101 uses the VideoCapture function in the computer vision library OpenCV to read the video to be detected and determine the total number of frames of the video to be detected; S102. Calculate the interval between video frames to be extracted based on the total number of frames of the video to be detected, and evenly extract multiple video frames from the video to be detected; S103. Perform data normalization processing on the multiple video frames to convert them into floating-point data.

3. The method for detecting deep fake videos by integrating multimodal information according to claim 1, wherein: S2 obtains the audio file in the video to be detected and converts the audio file into a MEL spectrogram, including: S201. Read the video file to be detected by calling the VideoFileClip function in the video processing library MoviePy, obtain the audio file in the video file to be detected, and save the audio file in WAV format; S202. Load the WAV audio file by calling the audio processing library Librosa library and store the audio data in the WAV audio file in the form of a time series; S203. By calling the melspectrogram function in the audio processing library Librosa library, the audio data in the WAV audio file is converted into a MEL spectrogram, and the floating-point data of the MEL spectrogram is normalized into integer-valued data; S204 . Resize the normalized MEL spectrum graph, and scale the normalized MEL spectrum graph to a specified target size.

4. The method for detecting deep fake videos by integrating multimodal information according to claim 1, wherein: S3 obtains the video frame feature vector X of the detection video v ,include: S301. Divide each of the plurality of video frames input into the Video Swin Transforme model into a plurality of small image blocks, and convert the plurality of small image blocks into high-dimensional feature vectors through a linear embedding layer in the Video Swin Transforme model; S302. Introduce temporal coding to embed the temporal sequence information between the multiple video frames into the high-dimensional feature vector, and obtain the video frame feature vector X of the detection video after processing by the Video Swin Transforme model. v .

5. The method for detecting deep fake videos by integrating multimodal information according to claim 1, wherein: In S3, the audio feature vector X of the audio MEL spectrogram is obtained a ,include: S303. Input the audio MEL spectrogram into the pre-trained ResNet18 model, and perform deep feature extraction on the audio MEL spectrogram through multiple convolutional layers and residual modules of the ResNet18 model; S304. Through the global average pooling layer of the ResNet18 model, the audio MEL spectrogram is compressed into an audio feature vector X a , the audio feature vector X a Contains the time-frequency information of the audio signal.

6. The method for detecting deep fake videos by integrating multimodal information according to claim 1, wherein: The multimodal fusion feature F' obtained after dimensionality reduction in S8 includes: S801. Input the multimodal fusion feature F into the batch normalization layer to obtain a multimodal fusion feature F1, ensuring that the multimodal fusion feature F1 has zero mean and unit variance; S802. Input the multimodal fusion feature F1 into the fully connected layer, and perform dimensionality reduction processing on the multimodal fusion feature F1 to obtain the multimodal fusion feature F2; S803. Input the multimodal fusion feature F2 into the batch normalization layer to obtain a multimodal fusion feature F3; S804. The multimodal fusion feature F3 is nonlinearly transformed by the Tanh activation function to obtain a multimodal fusion feature F4, where: S805. Input the multimodal fusion feature F4 to the output layer, and compress the multimodal fusion feature F4 into a multimodal fusion feature F', wherein the multimodal fusion feature F' is a two-dimensional vector.

7. A deep fake video detection device that integrates multimodal information, employing the deep fake video detection method that integrates multimodal information as claimed in any one of claims 1 to 6, comprising: Data processing module 210: pre-processes the video to be detected, obtains multiple video frames of the video to be detected and an audio file in the video to be detected, and converts the audio file into a MEL spectrogram; Feature vector extraction module 220: converting the multiple video frames of the video to be detected and the audio file into a MEL spectrum to obtain the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a ; Multimodal attention fusion module 230: according to the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a , the video frame feature vector X of the video to be detected v and the audio feature vector X of the audio MEL spectrogram a Perform multimodal cross-attention feature fusion to obtain multimodal fusion feature F; Video forgery detection module 240: Input the multimodal fusion feature F into the multi-layer fully connected neural network D to obtain the multimodal fusion feature F' after dimensionality reduction, and input the multimodal fusion feature F' after dimensionality reduction into the classifier to obtain the classification result Determine whether the video to be detected is a forged video.

8. An electronic device, characterized in that: include: memory and processor; wherein the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the deep fake video detection method for fusing multimodal information as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that A plurality of instructions are stored therein, and the instructions are loaded by a processor, so that the processor executes the deep fake video detection method for fusing multimodal information as claimed in any one of claims 1 to 6.

Images