Method, device and computer equipment for multi-modal detection of deepfake videos

Abstract

The application relates to a multi-modal deep fake video detection method, device and computer equipment. The method comprises the following steps: extracting a visual mode and an auditory mode of an arbitrary type of video, and converting the auditory mode into a corresponding mel spectrum; performing feature extraction on a video block obtained by segmenting the visual mode and an audio group obtained by segmenting the mel spectrum, respectively, to obtain video features and audio features; inputting the video features and the audio features into a pre-constructed synchronization model to obtain a bidirectional delay distribution of audio and video; forming an initial distribution of an original video by collecting the bidirectional delay distribution, inputting the initial distribution into a pre-constructed short-time dependence estimation model to estimate a subsequent video distribution, and performing back propagation training on the short-time dependence estimation model by calculating a loss to obtain a trained short-time dependence estimation model for fake detection of the visual mode and the auditory mode. The method can consider fake detection of both visual and auditory modes, and improve fake detection performance.

Description

Technical Field

[0001] This application relates to the field of deepfake video detection technology, and in particular to a multimodal detection method, apparatus and computer equipment for deepfake videos. Background Technology

[0002] With the rapid evolution of generative artificial intelligence technology, the creation of multimedia content for video and audio has entered an unprecedented stage of rapid development. However, this technological leap has also been accompanied by the rise of deepfake technology. Deepfake-generated fake multimedia content is highly deceptive and misleading. Therefore, strengthening research and exploration of deepfake detection mechanisms to address diverse forgery methods and the wide range of multimedia content has become a critical issue that urgently needs to be solved.

[0003] Currently, multimodal deepfake detection technology is a research hotspot, achieving significant progress in identifying forged content by integrating visual and auditory analysis methods. However, these technologies often face the problem of overfitting to specific forgery types and datasets, limiting their generalization ability on unknown videos. To address this challenge, researchers have turned their attention to mining the inherent features common to deepfake technologies that can be generalized across domains; these features leave unique traces in tampered videos. However, such multimodal deepfake detection methods focus more on visual modality detection, often neglecting auditory modality forgery detection, resulting in poor performance in audio forgery scenarios. Furthermore, autoregressive models based on Transformer-decoder utilize information from a long future time frame when generating the probability distribution of the next frame. Since videos only exhibit stronger correlations in the short time dimension, they show significant inconsistencies and differences in the long time dimension, thus causing poor regression prediction results.

[0004] Therefore, how to simultaneously detect bidirectional inconsistencies between visual and auditory modes, take into account both visual and auditory modalities in forgery detection, and how to utilize the short-term temporal dependency modeling of synchronous probability distributions to enhance the representation learning ability of synchronous probability distributions are very important issues in the field of deep forgery detection. Summary of the Invention

[0005] Therefore, it is necessary to provide a multimodal detection method, apparatus, and computer equipment for deepfake videos to address the aforementioned technical problems.

[0006] A multimodal detection method for deepfake videos, the method comprising:

[0007] Extract the visual and auditory modalities of any type of video, and convert the auditory modalities into the corresponding Mel spectrum;

[0008] The video blocks obtained by segmenting the visual modality are input into the video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectrum are input into the audio encoder for feature extraction to obtain audio features.

[0009] Video and audio features are input into a pre-built synchronization model. The synchronization model is used to perform bidirectional synchronization distribution modeling and calculation of audio and video, and the output is the bidirectional delay distribution of audio to video and video to audio.

[0010] The initial distribution of the original video is formed by a bidirectional delay distribution. The initial distribution is then input into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video. Based on the loss between the estimated subsequent distribution and the initial distribution, the short-term dependency estimation model is trained by backpropagation to obtain a trained short-term dependency estimation model. Based on this trained short-term dependency estimation model, visual and auditory modal forgery detection is performed on the video to be detected.

[0011] In one embodiment, the visual and auditory modalities of any type of video are extracted, and the auditory modalities are converted into corresponding Mel spectra, including:

[0012] Get any type of video in T, H, and W represent the video's duration, height, and width, respectively, while C represents the number of channels per frame. Represent the space of real numbers;

[0013] Extract video The visual modality V and the auditory modality A are represented as follows:

[0014] V = {f1, f2, f3, ..., f n};

[0015] A = {a1, a2, a3, ..., a} n};

[0016] Among them, f n Let a represent the nth frame in visual modality V. n This represents the audio corresponding to each video frame, where n is the total number of frames in the visual modality V.

[0017] Convert auditory mode A into the corresponding Mel spectrum. Represented as

[0018]

[0019] Mel_spectrogram represents Mel spectrum processing.

[0020] In one embodiment, video blocks obtained from segmenting visual modalities are input into a video encoder for feature extraction to obtain video features, including:

[0021] The visual modality V is segmented into video blocks, each containing five adjacent frames, specifically represented as follows:

[0022]

[0023] in, Let represent the i-th video block, and fi represent the i-th frame in visual modality V;

[0024] Each video block is input into the VisualEncoder video encoder for feature extraction, resulting in video features, represented as...

[0025]

[0026] in, This represents the video feature corresponding to the i-th video block.

[0027] In one embodiment, the audio groups obtained by segmenting the Mel spectrum are input into an audio encoder for feature extraction to obtain audio features, including:

[0028] Mel spectrum After being divided into audio segments, five adjacent audio segments are bundled together to form an audio group, specifically represented as follows:

[0029]

[0030] in, This represents the i-th audio group. Represents the Mel spectrum The i-th audio segment obtained after segmentation;

[0031] Each audio group is input into the audio encoder (AudioEncoder) for feature extraction, yielding audio features, represented as follows:

[0032]

[0033] in, This represents the audio feature corresponding to the i-th audio group.

[0034] In one embodiment, video features and audio features are input into a pre-built synchronization model. The synchronization model is used to perform bidirectional audio-video synchronization distribution modeling and calculation, and the output is the bidirectional delay distribution of audio to video and video to audio, including:

[0035] The video features corresponding to each video block and the audio features corresponding to each audio group are input into a pre-built synchronization model. The synchronization model is then used to perform bidirectional audio-video synchronization distribution modeling and calculation, outputting the bidirectional delay distribution between audio and video, and between video and audio, represented as follows:

[0036]

[0037] in, This represents the delay distribution of the audio pairs corresponding to the i-th video block. MP_AviT represents the latency distribution of the audio pairs corresponding to the i-th audio group in the video; MP_AviT represents the synchronization model. This represents the video feature corresponding to the i-th video block. This represents the audio feature corresponding to the i-th audio group.

[0038] In one embodiment, the set of bidirectional delay distributions forms an initial distribution of the original video. This initial distribution is then input into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video, including:

[0039] The initial distribution Ψ(i) of the original video is obtained by combining the delay distributions of the audio pairs corresponding to all video blocks and the delay distributions of the audio pairs corresponding to all audio groups.

[0040]

[0041] in, This represents the combination of the video-to-audio delay distribution corresponding to the i-th video block and the audio-to-video delay distribution corresponding to the i-th audio group, where N represents the number of video blocks and audio groups.

[0042] The initial distribution Ψ(i) is input into a pre-built short-term dependency estimation model. Based on the short-term dependency estimation model, two convolutional encoding operations and two deconvolutional decoding operations are performed sequentially to estimate the subsequent distribution of the original video within a short time. Represented as

[0043]

[0044] Where Conv represents the convolutional encoding operation and Deconv represents the deconvolutional decoding operation.

[0045] In one embodiment, the short-term dependency estimation model is backpropagated and trained based on the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-term dependency estimation model, including:

[0046] Calculate the subsequent distribution of the original video using KL divergence. The loss between the initial distribution Ψ(i) and the initial distribution Loss is expressed as:

[0047]

[0048] Where KLDivLoss represents the loss function for calculating the KL divergence;

[0049] The short-term dependency estimation model is trained by backpropagation based on the loss, resulting in a well-trained short-term dependency estimation model.

[0050] A multimodal detection device for deepfake videos, the device comprising:

[0051] The audio-video bidirectional synchronization distribution modeling module is used to extract the visual and auditory modalities of arbitrary types of videos and convert the auditory modalities into corresponding Mel spectra. The video blocks obtained by segmenting the visual modalities are input into the video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectra are input into the audio encoder for feature extraction to obtain audio features. The video features and audio features are input into a pre-built synchronization model, and the bidirectional audio-video synchronization distribution modeling calculation is performed through the synchronization model to output the bidirectional delay distribution of audio to video and video to audio.

[0052] The short-term dependency estimation modeling module is used to form the initial distribution of the original video by aggregating the bidirectional delay distribution. The initial distribution is input into the pre-built short-term dependency estimation model to estimate the subsequent distribution of the video. Based on the loss between the estimated subsequent distribution and the initial distribution, the short-term dependency estimation model is backpropagated to train it, resulting in a trained short-term dependency estimation model. Based on the trained short-term dependency estimation model, visual modality and auditory modality forgery detection is performed on the video to be detected.

[0053] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program performing the following steps:

[0054] Extract the visual and auditory modalities of any type of video, and convert the auditory modalities into the corresponding Mel spectrum;

[0055] The video blocks obtained by segmenting the visual modality are input into the video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectrum are input into the audio encoder for feature extraction to obtain audio features.

[0056] Video and audio features are input into a pre-built synchronization model. The synchronization model is used to perform bidirectional synchronization distribution modeling and calculation of audio and video, and the output is the bidirectional delay distribution of audio to video and video to audio.

[0057] The initial distribution of the original video is formed by a bidirectional delay distribution. The initial distribution is then input into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video. Based on the loss between the estimated subsequent distribution and the initial distribution, the short-term dependency estimation model is trained by backpropagation to obtain a trained short-term dependency estimation model. Based on this trained short-term dependency estimation model, visual and auditory modal forgery detection is performed on the video to be detected.

[0058] The aforementioned multimodal detection method, apparatus, and computer equipment for deepfake videos have the following technical advantages compared to existing technologies:

[0059] 1. This application performs bidirectional audio-video synchronization distribution modeling and calculation on video features and audio features extracted from the video. It can determine the synchronization between video frames and audio frames based on the bidirectional audio-video delay distribution of the calculated output. It also considers two video forgery scenarios: video as the main focus and audio as the secondary focus, and audio as the main focus and video as the secondary focus. This overcomes the poor forgery detection performance caused by the imbalance of audio and video data and improves the forgery detection performance in audio forgery scenarios.

[0060] 2. This application inputs the initial distribution of the original video formed by the bidirectional delay distribution of the audio and video sets into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video in the short term. This effectively utilizes the short-term correlation of the video, improves the correlation of the video in the short-term dimension and the accuracy of the evaluation process, avoids the inconsistency and difference problems caused by long-term information dependence, significantly enhances the information utilization rate in the forgery detection process, and improves the performance of forgery video detection. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating a multimodal detection method for deepfake videos in one embodiment;

[0062] Figure 2 This is a flowchart illustrating the calculation of bidirectional audio and video delay distribution in one embodiment;

[0063] Figure 3 This is a schematic diagram illustrating the workflow of a short-time dependency estimation model in one embodiment;

[0064] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] In one embodiment, such as Figure 1 As shown, a multimodal detection method for deepfake videos is provided, including the following steps:

[0067] Step S1: Extract the visual and auditory modalities of any type of video, and convert the auditory modalities into the corresponding Mel spectra.

[0068] Step S2: Input the video blocks obtained from the segmentation of the visual modality into the video encoder for feature extraction to obtain video features; input the audio groups obtained from the segmentation of the Mel spectrum into the audio encoder for feature extraction to obtain audio features.

[0069] Step S3: Input the video features and audio features into the pre-built synchronization model, perform bidirectional synchronization distribution modeling and calculation of audio and video through the synchronization model, and output the bidirectional delay distribution of audio to video and video to audio.

[0070] Step S4: The bidirectional delay distribution is used to form the initial distribution of the original video. The initial distribution is input into the pre-built short-term dependency estimation model to estimate the subsequent distribution of the video. Based on the loss between the estimated subsequent distribution and the initial distribution, the short-term dependency estimation model is trained by backpropagation to obtain the trained short-term dependency estimation model. Based on the trained short-term dependency estimation model, visual modality and auditory modality forgery detection is performed on the video to be detected.

[0071] In one embodiment, the visual and auditory modalities of any type of video are extracted, and the auditory modalities are converted into corresponding Mel spectra, including:

[0072] Get any type of video in T, H, and W represent the video's duration, height, and width, respectively, while C represents the number of channels per frame. Represent the space of real numbers;

[0073] Extract video The visual modality V and the auditory modality A are represented as follows:

[0074] V = {f1, f2, f3, ..., fn} } ;

[0075] A = {a1, a2, a3, ..., a} n};

[0076] Among them, f n Let a represent the nth frame in visual modality V. n This represents the audio corresponding to each video frame, where n is the total number of frames in the visual modality V.

[0077] Convert auditory mode A into the corresponding Mel spectrum. Represented as

[0078]

[0079] Mel_spectrogram represents Mel spectrum processing.

[0080] In one embodiment, video blocks obtained from segmenting visual modalities are input into a video encoder for feature extraction to obtain video features, including:

[0081] The visual modality V is segmented into video blocks, each containing five adjacent frames, specifically represented as follows:

[0082]

[0083] in, f represents the i-th video block. i This represents the i-th frame in visual modality V;

[0084] Each video block is input into the VisualEncoder video encoder for feature extraction, resulting in video features, represented as...

[0085]

[0086] in, This represents the video feature corresponding to the i-th video block.

[0087] In one embodiment, the audio groups obtained by segmenting the Mel spectrum are input into an audio encoder for feature extraction to obtain audio features, including:

[0088] Mel spectrum After being divided into audio segments, five adjacent audio segments are bundled together to form an audio group, specifically represented as follows:

[0089]

[0090] in, This represents the i-th audio group. Represents the Mel spectrum The i-th audio segment obtained after segmentation;

[0091] Each audio group is input into the audio encoder (AudioEncoder) for feature extraction, yielding audio features, represented as follows:

[0092]

[0093] in, This represents the audio feature corresponding to the i-th audio group.

[0094] Furthermore, the video encoder used in this application differs from the audio encoder. The audio encoder incorporates the theoretical framework of VGG-M (Convolutional Neural Network-M), while the video encoder integrates the principles of ResNet (Residual Network).

[0095] In one embodiment, video features and audio features are input into a pre-built synchronization model. The synchronization model is used to perform bidirectional audio-video synchronization distribution modeling and calculation, and the output is the bidirectional delay distribution of audio to video and video to audio, including:

[0096] like Figure 2 As shown, the video features corresponding to each video block and the audio features corresponding to each audio group are input into a pre-built synchronization model. The synchronization model is used to perform bidirectional audio-video synchronization distribution modeling and calculation, outputting the bidirectional delay distribution between audio and video and between video and audio, represented as follows:

[0097]

[0098] in, This represents the delay distribution of the video to audio pair corresponding to the i-th video block, describing the alignment between the video and audio. This represents the delay distribution of the audio pairs corresponding to the i-th audio group, describing the audio-to-video alignment; MP_AviT represents the synchronization model. This represents the video feature corresponding to the i-th video block. This represents the audio feature corresponding to the i-th audio group.

[0099] In one embodiment, the set of bidirectional delay distributions forms an initial distribution of the original video. This initial distribution is then input into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video, including:

[0100] The initial distribution Ψ(i) of the original video is obtained by combining the delay distributions of the audio pairs corresponding to all video blocks and the delay distributions of the audio pairs corresponding to all audio groups.

[0101]

[0102] in, This represents the combination of the video-to-audio delay distribution corresponding to the i-th video block and the audio-to-video delay distribution corresponding to the i-th audio group, where N represents the number of video blocks and audio groups.

[0103] The initial distribution Ψ(i) is input into a pre-built short-term dependency estimation model. Based on the short-term dependency estimation model, two convolutional encoding operations and two deconvolutional decoding operations are performed sequentially to estimate the subsequent distribution of the original video within a short time. Represented as

[0104]

[0105] Where Conv represents the convolutional encoding operation and Deconv represents the deconvolutional decoding operation.

[0106] The workflow of the short-time dependency estimation model is as follows: Figure 3 As shown, this model utilizes the inherent short-term dependency property of the distribution to facilitate the evaluation of subsequent distributions.

[0107] In one embodiment, the short-term dependency estimation model is backpropagated and trained based on the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-term dependency estimation model, including:

[0108] The loss between the subsequent distribution e(i+1) and the initial distribution Ψ(i) of the original video is calculated using KL (relative entropy) divergence, and is expressed as:

[0109] Loss=KLDivLoss(e(i+1),Ψ(i));

[0110] Where KLDivLoss represents the loss function for calculating the KL divergence;

[0111] The short-term dependency estimation model is trained by backpropagation based on the loss, resulting in a well-trained short-term dependency estimation model.

[0112] In one embodiment, a multimodal detection device for deepfake videos is provided, comprising:

[0113] The audio-video bidirectional synchronization distribution modeling module is used to extract the visual and auditory modalities of arbitrary types of videos and convert the auditory modalities into corresponding Mel spectra. The video blocks obtained by segmenting the visual modalities are input into the video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectra are input into the audio encoder for feature extraction to obtain audio features. The video features and audio features are input into a pre-built synchronization model, and the bidirectional audio-video synchronization distribution modeling calculation is performed through the synchronization model to output the bidirectional delay distribution of audio to video and video to audio.

[0114] The short-term dependency estimation modeling module is used to form the initial distribution of the original video by aggregating the bidirectional delay distribution. The initial distribution is input into the pre-built short-term dependency estimation model to estimate the subsequent distribution of the video. Based on the loss between the estimated subsequent distribution and the initial distribution, the short-term dependency estimation model is backpropagated to train it, resulting in a trained short-term dependency estimation model. Based on the trained short-term dependency estimation model, visual modality and auditory modality forgery detection is performed on the video to be detected.

[0115] Specific limitations regarding the multimodal detection device for deepfake videos can be found in the limitations of the multimodal detection method for deepfake videos described above, and will not be repeated here. Each module in the aforementioned multimodal detection device for deepfake videos can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0116] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal detection method for deepfake videos. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0117] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the following steps:

[0119] Extract the visual and auditory modalities of any type of video, and convert the auditory modalities into the corresponding Mel spectrum;

[0120] The video blocks obtained by segmenting the visual modality are input into the video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectrum are input into the audio encoder for feature extraction to obtain audio features.

[0121] Video and audio features are input into a pre-built synchronization model. The synchronization model is used to perform bidirectional synchronization distribution modeling and calculation of audio and video, and the output is the bidirectional delay distribution of audio to video and video to audio.

[0122] The initial distribution of the original video is formed by a bidirectional delay distribution. The initial distribution is then input into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video. Based on the loss between the estimated subsequent distribution and the initial distribution, the short-term dependency estimation model is trained by backpropagation to obtain a trained short-term dependency estimation model. Based on this trained short-term dependency estimation model, visual and auditory modal forgery detection is performed on the video to be detected.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-modal method for deepfake video detection, the method comprising: The method comprises: extracting the visual modality and the auditory modality of any type of video, and converting the auditory modality into corresponding mel spectrum; inputting video blocks segmented from the visual modality into a video encoder for feature extraction to obtain video features, and inputting audio groups segmented from the mel spectrum into an audio encoder for feature extraction to obtain audio features inputting the video features and the audio features into a pre-constructed synchronization model, performing audio-video bidirectional synchronization distribution modeling calculation through the synchronization model, and outputting bidirectional delay distribution of audio to video and video to audio; forming an initial distribution of the original video by collecting the bidirectional delay distribution, inputting the initial distribution into a pre-constructed short-term dependency estimation model for video subsequent distribution estimation, and performing back propagation training on the short-term dependency estimation model according to the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-term dependency estimation model, and performing visual modality and auditory modality forgery detection on a to-be-detected video according to the trained short-term dependency estimation model; wherein inputting the video features and the audio features into a pre-constructed synchronization model, performing audio-video bidirectional synchronization distribution modeling calculation through the synchronization model, and outputting bidirectional delay distribution of audio to video and video to audio, comprises: inputting the video features corresponding to each video block and the audio features corresponding to each audio group into a pre-constructed synchronization model, performing audio-video bidirectional synchronization distribution modeling calculation through the synchronization model, and outputting bidirectional delay distribution of audio to video and video to audio, represented as ； in, Indicates the first i The latency distribution of the video pairs corresponding to each video block. Indicates the first i The delay distribution of audio pairs corresponding to each audio group in the video; This refers to the synchronization model. Indicates the first i Video features corresponding to each video block Indicates the first i The audio features corresponding to each audio group; wherein forming an initial distribution of the original video by collecting the bidirectional delay distribution, inputting the initial distribution into a pre-constructed short-term dependency estimation model for video subsequent distribution estimation, comprises: collecting the video-to-audio delay distribution corresponding to all video blocks and the audio-to-video delay distribution corresponding to all audio groups to obtain an initial distribution of the original video is represented as ； wherein, denotes the combination of the video-to-audio delay profile of the i first video block with the audio-to-video delay profile of the i first audio group, N denotes the number of video blocks and audio groups. The initial distribution The pre-constructed short-time dependence estimation model is inputted, two convolutional encoding and two deconvolutional decoding operations are sequentially performed according to the short-time dependence estimation model, and a subsequent distribution of the original video in a short time is estimated , is expressed as ； wherein, denotes a convolutional encoding operation, denotes an inverse convolutional decoding operation.

2. The method of claim 1, wherein, extracting the visual modality and the auditory modality of any type of video, and converting the auditory modality into corresponding mel spectrum, comprises: acquiring a video of any type wherein , , and denote the time length, height and width of the video, respectively, denotes the number of channels per frame of the video, denotes the real space; extracting a video of the visual modality V with the auditory modality A , respectively ； ； wherein, denotes the visual modality V of the i-th frame, denotes the audio corresponding to each frame of video, is the total number of frames in the visual modality V . The auditory modality A is converted to a corresponding mel-spectrum is represented as ； wherein denotes a mel-spectral processing.

3. The method of claim 2, wherein, inputting video blocks segmented from the visual modality into a video encoder for feature extraction to obtain video features, comprises: Splitting the visual modality V into video blocks, each video block containing five adjacent frames, denoted as ； wherein, represents the i video block, represents the V frame in the visual modality i ; inputting each video block into a video encoder performing feature extraction to obtain video features, denoted as ； wherein, represents the video feature corresponding to the i video block.

4. The method of claim 2, wherein, inputting audio groups segmented from the mel spectrum into an audio encoder for feature extraction to obtain audio features, comprises: The mel-spectrogram After the audio segments are divided, five adjacent audio segments are bundled to form an audio group, which is specifically represented as ； wherein, represents the i audio group, represents the audio segment obtained after i segmentation. inputting each audio group into an audio encoder performing feature extraction to obtain audio features, denoted as ； wherein, represents the audio feature corresponding to the i i th audio group.

5. The method of claim 1, wherein, performing back propagation training on the short-term dependency estimation model according to the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-term dependency estimation model, comprises: Utilizing KL divergence to compute a subsequent distribution of the original video loss between the initial distribution and the target distribution is represented as ； wherein, represents a loss function that computes KL the divergence; According to the loss The short-time dependency estimation model is trained by back propagation to obtain a trained short-time dependency estimation model.

6. A multimodal detection device for deepfake videos, characterized in that, The device comprises: an audio-video bidirectional synchronization distribution modeling module, configured to extract the visual modality and the auditory modality of any type of video, and convert the auditory modality into corresponding mel spectrum; input video blocks segmented from the visual modality into a video encoder for feature extraction to obtain video features, and input audio groups segmented from the mel spectrum into an audio encoder for feature extraction to obtain audio features; input the video features and the audio features into a pre-constructed synchronization model, perform audio-video bidirectional synchronization distribution modeling calculation through the synchronization model, and output bidirectional delay distribution of audio to video and video to audio. The short-time dependence estimation modeling module is configured to collect the bidirectional delay distribution to form an initial distribution of the original video, input the initial distribution into a pre-constructed short-time dependence estimation model to estimate a subsequent distribution of the video, and perform back propagation training on the short-time dependence estimation model according to a loss between the estimated subsequent distribution and the initial distribution, to obtain a trained short-time dependence estimation model, and perform visual mode and auditory mode forgery detection on a to-be-detected video according to the trained short-time dependence estimation model. The video features and the audio features are input into a pre-constructed synchronization model, bidirectional synchronization distribution modeling calculation is performed on the video features and the audio features by the synchronization model, and bidirectional delay distribution of audio to video and video to audio is output, including: The video features corresponding to each video block and the audio features corresponding to each audio group are input into a pre-constructed synchronization model, bidirectional synchronization distribution modeling calculation is performed on the video features and the audio features by the synchronization model, and bidirectional delay distribution of audio to video and video to audio is output, and is expressed as ； wherein, denotes a video-to-audio delay profile corresponding to the i video block, denotes an audio-to-video delay profile corresponding to the i audio group; denotes the synchronization model, denotes a video feature corresponding to the i video block, denotes an audio feature corresponding to the i audio group; The video features corresponding to each video block and the audio features corresponding to each audio group are input into a pre-constructed synchronization model, bidirectional synchronization distribution modeling calculation is performed on the video features and the audio features by the synchronization model, and bidirectional delay distribution of audio to video and video to audio is output, and is expressed as collecting the video-to-audio delay distribution corresponding to all video blocks and the audio-to-video delay distribution corresponding to all audio groups to obtain an initial distribution of the original video is represented as ； wherein, denotes the combination of the video-to-audio delay profile of the i first video block with the audio-to-video delay profile of the i first audio group, N denotes the number of video blocks and audio groups. The initial distribution The pre-constructed short-time dependence estimation model is inputted, two convolutional encoding and two deconvolutional decoding operations are sequentially performed according to the short-time dependence estimation model, and a subsequent distribution of the original video in a short time is estimated , is expressed as ； wherein, denotes a convolutional encoding operation, denotes an inverse convolutional decoding operation. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor implements the steps of the method in any one of claims 1 to 5 when executing the computer program. The processor implements the steps of the method in any one of claims 1 to 5 when executing the computer program.

Images