Multi-modal detection method and device for deeply-forged video and computer equipment

Abstract

The invention relates to a multi-mode detection method and device for a deep counterfeit video and computer equipment. The method comprises the following steps: extracting a visual mode and an auditory mode of any type of video, and converting the auditory mode into a corresponding Mel spectrum; respectively carrying out feature extraction on the video blocks obtained by segmenting the visual modality and the audio groups obtained by segmenting the Mel spectrum to obtain video features and audio features; inputting the video features and the audio features into a pre-constructed synchronization model to obtain audio and video bidirectional delay distribution; and integrating the bidirectional delay distribution to form initial distribution of the original video, inputting the initial distribution into a pre-constructed short-time dependency estimation model to carry out video subsequent distribution estimation, and carrying out back propagation training on the short-time dependency estimation model by calculating loss, and obtaining a trained short-time dependence estimation model to carry out counterfeiting detection on a visual mode and an auditory mode. By adopting the method, counterfeiting detection of both visual and auditory modes can be considered, and the counterfeiting detection performance is improved.

Description

Technical Field

[0001] The present application relates to the technical field of deep fake video detection, and in particular to a multimodal detection method, apparatus and computer equipment for deep fake videos. Background Art

[0002] With the rapid evolution of generative artificial intelligence technology, multimedia content creation for video and audio has entered an unprecedented high-speed development stage. However, this technological leap is also accompanied by the rise of deepfake technology. The fake multimedia content generated by deepfake technology is highly deceptive and misleading. Therefore, strengthening the research and exploration of deepfake detection mechanisms to cope with diverse counterfeiting methods and a wide range of multimedia content has become a key issue that needs to be solved urgently.

[0003] At present, multimodal deep fake detection technology has become a research hotspot. By integrating visual and auditory analysis methods, it has made significant progress in identifying forged content. However, these technologies often face the problem of overfitting to specific forgery types and data sets, which limits their generalization ability on unknown videos. To solve this problem, researchers have turned their attention to mining the intrinsic features that are common to deep fake technologies and can be generalized across domains. These features leave unique traces in tampered videos. However, such multimodal deep fake detection methods pay more attention to the detection of visual modalities, and often ignore the forgery detection of auditory modalities, resulting in poor performance for audio forgery scenarios. In addition, the autoregressive model based on Transformer-decoder uses long-term future information when generating the probability distribution of the next frame. Since videos only show stronger correlations in the short time dimension, there are large inconsistencies and differences in the long time dimension, which causes the problem of poor regression prediction effect.

[0004] Therefore, how to simultaneously detect two-way inconsistencies between video and audio, how to take into account forgery detection of both visual and auditory modalities, and how to use the short-time temporal dependency modeling of synchronization probability distribution to enhance the representation learning ability of synchronization probability distribution are very important issues in the field of deep fake detection. Summary of the invention

[0005] Based on this, it is necessary to provide a multimodal detection method, device and computer equipment for deep fake videos to address the above technical problems.

[0006] A multimodal detection method for deep fake videos, the method comprising:

[0007] Extract visual and auditory modalities of any type of video and convert the auditory modality into the corresponding Mel spectrum;

[0008] The video blocks obtained by segmenting the visual modality are input into a video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectrum are input into an audio encoder for feature extraction to obtain audio features;

[0009] The video features and audio features are input into a pre-built synchronization model, and the audio and video bidirectional synchronization distribution modeling and calculation are performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is output;

[0010] The bidirectional delay distributions are gathered to form the initial distribution of the original video, and the initial distribution is input into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video. According to the loss between the estimated subsequent distribution and the initial distribution, the short-time dependency estimation model is back-propagated and trained to obtain a trained short-time dependency estimation model. The visual and auditory modal forgery detection is performed on the video to be detected based on the trained short-time dependency estimation model.

[0011] In one embodiment, the visual modality and the auditory modality of any type of video are extracted, and the auditory modality is converted into a corresponding Mel spectrum, including:

[0012] Get any type of video in T, H and W represent the time length, height and width of the video respectively, and C represents the number of channels per frame of the video. represents the real number space;

[0013] Extract Video The visual modality V and auditory modality A of

[0014] V={f1,f2,f3,...,f n};

[0015] A={a1,a2,a3,...,a n};

[0016] Among them, f n represents the nth frame in the visual modality V, a n represents the audio corresponding to each frame of video, and n is the total number of frames in the visual modality V;

[0017] Convert the auditory modality A to the corresponding Mel spectrum Expressed as

[0018]

[0019] Among them, Mel_spectrogram represents Mel spectrum processing.

[0020] In one embodiment, the video blocks obtained by segmenting the visual modality are input into a video encoder for feature extraction to obtain video features, including:

[0021] The visual modality V is divided into video blocks, each of which contains five adjacent frames, specifically represented as

[0022]

[0023] in, represents the i-th video block, fi represents the i-th frame in the visual modality V;

[0024] Each video block is input into the video encoder VisualEncoder for feature extraction to obtain the video features, which are expressed as

[0025]

[0026] in, Represents the video features corresponding to the i-th video block.

[0027] In one embodiment, the audio group obtained by segmenting the Mel spectrum is input into an audio encoder for feature extraction to obtain audio features, including:

[0028] Mel spectrum After being segmented into audio segments, five adjacent audio segments are bundled to form an audio group, which is specifically represented as

[0029]

[0030] in, represents the i-th audio group, Represents Mel spectrum The i-th audio segment obtained after segmentation;

[0031] Input each audio group into the audio encoder AudioEncoder for feature extraction to obtain audio features, expressed as

[0032]

[0033] in, Represents the audio features corresponding to the i-th audio group.

[0034] In one embodiment, the video features and the audio features are input into a pre-built synchronization model, and the synchronization model is used to perform audio and video bidirectional synchronization distribution modeling and calculation, and the bidirectional delay distribution of audio to video and video to audio is output, including:

[0035] The video features corresponding to each video block and the audio features corresponding to each audio group are input into the pre-built synchronization model. The synchronization model is used to perform bidirectional synchronization distribution modeling and calculation of audio and video. The output is the bidirectional delay distribution of audio to video and video to audio, which is expressed as

[0036]

[0037] in, represents the delay distribution of the video to audio corresponding to the i-th video block, represents the delay distribution of the audio to video corresponding to the i-th audio group; MP_AviT represents the synchronization model, represents the video features corresponding to the i-th video block, Represents the audio features corresponding to the i-th audio group.

[0038] In one embodiment, the bidirectional delay distribution is aggregated to form an initial distribution of the original video, and the initial distribution is input into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video, including:

[0039] The delay distributions of video to audio corresponding to all video blocks and the delay distributions of audio to video corresponding to all audio groups are combined to obtain the initial distribution Ψ(i) of the original video, expressed as

[0040]

[0041] in, represents the combination of the video-to-audio delay distribution corresponding to the i-th video block and the audio-to-video delay distribution corresponding to the i-th audio group, and N represents the number of video blocks and audio groups;

[0042] The initial distribution Ψ(i) is input into the pre-built short-term dependency estimation model, and two convolution encoding and two deconvolution decoding operations are performed in sequence according to the short-term dependency estimation model to estimate the subsequent distribution of the original video in a short period of time Expressed as

[0043]

[0044] Among them, Conv represents the convolution encoding operation, and Deconv represents the deconvolution decoding operation.

[0045] In one embodiment, back propagation training is performed on the short-term dependency estimation model according to the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-term dependency estimation model, including:

[0046] Using KL divergence to calculate the subsequent distribution of the original video The loss Loss between the initial distribution Ψ(i) is expressed as

[0047]

[0048] Among them, KLDivLoss represents the loss function for calculating KL divergence;

[0049] The short-term dependency estimation model is back-propagated and trained according to the loss Loss to obtain a trained short-term dependency estimation model.

[0050] A multimodal detection device for deep fake videos, the device comprising:

[0051] The audio and video bidirectional synchronous distribution modeling module is used to extract the visual mode and auditory mode of any type of video, and convert the auditory mode into the corresponding Mel spectrum; the video block obtained by segmenting the visual mode is input into the video encoder for feature extraction to obtain video features, and the audio group obtained by segmenting the Mel spectrum is input into the audio encoder for feature extraction to obtain audio features; the video features and audio features are input into the pre-built synchronization model, and the audio and video bidirectional synchronous distribution modeling calculation is performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is obtained as output;

[0052] The short-time dependency estimation modeling module is used to aggregate the bidirectional delay distribution to form the initial distribution of the original video, input the initial distribution into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video, and perform back-propagation training on the short-time dependency estimation model based on the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-time dependency estimation model, and perform forgery detection of the visual and auditory modalities on the video to be detected based on the trained short-time dependency estimation model.

[0053] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0054] Extract visual and auditory modalities of any type of video and convert the auditory modality into the corresponding Mel spectrum;

[0055] The video blocks obtained by segmenting the visual modality are input into a video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectrum are input into an audio encoder for feature extraction to obtain audio features;

[0056] The video features and audio features are input into a pre-built synchronization model, and the audio and video bidirectional synchronization distribution modeling and calculation are performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is output;

[0057] The bidirectional delay distributions are gathered to form the initial distribution of the original video, and the initial distribution is input into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video. According to the loss between the estimated subsequent distribution and the initial distribution, the short-time dependency estimation model is back-propagated and trained to obtain a trained short-time dependency estimation model. The visual and auditory modal forgery detection is performed on the video to be detected based on the trained short-time dependency estimation model.

[0058] Compared with the prior art, the multimodal detection method, device and computer equipment for deep fake videos have the following technical effects:

[0059] 1. This application performs audio and video bidirectional synchronous distribution modeling and calculation on the video features and audio features extracted from the video. It can achieve the synchronization between video frames and audio frames based on the calculated output of the audio and video bidirectional delay distribution. At the same time, it takes into account two video forgery situations: video as the main and audio as the auxiliary and audio as the main and video as the auxiliary. It overcomes the poor forgery detection performance caused by the imbalance of audio and video data and improves the forgery detection performance in the audio forgery scenario.

[0060] 2. This application inputs the initial distribution of the original video formed by the bidirectional delay distribution of the collective audio and video into a pre-built short-term dependency estimation model to perform subsequent distribution estimation of the video in the short term, effectively utilizes the short-term correlation of the video, improves the correlation of the video in the short-time dimension and the accuracy of the evaluation process, avoids the inconsistency and difference problems caused by long-term information dependence, significantly enhances the information utilization rate in the forgery detection process, and improves the performance of forged video detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 1 is a flow chart of a multimodal detection method for deep fake videos in one embodiment;

[0062] Figure 2 A schematic diagram of a process for calculating bidirectional audio and video delay distribution in one embodiment;

[0063] Figure 3 A schematic diagram of the workflow of a short-term dependency estimation model in one embodiment;

[0064] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] In one embodiment, Figure 1 As shown, a multimodal detection method for deep fake videos is provided, comprising the following steps:

[0067] Step S1, extracting the visual modality and auditory modality of any type of video, and converting the auditory modality into the corresponding Mel spectrum.

[0068] Step S2: Input the video blocks obtained by segmenting the visual modality into a video encoder for feature extraction to obtain video features, and input the audio groups obtained by segmenting the Mel spectrum into an audio encoder for feature extraction to obtain audio features.

[0069] Step S3, input the video features and audio features into a pre-built synchronization model, perform audio and video bidirectional synchronization distribution modeling and calculation through the synchronization model, and output the bidirectional delay distribution of audio to video and video to audio.

[0070] Step S4, the bidirectional delay distribution is gathered to form the initial distribution of the original video, the initial distribution is input into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video, and the short-time dependency estimation model is back-propagated and trained according to the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-time dependency estimation model, and the visual and auditory modal forgery detection is performed on the video to be detected according to the trained short-time dependency estimation model.

[0071] In one embodiment, the visual modality and the auditory modality of any type of video are extracted, and the auditory modality is converted into a corresponding Mel spectrum, including:

[0072] Get any type of video in T, H and W represent the time length, height and width of the video respectively, and C represents the number of channels per frame of the video. represents the real number space;

[0073] Extract Video The visual modality V and auditory modality A of

[0074] V={f1,f2,f3,...,fn } ;

[0075] A={a1,a2,a3,…,a n};

[0076] Among them, f n represents the nth frame in the visual modality V, a n represents the audio corresponding to each frame of video, and n is the total number of frames in the visual modality V;

[0077] Convert the auditory modality A to the corresponding Mel spectrum Expressed as

[0078]

[0079] Among them, Mel_spectrogram represents Mel spectrum processing.

[0080] In one embodiment, the video blocks obtained by segmenting the visual modality are input into a video encoder for feature extraction to obtain video features, including:

[0081] The visual modality V is divided into video blocks, each of which contains five adjacent frames, specifically represented as

[0082]

[0083] in, represents the i-th video block, f i represents the i-th frame in the visual modality V;

[0084] Each video block is input into the video encoder VisualEncoder for feature extraction to obtain the video features, which are expressed as

[0085]

[0086] in, Represents the video features corresponding to the i-th video block.

[0087] In one embodiment, the audio group obtained by segmenting the Mel spectrum is input into an audio encoder for feature extraction to obtain audio features, including:

[0088] Mel spectrum After being segmented into audio segments, five adjacent audio segments are bundled to form an audio group, which is specifically represented as

[0089]

[0090] in, represents the i-th audio group, Represents Mel spectrum The i-th audio segment obtained after segmentation;

[0091] Input each audio group into the audio encoder AudioEncoder for feature extraction to obtain audio features, expressed as

[0092]

[0093] in, Represents the audio features corresponding to the i-th audio group.

[0094] Furthermore, the video encoder and audio encoder used in this application are different. The audio encoder combines the theoretical framework of VGG-M (Convolutional Neural Network-M), while the video encoder integrates the principle of ResNet (Residual Network).

[0095] In one embodiment, the video features and the audio features are input into a pre-built synchronization model, and the synchronization model is used to perform audio and video bidirectional synchronization distribution modeling and calculation, and the bidirectional delay distribution of audio to video and video to audio is output, including:

[0096] like Figure 2 As shown, the video features corresponding to each video block and the audio features corresponding to each audio group are input into the pre-built synchronization model, and the audio and video bidirectional synchronization distribution modeling and calculation are performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is output, which is expressed as

[0097]

[0098] in, represents the delay distribution of the video to audio corresponding to the i-th video block, describing the alignment of the video to audio; It represents the delay distribution of audio to video corresponding to the i-th audio group, describing the alignment of audio to video; MP_AviT represents the synchronization model, represents the video features corresponding to the i-th video block, Represents the audio features corresponding to the i-th audio group.

[0099] In one embodiment, the bidirectional delay distribution is aggregated to form an initial distribution of the original video, and the initial distribution is input into a pre-built short-term dependency estimation model to estimate the subsequent distribution of the video, including:

[0100] The delay distributions of video to audio corresponding to all video blocks and the delay distributions of audio to video corresponding to all audio groups are combined to obtain the initial distribution Ψ(i) of the original video, expressed as

[0101]

[0102] in, represents the combination of the video-to-audio delay distribution corresponding to the i-th video block and the audio-to-video delay distribution corresponding to the i-th audio group, and N represents the number of video blocks and audio groups;

[0103] The initial distribution Ψ(i) is input into the pre-built short-term dependency estimation model, and two convolution encoding and two deconvolution decoding operations are performed in sequence according to the short-term dependency estimation model to estimate the subsequent distribution of the original video in a short period of time Expressed as

[0104]

[0105] Among them, Conv represents the convolution encoding operation, and Deconv represents the deconvolution decoding operation.

[0106] The workflow of the short-term dependency estimation model is as follows: Figure 3 As shown, the model exploits the inherent short-term dependence properties of the distribution to facilitate the evaluation of subsequent distributions.

[0107] In one embodiment, back propagation training is performed on the short-term dependency estimation model according to the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-term dependency estimation model, including:

[0108] The KL (relative entropy) divergence is used to calculate the loss Loss between the subsequent distribution e(i+1) of the original video and the initial distribution Ψ(i), expressed as

[0109] Loss=KLDivLoss(e(i+1),Ψ(i));

[0110] Among them, KLDivLoss represents the loss function for calculating KL divergence;

[0111] The short-term dependency estimation model is back-propagated and trained according to the loss Loss to obtain a trained short-term dependency estimation model.

[0112] In one embodiment, a multimodal detection apparatus for deep fake videos is provided, comprising:

[0113] The audio and video bidirectional synchronous distribution modeling module is used to extract the visual mode and auditory mode of any type of video, and convert the auditory mode into the corresponding Mel spectrum; the video block obtained by segmenting the visual mode is input into the video encoder for feature extraction to obtain video features, and the audio group obtained by segmenting the Mel spectrum is input into the audio encoder for feature extraction to obtain audio features; the video features and audio features are input into the pre-built synchronization model, and the audio and video bidirectional synchronous distribution modeling calculation is performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is obtained as output;

[0114] The short-time dependency estimation modeling module is used to aggregate the bidirectional delay distribution to form the initial distribution of the original video, input the initial distribution into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video, and perform back-propagation training on the short-time dependency estimation model based on the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-time dependency estimation model, and perform forgery detection of the visual and auditory modalities on the video to be detected based on the trained short-time dependency estimation model.

[0115] For the specific definition of the multimodal detection device for deep fake videos, please refer to the definition of the multimodal detection method for deep fake videos above, which will not be repeated here. Each module in the above-mentioned multimodal detection device for deep fake videos can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0116] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal detection method for deep fake videos is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0117] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0119] Extract visual and auditory modalities of any type of video and convert the auditory modality into the corresponding Mel spectrum;

[0120] The video blocks obtained by segmenting the visual modality are input into a video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectrum are input into an audio encoder for feature extraction to obtain audio features;

[0121] The video features and audio features are input into a pre-built synchronization model, and the audio and video bidirectional synchronization distribution modeling and calculation are performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is output;

[0122] The bidirectional delay distributions are gathered to form the initial distribution of the original video, and the initial distribution is input into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video. According to the loss between the estimated subsequent distribution and the initial distribution, the short-time dependency estimation model is back-propagated and trained to obtain a trained short-time dependency estimation model. The visual and auditory modal forgery detection is performed on the video to be detected based on the trained short-time dependency estimation model.

[0123] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multimodal detection method for deep fake videos, It is characterized in that The method comprises: Extract visual and auditory modalities of any type of video and convert the auditory modality into the corresponding Mel-spectrogram; The video blocks obtained by segmenting the visual modality are input into a video encoder for feature extraction to obtain video features, and the audio groups obtained by segmenting the Mel spectrum are input into an audio encoder for feature extraction to obtain audio features. Input the video features and audio features into a pre-built synchronization model, perform audio and video bidirectional synchronization distribution modeling and calculation through the synchronization model, and output bidirectional delay distribution of audio to video and video to audio; The bidirectional delay distributions are assembled to form an initial distribution of the original video, the initial distribution is input into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video, and the short-time dependency estimation model is back-propagated and trained based on the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-time dependency estimation model, and the visual and auditory modal forgery detection is performed on the video to be detected based on the trained short-time dependency estimation model.

2. The method according to claim 1, It is characterized in that Extract the visual and auditory modalities of any type of video and convert the auditory modality into the corresponding Mel spectrum, including: Get any type of video in T, H and W represent the time length, height and width of the video respectively, and C represents the number of channels per frame of the video. represents the real number space; Extract Video The visual modality V and auditory modality A of V={f 1 ,f 2 ,f 3 ,...,f n }; A={a 1 ,a 2 ,a 3 ,...,a n }; Among them, f n represents the nth frame in the visual modality V, a n represents the audio corresponding to each frame of video, and n is the total number of frames in the visual modality V; Convert the auditory modality A into the corresponding Mel spectrum Expressed as Among them, Mel_spectrogram represents Mel spectrum processing.

3. The method according to claim 2, It is characterized in that The video blocks obtained by segmenting the visual modality are input into a video encoder for feature extraction to obtain video features, including: The visual modality V is divided into video blocks, each of which contains five adjacent frames, specifically represented as in, represents the i-th video block, f i represents the i-th frame in the visual modality V; Each video block is input into the video encoder VisualEncoder for feature extraction to obtain the video features, which are expressed as in, Represents the video features corresponding to the i-th video block.

4. The method according to claim 2, It is characterized in that The audio group obtained by dividing the Mel spectrum is input into an audio encoder for feature extraction to obtain audio features, including: Mel spectrum After being segmented into audio segments, five adjacent audio segments are bundled to form an audio group, which is specifically represented as in, represents the i-th audio group, Represents Mel spectrum The i-th audio segment obtained after segmentation; Input each audio group into the audio encoder AudioEncoder for feature extraction to obtain audio features, expressed as in, Represents the audio features corresponding to the i-th audio group.

5. The method according to claim 1, It is characterized in that The video features and audio features are input into a pre-built synchronization model, and the audio and video bidirectional synchronization distribution modeling and calculation are performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is output, including: The video features corresponding to each video block and the audio features corresponding to each audio group are input into the pre-built synchronization model. The synchronization model is used to perform bidirectional synchronization distribution modeling and calculation of audio and video. The output is the bidirectional delay distribution of audio to video and video to audio, which is expressed as in, represents the delay distribution of the video to audio corresponding to the i-th video block, represents the delay distribution of the audio to video corresponding to the i-th audio group; MP_AviT represents the synchronization model, represents the video features corresponding to the i-th video block, Represents the audio features corresponding to the i-th audio group.

6. The method according to claim 5, It is characterized in that The bidirectional delay distribution is assembled to form an initial distribution of the original video, and the initial distribution is input into a pre-built short-term dependency estimation model to estimate subsequent distribution of the video, including: The delay distributions of video to audio corresponding to all video blocks and the delay distributions of audio to video corresponding to all audio groups are combined to obtain the initial distribution Ψ(i) of the original video, expressed as in, represents the combination of the video-to-audio delay distribution corresponding to the i-th video block and the audio-to-video delay distribution corresponding to the i-th audio group, and N represents the number of video blocks and audio groups; The initial distribution Ψ(i) is input into a pre-built short-term dependency estimation model, and two convolution encoding and two deconvolution decoding operations are performed in sequence according to the short-term dependency estimation model to estimate the subsequent distribution e(i+1) of the original video in a short period of time, which is expressed as Among them, Conv represents the convolution encoding operation, and Deconv represents the deconvolution decoding operation.

7. The method according to claim 6, It is characterized in that According to the loss between the estimated subsequent distribution and the initial distribution, the short-term dependency estimation model is back-propagated and trained to obtain a trained short-term dependency estimation model, including: KL divergence is used to calculate the loss Loss between the subsequent distribution e(i+1) of the original video and the initial distribution Ψ(i), expressed as Loss=KLDivLoss(e(i+1),Ψ(i)); Among them, KLDivLoss represents the loss function for calculating KL divergence; The short-term dependency estimation model is back-propagated and trained according to the loss Loss to obtain a trained short-term dependency estimation model.

8. A multimodal detection device for deep fake videos, It is characterized in that The device comprises: The audio and video bidirectional synchronous distribution modeling module is used to extract the visual mode and auditory mode of any type of video, and convert the auditory mode into the corresponding Mel spectrum; the video block obtained by segmenting the visual mode is input into the video encoder for feature extraction to obtain video features, and the audio group obtained by segmenting the Mel spectrum is input into the audio encoder for feature extraction to obtain audio features; the video features and audio features are input into a pre-built synchronization model, and the audio and video bidirectional synchronous distribution modeling calculation is performed through the synchronization model, and the bidirectional delay distribution of audio to video and video to audio is obtained as output; The short-time dependency estimation modeling module is used to gather the bidirectional delay distribution to form the initial distribution of the original video, input the initial distribution into a pre-built short-time dependency estimation model to estimate the subsequent distribution of the video, and perform back-propagation training on the short-time dependency estimation model based on the loss between the estimated subsequent distribution and the initial distribution to obtain a trained short-time dependency estimation model, and perform forgery detection of the visual modality and the auditory modality on the video to be detected based on the trained short-time dependency estimation model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images