Deep fake detection method, device and equipment based on cross-modal mask modeling

Abstract

The present application relates to a method, apparatus and device for deep fake detection based on cross-modal mask modeling. The method comprises: constructing a deep fake detection model comprising a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; training and optimizing the deep fake detection model by integrating audio and video reconstruction loss and audio and video contrastive learning loss; inputting the audio and video sequence to be detected into the trained deep fake detection model, obtaining the cross-attention coding features of the audio and video to be detected according to the model output, inputting the cross-attention coding features of the audio and video to be detected into the classifier for audio and video matching correlation detection, and outputting the forgery detection results of the audio and video sequence. This method can utilize mask modeling technology to learn the representation of real videos and cross-fuse and contrastively learn the features of audio and video modalities to improve the forgery detection performance.

Description

Technical Field

[0001] The present application relates to the technical field of deep fake detection, and in particular to a deep fake detection method, apparatus and device based on cross-modal mask modeling. Background Art

[0002] Deep fake technology refers to the deceptive modification of a person's face in a video or image. This technology can produce high-fidelity videos that include fictional facial expressions, customized voice statements, or seamless face replacement in visual media. It has attracted widespread attention from various industries including entertainment, news, and art. There is an urgent need for extensive research on deep fake detection mechanisms to reduce the abuse of this technology.

[0003] Deepfake detection mechanisms aim to achieve accurate classification by identifying deceptive cues in videos or images. These techniques are generally divided into two categories: unimodal and multimodal approaches. Traditional unimodal detection methods focus solely on image-based content, carefully examining facial regions for artifacts, distortions, frequency variations, and other subtle spatial details that indicate manipulation. However, this approach is limited to static image data and struggles to handle the dynamic nature of complex multimedia content. In recent years, work on deepfake detection using multimodal techniques has achieved remarkable results. When dealing with audio forgery scenarios, it is important to consider not only visual artifacts but also auditory cues.

[0004] Existing studies have developed modality-specific detection models for visual and auditory data separately, and subsequently merged these classification results to produce a unified output. However, these methods separate multimodal information and ignore the interrelationships between different modalities. The latest research focuses on extracting global correlations between modalities by fusing features from the two modalities. For example, deep fake videos have huge differences in the time delay between audio and vision compared to real videos. An autoregressive model is used to fit the distribution of time delays between modes in real videos, and samples that exhibit abnormal time delays are subsequently labeled as deep fake videos. Although these methods exploit high-level semantic information from audio-visual pairs, they ignore the coupling of shallow details of heterogeneous modalities, which is crucial for multimodal deep fake detection.

[0005] Therefore, how to leverage the coupling of high-level semantic information from audio-visual pairs and shallow details from heterogeneous modalities to improve the performance of deepfake detection models remains a huge challenge. Summary of the Invention

[0006] Based on this, it is necessary to provide a deep fake detection method, device and equipment based on cross-modal mask modeling to address the above technical problems. This method uses mask modeling technology to learn the representation of real videos and cross-fuse and compare the features of audiovisual modalities to improve the fake detection performance.

[0007] A deep fake detection method based on cross-modal mask modeling, the method comprising:

[0008] A deep fake detection model is constructed, which includes a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein, the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and performs segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and outputs video features and audio features composed of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and feature decode the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0009] The audio and video reconstruction loss is constructed based on the audio and video reconstruction results output by the cross-attention fusion module. The audio and video comparative learning loss is constructed based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a well-trained deep fake detection model is obtained.

[0010] The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed. The cross-attention coding features of the audio and video to be detected are output, and the cross-attention coding features of the audio and video to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

[0011] In one embodiment, the multimodal mask modeling module extracts the video modality and audio modality from the audio and video sequence, and performs segmentation, mask modeling, position encoding, and feature encoding on the video modality and audio modality, respectively, to output video features and audio features each consisting of encoding tokens and encoding features, including:

[0012] Extract the video modality from the audio and video sequence, divide the video modality into video blocks according to the preset first step length and second step length, perform mask modeling on each video block, and tile and position encode all video blocks with mask value 1 in turn to obtain the unmasked video features after position encoding. The unmasked video features after position encoding and the learnable embedding vector cls of the video modality are combined. v Input to the VIT-based encoder for video feature encoding, and output the video features including video coding class tokens and video coding features;

[0013] Extract the audio modality from the audio and video sequence and convert it into the corresponding Mel spectrogram. According to the preset third step length, the Mel spectrogram is divided into unit blocks. Mask modeling is performed for each unit block. All unit blocks with mask value 1 are tiled and position encoded in turn to obtain the unmasked audio features after position encoding. The unmasked audio features after position encoding and the learnable embedding vector cls of the audio modality are combined. a The audio feature is input to the VIT-based encoder for audio feature encoding, and the output is an audio feature containing audio coding class tokens and audio coding features.

[0014] In one embodiment, the video modality is extracted from the audio and video sequence, the video modality is divided into video blocks according to the preset first step length and second step length, mask modeling is performed on each video block, and all video blocks with a mask value of 1 are tiled and position-coded in sequence to obtain unmasked video features after position coding, and the unmasked video features after position coding and the learnable embedding vector cls of the video modality are combined. v The video is input to the VIT-based encoder for video feature encoding, and the output is a video feature containing video coding class tokens and video coding features, including:

[0015] Extract audio and video sequences The video mode V in ; where, Represents the real number space to which the audio and video sequence belongs, D represents the depth of the audio and video sequence, H, W, and C correspond to the height, width, and number of channels of each frame respectively;

[0016] The video modality V is divided into video blocks α according to the preset first step length τ1 and second step length τ2 d,h,w , expressed as

[0017]

[0018] Where D′, H′, and W′ represent the total number of steps of D, H, and W, respectively, and d, h, and w represent the indexes of D, H, and W, respectively;

[0019] For each video block α d,h,w Perform mask modeling and transform each video block αd,h,w The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means shielding the video block corresponding to the mask. When it is 1, it means that the video block corresponding to the mask is retained;

[0020] Use the PatchEmb3D algorithm to tile all video blocks with a mask value of 1 to obtain the unmasked video feature Ω v , expressed as

[0021]

[0022] For unmasked video features Ω v Perform position encoding to obtain the unmasked video features after position encoding Expressed as

[0023]

[0024] in, represents the position encoding of each position pair (m,n) in the unmasked video feature, represents the unmasked video feature of each position pair (m,n), Sincospos represents the sinusoidal position encoding, m∈M and n∈N, M and N represent the height and width of the unmasked video feature respectively;

[0025] The unmasked video features after position encoding And the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output contains the video coding class token CLS v and video coding featuresξ v The video features are expressed as

[0026]

[0027] Among them, ViTEncoder represents the VIT-based encoder.

[0028] In one embodiment, the audio mode in the audio and video sequence is extracted and converted into a corresponding mel spectrogram, the mel spectrogram is divided into unit blocks according to a preset third step size, mask modeling is performed on each unit block, and all unit blocks with a mask value of 1 are tiled and position-encoded in sequence to obtain unmasked audio features after position encoding, and the unmasked audio features after position encoding and the learnable embedding vector cls of the audio mode are combined. aThe audio feature is input to the VIT-based encoder for audio feature encoding, and the output is an audio feature containing audio coding class tokens and audio coding features, including:

[0029] Extract the audio modality A from the audio and video sequence and convert the audio modality A into the corresponding Mel spectrogram A mel , and according to the preset third step length τ3, the Mel spectrogram A mel Divide into unit blocks Mel-spectrogram A mel and unit blocks Respectively expressed as

[0030]

[0031] Among them, Mel_spectrogram represents Mel spectrogram conversion, h and w represent the indexes of H and W respectively, Represents the real number space to which the audio and video sequence belongs, where H and W correspond to the height and width of each frame respectively;

[0032] For each unit block Perform mask modeling and transform each unit block The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the unit block corresponding to the mask is shielded. When it is 1, it means that the unit block corresponding to the mask is retained;

[0033] The PatchEmb2D algorithm is used to tile all unit blocks with a mask value of 1 to obtain unmasked audio features. Expressed as

[0034]

[0035] Unmasked audio features Perform position encoding to obtain the unmasked audio features after position encoding Expressed as

[0036]

[0037] in, Represents the position encoding of the unmasked audio feature, and Sincospos represents the sinusoidal position encoding;

[0038] Unmasked audio features after position encoding and the learnable embedding vector cls of the audio modality a Input to the VIT-based encoder for audio feature encoding, and the output includes audio coding class token CLS aWith audio coding features γ a The audio features are expressed as

[0039]

[0040] Among them, ViTEncoder represents the VIT-based encoder.

[0041] In one embodiment, the cross-attention fusion module fuses encoding features from video features and audio features to obtain fused encoding features, and performs cross-attention modeling based on the fused encoding features, including:

[0042] By fusing the video coding features ξ v and audio coding features in audio features γ a , get the fusion coding feature β va , denoted as β va =concat(ξ v ,γ a ), where concat represents fusion processing;

[0043] Based on the fusion coding feature β va Perform cross-attention modeling to obtain the video cross-attention encoding feature Φ v and audio cross-attention encoding features Φ a , respectively expressed as

[0044]

[0045] Among them, Q, K, V represent query vector, key vector and value vector respectively, and W Q 、W K and W V They represent the query vector, key vector, and value vector after conversion by the attention mechanism, Softmax represents the normalized exponential function, and the superscript T represents the transposition operation.

[0046] In one embodiment, the cross-attention fusion module decodes the video cross-attention encoding features and the audio cross-attention encoding features obtained by the cross-attention modeling, and outputs the audio and video reconstruction results, including:

[0047] The VIT-based decoder is used to convert the video cross-attention encoding feature Φ obtained by cross-attention modeling v and audio cross-attention encoding features Φ a Perform feature decoding separately to obtain video reconstruction results and audio reconstruction results Respectively expressed as

[0048]

[0049] Among them, VitDecoder represents a decoder based on VIT.

[0050] In one embodiment, the contrastive learning module fuses the encoding tokens in the video features and the audio features to obtain a mutual information matrix, and calculates the audio and video similarity based on the mutual information matrix, including:

[0051] Video Coding Class Tokens (CLS) by Fusion of Video Features v The audio coding class token CLS in the audio feature a , and get the mutual information matrix Ψ N×N , expressed as

[0052]

[0053] Wherein, the superscript T represents the transposition operation, and N represents the width of the unmasked video feature;

[0054] The mutual information matrix Ψ N×N The diagonal elements of are accumulated to obtain the audio and video similarity Similar(V,A), which is expressed as

[0055]

[0056] Among them, V and A represent the video mode and audio mode respectively, i,j represents the element in the i-th row and j-th column of the mutual information matrix.

[0057] In one embodiment, an audio and video reconstruction loss is constructed based on the audio and video reconstruction results output by the cross-attention fusion module, and an audio and video comparative learning loss is constructed based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained, including:

[0058] Construct audio and video reconstruction loss based on the video reconstruction results and audio reconstruction results output by the cross-attention fusion module Expressed as

[0059]

[0060] Among them, Loss v Represents the video reconstruction loss, Loss a represents the audio reconstruction loss, MAE represents the masked autoencoder, and Represent the video reconstruction results and audio reconstruction results respectively, Represents the video block α d,h,w The corresponding mask, Represents a unit block The corresponding mask;

[0061] Construct the audio and video contrast learning loss based on the audio and video similarity Similar(V,A) output by the contrast learning module Expressed as Among them, V and A represent video mode and audio mode respectively;

[0062] The final loss is constructed by combining audio and video reconstruction loss and audio and video contrast learning loss Expressed as Where λ represents the balance and The weight factor of

[0063] Based on the final loss The deep fake detection model is trained and optimized until a trained deep fake detection model is obtained.

[0064] A deep fake detection device based on cross-modal mask modeling, the device comprising:

[0065] A model construction unit is used to construct a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and output video features and audio features composed of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0066] The model training unit is used to construct an audio and video reconstruction loss based on the audio and video reconstruction results output by the cross-attention fusion module, and to construct an audio and video comparative learning loss based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained;

[0067] The forgery detection unit is used to input the audio and video sequence to be detected into the trained deep forgery detection model, extract the audio and video coding features to be detected according to the multimodal mask modeling module in the model, fuse the audio and video coding features to be detected according to the cross-attention fusion module and perform cross-attention modeling, output the audio and video cross-attention coding features to be detected, input the audio and video cross-attention coding features to be detected into the classifier for audio and video matching correlation detection, and output the forgery detection result of the audio and video sequence.

[0068] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0069] A deep fake detection model is constructed, which includes a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein, the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and performs segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and outputs video features and audio features composed of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and feature decode the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0070] The audio and video reconstruction loss is constructed based on the audio and video reconstruction results output by the cross-attention fusion module. The audio and video comparative learning loss is constructed based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a well-trained deep fake detection model is obtained.

[0071] The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed. The cross-attention coding features of the audio and video to be detected are output, and the cross-attention coding features of the audio and video to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

[0072] The above-mentioned deep fake detection method, device and equipment based on cross-modal mask modeling perform fake detection by constructing and training a deep fake detection model. Among them, the multimodal mask modeling module based on the model can adopt the cross-modal mask modeling method to utilize the intrinsic interaction of heterogeneous modalities to reconstruct video and audio features, enhance multimodal representation and reduce overfitting tendency, laying a solid data foundation for fake detection; the cross-attention fusion module based on the model can locally and finely fuse the shallow features of two heterogeneous modalities, promote the seamless connection and complementarity of information between heterogeneous modalities, and further enhance the model's sensitivity and recognition ability to fake content; the contrastive learning module based on the model can adopt the audio-visual contrast learning strategy to globally align video features and audio features, strengthen the consistency test between features. This strategy not only improves the model's global recognition ability of fake content, but also enhances the generalization performance of the model under different fake techniques, ensuring the stability and reliability of the fake detection results. This application constructs an efficient, accurate and robust deep fake detection model through multimodal mask modeling, cross-attention fusion and contrastive learning, providing strong technical support for coping with increasingly complex fake challenges. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 Schematic diagram of a process for deep fake detection based on cross-modal mask modeling in one embodiment;

[0074] Figure 2 A schematic diagram of the structure of a deep fake detection model in one embodiment;

[0075] Figure 3 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0077] In one embodiment, Figure 1 As shown, a deep fake detection method based on cross-modal mask modeling is provided, comprising the following steps:

[0078] Step S1, constructing a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein, the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and output video features and audio features composed of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix.

[0079] The structure of the deep fake detection model is as follows Figure 2 The workflow of the multimodal mask modeling module in the model includes: first, extracting the video modality from the audio and video sequence, dividing the video modality into video blocks according to the preset first step length and second step length, performing mask modeling on each video block, and then tiling and position encoding all video blocks with a mask value of 1 in turn to obtain the unmasked video features after position encoding, and embedding the unmasked video features after position encoding and the learnable embedding vector cls of the video modality. v The video feature is input to the VIT-based encoder for video feature encoding, and the output is a video feature containing video coding class tokens and video coding features. Then, the audio mode in the audio and video sequence is extracted and converted into the corresponding Mel spectrogram. The Mel spectrogram is divided into unit blocks according to the preset third step length. Mask modeling is performed for each unit block, and all unit blocks with mask values ​​1 are tiled and position encoded in turn to obtain the unmasked audio features after position encoding. The unmasked audio features after position encoding and the learnable embedding vector cls of the audio mode are combined. a The audio feature is input to the VIT-based encoder for audio feature encoding, and the output is an audio feature containing audio coding class tokens and audio coding features.

[0080] The specific workflow of the multimodal mask modeling module for video modality includes:

[0081] Extract audio and video sequences The video mode V in ; where, Represents the real number space to which the audio and video sequence belongs, D represents the depth of the audio and video sequence, H, W, and C correspond to the height, width, and number of channels of each frame respectively;

[0082] The video modality V is divided into video blocks α according to the preset first step length τ1 and second step length τ2d,h,w , expressed as

[0083]

[0084]

[0085] Where D′, H′, and W′ represent the total number of steps of D, H, and W, respectively, and d, h, and w represent the indexes of D, H, and W, respectively. Specifically, the first step length τ1 and the second step length τ2 are set to 2 and 16, respectively.

[0086] For each video block α d,h,w Perform mask modeling and transform each video block α d,h,w The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means shielding the video block corresponding to the mask. When it is 1, it means that the video block corresponding to the mask is retained;

[0087] The PatchEmb3D (three-dimensional patch embedding) algorithm is used to tile all video blocks with a mask value of 1 to obtain the unmasked video feature Ω v (Ω v is a two-dimensional matrix), expressed as

[0088]

[0089] For unmasked video features Ω v Perform position encoding to obtain the unmasked video features after position encoding Expressed as

[0090]

[0091] in, represents the position encoding of each position pair (m,n) in the unmasked video feature, represents the unmasked video feature of each position pair (m,n), Sincospos represents the sinusoidal position encoding, m∈M and n∈N, M and N represent the height and width of the unmasked video feature respectively;

[0092] The unmasked video features after position encoding And the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output contains the video coding class token CLS v and video coding featuresξ v The video features are expressed as

[0093]

[0094] Among them, ViTEncoder represents the VIT-based encoder.

[0095] The specific workflow of the multimodal mask modeling module for audio modality includes:

[0096] Extract the audio modality A from the audio and video sequence and convert the audio modality A into the corresponding Mel spectrogram A mel , and according to the preset third step length τ3, the Mel spectrogram A mel Divide into unit blocks Mel-spectrogram A mel and unit blocks are represented as

[0097]

[0098] Among them, Mel_spectrogram represents Mel spectrogram conversion, h and w represent the indexes of H and W respectively, Represents the real number space to which the audio and video sequence belongs, H and W correspond to the height and width of each frame respectively. width;

[0099] For each unit block Perform mask modeling and transform each unit block The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the unit block corresponding to the mask is shielded. When it is 1, it means that the unit block corresponding to the mask is retained;

[0100] The PatchEmb2D (two-dimensional patch embedding) algorithm is used to tile all unit blocks with a mask value of 1 to obtain unmasked audio features. Expressed as

[0101]

[0102] Unmasked audio features Perform position encoding to obtain the unmasked audio features after position encoding Expressed as

[0103]

[0104] in, Represents the position encoding of the unmasked audio feature, and Sincospos represents the sinusoidal position encoding;

[0105] Unmasked audio features after position encoding and the learnable embedding vector cls of the audio modalitya Input to the VIT-based encoder for audio feature encoding, and the output includes audio coding class token CLS a With audio coding features γ a The audio features are expressed as

[0106]

[0107] Among them, ViTEncoder represents the VIT-based encoder.

[0108] Specifically, the VIT-based encoder is a visual encoder based on the Transformer model, which includes two normalization layers, a multi-head attention mechanism layer, and a multi-layer perceptron (MLP).

[0109] The specific workflow of the cross-attention fusion module in the model includes:

[0110] By fusing the video coding features ξ v and audio coding features in audio features γ a , get the fusion coding feature β va , denoted as β va =concat(ξ v ,γ a ), where concat represents fusion processing;

[0111] Based on the fusion coding feature β va Perform cross-attention modeling to obtain the video cross-attention encoding feature Φ v and audio cross-attention encoding features Φ a , respectively expressed as

[0112]

[0113] Among them, Q, K, V represent query vector, key vector and value vector respectively, and W Q 、W K and W V They represent the query vector, key vector, and value vector after the attention mechanism conversion, Softmax represents the normalized exponential function, and the superscript T represents the transposition operation;

[0114] The VIT-based decoder is used to convert the video cross-attention encoding feature Φ obtained by cross-attention modeling v and audio cross-attention encoding features Φ a Perform feature decoding separately to obtain video reconstruction results and audio reconstruction results Respectively expressed as

[0115]

[0116] VitDecoder represents a VIT-based decoder. The structure of the VIT-based decoder is consistent with that of the VIT-based encoder.

[0117] The specific workflow of the contrastive learning module in the model includes:

[0118] Video Coding Class Tokens (CLS) by Fusion of Video Features v The audio coding class token CLS in the audio feature a , and get the mutual information matrix Ψ N×N , expressed as

[0119]

[0120] Wherein, the superscript T represents the transposition operation, and N represents the width of the unmasked video feature;

[0121] The mutual information matrix Ψ N×N The diagonal elements of are accumulated to obtain the audio and video similarity Similar(V,A), which is expressed as

[0122]

[0123] Among them, V and A represent the video mode and audio mode respectively, i,j represents the element in the i-th row and j-th column of the mutual information matrix.

[0124] In step S2, an audio and video reconstruction loss is constructed based on the audio and video reconstruction results output by the cross-attention fusion module, and an audio and video comparative learning loss is constructed based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained.

[0125] The specific implementation process of step S2 includes:

[0126] Construct audio and video reconstruction loss based on the video reconstruction results and audio reconstruction results output by the cross-attention fusion module Expressed as

[0127]

[0128] Among them, Loss v Represents the video reconstruction loss, Loss a represents the audio reconstruction loss, MAE represents the masked autoencoder, and Represent the video reconstruction results and audio reconstruction results respectively, Represents the video block αd,h,w The corresponding mask, Represents a unit block The corresponding mask;

[0129] Construct the audio and video contrast learning loss based on the audio and video similarity Similar(V,A) output by the contrast learning module Expressed as Among them, V and A represent video mode and audio mode respectively;

[0130] The final loss is constructed by combining audio and video reconstruction loss and audio and video contrast learning loss Expressed as Where λ represents the balance and The weight factor of

[0131] Based on the final loss The deep fake detection model is trained and optimized until a trained deep fake detection model is obtained.

[0132] In step S3, the audio and video sequence to be detected is input into the trained deep fake detection model, the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model, the audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed, and the audio and video cross-attention coding features to be detected are output. The audio and video cross-attention coding features to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

[0133] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0134] In one embodiment, a deep fake detection apparatus based on cross-modal mask modeling is provided, comprising:

[0135] A model construction unit is used to construct a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and output video features and audio features composed of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0136] The model training unit is used to construct an audio and video reconstruction loss based on the audio and video reconstruction results output by the cross-attention fusion module, and to construct an audio and video comparative learning loss based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained;

[0137] The forgery detection unit is used to input the audio and video sequence to be detected into the trained deep forgery detection model, extract the audio and video coding features to be detected according to the multimodal mask modeling module in the model, fuse the audio and video coding features to be detected according to the cross-attention fusion module and perform cross-attention modeling, output the audio and video cross-attention coding features to be detected, input the audio and video cross-attention coding features to be detected into the classifier for audio and video matching correlation detection, and output the forgery detection result of the audio and video sequence.

[0138] For the specific limitations of the deep fake detection device based on cross-modal mask modeling, please refer to the limitations of the deep fake detection method based on cross-modal mask modeling above, which will not be repeated here. The various modules in the above-mentioned deep fake detection device based on cross-modal mask modeling can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0139] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a deep fake detection method based on cross-modal mask modeling is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0140] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0142] A deep fake detection model is constructed, which includes a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein, the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and performs segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and outputs video features and audio features composed of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and feature decode the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0143] The audio and video reconstruction loss is constructed based on the audio and video reconstruction results output by the cross-attention fusion module. The audio and video comparative learning loss is constructed based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a well-trained deep fake detection model is obtained.

[0144] The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed. The cross-attention coding features of the audio and video to be detected are output, and the cross-attention coding features of the audio and video to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

[0145] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A deep fake detection method based on cross-modal mask modeling, characterized by: The method comprises: Construct a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and output video features and audio features both consisting of coding tokens and coding features; the cross-attention fusion module is used to fuse the coding features in the video features and audio features to obtain fused coding features, perform cross-attention modeling based on the fused coding features, and perform feature decoding on the modeled video cross-attention coding features and audio cross-attention coding features, and output audio and video reconstruction results; the contrastive learning module is used to fuse the coding tokens in the video features and audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix; Constructing an audio and video reconstruction loss based on the audio and video reconstruction results output by the cross-attention fusion module, and constructing an audio and video comparative learning loss based on the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained; The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed, and the audio and video cross-attention coding features to be detected are output. The audio and video cross-attention coding features to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

2. The method according to claim 1, characterized in that The multimodal mask modeling module extracts the video modality and audio modality from the audio and video sequence, and performs segmentation, mask modeling, position encoding, and feature encoding on the video modality and audio modality respectively, and outputs video features and audio features each consisting of encoding tokens and encoding features, including: Extract the video modality from the audio and video sequence, divide the video modality into video blocks according to the preset first step length and second step length, perform mask modeling on each video block, and tile and position encode all video blocks with mask value 1 in turn to obtain the unmasked video features after position encoding. The unmasked video features after position encoding and the learnable embedding vector cls of the video modality are used as the embedding vector. v Input to the VIT-based encoder for video feature encoding, and output the video features including video coding class tokens and video coding features; Extract the audio modality in the audio and video sequence and convert it into the corresponding Mel spectrogram. According to the preset third step size, the Mel spectrogram is divided into unit blocks. Mask modeling is performed on each unit block. All unit blocks with mask value 1 are tiled and position encoded in turn to obtain the unmasked audio features after position encoding. The unmasked audio features after position encoding and the learnable embedding vector cls of the audio modality are combined. a The audio feature is input to the VIT-based encoder for audio feature encoding, and the output is an audio feature containing audio coding class tokens and audio coding features.

3. The method according to claim 2, characterized in that Extract the video modality from the audio and video sequence, divide the video modality into video blocks according to the preset first step length and second step length, perform mask modeling on each video block, and tile and position encode all video blocks with mask value 1 in turn to obtain the unmasked video features after position encoding. The unmasked video features after position encoding and the learnable embedding vector cls of the video modality are used as the embedding vector. v The video is input to the VIT-based encoder for video feature encoding, and the output is a video feature containing video coding class tokens and video coding features, including: Extract audio and video sequences The video mode V in ; where, Represents the real number space to which the audio and video sequence belongs, D represents the depth of the audio and video sequence, H, W, and C correspond to the height, width, and number of channels of each frame respectively; The video modality V is divided into video blocks α according to the preset first step length τ1 and second step length τ2 d,h,w , expressed as Where D′, H′, and W′ represent the total number of steps of D, H, and W, respectively, and d, h, and w represent the indexes of D, H, and W, respectively; For each video block α d,h,w Perform mask modeling and transform each video block α d,h,w The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means shielding the video block corresponding to the mask. When it is 1, it means that the video block corresponding to the mask is retained; Use the PatchEmb3D algorithm to tile all video blocks with a mask value of 1 to obtain the unmasked video feature Ω v , expressed as For the unmasked video feature Ω v Perform position encoding to obtain the unmasked video features after position encoding Expressed as in, represents the position encoding of each position pair (m,n) in the unmasked video feature, represents the unmasked video feature of each position pair (m,n), Sincospos represents the sinusoidal position encoding, m∈M and n∈N, M and N represent the height and width of the unmasked video feature respectively; The unmasked video features after the position encoding And the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output contains the video coding class token CLS v and video coding featuresξ v The video features are expressed as Among them, ViTEncoder represents the VIT-based encoder.

4. The method according to claim 2, characterized in that Extract the audio modality in the audio and video sequence and convert it into the corresponding Mel spectrogram. According to the preset third step size, the Mel spectrogram is divided into unit blocks. Mask modeling is performed on each unit block. All unit blocks with mask value 1 are tiled and position encoded in turn to obtain the unmasked audio features after position encoding. The unmasked audio features after position encoding and the learnable embedding vector cls of the audio modality are combined. a The audio feature is input to the VIT-based encoder for audio feature encoding, and the output is an audio feature containing audio coding class tokens and audio coding features, including: Extract the audio mode A from the audio and video sequence and convert the audio mode A into the corresponding Mel spectrogram A mel , and the Mel spectrogram A is converted into mel Divide into unit blocks The mel-spectrogram A mel and unit blocks Respectively expressed as Among them, Mel_spectrogram represents Mel spectrogram conversion, h and w represent the indexes of H and W respectively, Represents the real number space to which the audio and video sequence belongs, where H and W correspond to the height and width of each frame respectively; For each unit block Perform mask modeling and transform each unit block The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the unit block corresponding to the mask is shielded. When it is 1, it means that the unit block corresponding to the mask is retained; The PatchEmb2D algorithm is used to tile all unit blocks with a mask value of 1 to obtain unmasked audio features. Expressed as For the unmasked audio features Perform position encoding to obtain the unmasked audio features after position encoding Expressed as in, Represents the position encoding of the unmasked audio feature, and Sincospos represents the sinusoidal position encoding; The unmasked audio features after encoding the position and the learnable embedding vector cls of the audio modality a Input to the VIT-based encoder for audio feature encoding, and the output includes audio coding class token CLS a With audio coding features γ a The audio features are expressed as Among them, ViTEncoder represents the VIT-based encoder.

5. The method according to claim 1, wherein The cross-attention fusion module fuses the coding features in the video features and the audio features to obtain a fused coding feature, and performs cross-attention modeling based on the fused coding feature, including: By fusing the video coding feature ξ in the video feature v and the audio coding feature γ in the audio feature a , get the fusion coding feature β va , denoted as β va =concat(ξ v ,γ a ), where concat represents fusion processing; Based on the fusion coding feature β va Perform cross-attention modeling to obtain the video cross-attention encoding feature Φ v and audio cross-attention encoding features Φ a , respectively expressed as Among them, Q, K, V represent query vector, key vector and value vector respectively, and W Q 、W K and W V They represent the query vector, key vector, and value vector after conversion by the attention mechanism, Softmax represents the normalized exponential function, and the superscript T represents the transposition operation.

6. The method according to claim 5, characterized in that The cross-attention fusion module decodes the video cross-attention encoding features and audio cross-attention encoding features obtained by cross-attention modeling, and outputs the audio and video reconstruction results, including: The VIT-based decoder is used to convert the video cross-attention encoding feature Φ obtained by cross-attention modeling v and audio cross-attention encoding features Φ a Perform feature decoding respectively to obtain the video reconstruction result l V And the audio reconstruction result l a , respectively expressed as Among them, VitDecoder represents a decoder based on VIT.

7. The method according to claim 1, characterized in that The contrastive learning module fuses the encoding tokens in the video features and the audio features to obtain a mutual information matrix, and calculates the audio and video similarity based on the mutual information matrix, including: By integrating the video features of the video coding class token CLS v The audio coding class token CLS in the audio feature a , and get the mutual information matrix Ψ N×N , expressed as Wherein, the superscript T represents the transposition operation, and N represents the width of the unmasked video feature; The mutual information matrix Ψ N×N The diagonal elements of are accumulated to obtain the audio and video similarity Similar(V,A), which is expressed as Among them, V and A represent the video mode and audio mode respectively, i,j represents the element in the i-th row and j-th column of the mutual information matrix.

8. The method according to claim 1, characterized in that Constructing an audio and video reconstruction loss based on the audio and video reconstruction results output by the cross-attention fusion module, constructing an audio and video comparative learning loss based on the audio and video similarity output by the comparative learning module, and training and optimizing the deep fake detection model by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained, including: The audio and video reconstruction loss is constructed based on the video reconstruction result and the audio reconstruction result output by the cross-attention fusion module. Expressed as Among them, Loss v Represents the video reconstruction loss, Loss a represents the audio reconstruction loss, MAE represents the masked autoencoder, l V and l a Represent the video reconstruction results and audio reconstruction results respectively, Represents the video block α d,h,w The corresponding mask, Represents a unit block The corresponding mask; Construct the audio and video contrast learning loss according to the audio and video similarity Similar(V,A) output by the contrast learning module Expressed as Among them, V and A represent video mode and audio mode respectively; The final loss is constructed by combining the audio and video reconstruction loss and the audio and video contrast learning loss Expressed as Where λ represents the balance and The weight factor of Based on the final loss The deep fake detection model is trained and optimized until a trained deep fake detection model is obtained.

9. A deep fake detection device based on cross-modal mask modeling, characterized in that: The device comprises: A model construction unit is used to construct a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and audio modality in the audio and video sequence, and perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and audio modality respectively, and output video features and audio features each consisting of coding tokens and coding features; the cross-attention fusion module is used to fuse the coding features in the video features and audio features to obtain a fused coding feature, perform cross-attention modeling based on the fused coding feature, and perform feature decoding on the modeled video cross-attention coding features and audio cross-attention coding features, and output an audio and video reconstruction result; the contrastive learning module is used to fuse the coding tokens in the video features and audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix; a model training unit, configured to construct an audio and video reconstruction loss based on the audio and video reconstruction results output by the cross-attention fusion module, construct an audio and video comparative learning loss based on the audio and video similarity output by the comparative learning module, and train and optimize the deep fake detection model by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained; The forgery detection unit is used to input the audio and video sequence to be detected into the trained deep forgery detection model, extract the audio and video coding features to be detected according to the multimodal mask modeling module in the model, fuse the audio and video coding features to be detected according to the cross-attention fusion module and perform cross-attention modeling, output the audio and video cross-attention coding features to be detected, input the audio and video cross-attention coding features to be detected into the classifier for audio and video matching correlation detection, and output the forgery detection result of the audio and video sequence.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images