Deep forgery detection method, device and equipment based on cross-modal mask modeling

Abstract

The invention relates to a depth forgery detection method, device and equipment based on cross-modal mask modeling. The method comprises the following steps: constructing a deep forgery detection model comprising a multi-mode mask modeling module, a trans-attention fusion module and a contrast learning module; performing training optimization on the deep forgery detection model by integrating the audio and video reconstruction loss and the audio and video comparison learning loss; and inputting a to-be-detected audio and video sequence into the trained deep forgery detection model, outputting to-be-detected audio and video trans-attention coding features according to the model, carrying out audio and video matching correlation detection by inputting the to-be-detected audio and video trans-attention coding features into the classifier, and outputting a forgery detection result of the audio and video sequence. By adopting the method, the characterization of a real video can be learned by utilizing a mask modeling technology, and the characteristics of an audio-visual mode are subjected to cross fusion and comparative learning, so that the forgery detection performance is improved.

Description

Technical Field

[0001] The present application relates to the technical field of deep fake detection, and in particular to a deep fake detection method, apparatus and device based on cross-modal mask modeling. Background Art

[0002] Deep fake technology refers to the deceptive modification of a person's face in a video or image. This technology can produce high-fidelity videos that include fictional facial expressions, customized voice statements, or seamless face replacement in visual media. It has attracted widespread attention from various industries including entertainment, news, and art. There is an urgent need for extensive research on deep fake detection mechanisms to reduce the abuse of this technology.

[0003] Deepfake detection mechanisms aim to achieve accurate classification by identifying deceptive cues in videos or images. These techniques are generally divided into two categories: unimodal and multimodal methods. Traditional unimodal detection methods focus only on image-based content, carefully examining facial regions for artifacts, distortions, frequency changes, and other subtle spatial details that indicate manipulation. However, this approach is limited to static image data and has difficulty handling the dynamics of complex multimedia content. In recent years, work on deepfake detection using multimodal techniques has achieved remarkable results. When dealing with audio forgery scenarios, it is important to consider not only visual artifacts but also auditory cues.

[0004] Existing studies have developed modality-specific detection models for visual and auditory data separately, and subsequently merged these classification results to produce a unified output. However, these methods separate multimodal information and ignore the interrelationships between different modalities. The latest research focuses on extracting global correlations between modalities by fusing features from both modalities. For example, deep fake videos have huge differences in the time delay between audio and vision compared to real videos. An autoregressive model is used to fit the distribution of time delays between modes in real videos, and samples that exhibit abnormal time delays are subsequently labeled as deep fake videos. Although these methods exploit high-level semantic information from audio-visual pairs, they ignore the coupling of shallow details of heterogeneous modalities, which is crucial for multimodal deep fake detection.

[0005] Therefore, how to leverage the coupling of high-level semantic information from audio-visual pairs and shallow details from heterogeneous modalities to improve the performance of deepfake detection models remains a huge challenge. Summary of the invention

[0006] Based on this, it is necessary to provide a deep fake detection method, device and equipment based on cross-modal mask modeling to address the above-mentioned technical problems. This method uses mask modeling technology to learn the representation of real videos and cross-fuse and contrast the features of audiovisual modalities to improve the fake detection performance.

[0007] A deep fake detection method based on cross-modal mask modeling, the method comprising:

[0008] A deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module is constructed; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively performs segmentation, mask modeling, position encoding and feature encoding processing on the video modality and the audio modality, and outputs video features and audio features both consisting of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and the audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and the audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0009] The audio and video reconstruction loss is constructed according to the audio and video reconstruction results output by the cross-attention fusion module, and the audio and video comparative learning loss is constructed according to the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained.

[0010] The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed, and the cross-attention coding features of the audio and video to be detected are output. The cross-attention coding features of the audio and video to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

[0011] In one embodiment, the multimodal mask modeling module extracts the video mode and the audio mode in the audio and video sequence, and performs segmentation, mask modeling, position encoding and feature encoding processing on the video mode and the audio mode respectively, and outputs video features and audio features composed of encoding class tokens and encoding features, including:

[0012] Extract the video modality in the audio and video sequence, divide the video modality into video blocks according to the preset first step length and second step length, perform mask modeling for each video block, and perform tiling and position encoding on all video blocks with mask value 1 in turn to obtain the unmasked video features after position encoding, and use the unmasked video features after position encoding and the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output the video features including video coding class tokens and video coding features;

[0013] Extract the audio mode in the audio and video sequence and convert it into the corresponding Mel spectrogram. Divide the Mel spectrogram into unit blocks according to the preset third step length. Perform mask modeling for each unit block, and perform tiling and position encoding on all unit blocks with mask value 1 in turn to obtain the unmasked audio features after position encoding. The unmasked audio features after position encoding and the learnable embedding vector cls of the audio mode are combined. a The audio feature is input to the VIT-based encoder for audio feature encoding, and the audio feature including the audio coding class token and the audio coding feature is output.

[0014] In one embodiment, a video modality is extracted from an audio and video sequence, the video modality is divided into video blocks according to a preset first step length and a second step length, mask modeling is performed for each video block, and all video blocks with a mask value of 1 are tiled and position-encoded in turn to obtain unmasked video features after position encoding, and the unmasked video features after position encoding and the learnable embedding vector cls of the video modality are converted into the unmasked video features after position encoding. v The video is input to the VIT-based encoder for video feature encoding, and the output is a video feature containing video coding class tokens and video coding features, including:

[0015] Extract audio and video sequences The video mode V in ; where represents the real number space to which the audio and video sequence belongs, D represents the depth of the audio and video sequence, and H, W, and C correspond to the height, width, and number of channels of each frame, respectively;

[0016] According to the preset first step length τ 1 With the second step length τ 2 Split the video modality V into video blocks α d,h,w , expressed as

[0017]

[0018] Where D′, H′ and W′ represent the total number of steps of D, H and W respectively, and d, h and w represent the indexes of D, H and W respectively;

[0019] For each video block αd,h,w Perform mask modeling and transform each video block α d,h,w The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the video block corresponding to the mask is shielded. When the value is 1, it means that the video block corresponding to the mask is retained;

[0020] The PatchEmb3D algorithm is used to tile all video blocks with a mask value of 1 to obtain the unmasked video feature Ω. v , expressed as

[0021]

[0022] For the unmasked video feature Ω v Perform position encoding to obtain the unmasked video features after position encoding Expressed as

[0023]

[0024] in, represents the position encoding of each position pair (m,n) in the unmasked video feature, represents the unmasked video feature of each position pair (m,n), Sincospos represents the sinusoidal position encoding, m∈M and n∈N, M and N represent the height and width of the unmasked video feature respectively;

[0025] The unmasked video features after position encoding and the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output contains the video coding class token CLS v Video coding features v The video features are expressed as

[0026]

[0027] Among them, ViTEncoder represents the encoder based on VIT.

[0028] In one embodiment, the audio mode in the audio and video sequence is extracted and converted into a corresponding mel-spectrogram, the mel-spectrogram is divided into unit blocks according to a preset third step length, mask modeling is performed for each unit block, and all unit blocks with a mask value of 1 are tiled and position-encoded in turn to obtain unmasked audio features after position encoding, and the unmasked audio features after position encoding and the learnable embedding vector cls of the audio mode are combined. aThe audio feature is input to the VIT-based encoder for audio feature encoding, and the audio features including audio coding class tokens and audio coding features are output, including:

[0029] Extract the audio mode A from the audio and video sequence and convert the audio mode A into the corresponding Mel spectrogram A mel , and according to the preset third step length τ 3 Mel-spectrogram A mel Divide into unit blocks Mel spectrum A mel and unit blocks Respectively expressed as

[0030]

[0031] Among them, Mel_spectrogram represents Mel spectrogram conversion, h and w represent the indexes of H and W respectively, Represents the real number space to which the audio and video sequence belongs, where H and W correspond to the height and width of each frame respectively;

[0032] For each unit block Perform mask modeling and transform each unit block The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the unit block corresponding to the mask is masked. When it is 1, it means that the unit block corresponding to the mask is retained;

[0033] The PatchEmb2D algorithm is used to tile all the unit blocks with a mask value of 1 to obtain the unmasked audio features. Expressed as

[0034]

[0035] Unmasked audio features Perform position encoding to obtain the unmasked audio features after position encoding Expressed as

[0036]

[0037] in, Represents the position encoding of the unmasked audio feature, Sincospos represents the sinusoidal position encoding;

[0038] Unmasked audio features after position encoding and the learnable embedding vector cls of the audio modality a Input to the VIT-based encoder for audio feature encoding, and the output includes the audio coding class token CLSa With audio coding features γ a The audio features are expressed as

[0039]

[0040] Among them, ViTEncoder represents the encoder based on VIT.

[0041] In one embodiment, the cross-attention fusion module fuses the coding features in the video features and the audio features to obtain a fused coding feature, and performs cross-attention modeling based on the fused coding feature, including:

[0042] By fusing the video coding features in the video features v Audio encoding features in audio features γ a , and obtain the fusion coding feature β va , denoted as β va =concat(ξ v ,γ a ), where concat represents fusion processing;

[0043] Based on the fusion coding feature β va Perform cross-attention modeling to obtain the video cross-attention encoding feature Φ v and audio cross-attention encoding feature Φ a , respectively expressed as

[0044]

[0045] Among them, Q, K, V represent the query vector, key vector and value vector respectively, and W Q , W K and W V They represent the query vector, key vector and value vector after the conversion by the attention mechanism, Softmax represents the normalized exponential function, and the superscript T represents the transposition operation.

[0046] In one embodiment, the cross-attention fusion module decodes the video cross-attention coding features and the audio cross-attention coding features obtained by the cross-attention modeling, and outputs the audio and video reconstruction results, including:

[0047] The VIT-based decoder is used to convert the video cross-attention encoding feature Φ obtained by cross-attention modeling v and audio cross-attention encoding feature Φ a Perform feature decoding respectively to obtain the video reconstruction result And the audio reconstruction results Respectively expressed as

[0048]

[0049] Among them, VitDecoder represents a decoder based on VIT.

[0050] In one embodiment, the contrastive learning module fuses the encoding tokens in the video features and the audio features to obtain a mutual information matrix, and calculates the audio and video similarity based on the mutual information matrix, including:

[0051] Video Coding Class Tokens (CLS) by Fusion of Video Features v The audio coding class token CLS in the audio feature a , and get the mutual information matrix Ψ N×N , expressed as

[0052]

[0053] Among them, the superscript T represents the transposition operation, and N represents the width of the unmasked video feature;

[0054] The mutual information matrix Ψ N×N The diagonal elements of are accumulated to obtain the audio and video similarity Similar(V,A), which is expressed as

[0055]

[0056] Among them, V and A represent the video mode and audio mode respectively, Ψ i,j represents the i-th row and j-th column element in the mutual information matrix.

[0057] In one embodiment, an audio and video reconstruction loss is constructed according to the audio and video reconstruction results output by the cross-attention fusion module, an audio and video comparative learning loss is constructed according to the audio and video similarity output by the comparative learning module, and the deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained, including:

[0058] Construct the audio and video reconstruction loss based on the video reconstruction results and audio reconstruction results output by the cross-attention fusion module Expressed as

[0059]

[0060] Among them, Loss v Represents the video reconstruction loss, Loss a represents the audio reconstruction loss, MAE represents the masked autoencoder, and Represent the video reconstruction results and audio reconstruction results respectively, Represents the video block α d,h,w The corresponding mask, Represents a unit block The corresponding mask;

[0061] Construct the audio and video contrast learning loss according to the audio and video similarity Similar(V,A) output by the contrast learning module Expressed as Among them, V and A represent the video mode and audio mode respectively;

[0062] The final loss is constructed by combining audio and video reconstruction loss and audio and video contrast learning loss Expressed as Among them, λ represents the balance and The weight factor of

[0063] Based on the final loss The deep fake detection model is trained and optimized until a trained deep fake detection model is obtained.

[0064] A deep fake detection device based on cross-modal mask modeling, the device comprising:

[0065] A model construction unit, for constructing a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and the audio modality, and output video features and audio features both consisting of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and the audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the video cross-attention encoding features and the audio cross-attention encoding features obtained by the modeling, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and the audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0066] A model training unit, used to construct an audio and video reconstruction loss according to the audio and video reconstruction results output by the cross-attention fusion module, construct an audio and video comparative learning loss according to the audio and video similarity output by the comparative learning module, and train and optimize the deep fake detection model by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained;

[0067] The forgery detection unit is used to input the audio and video sequence to be detected into the trained deep forgery detection model, extract the audio and video coding features to be detected according to the multimodal mask modeling module in the model, fuse the audio and video coding features to be detected according to the cross-attention fusion module and perform cross-attention modeling, output the audio and video cross-attention coding features to be detected, input the audio and video cross-attention coding features to be detected into the classifier to perform audio and video matching related detection, and output the forgery detection result of the audio and video sequence.

[0068] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0069] A deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module is constructed; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively performs segmentation, mask modeling, position encoding and feature encoding processing on the video modality and the audio modality, and outputs video features and audio features both consisting of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and the audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and the audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0070] The audio and video reconstruction loss is constructed according to the audio and video reconstruction results output by the cross-attention fusion module, and the audio and video comparative learning loss is constructed according to the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained.

[0071] The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed, and the cross-attention coding features of the audio and video to be detected are output. The cross-attention coding features of the audio and video to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

[0072] The above-mentioned deep fake detection method, device and equipment based on cross-modal mask modeling perform fake detection by constructing and training a deep fake detection model. Among them, the multimodal mask modeling module based on the model can adopt the cross-modal mask modeling method to utilize the intrinsic interaction of heterogeneous modalities to reconstruct video and audio features, enhance multimodal representation and reduce overfitting tendency, laying a solid data foundation for fake detection; the cross-attention fusion module based on the model can locally and finely fuse the shallow features of two heterogeneous modalities, promote the seamless connection and complementarity of information between heterogeneous modalities, and further enhance the model's sensitivity and recognition ability to fake content; the contrastive learning module based on the model can adopt the audio-visual contrastive learning strategy to align video features and audio features globally, and strengthen the consistency check between features. This strategy not only improves the model's global recognition ability of fake content, but also enhances the generalization performance of the model under different fake techniques, ensuring the stability and reliability of fake detection results. This application constructs an efficient, accurate and robust deep fake detection model through multimodal mask modeling, cross-attention fusion and contrastive learning, providing strong technical support for coping with increasingly complex fake challenges. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of a process of a deep fake detection method based on cross-modal mask modeling in one embodiment;

[0074] Figure 2 A schematic diagram of the structure of a deep fake detection model in one embodiment;

[0075] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0077] In one embodiment, Figure 1 As shown, a deep fake detection method based on cross-modal mask modeling is provided, comprising the following steps:

[0078] Step S1, constructing a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and the audio modality, and output video features and audio features composed of coding tokens and coding features; the cross-attention fusion module is used to fuse the coding features in the video features and the audio features to obtain fused coding features, perform cross-attention modeling based on the fused coding features, and feature decode the video cross-attention coding features and audio cross-attention coding features obtained by the modeling, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the coding tokens in the video features and the audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix.

[0079] The structure of the deep fake detection model is as follows Figure 2 The workflow of the multimodal mask modeling module in the model includes: first, extract the video modality in the audio and video sequence, divide the video modality into video blocks according to the preset first step length and second step length, perform mask modeling for each video block, and perform tiling and position encoding on all video blocks with mask values ​​of 1 in turn to obtain the unmasked video features after position encoding, and embed the unmasked video features after position encoding and the learnable embedding vector cls of the video modality v The video feature is input to the VIT-based encoder for video feature encoding, and the video feature containing the video coding class token and the video coding feature is output. Then, the audio mode in the audio and video sequence is extracted and converted into the corresponding Mel spectrogram. The Mel spectrogram is divided into unit blocks according to the preset third step length. Mask modeling is performed for each unit block, and all unit blocks with mask values ​​1 are tiled and position encoded in turn to obtain the unmasked audio feature after position encoding. The unmasked audio feature after position encoding and the learnable embedding vector cls of the audio mode are combined. a The audio feature is input to the VIT-based encoder for audio feature encoding, and the audio feature including the audio coding class token and the audio coding feature is output.

[0080] Among them, the specific workflow of the multimodal mask modeling module for video modality includes:

[0081] Extract audio and video sequences The video mode V in ; where represents the real number space to which the audio and video sequence belongs, D represents the depth of the audio and video sequence, and H, W, and C correspond to the height, width, and number of channels of each frame, respectively;

[0082] According to the preset first step length τ 1 With the second step length τ2 Split the video modality V into video blocks α d,h,w , expressed as

[0083]

[0084]

[0085] Where D′, H′ and W′ represent the total number of steps of D, H and W respectively, d, h and w represent the indexes of D, H and W respectively; specifically, the first step length τ 1 With the second step length τ 2 Set to 2 and 16 respectively;

[0086] For each video block α d,h,w Perform mask modeling and transform each video block α d,h,w The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the video block corresponding to the mask is shielded. When the value is 1, it means that the video block corresponding to the mask is retained;

[0087] The PatchEmb3D (three-dimensional patch embedding) algorithm is used to tile all video blocks with a mask value of 1 to obtain the unmasked video feature Ω v (Ω v is a two-dimensional matrix), expressed as

[0088]

[0089] For the unmasked video feature Ω v Perform position encoding to obtain the unmasked video features after position encoding Expressed as

[0090]

[0091] in, represents the position encoding of each position pair (m,n) in the unmasked video feature, represents the unmasked video feature of each position pair (m,n), Sincospos represents the sinusoidal position encoding, m∈M and n∈N, M and N represent the height and width of the unmasked video feature respectively;

[0092] The unmasked video features after position encoding and the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output contains the video coding class token CLS v Video coding featuresv The video features are expressed as

[0093]

[0094] Among them, ViTEncoder represents the encoder based on VIT.

[0095] Among them, the specific workflow of the multimodal mask modeling module for the audio modality includes:

[0096] Extract the audio mode A from the audio and video sequence and convert the audio mode A into the corresponding Mel spectrogram A mel , and according to the preset third step length τ 3 Mel-spectrogram A mel Divide into unit blocks Mel spectrum A mel and unit blocks are represented as

[0097]

[0098] Among them, Mel_spectrogram represents Mel spectrogram conversion, h and w represent the indexes of H and W respectively, Represents the real number space to which the audio and video sequence belongs, H and W correspond to the height and width of each frame respectively. width;

[0099] For each unit block Perform mask modeling and transform each unit block The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the unit block corresponding to the mask is masked. When it is 1, it means that the unit block corresponding to the mask is retained;

[0100] The PatchEmb2D (two-dimensional patch embedding) algorithm is used to tile all unit blocks with a mask value of 1 to obtain unmasked audio features. Expressed as

[0101]

[0102] Unmasked audio features Perform position encoding to obtain the unmasked audio features after position encoding Expressed as

[0103]

[0104] in, Represents the position encoding of the unmasked audio feature, Sincospos represents the sinusoidal position encoding;

[0105] Unmasked audio features after position encoding and the learnable embedding vector cls of the audio modality a Input to the VIT-based encoder for audio feature encoding, and the output includes the audio coding class token CLS a With audio coding features γ a The audio features are expressed as

[0106]

[0107] Among them, ViTEncoder represents the encoder based on VIT.

[0108] Specifically, the VIT-based encoder is a visual encoder based on the Transformer model, which includes two normalization layers, a multi-head attention mechanism layer, and a multi-layer perceptron (MLP).

[0109] The specific workflow of the cross-attention fusion module in the model includes:

[0110] By fusing the video coding features in the video features v Audio encoding features in audio features γ a , and obtain the fusion coding feature β va , denoted as β va =concat(ξ v ,γ a ), where concat represents fusion processing;

[0111] Based on the fusion coding feature β va Perform cross-attention modeling to obtain the video cross-attention encoding feature Φ v and audio cross-attention encoding feature Φ a , respectively expressed as

[0112]

[0113] Among them, Q, K, V represent the query vector, key vector and value vector respectively, and W Q , W K and W V They represent the query vector, key vector, and value vector after the attention mechanism conversion, Softmax represents the normalized exponential function, and the superscript T represents the transposition operation;

[0114] The VIT-based decoder is used to convert the video cross-attention encoding feature Φ obtained by cross-attention modeling v and audio cross-attention encoding feature Φ a Perform feature decoding respectively to obtain the video reconstruction result And the audio reconstruction results Respectively expressed as

[0115]

[0116] Among them, VitDecoder represents a VIT-based decoder. The structure of the VIT-based decoder is consistent with the structure of the VIT-based encoder.

[0117] The specific workflow of the contrastive learning module in the model includes:

[0118] Video Coding Class Tokens (CLS) by Fusion of Video Features v The audio coding class token CLS in the audio feature a , and get the mutual information matrix Ψ N×N , expressed as

[0119]

[0120] Among them, the superscript T represents the transposition operation, and N represents the width of the unmasked video feature;

[0121] The mutual information matrix Ψ N×N The diagonal elements of are accumulated to obtain the audio and video similarity Similar(V,A), which is expressed as

[0122]

[0123] Among them, V and A represent the video mode and audio mode respectively, Ψ i,j represents the i-th row and j-th column element in the mutual information matrix.

[0124] Step S2: construct an audio and video reconstruction loss based on the audio and video reconstruction results output by the cross-attention fusion module, construct an audio and video comparative learning loss based on the audio and video similarity output by the comparative learning module, and train and optimize the deep fake detection model by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained.

[0125] The specific implementation process of step S2 includes:

[0126] Construct the audio and video reconstruction loss based on the video reconstruction results and audio reconstruction results output by the cross-attention fusion module Expressed as

[0127]

[0128] Among them, Loss v Represents the video reconstruction loss, Loss a represents the audio reconstruction loss, MAE represents the masked autoencoder, and Represent the video reconstruction results and audio reconstruction results respectively, Represents the video block α d,h,w The corresponding mask, Represents a unit block The corresponding mask;

[0129] Construct the audio and video contrast learning loss according to the audio and video similarity Similar(V,A) output by the contrast learning module Expressed as Among them, V and A represent the video mode and audio mode respectively;

[0130] The final loss is constructed by combining audio and video reconstruction loss and audio and video contrast learning loss Expressed as Among them, λ represents the balance and The weight factor of

[0131] Based on the final loss The deep fake detection model is trained and optimized until a trained deep fake detection model is obtained.

[0132] Step S3: input the audio and video sequence to be detected into the trained deep fake detection model, extract the audio and video coding features to be detected according to the multimodal mask modeling module in the model, fuse the audio and video coding features to be detected according to the cross-attention fusion module and perform cross-attention modeling, output the audio and video cross-attention coding features to be detected, input the audio and video cross-attention coding features to be detected into the classifier to perform audio and video matching related detection, and output the forgery detection result of the audio and video sequence.

[0133] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0134] In one embodiment, a deep fake detection device based on cross-modal mask modeling is provided, comprising:

[0135] A model construction unit, for constructing a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively perform segmentation, mask modeling, position encoding and feature encoding processing on the video modality and the audio modality, and output video features and audio features both consisting of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and the audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the video cross-attention encoding features and the audio cross-attention encoding features obtained by the modeling, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and the audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0136] A model training unit, used to construct an audio and video reconstruction loss according to the audio and video reconstruction results output by the cross-attention fusion module, construct an audio and video comparative learning loss according to the audio and video similarity output by the comparative learning module, and train and optimize the deep fake detection model by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained;

[0137] The forgery detection unit is used to input the audio and video sequence to be detected into the trained deep forgery detection model, extract the audio and video coding features to be detected according to the multimodal mask modeling module in the model, fuse the audio and video coding features to be detected according to the cross-attention fusion module and perform cross-attention modeling, output the audio and video cross-attention coding features to be detected, input the audio and video cross-attention coding features to be detected into the classifier to perform audio and video matching related detection, and output the forgery detection result of the audio and video sequence.

[0138] For the specific limitations of the deep fake detection device based on cross-modal mask modeling, please refer to the limitations of the deep fake detection method based on cross-modal mask modeling above, which will not be repeated here. Each module in the above-mentioned deep fake detection device based on cross-modal mask modeling can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0139] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a deep counterfeit detection method based on cross-modal mask modeling is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0140] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0142] A deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module is constructed; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively performs segmentation, mask modeling, position encoding and feature encoding processing on the video modality and the audio modality, and outputs video features and audio features both consisting of encoding tokens and encoding features; the cross-attention fusion module is used to fuse the encoding features in the video features and the audio features to obtain fused encoding features, perform cross-attention modeling based on the fused encoding features, and perform feature decoding on the modeled video cross-attention encoding features and audio cross-attention encoding features, and output the audio and video reconstruction results; the contrastive learning module is used to fuse the encoding tokens in the video features and the audio features to obtain the mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix;

[0143] The audio and video reconstruction loss is constructed according to the audio and video reconstruction results output by the cross-attention fusion module, and the audio and video comparative learning loss is constructed according to the audio and video similarity output by the comparative learning module. The deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained.

[0144] The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed, and the cross-attention coding features of the audio and video to be detected are output. The cross-attention coding features of the audio and video to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements may be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A deep fake detection method based on cross-modal mask modeling, characterized in that: The method comprises: A deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module is constructed; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively performs segmentation, mask modeling, position coding and feature coding processing on the video modality and the audio modality, and outputs video features and audio features both consisting of coding tokens and coding features; the cross-attention fusion module is used to fuse the coding features in the video features and the audio features to obtain fused coding features, perform cross-attention modeling based on the fused coding features, and perform feature decoding on the modeled video cross-attention coding features and audio cross-attention coding features, and output audio and video reconstruction results; the contrastive learning module is used to fuse the coding tokens in the video features and the audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix; Constructing an audio and video reconstruction loss according to the audio and video reconstruction result output by the cross-attention fusion module, constructing an audio and video comparative learning loss according to the audio and video similarity output by the comparative learning module, and training and optimizing the deep fake detection model by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained; The audio and video sequence to be detected is input into the trained deep fake detection model, and the audio and video coding features to be detected are extracted according to the multimodal mask modeling module in the model. The audio and video coding features to be detected are fused according to the cross-attention fusion module and cross-attention modeling is performed, and the audio and video cross-attention coding features to be detected are output. The audio and video cross-attention coding features to be detected are input into the classifier for audio and video matching related detection, and the forgery detection result of the audio and video sequence is output.

2. The method according to claim 1, characterized in that The multimodal mask modeling module extracts the video mode and the audio mode in the audio and video sequence, and performs segmentation, mask modeling, position coding and feature coding processing on the video mode and the audio mode respectively, and outputs video features and audio features composed of coding class tokens and coding features, including: Extract the video modality in the audio and video sequence, divide the video modality into video blocks according to the preset first step length and second step length, perform mask modeling for each video block, and perform tiling and position encoding on all video blocks with mask value 1 in turn to obtain the unmasked video features after position encoding, and embed the unmasked video features after position encoding and the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output the video features including video coding class tokens and video coding features; Extract the audio mode in the audio and video sequence and convert it into the corresponding Mel spectrogram, divide the Mel spectrogram into unit blocks according to the preset third step length, perform mask modeling for each unit block, and perform tiling and position encoding on all unit blocks with mask value 1 in turn to obtain the unmasked audio features after position encoding, and embed the unmasked audio features after position encoding and the learnable embedding vector cls of the audio mode a The audio feature is input to the VIT-based encoder for audio feature encoding, and the audio feature including the audio coding class token and the audio coding feature is output.

3. The method according to claim 2, characterized in that Extract the video modality in the audio and video sequence, divide the video modality into video blocks according to the preset first step length and second step length, perform mask modeling for each video block, and perform tiling and position encoding on all video blocks with mask value 1 in turn to obtain the unmasked video features after position encoding, and embed the unmasked video features after position encoding and the learnable embedding vector cls of the video modality v The video is input to the VIT-based encoder for video feature encoding, and the output is a video feature containing video coding class tokens and video coding features, including: Extract audio and video sequences The video mode V in ; where represents the real number space to which the audio and video sequence belongs, D represents the depth of the audio and video sequence, and H, W, and C correspond to the height, width, and number of channels of each frame, respectively; The video modality V is divided into video blocks α according to the preset first step length τ1 and second step length τ2 d,h,w , expressed as Where D′, H′ and W′ represent the total number of steps of D, H and W respectively, and d, h and w represent the indexes of D, H and W respectively; For each video block α d,h,w Perform mask modeling and transform each video block α d,h,w The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the video block corresponding to the mask is shielded. When the value is 1, it means that the video block corresponding to the mask is retained; The PatchEmb3D algorithm is used to tile all video blocks with a mask value of 1 to obtain the unmasked video feature Ω. v , expressed as For the unmasked video feature Ω v Perform position encoding to obtain the unmasked video features after position encoding Expressed as in, represents the position encoding of each position pair (m,n) in the unmasked video feature, represents the unmasked video feature of each position pair (m,n), Sincospos represents the sinusoidal position encoding, m∈M and n∈N, M and N represent the height and width of the unmasked video feature respectively; The unmasked video features after encoding the position and the learnable embedding vector cls of the video modality v Input to the VIT-based encoder for video feature encoding, and output contains the video coding class token CLS v Video coding features v The video features are expressed as Among them, ViTEncoder represents the encoder based on VIT.

4. The method according to claim 2, characterized in that: Extract the audio mode in the audio and video sequence and convert it into the corresponding Mel spectrogram, divide the Mel spectrogram into unit blocks according to the preset third step length, perform mask modeling for each unit block, and perform tiling and position encoding on all unit blocks with mask value 1 in turn to obtain the unmasked audio features after position encoding, and embed the unmasked audio features after position encoding and the learnable embedding vector cls of the audio mode a The audio feature is input to the VIT-based encoder for audio feature encoding, and the audio features including audio coding class tokens and audio coding features are output, including: Extract the audio mode A from the audio and video sequence and convert the audio mode A into the corresponding Mel spectrogram A mel , and according to the preset third step length τ3, the Mel-spectrogram A mel Divide into unit blocks The mel-spectrogram A mel and unit blocks Respectively expressed as Among them, Mel_spectrogram represents Mel spectrogram conversion, h and w represent the indexes of H and W respectively, Represents the real number space to which the audio and video sequence belongs, where H and W correspond to the height and width of each frame respectively; For each unit block Perform mask modeling and transform each unit block The corresponding mask is specified as in, The value is 0 or 1. When it is 0, it means that the unit block corresponding to the mask is masked. When it is 1, it means that the unit block corresponding to the mask is retained; The PatchEmb2D algorithm is used to tile all the unit blocks with a mask value of 1 to obtain the unmasked audio features. Expressed as For the unmasked audio features Perform position encoding to obtain the unmasked audio features after position encoding Expressed as in, Represents the position encoding of the unmasked audio feature, Sincospos represents the sinusoidal position encoding; The unmasked audio features after encoding the position and the learnable embedding vector cls of the audio modality a Input to the VIT-based encoder for audio feature encoding, and the output includes the audio coding class token CLS a With audio coding features γ a The audio features are expressed as Among them, ViTEncoder represents the encoder based on VIT.

5. The method according to claim 1, characterized in that The cross-attention fusion module fuses the coding features in the video features and the audio features to obtain a fused coding feature, and performs cross-attention modeling based on the fused coding feature, including: By fusing the video coding feature ξ in the video feature v The audio coding feature γ a , and obtain the fusion coding feature β va , denoted as β va =concat(ξ v ,γ a ), where concat represents fusion processing; Based on the fusion coding feature β va Perform cross-attention modeling to obtain the video cross-attention encoding feature Φ v and audio cross-attention encoding feature Φ a , respectively expressed as Among them, Q, K, V represent the query vector, key vector and value vector respectively, and W Q , W K and W V They represent the query vector, key vector and value vector after the conversion by the attention mechanism, Softmax represents the normalized exponential function, and the superscript T represents the transposition operation.

6. The method according to claim 5, characterized in that The cross-attention fusion module decodes the video cross-attention coding features and the audio cross-attention coding features obtained by the cross-attention modeling, and outputs the audio and video reconstruction results, including: The VIT-based decoder is used to convert the video cross-attention encoding feature Φ obtained by cross-attention modeling v and audio cross-attention encoding feature Φ a Perform feature decoding respectively to obtain the video reconstruction result l V And the audio reconstruction result l a , respectively expressed as Among them, VitDecoder represents a decoder based on VIT.

7. The method according to claim 1, characterized in that The contrastive learning module fuses the coding tokens in the video features and the audio features to obtain a mutual information matrix, and calculates the audio and video similarity based on the mutual information matrix, including: By integrating the video features, the video coding class token CLS v The audio coding class token CLS in the audio feature a , and get the mutual information matrix Ψ N×N , expressed as Among them, the superscript T represents the transposition operation, and N represents the width of the unmasked video feature; The mutual information matrix Ψ N×N The diagonal elements of are accumulated to obtain the audio and video similarity Similar(V,A), which is expressed as Among them, V and A represent the video mode and audio mode respectively, Ψ i,j represents the i-th row and j-th column element in the mutual information matrix.

8. The method according to claim 1, characterized in that The audio and video reconstruction loss is constructed according to the audio and video reconstruction result output by the cross-attention fusion module, the audio and video contrast learning loss is constructed according to the audio and video similarity output by the contrast learning module, and the deep fake detection model is trained and optimized by combining the audio and video reconstruction loss and the audio and video contrast learning loss until a trained deep fake detection model is obtained, including: The audio and video reconstruction loss is constructed according to the video reconstruction result and the audio reconstruction result output by the cross-attention fusion module Expressed as Among them, Loss v Represents the video reconstruction loss, Loss a represents the audio reconstruction loss, MAE represents the masked autoencoder, l V and l a Represent the video reconstruction results and audio reconstruction results respectively, Represents the video block α d,h,w The corresponding mask, Represents a unit block The corresponding mask; The audio and video contrast learning loss is constructed based on the audio and video similarity Similar(V,A) output by the contrast learning module. Expressed as Among them, V and A represent the video mode and audio mode respectively; The final loss is constructed by combining the audio and video reconstruction loss and the audio and video contrast learning loss. Expressed as Among them, λ represents the balance and The weight factor of Based on the final loss The deep fake detection model is trained and optimized until a trained deep fake detection model is obtained.

9. A deep fake detection device based on cross-modal mask modeling, characterized in that: The device comprises: A model construction unit, for constructing a deep fake detection model including a multimodal mask modeling module, a cross-attention fusion module and a contrastive learning module; wherein the multimodal mask modeling module is used to extract the video modality and the audio modality in the audio and video sequence, and respectively perform segmentation, mask modeling, position coding and feature coding processing on the video modality and the audio modality in turn, and output video features and audio features both consisting of coding tokens and coding features; the cross-attention fusion module is used to fuse the coding features in the video features and the audio features to obtain fused coding features, perform cross-attention modeling based on the fused coding features, and perform feature decoding on the modeled video cross-attention coding features and audio cross-attention coding features, and output audio and video reconstruction results; the contrastive learning module is used to fuse the coding tokens in the video features and the audio features to obtain a mutual information matrix, and calculate the audio and video similarity based on the mutual information matrix; A model training unit, configured to construct an audio and video reconstruction loss according to the audio and video reconstruction result output by the cross-attention fusion module, construct an audio and video comparative learning loss according to the audio and video similarity output by the comparative learning module, and train and optimize the deep fake detection model by combining the audio and video reconstruction loss and the audio and video comparative learning loss until a trained deep fake detection model is obtained; The forgery detection unit is used to input the audio and video sequence to be detected into the trained deep forgery detection model, extract the audio and video coding features to be detected according to the multimodal mask modeling module in the model, fuse the audio and video coding features to be detected according to the cross-attention fusion module and perform cross-attention modeling, output the audio and video cross-attention coding features to be detected, input the audio and video cross-attention coding features to be detected into the classifier to perform audio and video matching related detection, and output the forgery detection result of the audio and video sequence.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images