Multi-mode depth forgery detection method and system

Abstract

The invention discloses a multi-modal deep forgery detection method and system, relates to the technical field of forgery detection, and solves the technical problems of unbiased emphasis and non-negligible modal information conflict during multi-modal fusion. The method comprises a multi-modal deep counterfeiting detection model, the model extracts video modal features and audio modal features from video data and audio data respectively, then deep extraction is carried out on the video modal features and the audio modal features respectively, and fusion is carried out by using a fusion weight. And finally, obtaining a prediction result according to the fused features. When the multi-modal deep forgery detection model is trained, adaptive modal contrast learning is carried out based on the audio modal features and the video modal features to obtain adaptive modal contrast loss; according to the method, the orthogonal constraint is introduced, so that the gradients of different modes are ensured not to interfere with each other during updating, and the diversity and complementarity of mode fusion are improved while the conflict between the single-mode gradient and the multi-mode gradient is solved.

Description

Technical Field

[0001] The present invention relates to the field of forgery detection technology, and in particular to a multimodal deep forgery detection method and system. Background Art

[0002] In recent years, the rapid development of computer vision and deep learning has made it possible to create complex deepfake videos. Multimodal audio and video deepfakes, in particular, are becoming increasingly rampant. The rapid spread of audio and video deepfakes online not only infringes on personal privacy but can also pose social security risks.

[0003] However, the modal fusion of some existing multimodal deep fake detection methods is still too simple. They usually adopt a linear splicing method that ignores importance distribution and equivalently fuses the features of each modality. This strategy fails to fully explore and utilize the potential complementary information between each single modality, and also lacks the use of deep information.

[0004] During the fusion process, the lack of in-depth feature interaction and cross-modal relationship modeling may lead to a bias towards one modality, which in turn affects the overall detection performance, resulting in significant pain points in areas such as network security, identity authentication, and media content review. Summary of the Invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a multimodal deep fake detection method and system to solve the technical problems of biased emphasis and non-negligible modal information conflict during multimodal fusion.

[0006] A multimodal deep fake detection method includes: a multimodal deep fake detection model, wherein the model extracts video modal features and audio modal features from video data and audio data, respectively, then performs deep extraction on the video modal features and audio modal features, and then fuses them using fusion weights, and finally obtains a prediction result based on the fused features; when training the multimodal deep fake detection model, adaptive modal contrast learning is performed based on the audio modal features and video modal features to obtain an adaptive modal contrast loss, and adaptive temperature parameters and fusion weights are designed.

[0007] Furthermore, the adaptive modality contrastive learning includes inter-modality contrastive learning and intra-modality contrastive learning, and the adaptive modality contrastive loss is expressed as follows:

[0008]

[0009] Where, L MAC is the adaptive modality contrast loss, L va is the contrastive learning loss from video to audio, L av is the audio-to-video contrastive learning loss, Lvv is the video-to-video contrastive learning loss, L aa For audio to audio comparison; L av The expression is as follows. Other contrastive learning loss expressions are similar:

[0010]

[0011]

[0012] Where, E p(A,V) is the expected value, is the unnormalized similarity score of the ith positive pair, is the unnormalized similarity score between the i-th positive pair and the k-th negative pair, τ is the temperature parameter, is the feature representation of the i-th audio sample, is the feature representation of the i-th video sample, is the feature representation of the kth negative sample video, and m is the margin.

[0013] Furthermore, the temperature parameter is defined by combining Bayesian estimation and attention mechanism, and is expressed as follows:

[0014] τ t =γ t ·τ t-1 +(1-γ t )·τ new

[0015] γ t =σ(MLP([h t-1 ,Δτ t-1 ,grad*τL*t-1]))

[0016]

[0017] Where τ is the temperature parameter, τ0 is the temperature parameter of the historical state, γ t is the timing adjustment factor, φ k (S) is a set of basis functions, σ is the Sigmoid function, MLP is the multi-layer perceptron, h t-1 is the historical state code, Δτ t-1 is the temperature change, grad is the gradient, L is the loss function, τ t-1 is the temperature at time t-1, τ new is the new temperature, β k is a hyperparameter.

[0018] Furthermore, the fusion weight is formed using clustering information, and the expression is as follows:

[0019]

[0020] Let m∈v,a, where v represents the video mode, a represents the audio mode, γ is a hyperparameter, and MHAttention m It is a multi-head attention mechanism. For the comprehensive distance metric, the expression is as follows:

[0021]

[0022] Where, is the distance, k∈a,v; Effectively measure the true distance between the sample and the cluster center, is the latent space feature, is the cluster center, β is the trade-off parameter, and is the maximum distance of the corresponding mode, used for normalization.

[0023] At the same time, it captures the statistical distribution differences and directional differences between samples and cluster centers. By introducing a trade-off parameter β, it adjusts the contribution of these two distances in measuring the statistical distance of samples from cluster centers and the directional similarity in feature space, thereby obtaining a more comprehensive evaluation of the relationship between samples and cluster centers.

[0024] Furthermore, the multimodal deepfake detection model includes a multi-scale video feature extractor, which performs deep extraction of the video modality features to capture long-range dependencies and multi-scale information. The expression is as follows:

[0025]

[0026]

[0027] Where, is the video modality feature, N is the normalized feature, LN is the normalization operation, λ1 and λ2 are learnable scaling factors, f1, f2, f3, f4, f5, f6 are convolution operations, and MSTLKA is the multi-scale spatiotemporal attention block, which is expressed as follows:

[0028]

[0029] Where, X i Characterized by G i is a gating mechanism, and STLKA is a spatiotemporal attention block, which is used to deeply extract video features. The expression is as follows:

[0030] STLKA(X)=f PW (f DWD (f DW (X)))

[0031] Where, f DW (·) is a 3D depthwise convolution with a kernel size of (2d-1)×(2d-1)×(2d-1); f DWD (·) is the kernel size used 3D dilated depth convolution; f PW (·) is a point convolution used to reduce the number of channels; GSAU-video is a video gated spatial attention unit that dynamically adjusts the weights of attention at different scales. The expression is as follows:

[0032]

[0033] Where X and Y are input features.

[0034] Furthermore, the multimodal deep fake detection model includes a multi-scale audio feature extractor, which performs deep extraction of the audio modality features, as expressed as follows:

[0035]

[0036] Where, is the audio modal feature, N is the normalized feature, LN is the normalization operation, λ1 and λ2 are learnable scaling factors, f1, f2, f3, f4, f5, f6 are convolution operations, and MFTLKA is the multi-scale frequency-time large kernel attention, which is expressed as follows:

[0037]

[0038] Where, X i Characterized by G i is a gating mechanism, and FTLKA is a frequency-time large kernel attention, which is expressed as follows:

[0039] FTLKA(X)=f PW (f DWD (f DW (X)))

[0040] Where, f DW (·): 2D depth convolution in time and frequency; f DWD (·) is the depth-expanded convolution to capture a larger receptive field; f PW (·) is the point convolution to reduce the number of channels; GSAU-audio is the gated frequency-time attention unit, which dynamically adjusts the attention of different scales through frequency and time gating mechanisms to enhance the capture of various frequency components and time patterns. The expression is as follows:

[0041]

[0042] Where X and Y are input features.

[0043] Furthermore, the fused features are expressed as follows:

[0044]

[0045] Where, is the fused feature, is the video weight, is the audio weight, V i-1 is the video feature, A i-1 is the audio feature;

[0046] Using fusion features Guide deep feature learning, the fusion of complementary modal information and global information:

[0047]

[0048] Among them, K∈V,A.

[0049] Furthermore, the multimodal deepfake detection model includes a video encoder and an audio encoder. The video encoder processes video data V, including dividing the video into blocks according to given video and audio pairs (V, A) to P; and then encodes the video to obtain global video features, namely video modality features;

[0050] The audio encoder processes the audio data A, including converting the audio signal A into a spectrogram S(t,f), calculating the spectrogram S(t,f) to obtain the Mel-spectrogram M(t,m), taking the logarithm of the Mel-spectrogram M(t,m) to obtain logM(t,m), and finally encoding the logarithmic Mel-frequency spectrum to obtain the global audio features, namely the audio modal features.

[0051] Furthermore, the training of the multimodal deepfake detection model also includes an orthogonalized Pareto optimization stage, specifically:

[0052] Stage 1: Identifying the multimodal loss gradient g multi and the unimodal loss gradient g uni Conflicts between

[0053] Phase 2: Optimize the gradient update directions of the two single-modality encoders using Pareto sum orthogonalization formulas to resolve modality gradient conflicts and promote information independence and complementarity between modalities.

[0054] The first stage includes: defining the loss of identifying unimodal and multimodal: L i and L j ; The Pareto optimization problem of each group is expressed as:

[0055]

[0056] Where, and They are the loss functions L i and L j Gradient of model parameters, α i and α j is the weight coefficient;

[0057] The second stage includes: defining the orthogonal Pareto optimization:

[0058]

[0059] Where λ orth is the regularization parameter;

[0060] Using constraints, simplify the objective function:

[0061]

[0062] Where g i 、g j is the gradient, A and B are the intermediate calculation amounts, A=|g i | 2 -2g i ·g j +|g j | 2 =|g i -g j | 2 , B=2g i ·g j -2|g j | 2 ;

[0063] Derive the optimal solution of the combined gradient:

[0064]

[0065] Where, is the optimal solution of the combined gradient;

[0066] Combine the optimization of multiple groups to obtain the optimal combination gradient Therefore, the final integrated gradient h is defined as:

[0067]

[0068] Where, is the optimal combined gradient, and h is the comprehensive gradient.

[0069] A multimodal deep fake detection system, including: feature extraction module, contrastive learning module, multi-scale joint fusion module, orthogonalized Pareto optimization module and prediction module,

[0070] The feature extraction module includes a video encoder for encoding video data to generate video modality features, and an audio encoder for encoding audio data to generate audio modality features;

[0071] The contrastive learning module balances and utilizes information within and between modalities through adaptive modality contrast learning, wherein the modalities include audio and video modalities, and utilizes clustering information to form fusion weights;

[0072] The multi-scale joint fusion module performs deep extraction on the video modality features and the audio modality features respectively and then fuses them using the fusion weights;

[0073] The orthogonalized Pareto optimization module identifies conflicts between multimodal loss gradients and unimodal loss gradients, and optimizes the gradient update directions of the two unimodal encoders using Pareto and orthogonalization formulas to resolve modal gradient conflicts and promote information independence and complementarity between modalities.

[0074] The prediction module obtains a prediction result based on the fused features.

[0075] The beneficial effects of the present invention include:

[0076] (1) This paper designs a true-false classification based on audio and video. By introducing a unified mapping space for comparative clustering processing, the mutual understanding and representation consistency between different modal data (such as audio and video) are improved. At the same time, an adaptive temperature adjustment mechanism is introduced. This mechanism allows the model to automatically adjust the temperature parameter in the contrast loss function according to the current learning stage, and constructs an adaptive temperature adjustment intra-modal and cross-modal comparative learning module. Subsequently, the video and audio information is used to generate weight parameters for modal fusion, thereby optimizing the fusion effect of multimodal information.

[0077] (2) The present invention designs an orthogonalization-multimodal Pareto module. By introducing orthogonal constraints, it ensures that the gradients of different modalities do not interfere with each other during updating. While resolving the conflict between single-modal and multimodal gradients, it improves the diversity and complementarity of modal fusion.

[0078] (3) Experiments have shown that the model designed by the present invention achieves optimal performance in terms of accuracy on multiple data sets. At the same time, multiple ablation experiments have fully demonstrated the effectiveness of the design module of the present invention.

[0079] (4) The present invention significantly improves the performance of forgery detection and its effectiveness in key multimedia applications, such as network security, content authentication, and media auditing, through the deep fusion and consistent representation of audio and video modalities. Specifically, forged content (such as deep forged videos or synthesized audio) usually has inconsistent features between different modalities, such as the synchronization of lip shape and voice, the matching degree of facial expression and audio emotion, etc. By introducing a unified mapping space and an adaptive temperature adjustment mechanism, the present invention can more accurately capture the subtle differences between these modalities, thereby effectively identifying forged content. In addition, the orthogonalization-multimodal Pareto module further enhances the model's sensitivity to forgery features, ensures the information complementarity and diversity of different modalities, and avoids noise interference from a single modality. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a flowchart of a multimodal deep fake detection method involved in an embodiment of the present application.

[0081] Figure 2 This is a flow chart of the feature extraction stage involved in the embodiment of this application.

[0082] Figure 3 This is a flow chart of the comparative learning phase involved in the embodiment of this application.

[0083] Figure 4 This is a flowchart of the multi-scale joint fusion stage involved in the embodiment of the present application.

[0084] Figure 5 This is a flow chart of the orthogonalized Pareto optimization stage involved in the embodiments of the present application.

[0085] Figure 6 This is a flow chart of the orthogonalized Pareto optimization stage involved in this embodiment. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0087] Example 1

[0088] A multimodal deep fake detection method includes: a multimodal deep fake detection model, wherein the model extracts video modal features and audio modal features from video data and audio data, respectively, then performs deep extraction on the video modal features and audio modal features, and then fuses them using fusion weights, and finally obtains a prediction result based on the fused features; when training the multimodal deep fake detection model, adaptive modal contrast learning is performed based on the audio modal features and video modal features to obtain an adaptive modal contrast loss, and adaptive temperature parameters and fusion weights are designed.

[0089] In another embodiment, the adaptive modality contrastive learning includes inter-modality contrastive learning and intra-modality contrastive learning, and the adaptive modality contrastive loss is expressed as follows:

[0090]

[0091] Where, L MAC is the adaptive modality contrast loss, L va is the contrastive learning loss from video to audio, L av is the audio-to-video contrastive learning loss, L vv is the video-to-video contrastive learning loss, L aa This is audio-to-audio comparison; its main function is to distinguish between video and audio, bringing positive samples closer and negative samples further away, helping the model distinguish features, better classify, and to some extent alleviate inter-modal conflicts. In addition, the subsequent fusion weights are also calculated based on contrastive learning.

[0092] L av The expression is as follows:

[0093]

[0094] Where, E p(A,V) is the expected value, is the unnormalized similarity score of the ith positive pair, is the unnormalized similarity score between the i-th positive pair and the k-th negative pair, τ is the temperature parameter, is the feature representation of the i-th audio sample, is the feature representation of the i-th video sample, is the feature representation of the kth negative sample video, and m is the margin.

[0095] Other contrastive learning loss expressions are similar, differing only in the feature representations to which they are applied.

[0096] In another embodiment, the temperature parameter is defined by combining Bayesian estimation and attention mechanism, as shown below:

[0097] τ t=γ t ·τ t-1 +(1-γ t )·τ new

[0098] γ t =σ(MLP([h t-1 ,Δτ t-1 ,grad*τL*t-1]))

[0099]

[0100] Where τ is the temperature parameter, τ0 is the temperature parameter of the historical state, γ t is the timing adjustment factor, φ k (S) is a set of basis functions, σ is the Sigmoid function, MLP is the multi-layer perceptron, h t-1 is the historical state code, Δτ t-1 is the temperature change, grad is the gradient, L is the loss function, τ t-1 is the temperature at time t-1, τ new is the new temperature, β k is a hyperparameter.

[0101] In another embodiment, the fusion weight is formed using clustering information, and is expressed as follows:

[0102]

[0103] Let m∈v,a, where v represents the video mode, a represents the audio mode, γ is a hyperparameter, and MHAttention m It is a multi-head attention mechanism that can more comprehensively capture the correlation between features:

[0104] For the comprehensive distance metric, the expression is as follows:

[0105]

[0106] Where, is the distance, k∈a,v; Effectively measure the true distance between the sample and the cluster center, is the latent space feature, is the cluster center, β is the trade-off parameter, and is the maximum distance of the corresponding mode, used for normalization.

[0107] At the same time, it captures the statistical distribution differences and directional differences between samples and cluster centers. By introducing a trade-off parameter β, it adjusts the contribution of these two distances in measuring the statistical distance of samples from cluster centers and the directional similarity in feature space, thereby obtaining a more comprehensive evaluation of the relationship between samples and cluster centers.

[0108] In another embodiment, the multimodal deepfake detection model includes a multi-scale video feature extractor, which performs deep extraction of the video modality features to capture long-range dependencies and multi-scale information, as expressed as follows:

[0109]

[0110] Where, is the video modality feature, N is the normalized feature, LN is the normalization operation, λ1 and λ2 are learnable scaling factors, f1, f2, f3, f4, f5, f6 are convolution operations, and MSTLKA is the multi-scale spatiotemporal attention block, which is expressed as follows:

[0111]

[0112] Where, X i Characterized by G i is a gating mechanism, and STLKA is a spatiotemporal attention block, which is used to deeply extract video features. The expression is as follows:

[0113] STLKA(X)=f PW (f DWD (f DW (X)))

[0114] Where, f DW (·) is a 3D depthwise convolution with a kernel size of (2d-1)×(2d-1)×(2d-1); f DWD (·) is the kernel size used 3D dilated depth convolution; f PW (·) is a point convolution used to reduce the number of channels; GSAU-video is a video gated spatial attention unit that dynamically adjusts the weights of attention at different scales. The expression is as follows:

[0115]

[0116] Where X and Y are input features.

[0117] In another embodiment, the multimodal deep fake detection model includes a multi-scale audio feature extractor, which performs deep extraction of the audio modality features, as expressed as follows:

[0118]

[0119] Where, is the audio modal feature, N is the normalized feature, LN is the normalization operation, λ1 and λ2 are learnable scaling factors, f1, f2, f3, f4, f5, f6 are convolution operations, and MFTLKA is the multi-scale frequency-time large kernel attention, which is expressed as follows:

[0120]

[0121] Where, X i Characterized by G i is a gating mechanism, and FTLKA is a frequency-time large kernel attention, which is expressed as follows:

[0122] FTLKA(X)=f PW (f DWD (f DW (X)))

[0123] Where, f DW (·): 2D depth convolution in time and frequency; f DWD (·) is the depth-expanded convolution to capture a larger receptive field; f PW (·) is the point convolution to reduce the number of channels; GSAU-audio is the gated frequency-time attention unit, which dynamically adjusts the attention of different scales through frequency and time gating mechanisms to enhance the capture of various frequency components and time patterns. The expression is as follows:

[0124]

[0125] Where X and Y are input features.

[0126] In another embodiment, the fused features are represented as follows:

[0127]

[0128] Where, is the fused feature, is the video weight, is the audio weight, V i-1 is the video feature, A i-1 is the audio feature;

[0129] Using fusion features Guide deep feature learning, the fusion of complementary modal information and global information:

[0130]

[0131] Among them, K∈V,A.

[0132] In another embodiment, the multimodal deepfake detection model includes a video encoder and an audio encoder. The video encoder processes video data V, including dividing the video into blocks according to given video and audio pairs (V, A) to P; and then encodes the video to obtain global video features, namely video modality features;

[0133] A1, video encoder E v , used to encode the video data V and generate global video features

[0134] A101. For a given video and audio pair (V, A)-P, divide the video into blocks;

[0135] A102, and through the video encoder E v Encode the self-attention layer and feed-forward network;

[0136] The audio encoder processes the audio data A, including converting the audio signal A into a spectrogram S(t,f), calculating the spectrogram S(t,f) to obtain the Mel-spectrogram M(t,m), taking the logarithm of the Mel-spectrogram M(t,m) to obtain logM(t,m), and finally encoding the logarithmic Mel-frequency spectrum to obtain the global audio feature, namely the audio modal feature. Specifically,

[0137] A2, audio encoder E a , used to encode the audio signal A and generate global audio features

[0138] A201, short-time Fourier transform (STFT) module, for converting the audio signal A into a spectrogram S(t,f);

[0139] A202, Mel filter bank, used to calculate the Mel spectrum map M(t,m);

[0140] A203, a logarithmic Mel frequency spectrum calculation module, is used to take the logarithm of the Mel frequency spectrum to obtain logM(t,m);

[0141] A204, self-attention layer and feedforward network are used to encode the logarithmic Mel frequency spectrum and generate global audio features

[0142] In another embodiment, training the multimodal deepfake detection model also includes an orthogonalized Pareto optimization phase, specifically:

[0143] Stage 1: Identifying the multimodal loss gradient g multi and the unimodal loss gradient g uni Conflicts between

[0144] Phase 2: Optimize the gradient update directions of the two single-modality encoders using Pareto sum orthogonalization formulas to resolve modality gradient conflicts and promote information independence and complementarity between modalities.

[0145] The first stage includes: defining the loss of identifying unimodal and multimodal: L i and L j ; The Pareto optimization problem of each group is expressed as:

[0146]

[0147] Where, and They are the loss functions L i and L j Gradient of model parameters, α i and α j is the weight coefficient;

[0148] The second stage includes: defining the orthogonal Pareto optimization:

[0149]

[0150] Where λ orth is the regularization parameter;

[0151] Using constraints, simplify the objective function:

[0152]

[0153] Where g i 、g j is the gradient, A and B are the intermediate calculation amounts, A=|g i | 2 -2g i ·g j +|g j | 2 =|g i -g j | 2 , B=2g i ·g j -2|g j | 2 ;

[0154] Derive the optimal solution of the combined gradient:

[0155]

[0156] Where, is the optimal solution of the combined gradient;

[0157] Combine the optimization of multiple groups to obtain the optimal combination gradient Therefore, the final integrated gradient h is defined as:

[0158]

[0159] Where, is the optimal combined gradient, and h is the comprehensive gradient.

[0160] In another embodiment, the deep fake detection industry is mainly divided into upstream, midstream, and downstream, specifically including the following industrial chains:

[0161] The deepfake detection industry is primarily upstream: core technology and component suppliers. Hardware vendors provide high-performance computing hardware, such as GPUs and TPUs, to support large-scale model training and efficient inference. Research institutions, universities, and technology companies focus on developing advanced machine learning algorithms and deep learning models, the core technologies for detecting and analyzing deepfake content.

[0162] The midstream of the industry chain builds a comprehensive deep fake detection platform for detection system integrators and service providers, integrating multiple detection technologies and algorithms to provide users with a one-stop solution and develop specialized applications or plug-ins suitable for deep fake content identification in different scenarios, such as social media monitoring and news media verification.

[0163] This system is located in the upper and middle reaches of the deep fake detection industry chain, developing and optimizing advanced deep learning models to form a deep fake detection system for multimedia scenarios such as social media and news media;

[0164] The downstream of the industry chain is composed of deep fake detection users and service groups. For example, social media platforms in the downstream of the industry chain are Figure 1-2 As shown, it includes the feature extraction stage, contrast learning stage, multi-scale joint fusion stage, orthogonalized Pareto optimization stage and classification prediction stage; using the video encoder E v and audio encoder E a , encode audio and video data to generate global features; use adaptive modal contrast learning, utilize clustering information to form fusion weights, distinguish and cluster positive and negative samples; use large kernel convolution and gating mechanism for cross-modal fusion of global and deep features; use Pareto and orthogonalization formulas to optimize the gradient update direction of two single-modal encoders to promote information independence and complementarity between modalities; use MLP fully connected neural network, composed of multiple layers, to extract high-order abstract representations from input features and ultimately output a classification result.

[0165] In this embodiment, the feature extraction stage is as follows Figure 3 The feature extraction stage is as follows: the manager divides the video into blocks for the given video and audio pair (V, A)~P, and passes the video encoder E vThe self-attention layer and feedforward network are used to encode the audio signal A into the Mel spectrum M(t,m). The logarithmic Mel frequency spectrum calculation module is used to take the logarithm of the Mel spectrum to obtain logM(t,m); the self-attention layer and feedforward network are used to encode the logarithmic Mel frequency spectrum and generate global audio features.

[0166] In this embodiment, the comparative learning stage is as follows Figure 4 As shown. The specific steps of the comparative learning phase are: calculating similarity, defining and A margin m is introduced when calculating the logits of negative pairs. The similarity of positive pairs is not affected by the margin and continues to be maximized. The logits of negative pairs are subtracted by m. When m is positive, the value of exp(·) increases, the contribution of negative pairs to the denominator increases, and the loss also increases. The model is forced to reduce the similarity of negative pairs. Combining Bayesian estimation and the attention mechanism, a new temperature parameter calculation formula is defined. To ensure the effectiveness of the internal information of each single modality, intra-modal contrastive learning is further performed in video and audio. All losses are integrated to form an adaptive modality contrastive loss. By calculating a comprehensive distance metric, the relationship between samples and cluster centers is more comprehensively evaluated. This distance metric is then combined with a multi-head attention mechanism to calculate importance scores.

[0167] In this embodiment, the multi-scale joint fusion stage is as follows Figure 5 The multi-scale joint fusion stage is as follows: a multi-scale feature extractor is designed for video features to capture long-range dependencies and multi-scale information. First, the spatiotemporal attention block (STLKA) video features are used for deep extraction. For each group Apply STLKA of different scales and dynamically adjust the weights of attention of different scales through the gating mechanism (GSAU). Given the input audio features , where T represents the number of time frames, C is the number of channels, and F is the frequency dimension. The steps of feature extraction through the multi-scale frequency-time large kernel attention (MFTLKA) module are similar to video feature extraction, using frequency-time large kernel attention (FTLKA). For each group Applying FTLKA of different scales, through frequency and time gating mechanism, dynamically adjust the attention of different scales, enhance the capture of various frequency components and time patterns, and perform weighted fusion of deep video and audio features based on the calculated fusion weights to obtain the fused feature representation. Using fusion features Guide deep feature learning, the fusion of complementary modality information and global information.

[0168] In this embodiment, the orthogonalized Pareto optimization stage is as follows: Figure 6The orthogonalized Pareto optimization stage identifies the multimodal loss gradient g multi and the unimodal loss gradient g uni The conflict between them defines the orthogonal Pareto optimization Using constraints, simplify the objective function: Where A=|g i | 2 -2g i ·g j +|g j | 2 =|g i -g j | 2 , B=2g i ·g j -2|g j | 2 , derive the optimal solution of the combined gradient: Combining the optimization of multiple groups, for group 2, similarly, the optimal combined gradient can be obtained Therefore, the final integrated gradient h is defined as:

[0169] In this embodiment, the classification prediction stage includes the following steps: To effectively analyze and predict audio and video data, dedicated classifiers are designed for each modality. Each classifier is a fully connected neural network composed of multiple layers, designed to extract high-level abstract representations from the fused input features and ultimately output a binary classification result.

[0170] Experiments demonstrate that the model of this embodiment achieves optimal performance, with accuracy rates of 96.8% on DefakeAVMiT, 91.7% on FakeAVCeleb, and 97.9% on DFDC. In generalization experiments, when trained on DFDC, the test accuracy on DefakeAVMiT reaches 91.2%, and the test accuracy on FakeAVCeleb reaches 89.2%. Multiple ablation experiments fully demonstrate the effectiveness of our designed module.

[0171] In another embodiment, a multimodal deep fake detection system is provided. It should be understood that the system is similar to the above-mentioned Figure 2 The method embodiment corresponds to the following Figure 2 The various steps involved in the method embodiment and the specific functions of the system can be found in the description above. To avoid repetition, detailed description is appropriately omitted here.

[0172] A multimodal deepfake detection system, comprising: a feature extraction module, a contrastive learning module, a multi-scale joint fusion module, an orthogonalized Pareto optimization module, and a prediction module;

[0173] The feature extraction module includes a video encoder for encoding video data to generate video modality features, and an audio encoder for encoding audio data to generate audio modality features;

[0174] The contrastive learning module balances and utilizes information within and between modalities through adaptive modality contrast information, the modalities including audio and video modalities, and uses clustering information to form fusion weights;

[0175] The multi-scale joint fusion module performs deep extraction on the video modality features and the audio modality features respectively and then fuses them using the fusion weights;

[0176] The orthogonalized Pareto optimization module identifies the conflict between the multimodal loss gradient and the unimodal loss gradient, and optimizes the gradient update direction of the two unimodal encoders using the Pareto and orthogonalization formulas;

[0177] The prediction module obtains a prediction result based on the fused features.

[0178] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

Claims

1. A multimodal deep fake detection method, characterized in that: include: A multimodal deep fake detection model extracts video modal features and audio modal features from video data and audio data respectively, then performs deep extraction on the video modal features and audio modal features respectively and fuses them using fusion weights, and finally obtains prediction results based on the fused features; when training the multimodal deep fake detection model, adaptive modal contrast learning is performed based on audio modal features and video modal features to obtain adaptive modal contrast loss, and adaptive temperature parameters and fusion weights are designed.

2. A multimodal deep fake detection method according to claim 1, characterized in that: The adaptive modality contrastive learning includes inter-modality contrastive learning and intra-modality contrastive learning. The adaptive modality contrastive loss is expressed as follows: Where, L MAC is the adaptive modality contrast loss, I va is the contrastive learning loss for video to audio, I av is the audio-to-video contrastive learning loss, L vv is the video-to-video contrastive learning loss, L aa For audio to audio comparison; L av The expression is as follows: Where, E p(A,V) is the expected value, is the unnormalized similarity score of the ith positive pair, is the unnormalized similarity score between the i-th positive pair and the k-th negative pair, τ is the temperature parameter, is the feature representation of the i-th audio sample, is the feature representation of the i-th video sample, is the feature representation of the kth negative sample video, and m is the margin.

3. A multimodal deep fake detection method according to claim 1, characterized in that: The temperature parameter is defined by combining Bayesian estimation and attention mechanism, and is expressed as follows: t t =c t ·t t-1 +(1-c t )·t new c t =σ(MLP([h t-1 ,Dt t-1 ,grad*τL*t-1])) Where τ is the temperature parameter, τ0 is the temperature parameter of the historical state, γ t is the timing adjustment factor, φ k (S) is a set of basis functions, σ is the Sigmoid function, MLP is the multi-layer perceptron, h t-1 is the historical state code, Δτ t-1 is the temperature change, grad is the gradient, L is the loss function, τ t-1 is the temperature at time t-1, τ new is the new temperature, β k is a hyperparameter.

4. A multimodal deep fake detection method according to claim 1, characterized in that: The fusion weight is formed using clustering information and is expressed as follows: Let m∈v,a, v represents the video mode, a represents the audio mode, γ is a hyperparameter, MHAttention m It is a multi-head attention mechanism. For the comprehensive distance metric, the expression is as follows: Where, is the distance, k∈a,v; Effectively measure the true distance between the sample and the cluster center, is the latent space feature, is the cluster center, β is the trade-off parameter, and is the maximum distance between the video and audio modalities, used for normalization.

5. A multimodal deep fake detection method according to claim 1, characterized in that: The multimodal deepfake detection model includes a multi-scale video feature extractor, which performs deep extraction of the video modality features to capture long-range dependencies and multi-scale information. The expression is as follows: Where, is the video modality feature, N is the normalized feature, LN is the normalization operation, λ1 and λ2 are learnable scaling factors, f1, f2, f3, f4, f5, f6 are convolution operations, and MSTLKA is the multi-scale spatiotemporal attention block, which is expressed as follows: Where, X i Characterized by G i is a gating mechanism, and STLKA is a spatiotemporal attention block, which is used to deeply extract video features. The expression is as follows: STLKA(X)=f PW (f DWD (f DW (X))) Where, f DW (·) is a 3D depthwise convolution with a kernel size of (2d-1)×(2d-1)×(2d-1); f DWD (·) is the kernel size used 3D dilated depth convolution; f PW (·) is a point convolution used to reduce the number of channels; GSAU-video is a video gated spatial attention unit that dynamically adjusts the weights of attention at different scales. The expression is as follows: Where X and Y are input features.

6. A multimodal deep fake detection method according to claim 1, characterized in that: The multimodal deep fake detection model includes a multi-scale audio feature extractor, which performs deep extraction of the audio modality features, as expressed as follows: Where, is the audio modal feature, N is the normalized feature, LN is the normalization operation, λ1 and λ2 are learnable scaling factors, f1, f2, f3, f4, f5, and f6 are convolution operations, and MFTLKA is the multi-scale frequency-time large kernel attention, which is expressed as follows: Where, X i Characterized by G i is a gating mechanism, and FTLKA is a frequency-time large kernel attention, which is expressed as follows: FTLKA(X)=f PW (f DWD (f DW (X))) Where, f DW (·): 2D depth convolution in time and frequency; f DWD (·) is the depth-expanded convolution to capture a larger receptive field; f PW (·) is the point convolution to reduce the number of channels; GSAU-audio is the gated frequency-time attention unit, which dynamically adjusts the attention of different scales through frequency and time gating mechanisms to enhance the capture of various frequency components and time patterns. The expression is as follows: Where X and Y are input features.

7. A multimodal deep fake detection method according to claim 1, characterized in that: The fused features are expressed as follows: Where, is the fused feature, is the video weight, is the audio weight, V i-1 is the video feature, A i-1 is the audio feature; Using fusion features Guide deep feature learning, the fusion of complementary modal information and global information: Among them, K∈V,A.

8. A multimodal deep fake detection method according to claim 1, characterized in that: The multimodal deepfake detection model includes a video encoder and an audio encoder. The video encoder processes video data V, including dividing the video into blocks based on given video and audio pairs (V, A) to P; and then encodes the blocks to obtain global video features, namely video modality features. The audio encoder processes the audio data A, including converting the audio signal A into a spectrogram S(t,f), calculating the spectrogram S(t,f) to obtain the Mel-spectrogram M(t,m), taking the logarithm of the Mel-spectrogram M(t,m) to obtain logM(t,m), and finally encoding the logarithmic Mel-frequency spectrum to obtain the global audio features, namely the audio modal features.

9. A multimodal deep fake detection method according to claim 8, characterized in that: Training a multimodal deepfake detection model also includes an orthogonalized Pareto optimization phase, specifically: Stage 1: Identifying the multimodal loss gradient g multi and the unimodal loss gradient g uni Conflicts between Phase 2: Optimize the gradient update directions of the video encoder and audio encoder using Pareto and orthogonalization formulas to resolve modality gradient conflicts and promote information independence and complementarity between modalities. The first stage includes: defining the loss of identifying unimodal and multimodal: L i and L j ; The Pareto optimization problem of each group is expressed as: Where, and They are the loss function L i and L j Gradient of model parameters, α i and α j is the weight coefficient; The second stage includes: defining the orthogonal Pareto optimization: Where λ orth is the regularization parameter; Using constraints, simplify the objective function: Where g i 、g j is the gradient, A and B are the intermediate calculation amounts, A=|g i | 2 -2g i ·g j +|g j | 2 =|g i -g j | 2 , B=2g i ·g j -2|g j | 2 ; Derive the optimal solution of the combined gradient: Where, is the optimal solution of the combined gradient; Combine the optimization of multiple groups to obtain the optimal combination gradient Therefore, the final integrated gradient h is defined as: Where, is the optimal combined gradient, and h is the comprehensive gradient.

10. A multimodal deep fake detection system, characterized in that The multimodal deep fake detection method according to claims 1-9 can be implemented, comprising: a feature extraction module, a contrastive learning module, a multi-scale joint fusion module, an orthogonalized Pareto optimization module, and a prediction module. The feature extraction module includes a video encoder for encoding video data to generate video modality features, and an audio encoder for encoding audio data to generate audio modality features; The contrastive learning module learns to balance and utilize information within and between modalities through adaptive modality contrast, wherein the modalities include audio and video modalities, and utilizes clustering information to form fusion weights; The multi-scale joint fusion module performs deep extraction on the video modality features and the audio modality features respectively and then fuses them using the fusion weights; The orthogonalized Pareto optimization module identifies conflicts between multimodal loss gradients and unimodal loss gradients, and optimizes the gradient update directions of the two unimodal encoders using Pareto and orthogonalization formulas to resolve modal gradient conflicts and promote information independence and complementarity between modalities. The prediction module obtains a prediction result based on the fused features.

Images