Face deepfake detection method based on multi-modal fusion

Abstract

The application discloses a kind of face depth forgery detection methods based on multi-modal fusion, belong to artificial intelligence, computer vision and multimedia forensics field.For the generalization deficiency of existing detection method, audio and picture time sequence inconsistency problem, end-to-end detection scheme is proposed.The method fuses the time-aligned audio spectrum and continuous video frame mosaic picture in advance, forms multi-modal unified input to solidify audio and picture time sequence relationship.Global branch uses contrast learning pre-training visual Transformer, introduces modal bias and cross-modal attention bias to enhance feature distinction interaction;Local branch fuses local and global features through multi-scale region perception network, realizes adaptive weighted fusion by combining position coding, vector-level channel attention and scale attention, and is guided by region perception auxiliary loss to select discriminative features.The method provides an effective tool for content review, traceability forensics and identity authentication security.

Description

Technical Field

[0001] This invention relates to the field of face forgery detection. By jointly representing audio spectral features with consecutive video frames in the form of a pre-fused multimodal image, and combining the global semantic modeling capability and multi-scale local region perception mechanism of a pre-trained visual Transformer, effective detection of forgery traces such as audio-visual inconsistencies can be achieved. Background Technology

[0002] Early deepfake detection primarily focused on the visual traces left by the generative algorithm in the spatial domain, such as inconsistent resolution, stitching edges, or abnormal facial textures. With the maturation of GAN technology, generated images have become increasingly realistic on visual surfaces, and relying solely on spatial domain visual features has gradually reached its limits. Therefore, the detection approach has shifted from the "visible space" to the "invisible frequency domain." Research shows that operations such as upsampling disrupt the original frequency domain statistical regularities of images, allowing for the capture of the generative model's feature fingerprint through frequency domain perception. Simultaneously, some works have taken a geometric and temporal approach, utilizing unnatural deformations of facial features and discontinuous motion to model the image. Although these single-modal methods perform well on specific datasets, under low-quality compression, noise interference, or complex real-world scenarios, single cues are prone to degradation or loss, limiting robustness and generalization ability.

[0003] As visual forgery becomes increasingly sophisticated, research has shifted towards the biological consistency between audio and audiovisual modalities. Because human pronunciation involves the coordination of speech signals, lip movements, and facial muscles, even if generation algorithms can synthesize realistic images or sounds separately, they struggle to simultaneously and accurately reproduce the complex temporal and semantic correspondences between the two. Based on this, related research uses phenomena such as "lip-reading mismatch" and "audiovisual temporal misalignment" as detection criteria, driving detection technology from single-modal analysis to multi-modal fusion. Furthermore, research focus has evolved from simple feature splicing to deeper interactive fusion: on the one hand, dynamic weights are introduced to balance the contribution of audiovisual information; on the other hand, end-to-end synchronous fusion networks are constructed to capture fine-grained associations. With the rise of the Transformer architecture, modeling cross-modal long-range dependencies using attention mechanisms has also become an important direction. This evolution indicates that detection logic has gradually shifted from finding "where it looks fake" to judging "whether audiovisual behavior conforms to human biological laws."

[0004] In summary, while existing deepfake detection techniques have evolved from spatial / frequency domain single-modal features to audiovisual consistency modeling and deep multimodal interaction, they still have shortcomings in fine-grained temporal inconsistencies in synchronous forgeries. Therefore, it is still necessary to propose a face forgery detection method that can explicitly solidify audiovisual alignment relationships under a unified representation, while taking into account both global semantics and local details and possessing good generalization ability.

[0005] (1) Current fake face detection technology mainly relies on single modal data, lacks the utilization of deep correlation information between audio and video, ignores the inherent correlation between video and audio, resulting in the inability to effectively detect synchronization defects and emotional consistency.

[0006] (2) The differences in distribution density and expression intensity of multimodal features can easily lead to the suppression of sparse audio frequency domain fingerprints by high-dimensional visual features, and the difficulty in accurately aligning audio and video modalities on the time scale. (3) There are huge differences between video frames and audio signals in terms of sampling rate and physical time, and simple feature splicing is difficult to accurately correspond. Summary of the Invention

[0007] To address the shortcomings of existing deepfake detection methods, such as insufficient utilization of multimodal clues, limited semantic representation capabilities, and the ease with which local forgery traces are obscured, this invention proposes a face deepfake detection method based on multimodal fusion. The key technical aspects of this method include: (1) Pre-fusion-based graph construction and modality bias injection: This invention abandons the complex two-stream network and constructs a pre-fusion image containing audio spectrum and video frames. To solve the problem of a single Transformer processing heterogeneous data, this invention introduces a learnable modality bias vector in the feature embedding layer. This bias vector is explicitly added to the corresponding audio patch and video patch, enabling the model to clearly distinguish modal identities in the initial stage of feature extraction, thereby learning cross-modal differences more effectively in a unified feature space.

[0008] (2) Injected Attention Bias & Residual CLS To enhance the sensitivity of audio-visual association, this invention designs and injects a fixed attention bias matrix into the self-attention computation of the Transformer, artificially increasing the attention weight between the audio and video regions, guiding the model to focus on cross-modal interactive features rather than single-modal internal textures. Furthermore, a residual CLS connection is designed at the output, adding the fused features to the CLS token output by the original pre-trained model, thus utilizing the newly learned fake features while retaining the powerful global semantic representation capability of the pre-trained model.

[0009] (3) Vector-based SE Fusion Mechanism: To address the issue of dynamically changing contributions of audio and video features, this invention introduces a vector-based Squeeze-and-Excitation (SE) module before feature output. Unlike traditional two-dimensional feature map SE, this module directly operates on one-dimensional feature vectors, adaptively learning the importance weights of audio and video feature channels through compression and excitation operations, thereby achieving dynamic enhancement of key modal features and suppression of noise features.

[0010] (4) Scale-Positional Encoding in Region Awareness: To address the spatial information loss caused by local cropping, this invention introduces learnable spatial-scale positional encoding into the multi-scale region feature extraction network. Before concatenating local features from different scales and regions with global features, corresponding positional encoding vectors are added. This allows the network to explicitly perceive the physical region represented by each feature vector and its scaling level, thereby establishing a stable local and global alignment relationship.

[0011] (5) Region-Aware SE & Loss Optimization: To select the most discriminative clues from multi-scale features, this invention applies a vector-level channel attention mechanism to recalibrate the concatenated features in the region-aware module. Combined with the designed region-aware auxiliary loss (RALoss), the model is forced to expand the margin between the largest weight scale and other scales, prompting the model to automatically "focus" on the scale where the forgery is most obvious within each region, thus avoiding the degradation of feature weight averaging. Attached Figure Description Figure 1 This is a schematic diagram of the overall technical route structure of the present invention; Figure 2 This is a schematic diagram of the preliminary fusion image construction process for aligning audio and video temporal features and performing modal fusion according to the present invention; Figure 3 This is a schematic diagram of the multi-view construction structure of the present invention, which generates global input and multi-scale local input based on fused features; Figure 4 This is a schematic diagram of the global semantic modeling network structure that introduces modal bias and cross-modal attention mechanisms in this invention; Figure 5 This is a schematic diagram of the discrimination decision structure of the present invention, which combines global features and multi-scale local region features. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0013] To address the shortcomings of existing deepfake detection methods in utilizing cross-modal clues and exhibiting weak cross-modal interaction in forged scenarios, this invention proposes a face deepfake detection method based on multimodal fusion. The method employs a structure of "pre-process fusion of multimodal images + single-vision Transformer global modeling + multi-scale region-aware local modeling," thereby achieving a progressive synergy of "guided fusion, adaptive fusion, and complementary fusion" at the feature level.

[0014] Step 1: Multimodal Image Generation and Alignment Step 11: For each video segment to be processed, first determine its total number of frames (denoted as F). To cover the entire video content, select the starting positions of 10 time windows on the timeline (i.e., the number of extractions N = 10). These starting positions are approximately uniformly distributed within the effective interval [0, F - 5 - 1] to ensure that each time window contains a complete 5-frame (denoted as T = 5) continuous image. We group these selected starting frame indices into a set, and for each starting point k in the set, extract the corresponding 5-frame video sequence as the processing unit.

[0015] Step 12: For each of the 5 frames within the time window, perform color space conversion to adjust them to the RGB format consistent with the model input (the original video is usually in BGR format). Next, scale each frame to 500×500 pixels. After processing, stitch these 5 frames together horizontally to form a long, rectangular video frame stitching image (denoted as Xv). In this embodiment, since it contains 5 frames, the final size of the stitched Xv is: width 2500 × height 500.

[0016] In this embodiment, T=5, therefore X v The dimensions are 500×2500 (W,H).

[0017] Step 13: Read the audio signal corresponding to the video and obtain its time-domain data y and sampling rate sr. First, calculate the Mel power spectrum of the audio and convert it to a logarithmic scale (decibels) to generate a visualized spectrum image (denoted as S). To accurately map the audio and video in time, we need to calculate a mapping coefficient (rho), which is equal to the total width of the spectrum image divided by the total number of video frames F. For the starting frame k of each video time window, use the mapping coefficient to calculate the starting coordinates b and ending coordinates e of the audio. Based on these two coordinates, crop the corresponding spectrum segment and forcibly scale it to the exact same size as the "video frame stitching image", i.e., 500-2500, to obtain the aligned spectrum image (denoted as Xa).

[0018] Step 14: Place the processed spectrogram Xa in the upper half and the video stitched image Xv in the lower half, then stitch them together vertically to form the final pre-fusion multimodal image (denoted as X). This final fusion image X contains audiovisual bimodal information and has physical dimensions of 2500 width × 1000 height. All subsequent model inputs are processed based on this top-bottom structured fusion image.

[0019] Step 2: In this embodiment, the model is constructed from the fused image X through two inputs: a global image and a multi-scale set of local patches. After the image is read in, a color space conversion is performed to make it conform to the RGB specification of the pre-trained visual model input.

[0020] Step 21: Read the fused image X (storage format corresponds to the BGR channel order read by OpenCV) and convert it to RGB representation.

[0021] Step 22: We perform feature cropping only from the lower half of the fused image (i.e., the video region). The total height of the fused image is known to be 1000, and the video region corresponds to the interval between row coordinates 500 and 1000. Assume the number of regions matches the number of frames in the time window, i.e., a total of 5 regions. For each region r (from 0 to 4), we segment it in the width direction with a step size of 500 pixels, obtaining a basic local block of size 500×500. Subsequently, we uniformly scale these basic blocks to the standard input size of the model, 224×224. We define this set of views as the Scale 0 (basic scale) local view.

[0022] Step 23: To capture forgery traces of varying fineness, we perform a "center-progressive cropping and zoom-back" operation on the Scale 0 view, which has already been scaled to 224×224: Scale 1 (Medium Scale): Crop the image with the center as the reference, resulting in a cropped region of 168×168 pixels. Then, enlarge this region back to 224×224 pixels. Scale 2 (Fine Scale): Crop the image further towards the center, resulting in a cropped region of 102×102 pixels. Again, enlarge this region back to 224×224 pixels. Finally, for each region, we obtain a multi-scale local set containing three scales.

[0023] Step 24: Resample the complete RGB version of the fused image (including the upper audio and lower video portions) and uniformly adjust it to a fixed-size global image. The target size is set to 1120×1120 to fully preserve the overall structural relationship of the "spectrum-frame sequence" before it enters the network, serving as the input for the global branch.

[0024] Step 25: Unified Normalization. Perform unified normalization on the global image and all local views obtained in Step 2. For any input image tensor (whether global or local), subtract the mean and divide by the standard deviation. This yields the final input tensor of the model: the normalized global input and the set of multi-scale local inputs.

[0025] Step 3: Convolutional pre-downsampling adaptation and pre-trained visual Transformer encoding.

[0026] Step 31: Denote the global input image obtained in Step 2 as Xg. Before feeding it into the pre-trained visual Transformer, pre-sampling is performed through a convolutional layer to reduce the number of subsequent tokens and match the input specifications of the pre-trained model. The parameters of this convolutional layer are consistent with the code: the kernel size is fixed at 5, and the number of output channels is set to 3. The stride is chosen based on the configuration of the pre-trained model: when the model is configured with a 336-pixel input specification, the stride is 3; otherwise (i.e., the default configuration is a 224-pixel input specification), the stride is 5. After this operation, the pre-sampling output is: Step 32 (Loading the pre-trained visual Transformer encoder) A pre-trained visual Transformer encoder is loaded as the global branch backbone. This encoder uses visual priors obtained through large-scale contrastive learning to enhance its representation of high-level semantic consistency and global structural relationships.

[0027] Step 4: Modal Bias and Cross-Modal Attention Bias Injection Step 41: Input the pre-downsampling output obtained in Step 3 into the PatchEmbedding process of the visual Transformer to obtain a Patch Token sequence, concatenate it with the Class Token, and then overlay positional encoding. The final Token sequence contains three dimensions: batch size, total number of Tokens (i.e., 1 Class Token plus L Patch Tokens), and the dimension of the Tokens. The 0th Token is defined as the Class Token, and the rest are Patch Tokens.

[0028] Step 42: Since the fused image has a fixed spatial structure of "upper half audio spectrum, lower half video", after the Patch Tokens are unfolded in raster order, their two segments can approximately correspond to the spectrum region and the video region. The segmentation point is taken as half of the total number of Patch Tokens, thus defining the index set of the spectrum tokens and the index set of the video tokens (excluding the class token): that is, the first half of the sequence corresponds to the audio modality, and the second half corresponds to the video modality.

[0029] Step 43: Initialize a learnable modal bias parameter matrix. The modal bias injection process is as follows: Construct two independent sets of learnable bias vectors, corresponding to the spectral modality and video modality biases respectively. Perform broadcast addition on the token sequence grouped by modality: Tokens belonging to the spectral region are added to the audio bias vector; tokens belonging to the video region are added to the video bias vector. Perform broadcast addition on the token sequence grouped by modality:

[0030] The bias is updated end-to-end during training, enabling an explicit distinction between the "spectral modality subspace" and the "video modality subspace" to be formed early in the encoding process.

[0031] Step 44: Cross-modal attention bias injection (with a constant positive bias in the cross-blocks, injecting into the first few layers). Construct the attention bias matrix M, setting the constant bias to gamma (set to 1). The matrix assignment rule is: only assign gamma to the intersection region of the spectrum token and the video token, and assign 0 to the rest.

[0032] The matrix M is injected as an attention bias or mask into the self-attention computation of the first few layers (such as layers 0, 1, and 2) of the Transformer to enhance cross-modal information interaction.

[0033] Step 45: For the token sequence after injecting modal bias and attention bias, perform multi-layer self-attention and feedforward network calculations sequentially to obtain the encoded output sequence. Among them, t0 is the Class Token output, and the rest (t1 to tL) are the Patch Token outputs.

[0034] The spectrum token and video token are pooled separately by averaging, that is, by calculating the average value of the first half of the tokens (denoted as f-audio) and the average value of the second half of the tokens (denoted as f-video).

[0035] The two are concatenated to obtain the combined feature z. Vector-level channel attention (Vector-SE) is applied to z for recalibration: the weight vector w is calculated through two fully connected layers (containing ReLU activation in the middle and Sigmoid activation at the end), and then z and w are multiplied channel by channel. Subsequently, the recalibrated feature is linearly projected to obtain the global feature f-global. To enhance the global semantic stability, the Class Token t0 is mapped through the same projection space, and residuals are stacked using a learnable scaling factor lambda (initialized to 0.5) to finally update the global feature f-global.

[0036] Step 5: Multi-scale local feature extraction Step 51: Extract local features from the multi-scale local image set (containing different scales and regions) constructed in Step 2 using a residual network backbone. To adapt to small-batch training and improve stability, the backbone network uses Group Normalization instead of Batch Normalization (specifically set to 32 groups of GN).

[0037] Step 52: Flatten and stack all local blocks of all scales and regions in batch dimension, and feed them into the backbone network at once to obtain local feature vectors. In this embodiment, the dimension of the local feature vector is consistent with the output of the ResNet-50 backbone, which is 2048 dimensions, providing fine-grained evidence input for the subsequent "region-aware fusion (position encoding + vector attention + scale weight Softmax)".

[0038] Step 61: Obtain the local feature vector (dimension denoted as dl, i.e., 2048) for each scale and region from Step 5; obtain the global multimodal semantic features (dimension denoted as dg, i.e., 768) from Step 4. For each "scale-region" combination, concatenate the local and global features along the channel dimension to obtain the fused feature vector:

[0039] At this point, the total dimension d of the fused features is equal to the sum of the local and global dimensions. By introducing global multimodal semantic features into each local region, the local discrimination process is constrained by the global context, avoiding reliance solely on local texture information.

[0040] Step 62: Assign a learnable location encoding vector to each "scale-region" location and superimpose it onto the corresponding stitched feature. The location encoding is used to explicitly identify the scale level and region (frame order) identity corresponding to the fused feature, thereby preserving spatial and temporal location information during multi-scale, multi-region parallel modeling and avoiding the location information loss problem common in multi-cropping methods.

[0041] Step 63: Apply Vector-Level Channel Attention (Vector-SE) to the fused features after overlay position encoding, and adaptively recalibrate each channel to obtain recalibrated features. The specific calculation logic is as follows: multiply the fused features by the result processed by the channel scaling function g. The channel scaling function g consists of a multi-layer fully connected network and non-linear gating, and its output is the scaling coefficient for each channel. This recalibration process is used to enhance feature channels related to forgery detection and suppress redundant or noisy channels.

[0042] Step 64: For different scale features within the same region r, predict scale weights based on their recalibrated features, and perform Softmax normalization on the scale dimension s to obtain scale attention weights. These weights are calculated as the ratio of the current scale weight index to the sum of all scale weight indexes. The scale attention weights reflect the importance of different scale features for forgery detection within the same region, enabling the model to adaptively select scale features with stronger discriminative power.

[0043] Step 65: Weight the multi-scale features within the same region *r* according to the scale attention weights to obtain the region-level fused features. Then, aggregate the fused features of all regions to obtain the sample-level feature representation. This process corresponds to two aggregation steps: first, multiply and sum the features at each scale with their corresponding attention weights to obtain the region features; then, sum all the region features and divide by the total number of regions *R* (i.e., calculate the average) to obtain the sample-level feature representation.

[0044] The F-out is used as the input feature for subsequent classification and loss calculation.

[0045] Step 7: Classification and Discrimination Output Step 71: Input the final aggregated features into the fully connected classification layer to output category Logits; Step 72: Obtain the probability value of "real / fake" through Softmax, and output the fake probability as the detection result.

[0046] Step 81: Model training employs a total loss function composed of a weighted average of the main classification loss and the region-aware auxiliary loss. This function simultaneously constrains the final discrimination result and the intermediate-scale attention allocation behavior, and is defined as follows:

[0047] The cross-entropy classification loss is used to supervise the model's output in distinguishing between real and fake samples; the region-aware auxiliary loss is used to assist in optimizing local feature selection; and the weighting coefficients are used to balance the contributions of the two types of losses. In one implementation, the main classification loss adopts the label-smoothed cross-entropy form to reduce the risk of overfitting the model on the training set and improve its generalization ability across datasets.

[0048] In the multi-scale region-aware fusion process obtained in step 6, for each region r, the model obtains a corresponding set of scale attention weights. Two key metrics are defined from this set: one is the maximum attention weight across all scales within the region (denoted as alpha-max), and the other is the attention weight at the base scale (Scale 0) within the region (denoted as alpha-base).

[0049] The region-aware auxiliary loss is used to constrain the relationship between the two types of weights mentioned above, and it is defined as follows:

[0050] Where m is a preset margin hyperparameter. Through this loss term, the model is explicitly constrained: in each region, at least one scale's attention weight should be significantly higher than the base scale, thereby preventing scale attention from degenerating into an approximately uniform distribution during training and prompting the model to actively select more discriminative scale features.

[0051] To verify the effectiveness and feasibility of the method of the present invention, in one embodiment, the method of the present invention is trained and tested, and the detection performance is quantitatively evaluated using common classification evaluation indicators.

[0052] During the training phase, this invention uses supervised learning to train the model. Each training image is input into the model, the batch size is set to 16, and the AdamW optimizer is used for optimization. The initial learning rate is set to 1×10⁻⁶. -4 The learning rate was adjusted using a cosine annealing strategy, and finally reduced to 1×10. -6 .

[0053] During the testing phase, the length of the input image was set to 16, and the batch size was set to 16. Tests were performed using a test set of a dataset.

[0054] Table 1 shows the results of the four indicators, AUC, AP, ACC and F1, of this invention.

[0055] Table 1. Metrics on the AVLips dataset method Method of the present invention AUC 0.9662 AP 0.9579 ACC 0.9107 F1 0.9155 It should be noted that the specific embodiments described above are exemplary, and those skilled in the art can devise various solutions inspired by the disclosure of this invention. These solutions all fall within the scope of this invention and its protection. Those skilled in the art should understand that this specification and its accompanying drawings are illustrative and not intended to limit the scope of the claims. The scope of protection of this invention is defined by the claims and their equivalents.

Claims

1. A face deepfake detection method based on multimodal fusion, wherein the face deepfake detection method constructs a pre-fusion multimodal image from audio spectrum information aligned with a time window and continuous video frame information, and achieves real / fake discrimination based on global-local multi-view feature modeling, specifically including: Step 1: Data Acquisition and Time Window Alignment. Acquire the audio and video data of the object to be detected, and perform time alignment between the audio segments and the corresponding consecutive video frames within a given time window; Step 2: Construction of the initial fusion-type multimodal image. The audio data aligned with the time window is converted into an audio spectrogram, and the continuous video frames within the time window are stitched together horizontally to form a video frame stitching image. The audio spectrogram is placed in the upper half and the video frame stitching image is placed in the lower half for spatial stitching to obtain the initial fusion-type multimodal image. Step 3: Multi-view input generation and unified preprocessing: Generate a global image based on the previously fused multimodal images as the global branch input; Multiple local region blocks are extracted from the video region of the previously fused multimodal image, and a multi-scale local image set containing different spatial scales is constructed by progressive cropping from the center; a unified size transformation and normalization preprocessing are performed on the global image and the multi-scale local image set to form a multi-view input; Step 4: Global branch downsampling and visual Transformer encoding. The global image is input into the feature downsampling module for spatial downsampling to adapt to the input resolution and patch partitioning method of the pre-trained visual Transformer. The downsampled global image is then input into the feature encoding network based on the visual Transformer architecture for patch embedding and feature encoding to obtain the global token sequence. Step 5: Modal bias guidance and cross-modal attention bias injection. Based on the spatial structure of the previously fused multimodal images, the global token sequence is grouped into spectral modal tokens and video modal tokens, and learnable modal biases are applied to the tokens corresponding to different modalities. A cross-modal attention bias is introduced in the first few layers of feature encoding to enhance the information interaction between the spectral modal token and the video modal token. The cross-modal attention bias takes a non-zero value on the cross-attention terms corresponding to the spectral modal token and the video modal token, and a zero value on the other attention terms. Step 6: Global feature generation. Pool and fuse the spectral modality token and the video modality token respectively. The fusion includes concatenating the pooled features of different modalities, recalibrating the features through a channel attention mechanism, and obtaining the global features through linear mapping. Furthermore, residual fusion based on classification identifier features is introduced to enhance the stability of global feature representation; Step 7: Local branch feature extraction and multi-scale region perception fusion. The multi-scale local image set is input into a local feature extraction network for local feature extraction. The local feature extraction network adopts a residual connection structure and combines a normalization strategy suitable for mini-batch training. The obtained local features are fused with the global features, and positional information encoding for representing the region location and scale level is introduced. Channel attention recalibration is performed on the fused features, and attention weights are calculated in the scale dimension to achieve adaptive weighted fusion of multi-scale features, resulting in region aggregation features. Step 8: Discrimination Output and Training Supervision. Input the aggregated features of the region into the discrimination model and output the discrimination result of the real or fake object to be detected. During the model training process, a weighted combination of the main loss function and the auxiliary loss function is used as the supervision signal. The auxiliary loss function is used to constrain the difference in attention allocation of features at different scales and is combined with one or more training stabilization strategies to improve training stability and generalization performance.

Images