Deep forgery detection method and system based on audio and video multi-mode fusion

Abstract

The invention belongs to the technical field of multimedia security, and particularly relates to a deep forgery detection method and system based on audio and video multi-mode fusion. The method comprises the following steps: extracting lip motion space-time characteristics from a video stream through dynamic ROI (Region of Interest) cutting, and processing an audio stream through fast Fourier transform and a Mel filter bank in sequence to obtain audio spectrum characteristics; reconstructing and generating a corresponding audio feature based on the spatio-temporal feature of lip motion, and fusing the audio spectrum feature and the generated audio feature by adopting a bidirectional cross-modal attention mechanism to obtain an attention fusion feature; a Mel spectrogram of the audio stream is acquired, and feature maps of different scales are extracted from the Mel spectrogram and the video stream by using a feature extraction model formed by a convolutional layer and are fused to obtain multi-scale features; after the attention fusion feature and the multi-scale fusion feature are flattened, performing weighted fusion after channel dimension splicing, and outputting a forgery probability through a multi-layer perceptron. The video detection performance is improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia security technology, and in particular relates to a deep fake detection method and system based on audio and video multimodal fusion. Background Art

[0002] In recent years, with the rapid development of artificial intelligence (AI) and the widespread adoption of the internet and digital products, the acquisition, editing, and transmission of visual media (such as images and videos) have become easier than ever before. Generative AI has made significant breakthroughs, particularly in the field of image and video generation. Advanced techniques such as Generative Adversarial Networks (GANs) and diffusion models have enabled high-quality, highly realistic synthetic content. However, the rapid development of these technologies has also led to the increasing prevalence of malicious applications such as forgery and tampering.

[0003] Traditional image and video processing techniques primarily rely on the analysis of visual features, using methods such as local texture, pixel statistics, and edge detection to capture the characteristic information of images and thus distinguish true from false content. With the successful application of deep learning in computer vision, detection methods based on convolutional neural networks (CNNs) have achieved certain results in detecting forged images using generative models such as ProGAN, StyleGAN, and BigGAN. However, due to significant differences in generation strategies, content preservation, and artifact characteristics among generative models, existing single-modality detection methods often lack generalization capabilities across models and scenarios, making it difficult to meet the challenges posed by new generative technologies.

[0004] On the other hand, in recent years, deep fake technology has gradually penetrated into multimodal fields such as video and audio. In particular, in deep fake videos, forgers often use open source tools and massive amounts of celebrity footage to modify information such as facial features and lip movements, and supplemented by audio reconstruction, so that the fake videos achieve a very high degree of simulation in both visual and auditory aspects. This not only greatly increases the difficulty of forgery detection, but also makes traditional detection technologies that rely on single modal features show obvious limitations in complex scenarios. For example, when faced with fake videos with inconsistent audio and video information and obvious local spatiotemporal changes, detection methods based on visual features often find it difficult to capture subtle differences across modalities, resulting in a significant decrease in detection performance.

[0005] Currently, deepfake detection technology is in a fierce competition between technical attacks and defenses. Existing methods mostly rely on standard dataset training and achieve high detection accuracy in specific scenarios. However, as forgery technology continues to evolve and forged content becomes more diverse and cross-modal, traditional detection methods become inadequate. Traditional detection technology has the following major problems:

[0006] 1) Reliance on single modality information: Most existing detection methods focus primarily on visual information from images or videos, neglecting the complementary role of other modalities such as audio and text. This makes it difficult to fully capture forgery traces in multimodal forgery scenarios.

[0007] 2) Local features are affected by semantic interference: When using local texture and pixel statistics for feature extraction, they are often affected by the semantic structure of the image content, making it difficult to distinguish the essential differences between the generated image and the real image.

[0008] 3) Insufficient cross-model generalization capability: Existing detection models are usually trained on image data of specific generative models, and are difficult to adapt to image forgeries of different generation strategies (such as GAN and diffusion models). The detection performance across scenes and across generation models is significantly reduced.

[0009] 4) Insufficient multimodal information fusion: Faced with multimodal forgery techniques such as audio and video synchronization and spatiotemporal inconsistency, simple feature splicing or weighted fusion methods often cannot fully explore the deep semantic associations between different modalities, resulting in less robust detection results. Summary of the Invention

[0010] To address the problems of existing deepfake detection technologies, such as insufficient modeling of cross-modal correlations between audio and video, limited ability to capture spatiotemporal inconsistencies, and lack of explainable localization of tampered areas, this paper proposes a deepfake detection method based on multimodal fusion of audio and video, which specifically includes the following steps:

[0011] Receive video and audio streams, extract lip motion spatiotemporal features from the video stream through dynamic ROI cropping, and process the audio stream through fast Fourier transform and Mel filter bank to obtain audio spectrum features;

[0012] Based on the spatiotemporal features of lip motion, the corresponding audio features are reconstructed and generated. The bidirectional cross-modal attention mechanism is used to fuse the audio spectrum features with the generated audio features to obtain the attention fusion features.

[0013] Obtain the Mel-spectrogram of the audio stream, and use a feature extraction model composed of convolutional layers to extract feature maps of different scales from the Mel-spectrogram and video stream, and fuse them to obtain multi-scale features;

[0014] The attention fusion features and multi-scale fusion features are flattened and spliced ​​in the channel dimension before weighted fusion is performed, and a forgery probability is output through a multi-layer perceptron.

[0015] Furthermore, dynamic ROI cropping is used to extract the spatiotemporal features of lip motion, including: using the HRNet network to detect 68 key points on the face and constructing a minimum bounding box with 12 key points in the lip area; using the optical flow method to compensate for inter-frame motion, and performing motion smoothing within 5 consecutive frames with a fixed size of 128×128 pixels; when the head deflection angle is detected to exceed 15 degrees, the 3D face mesh reconstruction module is activated to generate an orthographic projection.

[0016] Furthermore, the reconstruction and generation of corresponding audio features based on the spatiotemporal features of lip movements consists of two stages. The first stage uses the Lip2Speech model to generate the fundamental frequency contour based on the lip movement features. The second stage uses the WaveGlow model to synthesize the time-frequency details. The loss function of the audio feature generator is:

[0017]

[0018] in, is the audio feature generator loss function; S real Represents the audio spectrum characteristics; S fake Indicates that the corresponding audio features are reconstructed based on the spatiotemporal features of lip movements; ||·||1 indicates the L1 norm. Denotes the mean; D(S fake ) represents a decision device, which is used to determine whether the input audio is an audio spectrum feature. If it is an audio spectrum feature, D(S fake )=1, otherwise D(S fake )=0.

[0019] Furthermore, the process of comparing the original and reconstructed audio using a bidirectional cross-modal attention mechanism includes:

[0020] The spatial and temporal features of lip movements are used as query vectors, and the audio spectrum features are used as key vectors and value vectors respectively to calculate the visual-audio attention weights.

[0021] The corresponding audio features reconstructed based on the spatiotemporal features of lip motion are used as the query vector, and the spatiotemporal features of lip motion are used as the key vector and value vector respectively to calculate the audio-visual attention weight;

[0022] The visual-audio attention weight and the audio-visual attention weight are concatenated together and then passed through a linear layer and a sigmoid activation layer to generate a gating weight. The gating weight is used to fuse the visual-audio attention weight and the audio-visual attention weight to obtain the attention fusion feature.

[0023] Furthermore, shallow features, middle features, and deep features are extracted from the Mel-spectrogram and the video stream, and fused to obtain shallow fusion features, middle fusion features, and deep fusion features. The shallow fusion features, middle fusion features, and deep fusion features are fused to obtain multi-scale features. The acquisition of multi-scale features includes:

[0024] Perform global average pooling on shallow fusion features and middle fusion features respectively;

[0025] The pooled features are concatenated with the deep fusion features to obtain a multi-scale fusion feature vector;

[0026] The multi-scale fusion feature vector is input into the multi-layer perceptron to obtain the multi-scale features.

[0027] Furthermore, the process of using ResNet50 and visual encoder to extract shallow features, middle features and deep features from the video stream and Mel-spectrogram respectively and fusing them to obtain shallow fusion features, middle fusion features and deep fusion features specifically includes:

[0028] Extract edge texture features from video stream using the first convolutional layer of ResNet50 Use visual encoder to extract low-frequency features from Mel spectrogram Concatenate the two features in the channel dimension to obtain shallow features

[0029] The convolutional layer in the middle of ResNet50 outputs visual motion features The convolutional layer in the middle of the visual encoder outputs audio harmonic features The two features are added element by element and processed by 3×3 convolution and softmax function to obtain the middle layer features.

[0030] The last fully connected layer of ResNet50 outputs a visual high-order semantic vector The audio context vector of the final fully connected output of the visual encoder After splicing together, input into two layers of cascaded multi-layer perceptron to obtain deep features

[0031] Furthermore, when the forgery probability S output When S > 0.5, the video is judged to be forged; when S output When ≤0.5, the video is judged to be real, and the feature fusion and prediction process uses the cross entropy loss function as the loss for training:

[0032]

[0033] Among them, y is the true label, S outputis the probability of falsification.

[0034] Furthermore, the average energy difference ΔE between the audio spectrum feature and the reconstructed audio spectrum feature in the 400-4000Hz speech main frequency band is calculated. avg When the difference exceeds a set threshold, it is marked as an audio anomaly. The lip motion vector is extracted using the optical flow method and dynamically time-warped with the standard lip template corresponding to the speech content. If the timing deviation Δt exceeds a set threshold, it is marked as a visual anomaly. The audio anomaly, visual anomaly, and forgery probability are weighted and fused. When the weighted fusion value is greater than the set threshold, the audio is judged to be forged. The process of weighted fusion of audio anomaly, visual anomaly, and forgery probability includes:

[0035] S final =0.5·S output +0.25·S audio +0.25·S visual

[0036] Among them, S final The final forgery probability is obtained by weighted fusion of audio anomaly, visual anomaly and forgery probability; S output is the forgery probability output by the multi-layer perceptron; S audio is the audio anomaly secondary distribution parameter. When there is an audio anomaly, the parameter value is 1, otherwise it is 0; S visual is the visual anomaly quadratic distribution parameter. When there is an audio anomaly, the parameter value is 1, otherwise it is 0.

[0037] Furthermore, when it is determined that the audio is forged, the forged area is located. That is, the contribution of each part of the feature map, which is weighted fusion after flattening the attention fusion feature and the multi-scale fusion feature and then splicing it in the channel dimension, to the obtained forgery probability is calculated. Then, a morphological closing operation is performed on the area where the contribution is greater than 0.7, and the connected area with an area of ​​more than 100 pixels is extracted. The confidence of the abnormal area is calculated. If the confidence is greater than the set threshold, the video and audio in the area do not match. The formula for calculating the confidence of the abnormal area is:

[0038]

[0039] Among them, N abnormal is the number of abnormal pixels in the connected area with an area greater than 100 pixels, N total is the number of pixels in a connected region with an area greater than 100 pixels; ΔE audio is the audio difference peak.

[0040] The present invention also proposes a deep fake detection system based on audio and video multimodal fusion, which is used to implement a deep fake detection method based on audio and video multimodal fusion, including:

[0041] A data acquisition module is used to obtain the video stream and audio stream of the data to be detected;

[0042] The feature extraction module is used to extract the spatiotemporal features of lip movements from the video stream through dynamic ROI cropping, and then process the audio stream through fast Fourier transform and Mel filter bank to obtain audio spectrum features;

[0043] A feature reconstruction module is used to reconstruct and generate corresponding audio features based on the spatiotemporal features of lip movements;

[0044] The first feature fusion module is used to fuse the audio spectrum features and the generated audio features using a bidirectional cross-modal attention mechanism to obtain attention fusion features;

[0045] The second feature fusion module is used to use a feature extraction model composed of convolutional layers to extract feature maps of different scales from the Mel spectrum map and the video stream, and fuse them to obtain multi-scale features;

[0046] The prediction module performs weighted fusion based on the flattened attention fusion features and multi-scale fusion features, spliced ​​in the channel dimension, and outputs a forgery probability through a multi-layer perceptron.

[0047] Furthermore, it also includes an output module for performing weighted fusion based on audio anomalies, visual anomalies and forgery probability to obtain a final forgery probability. If the final forgery probability is greater than 0.5, it is determined that the data to be detected has an anomaly.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1) Improved detection performance: Through cross-modal generative reconstruction (Lip2Speech-WaveGlow cascade model) and a bidirectional cross-attention mechanism, we achieved a detection accuracy of 98.7% on the FakeAVCeleb dataset. Compared with traditional methods, our ability to capture subtle lip movement-speech timing deviations (≥20ms) and spectral energy differences (≥3dB) is significantly improved.

[0050] 2) Enhanced cross-modal detection capabilities: Dynamic ROI cropping (128×128 pixel adaptive region) and a layered dynamic fusion architecture (shallow stitching + mid-layer attention + deep MLP) effectively cope with complex scenarios such as head deflection (≤15°), noise interference (SNR≥20dB), and compression artifacts (CRF=23). The cross-model (GAN / diffusion model) detection accuracy fluctuation is ≤±3%;

[0051] 3) Accurate and explainable spatiotemporal positioning: Based on heat map connectivity analysis (activation value > 0.7, area > 100 pixels) and dynamic time regularization (continuous anomaly ≥ 0.5 seconds), 5×5 pixel-level spatial positioning and

[0052] ±8.3ms timestamp marking, visual report synchronously outputs tampered area superimposed video and abnormal audio waveform;

[0053] 4) Plug-and-play capability enhancements extend existing model functionality through lightweight plug-ins to support scenarios such as social media review, judicial evidence collection, and model upgrades.

[0054] 4) System modular design: The audio and video deep fake detection system proposed in this invention includes an audio and video feature extraction module, a cross-modal feature fusion module, a multi-layer feature fusion module, and a detection and positioning module. The modules are seamlessly connected through standard interfaces, facilitating promotion and application in various practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of a deep fake detection method based on audio and video fusion in the present invention;

[0056] Figure 2 Schematic diagram of local spatiotemporal video content audio feature reconstruction (Lip2Speech+WaveGlow) in the present invention;

[0057] Figure 3 A schematic diagram of a method for detecting audio and video inconsistency measurement forgery in the present invention;

[0058] Figure 4 This is a schematic diagram of an application of a plug-and-play audio and video modality consistency forgery detection plug-in in the present invention. DETAILED DESCRIPTION

[0059] The technical solution of the present invention is described in detail below in conjunction with the embodiments and drawings. This embodiment is implemented based on the technical solution of the present invention, but the protection scope of the present invention is not limited to the following embodiments.

[0060] The present invention proposes a deep fake detection method based on audio and video multimodal fusion, which specifically includes the following steps:

[0061] Receive video and audio streams, extract lip motion spatiotemporal features from the video stream through dynamic ROI cropping, and process the audio stream through fast Fourier transform and Mel filter bank to obtain audio spectrum features;

[0062] Based on the spatiotemporal features of lip motion, the corresponding audio features are reconstructed and generated. The bidirectional cross-modal attention mechanism is used to fuse the audio spectrum features with the generated audio features to obtain the attention fusion features.

[0063] Obtain the Mel-spectrogram of the audio stream, and use a feature extraction model composed of convolutional layers to extract feature maps of different scales from the Mel-spectrogram and video stream, and fuse them to obtain multi-scale features;

[0064] The attention fusion features and multi-scale fusion features are flattened and spliced ​​in the channel dimension before weighted fusion is performed, and a forgery probability is output through a multi-layer perceptron.

[0065] This embodiment achieves high-precision and robust audio and video forgery detection and spatiotemporal localization through audio and video feature extraction, cross-modal generative reconstruction, layered dynamic fusion, and multi-dimensional inconsistency verification, while also providing interpretable visual analysis results. Furthermore, this method supports plug-and-play deployment, seamlessly enhancing the cross-modal analysis capabilities of existing detection models.

[0066] The core solution of the present invention includes:

[0067] 1) Dynamic lip feature extraction: The HRNet network is used to locate facial key points. Optical flow motion compensation and 3D face mesh reconstruction are combined to adaptively crop the 128×128 pixel lip region to address feature distortion caused by head deflection.

[0068] 2) Cross-modal Audio Reconstruction and Alignment: The Lip2Speech model generates a fundamental frequency contour based on lip movement features, and the WaveGlow model further synthesizes time-frequency details. A bidirectional cross-attention mechanism (visual→audio, audio→visual paths) is employed to compare the timing deviations and spectral energy differences between the original audio spectral features and the generated audio features (main frequency band 400-4000Hz, threshold 3dB). This process effectively aligns the timing differences between the audio and visual features, improving the accuracy of the reconstructed audio.

[0069] 3) Hierarchical Dynamic Feature Fusion: Through shallow, mid-level, and deep-level multi-scale feature fusion modules, we combine features from different levels of visual and audio modalities for effective cross-modal fusion. The shallow layer combines edge texture features from layers 1-3 of the visual encoder with audio mel-spectrogram features. The mid-level layer weights motion and harmonic features using spatial attention maps. The deep layer maps high-level semantic associations between visual and audio using multi-layer perceptrons, ultimately generating a 128-dimensional fused feature.

[0070] 4) Spatiotemporal Joint Localization: Based on Grad-CAM technology, heat maps are generated. Gradient-weighted activation values ​​are calculated for feature maps. Morphological closing operations are used to remove noise and extract abnormal regions. Connected domain analysis, combined with the Dynamic Time Warping (DTW) algorithm, accurately detects tampered areas in the video and outputs pixel-level coordinates and millisecond timestamps.

[0071] 5) Plug-and-play enhancement plug-in: Encapsulates cross-modal reconstruction alignment and layered fusion modules and integrates them into existing models (such as AVAD and LipForensics) through PyTorch standardized interfaces to achieve enhanced performance and interpretability.

[0072] like Figure 1 As shown in the figure, the core process of the present invention covers four modules: dynamic lip feature extraction, cross-modal generative reconstruction, layered dynamic feature fusion and dual verification of audio and video inconsistency. When used as an independent detection system, it outputs a tampered area heat map and millisecond-level abnormal timestamp; when packaged as a plug-and-play plug-in ( Figure 4 ), by splicing cross-modal alignment features with the unimodal features of the original model, the detection accuracy of baseline models such as ResNet is improved by 12-18%, and the new spatiotemporal positioning function is added, and the interpretability is significantly improved.

[0073] This embodiment provides a specific implementation method, and the complete steps of the method are as follows:

[0074] Step 1) Audio and video feature extraction

[0075] The input video stream passes through the HRNet network to detect 68 key points on the face, focusing on 12 key points in the lip area to construct a minimum bounding box. The head deflection angle θ is calculated using the following formula:

[0076]

[0077] Where v1 and v2 are the left and right zygomatic point vectors, respectively, and ||·|| represents the norm. When θ>15°, a frontal view projection is generated based on the 3DMorphable Model, and a standardized lip ROI of 128×128 pixels is output.

[0078] The video encoder uses a 3D-ResNet18 network to extract lip movement features. Where T is the time step.

[0079] The audio stream is filtered by pre-emphasis (coefficient 0.97), framed and windowed (25ms Hamming window, frame shift 10ms), and the Mel spectrum is calculated. Focusing on the main frequency band of 400-4000Hz. The Speaker encoder extracts audio features through a one-dimensional convolutional layer.

[0080] Step 2) Cross-modal reconstruction and alignment

[0081] like Figure 2 , the audio feature f a and spatiotemporal characteristics f v After splicing and fusion, the audio decoder is used to reconstruct the generator (AudioDecoder) to generate the reconstructed audio spectrum S fake The Lip2Speech model outputs the fundamental frequency contour. The WaveGlow model further synthesizes time-frequency details, and its loss function is:

[0082]

[0083] The discriminator D(·) uses a 5-layer spectral normalization convolutional network with a 256×256 linear spectrum as input. During the training of the generator, a discriminator is used to determine whether the generated audio is real or fake.

[0084] In the bidirectional cross-attention mechanism, the visual→audio path uses the spatiotemporal features of lip movement as the query vector in the attention mechanism of this path. The audio spectrum feature is used as the key vector in the attention mechanism of this path Sum vector Compute temporal attention weights for the vision→audio path:

[0085]

[0086] In the audio→visual path, the corresponding audio features are reconstructed based on the spatiotemporal features of lip movement as the query vector in the attention mechanism of this path. The spatiotemporal features of lip movement are used as the key vector in the attention mechanism of this path Value vector Calculate the temporal attention weights for the audio→visual path:

[0087]

[0088] The dual-path outputs are fused through gate control, namely:

[0089] f fusion =g⊙f v2a +(1-g)⊙f a2v

[0090] Among them, f fusion is the dual-path output, ⊙ represents the Hadamard product. The gate value g is generated by the concatenated features through the Sigmoid function and is expressed as:

[0091] g=σ(W g [f v2a ;f a2v ]+b g )

[0092] in, is the learnable weight parameter, b g is a learnable bias parameter, and σ is a Sigmoid function.

[0093] Step 3) Hierarchical dynamic feature fusion

[0094] Shallow fusion combines visual edge features and audio fundamental frequency characteristics Channel concatenation into 96-dimensional features

[0095] Mid-level fusion of visual motion features and audio harmonic characteristics Perform element-by-element addition and generate the spatial attention map W through 3×3 convolution:

[0096] Get 64-dimensional features in It means element-by-element addition;

[0097] Deep fusion combines visual high-order semantic vectors With audio context vector Input two layers of MLP and output 128-dimensional fusion features

[0098] Step 4) Cross-scale attention mechanism

[0099] In order to achieve effective information interaction between multi-scale features, this paper introduces a cross-scale attention mechanism to further enhance the feature fusion effect between different scales, specifically including:

[0100] First, the shallow fusion features and mid-level fusion features Perform global average pooling (GlobalAverage Pooling) to obtain a fixed-length vector and

[0101] The shallow features after pooling Mid-level features Fusion with deep features Splicing to form a unified multi-scale fusion feature vector

[0102] The multi-scale fusion feature vector f multi Input to the MLP layer for feature weighting processing, and finally output the unified fusion feature representation f final =MLP(f multi ). According to the attention fusion feature f fusion and multi-scale fusion features f final After flattening, they are concatenated in the channel dimension and weighted fused, and a forgery probability is output through a multi-layer perceptron.

[0103] Step 5) Inconsistency Measurement (Classification and Detection Enhancement)

[0104] like Figure 3 ,When detecting forgery, we first use cross-modal fusion features to make the distinction. The feature f output by the hierarchical dynamic fusion module fusion Fusion feature f with deep semantics final The concatenation is performed and input into a two-layer fully connected multi-layer perceptron for processing, and the forgery probability S is generated through the Sigmoid activation function. output This value indicates the consistency of the audio and video modalities and ultimately determines whether the video is forged. Specifically, if S output >0.5, the video is judged to be fake.

[0105] To enhance the accuracy of forgery detection, this step further combines audio and visual inconsistency metrics. Audio inconsistency is calculated by calculating the average energy difference ΔE between the original and reconstructed audio in the main speech frequency band of 400-4000Hz. avg To achieve this, the formula for calculating the energy difference of the main frequency band of audio inconsistency is:

[0106]

[0107] Among them, S real (t) represents the average energy of the original audio in the 400-4000Hz speech main frequency band at time t; S fake (t) represents the average energy of the reconstructed audio in the 400-4000 Hz main speech frequency band at time t; ||·||2 is the L2 norm.

[0108] When ΔE avg When >3dB, it is marked as abnormal, that is:

[0109]

[0110] Visual inconsistency is solved by matching the lip optical flow vector M with the standard lip template T through the DTW algorithm:

[0111]

[0112] Among them, Δt is the timing deviation; M(i) represents the lip optical flow vector of the i-th sampling point; T(j) represents the standard lip template of the j-th sampling point, and π is the optimal alignment path connecting two sequences in the DTW algorithm, that is, the lip optical flow vector M(i) of the i-th sampling point represents the matching relationship between the standard lip template T(j) of the j-th sampling point. Its purpose is to eliminate the timing difference through nonlinear mapping, thereby quantifying the visual inconsistency.

[0113] Tampering is determined when the timing deviation exceeds 20ms, that is:

[0114]

[0115] Finally, the comprehensive S output 、S audio and S visual Score, get S final , the formula is as follows:

[0116] S final =0.5·S output +0.25·S audio +0.25·S visual

[0117] When S final >0.5, triggering the forgery judgment, and considering the video to be forged; if S final If ≤0.5, the video is judged to be real.

[0118] Step 6) Spatiotemporal positioning

[0119] The spatiotemporal localization module accurately locates tampered areas in the video by generating a heat map and subsequent processing steps. This module combines gradient weighting technology to determine the areas in the video frame that are most likely to have been tampered with and provides the precise location of the tampered areas with spatiotemporal coordinates and anomaly timestamps.

[0120] First, generate the heat map H heatmap It is achieved through the gradient weighting method. Specifically, the heat map is calculated by calculating the activation value α of each pixel i , the activation value represents the importance of the position for forgery detection. The activation value A of each pixel in the feature map i is also considered in this process, and the formula is as follows:

[0121]

[0122] The generated heat map assigns a value to each pixel location, indicating the probability that the area has been tampered with. Regions with higher activation values ​​are more likely to be forged regions.

[0123] Next, we post-process the generated heatmap using a morphological closing operation to filter out connected regions with activation values ​​greater than 0.7 and an area greater than 100 pixels. These regions are considered potential tampering regions. During this process, the morphological closing operation removes small noisy regions while retaining larger, connected tampering regions.

[0124] For these connected regions, the abnormal confidence of each region is further calculated. The confidence of the abnormal region C is based on the number of abnormal pixels N in the region. abnormal and the total number of pixels N totalThe confidence calculation formula is as follows:

[0125]

[0126] Among them, n abnormal is the number of abnormal pixels, N total is the total number of pixels, ΔE audio The audio difference peaks represent the discrepancies between the video and its audio spectrum. Through this process, the spatiotemporal localization module not only provides the spatial coordinates of the tampered area but also the timestamp of the anomaly. This allows the system to precisely pinpoint the exact location and time of the forgery and generate a corresponding visual report for further review by analysts.

[0127] Step 7) Plug-and-play plugin integration

[0128] like Figure 4 As shown, this method can be encapsulated as a lightweight enhancement plug-in that can be seamlessly integrated into existing benchmark models or state-of-the-art methods through PyTorch's standardized interface. By combining cross-modal alignment features with the unimodal features of the original model, the plug-in adds spatiotemporal localization capabilities, significantly improving the detection performance and interpretability of the original model (outputting heatmaps and marking abnormal time periods) while maintaining low computational overhead.

[0129] Step 8) Experimental verification

[0130] Experimental results show that the proposed method has achieved excellent performance in a variety of forged video detection scenarios, significantly outperforming existing methods in terms of accuracy, recall, AUC value, F1 value and other indicators, especially in cross-modal inconsistency detection, showing stronger robustness and generalization ability.

[0131] Table 1 Detection performance of the method of the present invention on multiple data sets

[0132]

[0133] In summary, the present invention proposes a dual-effect fusion deep fake detection solution, which can not only achieve fine-grained capture of spectral energy differences (≥3dB) and lip movement-speech timing deviations (≥20ms) through dynamic lip feature extraction, cross-modal audio reconstruction (Lip2Speech-WaveGlow) and bidirectional cross-attention mechanism, and achieve a detection accuracy of 98.79% on the FakeAVCeleb dataset. It can also be encapsulated as a lightweight plug-in and seamlessly integrated into existing models such as AVAD and LipForensics through the PyTorch / TensorFlow interface, achieving a significant improvement in the detection performance and interpretability of the original model while keeping the original system architecture unchanged.

[0134] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A deep fake detection method based on audio and video multimodal fusion, characterized by: The specific steps include: Receive video and audio streams, extract lip motion spatiotemporal features from the video stream through dynamic ROI cropping, and process the audio stream through fast Fourier transform and Mel filter bank to obtain audio spectrum features; Based on the spatiotemporal features of lip motion, the corresponding audio features are reconstructed and generated. The bidirectional cross-modal attention mechanism is used to fuse the audio spectrum features with the generated audio features to obtain the attention fusion features. Obtain the Mel-spectrogram of the audio stream, and use a feature extraction model composed of convolutional layers to extract feature maps of different scales from the Mel-spectrogram and video stream, and fuse them to obtain multi-scale features; The attention fusion features and multi-scale fusion features are flattened and spliced ​​in the channel dimension before weighted fusion is performed, and a forgery probability is output through a multi-layer perceptron.

2. The method according to claim 1, characterized in that Extracting spatiotemporal features of lip motion through dynamic ROI cropping includes: using the HRNet network to detect 68 key points on the face and constructing a minimum bounding box with 12 key points in the lip area; compensating for inter-frame motion through the optical flow method and performing motion smoothing within 5 consecutive frames at a fixed size of 128×128 pixels; and starting the 3D face mesh reconstruction module to generate an orthographic projection when the head deflection angle is detected to exceed 15 degrees.

3. The method according to claim 1 or 2, characterized in that The reconstruction and generation of corresponding audio features based on the spatiotemporal features of lip movement consists of two stages. The first stage uses the Lip2Speech model to generate the fundamental frequency contour based on the lip movement features. The second stage uses the WaveGlow model to synthesize the time-frequency details. The loss function of the audio feature generator is: in, is the audio feature generator loss function; S real Represents the audio spectrum characteristics; S fake Indicates that the corresponding audio features are reconstructed based on the spatiotemporal features of lip movements; ||·||1 indicates the L1 norm. Indicates the mean; D(S fake ) represents a decision device, which is used to determine whether the input audio is an audio spectrum feature. If it is an audio spectrum feature, D(S fake )=1, otherwise D(S fake )=0.

4. The method according to claim 1, wherein The process of comparing the original and reconstructed audio using a bidirectional cross-modal attention mechanism includes: The spatial and temporal features of lip movements are used as query vectors, and the audio spectrum features are used as key vectors and value vectors respectively to calculate the visual-audio attention weights. The corresponding audio features reconstructed based on the spatiotemporal features of lip motion are used as the query vector, and the spatiotemporal features of lip motion are used as the key vector and value vector respectively to calculate the audio-visual attention weight; The visual-audio attention weight and the audio-visual attention weight are concatenated together and then passed through a linear layer and a sigmoid activation layer to generate a gating weight. The gating weight is used to fuse the visual-audio attention weight and the audio-visual attention weight to obtain the attention fusion feature.

5. The method according to claim 1, wherein Shallow features, middle features, and deep features are extracted from the Mel spectrum graph and video stream, and then fused to obtain shallow fusion features, middle fusion features, and deep fusion features. Shallow fusion features, middle fusion features, and deep fusion features are then fused to obtain multi-scale features. The acquisition of multi-scale features includes: Perform global average pooling on shallow fusion features and middle fusion features respectively; The pooled features are concatenated with the deep fusion features to obtain a multi-scale fusion feature vector; The multi-scale fusion feature vector is input into the multi-layer perceptron to obtain the multi-scale features.

6. The method according to claim 5, characterized in that The process of using ResNet50 and visual encoder to extract shallow features, middle features and deep features from video stream and Mel spectrum respectively and fusing them to obtain shallow fusion features, middle fusion features and deep fusion features includes: Extract edge texture features from video stream using the first convolutional layer of ResNet50 Use visual encoder to extract low-frequency features from Mel spectrogram Concatenate the two features in the channel dimension to obtain shallow features The convolutional layer in the middle of ResNet50 outputs visual motion features The convolutional layer in the middle of the visual encoder outputs audio harmonic features The two features are added element by element and processed by 3×3 convolution and softmax function to obtain the middle layer features. The last fully connected layer of ResNet50 outputs a visual high-order semantic vector The audio context vector of the final fully connected output of the visual encoder After splicing together, input into two layers of cascaded multi-layer perceptron to obtain deep features 7. The method according to claim 1, characterized in that When the forgery probability S output When S > 0.5, the video is judged to be forged; when S output When ≤0.5, the video is judged to be real, and the feature fusion and prediction process uses the cross entropy loss function as the loss for training: Among them, y is the true label, S output is the probability of falsification.

8. The method according to claim 1 or 7, characterized in that Calculate the average energy difference ΔE between the audio spectrum features and the reconstructed audio spectrum features in the 400-4000Hz speech main frequency band avg ,When the difference exceeds the set threshold, it is marked as an audio anomaly; The lip motion vector is extracted using the optical flow method and dynamically time-warped with the standard lip template corresponding to the speech content. If the timing deviation Δt exceeds a set threshold, it is marked as a visual anomaly. The audio anomaly, visual anomaly, and forgery probability are weighted and fused. If the weighted fusion value is greater than the set threshold, the audio is judged to be forged. The process of weighted fusion of audio anomaly, visual anomaly, and forgery probability includes: S final =0.5·S output +0.25·S audio +0.25·S visual Among them, S final The final forgery probability is obtained by weighted fusion of audio anomaly, visual anomaly and forgery probability; S output is the forgery probability output by the multi-layer perceptron; S audio is the audio anomaly secondary distribution parameter. When there is an audio anomaly, the parameter value is 1, otherwise it is 0; S visual is the visual anomaly quadratic distribution parameter. When there is an audio anomaly, the parameter value is 1, otherwise it is 0.

9. The method according to claim 1, characterized in that When it is determined that the audio is forged, the forged area is located. That is, the contribution of each part of the feature map, which is weighted fusion after flattening the attention fusion feature and the multi-scale fusion feature and then splicing it in the channel dimension, to the obtained forgery probability is calculated. Then, a morphological closing operation is performed on the area where the contribution is greater than 0.7, and the connected area with an area of ​​more than 100 pixels is extracted. The confidence of the abnormal area is calculated. If the confidence is greater than the set threshold, the video and audio in the area do not match. The formula for calculating the confidence of the abnormal area is: Among them, N abnormal is the number of abnormal pixels in the connected area with an area greater than 100 pixels, N total is the number of pixels in a connected region with an area greater than 100 pixels; ΔE audio is the audio difference peak.

10. A deep fake detection system based on audio and video multimodal fusion, characterized by: For implementing the method according to claim 1, comprising: A data acquisition module is used to obtain the video stream and audio stream of the data to be detected; The feature extraction module is used to extract the spatiotemporal features of lip movements from the video stream through dynamic ROI cropping, and then process the audio stream through fast Fourier transform and Mel filter bank to obtain audio spectrum features; A feature reconstruction module is used to reconstruct and generate corresponding audio features based on the spatiotemporal features of lip movements; The first feature fusion module is used to fuse the audio spectrum features and the generated audio features using a bidirectional cross-modal attention mechanism to obtain attention fusion features; The second feature fusion module is used to use a feature extraction model composed of convolutional layers to extract feature maps of different scales from the Mel spectrum map and the video stream, and fuse them to obtain multi-scale features; The prediction module performs weighted fusion based on the flattened attention fusion features and multi-scale fusion features, spliced ​​in the channel dimension, and outputs a forgery probability through a multi-layer perceptron.

Images