A multimodal fake video detection method based on multi-head additive cross-attention mechanism
By extracting video and audio features through a multi-head additive cross-attention mechanism and performing cross-modal fusion, the shortcomings of single-modal detection in existing technologies are overcome, and high-precision identification of complex forged videos is achieved.
Patent Information
- Application Number
- CN202511117722.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-11
- Publication Date
- 2025-12-02
- Estimated Expiration
- 2045-08-11
AI Technical Summary
Existing methods for detecting fake videos focus too much on unimodal information and neglect audio information, resulting in low detection accuracy, poor generalization ability, long training cycles, and large model parameters, making it unable to effectively identify complex fake videos.
A multi-head additive cross-attention mechanism is adopted to extract video and audio features through a 3D convolutional network and filter bank, perform cross-modal feature alignment and fusion, and use cosine similarity loss function and dynamic gradient weighted optimization to make four-class forgery type decisions.
It significantly improves the recognition accuracy of multimodal fake videos, reduces the problems of weak generalization ability and long training cycle of the model, and achieves efficient recognition of complex fake videos.
Smart Images
Figure CN120635786B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of video forgery detection technology, and in particular relates to a multimodal forgery video detection method based on a multi-head additive cross-attention mechanism. Background Technology
[0002] With the rapid development of AIGC technology and the maturation of existing forgery methods, more and more forgery traces are becoming difficult to detect, making the detection of forged videos increasingly challenging. Thanks to the tremendous success of generative image processing technologies (VAEs) and generative adversarial networks (GANs), face manipulation has become a hot new research topic in recent years. Face swapping or partial forgery techniques, such as deepfakes and face swapping, allow even those without specialized skills or equipment to generate high-quality fake faces. Therefore, developing effective face forgery detection methods has become particularly important.
[0003] Current research largely focuses on unimodal forgery methods, overemphasizing video information while neglecting audio-related information and the subtle connections between video and audio, which are crucial for video forgery detection. Existing forgery detection methods face problems such as long training cycles, low detection accuracy, poor generalization ability, large model parameters, and long inference times. With the continuous development and increasing complexity of forgery techniques, traditional detection methods are gradually failing to meet practical needs, thus necessitating new and more efficient forgery detection technologies. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention proposes a multimodal forged video detection method based on a multi-head additive cross-attention mechanism, thereby resolving the issues present in the prior art.
[0005] Firstly, to achieve the above objectives, this invention provides a multimodal forged video detection method based on a multi-head additive cross-attention mechanism, comprising the following steps:
[0006] S1. Multimodal data preprocessing:
[0007] Video modal processing includes decomposing the video stream into a sequence of single-frame images, locating the face region, and cropping the face image;
[0008] Audio modal processing includes extracting audio waveforms to generate Mel spectrograms and then slicing them into segments;
[0009] S2. Multimodal feature extraction:
[0010] Video feature extraction employs a 3D convolutional network for block sampling, extracting spatiotemporal features and calculating inter-frame motion differences;
[0011] Audio feature extraction is based on generating spectral features using a filter bank, combined with a residual network to capture contextual information;
[0012] S3. Cross-modal feature alignment and fusion:
[0013] Map audio features to a latent space aligned with the video dimension;
[0014] After applying temporal encoding to video features, they are input into a multi-head additive cross-attention module along with audio features:
[0015] Audio-dominant features are generated using audio features as query vectors and video features as key-value pairs.
[0016] Video-dominant features are generated using video features as query vectors and audio features as key-value pairs.
[0017] Concatenate and fuse the original features with the bidirectional interactive features;
[0018] S4. Comparative Learning and Classification Decision-Making:
[0019] Construct a cosine similarity loss function for audio and video to constrain feature distribution;
[0020] The fused features are dynamically weighted, optimized, and compressed over time to output the probability of four types of forgery.
[0021] Optionally, the process of S1 includes:
[0022] Video data is extracted in frames at a fixed sampling rate and divided into non-overlapping time periods;
[0023] The audio spectrum is segmented along the time axis using a Hamming window.
[0024] The number of sub-spectrum segments is aligned with the number of video time segments.
[0025] Optionally, the video feature extraction process in S2 includes:
[0026] Perform horizontal and vertical bidirectional slicing operations on video frames;
[0027] Multi-scale strip convolution is used to aggregate spatiotemporal information;
[0028] An attention map was generated using analysis of variance to highlight the micro-movements of the eyes and the distortion of the lips.
[0029] Optionally, the feature alignment process in S3 includes:
[0030] Map audio feature time steps to video time series into a shared latent space;
[0031] Projected onto a unified representation space through a fully connected layer;
[0032] The first fusion feature is generated by splicing along the channel dimension.
[0033] Optionally, the operation of the multi-head additive cross-attention module in S3 includes:
[0034] The query vector is split into multiple attention heads, and weights are calculated independently.
[0035] Attention weights for each head are weighted and fused with value vectors;
[0036] Integrate original features and aggregated features through residual connections.
[0037] Optionally, the four-category forgery types in S4 include:
[0038] Audiovisual double authenticity, audiovisual double deception, video forgery with real audio, real video with forged audio;
[0039] The classification decision adopts a multi-granularity discrimination mechanism to integrate audio binary classification, video binary classification and multimodal fusion branch output.
[0040] Secondly, the present invention also provides a multimodal forged video detection system based on a multi-head additive cross-attention mechanism, used to implement a multimodal forged video detection method based on a multi-head additive cross-attention mechanism, the system comprising:
[0041] The preprocessing module is used to decompose the input video stream into single-frame image sequences and locate the face region, while extracting the audio waveform to generate a Mel spectrogram;
[0042] The feature extraction module is used to extract spatiotemporal features by sampling video frames in blocks using a 3D convolutional network, and to generate audio spectral features based on a filter bank.
[0043] A cross-modal alignment module is used to map audio features to a latent space aligned with the video dimension;
[0044] The attention fusion module is used to perform bidirectional multi-head additive cross-attention calculation on video features and audio features with applied temporal coding, generate audio-dominant features and video-dominant features, and cascade and fuse the original features with bidirectional interactive features.
[0045] The classification decision module is used to construct the audio and video cosine similarity loss function to constrain the feature distribution, and output the four-class forgery type probability after dynamic weighting and optimization of the fused features.
[0046] Optionally, the preprocessing module includes:
[0047] A video segmentation unit is used to extract video frames at a fixed sampling rate and divide them into non-overlapping time segments;
[0048] The spectrum slicing unit is used to slice the Mel spectrogram along the time axis using a Hamming window, so that the number of sub-spectral segments is aligned with the video time period.
[0049] Thirdly, the present invention also provides a computer terminal device, comprising:
[0050] One or more processors;
[0051] A memory, coupled to the processor, for storing one or more programs;
[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement a multimodal fake video detection method based on a multi-head additive cross-attention mechanism.
[0053] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements a multimodal forged video detection method based on a multi-head additive cross-attention mechanism.
[0054] Compared with the prior art, the present invention has the following advantages and technical effects:
[0055] This invention provides a multimodal forged video detection method based on a multi-head additive cross-attention mechanism. This method achieves latent spatial alignment between audio features and video dimensions through asymmetric projection, and utilizes a multi-head additive cross-attention mechanism to complete cross-modal deep interaction, effectively capturing texture anomalies and audio-video temporal mismatch features in forged videos. A cosine similarity loss function is used to constrain the density of homologous features and the separability of heterologous features, combined with dynamic gradient weighting to optimize the four-class classification decision process. This method significantly improves the recognition accuracy of multimodal forged attacks in open environments, overcomes the deficiency of single-modal detection in capturing temporal mismatch features, and reduces the problems of weak model generalization ability and long training cycles caused by cross-modal interaction redundancy. Attached Figure Description
[0056] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:
[0057] Figure 1 This is a flowchart of an embodiment of the present invention;
[0058] Figure 2 This is a flowchart of the video preprocessing process according to an embodiment of the present invention;
[0059] Figure 3 This is a structural diagram of the video feature extraction module according to an embodiment of the present invention;
[0060] Figure 4 This is a structural diagram of the audio feature extraction module according to an embodiment of the present invention;
[0061] Figure 5 This is a structural diagram of the multi-head additive cross-attention module according to an embodiment of the present invention;
[0062] Figure 6 This is a schematic diagram illustrating the trend of detection accuracy on the FakeAVCeleb test set as the number of training iterations increases, according to an embodiment of the present invention.
[0063] Figure 7 This is a schematic diagram illustrating the trend of the detection AUC on the FakeAVCeleb test set as the number of training iterations increases according to an embodiment of the present invention.
[0064] Figure 8 This embodiment of the invention uses the t-SNE nonlinear dimensionality reduction algorithm to show the distribution of the final classification layer features in two-dimensional space. t-SNE is a commonly used nonlinear dimensionality reduction algorithm that is widely used in the visualization of high-dimensional data. Detailed Implementation
[0065] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0066] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0067] To address the shortcomings of existing methods, this invention proposes a multimodal video forgery detection method and system based on a multi-head additive cross-attention mechanism. This method extracts spatiotemporal and spectral features through video and audio feature extractors and achieves feature alignment through an asymmetric projection mechanism. The multi-head additive cross-attention module is used for deep interaction and feature fusion to identify anomalous textures and temporal mismatches in forged videos, thereby achieving accurate identification of video forgeries.
[0068] Example 1
[0069] like Figure 1 As shown, this embodiment provides a multimodal forged video detection method based on a multi-head additive cross-attention mechanism, including:
[0070] S1. Multimodal data preprocessing:
[0071] Video modal processing includes decomposing the video stream into a sequence of single-frame images, locating the face region, and cropping the face image;
[0072] Audio modal processing includes extracting audio waveforms to generate Mel spectrograms and then slicing them into segments;
[0073] S2. Multimodal feature extraction:
[0074] Video feature extraction employs a 3D convolutional network for block sampling, extracting spatiotemporal features and calculating inter-frame motion differences;
[0075] Audio feature extraction is based on generating spectral features using a filter bank, combined with a residual network to capture contextual information;
[0076] S3. Cross-modal feature alignment and fusion:
[0077] Map audio features to a latent space aligned with the video dimension;
[0078] After applying temporal encoding to video features, they are input into a multi-head additive cross-attention module along with audio features:
[0079] Audio-dominant features are generated using audio features as query vectors and video features as key-value pairs.
[0080] Video-dominant features are generated using video features as query vectors and audio features as key-value pairs.
[0081] Concatenate and fuse the original features with the bidirectional interactive features;
[0082] S4. Comparative Learning and Classification Decision-Making:
[0083] Construct a cosine similarity loss function for audio and video to constrain feature distribution;
[0084] The fused features are dynamically weighted, optimized, and compressed over time to output the probability of four types of forgery.
[0085] Specifically, multimodal data preprocessing includes video modal processing, which decomposes the video into an image sequence, performs rotation, cropping and normalization, and uses facial key point detection to extract face images; audio modal processing involves extracting audio waveforms, generating Mel spectrograms, and performing time slicing.
[0086] Multimodal feature extraction uses a 3D convolutional network and a SincNet filter bank to extract video and audio features, respectively.
[0087] Cross-modal feature alignment and fusion calculates feature correlations through asymmetric projection and multi-head additive cross-attention modules.
[0088] Contrastive learning and classification decision-making constrain the feature distribution through the cosine similarity loss function, and optimize the feature space through dynamic gradient weighting and temporal global average pooling to output the forgery type.
[0089] Specifically, by combining the subtle relationship between video and audio to identify forgeries, a good counterfeit detection effect can be achieved.
[0090] As one implementation method in this embodiment, the process of S1 includes:
[0091] Video data is extracted in frames at a fixed sampling rate and divided into non-overlapping time periods;
[0092] The audio spectrum is segmented along the time axis using a Hamming window.
[0093] The number of sub-spectrum segments is aligned with the number of video time segments.
[0094] As one implementation method in this embodiment, the video feature extraction process in S2 includes:
[0095] Perform horizontal and vertical bidirectional slicing operations on video frames;
[0096] Multi-scale strip convolution is used to aggregate spatiotemporal information;
[0097] An attention map was generated using analysis of variance to highlight the micro-movements of the eyes and the distortion of the lips.
[0098] Specifically, such as Figure 2 and Figure 3 As shown, the video preprocessing process includes:
[0099] In the video data preprocessing process, the VideoFileClip and AudioFileClip functions from the moviepy library are first used to separate audio and video. Then, the OpenCV library is used to extract frames from the video. Next, the RetinaFace model is applied for face detection, accurately extracting face regions from each frame of the video. Then, based on the video's duration, it is segmented into a sequence of independent images containing faces. This allows the model to ignore irrelevant background information and focus on detecting forgery in face regions, thereby enhancing the model's performance in distinguishing fake videos.
[0100] Specifically, such as Figure 4 As shown, the audio preprocessing process includes:
[0101] In the audio data preprocessing stage, Mel frequency-scale dynamic filtering technology was employed. By applying the Mel frequency conversion algorithm, a frequency domain feature extraction mechanism consisting of 50 equally spaced bandpass filters was constructed, as shown in the following formula:
[0102] ;
[0103] ;
[0104] in, For frequencies in the Mel frequency scale, The frequency is Hertz;
[0105] and These represent the highest and lowest frequency components of the signal that the bandpass filter allows to pass, respectively. The time coordinates of the audio signal. This represents the time-domain response of an ideal bandpass filter. These are the time sampling points of the filter, used to generate the filter's time-domain response.
[0106] Subsequently, by randomly shielding several consecutive filter channels, the shielding length... The values are determined randomly. For feature generation, a Hamming window weighting method was used, combined with one-dimensional convolution operations, ultimately outputting a time-frequency energy distribution map containing 50 channels.
[0107] As one implementation method in this embodiment, the feature alignment process in S3 includes:
[0108] Map audio feature time steps to video time series into a shared latent space;
[0109] Projected onto a unified representation space through a fully connected layer;
[0110] The first fusion feature is generated by splicing along the channel dimension.
[0111] Specifically, such as Figure 5 As shown, the multimodal feature extraction process includes:
[0112] In the multimodal feature extraction stage, video feature extraction employs a multi-scale spatiotemporal convolutional network, which separates spatial and motion features using horizontally and vertically separable convolutional kernels, as shown in the following formula:
[0113] ;
[0114] ;
[0115] ;
[0116] in, It represents multi-scale video motion features and is used to enhance the spatiotemporal coupling relationship of motion trajectories. and These represent the lateral motion trajectory and vertical facial expression changes extracted after depthwise separable convolution, respectively. This represents the spatiotemporal feature vector of the input video.
[0117] The extracted multi-scale video motion feature maps are processed through spatial self-attention, and the feature variance mask is calculated using the following formula:
[0118] ;
[0119] ;
[0120] ;
[0121] in, Represents a binary mask matrix. The number of channels in the motion feature map. This represents the mean of the motion feature map along the channel dimension. Represents the input feature map, This means that the variance vector is calculated along the channel dimension for each video position, and is dynamically selected based on the height of the feature map. Threshold as The formula is as follows:
[0122] ;
[0123] Where k is used to dynamically determine the threshold. The parameters are used to select the H values with the largest variance in the feature map as thresholds. Let H represent the H-th largest eigenvalue, used to generate a binary mask matrix I. This mask suppresses low-variance regions, thereby focusing on micro-expression anomalies in the lips.
[0124] Audio feature extraction uses the following methods: Output 50-channel frequency domain feature map The process involves SELU activation, skip connections, and max pooling in sequence, as shown in the following formula:
[0125] ;
[0126] in, Indicates the enhanced audio features, For activation function, This represents the learnable parameter matrix. After obtaining the enhanced audio features, the audio feature nodes are expanded to the video frame rate using linear projection, as shown in the following formula:
[0127] ;
[0128] in This represents the audio features aligned with the video features. It is a fully connected layer. This represents the audio characteristics of the input.
[0129] The entire process is carried out through spatiotemporal feature coupling, and finally the aligned video and audio features are obtained, which can effectively distinguish the inconsistencies in texture features in the spatial and temporal domains between real and fake videos.
[0130] As one implementation method in this embodiment, the working process of the multi-head additive cross-attention module in S3 includes:
[0131] The query vector is split into multiple attention heads, and weights are calculated independently.
[0132] Attention weights for each head are weighted and fused with value vectors;
[0133] Integrate original features and aggregated features through residual connections.
[0134] Specifically, the multimodal feature fusion process includes:
[0135] In the multimodal feature fusion stage, based on the multi-head additive cross-modal attention mechanism, fusion-level features from audio to vision and from vision to audio are generated.
[0136] Taking audio-to-video fusion-level features as an example, with visual features as the query vector and audio features as key-value pairs, the cross-modal correlation is calculated using the following formula:
[0137] ;
[0138] in, For audio query vectors, For video key vectors, It is a video value vector. All are multi-projection weight matrices. Indicates audio features, Represent video features, then... Split into There are 1 attention head, and each head calculates its attention weight independently, using the following formula:
[0139] ;
[0140] ;
[0141] in, This means splitting the feature channel C into H independent subspaces. This is the learnable parameter vector for the h-th head, used for dynamically weighting different dimensions of the query vector. This represents the attention log score of the h-th attention head. This represents the weight generated by the h-th attention head. Let Query represent the query vector of the h-th attention head, which is generated by linear projection of audio features.
[0142] The attention weights obtained for each head The aggregated features are obtained by weighted fusion with the video value vector V. The formula is shown below:
[0143] ;
[0144] Finally, through two layers The activated fully connected network is used to perform a non-linear transformation on the attention output to obtain the enhanced features. and with original audio features Features after aggregation Residual connections are performed to obtain the final fused features. :
[0145] ;
[0146] ;
[0147] in, It is a learnable weight matrix used to linearly transform the element-wise product of the video key vector and the attention weights.
[0148] As one implementation method in this embodiment, the four-category forgery types in S4 include:
[0149] Audiovisual double authenticity, audiovisual double deception, video forgery with real audio, real video with forged audio;
[0150] The classification decision adopts a multi-granularity discrimination mechanism to integrate audio binary classification, video binary classification and multimodal fusion branch output.
[0151] Specifically, the counterfeit detection process includes:
[0152] In the forgery detection phase, visual and audio unimodal classifiers are trained to obtain the binary forgery probability of audio and video outputs. and The formula is as follows:
[0153] ;
[0154] ;
[0155] in, and These represent the probabilities of video and audio forgery obtained through reasoning, respectively.
[0156] Audio and visual features are processed through a multi-head additive cross-attention mechanism to interactively generate fused features of audio→visual and visual→audio, which are then mapped to auxiliary decision signals. and The formula is as follows:
[0157] ;
[0158] ;
[0159] Finally, the five-dimensional feature vectors are dynamically weighted and fused to obtain the final four-class classification probabilities:
[0160] ;
[0161] ;
[0162] in, It is a weight matrix for dynamically weighted fusion of five-dimensional feature vectors. It is a fused feature vector obtained by splicing the original audio features and video features.
[0163] In this embodiment, the FakeAVCeleb dataset is used as an example dataset. FakeAVCeleb is an open-source multimodal forged video dataset that is widely used in the field of deepfake detection.
[0164] As one implementation method in this embodiment, the process of processing video data to obtain a face region image in a single frame, and constructing a 5-tuple data using a data loader, includes:
[0165] Output data structure: Returns the address of the processed frame file, 40 frame image vectors, 4-second audio vector, four-category labels, video label, and audio label. The processed frame file contains consecutive frames with the same number of frames.
[0166] Specifically, such as Figure 1 As shown, the processing steps in the video data preprocessing module include:
[0167] 1.1 Frame Extraction:
[0168] The specific steps for slicing the input video using the OpenCV library are as follows:
[0169] Load the video file and parse its frame rate and total number of frames, then dynamically divide the sampling interval based on the total number of frames:
[0170] For the 60-70 frame sample: it is divided into 6 sub-intervals, with 6 frames taken from each of the first 6 intervals and 4 frames taken from the last interval.
[0171] For the 70-80 frame sample: it is divided into 7 sub-intervals, with 5 frames taken from each of the first 7 intervals and 5 frames taken from the last interval.
[0172] For the 80-90 frame sample: it is divided into 8 sub-intervals, with 4 frames taken from each of the first 8 intervals and 4 frames taken from the last interval.
[0173] For the 90-100 frame samples: divide them into 9 sub-intervals, take 4 frames from each of the first 9 intervals, and take 4 frames from the last interval.
[0174] For samples with more than 100 frames: divide them into 10 sub-intervals, and take 4 frames from each interval.
[0175] The above sampling operations must also meet the following conditions:
[0176] (1) When the number of valid frames in a sub-interval is greater than or equal to the number of sampled frames, non-repeating frames should be randomly selected.
[0177] (2) When the number of valid frames in a sub-interval is less than the number of sampled frames, repeated sampling should be allowed.
[0178] (3) For invalid intervals, they need to be filled with all-zero tensors.
[0179] As one implementation method in this embodiment, the process of jointly optimizing cosine similarity and multi-task loss includes:
[0180] 2.1 Cross-modal cosine contrast loss:
[0181] The formula for contrast loss is as follows:
[0182] ;
[0183]
[0184] in, The loss function is used to measure the degree of matching between video and audio features. For batches, Used to indicate whether the video and audio match in the i-th sample. Let i be the video feature vector of the i-th sample. Let be the audio feature vector of the i-th sample. This represents the cosine similarity between video and audio features. Then, boundary parameters and a one-sided penalty mechanism are set to ensure that the similarity of anomalous samples remains below a predetermined threshold, effectively avoiding over-optimization of real samples.
[0185] 2.2 Single-modal binary classification loss:
[0186] The unimodal binary classification loss between samples is constructed as follows:
[0187]
[0188]
[0189] in, and These represent the probabilities of video and audio forgery predicted independently by the model for a single sample, respectively. The binary classification loss value for video modalities. The binary classification loss value for audio modalities. Let i be the real video label for the i-th sample. Let be the true audio label for the i-th sample.
[0190] 2.3 Four-class cross-entropy loss:
[0191] The formula for constructing the quaternary cross-entropy loss between samples is as follows:
[0192]
[0193] in, The cross-entropy loss for four-class classification is a loss function used to measure the difference between the model's four-class classification results and the true labels. To represent the true label value of the i-th sample in category c, For category indexing, the combination of video and audio is mapped to four categories: Category A, where both video and audio are real; Category B, where both video and audio are fake; Category C, where the video is fake but the audio is real; and Category D, where the video is real but the audio is fake.
[0194] 2.4 Joint Optimization Objective:
[0195] The total losses are as follows:
[0196] ;
[0197] in, All of these represent the loss weight hyperparameters in multi-task learning. During the pre-training phase, the parameters of the video residual compression network are frozen to prioritize cross-modal alignment optimization. In the fine-tuning phase, the model will be able to grasp the feature differences between real videos and audio and fake videos and audio, thus providing strong support for subsequent fake video identification learning.
[0198] 2.5 Results Analysis:
[0199] Hyperparameters of the model Set to 1, 1, and 0.3, and in Figure 6 The figure shows the trend of detection accuracy as the number of training iterations increases on the FakeAVCeleb dataset. Although there are slight fluctuations in the middle, the overall trend is a continuous increase and eventually stabilizes at around 99%, indicating that the model performs well in generalization on the test set.
[0200] like Figure 7 The graph shows the trend of the detection AUC value on the FakeAVCeleb dataset as the number of training iterations increases. As can be seen from the graph, the long-term trend of AUC is a continuous increase, approaching 99.9%, indicating that the model has extremely high reliability in classifying positive and negative samples.
[0201] like Figure 8 The figure shows t-SNE visualizations for four types of samples. The four colors in the figure represent different sample types, and their specific meanings are as follows:
[0202] Purple (RV-RA): Positive samples with a label value of 0;
[0203] Blue (FV-FA): Samples where both video and audio have been tampered with, with a label value of 1;
[0204] Green (FV-RA): Samples with authentic audio and manipulated video, labeled with a value of 2;
[0205] Yellow (RV-FA): Samples with authentic video but altered audio, labeled with a value of 3.
[0206] Based on this, this invention provides a multimodal forged video detection method based on a multi-head additive cross-attention mechanism. By constructing a video feature extractor and an audio feature extractor, spatiotemporal features of the video modality and spectral features of the audio modality are extracted respectively. During the cross-modal interaction stage, an asymmetric projection mechanism is employed to map the dynamic temporal changes of the audio to a latent space aligned with the video dimension. After applying temporal position encoding to the video features, they are input together with the audio features into the multi-head additive cross-attention module, achieving deep interaction and fusion of cross-modal features and effectively capturing texture anomalies and audio-video temporal mismatch features in forged videos. Cosine similarity is used to calculate the matching loss between the audio and video outputs, comprehensively considering the supervised learning of the four-class classification detection main task and the collaborative supervision mechanism of the audio-video sub-task detection, achieving high-precision detection in complex forged scenarios. Based on this, the present invention overcomes the shortcomings of traditional single-modal detection methods in capturing audio and video temporal mismatch features, and solves the problems of weak model generalization ability and long training cycle caused by cross-modal feature interaction redundancy in the prior art. It realizes real-time and accurate identification of multimodal attacks such as deepfakes and voice replacement in open environments, and provides a high-precision solution for deepfake identification.
[0207] Example 2
[0208] Based on the same general inventive concept, this invention also provides a multimodal forged video detection system based on a multi-head additive cross-attention mechanism. The multimodal forged video detection system based on a multi-head additive cross-attention mechanism provided by this invention is described below. The multimodal forged video detection system based on a multi-head additive cross-attention mechanism described below can be referred to in conjunction with the multimodal forged video detection method based on a multi-head additive cross-attention mechanism described above. The system includes:
[0209] The video processing module is used for audio-visual separation and 68-point facial key point detection;
[0210] The feature extraction module is used to extract audio mel spectrum features and video spatiotemporal features, and to achieve feature dimensionality reduction through a residual compression network;
[0211] A cross-modal fusion module is used to achieve cross-modal feature interaction based on a multi-head attention mechanism;
[0212] The forgery detection module is used for joint optimization of multi-task losses and dynamic threshold classification decisions.
[0213] As one implementation method in this embodiment, the video processing module includes:
[0214] Multimodal separation unit: Used to separate video streams and audio streams, generating independent temporal features.
[0215] Face detection unit: Based on the RetinaFace model, it detects 68 facial key points and extracts geometric dynamic features.
[0216] As one implementation method in this embodiment, the data construction unit includes:
[0217] The dataset was divided into four classes: A (real video + real audio), B (fake video + fake audio), C (fake video + real audio), and D (real video + fake audio). The training and testing sets were split in a 3:1 ratio, and frequency domain masking and temporal rearrangement were used to augment the data.
[0218] As one implementation method in this embodiment, the feature extraction module includes:
[0219] Audio feature extraction unit: 50 Mel-scale bandpass filter banks are generated using SincConv layers, and the frequency domain features are nonlinearly reduced using a hierarchical residual compression network;
[0220] Video feature extraction unit: Based on horizontal-vertical separable convolution kernels, it separates spatial texture and motion trajectory features, and suppresses background noise areas through variance-driven spatial self-attention mechanism, focusing on abnormal micro-expressions of the lips.
[0221] As one implementation method in this embodiment, the spatiotemporal feature processing module includes:
[0222] Timing alignment unit: A linear projection matrix is used to extend the 4-node audio features to 10 nodes to achieve time synchronization with the video frame rate and control the timing error within 2 frames.
[0223] As one implementation method in this embodiment, the feature fusion module includes:
[0224] Parameter Freezing Unit: Freezes the parameters of the audio SincConv filter bank and the video residual network during the cross-modal fusion stage;
[0225] Cross-modal attention unit: Based on the additive cross-attention mechanism, it calculates the attention weights for any two modalities;
[0226] Residual fusion unit: Integrates cross-modal features through layer-normalized residual connections.
[0227] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.
[0228] Through the above implementation methods, the problem of multimodal video forgery detection based on multi-head additive cross-attention mechanism in related technologies is solved. Based on this, the present invention overcomes the shortcomings of traditional single-modal detection methods in capturing audio and video temporal mismatch features, and solves the problems of weak model generalization ability and long training cycle caused by cross-modal feature interaction redundancy in the prior art. It realizes real-time and accurate identification of multimodal attacks such as deepfakes and voice replacement in open environments, and provides a high-precision solution for deepfake identification.
[0229] It should be understood that the multimodal spoofing video detection system based on the multi-head additive cross-attention mechanism provided in this embodiment of the invention possesses all the advantages of the multimodal spoofing video detection method based on the multi-head additive cross-attention mechanism provided in the above embodiments.
[0230] Example 3
[0231] In this embodiment, a computer terminal device is provided, including:
[0232] One or more processors;
[0233] A memory, coupled to the processor, for storing one or more programs;
[0234] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods in the above embodiments.
[0235] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in the above embodiments.
[0236] In this embodiment, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the methods described in the above embodiments.
[0237] The aforementioned program can run on a processor or be stored in memory (or a computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0238] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented using different modules, and different steps can be implemented using different modules.
[0239] This embodiment provides such an apparatus or system. The system, referred to as a multimodal forged video detection system based on a multi-head additive cross-attention mechanism, includes:
[0240] The preprocessing module is used to decompose the input video stream into single-frame image sequences and locate the face region, while extracting the audio waveform to generate a Mel spectrogram;
[0241] The feature extraction module is used to extract spatiotemporal features by sampling video frames in blocks using a 3D convolutional network, and to generate audio spectral features based on a filter bank.
[0242] A cross-modal alignment module is used to map audio features to a latent space aligned with the video dimension;
[0243] The attention fusion module is used to perform bidirectional multi-head additive cross-attention calculation on video features and audio features with applied temporal coding, generate audio-dominant features and video-dominant features, and cascade and fuse the original features with bidirectional interactive features.
[0244] The classification decision module is used to construct the audio and video cosine similarity loss function to constrain the feature distribution, and output the four-class forgery type probability after dynamic weighting and optimization of the fused features.
[0245] As one implementation method in this embodiment, the preprocessing module includes:
[0246] A video segmentation unit is used to extract video frames at a fixed sampling rate and divide them into non-overlapping time segments;
[0247] The spectrum slicing unit is used to slice the Mel spectrogram along the time axis using a Hamming window, so that the number of sub-spectral segments is aligned with the video time period.
[0248] As one implementation method in this embodiment, the feature extraction module includes:
[0249] The bidirectional differential unit is used to perform horizontal and vertical slicing operations on video frames and calculate inter-frame motion differences;
[0250] The variance analysis unit is used to aggregate spatiotemporal information through multi-scale strip convolution to generate attention maps that highlight eye micro-movements and lip distortion features.
[0251] As one implementation method in this embodiment, the cross-modal alignment module includes:
[0252] The spatial mapping unit is used to map the audio feature time step to the video time series into a shared latent space;
[0253] The feature splicing unit is used to project onto the unified representation space through the fully connected layer and splice along the channel dimension to generate the first fused feature.
[0254] As one implementation method in this embodiment, the attention fusion module includes:
[0255] Multi-head computation unit, used to split the query vector into multiple attention heads and calculate weights independently;
[0256] The residual integration unit is used to weight and fuse the attention weights and value vectors of each head, and integrate the original features and aggregated features through residual connections.
[0257] As one implementation method in this embodiment, the classification decision module includes:
[0258] A multi-granularity discrimination unit is used to integrate the outputs of the audio binary classification branch, the video binary classification branch, and the multimodal fusion branch;
[0259] The four-class output unit is used to generate four types of probabilities: audiovisual double true, audiovisual double false, video fake-audio true, and video true-audio fake.
[0260] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.
[0261] The above implementation method solves the problem of multimodal forged video detection based on multi-head additive cross-attention mechanism in related technologies, thereby ensuring that the problems existing in the prior art are resolved.
[0262] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A multimodal forged video detection method based on a multi-head additive cross-attention mechanism, characterized in that, Includes the following steps: S1. Multimodal data preprocessing: Video modal processing includes decomposing the video stream into a sequence of single-frame images, locating the face region, and cropping the face image; Audio modal processing includes extracting audio waveforms to generate Mel spectrograms and then slicing them into segments; S2. Multimodal feature extraction: Video feature extraction employs a 3D convolutional network for block sampling, extracting spatiotemporal features and calculating inter-frame motion differences; Audio feature extraction is based on generating spectral features using a filter bank, combined with a residual network to capture contextual information; The video feature extraction process in S2 includes: Perform horizontal and vertical bidirectional slicing operations on video frames; Multi-scale strip convolution is used to aggregate spatiotemporal information; An attention map was generated using analysis of variance to highlight the micro-movements of the eyes and the distortion of the lips. S3. Cross-modal feature alignment and fusion: Map audio features to a latent space aligned with the video dimension; After applying temporal encoding to video features, they are input into a multi-head additive cross-attention module along with audio features: Audio-dominant features are generated using audio features as query vectors and video features as key-value pairs. Video-dominant features are generated using video features as query vectors and audio features as key-value pairs. Concatenate and fuse the original features with the bidirectional interactive features; The working process of the multi-head additive cross-attention module in S3 includes: The query vector is split into multiple attention heads, and weights are calculated independently. Attention weights for each head are weighted and fused with value vectors; Integrate original features and aggregated features through residual connections; Using visual features as the query vector and audio features as key-value pairs, the cross-modal correlation is calculated using the following formula: ； in, For audio query vectors, For video key vectors, It is a video value vector. All are multi-projection weight matrices. Indicates audio features, Represent video features, then... Split into There are 1 attention head, and each head calculates its attention weight independently, using the following formula: ； ； in, This means splitting the feature channel C into H independent subspaces. This is the learnable parameter vector for the h-th head, used for dynamically weighting different dimensions of the query vector; This represents the logarithmic score of the attention head at the h-th attention head. This represents the weight generated by the h-th attention head; Let Query represent the query vector of the h-th attention head, which is generated by linear projection of audio features; The attention weights obtained for each head The aggregated features are obtained by weighted fusion with the video value vector V. The formula is shown below: ； Finally, through two layers The activated fully connected network is used to perform a non-linear transformation on the attention output to obtain the enhanced features. and with original audio features Features after aggregation Residual connections are performed to obtain the final fused features. : ； ； in, It is a learnable weight matrix used to linearly transform the element-wise product of the video key vector and the attention weights; S4. Comparative Learning and Classification Decision-Making: Construct a cosine similarity loss function for audio and video to constrain feature distribution; The fused features are dynamically weighted, optimized, and compressed over time to output the probability of four types of forgery.
2. The method according to claim 1, characterized in that, The process S1 includes: Video data is extracted in frames at a fixed sampling rate and divided into non-overlapping time periods; The audio spectrum is segmented along the time axis using a Hamming window. The number of sub-spectrum segments is aligned with the number of video time segments.
3. The method according to claim 1, characterized in that, The feature alignment process in S3 includes: Map audio feature time steps to video time series into a shared latent space; Projected onto a unified representation space through a fully connected layer; The first fusion feature is generated by splicing along the channel dimension.
4. The method according to claim 1, characterized in that, The four types of forgery in S4 include: Audiovisual double authenticity, audiovisual double deception, video forgery with real audio, real video with forged audio; The classification decision adopts a multi-granularity discrimination mechanism to integrate audio binary classification, video binary classification and multimodal fusion branch output.
5. A multimodal forged video detection system based on a multi-head additive cross-attention mechanism, characterized in that, The system for implementing the method according to any one of claims 1-4 comprises: The preprocessing module is used to decompose the input video stream into single-frame image sequences and locate the face region, while extracting the audio waveform to generate a Mel spectrogram; The feature extraction module is used to extract spatiotemporal features by sampling video frames in blocks using a 3D convolutional network, and to generate audio spectral features based on a filter bank. A cross-modal alignment module is used to map audio features to a latent space aligned with the video dimension; The attention fusion module is used to perform bidirectional multi-head additive cross-attention calculation on video features and audio features with applied temporal coding, generate audio-dominant features and video-dominant features, and cascade and fuse the original features with bidirectional interactive features. The classification decision module is used to construct the audio and video cosine similarity loss function to constrain the feature distribution, and output the four-class forgery type probability after dynamic weighting and optimization of the fused features.
6. The system according to claim 5, characterized in that, The preprocessing module includes: A video segmentation unit is used to extract video frames at a fixed sampling rate and divide them into non-overlapping time segments; The spectrum slicing unit is used to slice the Mel spectrogram along the time axis using a Hamming window, so that the number of sub-spectral segments is aligned with the video time period.
7. A computer terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal forged video detection method based on the multi-head additive cross-attention mechanism as described in any one of claims 1-4.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal forged video detection method based on the multi-head additive cross-attention mechanism as described in any one of claims 1-4.