Multi-modal forged video detection method based on multi-head addition cross attention mechanism

Abstract

The invention discloses a multi-mode counterfeit video detection method based on a multi-head addition cross attention mechanism, and belongs to the technical field of video counterfeit detection. The method comprises the following steps: preprocessing a video stream, decomposing a single-frame positioning face, and extracting an audio to generate a Mel spectrogram slice; the 3D convolutional network extracts video spatio-temporal features and motion differences, and the filter bank extracts audio features in combination with the residual network; audio features are mapped to a video alignment space through asymmetric projection, the video features are subjected to bidirectional interaction with an audio input multi-head addition cross attention module after being subjected to time sequence coding, and audio dominant and video dominant features are generated and are cascaded and fused with original features; and constructing cross-modal similarity loss constraint feature distribution, and fusing feature dynamic weighting and time sequence compression to output four classification probabilities of audio-visual double true, audio-visual double pseudo, video pseudo-audio true and video pseudo-audio pseudo. The multi-mode counterfeiting recognition precision is improved, and texture abnormity and audio and video mismatch features are captured.

Description

Technical Field

[0001] The present invention belongs to the technical field of video forgery detection, and in particular relates to a multimodal forged video detection method based on a multi-head addition cross attention mechanism. Background Art

[0002] With the rapid development of AIGC technology and the maturity of existing forgery methods, more and more forgery traces are becoming difficult to detect, making the detection of forged videos increasingly difficult. Thanks to the tremendous success of generative techniques (VAEs) and generative adversarial networks (GANs), face manipulation has become an emerging and hot research topic in recent years. Face swapping or partial forgery techniques, such as deepfakes and face swapping (FaceSwap), allow people without specialized skills or equipment to generate high-quality fake faces. Therefore, developing effective methods for detecting face forgery has become particularly important.

[0003] Current research has largely focused on single-modal forgery detection methods, overly focusing on video information while neglecting relevant audio information and the subtle connections between video and audio, which are crucial for video forgery detection. Existing forgery detection methods face challenges such as long training cycles, low detection accuracy, poor generalization, large model parameters, and long inference times. As forgery technology continues to evolve and become increasingly complex, traditional detection methods are increasingly unable to meet practical needs. Therefore, new, more efficient forgery detection technologies are urgently needed. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a multimodal forged video detection method based on a multi-head addition cross attention mechanism to solve the problems existing in the above-mentioned prior art.

[0005] In a first aspect, to achieve the above-mentioned objectives, the present invention provides a multimodal forged video detection method based on a multi-head additive cross attention mechanism, comprising the following steps:

[0006] S1. Multimodal data preprocessing:

[0007] Video modality processing includes decomposing the video stream into a single-frame image sequence, locating the face area and capturing the face image;

[0008] Audio modality processing includes extracting audio waveforms to generate Mel-spectrograms and slicing them into segments;

[0009] S2. Multimodal feature extraction:

[0010] Video feature extraction uses 3D convolutional network block sampling to extract spatiotemporal features and calculate inter-frame motion differences;

[0011] Audio feature extraction generates spectral features based on filter banks and combines them with residual networks to capture contextual information;

[0012] S3. Cross-modal feature alignment and fusion:

[0013] Mapping audio features to a latent space aligned with video dimensions;

[0014] After applying temporal encoding to the video features, they are input into the multi-head addition cross attention module with the audio features:

[0015] Generate audio dominant features using audio features as query vectors and video features as key-value pairs;

[0016] Generate video dominant features using video features as query vectors and audio features as key-value pairs;

[0017] Cascade fusion of original features and bidirectional interaction features;

[0018] S4. Contrastive learning and classification decision:

[0019] Construct audio and video cosine similarity loss function to constrain feature distribution;

[0020] The fused features are dynamically weighted, optimized, and time-series compressed to output four-category forgery type probabilities.

[0021] Optionally, the process of S1 includes:

[0022] The video data is framed at a fixed sampling rate and divided into non-overlapping time segments;

[0023] The audio spectrum is segmented and sliced ​​along the time axis using a Hamming window;

[0024] The number of sub-spectral segments is aligned with the number of video time segments.

[0025] Optionally, the process of extracting video features in S2 includes:

[0026] Perform horizontal and vertical bidirectional slicing operations on video frames;

[0027] Use multi-scale strip convolution to aggregate spatiotemporal information;

[0028] The attention map generated by variance analysis highlights the features of eye micro-movement and lip distortion.

[0029] Optionally, the feature alignment process in S3 includes:

[0030] Mapping audio feature time steps and video time series into a shared latent space;

[0031] Projected to a unified representation space through a fully connected layer;

[0032] The first fusion feature is generated by concatenating along the channel dimension.

[0033] Optionally, the working process of the multi-head addition cross attention module in S3 includes:

[0034] Split the query vector into multiple attention heads and calculate the weights independently;

[0035] Weighted fusion of each head’s attention weight and value vector;

[0036] The original features and aggregated features are integrated through residual connections.

[0037] Optionally, the four categories of forgery in S4 include:

[0038] Audiovisual double truth, audiovisual double fake, video fake - audio real, video real - audio fake;

[0039] The classification decision adopts a multi-granularity discrimination mechanism to integrate audio binary classification, video binary classification and multimodal fusion branch output.

[0040] In a second aspect, the present invention further provides a multimodal forged video detection system based on a multi-head additive cross attention mechanism, which is used to implement a multimodal forged video detection method based on a multi-head additive cross attention mechanism. The system comprises:

[0041] The preprocessing module is used to decompose the input video stream into a single-frame image sequence and locate the face area, while extracting the audio waveform to generate a Mel-spectrogram;

[0042] The feature extraction module is used to extract spatiotemporal features by sampling video frames in blocks through a 3D convolutional network and generate audio spectrum features based on the filter bank;

[0043] A cross-modal alignment module to map audio features to a latent space aligned with the video dimensions;

[0044] The attention fusion module is used to perform bidirectional multi-head addition cross-attention calculation on the video features and audio features with temporal encoding, generate audio dominant features and video dominant features, and cascade fuse the original features with the bidirectional interaction features;

[0045] The classification decision module is used to construct the audio and video cosine similarity loss function to constrain the feature distribution, dynamically weighted and optimized the fusion features, and output the four-category forgery type probability.

[0046] Optionally, the preprocessing module includes:

[0047] A video segmentation unit, configured to capture video frames at a fixed sampling rate and divide the frames into non-overlapping time segments;

[0048] The spectrum slicing unit is used to slice the Mel spectrum graph along the time axis using a Hamming window, so that the number of sub-spectrum segments is aligned with the video time period.

[0049] In a third aspect, the present invention further provides a computer terminal device, comprising:

[0050] one or more processors;

[0051] a memory, coupled to the processor, for storing one or more programs;

[0052] When the one or more programs are executed by the one or more processors, the one or more processors implement a multimodal forged video detection method based on a multi-head addition cross attention mechanism.

[0053] In a fourth aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements a multimodal forged video detection method based on a multi-head addition cross attention mechanism.

[0054] Compared with the prior art, the present invention has the following advantages and technical effects:

[0055] The present invention provides a multimodal forged video detection method based on a multi-head additive cross-attention mechanism. The present invention realizes the latent space alignment of audio features and video dimensions through asymmetric projection, and uses the multi-head additive cross-attention mechanism to complete cross-modal deep interaction, effectively capturing the texture anomalies and audio and video temporal mismatch features of forged videos. The cosine similarity loss function is used to constrain the closeness of homologous features and the separability of heterologous features, combined with dynamic gradient weighting to optimize the four-classification decision process. This method significantly improves the recognition accuracy of multimodal forgery attacks in open environments, overcomes the defect of single-modal detection that cannot capture temporal mismatch features, and reduces the problems of weak model generalization ability and long training cycle caused by cross-modal interaction redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0057] Figure 1 is a flow chart of an embodiment of the present invention;

[0058] Figure 2 This is a video preprocessing flow chart of an embodiment of the present invention;

[0059] Figure 3 This is a structural diagram of a video feature extraction module according to an embodiment of the present invention;

[0060] Figure 4 This is a structural diagram of an audio feature extraction module according to an embodiment of the present invention;

[0061] Figure 5 This is a structural diagram of a multi-head addition cross attention module according to an embodiment of the present invention;

[0062] Figure 6 Schematic diagram of the changing trend of the detection accuracy of an embodiment of the present invention on the FakeAVCeleb test set as the number of training iterations increases;

[0063] Figure 7 This is a schematic diagram showing the changing trend of the detection AUC on the FakeAVCeleb test set as the number of training iterations increases;

[0064] Figure 8 In this embodiment of the present invention, the distribution of the final classification layer features in two-dimensional space is analyzed by the t-SNE nonlinear dimensionality reduction algorithm, where t-SNE is a commonly used nonlinear dimensionality reduction algorithm that is widely used in the visualization of high-dimensional data. DETAILED DESCRIPTION

[0065] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0066] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] In light of the shortcomings of existing methods, this paper proposes a multimodal video forgery detection method and system based on a multi-head additive cross-attention mechanism. This method extracts spatiotemporal and spectral features using video and audio feature extractors and achieves feature alignment through an asymmetric projection mechanism. The multi-head additive cross-attention module deeply interacts and fuses features to identify anomalous textures and temporal mismatches in forged videos, thereby accurately identifying video forgeries.

[0068] Example 1

[0069] like Figure 1 As shown, this embodiment provides a multimodal forged video detection method based on a multi-head addition cross attention mechanism, including:

[0070] S1. Multimodal data preprocessing:

[0071] Video modality processing includes decomposing the video stream into a single-frame image sequence, locating the face area and capturing the face image;

[0072] Audio modality processing includes extracting audio waveforms to generate Mel-spectrograms and slicing them into segments;

[0073] S2. Multimodal feature extraction:

[0074] Video feature extraction uses 3D convolutional network block sampling to extract spatiotemporal features and calculate inter-frame motion differences;

[0075] Audio feature extraction generates spectral features based on filter banks and combines them with residual networks to capture contextual information;

[0076] S3. Cross-modal feature alignment and fusion:

[0077] Mapping audio features to a latent space aligned with video dimensions;

[0078] After applying temporal encoding to the video features, they are input into the multi-head addition cross attention module with the audio features:

[0079] Generate audio dominant features using audio features as query vectors and video features as key-value pairs;

[0080] Generate video dominant features using video features as query vectors and audio features as key-value pairs;

[0081] Cascade fusion of original features and bidirectional interaction features;

[0082] S4. Contrastive learning and classification decision:

[0083] Construct audio and video cosine similarity loss function to constrain feature distribution;

[0084] The fused features are dynamically weighted, optimized, and time-series compressed to output four-category forgery type probabilities.

[0085] Specifically, multimodal data preprocessing includes video modality processing, which decomposes the video into image sequences, rotates, crops and normalizes them, and uses facial key point detection to capture facial images; audio modality processing involves extracting audio waveforms, generating Mel-spectrograms, and performing time slicing.

[0086] Multimodal feature extraction extracts video and audio features through 3D convolutional networks and SincNet filter banks respectively.

[0087] Cross-modal feature alignment and fusion calculate feature correlations through asymmetric projection and multi-head addition cross-attention modules.

[0088] Contrastive learning and classification decision-making constrain the feature distribution through the cosine similarity loss function, and optimize the feature space through dynamic gradient weighting and temporal global average pooling to output the forgery type.

[0089] Specifically, by combining the subtle relationship between video and audio to perform forgery identification, a good forgery identification effect can be achieved.

[0090] As an implementation method in this embodiment, the process of S1 includes:

[0091] The video data is framed at a fixed sampling rate and divided into non-overlapping time segments;

[0092] The audio spectrum is segmented and sliced ​​along the time axis using a Hamming window;

[0093] The number of sub-spectral segments is aligned with the number of video time segments.

[0094] As an implementation method in this embodiment, the process of extracting video features in S2 includes:

[0095] Perform horizontal and vertical bidirectional slicing operations on video frames;

[0096] Use multi-scale strip convolution to aggregate spatiotemporal information;

[0097] The attention map generated by variance analysis highlights the features of eye micro-movement and lip distortion.

[0098] Specifically, such as Figure 2 and Figure 3 As shown in Figure 2, the video preprocessing process includes:

[0099] During video data preprocessing, the video data is first separated from the audio using the VideoFileClip and AudioFileClip functions in the moviepy library. OpenCV is then used to extract frames from the video. Next, the RetinaFace model is applied for face detection, accurately extracting the face region from each frame. The video is then segmented into separate image sequences containing faces based on their duration. This allows the model to ignore irrelevant background information and focus on detecting forgeries in the face region, thereby enhancing its performance in distinguishing forged videos.

[0100] Specifically, such as Figure 4 As shown in Figure 1, the audio preprocessing process includes:

[0101] In the audio data preprocessing stage, the Mel frequency scale dynamic filtering technology is adopted. By applying the Mel frequency conversion algorithm, a frequency domain feature extraction mechanism consisting of 50 equally spaced bandpass filters is constructed. The formula is as follows:

[0102] ;

[0103] ;

[0104] in, is the frequency on the Mel frequency scale, is the frequency in Hertz;

[0105] and They represent the highest and lowest frequency components of the signal allowed to pass by the bandpass filter. is the time coordinate of the audio signal, represents the time domain response of an ideal bandpass filter, is the time sampling point of the filter, which is used to generate the time domain response of the filter.

[0106] Subsequently, by randomly shielding several consecutive filter channels, the shielding length In terms of feature generation, the Hamming window weighting method is adopted, combined with a one-dimensional convolution operation, and finally a time-frequency energy distribution map containing 50 channels is output.

[0107] As an implementation method in this embodiment, the feature alignment process in S3 includes:

[0108] Mapping audio feature time steps and video time series into a shared latent space;

[0109] Projected to a unified representation space through a fully connected layer;

[0110] The first fusion feature is generated by concatenating along the channel dimension.

[0111] Specifically, such as Figure 5 As shown in Figure 2, the multimodal feature extraction process includes:

[0112] In the multimodal feature extraction stage, video feature extraction uses a multi-scale spatiotemporal convolutional network to separate spatial and motion features through horizontally-vertically separable convolution kernels. The formula is as follows:

[0113] ;

[0114] ;

[0115] ;

[0116] in, Represents multi-scale video motion features and is used to enhance the spatiotemporal coupling relationship of motion trajectories. and They represent the horizontal motion trajectory and vertical expression changes extracted after depthwise separable convolution, Represents the input video spatiotemporal feature vector.

[0117] The extracted multi-scale video motion feature map is subjected to spatial self-attention to calculate the feature variance mask. The formula is as follows:

[0118] ;

[0119] ;

[0120] ;

[0121] in, represents the binary mask matrix, is the number of channels of the motion feature map, is the mean of the motion feature map along the channel dimension, represents the input feature map, Indicates that the variance vector is calculated along the channel dimension for each video position, and is dynamically selected based on the height of the feature map. The threshold is , the formula is as follows:

[0122] ;

[0123] Among them, k is used to dynamically determine the threshold The parameter is used to select the H values ​​with the largest variance in the feature map as the threshold. Represents the Hth largest eigenvalue, which is used to generate the binary mask matrix I. The mask suppresses the low variance area, thereby focusing on the abnormal micro-expressions of the lips.

[0124] Audio feature extraction is done by Output 50-channel frequency domain feature map , perform SELU activation, skip connection and maximum pooling in sequence, the formula is as follows:

[0125] ;

[0126] in, represents the enhanced audio features, is the activation function, Represents the learnable parameter matrix. After obtaining the enhanced audio features, linear projection is used to expand the audio feature nodes to the video frame rate. The formula is as follows:

[0127] ;

[0128] in represents the audio features aligned with the video features, is the fully connected layer, Represents the input audio features.

[0129] The entire process is carried out through the coupling of spatiotemporal features, and finally the aligned video and audio features are obtained, which can effectively distinguish the inconsistency of texture features in the spatial and temporal domains between real and forged videos.

[0130] As an implementation method in this embodiment, the working process of the multi-head addition cross attention module in S3 includes:

[0131] Split the query vector into multiple attention heads and calculate the weights independently;

[0132] Weighted fusion of each head’s attention weight and value vector;

[0133] The original features and aggregated features are integrated through residual connections.

[0134] Specifically, the multimodal feature fusion process includes:

[0135] In the multimodal feature fusion stage, audio-to-vision and vision-to-audio fusion-level features are generated based on the multi-head addition cross-modal cross-attention mechanism.

[0136] Taking the fusion-level features from audio to video as an example, we use visual features as query vectors and audio features as key-value pairs to calculate cross-modal correlation. The formula is as follows:

[0137] ;

[0138] in, is the audio query vector, is the video key vector, is the video value vector, are all multi-head projection weight matrices, Represents audio features, Represents the video features, then Split into Each attention head calculates the attention weight independently, and the formula is as follows:

[0139] ;

[0140] ;

[0141] in, Indicates splitting the feature channel C into H independent subspaces, is the learnable parameter vector of the h-th head, which is used to dynamically weight the different dimensions of the query vector. represents the logarithmic score of the h-th attention head. represents the weight generated by the h-th attention head. The query vector Query represents the h-th attention head, which is generated by linear projection of audio features.

[0142] The attention weights obtained by each head Perform weighted fusion with the video value vector V to obtain the aggregated features , the formula is as follows:

[0143] ;

[0144] Finally, through two layers The activated fully connected network performs nonlinear transformation on the attention output to obtain enhanced features and the original audio features Aggregated features Perform residual connection to obtain the final fusion feature :

[0145] ;

[0146] ;

[0147] in, is a learnable weight matrix that linearly transforms the element-wise product of the video key vector and the attention weight.

[0148] As an implementation in this embodiment, the four categories of forgery types in S4 include:

[0149] Audiovisual double truth, audiovisual double fake, video fake - audio real, video real - audio fake;

[0150] The classification decision adopts a multi-granularity discrimination mechanism to integrate audio binary classification, video binary classification and multimodal fusion branch output.

[0151] Specifically, the counterfeit detection process includes:

[0152] In the forgery detection stage, the forgery probability of audio and video output is obtained by training the visual and audio unimodal classifiers. and , the formula is as follows:

[0153] ;

[0154] ;

[0155] in, and They represent the inferred video and audio forgery probabilities respectively.

[0156] The audio and visual features are interactively generated through a multi-head addition cross-attention mechanism to generate audio→visual and visual→audio fusion features, which are mapped into auxiliary decision signals. and , the formula is as follows:

[0157] ;

[0158] ;

[0159] Finally, the five-dimensional feature vectors are dynamically weighted and fused to obtain the final four-category probability:

[0160] ;

[0161] ;

[0162] in, is the weight matrix for dynamic weighted fusion of five-dimensional feature vectors, It is the fused feature vector obtained by concatenating the original audio features and video features.

[0163] In this example, the FakeAVCeleb dataset is used as an example dataset. FakeAVCeleb is an open-source multimodal forged video dataset that is widely used in the field of deepfake detection.

[0164] As an implementation method of this embodiment, the video data is processed to obtain a face region image in a single frame image, and the process of constructing quintuple data through a data loader includes:

[0165] Output data structure: Returns the address of the processed frame file, 40-frame image vectors, 4-second audio vectors, four-category labels, video labels, and audio labels. The processed frame files all contain the same number of consecutive frame images.

[0166] Specifically, such as Figure 1 As shown, the processing process in the video data preprocessing module includes:

[0167] 1.1 Frame extraction:

[0168] Use the OpenCV library to slice the input video. The specific steps are as follows:

[0169] Load the video file and analyze its frame rate and total number of frames, and dynamically divide the sampling interval according to the total number of frames:

[0170] For 60-70 frame samples: Divide into 6 sub-intervals, take 6 frames in each of the first 6 intervals, and take 4 frames in the last interval.

[0171] For 70-80 frame samples: divide it into 7 sub-intervals, take 5 frames from each of the first 7 intervals, and take 5 frames from the last interval.

[0172] For 80-90 frame samples: Divide into 8 sub-intervals, take 4 frames from each of the first 8 intervals, and take 4 frames from the last interval.

[0173] For 90-100 frame samples: divide it into 9 sub-intervals, take 4 frames from each of the first 9 intervals, and take 4 frames from the last interval.

[0174] For samples with more than 100 frames: divide them into 10 sub-intervals, and take 4 frames in each sub-interval.

[0175] The above sampling operations must also meet the following conditions:

[0176] (1) When the number of valid frames in a subinterval is greater than or equal to the number of sampling frames, non-repeated frames should be randomly selected.

[0177] (2) When the number of valid frames in a subinterval is less than the number of sampling frames, repeated sampling should be allowed.

[0178] (3) For invalid intervals, they need to be filled with all-zero tensors.

[0179] As an implementation method in this embodiment, the process of jointly optimizing cosine similarity and multi-task loss includes:

[0180] 2.1 Cross-modal cosine contrast loss:

[0181] The contrast loss formula is as follows:

[0182] ;

[0183]

[0184] in, is a loss function that measures the degree of matching between video and audio features. For batches, Used to indicate whether the video and audio in the i-th sample match. is the video feature vector of the i-th sample, is the audio feature vector of the i-th sample, represents the cosine similarity between video and audio features. Subsequently, boundary parameters and a unilateral penalty mechanism are set to ensure that the similarity of abnormal samples is below a predetermined threshold, effectively avoiding over-optimization of real samples.

[0185] 2.2 Unimodal binary classification loss:

[0186] The unimodal binary classification loss between samples is constructed as follows:

[0187]

[0188]

[0189] in, and Represent the video and audio forgery probabilities predicted by the model for a single sample independently, is the binary classification loss value of the video modality, is the binary classification loss value of the audio modality, is the true video label of the i-th sample, is the true audio label of the i-th sample.

[0190] 2.3 Four-category cross entropy loss:

[0191] The quaternary cross entropy loss formula between constructed samples is as follows:

[0192]

[0193] in, To represent the four-category cross entropy loss, it is used to measure the loss function of the model for the difference between the four-category results and the true labels. To represent the true label value of the i-th sample in category c, For category indexing, the combination of video and audio is mapped into four categories: Category A, which means that both the video and audio are real; Category B, which means that both the video and audio are fake; Category C, which means that the video is fake and the audio is real; Category D, which means that the video is real and the audio is fake.

[0194] 2.4 Joint optimization objectives:

[0195] The total losses are as follows:

[0196] ;

[0197] in, Both represent loss weight hyperparameters in multi-task learning. During pre-training, the parameters of the video residual compression network are frozen to prioritize cross-modal alignment optimization. During fine-tuning, the model will be able to grasp the characteristic differences between real video and audio and forged video and audio, providing strong support for subsequent forged video identification learning.

[0198] 2.5 Results Analysis

[0199] The model's hyperparameters Set to 1, 1, and 0.3, and Figure 6 Figure 2 shows the detection accuracy trend on the FakeAVCeleb dataset as training iterations increase. Despite slight fluctuations, the overall trend continues to rise, ultimately stabilizing at around 99%, demonstrating the model's strong generalization ability on the test set.

[0200] like Figure 7 The following figure shows the trend of the detection AUC value on the FakeAVCeleb dataset as the number of training iterations increases. As can be seen from the figure, the long-term trend of AUC continues to rise and approaches 99.9%, which shows that the model is highly reliable in classifying positive and negative samples.

[0201] like Figure 8 As shown in the figure, a t-SNE visualization of four types of samples is shown. The four colors in the figure represent different sample types, and their specific meanings are as follows:

[0202] Purple (RV-RA): positive sample, label value is 0;

[0203] Blue (FV-FA): Samples where both video and audio have been tampered with, with a label value of 1;

[0204] Green (FV-RA): Sample with authentic audio and tampered video, with a label value of 2;

[0205] Yellow (RV-FA): Samples with authentic video and tampered audio, with a label value of 3.

[0206] Based on this, an embodiment of the present invention provides a multimodal forged video detection method based on a multi-head additive cross-attention mechanism. By constructing a video feature extractor and an audio feature extractor, the spatiotemporal features of the video modality and the spectral features of the audio modality are extracted respectively. In the cross-modal interaction stage, an asymmetric projection mechanism is adopted to map the dynamic changes of the audio timing to a latent space aligned with the video dimension. After applying temporal position encoding to the video features, they are input into the multi-head additive cross-attention module together with the audio features to achieve deep interaction and fusion of cross-modal features, effectively capturing the texture anomalies and audio and video timing mismatch features of forged videos. Cosine similarity is used to calculate the matching loss between audio and video outputs, and the supervised learning of the four-category detection main task and the collaborative supervision mechanism of audio and video subtask detection are comprehensively considered to achieve high-precision detection in complex forgery scenarios. Based on this, the present invention overcomes the defect of traditional single-modal detection methods in insufficiently capturing the characteristics of audio and video timing mismatches, and solves the problems of weak model generalization ability and long training cycle caused by cross-modal feature interaction redundancy in the existing technology. It realizes real-time and accurate recognition of multimodal attacks such as deep fakes and voice replacement in an open environment, providing a high-precision solution for deep fake identification.

[0207] Example 2

[0208] Based on the same general inventive concept, the present invention also provides a multimodal forged video detection system based on a multi-head additive cross-attention mechanism. The multimodal forged video detection system based on a multi-head additive cross-attention mechanism provided by the present invention is described below. The multimodal forged video detection system based on a multi-head additive cross-attention mechanism described below can be used in conjunction with the multimodal forged video detection method based on a multi-head additive cross-attention mechanism described above. The system includes:

[0209] Video processing module for audio and video separation and 68-point facial key point detection;

[0210] Feature extraction module, used to extract audio Mel spectrum features and video spatiotemporal features, and achieve feature dimensionality reduction through residual compression network;

[0211] Cross-modal fusion module, used to achieve cross-modal feature interaction based on multi-head attention mechanism;

[0212] Forgery detection module for joint optimization of multi-task losses and dynamic threshold classification decision.

[0213] As an implementation method of this embodiment, the video processing module includes:

[0214] Multimodal separation unit: used to separate video stream and audio stream and generate independent time series features

[0215] Face detection unit: detects 68 facial key points based on the RetinaFace model and extracts geometric dynamic features

[0216] As an implementation method in this embodiment, the data construction unit includes:

[0217] The dataset is divided into Class A (real video + real audio), Class B (fake video + fake audio), Class C (fake video + real audio), and Class D (real video + fake audio). The training / test sets are divided into 3:1, and frequency domain masking and time reordering are used to enhance the data.

[0218] As an implementation method of this embodiment, the feature extraction module includes:

[0219] Audio feature extraction unit: A SincConv layer is used to generate 50 Mel-scale bandpass filter banks, and a hierarchical residual compression network is used to perform nonlinear dimensionality reduction on frequency domain features.

[0220] Video feature extraction unit: Separates spatial texture and motion trajectory features based on horizontally and vertically separable convolution kernels, suppresses background noise areas through variance-driven spatial self-attention mechanism, and focuses on abnormal lip micro-expressions.

[0221] As an implementation method of this embodiment, the spatiotemporal feature processing module includes:

[0222] Timing alignment unit: A linear projection matrix is ​​used to expand the 4-node audio feature to 10 nodes, achieving time synchronization with the video frame rate and controlling the timing error within 2 frames.

[0223] As an implementation method of this embodiment, the feature fusion module includes:

[0224] Parameter freezing unit: freezes the audio SincConv filter bank and video residual network parameters during the cross-modal fusion stage;

[0225] Cross-modal attention unit: Based on the additive cross-attention mechanism, it calculates the attention weights of any two modalities;

[0226] Residual Fusion Unit: Integrating cross-modal features through layer-normalized residual connections:

[0227] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0228] Through the above implementation, the problem of multimodal video forgery detection based on the multi-head addition cross-attention mechanism in the related art is solved. Based on this, the present invention overcomes the defect of traditional single-modal detection methods in insufficiently capturing the audio and video timing mismatch features, and solves the problems of weak model generalization ability and long training cycle caused by cross-modal feature interaction redundancy in the existing technology. It realizes real-time and accurate recognition of multimodal attacks such as deep fakes and voice replacement in an open environment, providing a high-precision solution for deep fake identification.

[0229] It should be understood that the multimodal forged video detection system based on the multi-head additive cross-attention mechanism provided by the embodiment of the present invention has all the advantages of the multimodal forged video detection method based on the multi-head additive cross-attention mechanism provided by the above-mentioned embodiment.

[0230] Example 3

[0231] In this embodiment, a computer terminal device is provided, including:

[0232] one or more processors;

[0233] a memory, coupled to the processor, for storing one or more programs;

[0234] When the one or more programs are executed by the one or more processors, the one or more processors implement the methods in the above embodiments.

[0235] In this embodiment, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method in the above embodiment is implemented.

[0236] In this embodiment, an electronic device is further provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0237] The above program can be executed in a processor or stored in a memory (or computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and can be implemented using any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0238] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0239] This embodiment provides such a device or system. The system is called a multimodal forged video detection system based on a multi-headed additive cross-attention mechanism, and includes:

[0240] The preprocessing module is used to decompose the input video stream into a single-frame image sequence and locate the face area, while extracting the audio waveform to generate a Mel-spectrogram;

[0241] The feature extraction module is used to extract spatiotemporal features by sampling video frames in blocks through a 3D convolutional network and generate audio spectrum features based on the filter bank;

[0242] A cross-modal alignment module to map audio features to a latent space aligned with the video dimensions;

[0243] The attention fusion module is used to perform bidirectional multi-head addition cross-attention calculation on the video features and audio features with temporal encoding, generate audio dominant features and video dominant features, and cascade fuse the original features with the bidirectional interaction features;

[0244] The classification decision module is used to construct the audio and video cosine similarity loss function to constrain the feature distribution, dynamically weighted and optimized the fusion features, and output the four-category forgery type probability.

[0245] As an implementation method in this embodiment, the preprocessing module includes:

[0246] A video segmentation unit, configured to capture video frames at a fixed sampling rate and divide the frames into non-overlapping time segments;

[0247] The spectrum slicing unit is used to slice the Mel spectrum graph along the time axis using a Hamming window, so that the number of sub-spectrum segments is aligned with the video time period.

[0248] As an implementation method of this embodiment, the feature extraction module includes:

[0249] A bidirectional difference unit for performing horizontal and vertical slicing operations on video frames and calculating inter-frame motion differences;

[0250] The variance analysis unit is used to aggregate spatiotemporal information through multi-scale strip convolution to generate an attention map to highlight the characteristics of eye micro-movements and lip distortion.

[0251] As an implementation method of this embodiment, the cross-modal alignment module includes:

[0252] A spatial mapping unit for mapping audio feature time steps and video time series into a shared latent space;

[0253] The feature splicing unit is used to project the features into a unified representation space through a fully connected layer and to splice along the channel dimension to generate the first fusion feature.

[0254] As an implementation in this embodiment, the attention fusion module includes:

[0255] Multi-head computing unit, used to split the query vector into multiple attention heads and independently calculate the weights;

[0256] The residual integration unit is used to weightedly fuse the attention weights and value vectors of each head, and integrate the original features and aggregated features through residual connections.

[0257] As an implementation method of this embodiment, the classification decision module includes:

[0258] Multi-granularity discrimination unit, used to integrate the outputs of the audio binary classification branch, the video binary classification branch, and the multimodal fusion branch;

[0259] The four-class output unit is used to generate four types of probabilities: audio-visual double truth, audio-visual double false, video false-audio true, and video true-audio false.

[0260] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0261] Through the above implementation, the problem of multimodal forged video detection based on the multi-head addition cross attention mechanism in the related art is solved, thereby ensuring that the problems existing in the existing technology are solved.

[0262] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A multimodal forged video detection method based on a multi-head additive cross attention mechanism, characterized in that: The following steps are involved: S1. Multimodal data preprocessing: Video modality processing includes decomposing the video stream into a single-frame image sequence, locating the face area and capturing the face image; Audio modality processing includes extracting audio waveforms to generate Mel-spectrograms and slicing them into segments; S2. Multimodal feature extraction: Video feature extraction uses 3D convolutional network block sampling to extract spatiotemporal features and calculate inter-frame motion differences; Audio feature extraction generates spectral features based on filter banks and combines them with residual networks to capture contextual information; S3. Cross-modal feature alignment and fusion: Mapping audio features to a latent space aligned with video dimensions; After applying temporal coding to the video features, they are input into the multi-head addition cross attention module with the audio features: Generate audio dominant features using audio features as query vectors and video features as key-value pairs; Generate video dominant features using video features as query vectors and audio features as key-value pairs; Cascade fusion of original features and bidirectional interaction features; S4. Contrastive learning and classification decision: Construct audio and video cosine similarity loss function to constrain feature distribution; The fused features are dynamically weighted, optimized, and time-series compressed to output four-category forgery type probabilities.

2. The method according to claim 1, characterized in that The S1 process includes: The video data is framed at a fixed sampling rate and divided into non-overlapping time segments; The audio spectrum is segmented and sliced ​​along the time axis using a Hamming window; The number of sub-spectral segments is aligned with the number of video time segments.

3. The method according to claim 1, characterized in that The process of video feature extraction in S2 includes: Perform horizontal and vertical bidirectional slicing operations on video frames; Use multi-scale strip convolution to aggregate spatiotemporal information; The attention map generated by variance analysis highlights the features of eye micro-movement and lip distortion.

4. The method according to claim 1, wherein The feature alignment process in S3 includes: Mapping audio feature time steps and video time series into a shared latent space; Projected to a unified representation space through a fully connected layer; The first fusion feature is generated by concatenating along the channel dimension.

5. The method according to claim 1, wherein The working process of the multi-head addition cross attention module in S3 includes: Split the query vector into multiple attention heads and calculate the weights independently; Weighted fusion of each head’s attention weight and value vector; The original features and aggregated features are integrated through residual connections.

6. The method according to claim 1, characterized in that The four types of forgery in S4 include: Audiovisual double truth, audiovisual double fake, video fake - audio real, video real - audio fake; The classification decision adopts a multi-granularity discrimination mechanism to integrate audio binary classification, video binary classification and multimodal fusion branch output.

7. A multimodal forged video detection system based on multi-head additive cross attention mechanism, characterized by: The system comprises: The preprocessing module is used to decompose the input video stream into a single-frame image sequence and locate the face area, while extracting the audio waveform to generate a Mel-spectrogram; The feature extraction module is used to extract spatiotemporal features by sampling video frames in blocks through a 3D convolutional network and generate audio spectrum features based on the filter bank; A cross-modal alignment module to map audio features to a latent space aligned with the video dimensions; The attention fusion module is used to perform bidirectional multi-head addition cross-attention calculation on the video features and audio features with temporal encoding, generate audio dominant features and video dominant features, and cascade fuse the original features with the bidirectional interaction features; The classification decision module is used to construct the audio and video cosine similarity loss function to constrain the feature distribution, dynamically weighted and optimized the fusion features, and output the four-category forgery type probability.

8. The system according to claim 7, characterized in that The pre-processing module comprises: A video segmentation unit, configured to capture video frames at a fixed sampling rate and divide the frames into non-overlapping time segments; The spectrum slicing unit is used to slice the Mel spectrum graph along the time axis using a Hamming window, so that the number of sub-spectrum segments is aligned with the video time period.

9. A computer terminal device, characterized in that: include: one or more processors; a memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multimodal forged video detection method based on the multi-head addition cross attention mechanism as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal forged video detection method based on a multi-head addition cross attention mechanism as described in any one of claims 1 to 6 is implemented.

Images