A dual-path deep forgery detection method and device using a large multi-modal model

Abstract

This application provides a method and apparatus for deepfake video detection based on a multimodal large language model. The method includes: acquiring video samples; calculating multimodal auxiliary loss values ​​and cross-modal consistency loss values ​​for the video samples; fine-tuning the large language model based on the multimodal auxiliary loss values ​​and cross-modal consistency loss values ​​to obtain a fine-tuned large language model for deepfake detection; acquiring the video to be detected; and using the fine-tuned large language model to detect the video to be detected, obtaining a detection result indicating whether the video to be detected is a fake video. This application uses multimodal loss to model forgery traces within a single modality, and simultaneously uses KL divergence loss to verify the consistency between audio and video modalities, achieving stronger cross-modal forgery detection capabilities.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a dual-path deep pseudo-detection method and apparatus utilizing a large multimodal model. Background Technology

[0002] The rapid development of generative artificial intelligence has propelled deepfake technology from early single-modal visual manipulation to complex audiovisual collaborative forgery, seriously threatening the authenticity of digital media. Traditional detection methods often have two limitations: first, they mainly rely on single-modal forgery traces for analysis, making it difficult to capture subtle inconsistencies across modalities; second, they mostly focus on the coherence between video and audio modalities, while relatively ignoring potential forgery clues within each modality. Furthermore, limited by the number of model parameters and network structure, existing methods, even at the single-modal detection level, still have room for further improvement through enhanced computing power and optimized architecture. Summary of the Invention

[0003] This application describes a dual-path deep pseudo-detection method and apparatus utilizing a large-scale multimodal model, which can solve the aforementioned technical problems.

[0004] According to the first aspect, a deepfake video detection method based on a multimodal large language model is provided, including:

[0005] Obtain video samples;

[0006] The video feature sequence of the video sample is extracted by the video encoder of the large language model, and the audio feature sequence of the video sample is extracted by the audio encoder of the large language model.

[0007] The binary classification prediction results of the video feature sequence and the binary classification prediction results of the audio feature sequence are obtained respectively.

[0008] Based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples, calculate the multimodal auxiliary loss value;

[0009] The video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence are determined respectively, and the cross-modal consistency loss value is calculated based on the symmetric KL divergence between the video feature distribution and the audio feature distribution.

[0010] Based on the multimodal auxiliary loss value and the cross-modal consistency loss value, the large language model is fine-tuned to obtain a fine-tuned large language model for deepfake detection;

[0011] The video to be detected is acquired, and the fine-tuned large language model is used to detect the video to obtain the detection result of whether the video to be detected is a fake video.

[0012] In some embodiments, it also includes:

[0013] The parameters of the video encoder and the audio encoder are adjusted based on the multimodal auxiliary loss value and the cross-modal consistency loss value.

[0014] In some embodiments, calculating the multimodal auxiliary loss value based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples includes:

[0015] The multimodal auxiliary loss value is calculated using the following formula:

[0016]

[0017] in, It is the multimodal auxiliary loss value, It is the cross-entropy loss function. It is the binary classification prediction result of the video feature sequence. The binary classification prediction result of the audio feature sequence. The true label for the video sample.

[0018] In some embodiments, determining the video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence respectively specifically includes:

[0019] Perform feature alignment on the video feature sequence and the audio feature sequence;

[0020] Applying probability distribution functions to the video feature sequence and the audio feature sequence respectively, the video feature distribution of the video feature sequence is obtained. and the audio feature distribution ,in, , and These are the video feature sequence and the audio feature sequence, respectively.

[0021] In some more specific embodiments, calculating the cross-modal consistency loss value based on the symmetric KL divergence between the video feature distribution and the audio feature distribution includes:

[0022] Calculated using the following formula:

[0023]

[0024] in, It is the cross-modal consistency loss value.

[0025] In some more specific embodiments, adjusting the parameters of the large language model based on the multimodal auxiliary loss value and the cross-modal consistency loss value specifically includes:

[0026] The total loss function is determined by weighting the multimodal auxiliary loss value, the cross-modal consistency loss value, and the main classification loss value of the large language model.

[0027] The parameters of the large language model are optimized by minimizing the total loss function.

[0028] In some more specific embodiments, the detection result of whether the video to be detected is a fake video includes: the determination result of whether the video to be detected is a fake video, and the fake probability for the video modality and the audio modality respectively.

[0029] According to the second aspect, a deepfake video detection device based on a multimodal large language model is provided, comprising:

[0030] The sample acquisition module is used to acquire video samples;

[0031] The feature extraction module is used to extract the video feature sequence of the video sample through the video encoder of the large language model, and to extract the audio feature sequence of the video sample through the audio encoder of the large language model.

[0032] A modality-specific prediction module is used to obtain the binary classification prediction results of the video feature sequence and the binary classification prediction results of the audio feature sequence, respectively.

[0033] The multimodal loss calculation module is used to calculate the multimodal auxiliary loss value based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples.

[0034] The cross-modal loss calculation module is used to determine the video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence, respectively, and calculate the cross-modal consistency loss value based on the symmetric KL divergence between the video feature distribution and the audio feature distribution;

[0035] The model training module is used to fine-tune the large language model based on the multimodal auxiliary loss value and the cross-modal consistency loss value to obtain a fine-tuned large language model for deep forgery detection.

[0036] The video detection module is used to acquire the video to be detected and to use the fine-tuned large language model to detect the video to be detected, so as to obtain the detection result of whether the video to be detected is a fake video.

[0037] In some embodiments, the model training module is used to adjust the parameters of the video encoder and the audio encoder based on the multimodal auxiliary loss value and the cross-modal consistency loss value.

[0038] In some embodiments, the multimodal loss calculation module is used to calculate the multimodal auxiliary loss value using the following formula:

[0039]

[0040] in, It is the multimodal auxiliary loss value, It is the cross-entropy loss function. It is the binary classification prediction result of the video feature sequence. The binary classification prediction result of the audio feature sequence. The true label for the video sample.

[0041] In some embodiments, the cross-modal loss calculation module is used to perform feature alignment on the video feature sequence and the audio feature sequence;

[0042] Applying probability distribution functions to the video feature sequence and the audio feature sequence respectively, the video feature distribution of the video feature sequence is obtained. and the audio feature distribution ,in, , and These are the video feature sequence and the audio feature sequence, respectively.

[0043] In some embodiments, the cross-modal loss calculation module is used to calculate using the following formula:

[0044]

[0045] in, It is the cross-modal consistency loss value.

[0046] In some embodiments, the model training module is used to determine the total loss function based on the weighted sum of the multimodal auxiliary loss value, the cross-modal consistency loss value, and the main classification loss value of the large language model;

[0047] The parameters of the large language model are optimized by minimizing the total loss function.

[0048] In some embodiments, the detection result of whether the video to be detected is a fake video includes: the determination result of whether the video to be detected is a fake video, and the fake probability for the video modality and the audio modality respectively.

[0049] According to a third aspect, a computer storage medium is provided, on which a computer program is stored, which, when executed by one or more processors, implements the deepfake video detection method based on a multimodal large language model as described in any of the above embodiments.

[0050] According to a fourth aspect, an electronic device is provided, including a memory and one or more processors, wherein a computer program is stored on the memory, and the computer program, when executed by the one or more processors, implements the deepfake video detection method based on a multimodal large language model as described in any of the above embodiments.

[0051] In the methods and systems provided in the embodiments of this specification, a large language model is used as the core inference engine, and two loss functions are jointly optimized to improve the anti-spoofing performance: on the one hand, multimodal loss is used to model forgery traces within a single modality; on the other hand, KL divergence loss is used to verify the consistency between audio and video modalities. The performance of both methods is significantly improved, demonstrating stronger cross-modal forgery detection capabilities. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This diagram illustrates the framework of a multimodal video forgery detection method based on a large language model, as provided in an embodiment of this specification.

[0054] Figure 2 This diagram illustrates a flowchart of a multimodal video forgery detection method based on a large language model, as provided in an embodiment of this specification.

[0055] Figure 3 This is a schematic diagram comparing the benchmark methods for deepfake detection with the present invention on the IDForge dataset;

[0056] Figure 4 This is a schematic diagram comparing the benchmark methods for deepfake detection with the present invention on the IDForge dataset;

[0057] Figure 5 This diagram illustrates a module schematic of a multimodal video forgery detection device based on a large language model, as provided in an embodiment of this specification. Detailed Implementation

[0058] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0060] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0061] In the description of the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more.

[0062] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0063] The rapid development of generative artificial intelligence is propelling deepfake technology into a new phase, evolving from initial unimodal visual forgery limited to images or videos to cross-modal audiovisual collaborative forgery. This technology can generate highly consistent and realistic combinations of video and audio, making the forged content visually and aurally indistinguishable to the average person.

[0064] Against this backdrop, traditional detection methods are increasingly revealing their limitations, primarily in the following aspects: Many existing methods still mainly analyze single modalities (such as video or audio), relying on forgery traces within that modality, such as facial distortions, unnatural lighting, and inter-frame inconsistencies in images, or spectral anomalies and unreasonable background noise in audio. However, in multimodal forgery, forgers may optimize visual and auditory information separately, making traces within a single modality extremely subtle or even undetectable. Relying solely on single-modal analysis often makes it difficult to capture subtle inconsistencies across modalities, such as slight misalignments between lip movements and speech, or mismatches between emotional expression and intonation. While other methods attempt to detect from a multimodal collaborative perspective, such as comparing the temporal synchronization or semantic consistency of video footage and corresponding audio, these methods often overemphasize intermodal inconsistencies while neglecting potentially detectable forgery traces that may still exist within each modality. In reality, even highly coordinated cross-modal forgeries may still retain inherent flaws in the generative model itself within a single modality, such as repetitive patterns in local textures in vision or anomalous attenuation in specific frequency bands in audio. Ignoring these intramodal traces could cause detection systems to quickly fail as forgery techniques improve.

[0065] Furthermore, traditional detection models are often limited by parameter size and model structure design, making it difficult to simultaneously achieve efficient multimodal fusion and refined single-modal analysis. Many methods simplify the feature extraction or fusion stages to control computational costs, resulting in an inability to fully capture subtle forgery features. Moreover, a single model often struggles to balance detection speed and accuracy, especially when dealing with high-resolution, long-duration audiovisual data, where a significant trade-off exists between detection efficiency and coverage. Even in single-modal detection tasks, existing methods can still achieve significant improvements through more powerful computing capabilities, more optimized network architectures (such as attention mechanisms and multi-scale analysis), and more abundant training data.

[0066] To address the aforementioned issues, this invention proposes a multimodal video forgery detection method based on a large-scale language model. This method utilizes a large-scale language model as the central inference engine to systematically model and verify forged content. Specifically, by designing multimodal loss and KL divergence loss, it accurately captures forgery traces within a single modality and misalignments and inconsistencies across audiovisual modalities, thereby fully mining and fusing multi-dimensional forgery information and significantly improving the detection method's discriminative power and robustness.

[0067] like Figure 1 The diagram shows a framework diagram of a multimodal video forgery detection method based on a large language model proposed in this invention.

[0068] The framework diagram mainly comprises three modules: a large-scale language model framework and low-rank adaptive fine-tuning module, a multimodal loss module for learning single-modal forgery traces, and a KL divergence loss module for learning intermodal inconsistencies. The structure of each module is described below.

[0069] In the large-scale language model framework and low-rank adaptation fine-tuning module, a pre-trained large-scale language model, such as Qwen2.5-Omni, can be used, combined with the low-rank adaptation method for efficient parameter fine-tuning. This maintains its strong language understanding capabilities while enhancing its perception of multimodal forgery features. The input is image and audio encoded and then feature-aligned. The low-rank adaptation method introduces a minimal set of trainable parameters while freezing the original pre-trained weights, enabling the model to quickly adapt to new types of forgery samples even under low-resource conditions, thus improving generalization performance and inference consistency.

[0070] In the multimodal loss module used for learning single-modal forgery artifacts, to ensure the model is sensitive to such single-modal forgery artifacts, an auxiliary multimodal loss function is used to augment the standard alignment loss. This design guides each modality-specific encoder to learn discriminative features representing artifacts within its corresponding modality. For audio and video modalities, and This represents the feature sequences extracted by the video and audio encoders, which are processed through separate fully connected layers. and This process generates binary classification predictions for the final counterfeit detection task.

[0071]

[0072] The final loss result of the multimodal loss is obtained by calculating the cross-entropy loss from the predictions and original labels generated above:

[0073]

[0074] in Representing a true label, this auxiliary objective enables each encoder to learn strong individual discrimination capabilities. Visual encoders learn to focus on spatial and temporal artifacts specific to video operations (e.g., unnatural blending in face swaps), while audio encoders learn to identify anomalies in acoustic signals (e.g., synthetic speech quality or inconsistent phoneme generation). This is crucial for detecting deepfakes, where one modality is real and the other is fake, because the encoder for the corrupted modality must still be able to recognize its inherent artifacts.

[0075] In the KL divergence loss module for learning intermodal inconsistencies, a cross-modal alignment module is employed to effectively capture the shared modal semantic information between the audio and video streams. The basic assumption is that for real video, the high-level semantic representations of the audio and visual modalities should remain consistent and aligned in a shared latent space. For example, lip movements observed in the video should correspond precisely to speech in the audio track. Any significant discrepancies in this alignment can be a strong indicator of forgery.

[0076] Based on the information theory principle of maximizing intermodal mutual information, KL divergence is used to minimize the distributional difference between audio and visual feature distributions. and As feature representations of video and audio, we first obtain their possible distributions using the softmax function:

[0077]

[0078] Then, the KL divergence loss is calculated to measure the difference between one distribution and another. Specifically, this loss is defined as the symmetric KL divergence between the two distributions:

[0079]

[0080] This approach ensures that the model penalizes inconsistencies in both directions—when video and audio don't match, and when audio and video don't match. This is achieved by minimizing... This explicitly encourages models to generate semantically coherent audio and visual representations, effectively bringing the distribution of consistent audiovisual pairs in the shared latent space closer together. This process is crucial for introducing semantically inconsistent forgeries, such as mismatches between spoken phonemes and visual lip shapes.

[0081] In summary, as Figure 1 As shown, video and audio are simultaneously input into the system. The visual encoder and audio encoder extract deep features of their respective modalities and represent them as structured information. The multimodal loss module calculates the loss values ​​of video features and audio features to obtain the multimodal loss value, while the KL divergence loss module calculates the KL divergence loss between the two stream features. In the fine-tuning module, the large language model is fine-tuned using the multimodal loss and KL divergence loss. When the video to be tested is input into the fine-tuned large language model, all features and preset guidance statements are input into the Qwen2.5-Omni thinking layer. Based on the falsification detection knowledge obtained after fine-tuning, the model comprehensively judges the multimodal evidence, as shown in the example: "Video is true, audio is false".

[0082] The bottom part of the figure illustrates the system's tuning strategy for the pre-trained large language model Qwen2.5-Omni. The "frozen" parts, marked in the figure, include the visual encoder, audio encoder, dictionary, and feature parts, indicating that most of the original pre-trained parameters are kept unchanged during fine-tuning to preserve its powerful general knowledge and reasoning capabilities. The "trainable" parts, marked in the figure, include the visual encoder and audio encoder, whose parameters can be optimized during training. The "low-rank adaptation" parts, marked in the figure, include the Qwen2.5-Omni thinking layer, which adapts the model to the specific task of video authenticity detection by training only a small number of low-rank parameter matrices. This method achieves efficient parameter fine-tuning, enabling a large model to acquire professional anti-spoofing capabilities with minimal training cost.

[0083] like Figure 2 As shown, this invention proposes a multimodal video forgery detection method based on a large-scale language model. Specifically, it includes the following steps:

[0084] 210: Obtain video samples, extract video feature sequences from the video samples using a video encoder, and extract audio feature sequences from the video samples using an audio encoder.

[0085] Specifically, abstract representations of visual and audio data are extracted from the original video samples to provide a foundation for subsequent multimodal analysis. The process can be divided into three steps:

[0086] First, the system acquires the video sample file to be analyzed. It preprocesses the file, such as standardizing the frame rate, resolution, and audio sampling rate, and then decapsulates it into a continuous sequence of image frames and synchronized audio waveform data.

[0087] The preprocessed video frame sequence is then input into a dedicated video encoder, which can be a pre-trained model based on a 3D convolutional neural network or a visual Transformer, or a video encoder within a large language model. This encoder analyzes the spatiotemporal relationships between frames segment by segment or as a whole, outputting a video feature sequence containing temporal information, where each feature vector represents the visual content and dynamic information within a specific time period.

[0088] Synchronized audio waveform data is input into an audio encoder, which can be a pre-trained model based on a convolutional network or an audio Transformer. The encoder analyzes the spectral and temporal characteristics of the audio and outputs a sequence of audio features aligned with the time axis, where each feature vector represents the acoustic characteristics within the corresponding time segment.

[0089] Ultimately, two feature sequences aligned in the time dimension are obtained: the video feature sequence and the audio feature sequence, which together constitute the multimodal representation of the video sample.

[0090] In this embodiment, the video encoder and audio encoder can be part of a large language model. Video samples are processed using the video and audio encoding capabilities that can utilize the large language model.

[0091] Specifically, the pre-trained large-scale multimodal model Qwen2.5-Omni can be used as the large language model. To achieve task-specific adaptation with limited computational resources, a low-rank adaptation method is employed for efficient parameter fine-tuning. For example, a trainable low-rank matrix can be injected as a bypass to the linear projection layer of the original Transformer network.

[0092] 220: Obtain the binary classification prediction results for the video feature sequence and the audio feature sequence, respectively.

[0093] Specifically, independent preliminary authenticity assessments are performed on visual and audio information separated from the same video sample. First, the visual and audio tracks are examined separately for signs of forgery. This helps pinpoint the specific modality in which the forgery occurred—whether the image was tampered with, the audio was replaced, or both were problematic. It also provides preliminary, intramodal evidence for subsequent, more complex cross-modal consistency analysis. The two independent assessment results can be likened to two "preliminary identification reports."

[0094] Train or fine-tune a dedicated binary classifier for each of the video and audio feature sequences. This classifier is typically a lightweight neural network module, such as a multilayer perceptron (MLP) or a one-dimensional convolutional network, appended to the corresponding encoder.

[0095] The video feature sequence is input into a video classifier. This classifier comprehensively analyzes the temporal patterns and spatial features of the sequence, and finally aggregates a single judgment representing the visual authenticity of the entire video.

[0096] The audio feature sequence is input into an audio classifier. This classifier analyzes the spectral features and temporal evolution of the audio, and finally aggregates a single judgment representing the authenticity of the entire audio segment.

[0097] The output of each branch is typically a two-dimensional vector, which, after passing through a probability distribution function, is converted into two probability values: the probability of being real and the probability of being fake. Finally, two clear binary classification predictions are obtained: video prediction result: (probability of being real, probability of being fake), and audio prediction result: (probability of being real, probability of being fake). If the probability of being fake is high in the video prediction result, while the probability of being real is high in the audio prediction result, it strongly suggests that the sample may be a deepfake where "only the video has been manipulated."

[0098] These two predicted probability values ​​can serve as important features, inputting them along with the original feature sequence into subsequent decision fusion modules or large-scale language model inference engines. The model can then use these values ​​to reason; for example, if the video modality shows a high probability of being fake, while the audio modality shows a high probability of being genuine, and there is a significant inconsistency between the two, the model can comprehensively determine that the content is fake.

[0099] In this embodiment, video feature sequences and audio feature sequences can be input into different fully connected layers to obtain modality-specific binary classification prediction results.

[0100] Specifically, the video feature sequence and the audio feature sequence are connected through separate fully connected layers. and The data is then passed on to obtain a binary classification prediction for the final counterfeit detection task. It is a prediction result regarding whether a video is fake. It is a prediction result regarding whether the audio is fake.

[0101]

[0102] 230: Calculate the multimodal auxiliary loss value based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples.

[0103] Specifically, this application utilizes the overall true labels of the samples to simultaneously supervise two independent identification branches, namely video and audio, and may introduce intermodal consistency constraints to train a more robust and internally coordinated multimodal anti-spoofing system.

[0104] The video prediction result is a two-dimensional probability vector obtained through a video classifier, and the audio prediction result is a two-dimensional probability vector obtained through an audio classifier.

[0105] Multimodal auxiliary loss is a combination of losses, including video loss and audio loss. The video loss uses a standard classification loss function, such as binary cross-entropy loss, to calculate the difference between the video prediction and the true label y. The audio loss calculates the difference between the audio prediction and the same true label y. Therefore, even if the forgery artifacts in a modality are very weak, this loss forces the classifier for that modality to work harder to learn discern cues from the features.

[0106] Specifically, ,in, It is the loss value for the video prediction results. It is the loss value for the audio prediction results.

[0107] The auxiliary training objective aims to endow each modal encoder with independent forgery detection capabilities. Specifically, the visual encoder is constrained to learn and pay attention to spatiotemporal anomalies unique to video operations, such as unnatural boundaries of facial fusion areas or inconsistent lighting transitions during face swapping; while the audio encoder is guided to identify heterogeneous features in acoustic signals, such as the spectral characteristics of non-human voices or inconsistencies between phoneme sequences in synthesized speech.

[0108] This is crucial for dealing with mixed-modal forgery scenarios. In deepfake content, it is common for one modality to be altered while another remains genuine. Through this training method, even if the other modality is not corrupted, the encoder corresponding to the altered modality can still effectively identify the inherent forgery traces within that modality using its learned discriminative ability, thus providing the system with critical authentication evidence.

[0109] 240: Determine the video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence, respectively.

[0110] Specifically, the video feature sequence and the audio feature sequence can be aligned first.

[0111] The possible distributions of video feature sequences and audio feature sequences can be obtained using the softmax function. .

[0112] Audio and video feature sequences describe dynamic information that changes over time. Feature distributions, on the other hand, use statistical methods to capture the overall clustering, dispersion, and structural patterns of these features in the vector space. This helps to detect forgery traces that are weak at individual moments but statistically significant on an overall scale.

[0113] After feature alignment, methods such as KL divergence, Wasserstein distance, and maximum mean difference can be used to quantify the differences between video and audio feature distributions, thereby achieving a quantitative assessment of whether the audio and video originate from the same source. Distribution-based analysis is less sensitive to local noise and fluctuations and is better able to capture the essential statistical distortions inherent in forgery.

[0114] The distribution can be determined by using mathematical models to approximate or characterize the statistical properties of features. For example, a multidimensional Gaussian distribution can be used to determine the distribution of video features and audio features. Alternatively, the entire feature sequence can be input into a statistical pooling layer, which not only calculates the mean and standard deviation but may also calculate higher-order statistics, outputting a fixed-length vector as a representative of the distribution.

[0115] 250: Calculate the KL divergence loss value using the distribution of video feature sequences and audio feature sequences.

[0116] Specifically, the KL divergence loss is calculated to measure the difference between one distribution and another; this loss is defined as the symmetric KL divergence between the two distributions:

[0117]

[0118] This symmetric KL divergence loss imposes constraints on bidirectional inconsistencies between video and audio. Specifically, it penalizes not only mismatches between video and audio content but also mismatches between audio and video content. By minimizing this loss function, the large language model is explicitly guided to generate semantically highly consistent audiovisual feature representations, thereby tightly aligning the distribution of genuine, consistent audiovisual sample pairs in the shared latent space. This enables the detection of deepfake content containing semantic inconsistencies. For example, when there is a significant mismatch between the speech content and the speaker's lip movements in a fake video—i.e., phoneme-visual asynchrony—this loss can sensitively capture such cross-modal semantic breaks, thus providing an effective discriminative signal for the identification system.

[0119] 260: The parameters of the large language model are adjusted using multimodal loss and KL divergence loss to obtain a fine-tuned large language model.

[0120] Specifically, the large language model uses a pre-trained large language model (Qwen2.5-Omni) and combines it with the Low-Rank Adaptation method for efficient parameter fine-tuning, which enhances the perception of multimodal forgery features while maintaining its powerful language understanding capabilities.

[0121] By introducing a minimal set of trainable parameters through a low-rank adaptation method, while freezing the original pre-trained weights, the model can still quickly adapt to new types of fake samples under low-resource conditions, thereby improving generalization performance and inference consistency.

[0122] 270: Obtain the video to be detected, use the trained large language model to detect the video to obtain the detection result of whether the video to be detected is a fake video.

[0123] Specifically, the system receives the video file to be tested, which can be an uploaded file, a video stream link, or a live stream. It then parses the video, separating the video and audio components.

[0124] The video is input into a trained video encoder, which outputs a sequence of video features with temporal correlation, and obtains a mathematical representation of the video feature distribution through statistical pooling.

[0125] Synchronized audio data is input into a trained audio encoder, which outputs the corresponding audio feature sequence and its audio feature distribution.

[0126] The video feature sequence and audio feature sequence are respectively processed by a pre-trained unimodal binary classifier to generate independent preliminary prediction probabilities, namely the video forgery probability and the audio forgery probability.

[0127] All intermediate results—video feature sequences, audio feature sequences, video forgery probabilities, and audio forgery probabilities—are integrated into a structured multimodal evidence report according to a predefined template. The large language model is explicitly instructed to act as a deepfake identification expert, and its output format is specified. The structured evidence report is input into a fine-tuned large language model, such as a fine-tuned Qwen2.5-Omni model. Leveraging its acquired expertise, the model performs deep reasoning, analyzing statistical anomalies within each modality's features and consistency relationships across modalities. Examples include audio-visual synchronization and semantic consistency. Logical connections and conflicts between unimodal predictions are also examined. The final detection result is generated, including a determination of whether the video is forged and the forgery probabilities for both the video and audio modalities. For example, if a video is determined to be forged, the reason given is that there is a significant discrepancy between the lip movements of the person in the video and the audio phonemes at a given time point.

[0128] Based on the above embodiments, a large language model is used as the core inference engine, and the forgery detection performance is improved by jointly optimizing two loss functions: on the one hand, multimodal loss is used to model forgery traces within a single modality; on the other hand, KL divergence loss is used to verify the consistency between audio and video modalities. The detection performance is significantly improved, demonstrating stronger cross-modal forgery detection capabilities.

[0129] like Figure 3 and Figure 4 As shown, the effectiveness of the proposed method is verified using two public datasets, IDForge and FakeAVCeleb. The specific composition of the two datasets is as follows:

[0130] The IDForge dataset contains 249,138 video clips, each no longer than 20 seconds, with a resolution of 1280×720. Of these, 169,311 are fake videos and 79,827 are genuine videos.

[0131] The FakeAVCeleb dataset: Based on 490 real samples, it uses a variety of advanced deepfake techniques to generate synthetic content, ultimately containing more than 20,000 fake videos.

[0132] For baseline comparison, several models that have performed well on the two datasets in recent years were selected as baseline methods. The evaluation metrics followed the evaluation system widely used in this field, mainly including accuracy (ACC), mean accuracy (mAP), and area under the ROC curve (AUC).

[0133] like Figure 3 and Figure 4 As shown in the table, the method in this embodiment outperforms all baseline models on both the IDForge and FakeAVCeleb datasets. The proposed method significantly improves upon previous benchmark models in terms of accuracy (ACC), mean accuracy (mAP), and area under the ROC curve (AUC), indicating that the method in this embodiment possesses stronger identification capabilities and robustness in identifying multimodal forged content.

[0134] Figure 5 This is a schematic diagram of a module for a deepfake video detection device based on a multimodal large language model. (Example:) Figure 5 As shown, a deepfake video detection device based on a multimodal large language model includes:

[0135] The sample acquisition module is used to acquire video samples;

[0136] The feature extraction module is used to extract the video feature sequence of the video sample through the video encoder of the large language model, and to extract the audio feature sequence of the video sample through the audio encoder of the large language model.

[0137] A modality-specific prediction module is used to obtain the binary classification prediction results of the video feature sequence and the binary classification prediction results of the audio feature sequence, respectively.

[0138] The multimodal loss calculation module is used to calculate the multimodal auxiliary loss value based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples.

[0139] The cross-modal loss calculation module is used to determine the video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence, respectively, and calculate the cross-modal consistency loss value based on the symmetric KL divergence between the video feature distribution and the audio feature distribution;

[0140] The model training module is used to fine-tune the large language model based on the multimodal auxiliary loss value and the cross-modal consistency loss value to obtain a fine-tuned large language model for deep forgery detection.

[0141] The video detection module is used to acquire the video to be detected and to use the fine-tuned large language model to detect the video to be detected, so as to obtain the detection result of whether the video to be detected is a fake video.

[0142] In some embodiments, the model training module is used to adjust the parameters of the video encoder and the audio encoder based on the multimodal auxiliary loss value and the cross-modal consistency loss value.

[0143] In some embodiments, the multimodal loss calculation module is used to calculate the multimodal auxiliary loss value using the following formula:

[0144]

[0145] in, It is the multimodal auxiliary loss value, It is the cross-entropy loss function. It is the binary classification prediction result of the video feature sequence. The binary classification prediction result of the audio feature sequence. The true label for the video sample.

[0146] In some embodiments, the cross-modal loss calculation module is used to perform feature alignment on the video feature sequence and the audio feature sequence;

[0147] Applying probability distribution functions to the video feature sequence and the audio feature sequence respectively, the video feature distribution of the video feature sequence is obtained. and the audio feature distribution ,in, , and These are the video feature sequence and the audio feature sequence, respectively.

[0148] In some embodiments, the cross-modal loss calculation module is used to calculate using the following formula:

[0149]

[0150] in, It is the cross-modal consistency loss value.

[0151] In some embodiments, the model training module is used to determine the total loss function based on the weighted sum of the multimodal auxiliary loss value, the cross-modal consistency loss value, and the main classification loss value of the large language model;

[0152] The parameters of the large language model are optimized by minimizing the total loss function.

[0153] In some embodiments, the detection result of whether the video to be detected is a fake video includes: the determination result of whether the video to be detected is a fake video, and the fake probability for the video modality and the audio modality respectively.

[0154] In summary, the embodiments of this application effectively combine intramodal artifact analysis with intermodal consistency, integrating the two existing anti-counterfeiting schemes to make fuller use of forged information; employing a large language model introduces the high computing power of a large language model into the anti-counterfeiting field, significantly improving the anti-counterfeiting method's ability to distinguish complex forgeries; and exhibiting stable performance in both seen and unseen scenarios, demonstrating the best discrimination and generalization capabilities in the current field of multimodal forgery identification.

[0155] According to another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, it implements the deepfake video detection method based on a multimodal large language model as described above.

[0156] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a deepfake video detection method based on a multimodal large language model.

[0157] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0158] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A deepfake video detection method based on a multimodal large language model, characterized in that, include: Obtain video samples; The video feature sequence of the video sample is extracted by the video encoder of the large language model, and the audio feature sequence of the video sample is extracted by the audio encoder of the large language model. The binary classification prediction results of the video feature sequence and the binary classification prediction results of the audio feature sequence are obtained respectively. Based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples, calculate the multimodal auxiliary loss value; The video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence are determined respectively, and the cross-modal consistency loss value is calculated based on the symmetric KL divergence between the video feature distribution and the audio feature distribution. Based on the multimodal auxiliary loss value and the cross-modal consistency loss value, the large language model is fine-tuned to obtain a fine-tuned large language model for deepfake detection; The video to be detected is acquired, and the fine-tuned large language model is used to detect the video to obtain the detection result of whether the video to be detected is a fake video.

2. The method according to claim 1, characterized in that, Also includes: The parameters of the video encoder and the audio encoder are adjusted based on the multimodal auxiliary loss value and the cross-modal consistency loss value.

3. The method according to claim 1, characterized in that, The step of calculating the multimodal auxiliary loss value based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples includes: The multimodal auxiliary loss value is calculated using the following formula: in, It is the multimodal auxiliary loss value, It is the cross-entropy loss function. It is the binary classification prediction result of the video feature sequence. The binary classification prediction result of the audio feature sequence. The true label for the video sample.

4. The method according to claim 1, characterized in that, The step of determining the video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence specifically includes: Perform feature alignment on the video feature sequence and the audio feature sequence; Applying probability distribution functions to the video feature sequence and the audio feature sequence respectively, the video feature distribution of the video feature sequence is obtained. and the audio feature distribution ,in, , and These are the video feature sequence and the audio feature sequence, respectively.

5. The method according to claim 1, characterized in that, The calculation of the cross-modal consistency loss value based on the symmetric KL divergence between the video feature distribution and the audio feature distribution includes: Calculated using the following formula: in, It is the cross-modal consistency loss value.

6. The method according to claim 1, characterized in that, The adjustment of the parameters of the large language model based on the multimodal auxiliary loss value and the cross-modal consistency loss value specifically includes: The total loss function is determined by weighting the multimodal auxiliary loss value, the cross-modal consistency loss value, and the main classification loss value of the large language model. The parameters of the large language model are optimized by minimizing the total loss function.

7. The method according to claim 1, characterized in that, The detection result of whether the video to be detected is a fake video includes: the determination result of whether the video to be detected is a fake video, and the fake probability for the video modality and the audio modality respectively.

8. A deepfake video detection device based on a multimodal large language model, characterized in that, include: The sample acquisition module is used to acquire video samples; The feature extraction module is used to extract the video feature sequence of the video sample through the video encoder of the large language model, and to extract the audio feature sequence of the video sample through the audio encoder of the large language model. A modality-specific prediction module is used to obtain the binary classification prediction results of the video feature sequence and the binary classification prediction results of the audio feature sequence, respectively. The multimodal loss calculation module is used to calculate the multimodal auxiliary loss value based on the binary classification prediction results of the video feature sequence, the binary classification prediction results of the audio feature sequence, and the true labels of the video samples. The cross-modal loss calculation module is used to determine the video feature distribution of the video feature sequence and the audio feature distribution of the audio feature sequence, respectively, and calculate the cross-modal consistency loss value based on the symmetric KL divergence between the video feature distribution and the audio feature distribution; The model training module is used to fine-tune the large language model based on the multimodal auxiliary loss value and the cross-modal consistency loss value to obtain a fine-tuned large language model for deep forgery detection. The video detection module is used to acquire the video to be detected and to use the fine-tuned large language model to detect the video to be detected, so as to obtain the detection result of whether the video to be detected is a fake video.

9. An electronic device, characterized in that, The device includes a memory and one or more processors, wherein the memory stores a computer program that, when executed by the one or more processors, implements the deepfake video detection method based on a multimodal large language model as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the deepfake video detection method based on a multimodal large language model as described in any one of claims 1 to 7.

Images