A method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion
The four-modal collaborative forgery detection method, which integrates ambiguity gating and modal hierarchical fusion, solves the problems of blind spots and inaccurate localization in existing technologies. It achieves unified collaborative decision-making and fine-grained localization of four-modal information, thereby improving the robustness and interpretability of detection.
Patent Information
- Application Number
- CN202611558261.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-09
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal forgery detection methods struggle to integrate information from four channels—image, text, video, and audio—simultaneously, making them incapable of handling four-modal collaborative forgery. Furthermore, they lack fine-grained localization of the tampered area. Traditional methods lose ambiguous signals when forcing modal alignment, resulting in detection blind spots and inaccurate localization.
A four-modal collaborative forgery detection and localization method based on ambiguity gating and modal hierarchical fusion is adopted. Through multimodal coding, cross-modal ambiguity scoring, image-text gating fusion, and audio-video fusion representation, the four-modal evidence chain is collaboratively judged in a unified semantic space. The information differences between modalities are quantified by variational modeling and KL divergence, and the fusion weight is dynamically adjusted. The tampered area is output by combining image, text, and audio-video forgery localization head.
It effectively solves the blind spot of four-modal collaborative forgery detection, improves the interpretability and forensic value of detection results, significantly enhances robustness to JPEG compression and additive noise, and achieves precise location of tampering.
Smart Images

Figure CN123116174A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal forgery detection technology, specifically to a method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion. Background Technology
[0002] The rapid development of generative artificial intelligence has lowered the barrier to creating high-fidelity images, natural language text, realistic videos, and synthesized speech to an all-time low. The widespread adoption of some generative AI technologies has made comprehensive forgery of images, text, videos, and audio a reality. For example, large language models such as BERT and GPT can generate semantically coherent fake text in batches; diffusion models and GAN families have achieved photorealistic image synthesis; face replacement tools like Inswapper and expression-driven models like LivePortrait can generate forged videos that are difficult to distinguish visually; and voice cloning systems like CAM++ and BigVGAN can replicate the timbre and rhythm of a target speaker with just a few seconds of reference audio. While these technologies have legitimate uses, their misuse has posed a systemic threat to the credibility of news, personal reputation, and even the social information ecosystem, prompting academia and industry to invest heavily in developing deepfake detection technologies.
[0003] Existing multimodal forgery detection methods mainly fall into two technical routes: first, image-text forgery detection identifies tampering traces in image-text pairs through cross-modal semantic consistency; second, audio-video forgery detection utilizes the temporal synchronization of audio and video as a criterion. Both routes have made some progress, but they share the following structural defects:
[0004] I. Multimodal forgery often involves four channels simultaneously: image, text, video, and audio. Existing methods either only handle image-text bimodality or only audio-video bimodality, lacking a unified detection framework that can integrate four modalities at the same time. When forgers construct semantic loops between the four modalities, bimodal detectors cannot cope with four-modal collaborative forgery and have difficulty perceiving abnormal signals from the other two modalities, forming a detection blind spot.
[0005] Second, cross-modal alignment tends to force different modalities to converge in semantic space and actively discard samples that are inconsistent between modalities. However, there are natural information differences between text, images, audio and video in real news. Forced alignment will misjudge normal differences as forgery and deliberately eliminate the ambiguity that may originally exist between modalities. The lack of ambiguity may be a signal of forgery, resulting in a lack of the most discriminative evidence.
[0006] Third, existing models only output true / false binary classification decisions and do not provide information on where the tampering occurred, which limits the detection task to true / false discrimination and lacks fine-grained localization of the tampered area;
[0007] Therefore, it is necessary to invent a four-modal collaborative forgery detection and localization method based on ambiguity gating and modal hierarchical fusion to solve the above problems. Summary of the Invention
[0008] The purpose of this invention is to provide a method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion, so as to solve the above-mentioned shortcomings in the technology.
[0009] To achieve the above objectives, the present invention provides the following technical solution: a method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion, comprising the following steps:
[0010] Step 1: Given an image-text pair and an audio-video pair, perform multimodal coding to obtain the embedding representation of each modality;
[0011] Step 2: Based on image embedding sequence Text embedding sequence The spatiotemporal characteristics of video Spatiotemporal characteristics of audio Multimodal fusion is performed to obtain cross-modal ambiguity scores. ;
[0012] Step 3: Based on cross-modal ambiguity scores Image-text gating fusion is performed to obtain the fused representation on the image side. ;
[0013] Step 4: Spatiotemporal features based on video Spatiotemporal characteristics of audio and cross-modal ambiguity scores The audio and video fusion representation is obtained. ;
[0014] Step 5: Based on the global semantic vector of the image after image-text fusion Global semantic vector after audio and video fusion Cross-branch contrastive learning is performed to obtain the cross-branch contrastive loss. ;
[0015] Step 6: Image-based fusion representation and audio-visual fusion representation To perform image forgery localization, the image forgery localization loss is obtained. ;
[0016] Step 7: Based on text embedding sequence Image embedding sequence and audio-visual fusion representation To locate and forge text.
[0017] Step 8: Audio-video fusion representation Text-side multimodal representation They used audio and video spoofing to locate individuals.
[0018] Step 9: Based on all losses, calculate the overall loss of the model and perform joint optimization training.
[0019] Compared with the prior art, the beneficial effects of the present invention are:
[0020] This invention simultaneously receives and encodes four modal information (image, text, video, and audio), and performs cross-modal alignment and gating fusion within the image-text branch and the audio-video branch respectively before performing cross-branch comparative learning. This enables the four modal evidence chains to collaboratively participate in the judgment within a unified semantic space, thereby effectively solving the problem that existing methods are limited to bimodal modality and have difficulty dealing with four-modal collaborative forgery.
[0021] By using variational modeling, the single-modal features are parameterized into a Gaussian distribution, and the information difference between modalities is quantified by KL divergence to obtain a cross-modal ambiguity score. This ambiguity score is then used as a sample-level and local-level gating signal to adaptively adjust the fusion weight of single-modal features and cross-modal features. This transforms ambiguities that are traditionally discarded as noise into discriminative clues, effectively suppressing false associations created by forgers through forced alignment. This solves the problem that existing alignment strategies force modal convergence and lose semantic ambiguity.
[0022] By employing three localization heads—image forgery region localization, text forgery word localization, and audio / video forgery segment localization—the invention outputs image tampering bounding boxes, text forgery words, and audio / video forgery time segments, respectively. This allows for accurate identification of the location of tampering while simultaneously determining authenticity. This addresses the problem of existing detection tasks being limited to authenticity discrimination and lacking fine-grained localization, significantly improving the interpretability and forensic value of the detection results. Furthermore, the four-modal semantic redundancy, ambiguity gating for noise propagation suppression, and semantic-level discrimination method make this invention more robust to signal degradation caused by JPEG compression and additive noise. Attached Figure Description
[0023] Figure 1 This is a flowchart of the present invention;
[0024] Figure 2 This is a statistical chart of the ITVA dataset of the present invention;
[0025] Figure 3 The image shows the detection and localization results of collaborative forgery by AGMF-Net of the present invention.
[0026] Figure 4 This is a schematic diagram comparing the attention maps on entity-level and attribute-level forged samples according to the present invention;
[0027] Figure 5 t-SNE visualization of the image representation extracted at different processing stages of the AGMF-Net of the present invention;
[0028] Figure 6 This is a visualization diagram of the location of a single-sample audio / video forged segment in this invention;
[0029] Figure 7 This is a schematic diagram comparing the temporal boundaries of the predicted forged fragment and the real forged fragment according to the present invention;
[0030] Figure 8 This is a boundary error distribution diagram of the audio and video forgery segment of the present invention. Detailed Implementation
[0031] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.
[0032] This invention provides the following: Figure 1-8 The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion, as shown, includes the following steps:
[0033] Step 1: Given an image-text pair and an audio-video pair, perform multimodal coding to obtain the embedding representation of each modality. The specific steps are as follows:
[0034] 1.1 Given an image-text pair Image encoder The image is segmented into multiple image patches, and these patches are encoded into an image embedding sequence using a self-attention layer and a feedforward network. The specific formula is as follows:
[0035] ;
[0036] in, For image, For text; For global semantic embedding of images, and , For the set of real numbers, For feature dimensions; For the corresponding Embedding of image patches, and , The total number of image blocks; For the first Embedding vectors of image patches, ; The joint data distribution of image-text pairs, i.e. ;
[0037] 1.2. Text encoder adopted Encode the text into a text embedding sequence. The specific formula is as follows:
[0038] ;
[0039] in, For text Global semantic embedding, and ; For the corresponding The embedding of text tags, and ; For the first The embedding vectors of text words, and , , The total number of words in the text;
[0040] 1.3 Given an audio / video pair Divide the video into Each segment is selected from [number] segments. Using keyframes as video-level input, and utilizing short-time Fourier transform to... The audio segments corresponding to each keyframe are processed to obtain the Mel spectrum as the audio level input;
[0041] Where V represents video and A represents audio;
[0042] 1.4. The Transformer module is used to perform initial encoding on the audio-level input to obtain the initial video embedding sequence. and audio initial embedding sequence The specific formula is as follows:
[0043] ;
[0044] in, For global semantic embedding of video, For audio global semantic embedding, For the first The first video clip One keyframe For the first Audio features corresponding to each video segment;
[0045] The Transformer module is a deep learning model that uses a self-attention mechanism to model the temporal context of video keyframe sequences and audio Mel-spectral feature sequences, capturing long-distance semantic dependencies between different video and audio segments. It also introduces global classification labels to encode video global semantic embeddings and audio global semantic embeddings, thereby outputting initial video embedding sequences and initial audio embedding sequences carrying temporal context information. This is a conventional existing technology and will not be described in detail here.
[0046] In step 1.4, average pooling can be performed on the initial embedded sequence of the video to ensure temporal coordination between audio and video modalities. The specific formula is as follows:
[0047] ;
[0048] in, For the first The first video clip One keyframe;
[0049] 1.5 Using a time encoder and space encoder Processing the initial embedded sequence of video and audio initial embedding sequence To obtain the spatiotemporal features of the video and the spatiotemporal characteristics of audio The specific steps are as follows:
[0050] 1.5.1 The time encoder embeds the sequence from the initial video sequence. and audio initial embedding sequence Extract time features from each part, and assume that the time encoder has a total of Layers, for modes , No. The update rule for the video or audio temporal features of a layer is as follows:
[0051] ;
[0052] in, ; For the sake of the bulls' self-attention, For layer normalization, For the first Layer-learnable temporal location encoding, To add element by element, ; For time encoder No. Video or audio temporal features extracted from the layer; For time encoder No. Video or audio temporal features extracted from the layer;
[0053] 1.5.2 Initialization , The time encoder output is recorded as and ;
[0054] in, For time encoder Temporal features of the video extracted from layer L; Time encoder The temporal features of the audio extracted from layer L;
[0055] 1.5.3 The spatial encoder extracts spatial features from the temporal features, and... Reconstructed into feature maps and divided into From image patches, an initial spatial embedding sequence is obtained. :
[0056] ;
[0057] in, For time encoder The time features extracted from the Lth layer, when hour, = ; for corresponding Image patch embedding, For learnable location encoding; This is an element-wise addition;
[0058] 1.5.4. Assume the spatial encoder has a total of Layers, for single-modal features , No. The update rule for the spatial features of a layer is as follows:
[0059] ;
[0060] in, , It is a feedforward layer; For the first The output of the layer space encoder belongs to the modal. Spatial characteristics; For the first The output of the layer space encoder belongs to the modal. Spatial characteristics;
[0061] 1.5.5 Calculate the spatiotemporal features of the video and audio, using the following formulas:
[0062] = ;
[0063] = ;
[0064] This represents the spatiotemporal characteristics of the video output by the Kth layer spatial encoder. This represents the spatiotemporal characteristics of the audio output by the Kth layer spatial encoder; For the first The output of the layer space encoder belongs to the modal Spatial characteristics, when hour, = ;when hour, = ;
[0065] In step 1, features are extracted from the four modalities by using an image encoder, a text encoder, and a spatiotemporal encoder respectively, to obtain a unified dimensional single-modal feature representation, providing a high-quality input foundation for subsequent cross-modal semantic interaction and fusion.
[0066] Step 2: Based on image embedding sequence Text embedding sequence The spatiotemporal characteristics of video Spatiotemporal characteristics of audio Multimodal fusion is performed to obtain cross-modal ambiguity scores. The specific steps are as follows:
[0067] 2.1 Construct a cross-modal image-text alignment module to embed images into the sequence. and text embedding sequence Mapping to the shared semantic space yields the image modality in the image-text shared semantic space. Text modalities in the semantic space shared with images and text The specific formula is as follows:
[0068] ;
[0069] ;
[0070] The image-text cross-modal alignment module includes an image mapping network. and text mapping network ;and and It consists of a modality-specific MLP and a modality-sharing layer. Used to preserve the unique information of each modality Used to project both into a shared semantic space;
[0071] in, , ; It is a set of image patch representation vectors in the image-text shared semantic space; This is the global semantic vector in image modality representation; This is the global semantic vector in the text modal representation; For text modal representation in the semantic space of image-text sharing; A set of text lexical representation vectors in a shared semantic space for images and text; For the first in the semantic space of image and text sharing Embedding vectors of text words; This is the embedding vector of the first text word in the image-text shared semantic space;
[0072] 2.2 The image-text cross-modal alignment module is trained through an auxiliary binary classification task, using cosine embedding loss. The alignment loss is constrained, and the specific formula is as follows:
[0073] ;
[0074] in, To define paired labels, if the image and text If they come from the same real news, then If they come from different real news sources, then ; , Indicates a positive sample pair. Indicates negative sample pairs; For boundary thresholds;
[0075] The cosine embedding loss maximizes the cosine similarity between positive sample pairs and forces the similarity of negative sample pairs below a specified boundary.
[0076] 2.3. The single-modal representation E is modeled as a Gaussian distribution using the reparameterization technique, and the differences between modes are calculated using KL divergence. Specifically:
[0077] The variational posterior of single-modal observations is expressed as: ;
[0078] in, Normal distribution, representing the variable Follow the mean variance is It follows a normal distribution; and the mean is... and variance It is obtained from a single-modal representation of a specific encoder network;
[0079] 2.4 For each sample Image variational posterior Text variational posterior Defined as:
[0080] ;
[0081] ;
[0082] Let be the global semantic vector in the image modal representation of the i-th sample in the batch; Let be the global semantic vector in the text modal representation of the i-th sample in the batch;
[0083] 2.5. Approximately obtained by sampling the marginal distribution across the entire dataset. and The specific formula is as follows:
[0084] ;
[0085] in, The edge distribution of image modalities across the entire dataset. The marginal distribution of the text modality across the entire dataset is represented by both the marginal distribution and the marginal distribution of the modality, which are obtained by the variational posterior mean of each sample within the batch. This refers to the number of samples in a small batch, i.e., the batch size.
[0086] 2.5 Calculate the sample using normalized KL divergence Cross-modal ambiguity score The specific formula is as follows:
[0087] ;
[0088] in, KL divergence is used to measure the degree of difference between two probability distributions; cross-modal ambiguity score. The smaller the value, the closer the two single-modal distributions are, meaning the high degree of information overlap between the modes; conversely, the larger the value, the greater the information gap between the modes, meaning the strong complementarity. This cross-modal ambiguity score is used in the subsequent fusion module to dynamically adjust the contribution weights of single-modal features and cross-modal features.
[0089] In step 2, by mapping the image-text unimodal representation to a shared semantic space and quantifying cross-modal ambiguity, the information differences between modalities are transformed into quantifiable discriminative cues, providing fine-grained control signals for subsequent image-text fusion.
[0090] Step 3: Based on cross-modal ambiguity scores Image-text gating fusion is performed to obtain the fused representation on the image side. The specific steps are as follows:
[0091] 3.1. Image Modality Representation Based on Alignment and text modal representation Calculate cross-modal attention weights, and compute global textual information for both image global semantics and text global semantics. The specific formula is as follows:
[0092] ;
[0093] in, The global semantic vector in its image modality representation is used when calculating attention. , When calculating attention, the global semantic vector in its text modality representation is used. ; The normalization exponential function is used to map the input vector to a probability distribution. After mapping, the values of each element are in the range of (0,1), and the sum of all elements is 1. Transpose of a vector;
[0094] 3.2, based on and Calculate local text information of local semantics of images and local semantics of text. The specific formula is as follows:
[0095] ;
[0096] 3.3. Transfer global text information Aggregate to superior This yields a text-conditionalized global semantic vector of the image. The specific formula is as follows:
[0097] ;
[0098] 3.4. Transfer local text information Aggregation yields a set of text-conditionalized image patch representation vectors. The specific formula is as follows:
[0099] ;
[0100] in, Let be the text-conditionalized representation vector of the j-th image patch;
[0101] 3.5, based on , ,as well as The original features and cross-modal features are weighted and combined to obtain the global semantic vector of the image after image-text fusion. The specific formula is as follows:
[0102] ;
[0103] 3.6, Based on , ,as well as After introducing a patch-level gating mechanism to calculate local gating coefficients, weighted fusion is performed to obtain the fused image local block representation vector. The specific formula is as follows:
[0104] ;
[0105] in, for The first in A vector, for The first in The patch-level gating mechanism is implemented using a two-layer MLP, with the hidden layer dimension (i.e., the feature vector dimension) set to [value missing]. It uses GELU activation, and the output is mapped to a sigmoid function. , No. Local control coefficient of an image patch for:
[0106] ;
[0107] in, This represents vector concatenation. and All parameters are learnable, and the cross-modal ambiguity score is... As a scalar, it is broadcast to the network during splicing. Same dimension, For the Sigmoid function;
[0108] 3.7. Based on global semantic vectors of images and image local block representation vector The fusion representation of the image side is calculated. The specific formula is as follows:
[0109] ;
[0110] in, ;
[0111] In step 3.7, the generalization characteristics of this gating mechanism are analyzed, specifically as follows:
[0112] set up This is the hypothesis space for gateless fusion. To introduce the hypothesis space after gating, the gating coefficients are used. The input space is softly divided into a finite number of regions (each region corresponds to a set of activation modes). Can be regarded as Composition with a low-dimensional gated function class;
[0113] At this point, according to Rademacher's composite bound on complexity, we have ,in For the parameter dimensions of the gated network, Let be the number of samples. This upper bound indicates that when At this time, the introduction of the gating mechanism does not significantly increase the complexity of the hypothesis space, and the risk of overfitting is controlled; under the experimental settings... , hour, The conditions are met;
[0114] In step 3, an adaptive fusion of image and text features is achieved through a dual gating mechanism at the sample and patch levels. This mechanism can dynamically adjust the fusion weights based on modal differences at the global level, and accurately locate image regions that are inconsistent with the text semantics at the local level, providing a fine-grained feature foundation for subsequent forgery region localization.
[0115] Step 4: Spatiotemporal features based on video Spatiotemporal characteristics of audio and cross-modal ambiguity scores The audio and video fusion representation is obtained. The specific steps are as follows:
[0116] 4.1 Construct an audio-video cross-modal alignment module to align the spatiotemporal features of the video. and the spatiotemporal characteristics of audio Mapping to a shared semantic space yields video modal representations. and audio modal representation The specific formula is as follows:
[0117] ;
[0118] ;
[0119] in, The total number of time segments. This is a set of temporal segment representation vectors for audio modal representation in the audio-video shared semantic space; This refers to the global semantic vector representing the audio modality in the shared semantic space of audio and video. For audio number A representation vector for each time segment; It is the global semantic vector representing the video modality in the audio-video shared semantic space; This is a set of temporal segment representation vectors for video modal representation in the audio-video shared semantic space; For the video number A representation vector for each time segment;
[0120] The audio-video cross-modal alignment module includes a video mapping network. and audio mapping network ; and Both consist of a modality-specific MLP and a modality-sharing layer. The modality-specific MLP is used to preserve the unique information of video monomodality and audio monomodality, while the modality-sharing layer is used to project the video monomodality representation and audio monomodality representation onto a shared semantic space.
[0121] 4.2. Define the pairing tag as Using the alignment loss function The constraints are established for it, and the specific formula is as follows:
[0122] ;
[0123] Among them, if the video and audio If they come from the same real sample, then If it is a scrambled mismatch or asynchronous audio / video pair, then ; is the cosine similarity function, used to calculate the cosine angle between two vectors. Its value ranges from [−1, 1]. The larger the value, the more similar the two vectors are. This is a maximum value function used to find the larger of two values. This represents the boundary threshold in the audio / video alignment loss.
[0124] 4.3 Video Modal Representation and audio modal representation Cross-modal ambiguity score Adaptively fuse video and audio features to generate an audio-video fusion representation. The specific steps are as follows:
[0125] 4.3.1. Based on video modal representation and audio modal representation Calculate the cross-modal attention weights from video to audio;
[0126] The cross-modal attention weights from video to audio include the attention weights of the video global semantic vector to the audio global semantic vector. And the attention weights of video temporal segments to audio temporal segments. The specific calculation formula is as follows:
[0127] ;
[0128] ;
[0129] in, , , , ;
[0130] 4.3.2 Based on cross-modal attention weights from video to audio, audio information is aggregated into the video representation to generate audio-conditional video features;
[0131] Among them, audio-conditionalized video features include audio-conditionalized video global semantic vectors. and audio-conditional video temporal segment representation vector set The specific calculation formula is as follows:
[0132] ;
[0133] ;
[0134] in, This is the audio-conditional video global semantic vector; A set of audio-conditionalized video temporal segment representation vectors; For the first The representation vector of a video segment after audio enhancement. ; For the first A representation vector of an audio segment. ;
[0135] 4.3.3 Utilizing cross-modal ambiguity scoring The global semantic vector of video modal representation in the audio-video shared semantic space. With audio-conditional video global semantic vector Weighted fusion is performed to obtain the global semantic vector after audio and video fusion. :
[0136] ;
[0137] in, ;
[0138] 4.3.4. Audio-conditionalized set of video temporal segment representation vectors The set of temporal segment representation vectors for video modal representation in the audio-video shared semantic space. ,as well as A segment-level gating mechanism is introduced for weighted fusion to obtain the audio-video fused video. Each time segment represents a vector. :
[0139] ;
[0140] in, Furthermore, the segment-level gating network is implemented using a two-layer MLP, which enables each temporal segment to undergo consistent adaptive fusion based on local audio and video. This segment-level gating network, for the first... Each time segment has a gating coefficient. for:
[0141] ;
[0142] in, and All parameters are learnable, and the cross-modal ambiguity score is... As a scalar, it is broadcast to the network during splicing. Same dimension;
[0143] 4.3.5, Regarding Repeat step 4.3.4 for each time segment to obtain a set of time segment representation vectors. The specific formula is as follows:
[0144] ;
[0145] 4.3.5. Based on the time-series segment representation vector set Global semantic vector after audio and video fusion The audio-video fusion representation is calculated. The specific formula is as follows:
[0146] .
[0147] In step 4, audio and video cross-modal alignment and sample-level and segment-level dual gating fusion are used to enable audio and video features to adaptively interact in the shared semantic space. This not only preserves the fine-grained information of the single modality, but also enhances the cross-modal temporal consistency modeling capability, providing a high-quality fusion representation for audio and video forgery detection and localization.
[0148] Step 5: Based on the global semantic vector of the image after image-text fusion Global semantic vector after audio and video fusion Cross-branch contrastive learning is performed to obtain the cross-branch contrastive loss. The specific steps are as follows:
[0149] 5.1 Suppose a batch contains For the nth sample, for the nth Let the global semantic vector of the image after image-text fusion be denoted as _ samples. For the first For each sample, its global semantic vector after audio-video fusion is denoted as . ;
[0150] in, Batch size;
[0151] 5.2. Define the pairing tag as ;
[0152] in, If the first The first image and text sample and the first If multiple audio and video samples come from the same sample, then , indicates a positive sample pair, otherwise , which represents a negative sample pair;
[0153] 5.3. Global Semantic Vector of Image Based on Image-Text Fusion Global semantic vector after audio and video fusion Pairing tags Calculate cross-branch contrastive loss The specific formula is as follows:
[0154] ;
[0155] in, The cosine similarity function is used. The boundary threshold for image-to-audio-video cross-branch alignment; this cross-branch contrast loss. It can simultaneously optimize the alignment of positive sample pairs and the uniform distribution of negative sample pairs, and , Their optimization directions complement each other, jointly promoting the learning of discriminative cross-modal representations;
[0156] In step 5, cross-branch contrastive learning maps the text / image branch and the audio / video branch to a shared semantic space, bringing text / image representations from the same sample closer to each other and moving representations from different news sources further apart. This enhances the discriminativeness and consistency of the four-modal representations and provides text / image semantic constraints for subsequent audio / video localization.
[0157] Step 6: Image-based fusion representation and audio-visual fusion representation To perform image forgery localization, the image forgery localization loss is obtained. The specific steps are as follows:
[0158] 6.1. Enhanced image fusion representation through cross-attention to obtain audio-visual guided enhanced image representation. The specific formula is as follows:
[0159] ;
[0160] in, For standard attention operations, As for the key and value, its dimensions are: Through linear transformation and Dimensions Alignment; Enhanced image representation Used to locate image areas that are inconsistent with the audio and video content;
[0161] 6.2. Enhance the image representation The decomposition is performed, and the specific formula is as follows:
[0162] ;
[0163] in, To enhance the global semantic vector in the image representation, and ; For the corresponding A set of augmented representation vectors for local image patches, and ;
[0164] 6.3. Employ a learnable aggregate tag. As a query, the result is... Forged related partial information The specific formula is as follows:
[0165] ;
[0166] For scaling dot product attention functions; It is a learnable aggregation tag, and This is used to aggregate forgery-related clues from local features;
[0167] 6.4. Forging relevant partial information Input bounding box detector Calculate the loss for image forgery localization The specific formula is as follows:
[0168] ;
[0169] in, For generalized intersection and comparison of losses; This is a sigmoid function used to map any real number to the interval (0,1). Here, it is used to convert the output of the bounding box detector into normalized bounding box coordinates. This is a bounding box detector based on a multilayer perceptron, used to detect forged relevant local information. Mapped to the predicted bounding box coordinates of the image forgery region; The labels are the true bounding boxes of the forged regions in the image, and Normalized coordinates are used It means that among them The coordinates of the top left corner of the bounding box. The coordinates of the bottom right corner of the bounding box; Indicates the distribution of data Below image-text pairs The expected value is used here to calculate the average loss over the entire training set;
[0170] In step 6, the audio-video fusion representation is used as a key-value enhanced image fusion representation through cross-attention, and the forgery-related local information in local blocks of the image is aggregated using aggregation tags, thereby accurately locating the tampered areas in the image that are inconsistent with the audio-video content.
[0171] Step 7: Based on text embedding sequence Image embedding sequence and audio-visual fusion representation The specific steps for text forgery location are as follows;
[0172] 7.1 Text-based embedding A multimodal aggregator was used. In the The layer performs text self-attention, resulting in the first... The text representation output after the layer performs text self-attention is shown in the following formula:
[0173] ;
[0174] in, For multimodal aggregators The Middle The text representation output after the layer performs text self-attention, and , stacked together Layer interaction block, ; For multimodal aggregators The Middle The text representation output by the layer after performing text self-attention is used as the first... Input for layered text self-attention; It is a text embedding sequence, and ; For the first Multi-head text self-attention function for layers is used to model the contextual dependencies between different terms in a text sequence;
[0175] 7.2, Based on and image embedding sequences Perform cross-attention to enhance the text representation with image information, and obtain the first... Text representation enhanced by the original image information The specific formula is as follows:
[0176] ;
[0177] in, For the first Text representations enhanced from the original image information are embedded into a sequence via cross-attention. The visual information in the text is integrated into the text representation, and ; To scale the query matrix in the dot product attention function, here That is, using the current text representation as a query to retrieve relevant visual information in an image; This is a standard scaled dot product attention function, whose inputs are a query, a key, and a value, represented here as text. For querying, image embedding sequences Simultaneously serving as both key and value, the text's cross-modal attention to the image is calculated;
[0178] 7.3, Based on and and cross-modal ambiguity scores Construct a token-level gating tensor The specific formula is as follows:
[0179] ;
[0180] in, For the first The text term-level gated tensor of the layer, and Each element takes a value in the range of (0,1), and is used to control the fusion ratio of the original text features and the image enhancement features at each text word position; and These are learnable parameters;
[0181] 7.4 Token-based gated tensors ,right and Perform gating fusion to obtain the first Text representation after layer-gated fusion The specific formula is as follows:
[0182] ;
[0183] in, ;
[0184] 7.5, Based on the first Text representation after layer-gated fusion After passing through a feedforward network and connecting to the residual, the first... The output of self-attention in layer interaction blocks The specific formula is as follows:
[0185] ;
[0186] In step 7.1, It is the first The output of the layer's self-attention is used as an intermediate result of the layer's interaction block, and in step 7.4, It is the final output of the same layer after passing through the feedforward network and residual connections;
[0187] Steps 7.1 to 7.5 together constitute the first step. The complete calculation process of the layer interaction block, after the update in step 7.5, The value is updated to a richer multimodal representation;
[0188] in, For the first The feedforward network function of the layer consists of two linear transformation layers and a nonlinear activation function (such as GELU), used to perform gated fusion of the text representation. Further semantic transformation and feature mapping are performed; , This represents the total number of interaction blocks.
[0189] 7.6, in the The output of self-attention in layer interaction blocks Based on, stack The final text-side multimodal data is obtained after layering. The specific formula is as follows:
[0190] ;
[0191] 7.7 Text-based multimodal and audio-visual fusion representation By fusing audio and video information through cross-attention, a text representation enhanced with audio and video information is obtained. The specific formula is as follows:
[0192] ;
[0193] Among them, it is a multimodal aggregator and token detector Maintain momentum versions separately and The momentum update rule is ,in These are the current model parameters. Let be the momentum coefficient; the momentum version is denoted as . and , For the final text-side multimodal representation The momentum version, Text representation after audio and video information enhancement The momentum version; n cls For the enhanced global semantic vector, n token The enhanced text word representation vector set;
[0194] 7.8. Based on the enhanced global semantic vector n cls Binary classifiers were used respectively. and multi-label classifier The system performs image and text authenticity verification and forgery type identification, and obtains the image and text binary classification loss. and multi-classification loss of counterfeit types The calculation formula is as follows:
[0195] ;
[0196] ;
[0197] in, For cross-entropy loss, For labeling images and text to indicate authenticity. To forge type tags (entity-level FS+TS or attribute-level FA+TA); For image-text binary classification, the cross-entropy loss function is used to constrain the model to perform binary classification of image and text samples as real or fake. For multi-class classification loss of forgery types, the cross-entropy loss function is adopted to constrain the model to identify the forgery type (whether it is entity-level collaborative forgery or attribute-level collaborative forgery).
[0198] 7.9. Based on the enhanced text lexical representation vector set n token Using a token detector Predict the forgery probability of each token to obtain the text forgery lexical localization loss. The specific formula is as follows:
[0199] ;
[0200] in, For the loss function of locating forged lexical units in text, the cross-entropy loss function is used. For each text term output by the token detector, the probability of forgery is given. Forged lexical tags that are genuine, The cross-entropy loss function;
[0201] 7.10 Token-based detector and momentum version Calculate the momentum version of the KL divergence constraint The specific formula is as follows:
[0202] ;
[0203] in, Momentum KL divergence constraint loss for text forgery token localization is used to constrain the current token detector. Output and momentum version To keep the output consistent, in order to mitigate the overfitting problem caused by noise annotation;
[0204] 7.11 Forgery Probability Based on Textual Truncation KL divergence constraint and momentum version The weighted summation yields the final text forgery location loss. The specific formula is as follows:
[0205] ;
[0206] in, To balance the hyperparameters; Total loss for text forgery location;
[0207] In step 7, image visual information and text semantic information are adaptively fused word by word through text self-attention, image cross-attention and word-level gating fusion, so as to accurately locate the tampered text words and alleviate the overfitting problem caused by noise annotation.
[0208] Step 8: Audio-video fusion representation Text-side multimodal representation The specific steps for audio and video spoofing and location tracking are as follows:
[0209] 8.1 Audio-video fusion representation Text-side multimodal representation By fusing audio / video features with image / text features through cross-attention, an enhanced audio / video representation guided by image / text is obtained. The specific formula is as follows:
[0210] ;
[0211] in, Enhanced audio and video representation guided by images and text, and ; Its global semantic vector, ; Let it be the set of vectors representing its time-series segments, and ;
[0212] 8.2 Enhanced Audio and Video Representation Based on Graphic and Text Guidance The Middle Features of each time segment Input them into the binary classification head respectively and return head , obtained the Each time segment is a forged scalar probability. and the predicted left and right time offsets. :
[0213] ;
[0214] ;
[0215] in, For the Sigmoid function, and are learnable parameters, and , ; To predict the forged fragment relative to the first The time offset to the left of the center of each segment, To predict the forged fragment relative to the first The time offset to the right of the center of each segment; An activation function to ensure that the predicted offset is non-negative; For the regression head H r The learnable weight matrix, and : For the regression head H r The learnable bias vector, and :
[0216] 8.3 Adjust the left and right time offsets Reconstructing the predicted forged fragment The specific formula is as follows:
[0217] ;
[0218] in, For the first The center time of each time segment;
[0219] 8.4 Based on scalar probability and predicting fake fragments Calculate the loss for audio and video forgery detection and location. The specific formula is as follows:
[0220] ;
[0221] in, For fragment classification loss, For regression loss; The number of fake fragments in the current sample. This is the regression loss balance coefficient;
[0222] Among them, use Instead Normalization can prevent the location monitoring signal from being diluted by a large number of real segments when there are few fake segments;
[0223] 8.5 Based on scalar probability The segment classification loss is calculated using the following formula:
[0224] ;
[0225] in, This is a segment classification loss used to alleviate the problem of imbalanced positive and negative samples; if ,but ,otherwise ;like ,but ,otherwise ; It is a positive sample weighting factor, and ; This is the focus parameter; if the segment is fake, i.e. ,but If the segment is true, that is... ,but ; For the first The class weight factor for each time segment is used to balance the loss contribution of positive and negative samples: if ,but ;like ,but ;
[0226] 8.6. Regarding meeting the requirements The A time series segment, based on predicted fake segments and fake fragment tags The Center-DIoU loss was used to calculate the regression loss of the forged fragments. The specific formula is as follows:
[0227] ;
[0228] in, For the first The regression loss for each time segment is the Center-DIoU loss, used to constrain the prediction of fake segments. With fake fragment tags The time boundary error between them; For the first The predicted forged segments of each time segment are obtained by reconstructing the left and right time offsets, and , For the predicted start time, This is the predicted end time; For the first Forged fragment labels for each time segment, and , To fabricate the true start time of the clip, the unit is seconds; To simulate the true end time of the segment, the unit is seconds; ; The intersection-over-union ratio (IoU) between predicted time segments and actual time segments;
[0229] To predict the distance between the center of the segment and the center of the actual segment; To simultaneously cover the minimum closure interval length of both predicted and ground truth segments, and ; To prevent extremely small constants with a denominator of zero;
[0230] 8.7 Scalar Probability Based on Each Time Segment By extracting the maximum forgery probability (max) from all segments j Compare it with the judgment threshold After comparison, binarization is performed for discrimination, and aggregation is used to obtain the overall audio and video authenticity prediction. The specific formula is as follows:
[0231] ;
[0232] in, The overall fake labels for the audio and video predicted by the model are obtained by aggregating the segment-level prediction probabilities, and If the maximum forgery probability of all time series segments Exceeding the judgment threshold ,but This indicates that the audio / video sample has been determined to be fake; otherwise... , indicating that it is determined to be true; among them, For indicator functions, The threshold is used for judgment; if the probability of forgery of at least one segment exceeds the threshold, the audio / video sample is judged as forged; otherwise, it is judged as real.
[0233] 8.8 Based on scalar probability Greater than the threshold Candidate forgery fragment set The non-maximum suppression algorithm is used to iteratively retain the fragment with the highest confidence and remove redundant fragments that highly overlap with it, thus obtaining the final set of forged fragments. The specific formula is as follows:
[0234] ;
[0235] in, The overlap threshold, This represents the final set of forged fragments retained after NMS. This is the final set of forged fragments retained after nonmaximum suppression, containing all time-series fragments that were identified as forged and filtered. It is a non-maximum suppression function used to remove highly overlapping redundant segments in predicted fake segments and retain the segments with the highest confidence. The overlap threshold for non-maximum suppression is used to determine whether two segments are suppressed. When the cross-union ratio of two predicted segments exceeds this threshold, the segment with higher confidence is retained and the segment with lower confidence is suppressed. For the first The predicted start time of the forged fragment, in seconds. For the first The end time of the predicted fake segment is obtained by reconstructing the left and right time offsets output by the regression head. For the first The probability of forgery for each time segment is determined by the classification head. The output represents the confidence level that the fragment contains fake content;
[0236] In step 8, the audio and video temporal representation is enhanced by cross-attention guided by images and text, and the classification head and regression head are used to perform segment-level forgery probability prediction and temporal boundary regression, respectively, thereby achieving fine-grained localization of forged audio and video segments.
[0237] Step 9: Based on all losses, calculate the overall model loss and perform joint optimization training. The specific steps are as follows:
[0238] 9.1 Calculate the overall loss of the four-modal collaborative forgery detection and localization model based on ambiguity gating and modal hierarchical fusion. The specific formula is as follows:
[0239] ;
[0240] in, for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters;
[0241] 9.2. Overall model loss based on calculations It employs end-to-end supervised learning, updating model parameters through gradient backpropagation.
[0242] In step 9, the end-to-end training of the four-modal collaborative forgery detection and localization model is achieved by jointly optimizing eight losses: image-text alignment, audio-video alignment, cross-branch comparison, image localization, text localization, audio-video localization, and image-text discrimination and forgery type recognition.
[0243] Multimodal forgery dataset:
[0244] 1.1 Dataset Overview:
[0245] ITVA is the first benchmark dataset specifically designed for the detection and localization of four-modal collaborative forgery in text, image, audio, and video. Unlike existing single-modal / dual-modal datasets, ITVA's core feature lies in its "collaborative forgery" attribute—all four modalities of each forged sample are tampered with, and the forgery types of the text and image branches and audio and video branches are strictly consistent in semantics. Specifically, it is divided into two categories: entity-level collaborative forgery (face replacement + text entity replacement + video face replacement + audio identity cloning) and attribute-level collaborative forgery (expression editing + text sentiment inversion + video expression editing + audio sentiment tampering).
[0246] In addition, ITVA provides three levels of fine-grained annotation: bounding boxes of image forgery regions, lexical masks of text forgery, and time segments of audio and video forgery. The dataset covers four modalities: image, text, audio, and video, and includes real and forged samples, covering multiple fields such as politics, entertainment, and social events, combining diversity with the challenge of evidence collection.
[0247] 1.2 Source Data Acquisition:
[0248] This section describes how to construct a multimodal media sample library by paying to crawl multimedia content from multiple mainstream news platforms. Each sample From the original image News headlines Video clips and audio stream To ensure the diversity and authenticity of the data, the collected content covers multiple fields such as politics, entertainment, and social events. All data has been manually reviewed to remove low-quality or irrelevant samples.
[0249] 1.3 Sample Generation:
[0250] 1.3.1 Generation of Image and Text Forgery Samples:
[0251] Entity-level image-text forgery: This type of sample is generated by replacing the faces of entities in an image with the identities of the entities in the corresponding news text. First, a named entity recognition model is used to extract the information from the original news headline. The entity name `PER1` was identified, and an entity-level text forgery sample was created by replacing `PER1` with `PER2`. For the corresponding original image Using PER2 as the source face and PER1 as the target face, Inswapper was used for face replacement, and GFPGAN was used to enhance the face-swapping area, resulting in entity-level image forgery samples. Obtain by text truncation or completion The former 1 word, using Dimensional heat vector mark Forged words (TS operations) in the text, using bounding boxes mark Image forgery region (FS operation);
[0252] Attribute-level image-text forgery: This type of sample is generated by polarity shifting between facial expressions in images and corresponding sentiment descriptions in news text. First, Chinese-RoBERTa is used to shift the sentiment of news headlines... Based on sentiment polarity, texts are categorized as positive, negative, and neutral. Using the large-scale model GPT, positive sentiment texts are edited to negative sentiment, or negative / neutral sentiment texts are edited to positive sentiment, thus creating attribute-level text forgery samples. and use mark Forged tokens (TA operations) in the corresponding original image; Using LivePortrait, facial expressions under the source emotion are edited into the target emotion through expression control parameters, resulting in attribute-level image forgery samples. and use mark Image forgery region (FA operation) in the image.
[0253] 1.3.2 Generation of forged audio and video samples:
[0254] Video forgery: Video forgery corresponds to FR (Face Replacement) and FE (Face Editing) operations: First, the input video is parsed, and the target video is processed frame by frame. Each frame uses YOLOFace to detect face bounding boxes, and then 2DFAN4 is used to predict facial key points to complete face cropping, geometric alignment, and scale normalization. For entity-level forgery, InsWapper is used in the alignment space to inject the source identity features into the target face to generate a forged face with identity replacement (FR operation). For attribute-level forgery, LivePortrait is used to controllably edit the expression or pose of the target face according to preset attribute control values to generate a forged face with the same identity but altered attributes (FE operation). Both types of forgery results are further enhanced with GFPGAN for texture and detail enhancement. A fusion mask is constructed based on the face detection box and facial key points, and a boundary smoothing strategy is used to seamlessly paste the processed face region back into the original video frame.
[0255] Audio forgery: Audio forgery corresponds to AC (Audio Identity Cloning) and AE (Audio Sentiment Modification) operations: First, the input text is encoded, and CAM++ is used to extract speaker identity and timbre features from the reference audio. Wav2Vec2-BERT is used to extract the speech context representation, and a text sentiment analysis model is used to obtain the sentiment conditions. For entity-level audio forgery, semantic representations with the target speaker's timbre are generated by introducing the target speaker's features (AC operation). For attribute-level audio forgery, the speaker features are kept unchanged, and only the sentiment conditions are replaced or reset to generate semantic representations with altered sentiment expressions (AE operation). Finally, the semantic-level representation is converted into a Mel spectrum based on the conditional flow matching method, and BigVGAN is used to restore the Mel spectrum to a time-domain waveform to obtain the final synthesized speech.
[0256] Audio-video collaboration: Based on the aforementioned method, entity-level forged videos are paired with entity-level audio (FR+AC), and attribute-level forged videos are paired with attribute-level audio (FE+AE). Lip alignment is performed via LatentSync, and the resulting fusion forms a complete forged video. Through this pairing generation process, each forged sample is modified in all four modalities, and the forgery type of the text / image branch and the audio / video branch remains consistent—entity-level text / image forgery (FS+TS) is paired only with entity-level audio / video forgery (FR+AC), and attribute-level text / image forgery (FA+TA) is paired only with attribute-level audio / video forgery (FE+AE)—thus achieving true four-modal collaborative forgery; audio / video forged segment tags are used. Add annotations. The total number of time segments. For the first The tag indicating whether a segment is fake.
[0257] 1.4 Dataset Statistics:
[0258] The dataset contains 2533 image-text-audio-video pairs, including 1485 original samples and 1048 forged samples. The image-text forged samples consist of two types: entity-level collaborative forgery and attribute-level collaborative forgery, corresponding to FS+TS and FA+TA operations, respectively. In the attribute-level sample construction stage, this paper selected 135 pairs of positive sentiment, 273 pairs of neutral sentiment, and 116 pairs of negative sentiment samples from the original data, and generated 276 pairs of positive sentiment image-text samples and 248 pairs of negative sentiment image-text samples through sentiment reverse editing, for a total of 524 pairs of attribute-level image-text forgery samples. The audio-video forgery samples also cover both entity-level and attribute-level collaborative forgery, with 524 samples in each category.
[0259] like Figure 2 As shown, this is a statistical chart of the ITVA dataset, where (a) the figure shows the distribution of image forgery region size in entity-level and attribute-level forgery; (b) the figure shows the distribution of the number of forged text tokens; (c) the figure shows the sample distribution where the sentiment polarity shifts during the generation of attribute-level forgery samples ("+" indicates a shift from neutral / negative to positive; "-" indicates a shift from positive to negative); and (d) the figure shows the distribution of forgery duration for audio and video forgery samples. Figure 2 We can conclude that:
[0260] Image forgery region analysis: such as Figure 2 As shown in (a), both entity-level and attribute-level image forgery regions are mainly concentrated in the 18%–24% range of the entire image, with entity-level samples accounting for about 80% and attribute-level samples accounting for about 72.7%. This indicates that image tampering in ITVA is mainly concentrated in the core area of the face, which places high demands on the model's fine-grained region localization capabilities.
[0261] Text forgery token analysis: such as Figure 2 As shown in (b), entity-level text forgery is highly concentrated in 2-3 tokens, with 2 tokens accounting for 61% and 3 tokens accounting for 39%, which is consistent with the characteristics of replacing person names or identities; the distribution of attribute-level text forgery is basically concentrated in the range of 2-6 tokens, indicating that attribute-level forgery is not only a local word replacement, but also involves the collaborative modification of emotional or semantic attributes, and therefore relies more on context-level semantic modeling.
[0262] Emotional transfer analysis: such as Figure 2 As shown in (c), there are 276 pairs of positive transfer (FA+TA(+)) and 248 pairs of negative transfer (FA+TA(-)) in the attribute-level image-text samples, with the number of positive and negative transfers being relatively close. This design avoids the model being biased towards a single emotional direction, making attribute-level forgery detection more focused on the cross-modal consistency between emotional changes and image expressions;
[0263] Audio and video forgery duration analysis: such as Figure 2 As shown in (d), apart from the original 0-second sample, the forged samples mainly cover three durations: 2 seconds, 5 seconds, and 10 seconds, with 341, 342, and 364 samples respectively, which is relatively balanced overall. Short-term forgery requires the model to capture inconsistencies in local lip movements, voiceprints, and facial expressions, while long-term forgery tests the model's ability to model continuous temporal semantics.
[0264] In summary, the ITVA dataset not only exhibits high diversity in image region scale, text modification granularity, sentiment transfer direction, and audio / video forgery duration, but its collaborative forgery characteristics also provide a more valuable and realistic evaluation benchmark for detecting and locating collaborative forgeries involving images, text, and audio / video.
[0265] Verification experiment:
[0266] 2.1 Experimental Setup:
[0267] 2.1.1 Baseline Model:
[0268] First, we compare our invention, the AGMF-Net model, with the following 11 baseline models:
[0269] (1) Text forgery detection: BERT and Fast-DetectGPT. BERT adopts a dual-model architecture: the first model generates text embeddings as hidden knowledge, which are input into the second model along with the original text to output the forgery probability; Fast-DetectGPT describes the distribution difference between generated text and real text through conditional probability curvature. (2) Image forgery detection: MAT and TS. MAT uses multi-head attention to focus on local forgery traces on the face; TS detects forgery regions based on texture statistics and high-frequency distortion features. (3) Image forgery detection: CLIP, HAMMER and CSCL. CLIP constructs a cross-modal semantic space through large-scale image-text contrast learning; HAMMER introduces forgery perception contrast learning and cross-modal attention; CSCL models contextual consistency and semantic consistency in a unified manner. (4) Audio forgery detection: CFPRF, which uses a two-stage temporal localization framework to realize the boundary localization of audio forgery segments. (5) Video forgery detection: TFCU, which designs a three-module bidirectional mining framework to model inter-frame temporal clues. (6) Audio and video forgery detection: BA-TFD+ and MMMS-BA. BA-TFD+ combines boundary attention with a parallel cross-fusion network; MMMS-BA adaptively learns the contribution weights of each modality at each time step;
[0270] 2.1.2 Implementation details:
[0271] AGMF-Net uses a 12-layer ViT-B / 16 as the image encoder, Chinese-RoBERTa as the text encoder, and a TSE architecture for the audio and video encoders. The main hyperparameter settings are as follows: number of audio and video segments. Number of keyframes per segment ;Gated network hidden layer dimension Alignment boundary threshold Momentum coefficient Text localization loss balance coefficient Regression loss balance coefficient Focal Loss parameters , ; Determination threshold NMS threshold Loss weights to Set all to ;
[0272] The model is implemented based on PyTorch 1.10.0. Each baseline uses the original code and hyperparameters, and necessary detection and localization heads are added for fair comparison. All models are retrained on the ITVA dataset using an NVIDIA RTX 3090 (24 GB VRAM) server. The training, validation and test sets are divided in a 6:2:2 ratio.
[0273] 2.1.3 Evaluation Indicators:
[0274] The experiment evaluation used 18 indicators: (1) Image and text authenticity recognition and audio and video authenticity recognition: ACC, AUC, EER; (2) Image and text forgery type recognition: mAP, CF1, OF1; (3) Image region localization: IoUmean, IoU@50, IoU@75; (4) Text token localization: Precision, Recall, F1; (5) Audio and video forgery localization: AP@0.5, AP@0.75, AP@0.95, AR@1, AR@5, AR@10.
[0275] 2.2 Single-modal forgery detection performance:
[0276] First, the performance of each single-modal baseline is evaluated and compared with AGMF-Net. It should be noted that the text model only takes text as input, the image model only takes an image as input, while AGMF-Net takes complete text and image input, so the two are not directly comparable. The comparison of Table 1 (comparison of image / text single-modal detection and localization performance (%)) and Table 2 (comparison of audio / video single-modal detection and localization performance (%)) aims to quantify the performance improvement of multimodal input relative to single-modal input. The details of Table 1 are as follows;
[0277] Table 1:
[0278]
[0279] Note: "-" indicates that the task is not supported;
[0280] As shown in Table 1, AGMF-Net achieves competitive performance on text-image tasks. Among text detection models, Fast-DetectGPT performs best with an AUC of 91.52%, while AGMF-Net's AUC is 95.25% (3.73 percentage points higher). Text detection models perform poorly in attribute-level forgery because this type of forgery only changes sentiment polarity without disrupting language fluency, making it difficult for pure language models to capture. Among image detection models, TS performs best with an AUC of 96.21%, slightly higher than AGMF-Net, but AGMF-Net's EER is 5.12%, while TS's is 9.45% (4.33 percentage points lower). AGMF-Net's advantage stems from multimodal fusion—trained with complete text-image input, the model can learn more essential semantic representations using cross-modal consistency constraints, benefiting from this joint training even with single-modal output.
[0281] Table 2 details are as follows;
[0282] Table 2:
[0283]
[0284] As shown in Table 2, AGMF-Net outperforms all single-modal baselines in overall performance on audio and video tasks. The AR@1 of the video detection model TFCU is 55.24%, the AR@1 of the audio detection model CFPRF is 71.12%, and AGMF-Net reaches 79.37% (24.13 and 8.25 percentage points higher, respectively). TFCU can capture inter-frame temporal anomalies, and CFPRF can locate spectral boundary anomalies, but both lack cross-modal validation capabilities. The advantage of AGMF-Net comes from multimodal fusion—trained with complete audio and video input, the model can use the cross-modal consistency between lip movements and voiceprints as a discrimination criterion, and it benefits from this joint training even with single-modal output.
[0285] 2.3 Multimodal forgery detection performance:
[0286] This section evaluates the overall performance of AGMF-Net under complete multimodal input. Table 3 reports the results (%) of the image and text detection and localization tasks, and Table 4 reports the results (%) of the audio and video detection and localization tasks. Table 3 is detailed below:
[0287] Table 3:
[0288]
[0289] Table 3 reveals the asynchrony between classification accuracy and positioning accuracy; CSCL outperforms AGMF-Net (88.71%) in ACC (90.58%), but falls short in IoU@50 (71.76% vs. 74.55%) and IoU@75 (63.41% vs. 74.55%). The consistency learning objective of CSCL is lower than that of AGMF-Net (65.23%). This difference indicates that a model that can determine "whether it is consistent" may not be able to locate "where the inconsistency is". From the perspective of feature learning, classification tasks only require a global discriminant signal - the model can compress all modal information into a scalar to judge the truth or falsehood, while localization tasks require the model to retain the local discriminant signal of each token and each patch. CSCL's consistency learning objective naturally tends to global semantic modeling. Its loss function encourages the convergence of image and text representations as a whole, so its classification performance is excellent. However, the fine-grained difference information required for local localization is sacrificed in the feature aggregation process. AGMF-Net explicitly retains the features of each local unit through token-level and patch-level gating mechanisms, so that global discrimination and local localization can coexist in the same feature space.
[0290] Further analysis reveals the asymmetric advantage of Precision-Recall; AGMF-Net's Precision (81.85%) is 3.71 percentage points higher than CSCL (78.14%), but its Recall (78.38%) is slightly lower than CSCL (80.45%). This means that AGMF-Net adopts a more conservative strategy when locating forged tokens: it would rather miss an error than make a false positive. This preference stems from AGMF-Net's ambiguity gating mechanism—the gating value requires sufficient evidence of cross-modal semantic conflict to be activated; otherwise, the original single-modal features are retained. In actual evidence collection scenarios, this conservative strategy is reasonable: mislabeling a real token as forged will damage the credibility of the entire detection report, while missed forged tokens can still be discovered through subsequent manual review.
[0291] Table 4 details are as follows:
[0292] Table 4:
[0293]
[0294] As shown in Table 4, the most noteworthy metric is AP@0.95, which requires accurate localization of forged segments even with an Intersection over Union (IoU) threshold of 0.95. This is the most stringent standard for measuring temporal localization accuracy. AGMF-Net achieves 62.36% on this metric, a 15.68 percentage point improvement over MMMS-BA (46.68%). BA-TFD+ is almost ineffective on this metric (0.31%). The differences among the three reveal the fundamental differences in localization granularity: BA-TFD+'s boundary attention can only locate segments; while MMMS-BA can locate general areas, it is limited by fixed attention weights and struggles to accurately locate boundaries at the frame level; AGMF-Net's graph-text branch provides semantic anchors for temporal localization—when an audio segment shifts identity in a frame, the static semantics of the graph-text branch can help determine whether the shift is consistent with the overall content, thus advancing the localization accuracy from the second level to the frame level.
[0295] 2.4 Ablation Experiment:
[0296] To further understand the contributions of each component of AGMF-Net, the following series of ablation experiments were conducted.
[0297] 2.4.1 Modal complementarity verification:
[0298] To quantify the marginal contribution of each modality to the overall performance, two variants were constructed: M1 (removing audio and video modalities and retaining only text and image) and M2 (removing text and image modalities and retaining only audio and video). Tables 5 (text and image modal ablation experiments (%)) and 6 (audio and video modal ablation experiments (%)) report the results, with Table 5 detailed below:
[0299] Table 5:
[0300]
[0301] In Table 5, the decrease in Precision was most dramatic after removing the audio / video modality (from 81.85% to 72.29%, a decrease of 9.56%). Precision measures "how many regions predicted as fake are actually fake," and its sharp decline means that without audio / video information, the model misclassifies a large number of real regions as fake. Taking attribute-level forgery as an example, the pure text-image model can only observe "text sentiment becomes positive" and "image expression becomes happy," which are semantically consistent, so the model tends to label them as real. When inconsistencies occur between text and images, the model can only classify them as fake, but cannot determine whether such inconsistencies are genuine forgeries or normal expression differences. The value of the audio / video modality lies in providing multiple verification anchors for emotion and identity—emotion in the voice, rhythm in the lip movements—this information forms redundant verification with the text and images, greatly reducing false positives. In other words, the sharp decline in Precision indicates that the role of the audio / video modality is "verification," not "supplementation."
[0302] Table 6 details are as follows:
[0303] Table 6:
[0304]
[0305] In Table 6, AR@5 showed the most significant decrease (from 84.85% to 76.25%, a decrease of 8.60%). AR@5 measures "how many real and fake segments are covered in the top 5 predicted segments," reflecting the model's recall ability at a coarse-grained level. Its significant decrease means that without text and image information, the model even struggles to make a coarse-grained judgment on "which time periods might be faked." Pure audio-video models can only rely on the temporal consistency between lip movements and voiceprints for judgment, but when the lip-sync accuracy of the forger is high enough (such as LatentSync), this temporal clue will be weakened. The static semantics provided by the text and image modality—whether the facial identity is consistent with the text description, and whether the scene semantics match the news topic—provide additional contextual constraints for temporal localization, enabling the model to still lock suspicious segments through semantic contradictions when temporal clues are insufficient.
[0306] As can be seen from Tables 5 and 6, the value of the four modalities lies in the fact that different modalities provide different types of verification signals: images and text provide static semantic anchors, while videos and audios provide dynamic temporal cues; when two types of evidence point to the same conclusion, the reliability of the decision is significantly improved; when one type of evidence is weakened, the other type can still provide supplementary constraints; this explains why the four modal inputs benefit the most on "precision" metrics (Precision, AR@5).
[0307] 2.4.2 Ambiguity Gating Mechanism:
[0308] To verify the effectiveness of the ambiguity gating mechanism proposed in this invention, Table 7 (Cross-modal ambiguity gating ablation experiment (%)) shows the ablation results of three gating variants, as detailed in Table 7 below:
[0309] Table 7:
[0310]
[0311] As shown in Table 7, removing any gating branch leads to a decrease in AGMF-Net performance, but the decrease patterns differ, revealing the differentiated mechanisms of ambiguous gating; removing image-side gating (w / o) After removing the text-side gating, the image localization capability decreased most significantly: IoU@50 dropped from 74.55% to 71.48% (↓3.07%), and IoU@75 dropped from 65.23% to 62.89% (↓2.34%). This indicates that the core function of image-side gating is to suppress noise in textual conditional information, helping the model to filter out visual responses that are truly related to the forged region; removing text-side gating (w / o) After removing the audio / video gating (w / o), text token localization and image-text semantic discrimination were most affected: Recall decreased from 78.38% to 74.40% (↓3.98%), and OF1 decreased from 86.54% to 84.04% (↓2.50%). This indicates that text-side gating helps the model distinguish between "semantically natural text changes" and "semantic anomalies truly caused by forgery"; removing audio / video gating (w / o) further impacted the model's performance. After that, the audio and video positioning performance deteriorated: AP@0.95 decreased from 62.36% to 59.34% (↓3.02%), and AP@0.75 decreased from 72.38% to 69.54% (↓2.84%). This indicates that the audio and video gating is responsible for capturing the fine-grained temporal inconsistencies between lip movements, voiceprints, and emotions.
[0312] Table 7 shows that ambiguity gating is not a simple feature reweighting, but a core component of AGMF-Net in establishing a selective attention mechanism across different modalities. The complete model can dynamically adjust the weights of single-modal and cross-modal evidence according to sample characteristics, which is the key ability of AGMF-Net to distinguish between "true consistency" and "fake consistency".
[0313] 2.4.3 Components of the loss function:
[0314] Table 8 (Ablation Experiments of Each Component of the Training Objective (%)) analyzes the marginal contribution of each loss function to the performance of AGMF-Net. The experiment design is as follows: after removing a certain loss term from the complete model, the model is retrained, and the performance change is observed. The greater the change, the greater the contribution of that loss term. Table 8 details are as follows:
[0315] Table 8:
[0316]
[0317] The following conclusions can be drawn from Table 8:
[0318] 1. Semantic alignment loss establishes cross-modal mapping, affecting localization accuracy: Remove After (image and text alignment), IoU@75 decreased from 65.23% to 62.95% (↓2.28%); removing After (audio / video alignment), IoU@50 decreased from 74.55% to 72.44% (↓2.11%); removing After cross-branch comparison, the image-text AUC decreased from 95.25% to 93.38% (↓1.87%); the three losses essentially serve the same purpose: establishing fine-grained correspondences between different modality pairs; without alignment loss, the model can still determine authenticity from global features, but it cannot answer mapping questions requiring precise localization, such as "which patch in the image corresponds to which token in the text"; it is particularly noteworthy that... The impact of removing the image on audio-visual localization (AP@0.5 decreased from 80.29% to 75.55%, a decrease of 4.74%) was much greater than its impact on image-text localization. This indicates that the core beneficiary of cross-branch contrastive loss is the audio-visual branch—the static semantics of the image-text branch are passed to the audio-visual branch through contrastive learning, giving it additional spatial constraints in temporal localization.
[0319] Second: Location loss is irreplaceable: removal Afterwards, IoU@75 decreased from 65.23% to 0.00%; removed The F1 score for the following text decreased from 80.08% to 17.62%; (Removal) Afterwards, AP@0.95 decreased from 62.36% to 0.33%; the removal of these three loss terms did not lead to a performance degradation, but rather a collapse in the corresponding modality localization capability; this result reveals an important property: although image-text binary classification loss and forgery type classification loss can indirectly affect the feature representation of the localization module through gradient backpropagation, this indirect influence cannot replace the direct supervision of localization loss; the localization task requires the model to generate discriminative signals at each local unit (patch, token, time segment), which is a strong spatial supervision that can only be provided by direct pointwise loss;
[0320] 3. Forgery type classification loss plays a guiding role in localization: removal Subsequently, mAP plummeted from 94.73% to 18.62%, and IoU@50 dropped from 74.55% to 72.10% (↓2.45%). The significant decrease in mAP was expected, as the model lost direct supervisory signals to distinguish forgery types. However, the decrease in localization metrics (IoU@50 ↓2.45%) reveals a deeper mechanism: clarifying "how forgery occurs" (identity substitution vs. sentiment reversal) helps to accurately pinpoint "where forgery occurs." Entity-level forgery mainly focuses on facial regions and entity nouns, while attribute-level forgery focuses on expression regions and sentiment words. If the model lacks prior knowledge of forgery types, it needs to blindly search all potential regions, leading to a decrease in localization accuracy.
[0321] Furthermore, experiments retaining only the binary classification task further validated the necessity of multi-task learning: when only the binary classification task is retained... At that time, all localization metrics almost failed (IoU@75 dropped to 0.41%, AP@0.95 dropped to 0.31%), and type recognition was also completely lost (mAP dropped to 14.39%). The binary classification task only requires a global discriminant signal, causing the model's feature representation to collapse to the global level. However, the localization task requires the generation of discriminant signals on each local unit (patch, token). The two have an essential conflict in their requirements for feature representation. Only through multi-task joint optimization can the model learn a shared feature space that simultaneously supports global judgment and local localization.
[0322] 2.4.4 Noise robustness:
[0323] To evaluate the model's tolerance to common compression and noise interference, four levels of JPEG compression (quality factors of 100, 90, 60, and 30) were applied to images / videos during the testing phase, and four levels of additive Gaussian noise (variances of 0, 0.1, 0.2, and 0.3) were applied to audio. The corresponding results are shown in Table 9 (Image and text forgery detection performance (%) under different noise levels) and Table 10 (Audio and video forgery detection performance (%) under different noise levels), with Table 9 detailed below:
[0324] Table 9:
[0325]
[0326] Table 10 details are as follows:
[0327] Table 10:
[0328]
[0329] As can be seen from Tables 9 and 10:
[0330] In image-text tasks, with enhanced JPEG compression, AGMF-Net's IoU@50 decreased from 74.55% to 71.00%, a drop of 3.55 percentage points; HAMMER's decreased from 68.01% to 62.71%, a drop of 5.30 percentage points. AGMF-Net's localization accuracy remained stable under high compression, while HAMMER's decrease was more significant, indicating that semantic conflict-based localization is more resistant to compression than methods relying on texture details. In classification, CSCL's AUC decrease (2.24 percentage points) was slightly smaller than AGMF-Net's (3.08 percentage points), which may be related to the additional constraints imposed on semantic representation by its consistency learning.
[0331] In audio and video tasks, AGMF-Net's AP@0.75 decreased from 72.38% to 70.16%, a drop of 2.22 percentage points; MMMS-BA decreased from 63.23% to 58.90%, a drop of 4.33 percentage points; and BA-TFD+ decreased from 36.46% to 27.95%, a drop of 8.51 percentage points. AGMF-Net's stable performance is related to its segment-level gating mechanism—when audio is contaminated by noise, gating can reduce the fusion weight of that segment, preventing noise from spreading to the entire sequence; MMMS-BA and BA-TFD+ use fixed cross-modal attention, so once noise enters, it affects all segments, resulting in a significantly larger decrease in performance under high noise conditions.
[0332] In summary, AGMF-Net's resistance to compression and noise mainly comes from three aspects: four-modal semantic redundancy (Tables 5-6), gating to suppress noise propagation (Table 7), and semantic-level rather than signal-level discrimination criteria. Figure 5 These features are particularly useful in real-world scenarios where content has undergone multiple compression and transcoding processes.
[0333] 2.5 Visualization:
[0334] 2.5.1 Detection and Location of Image and Text Forgery:
[0335] Figure 3 This section presents the detection and localization results of AGMF-Net on collaborative forgery, showcasing its performance on typical samples. Figure 3 (a) The image shows the original image-text pair before entity-level forgery generation, with the title "Xiao Li reminds everyone to treat skincare issues scientifically"; (b) The image shows the image-text pair after entity-level forgery generation, where the forged text replaces "Xiao Li" with "Xiao Fang" (marked in red), and a face-swapping operation occurs in the image (red bounding box). AGMF-Net correctly identifies it as FS+TS and locates the forged token (highlighted in blue) and the forged region (blue bounding box); (c) The image shows the original image-text pair before attribute-level forgery generation, with the title "Xiao Li introduces the scenery of his hometown"; (d) The image shows the image-text pair after attribute-level forgery generation, where the forged text has a reversed sentiment (marked in red), and the image expression changes from neutral to enthusiastic (red bounding box). AGMF-Net correctly identifies it as FA+TA and locates the forged token (highlighted in blue) and the forged region (blue bounding box).
[0336] from Figure 3 As can be seen, AGMF-Net can simultaneously complete two tasks: forgery type identification and forgery region localization. In entity-level forgery (FS+TS) samples, the model correctly identifies the forgery type and accurately locates the replaced face region and the altered name. In attribute-level forgery (FA+TA) samples, the model correctly identifies the forgery type and locates the edited expression region and the word with reversed sentiment. The model's localization boundary of the forgery region highly coincides with the real altered region, verifying its fine-grained localization capability.
[0337] 2.5.2 Attention Distribution in Image and Text Forgery:
[0338] Figure 4 A comparison of attention maps on entity-level and attribute-level forged samples is presented, demonstrating the attention distribution of AGMF-Net on different forged samples; among them, Figure 4(a) The image shows the original image-text pair before entity-level forgery generation, with the title "Old Zhang talks about the most difficult kung fu moves, including stick-sticking, cross-legged rotation, etc."; (b) The image shows the image-text pair after entity-level forgery generation, with "Old Zhang" replaced by "Old Chen" (marked in red); (c) The image shows the attention distribution of AGMF-Net on entity-level forgery samples, with red and yellow representing high attention weight and medium attention weight, respectively; (d) The image shows the original image-text pair before attribute-level forgery generation, with the title "Xiao Liu guesses that oil prices will be adjusted"; (e) The image shows the image-text pair after attribute-level forgery generation, with the text sentiment changing from neutral to optimistic (marked in red), and the facial expression changing from neutral to happy; (f) The image shows the attention distribution of AGMF-Net on attribute-level forgery samples, with the highest attention concentrated in the lip area after the expression modification.
[0339] from Figure 4 It can be seen that the attention distribution of AGMF-Net exhibits a clear hierarchy. In entity-level forgery samples, the model assigns a moderate level of attention to the replaced face region, while assigning the highest level of attention to the background region. This is because although face replacement changes the identity, it retains the posture and expression, and remains semantically consistent with the replaced name in the text. However, there is an irreconcilable contradiction between the background region and the text description. In attribute-level forgery samples, the model focuses the highest attention on the lip region after the expression modification, which directly conflicts with the emotional expression in the audio.
[0340] This indicates that AGMF-Net's reasoning logic is: first, to use cross-modal consistency for preliminary screening, and then to focus on regions where semantic conflicts actually exist; this hierarchical reasoning mechanism is different from the traditional method's reliance on low-level artifacts.
[0341] 2.5.3 Feature Space for Image and Text Forgery:
[0342] Figure 5 The t-SNE visualization of the image representations extracted at different processing stages of AGMF-Net shows the changes in the feature representations of AGMF-Net at different processing stages. Among them, (a) shows the initial image embedding from the ViT encoder. (a) The images show severe overlap between different categories of samples; (b) The image embedding is the result of image-text pair fusion. (c) The samples begin to separate but still overlap; the figure shows the final image-text joint representation after hierarchical fusion with audio-video context. The three types of samples form three clearly separated clusters: blue nodes represent real / unaltered images, yellow nodes represent entity-level collaborative forgery (FS) images, and red nodes represent attribute-level collaborative forgery (FA) images. The darker the color, the higher the node density and the tighter the cluster.
[0343] from Figure 5 It can be seen that the feature representation of AGMF-Net exhibits a clear evolution trend with each processing stage; the initial ViT features ( Figure 5 In (a)), real samples, entity-level forgeries, and attribute-level forgeries are severely mixed, indicating that visual features alone cannot distinguish semantically coherent forgeries; after image-text fusion ( Figure 5 (b) The samples begin to separate but still overlap. Finally, after multimodal hierarchical fusion ( Figure 5 (c) The three types of samples form three clearly separated clusters;
[0344] at the same time, Figure 5 In (c), all entity-level forgery samples (regardless of whether they are generated by Inswapper or the named entity recognition model) cluster into one class, and all attribute-level forgery samples (regardless of whether they are generated by LivePortrait or GPT) cluster into another class. This indicates that AGMF-Net learns the high-level semantic differences of forgery types (identity substitution vs. sentiment reversal), rather than the low-level traces of a specific generator. Furthermore, entity-level forgery and attribute-level forgery are far apart in the semantic space, indicating that their cross-modal semantic deviation patterns are different—the former is a conflict at the identity level, and the latter is a contradiction at the sentiment level.
[0345] 2.5.4 Audio and Video Spoofing Location
[0346] To evaluate the temporal localization performance of AGMF-Net in audio and video forgery, this experiment selected a representative forgery sample from the ITVA test set and visualized its segment-level prediction and boundary regression results. Specifically, Figure 6 and Figure 7 For the same test sample, the fragment-level forgery probability distribution and the comparison between the predicted fragment and the real fragment are shown respectively;
[0347] Figure 6 Visualize the location of forged audio and video segments in a single sample, specifically the segment-level forgery probability distribution. The horizontal axis represents time, the vertical axis represents the predicted forgery probability of each segment, the steel blue bars represent the predicted forgery probability within the real forgery interval, the light coral bars represent the non-forgery area, the light blue shaded area represents the real forgery interval, and the red dashed line represents the decision threshold θ=0.5.
[0348] Figure 7 To compare the temporal boundaries of the predicted fake fragments with the real fake fragments, the red bars represent the real (GT) fragments, and the steel blue bars represent the predicted (Pred) fragments obtained after boundary regression and nonmaximum suppression (NMS).
[0349] like Figure 6As shown, AGMF-Net produced a stable high-confidence response in the real-forged interval (0-5 seconds), with all segment-level probabilities exceeding the threshold θ=0.5 and a mean of 0.754. In contrast, all probabilities in the non-forged interval (5-10 seconds) were below 0.05, with a mean of 0.030. At the boundary between 4.5-5.0 seconds and 5.0-5.5 seconds, the probability dropped sharply from 0.733 to 0.033, further demonstrating that the model can clearly distinguish between forged and real segments.
[0350] like Figure 7 As shown, the predicted interval [0.02 seconds, 4.93 seconds] closely matches the actual interval [0.00 seconds, 4.97 seconds], with initial boundary error and ending boundary error of only 0.02 seconds and 0.04 seconds, respectively; the time intersection-union ratio (tIoU) reaches 0.98, indicating that the two are almost completely overlapping.
[0351] The accuracy of the boundary regression was further evaluated on all test samples. Figure 8 The boundary error distribution for audio / video forgery segments is shown, displaying the boundary error distribution of predicted segments that successfully match the real segments (tIoU ≥ 0.5). The horizontal axis represents the boundary error type: the initial boundary error Δs and the final boundary error Δe; the vertical axis represents the error value in seconds; the light blue box represents the initial error distribution, and the light pink box represents the final error distribution; the box boundaries represent the first and third quartiles, the dark blue center line represents the median, the whisker extends to 1.5 times the interquartile range, and the pink scatter points represent outliers.
[0352] like Figure 8 As shown, the medians of the initial and final boundary errors were 0.0187 seconds and 0.0177 seconds, respectively, both below 0.02 seconds. The first and third quartiles of the initial error were 0.0068 seconds and 0.0525 seconds, respectively, and the first and third quartiles of the final error were 0.0045 seconds and 0.0384 seconds, respectively, indicating that the boundary errors of more than 75% of successful matching segments were controlled within approximately 0.05 seconds. Compared to the initial error (0.0457 seconds), the interquartile range of the final error (0.0339 seconds) was more concentrated. However, it contains more outliers, with a maximum error of 0.3150 seconds. This indicates that although the model can achieve stable boundary prediction on most samples, in a few complex cases, the end position of the forged segment may be more easily affected by factors such as smooth audio-visual transitions or the gradual fading of forgery traces. It can be seen that AGMF-Net can not only identify the approximate time range of forgery through segment-level classification, but also achieve fine-grained temporal boundary regression, with a median error of about 0.018 seconds, and more than 75% of the errors are within 0.05 seconds.
[0353] This invention proposes for the first time a novel problem: four-modal collaborative forgery detection and localization involving text, images, audio, and video—that is, all four modalities are simultaneously tampered with, and the forgery types of text, images, and audio / video are consistent. An AGMF-Net model based on ambiguity gating and modality hierarchical fusion is designed. For this task, the first four-modal collaborative forgery dataset, ITVA, is constructed. Experimental results show that AGMF-Net significantly outperforms existing methods in detection and localization. Ablation experiments verify the necessity of four-modal input, the effectiveness of cross-branch hierarchical fusion, and the contribution of the ambiguity gating mechanism. The current model assumes that all four modalities are complete and usable; future research will explore robust detection schemes under modality loss conditions. Furthermore, the data and model in this paper are based on Chinese news scenarios; their generalization ability in cross-language and cross-cultural scenarios needs further verification.
[0354] In summary, this system simultaneously receives and encodes four modal inputs: image, text, video, and audio. It performs image-text alignment and ambiguity quantization, and audio-video alignment and gating fusion, respectively. Based on cross-branch contrastive learning, it aligns the image-text branch and the audio-video branch to a shared semantic space, thereby integrating four modal information and eliminating the blind spot in bimodal detection. Furthermore, it employs reparameterization techniques to model single-modal features as Gaussian distributions and calculates cross-modal ambiguity scores using KL divergence, transforming ambiguity from noise into discriminative cues. Finally, through cross-attention aggregation, token-level gating fusion, fragment-level classification, and boundary regression, it simultaneously outputs image tampering bounding boxes, text forgery words, and audio-video forgery time segments, thus achieving three-granularity fine-grained localization. Specifically:
[0355] This invention can simultaneously receive four modal inputs: image, text, video, and audio. It can detect collaborative forgery samples in which all four modalities are simultaneously tampered with and the forgery types of the image / text branch and the audio / video branch are consistent. It solves the problem that the modal coverage of existing methods is limited to two modalities and cannot deal with four-modal collaborative forgery. It breaks through the blind spot of two-modal detection and realizes four-modal collaborative forgery detection and localization.
[0356] By using variational modeling, the single-modal features are parameterized into a Gaussian distribution, and the information difference between modalities is quantified by KL divergence to obtain cross-modal ambiguity scores. These scores are then used as control signals for dynamic fusion. By utilizing the distribution differences of ambiguity scores on real and fake samples, the model can distinguish between "real consistency" and "fake consistency," transforming cross-modal ambiguity into discriminative clues, suppressing false associations, effectively inhibiting false associations created by forgers through forced alignment, and improving detection accuracy.
[0357] By adopting a hierarchical fusion strategy: firstly, cross-modal alignment and gating fusion of the text-image pair and the audio-video pair are completed separately, and then the two branches are aligned to the shared semantic space through cross-branch contrastive learning. This not only preserves the fine-grained information within each modality, but also establishes a unified representation of the four modalities, solving the problem of loss of modality-specific structural information caused by simple splicing or one-time fusion.
[0358] By simultaneously outputting localization results at three granularities—image tampering bounding boxes, text forgery words, and audio / video forgery time segments—it not only determines authenticity but also accurately answers "where was tampered with," significantly improving the interpretability and evidentiary value of the detection results. This achieves three-granularity tampering localization, meeting the needs of practical evidence collection.
[0359] By using semantic-level cross-modal conflict discrimination instead of relying on low-level textures or signal artifacts, and by suppressing noise propagation between modalities through sample-level and local-level gating mechanisms, it has better resistance to common signal degradation such as JPEG compression and additive noise than existing methods. It is more suitable for real-world scenarios where content has undergone multiple compression and transcoding, and has stronger robustness to compression and noise interference.
[0360] By jointly optimizing eight loss functions—text-image alignment, audio-video alignment, cross-branch comparison, image localization, text localization, audio-video localization, text-image binary classification, and forgery type multi-classification—the model learns a shared feature space that simultaneously supports global judgment and local localization. This avoids the problem of feature representation collapsing to the global level due to a single binary classification task, thereby improving the overall performance of end-to-end multi-task joint optimization.
[0361] By constructing a benchmark dataset containing four modal samples of images, text, audio, and video, covering two types of collaborative forgery at the entity and attribute levels, and providing complete annotations at three granularities—image bounding boxes, text forgery lexical units, and audio / video forgery time segments—a reliable data foundation is provided for model training and performance evaluation.
[0362] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used above are only some embodiments described in this invention. Obviously, those skilled in the art can obtain other drawings based on these drawings.
[0363] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.
Claims
1. A method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion, characterized in that: Includes the following steps: Step 1: Given an image-text pair and an audio-video pair, perform multimodal coding to obtain the embedding representation of each modality; Step 2: Based on image embedding sequence Text embedding sequence The spatiotemporal characteristics of video Spatiotemporal characteristics of audio Multimodal fusion is performed to obtain cross-modal ambiguity scores. ; Step 3: Based on cross-modal ambiguity scores Image-text gating fusion is performed to obtain the fused representation on the image side. ; Step 4: Spatiotemporal features based on video Spatiotemporal characteristics of audio and cross-modal ambiguity scores The audio and video fusion representation is obtained. ; Step 5: Based on the global semantic vector of the image after image-text fusion Global semantic vector after audio and video fusion Cross-branch contrastive learning is performed to obtain the cross-branch contrastive loss. ; Step 6: Image-based fusion representation and audio-visual fusion representation To perform image forgery localization, the image forgery localization loss is obtained. ; Step 7: Based on text embedding sequence Image embedding sequence and audio-visual fusion representation To locate text forgery; Step 8: Audio-video fusion representation Text-side multimodal representation To perform audio and video spoofing and location tracking; Step 9: Based on all losses, calculate the overall loss of the model and perform joint optimization training.
2. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 1, given an image-text pair and an audio-video pair, multimodal coding is performed to obtain the embedding representation of each modality. The specific steps are as follows: 1.1 Given an image-text pair Image encoder The image is segmented into multiple image patches, and these patches are encoded into an image embedding sequence using a self-attention layer and a feedforward network. The specific formula is as follows: ; in, For image, For text; For global semantic embedding of images, and , For the set of real numbers, For feature dimensions; For the corresponding Embedding of image patches, and , The total number of image blocks; For the first Embedding vectors of image patches, ; The joint data distribution of image-text pairs, i.e. ; 1.
2. Text encoder adopted Encode the text into a text embedding sequence. The specific formula is as follows: ; in, For text Global semantic embedding, and ; For the corresponding The embedding of text tags, and ; For the first The embedding vectors of text words, and , , The total number of words in the text; 1.3 Given an audio / video pair Divide the video into Each segment is selected from [number] segments. Using keyframes as video-level input, and utilizing short-time Fourier transform to... The audio segments corresponding to each keyframe are processed to obtain the Mel spectrum as the audio level input; Where V represents video and A represents audio; 1.
4. The Transformer module is used to perform initial encoding on the audio-level input to obtain the initial video embedding sequence. and audio initial embedding sequence The specific formula is as follows: ; in, For global semantic embedding of video, For audio global semantic embedding, For the first The first video clip One keyframe For the first Audio features corresponding to each video segment; In this process, average pooling is performed on the initial embedded sequence of the video to ensure temporal coordination between audio and video modalities. The specific formula is as follows: ; in, For the first The first video clip One keyframe; 1.5 Using a time encoder and space encoder Processing the initial embedded sequence of video and audio initial embedding sequence To obtain the spatiotemporal features of the video and the spatiotemporal characteristics of audio The specific steps are as follows: 1.5.1 The time encoder embeds the sequence from the initial video sequence. and audio initial embedding sequence Extract time features from each part, and assume that the time encoder has a total of Layers, for modes , No. The update rule for the video or audio temporal features of a layer is as follows: ; in, ; For the sake of the bulls' self-attention, For layer normalization, For the first Layer-learnable temporal location encoding, To add element by element, ; For time encoder No. Video or audio temporal features extracted from the layer; For time encoder No. Video or audio temporal features extracted from the layer; 1.5.2 Initialization , The time encoder output is recorded as and ; in, For time encoder Temporal features of the video extracted from layer L; Time encoder The temporal features of the audio extracted from layer L; 1.5.3 The spatial encoder extracts spatial features from the temporal features, and... Reconstructed into feature maps and divided into From image patches, an initial spatial embedding sequence is obtained. : ; in, For time encoder The time features extracted from the Lth layer, when hour, = ; for corresponding Image patch embedding, For learnable location encoding; This is an element-wise addition; 1.5.
4. Assume the spatial encoder has a total of Layers, for single-modal features , No. The update rule for the spatial features of a layer is as follows: ; in, , It is a feedforward layer; For the first The output of the layer space encoder belongs to the modal. Spatial characteristics; For the first The output of the layer space encoder belongs to the modal. Spatial characteristics; 1.5.5 Calculate the spatiotemporal features of the video and audio, using the following formulas: = ; = ; This represents the spatiotemporal characteristics of the video output by the Kth layer spatial encoder. This represents the spatiotemporal characteristics of the audio output by the Kth layer spatial encoder; For the first The output of the layer space encoder belongs to the modal Spatial characteristics, when hour, = ;when hour, = .
3. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 2, based on the image embedding sequence Text embedding sequence The spatiotemporal characteristics of video Spatiotemporal characteristics of audio Multimodal fusion is performed to obtain cross-modal ambiguity scores. The specific steps are as follows: 2.1 Construct a cross-modal image-text alignment module to embed images into the sequence. and text embedding sequence Mapping to the shared semantic space yields the image modality in the image-text shared semantic space. Text modalities in the semantic space shared with images and text The specific formula is as follows: ; ; The image-text cross-modal alignment module includes an image mapping network. and text mapping network ;and and It consists of a modality-specific MLP and a modality-sharing layer. Used to preserve the unique information of each modality Used to project both into a shared semantic space; in, , ; It is a set of image patch representation vectors in the image-text shared semantic space; This is the global semantic vector in image modality representation; This is the global semantic vector in the text modal representation; For text modal representation in the semantic space of image-text sharing; A set of text lexical representation vectors in a shared semantic space for images and text; For the first in the semantic space of image and text sharing Embedding vectors of text words; This is the embedding vector of the first text word in the image-text shared semantic space; 2.2 The image-text cross-modal alignment module is trained through an auxiliary binary classification task, using cosine embedding loss. The alignment loss is constrained, and the specific formula is as follows: ; in, To define paired labels, if the image and text If they come from the same real news, then If they come from different real news sources, then ; , Indicates a positive sample pair. Indicates negative sample pairs; For boundary thresholds; 2.
3. The single-modal representation E is modeled as a Gaussian distribution using the reparameterization technique, and the differences between modes are calculated using KL divergence. Specifically: The variational posterior of single-modal observations is expressed as: ; in, Normal distribution, representing the variable Follow the mean variance is The normal distribution; 2.4 For each sample Image variational posterior Text variational posterior Defined as: ; ; Let be the global semantic vector in the image modal representation of the i-th sample in the batch; Let be the global semantic vector in the text modal representation of the i-th sample in the batch; 2.
5. Approximately obtained by sampling the marginal distribution across the entire dataset. and The specific formula is as follows: ; in, The edge distribution of image modalities across the entire dataset. The marginal distribution of text modalities across the entire dataset; This refers to the number of samples in a small batch, i.e., the batch size. 2.5 Calculate the sample using normalized KL divergence Cross-modal ambiguity score The specific formula is as follows: ; in, Let KL divergence be denoted as KL divergence.
4. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 3, based on cross-modal ambiguity scores Image-text gating fusion is performed to obtain the fused representation on the image side. The specific steps are as follows: 3.
1. Image Modality Representation Based on Alignment and text modal representation Calculate cross-modal attention weights, and compute global textual information for both image global semantics and text global semantics. The specific formula is as follows: ; in, The global semantic vector in its image modality representation is used when calculating attention. , When calculating attention, the global semantic vector in its text modality representation is used. ; The normalization exponential function is used to map the input vector to a probability distribution. After mapping, the values of each element are in the range of (0,1), and the sum of all elements is 1. Transpose of a vector; 3.2, based on and Calculate local text information of local semantics of images and local semantics of text. The specific formula is as follows: ; 3.
3. Transfer global text information Aggregate to superior This yields a text-conditionalized global semantic vector of the image. The specific formula is as follows: ; 3.
4. Transfer local text information Aggregation yields a set of text-conditionalized image patch representation vectors. The specific formula is as follows: ; in, Let be the text-conditionalized representation vector of the j-th image patch; 3.5, based on , ,as well as The original features and cross-modal features are weighted and combined to obtain the global semantic vector of the image after image-text fusion. The specific formula is as follows: ; 3.6, Based on , ,as well as After introducing a patch-level gating mechanism to calculate local gating coefficients, weighted fusion is performed to obtain the fused image local block representation vector. The specific formula is as follows: ; in, for The first in A vector, for The first in One vector; The patch-level gating mechanism is implemented using a two-layer MLP, with the hidden layer dimension (i.e., the feature vector dimension) set to... It uses GELU activation, and the output is mapped to a sigmoid function. , No. Local control coefficient of an image patch for: ; in, This represents vector concatenation. and All parameters are learnable, and the cross-modal ambiguity score is... As a scalar, it is broadcast to the network during splicing. Same dimension, For the Sigmoid function; 3.
7. Based on global semantic vectors of images and image local block representation vector The fusion representation of the image side is calculated. The specific formula is as follows: ; in, .
5. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 4, based on the spatiotemporal features of the video Spatiotemporal characteristics of audio and cross-modal ambiguity scores The audio and video fusion representation is obtained. The specific steps are as follows: 4.1 Construct an audio-video cross-modal alignment module to align the spatiotemporal features of the video. and the spatiotemporal characteristics of audio Mapping to a shared semantic space yields video modal representations. and audio modal representation The specific formula is as follows: ; ; in, The total number of time segments. This is a set of temporal segment representation vectors for audio modal representation in the audio-video shared semantic space; This refers to the global semantic vector representing the audio modality in the shared semantic space of audio and video. For audio number A representation vector for each time segment; It is the global semantic vector representing the video modality in the audio-video shared semantic space; This is a set of temporal segment representation vectors for video modal representation in the audio-video shared semantic space; For the video number The representation vectors of each temporal segment; the audio-video cross-modal alignment module includes a video mapping network. and audio mapping network ; 4.
2. Define the pairing tag as Using the alignment loss function The constraints are established for it, and the specific formula is as follows: ; Among them, if the video and audio If they come from the same real sample, then If it is a scrambled mismatch or asynchronous audio / video pair, then ; is the cosine similarity function, with values ranging from [−1, 1]; It is a function for maximizing the value; This represents the boundary threshold in the audio / video alignment loss. 4.3 Video Modal Representation and audio modal representation Cross-modal ambiguity score Adaptively fuse video and audio features to generate an audio-video fusion representation. The specific steps are as follows: 4.3.
1. Based on video modal representation and audio modal representation Calculate the cross-modal attention weights from video to audio; The cross-modal attention weights from video to audio include the attention weights of the video global semantic vector to the audio global semantic vector. And the attention weights of video temporal segments to audio temporal segments. The specific calculation formula is as follows: ; ; in, , , , ; 4.3.2 Based on cross-modal attention weights from video to audio, audio information is aggregated into the video representation to generate audio-conditional video features; Among them, audio-conditionalized video features include audio-conditionalized video global semantic vectors. and audio-conditional video temporal segment representation vector set The specific calculation formula is as follows: ; ; in, This is the audio-conditional video global semantic vector; A set of audio-conditionalized video temporal segment representation vectors; For the first The representation vector of a video segment after audio enhancement. ; For the first A representation vector of an audio segment. ; 4.3.3 Utilizing cross-modal ambiguity scoring The global semantic vector of video modal representation in the audio-video shared semantic space. With audio-conditional video global semantic vector Weighted fusion is performed to obtain the global semantic vector after audio and video fusion. : ; in, ; 4.3.
4. Audio-conditionalized set of video temporal segment representation vectors The set of temporal segment representation vectors for video modal representation in the audio-video shared semantic space. ,as well as A segment-level gating mechanism is introduced for weighted fusion to obtain the audio-video fused video. Each time segment represents a vector. : ; in, Furthermore, the segment-level gating network is implemented using a two-layer MLP, which enables each temporal segment to undergo consistent adaptive fusion based on local audio and video. This segment-level gating network, for the first... Each time segment has a gating coefficient. for: ; in, and All parameters are learnable, and the cross-modal ambiguity score is... As a scalar, it is broadcast to the network during splicing. Same dimension; 4.3.5, Regarding Repeat step 4.3.4 for each time segment to obtain a set of time segment representation vectors. The specific formula is as follows: ; 4.3.
5. Based on the time-series segment representation vector set Global semantic vector after audio and video fusion The audio-video fusion representation is calculated. The specific formula is as follows: 。 6. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 5, based on the global semantic vector of the image after image-text fusion... Global semantic vector after audio and video fusion Cross-branch contrastive learning is performed to obtain the cross-branch contrastive loss. The specific steps are as follows: 5.1 Suppose a batch contains For the nth sample, for the nth Let the global semantic vector of the image after image-text fusion be denoted as _ samples. For the first For each sample, its global semantic vector after audio-video fusion is denoted as . ; in, Batch size; 5.
2. Define the pairing tag as ; in, If the first The first image and text sample and the first If multiple audio and video samples come from the same sample, then , indicates a positive sample pair, otherwise , which represents a negative sample pair; 5.
3. Global Semantic Vector of Image Based on Image-Text Fusion Global semantic vector after audio and video fusion Pairing tags Calculate cross-branch contrastive loss The specific formula is as follows: ; in, The cosine similarity function is used. This is the boundary threshold for cross-branch alignment of text / image and audio / video.
7. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 6, the fusion representation based on the image side is... and audio-visual fusion representation To perform image forgery localization, the image forgery localization loss is obtained. The specific steps are as follows: 6.
1. Enhanced image fusion representation through cross-attention to obtain audio-visual guided enhanced image representation. The specific formula is as follows: ; in, For standard attention operations, As for the key and value, its dimensions are: Through linear transformation and Dimensions Alignment; 6.
2. Enhance the image representation The decomposition is performed, and the specific formula is as follows: ; in, To enhance the global semantic vector in the image representation, and ; For the corresponding A set of augmented representation vectors for local image patches, and ; 6.
3. Employ a learnable aggregate tag. As a query, the result is... Forged related partial information The specific formula is as follows: ; For scaling dot product attention functions; It is a learnable aggregation tag, and ; 6.
4. Forging relevant partial information Input bounding box detector Calculate the loss for image forgery localization The specific formula is as follows: ; in, For generalized intersection and comparison of losses; It is a sigmoid function; A bounding box detector based on a multilayer perceptron; The labels are the true bounding boxes of the forged regions in the image, and Normalized coordinates are used It means that among them The coordinates of the top left corner of the bounding box. The coordinates of the bottom right corner of the bounding box; Indicates the distribution of data Below image-text pairs Take the expected value.
8. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 7, based on the text embedding sequence Image embedding sequence and audio-visual fusion representation The specific steps for text forgery location are as follows: 7.1 Text-based embedding A multimodal aggregator was used. In the The layer performs text self-attention, resulting in the first... The text representation output after the layer performs text self-attention is shown in the following formula: ; in, For multimodal aggregators The Middle The text representation output after the layer performs text self-attention, and , stacked together Layer interaction block, ; For multimodal aggregators The Middle The text representation output after the layer performs text self-attention; It is a text embedding sequence, and ; For the first Multi-head text self-attention function for layers; 7.2, Based on and image embedding sequences Perform cross-attention to enhance the text representation with image information, and obtain the first... Text representation enhanced by the original image information The specific formula is as follows: ; in, For the first The text representation after the original image information is enhanced, and ; To scale the query matrix in the dot product attention function, here ; This is a standard scaled dot product attention function, whose inputs are query, key, and value. 7.3, Based on and and cross-modal ambiguity scores Construct a token-level gating tensor The specific formula is as follows: ; in, For the first The text term-level gated tensor of the layer, and Each element takes a value in the range (0,1); and These are learnable parameters; 7.4 Token-based gated tensors ,right and Perform gating fusion to obtain the first Text representation after layer-gated fusion The specific formula is as follows: ; in, ; 7.5, Based on the first Text representation after layer-gated fusion After passing through a feedforward network and connecting to the residual, the first... The output of self-attention in layer interaction blocks The specific formula is as follows: ; In step 7.1, It is the first The output of the layer's self-attention is used as an intermediate result of the layer's interaction block, and in step 7.4, It is the final output of the same layer after passing through the feedforward network and residual connections, i.e. The value is updated to a richer multimodal representation; in, For the first The feedforward network function of the layer, , This represents the total number of interaction blocks. 7.6, in the The output of self-attention in layer interaction blocks Based on, stack The final text-side multimodal data is obtained after layering. The specific formula is as follows: ; 7.7 Text-based multimodal and audio-visual fusion representation By fusing audio and video information through cross-attention, a text representation enhanced with audio and video information is obtained. The specific formula is as follows: ; Among them, it is a multimodal aggregator and token detector Maintain momentum versions separately and The momentum update rule is ,in These are the current model parameters. Let be the momentum coefficient; the momentum version is denoted as . and , For the final text-side multimodal representation The momentum version, Text representation after audio and video information enhancement The momentum version; n cls For the enhanced global semantic vector, n token The enhanced text word representation vector set; 7.
8. Based on the enhanced global semantic vector n cls Binary classifiers were used respectively. and multi-label classifier The system performs image and text authenticity verification and forgery type identification, and obtains the image and text binary classification loss. and multi-classification loss of counterfeit types The calculation formula is as follows: ; ; in, For cross-entropy loss, For labeling images and text to indicate authenticity. Forged type label; For image-text binary classification loss; For multiple classification losses due to forgery types; 7.
9. Based on the enhanced text lexical representation vector set n token Using a token detector Predict the forgery probability of each token to obtain the text forgery lexical localization loss. The specific formula is as follows: ; in, For the loss function of locating forged lexical units in text, the cross-entropy loss function is used. For each text term output by the token detector, the probability of forgery is given. Forged lexical tags that are genuine, The cross-entropy loss function; 7.10 Token-based detector and momentum version Calculate the momentum version of the KL divergence constraint The specific formula is as follows: ; in, Momentum-KL divergence-constrained loss for locating forged word tokens in text; 7.11 Forgery Probability Based on Textual Truncation KL divergence constraint and momentum version The weighted summation yields the final text forgery location loss. The specific formula is as follows: ; in, To balance the hyperparameters; Total loss for text forgery location.
9. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 8, based on audio and video fusion representation Text-side multimodal representation The specific steps for audio and video spoofing and location tracking are as follows: 8.1 Audio-video fusion representation Text-side multimodal representation By fusing audio / video features with image / text features through cross-attention, an enhanced audio / video representation guided by image / text is obtained. The specific formula is as follows: ; in, Enhanced audio and video representation guided by images and text, and ; Its global semantic vector, ; Let it be the set of vectors representing its time-series segments, and ; 8.2 Enhanced Audio and Video Representation Based on Graphic and Text Guidance The Middle Features of each time segment Input them into the binary classification head respectively and return head , obtained the Each time segment is a forged scalar probability. and the predicted left and right time offsets. : ; ; in, For the Sigmoid function, and are learnable parameters, and , ; To predict the forged fragment relative to the first The time offset to the left of the center of each segment, To predict the forged fragment relative to the first The time offset to the right of the center of each segment; An activation function to ensure that the predicted offset is non-negative; For the regression head H r The learnable weight matrix, and : For the regression head H r The learnable bias vector, and : 8.3 Adjust the left and right time offsets Reconstructing the predicted forged fragment The specific formula is as follows: ; in, For the first The center time of each time segment; 8.4 Based on scalar probability and predicting fake fragments Calculate the loss for audio and video forgery detection and location. The specific formula is as follows: ; in, For fragment classification loss, For regression loss; The number of fake fragments in the current sample. This is the regression loss balance coefficient; 8.5 Based on scalar probability The segment classification loss is calculated using the following formula: ; in, For segment classification loss; if ,but ,otherwise ;like ,but ,otherwise ; It is a positive sample weighting factor, and ; This is the focus parameter; if the segment is fake, i.e. ,but If the segment is true, that is... ,but ; For the first The category weight factor for each time segment; if ,but ;like ,but ; 8.
6. Regarding meeting the requirements The A time series segment, based on predicted fake segments and fake fragment tags The Center-DIoU loss was used to calculate the regression loss of the forged fragments. The specific formula is as follows: ; in, For the first The regression loss for each time segment is the Center-DIoU loss, used to constrain the prediction of fake segments. With fake fragment tags The time boundary error between them; For the first Predicted fake segments for each time segment, and , For the predicted start time, This is the predicted end time; For the first Forged fragment labels for each time segment, and , To fabricate the true start time of the clip, the unit is seconds; To simulate the true end time of the segment, the unit is seconds; ; The intersection-over-union ratio (IoU) between predicted time segments and actual time segments; To predict the distance between the center of the segment and the center of the actual segment; To simultaneously cover the minimum closure interval length of both predicted and ground truth segments, and ; To prevent extremely small constants with a denominator of zero; 8.7 Scalar Probability Based on Each Time Segment By extracting the maximum forgery probability (max) from all segments j Compare it with the judgment threshold After comparison, binarization is performed for discrimination, and aggregation is used to obtain the overall audio and video authenticity prediction. The specific formula is as follows: ; in, The overall fake labels for the audio and video predicted by the model are obtained by aggregating the segment-level prediction probabilities, and If the maximum forgery probability of all time series segments Exceeding the judgment threshold ,but This indicates that the audio / video sample has been determined to be fake; otherwise... , indicating that it is determined to be true; among them, For indicator functions, The threshold is used for judgment; if the probability of forgery of at least one segment exceeds the threshold, the audio / video sample is judged as forged; otherwise, it is judged as real. 8.8 Based on scalar probability Greater than the threshold Candidate forgery fragment set The non-maximum suppression algorithm is used to iteratively retain the fragment with the highest confidence and remove redundant fragments that highly overlap with it, thus obtaining the final set of forged fragments. The specific formula is as follows: ; in, The overlap threshold, This represents the final set of forged fragments retained after NMS. This is the final set of forged fragments retained after nonmaximum suppression; It is a nonmaximum suppression function; The overlap threshold for non-maximum suppression; For the first The predicted start time of the forged fragment, in seconds. For the first The predicted end time of the forged segment, in seconds; For the first The probability of forgery for each time segment.
10. The method for four-modal collaborative forgery detection and localization based on ambiguity gating and modal hierarchical fusion according to claim 1, characterized in that: In step 9, based on all losses, the overall model loss is calculated, and joint optimization training is performed. The specific steps are as follows: 9.1 Calculate the overall loss of the four-modal collaborative forgery detection and localization model based on ambiguity gating and modal hierarchical fusion. The specific formula is as follows: ; in, for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; for The balancing hyperparameters; 9.
2. Overall model loss based on calculations It employs end-to-end supervised learning, updating model parameters through gradient backpropagation.
