Method for constructing a multi-modal media tampering detection and localization model and application thereof

Abstract

This application relates to the field of artificial intelligence technology, and in particular to a method for constructing a multimodal media tampering detection and localization model and its application. On one hand, it utilizes mask-guided probability field constraints for consistency calibration, correcting the semantic saliency bias of the pre-trained model and guiding the model's perceptual focus from salient semantic regions to physically tampered regions, thereby improving the accuracy of capturing fine-grained tampering. On the other hand, it proposes constructing manifold geometric constraints and explicitly widening the distribution gap between normal and abnormal features through a manifold exclusion mechanism, sharpening the localization edges, and thus overcoming the inaccuracy in localization caused by heatmap diffusion. Furthermore, it proposes decoupling optimization within a task-specific subspace to block gradient interference from the localization branch to the global detection backbone, and combines this with a cross-modal consistency-guided feature aggregation mechanism to achieve enhanced synergistic performance between the detection and localization tasks, thereby improving the detection and localization accuracy of the multimodal tampering model. The aim is to address the problem of how to improve the detection and localization accuracy of multimodal tampering models.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for constructing a multimodal media tampering detection and localization model and its application. Background Technology

[0002] With the development of generative artificial intelligence technology, media forgery methods have evolved from single-modal manipulation to multimodal media forgery involving both images and text. Compared to single-modal forgery, this type of forgery is not only more visually realistic but also maintains semantic consistency between images and text, thus significantly improving the concealment and deceptiveness of the forged content. Existing multimodal forgery detection and localization methods typically employ shared encoders and cross-modal interaction structures to jointly model images and text for authenticity discrimination and forgery region localization.

[0003] However, during the conception and implementation of this application, the inventors discovered that: Firstly, the traditional shared encoder architecture causes the gradients of the detection and localization tasks to interfere with each other in the shared feature space. The detection task is biased towards global features, while the localization task focuses on details. The gradient conflict during backpropagation limits the optimization effect of the model and affects the final detection and localization accuracy. Secondly, cross-modal semantic alignment for feature learning results in a strong semantic saliency bias in the pre-trained model, causing the model to focus on salient regions in the image while ignoring tampered regions. Especially when dealing with fine-grained tampering, attention drift makes it difficult to accurately capture the forged regions, affecting detection accuracy. Thirdly, due to the lack of manifold geometric constraints, traditional methods are prone to boundary blurring or heatmap diffusion in fine-grained localization, causing the features of the forged regions to highly overlap with the normal regions, making it impossible for the model to accurately distinguish the tampered regions at the pixel level.

[0004] The technical limitations mentioned above in these three aspects mean that current common multimodal forgery detection and localization methods suffer from insufficient detection accuracy to meet practical application requirements. Therefore, this application proposes a novel method for constructing a multimodal media tampering detection and localization model, aiming to overcome the aforementioned technical deficiencies and improve the detection accuracy of the multimodal forgery model. Summary of the Invention

[0005] The main purpose of this application is to provide a method for constructing a multimodal media tampering detection and localization model, aiming to solve the problem of how to improve the detection accuracy of multimodal forgery models.

[0006] To achieve the above objectives, this application provides a method for constructing a multimodal media tampering detection and localization model, the method comprising the following steps:

[0007] S10: Obtain test samples containing image modal data and text modal data, and construct a training set containing global true / false labels, pixel-level tampering masks, and text manipulation state annotations;

[0008] S20, the image modal data is divided into multiple non-overlapping or overlapping image patch features by the image encoder, and each image patch feature is mapped to a corresponding feature vector to form a multi-dimensional visual feature sequence. The image encoder adopts a deep neural network model with global or local feature extraction capabilities.

[0009] S30, the text modal data is transformed into a set of ordered tokens by the text encoder, and each ordered token is mapped to a corresponding text tag feature to form a text feature vector;

[0010] S40, the cross-modal interaction layer calculates the mutual attention between the multidimensional visual feature sequence and the text feature vector to generate an original cross-modal attention map, wherein the cross-modal interaction layer calculates the correlation between the image and the text based on the self-attention mechanism to generate a two-dimensional attention matrix representing the cross-modal relationship between the image patch and the text tag;

[0011] S50, calculate the KL divergence between the original cross-modal attention map and the physical manipulation probability field, and constrain the alignment of the original attention map and the manipulation probability field according to the KL divergence, so as to achieve attention focusing on the tampered region. The physical manipulation probability field is obtained by convolving the pixel-level tampering mask with a two-dimensional Gaussian kernel.

[0012] S60, the image patch features are projected onto a low-dimensional manifold space, and the manifold repulsion loss between the tampered patch and the normal prototype features is calculated and minimized to enhance the distinguishability between abnormal features and normal regions in the image;

[0013] S70 introduces a gradient isolation operator in the localization branch to block the propagation of the localization task gradient to the shared backbone network, so that the localization task and the detection task can be decoupled and optimized in the task-specific subspace to avoid gradient interference.

[0014] S80, construct a joint loss function including detection loss, localization loss, calibration loss and rejection loss to train the original model, optimize the model parameters, and obtain a multimodal media tampering detection and localization model.

[0015] Optionally, S50 includes:

[0016] S51, introducing a temperature coefficient The original attention score of the original cross-modal attention map Perform redistribution processing to generate a calibrated predicted probability distribution. :

[0017]

[0018] in:

[0019]

[0020] In the formula, Representing image patch features Text markup features The original attention scores between them; and They are respectively and The linear projection matrix, For feature dimensions;

[0021] S52, calculate the calibrated predicted probability distribution using KL divergence. With physical probability field Loss of perceptual consistency between manipulations :

[0022]

[0023] In the formula, Indicates KL divergence;

[0024] S53, by minimizing the loss of operational perception consistency This is to guide the alignment of the original attention map with the manipulation probability field.

[0025] Optionally, the S60 includes:

[0026] S61, utilizing a nonlinear projection operator Image patch features Mapping to a discriminative low-dimensional manifold space yields manifold features. :

[0027]

[0028] S62, Select the target image patch features with a consistency score higher than a preset score threshold as the normal reference set. By analyzing the manifold features within the normal reference set Perform mean aggregation to construct a prototype of local normal features. :

[0029]

[0030] S63, Calculate the features of the patch to be detected. With local normal feature prototype Cosine similarity between Patch set for known tampered regions Introducing a repulsive boundary Comparison adjustment coefficient Calculate the modeling loss of local semantic miscoordination. :

[0031] .

[0032] S64, Minimize the modeling loss for local semantic discrepancy This is to cause the features of the tampered region to be updated in a direction that deviates from the normal prototype, thereby explicitly widening the distribution gap between normal and abnormal features in the manifold space.

[0033] Optionally, S70 includes:

[0034] S71, in multimodal global representation Above, fine-grained image tampering classification is constructed respectively. Fine-grained text tampering classification and global true / false binary classification tasks Three physically isolated task projection heads, each executing its own prediction output:

[0035]

[0036]

[0037]

[0038] In the formula, , and These represent the fine-grained tampering classification results for image modalities, the fine-grained tampering classification results for text modalities, and the global binary classification results, respectively.

[0039] S72, the corresponding decoupling loss function is calculated using cross-entropy. , and :

[0040]

[0041]

[0042]

[0043] Among them, when image label The time indicates that it has not been tampered with; text label When it indicates that it has not been tampered with; when the global tag , When, it indicates that the image and text are real and fake respectively, and EH() represents the cross-entropy function;

[0044] S73, with decoupling loss function , and With the goal of minimizing, update the parameters of the shared backbone network.

[0045] Optionally, the S70 also includes:

[0046] S74, calculate the co-scoring of image patch features and text tag features to generate gating weights;

[0047] S75, the image patch features and text marker features are weighted and aggregated according to the gating weights to generate enhanced global judgment features; wherein, the gating weights are generated based on cross-modal consistency scores, and the enhanced aggregation of potential tampering regions is achieved by enhancing the weights of low consistency regions and suppressing the weights of high consistency regions.

[0048] Optionally, the joint loss function The expression is:

[0049] in, For fine-grained image tampering classification loss, For fine-grained text tampering classification loss, For global binary classification loss, For local positioning loss, For consistency calibration loss; For manifold repulsion loss; Model the loss for consistency; , , , and These are the preset weighting coefficients for each loss item.

[0050] Furthermore, to achieve the above objectives, this application also provides a multimodal media tampering detection and localization model, obtained using the construction method of the multimodal media tampering detection and localization model as described in any of the preceding claims, wherein the multimodal media tampering detection and localization model includes:

[0051] Feature extraction and interaction module: used to extract image patch features and text tag features, and generate the original cross-modal attention map;

[0052] Consistency calibration module: used to guide attention to the physically tampered region through KL divergence constraints;

[0053] Manifold repulsion module: used to separate tampered features from normal features in a low-dimensional space;

[0054] Task decoupling module: Used to isolate and locate task gradients by stopping gradient operators, enabling parallel optimization of multiple tasks;

[0055] Inference Deployment Module: Used to perform lightweight detection and localization by dynamically removing auxiliary units during the inference phase.

[0056] In addition, to achieve the above objectives, this application also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multimodal media tampering detection and location method as described in any of the preceding claims.

[0057] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the multimodal media tampering detection and location method as described in any of the preceding claims.

[0058] In addition, to achieve the above objectives, this application also provides a multimodal media tampering detection and localization model as described above, which can be applied to the detection of deepfake videos, social media content review, news authenticity verification, and video surveillance systems.

[0059] This application has at least the following beneficial effects:

[0060] 1. Consistency calibration is performed using probability field constraints guided by masks to correct the semantic saliency bias of the pre-trained model, and to guide the model’s perception focus from the salient semantic region to the physically tampered region, thereby improving the accuracy of capturing fine-grained tampering.

[0061] 2. A proposal is made to construct manifold geometric constraints and explicitly widen the distribution gap between normal and abnormal features through the manifold repulsion mechanism, thereby sharpening the positioning edge and overcoming the inaccuracy of positioning caused by heatmap dispersion.

[0062] 3. We propose to improve the detection accuracy of the multimodal forgery model by decoupling optimization in the task-specific subspace to block the gradient interference of the localization branch on the global detection backbone. This will enhance the collaborative performance of the detection and localization tasks. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating the method for constructing a multimodal media tampering detection and localization model according to an embodiment of this application;

[0064] Figure 2This is a schematic diagram of the overall architecture of the construction method of the multimodal media tampering detection and localization model involved in the embodiments of this application.

[0065] Figure 3 This is a schematic diagram illustrating the visual-language dual-stream coding and the topological relationships of each core module involved in the embodiments of this application;

[0066] Figure 4 This is a detailed schematic diagram illustrating the mechanism of the Manipulation Perceptual Consistency Calibration (MICC) module involved in the embodiments of this application, showing the alignment process between attention distribution and physical probability field;

[0067] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0068] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0069] First Embodiment

[0070] Reference Figures 1 to 4 This embodiment provides a method for constructing a multimodal media tampering detection and localization model, which is built on a deep pre-trained visual and linguistic dual-stream backbone network. Specifically, on the visual side, a pre-trained ViT is used as the basic image encoder to transform the input pixel-level image into a patch feature sequence with high-level semantic information; on the text side, a BERT-base model is used as the semantic extractor to perform word embedding and contextual feature capture on the corresponding text description.

[0071] The method includes the following steps:

[0072] S10: Obtain test samples containing image modal data and text modal data, and construct a training set containing global true / false labels, pixel-level tampering masks, and text manipulation state annotations;

[0073] In this embodiment, basic data is first constructed by collecting diverse image modalities I and associated text modalities T, specifically including the following steps:

[0074] S11, sets the global true / false label for binary data. ,in This indicates that the image and text have been altered. To represent reality;

[0075] S12, Construct a pixel-level tampering mask with the same resolution as the input image. Accurately mark the geometric outline of physically altered areas (such as face swapping, region erasure, etc.);

[0076] S13 performs token-level annotation on the text sequence to distinguish between normal tokens and forged tokens that have been semantically manipulated, thereby forming a dual alignment supervision signal that combines spatial and semantic elements.

[0077] S20, the image modal data is divided into multiple non-overlapping or overlapping image patch features by the image encoder, and each image patch feature is mapped to a corresponding feature vector to form a multi-dimensional visual feature sequence. The image encoder adopts a deep neural network model with global or local feature extraction capabilities.

[0078] In this step, the image is patched and globally modeled using a visual encoder.

[0079] In some alternative implementations, this step may be implemented using the following specific steps:

[0080] S21, the input image modality I is divided into non-overlapping patches of fixed size, and mapped to image patch features through a linear projection layer to form a patch embedding vector;

[0081] S22, inject a visual class marker at the beginning of the sequence of the patch embedding vector. Used to aggregate global information and overlay spatial location codes. To preserve the geometric topology of the image;

[0082] S23, input the sequence into the Transformer encoder, use the self-attention mechanism to capture long-range dependencies between patches, and output a multi-dimensional visual feature sequence containing global and local features:

[0083]

[0084] S30, the text modal data is transformed into a set of ordered tokens by the text encoder, and each ordered token is mapped to a corresponding text tag feature to form a text feature vector;

[0085] In this step, the associated text is tokenized and vectorized using a text encoder.

[0086] It is understood that step S30 can be executed synchronously with step S20 or in a certain order, and this embodiment does not impose any restrictions.

[0087] In some alternative implementations, this step may be implemented using the following specific steps:

[0088] S31, split the input text T into a discrete token sequence and insert a text class token at the beginning position. ;

[0089] S32, Integrating Text Position Embedding By embedding paragraphs, a feature representation with structured information is constructed for each token, resulting in ordered tokens;

[0090] S33 models the execution context of ordered tokens using a bidirectional Transformer architecture, captures manipulation cues in the text, and ultimately outputs a text feature vector:

[0091]

[0092] In the formula, Textual tagging features representing ordered token mappings.

[0093] S40, the cross-modal interaction layer calculates the mutual attention between the multidimensional visual feature sequence and the text feature vector to generate an original cross-modal attention map, wherein the cross-modal interaction layer calculates the correlation between the image and the text based on the self-attention mechanism to generate a two-dimensional attention matrix representing the cross-modal relationship between the image patch and the text tag;

[0094] After completing the data preprocessing steps described above, this step fuses the obtained multidimensional visual feature sequence with the text feature vector, and uses a cross-modal interaction layer to establish a low-level association between the visual patch and the text tag.

[0095] In some alternative implementations, this step may be implemented using the following specific steps:

[0096] S41, visual patch features Text markup features respectively through linear projection matrix and Mapping to a unified metric space, calculate the scaled dot product score between the two. :

[0097]

[0098] in, For feature dimensions.

[0099] S42, the score matrix is ​​mapped using the Softmax operator to obtain the original cross-modal attention weights. .

[0100] It should be noted that this weight is used to characterize the degree of attention the pre-trained model pays to the semantic relationship between the image and text in the initial state.

[0101] S50, calculate the KL divergence between the original cross-modal attention map and the physical manipulation probability field, and constrain the alignment of the original attention map and the manipulation probability field according to the KL divergence, so as to achieve attention focusing on the tampered region. The physical manipulation probability field is obtained by convolving the pixel-level tampering mask with a two-dimensional Gaussian kernel.

[0102] In this step, as an innovative step proposed in this embodiment, we propose to physically calibrate the attention distribution by manipulating perceptual consistency calibration to address the semantic bias of the pre-trained backbone network.

[0103] Further and optionally, it specifically includes the following steps:

[0104] S51, introducing a temperature coefficient The original attention score of the original cross-modal attention map Perform redistribution processing to generate a calibrated predicted probability distribution. :

[0105]

[0106] in, That is, the scaled dot product score in step S41:

[0107]

[0108] In the formula, Representing image patch features Text markup features The original attention scores between them; and They are respectively and The linear projection matrix, For feature dimensions;

[0109] It should be noted that by adjusting the temperature coefficient This helps suppress background noise in unaltered regions, allowing the distribution to focus more on high-response areas.

[0110] S52, calculate the calibrated predicted probability distribution using KL divergence. With physical probability field Loss of perceptual consistency between manipulations :

[0111]

[0112] In the formula, Indicates KL divergence;

[0113] S53, by minimizing the loss of operational perception consistency This is to guide the alignment of the original attention map with the manipulation probability field.

[0114] It should be noted that in step S53, by minimizing this loss, the model's attention focus is forcibly shifted from "salient semantic objects" to "physical tampering traces", thereby achieving consistent calibration of the perceptual focus.

[0115] In this embodiment, the physical probability field It was designed to address the gradient discontinuity problem in hard masks. For example, the physical probability field... The mathematical expression is:

[0116]

[0117] In the formula, To tamper with the mask at the pixel level, It is a two-dimensional Gaussian kernel.

[0118] S60, the image patch features are projected onto a low-dimensional manifold space, and the manifold repulsion loss between the tampered patch and the normal prototype features is calculated and minimized to enhance the distinguishability between abnormal features and normal regions in the image;

[0119] In this step, as another innovative step proposed in this embodiment, in order to avoid the lack of manifold geometric constraints causing the model to be unable to accurately distinguish the tampered region at the pixel level, this embodiment proposes a method to enhance the distinguishability between the tampered region and the normal region in a low-dimensional manifold space.

[0120] Further and optionally, this step includes:

[0121] S61, utilizing a nonlinear projection operator Image patch features Mapping to a discriminative low-dimensional manifold space yields manifold features. :

[0122]

[0123] It should be noted that the mapping process in step S61 aims to remove redundant semantic information that is unrelated to tampering, while preserving the underlying physical consistency characteristics.

[0124] S62, Select the target image patch features with a consistency score higher than a preset score threshold as the normal reference set. By analyzing the manifold features within the normal reference set Perform mean aggregation to construct a prototype of local normal features. :

[0125]

[0126] It should be noted that the local normal feature prototype represents the standardized physical properties of the unmanipulated regions in the current image.

[0127] S63, Calculate the features of the patch to be detected. With local normal feature prototype Cosine similarity between Patch set for known tampered regions Introducing a repulsive boundary Comparison adjustment coefficient Calculate the modeling loss of local semantic miscoordination. :

[0128]

[0129] It should be noted that cosine similarity Used to quantify the degree of semantic discrepancy in local regions.

[0130] S64, Minimize the modeling loss for local semantic discrepancy This is to cause the features of the tampered region to be updated in a direction that deviates from the normal prototype, thereby explicitly widening the distribution gap between normal and abnormal features in the manifold space.

[0131] The mechanism proposed in step S60 solves the problem of blurred edges in the detection heatmap caused by feature overlap, and significantly improves the boundary accuracy of the positioning results. The effect of this part will be verified in subsequent embodiments.

[0132] S70 introduces a gradient isolation operator in the localization branch to block the propagation of the localization task gradient to the shared backbone network, so that the localization task and the detection task can be decoupled and optimized in the task-specific subspace to avoid gradient interference.

[0133] In this step, to avoid mutual interference between the gradients of the detection and localization tasks in the shared feature space, a core innovative step in this embodiment is to perform feature decoupling and evidence enhancement within the task-specific space, thereby eliminating gradient interference between different tasks and strengthening cross-modal consistency. Further and optionally, this step includes:

[0134] First, implement a task space decoupling mechanism based on physical isolation:

[0135] S71, in multimodal global representation Above, fine-grained image tampering classification is constructed respectively. Fine-grained text tampering classification and global true / false binary classification tasks Three physically isolated task projection heads, each executing its own prediction output:

[0136]

[0137]

[0138]

[0139] In the formula, , and These represent the fine-grained tampering classification results for image modalities, the fine-grained tampering classification results for text modalities, and the global binary classification results, respectively.

[0140] S72, the corresponding decoupling loss function is calculated using cross-entropy. , and :

[0141]

[0142]

[0143]

[0144] Among them, when image label The time indicates that it has not been tampered with; text label When it indicates that it has not been tampered with; when the global tag , When, it indicates that the image and text are real and fake respectively, and EH() represents the cross-entropy function;

[0145] S73, with decoupling loss function , and With the goal of minimizing, update the parameters of the shared backbone network.

[0146] It should be noted that this mechanism uses physically isolated task heads to block the backhaul of the localization task gradient to the shared backbone, ensuring the stability of the representation.

[0147] On the other hand, also to avoid the gradients of the detection and localization tasks interfering with each other in the shared feature space, this embodiment proposes to use a consistency gating signal to guide adaptive feature fusion, capturing cross-modal inconsistency evidence from multiple perspectives to strengthen the global representation. Specifically, it includes the following steps:

[0148] S74, calculate the co-scoring of image patch features and text tag features to generate gating weights;

[0149] S75, the image patch features and text marker features are weighted and aggregated according to the gating weights to generate enhanced global decision features.

[0150] For example, taking image patch features as an example, the following example demonstrates the process of generating global decision features by weighted aggregation of image patch features based on gating weights:

[0151] First, a set of learnable query vectors is introduced for the image modality.

[0152]

[0153] Calculate the first Each query and the characteristics of each image patch Initial attention weights :

[0154]

[0155] Using consistency scoring Through monotonic functions Construct a reweighting mechanism to obtain the corrected attention weights. :

[0156]

[0157] This operation is performed via The inverse mapping property of the function explicitly suppresses highly consistent background regions, allowing attention to be highly focused on potentially conflicting regions with low consistency, ultimately yielding the first... Aggregate representation of a query:

[0158] Similarly, the attention weights after correction for text tag features are obtained. .

[0159] Finally, these query features focused on evidence of tampering are fused with the original multimodal [CLS] representation to generate an enhanced global representation. : This new characterization Will replace the original Input into the category header.

[0160] Through this consistency-guided aggregation, the model successfully achieves accurate capture and reinforcement of "cross-modal inconsistent evidence" in a decoupled architecture.

[0161] S80, construct a joint loss function including detection loss, localization loss, calibration loss and rejection loss to train the original model, optimize the model parameters, and obtain a multimodal media tampering detection and localization model.

[0162] In this embodiment, to enhance the model's generalization ability to unknown forgery types, the training process is modeled as a constrained multi-objective optimization problem, and the network parameters are updated by constructing a global joint loss function. The global loss function constructed in this embodiment... It consists of a task-driven loss term and a physical constraint regularization term, and its mathematical expression is:

[0163] The task-driven loss includes a fine-grained loss based on the generated image. Fine-grained text loss and global binary classification loss and the localization loss used to optimize localization tasks. The physical constraint regularization term includes a consistency calibration loss for globally distributed calibration. Manifold repulsion loss used for local manifold constraints And consistent modeling loss used to ensure semantic consistency of decoupled features. By presetting weighting coefficients , , , , Balance the contribution of each loss to the model's discriminative power.

[0164] and Apply global distribution calibration and local manifold constraints respectively to prevent attention collapse and sharpen the decision boundary; Ensure the semantic consistency of decoupled features.

[0165] In the technical solution provided in this embodiment, on the one hand, consistency calibration is performed using probability field constraints guided by a mask to correct the semantic saliency bias of the pre-trained model, guiding the model's perception focus from the salient semantic region to the physically tampered region, thereby improving the accuracy of capturing fine-grained tampering; on the other hand, by constructing manifold geometric constraints and introducing a manifold repulsion mechanism, the distribution spacing between normal and abnormal features is explicitly widened, and the localization boundary is sharpened, thereby overcoming the inaccurate localization caused by heatmap diffusion; furthermore, by performing decoupling optimization in the task-specific subspace, the gradient interference of the localization branch on the global detection backbone is blocked, thereby achieving collaborative optimization of the detection task and the localization task, and improving the detection accuracy of the multimodal forgery model.

[0166] Verification of Examples

[0167] Based on the first embodiment, this embodiment compares the proposed method for constructing a multimodal media tampering detection and localization model (Ours) with nine recent representative methods across 12 indicators, as shown in Table 1.

[0168] Table 1. Performance comparison with state-of-the-art methods on the DGM4 dataset

[0169]

[0170] Experimental results show that the present invention performs best in global detection metrics, with the average error rate (EER) reduced to 12.08%. This directly confirms the effectiveness of the gradient isolation mechanism of the MDFM module, that is, by blocking the high-frequency gradient backpropagation of the localization task, the interference of local details on the global discriminative representation is eliminated, ensuring the decision stability of the model at complex manipulation boundaries.

[0171] In terms of high-precision image localization metrics, this invention achieves 86.61%, which is approximately 10.58% higher than the existing best methods (such as HAMMER++). This improvement is attributed to the manifold learning strategy of the LSDM module: by treating tampered regions as manifold outliers and rejecting them, the model is forced to form sharp decision boundaries, thereby achieving accurate capture of minute physical tampering traces.

[0172] Experimental data show that the present invention achieves leading performance in multiple metrics, including multi-label classification (OF1 accuracy of 83.51%) and text localization, with an average performance improvement of 2.68%. These results demonstrate that the framework described in this invention, through regularization constraints (MICC, LSDM) during the training phase, successfully resolves the multi-task conflict and boundary ambiguity issues without increasing computational overhead during the inference phase, thus overcoming the performance bottleneck of refined multimodal forensics.

[0173] Furthermore, as an implementation scheme, the embodiments of this application also involve a multimodal media tampering detection and localization model. This model is obtained using the construction method of the multimodal media tampering detection and localization model described above. The multimodal media tampering detection and localization model includes:

[0174] Feature extraction and interaction module: used to extract image patch features and text tag features, and generate the original cross-modal attention map;

[0175] Consistency calibration module: used to guide attention to the physically tampered region through KL divergence constraints;

[0176] Manifold repulsion module: used to separate tampered features from normal features in a low-dimensional space;

[0177] Task decoupling module: Used to isolate and locate task gradients by stopping gradient operators, enabling parallel optimization of multiple tasks;

[0178] Inference Deployment Module: Used to perform lightweight detection and localization by dynamically removing auxiliary units during the inference phase.

[0179] Furthermore, as one implementation, this application also relates to an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the multimodal media tampering detection and location method as described in any of the preceding claims.

[0180] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0181] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the multimodal media tampering detection and location method described in the above embodiments.

[0182] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0183] Furthermore, as an implementation scheme, the embodiments of this application also involve a multimodal media tampering detection and localization model as described above, which is applied in the detection of deepfake videos, social media content review, news authenticity verification, and video surveillance systems.

[0184] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0185] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0189] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0190] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for constructing a multimodal media tampering detection and localization model, characterized in that, The method includes the following steps: S10: Obtain test samples containing image modal data and text modal data, and construct a training set containing global true / false labels, pixel-level tampering masks, and text manipulation state annotations; S20, the image modal data is divided into multiple non-overlapping or overlapping image patch features by the image encoder, and each image patch feature is mapped to a corresponding feature vector to form a multi-dimensional visual feature sequence. The image encoder adopts a deep neural network model with global or local feature extraction capabilities. S30, the text modal data is transformed into a set of ordered tokens by the text encoder, and each ordered token is mapped to a corresponding text tag feature to form a text feature vector; S40, the cross-modal interaction layer calculates the mutual attention between the multidimensional visual feature sequence and the text feature vector to generate an original cross-modal attention map, wherein the cross-modal interaction layer calculates the correlation between the image and the text based on the self-attention mechanism to generate a two-dimensional attention matrix representing the cross-modal relationship between the image patch and the text tag; S50, calculate the KL divergence between the original cross-modal attention map and the physical manipulation probability field, and constrain the alignment of the original attention map and the manipulation probability field according to the KL divergence, so as to achieve attention focusing on the tampered region. The physical manipulation probability field is obtained by convolving the pixel-level tampering mask with a two-dimensional Gaussian kernel. S60, the image patch features are projected onto a low-dimensional manifold space, and the manifold repulsion loss between the tampered patch and the normal prototype features is calculated and minimized to enhance the distinguishability between abnormal features and normal regions in the image; S70 introduces a gradient isolation operator in the localization branch to block the propagation of the localization task gradient to the shared backbone network, so that the localization task and the detection task can be decoupled and optimized in the task-specific subspace to avoid gradient interference. S80, construct a joint loss function including detection loss, localization loss, calibration loss and rejection loss to train the original model, optimize the model parameters, and obtain a multimodal media tampering detection and localization model.

2. The method for constructing a multimodal media tampering detection and localization model as described in claim 1, characterized in that, The S50 includes: S51, introducing a temperature coefficient The original attention score of the original cross-modal attention map Perform redistribution processing to generate a calibrated predicted probability distribution. : ； in: ； In the formula, Representing image patch features Text markup features The original attention scores between them; and They are respectively and The linear projection matrix, For feature dimensions; S52, calculate the calibrated predicted probability distribution using KL divergence. With physical probability field Loss of perceptual consistency between manipulations : ； wherein denotes the KL divergence; S53, minimizing the manipulation-aware consistency loss by to guide the alignment of the original attention map with the manipulation probability field. 3.The method of claim 1, wherein, The S60 includes: S61, using a nonlinear projection operator map the image patch features to a discriminative low-dimensional manifold space to obtain manifold features : ； S62, Select the target image patch features with a consistency score higher than a preset score threshold as the normal reference set. By analyzing the manifold features within the normal reference set Perform mean aggregation to construct a prototype of local normal features. : ； S63, Calculate the features of the patch to be detected. With local normal feature prototype Cosine similarity between Patch set for known tampered regions Introducing a repulsive boundary Comparison adjustment coefficient Calculate the modeling loss of local semantic miscoordination. : ； S64, Minimize the modeling loss for local semantic discrepancy This is to cause the features of the tampered region to be updated in a direction that deviates from the normal prototype, thereby explicitly widening the distribution gap between normal and abnormal features in the manifold space.

4. The method for constructing a multimodal media tampering detection and localization model as described in claim 1, characterized in that, The S70 includes: S71, in multimodal global representation Above, fine-grained image tampering classification is constructed respectively. Fine-grained text tampering classification and global true / false binary classification tasks Three physically isolated task projection heads, each executing its own prediction output: ； ； ； In the formula, , and These represent the fine-grained tampering classification results for image modalities, the fine-grained tampering classification results for text modalities, and the global binary classification results, respectively. S72, the corresponding decoupling loss function is calculated using cross-entropy. , and : ； ； ； Among them, when image label The time indicates that it has not been tampered with; text label When it indicates that it has not been tampered with; when the global tag , When, it indicates that the image and text are real and fake respectively, and EH() represents the cross-entropy function; S73, with decoupling loss function , and With the goal of minimizing, update the parameters of the shared backbone network.

5. The method for constructing a multimodal media tampering detection and localization model as described in claim 1, characterized in that, The S70 also includes: S74, calculate the cross-modal consistency score of image patch features and text tag features, and generate gating weights based on the consistency score, wherein the gating weights are used to characterize the degree of semantic consistency between different modal features; S75, the image patch features and text marker features are weighted and aggregated according to the gating weights to generate enhanced global judgment features; wherein, the gating weights are generated based on cross-modal consistency scores, and the enhanced aggregation of potential tampering regions is achieved by enhancing the weights of low consistency regions and suppressing the weights of high consistency regions.

6. The method for constructing a multimodal media tampering detection and localization model as described in claim 1, characterized in that, Joint loss function The expression is: ； in, For fine-grained image tampering classification loss, For fine-grained text tampering classification loss, For global binary classification loss, For local positioning loss, For consistency calibration loss; For manifold repulsion loss; Model the loss for consistency; , , , and These are the preset weighting coefficients for each loss item.

7. A multimodal media tampering detection and localization model, characterized in that, The multimodal media tampering detection and localization model is obtained by using the construction method of any one of claims 1 to 6, wherein the multimodal media tampering detection and localization model includes: Feature extraction and interaction module: used to extract image patch features and text tag features, and generate the original cross-modal attention map; Consistency calibration module: used to guide attention to the physically tampered region through KL divergence constraints; Manifold repulsion module: used to separate tampered features from normal features in a low-dimensional space; Task decoupling module: Used to isolate and locate task gradients by stopping gradient operators, enabling parallel optimization of multiple tasks; Inference Deployment Module: Used to perform lightweight detection and localization by dynamically removing auxiliary units during the inference phase.

8. An electronic device comprising a processor and a memory, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the multimodal media tampering detection and location method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal media tampering detection and location method according to any one of claims 1 to 6.

10. The application of the multimodal media tampering detection and localization model as described in claim 7 in the detection of deepfake videos, social media content review, news authenticity verification, and video surveillance systems.

Images