Multi-mode media tampering detection method, system and equipment based on multi-view comparative learning and medium

Abstract

The invention belongs to the technical field of multimedia analysis, and discloses a multi-modal media tampering detection method, system and device based on multi-view comparative learning and a medium, and the method comprises the steps: obtaining a training data set which comprises a training image-training text pair and a corresponding tampering category label; a cross encoder is introduced on the basis of a vision-language model, a plurality of multi-layer sensor head structures are arranged, and three kinds of comparative learning of noise enhancement, prototype-based and multi-label tampering classification are designed to obtain an initial multi-view comparative learning framework; training the initial multi-view comparative learning framework based on the training data set to obtain a trained multi-view comparative learning framework; and based on the trained multi-view contrast learning framework, executing a tampering detection task of the to-be-detected image-text to the data. According to the technical scheme, the accuracy and robustness of multi-label classification can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia analysis technology, and in particular relates to a multimodal media tampering detection method, system, device and medium based on multi-view comparative learning. Background Art

[0002] With the advancement of generative models, highly realistic fake content can be generated, leading to multimodal media manipulation (DGM) 4 ) problem is becoming increasingly serious. Existing methods usually align the original image-text pairs through contrastive learning and use a cross encoder to aggregate information from image and text modalities for detection and localization. However, these methods have the following two major problems:

[0003] 1. Dispersed Manipulation Distribution (DMD): Since all types of tampering are considered as a single category, the tampering samples are dispersed in the feature space, making it difficult to effectively distinguish different types of tampering.

[0004] 2. Noise Interference Confusion (NIC): The mixture of noise and subtle tampering traces in the data causes the model to mistakenly classify the original sample as a tampered sample, reducing the classification accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal media tampering detection method, system, device and medium based on multi-view contrast learning to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention provides a multimodal media tampering detection method based on multi-view contrastive learning, comprising:

[0007] Obtaining a training dataset, wherein the training dataset includes training image-training text pairs and corresponding tampering category labels;

[0008] Based on the vision-language model, we introduce a cross encoder, set up several multi-layer perceptron head structures, and design three types of contrastive learning: noise enhancement, prototype-based, and multi-label tampering classification, to obtain the initial multi-view contrastive learning framework.

[0009] The initial multi-view contrast learning framework is trained based on the training data set to obtain a trained multi-view contrast learning framework; and a tampering detection task of the image-text pair data to be detected is performed based on the trained multi-view contrast learning framework.

[0010] Optionally, the training of the initial multi-view contrastive learning framework based on the training dataset specifically includes:

[0011] The training image is input into the image encoder in the initial multi-view contrastive learning framework to extract image embedding, and the training text is input into the text encoder in the initial multi-view contrastive learning framework to extract global embedding and local embedding; the outputs of the image encoder and the text encoder are input into the cross encoder for aggregation processing, and the detection and positioning task is performed based on the aggregated data, and training is performed according to the target loss function to obtain a trained multi-view contrastive learning framework; wherein the target loss function includes noise enhancement alignment learning loss, multi-label contrastive learning loss, prototype-guided contrastive learning loss, binary classification loss, image tampering positioning loss, multi-label classification loss and text tampering positioning loss.

[0012] Optionally, the process of training according to the noise-augmented alignment learning loss specifically includes:

[0013] Determining a hard-to-distinguish positive sample image corresponding to the image embedding, calculating a mean and a standard deviation of the hard-to-distinguish positive sample image, generating a Gaussian noise vector based on the calculated mean and standard deviation, and adding the Gaussian noise vector to the image embedding to obtain a noisy image embedding;

[0014] The similarity distribution matching loss between the image embedding and the text embedding after adding noise is calculated based on cosine similarity. The similarity distribution matching loss is normalized according to the degree of noise interference to obtain the noise-enhanced alignment learning loss. The noise-enhanced alignment learning loss is used to guide the model to minimize the interference caused by noise and improve the accuracy of alignment.

[0015] Optionally, the process of training according to the multi-label contrastive learning loss specifically includes:

[0016] Based on the momentum contrast framework, two queues are used to maintain the embeddings input into the initial multi-view contrastive learning framework.

[0017] According to the InfoNCE loss function, the initial multi-view contrastive learning framework performs contrastive learning of global and local views, and obtains the global view contrast loss and local view contrast loss;

[0018] Combining the global view contrast loss and the local view contrast loss, a multi-label contrast loss is obtained; the multi-label contrast loss is used to combine information from the global view and the local view to improve the detection and positioning performance of the initial multi-view contrast learning framework.

[0019] Optionally, the process of training based on the prototype-guided contrastive learning loss specifically includes:

[0020] Initialize the prototype vector of each tampering category to the mean of the corresponding category sample embedding;

[0021] Use a cross encoder to fuse image data, text data, and prototype embedding to update the prototype vector;

[0022] Based on the contrast loss function, the alignment effects between image data and text data and prototype embeddings are optimized respectively, so that the embeddings of samples of the same category are close to the corresponding prototype vectors, and the embeddings of samples of different categories are far away from the prototype vectors of other categories; for other categories, their contrast losses are calculated symmetrically by replacing the corresponding features respectively.

[0023] A multimodal media tampering detection system based on multi-view contrastive learning, comprising:

[0024] A data acquisition module is used to obtain a training data set, wherein the training data set includes training image-training text pairs and corresponding tampering category labels;

[0025] A model construction and training module is configured to introduce a cross encoder based on the vision-language model, set up several multi-layer perceptron head structures, and design three types of contrastive learning: noise enhancement, prototype-based, and multi-label tampering classification, to obtain an initial multi-view contrastive learning framework; and train the initial multi-view contrastive learning framework based on the training dataset to obtain a trained multi-view contrastive learning framework.

[0026] The model application module is used to perform tampering detection tasks on image-text pairs based on the trained multi-view contrastive learning framework.

[0027] An electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform a multimodal media tampering detection method based on multi-view contrast learning.

[0028] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a multimodal media tampering detection method based on multi-view contrast learning.

[0029] The technical effects of the present invention are:

[0030] By leveraging multi-view contrastive learning, this paper not only enhances the differences between tampering categories but also reduces the distance within a category, thereby improving detection performance. Compared with UFAFormer and CrUr, this paper achieves better results on multiple key indicators while maintaining low model complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0032] Figure 1 Schematic diagram of the multi-view contrastive learning framework (MPCL) model structure in an embodiment of the present invention;

[0033] Figure 2 Schematic diagram of the results of each training cycle of the MPCL model in an embodiment of the present invention;

[0034] Figure 3 This is an implementation flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0035] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as limiting the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0036] It should be understood that the terms described herein are intended only to describe particular embodiments and are not intended to limit the present invention. In addition, for numerical ranges herein, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Each smaller range between any intermediate value within a stated value or stated range and any other stated value or intermediate value within the stated range is also encompassed by the present invention. The upper and lower limits of these smaller ranges may be independently included or excluded within the scope.

[0037] It will be apparent to those skilled in the art that various modifications and variations may be made to the specific embodiments of the present invention without departing from the scope or spirit of the invention. Other embodiments will be apparent to those skilled in the art from the present invention. The present description and examples are intended to be illustrative only.

[0038] The words “include,” “including,” “have,” “contain,” etc. used in this article are open-ended terms, meaning including but not limited to.

[0039] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] like Figure 1 - Figure 3As shown, this embodiment provides a multimodal media tampering detection method based on multi-view contrastive learning, including: obtaining a training data set, the training data set including training image-training text pairs and corresponding tampering category labels; introducing a cross encoder based on a vision-language model, setting several multi-layer perceptron head structures, and designing three types of contrastive learning: noise enhancement, prototype-based, and multi-label tampering classification, to obtain an initial multi-view contrastive learning framework; training the initial multi-view contrastive learning framework based on the training data set to obtain a trained multi-view contrastive learning framework; and performing a tampering detection task on the image-text pair data to be detected based on the trained multi-view contrastive learning framework.

[0041] This embodiment aims to provide an improved multi-view contrastive learning framework (MPCL) to effectively detect and locate media tampering in image and text dual modalities, and improve the accuracy and robustness of multi-label classification by solving the problems of fragmented tampering distribution and noise interference confusion in existing methods.

[0042] To achieve the above objectives, this embodiment provides a multi-view contrastive learning framework (MPCL), which includes the following three key components:

[0043] 1. Noise-Augmented Alignment Learning (NAAL):

[0044] Simulate the noise in the original samples, optimize the global similarity distribution, and improve the robustness of the model to noise interference.

[0045] Noise-augmented representations are generated by adding Gaussian noise based on sample mean and variance to the original image embedding.

[0046] The Similarity Distribution Matching (SDM) loss function is used to ensure the consistency of the global similarity distribution.

[0047] 2. Multi-Label Contrastive Learning (MLCL):

[0048] The embeddings of different tampering types are clustered separately to enhance the accuracy of multi-label classification.

[0049] Through comparative learning of global and local views, fine-grained tampering traces are captured, ensuring that the features of different tampering categories are more separated.

[0050] 3. Prototype-Guided Contrastive Learning (PCL):

[0051] We use a learnable prototype vector to represent each tampering category, attracting embeddings of the same category and repelling embeddings of different categories.

[0052] The prototype vector interacts with image and text embeddings through a cross-modal encoder to enhance the discriminativeness of features.

[0053] This embodiment uses the Hierarchical Multimodal Tampering Reasoning Transformer (HAMMER) as the baseline model of this embodiment. Figure 1 As shown, the MPCL framework of this embodiment includes three encoders - image encoder, text encoder and cross encoder - and multiple multi-layer perceptron (MLP) heads for classification and localization. Given an image-text pair (I i ,T i ), in this embodiment, the image I is first i Split into a series of M non-overlapping patches and feed them into an image encoder to extract the image embedding v cls ,v1,…,v M , where v cls as [CLS] tag. Similarly, for the input text T i , this embodiment applies a text encoder to obtain global and local embeddings t cls ,t1,…,t N , where N is the sequence length. This embodiment initializes the learnable query {p1, p2, ..., p5} as a prototype embedding, where 5 corresponds to the five categories in the dataset (original and four tampered ones), and each prototype attempts to match the corresponding input image or text. To solve the DMD and NIC problems, this embodiment further adopts noise-enhanced alignment learning Multi-label contrastive learning and prototype-guided comparative learning In addition, the cross encoder aggregates image, prototype, and text embeddings for prediction tasks, using the last six layers of BERT combined with masked language modeling Enhance cross-modal interaction. Following HAMMER, this embodiment uses the aggregation representation {m cls ,m1,…,m M} for binary classification Image tampering location Multi-label classification and text tampering positioning For simplicity, this embodiment refers to these losses as The overall MPCL loss function formula is:

[0054]

[0055] Where β, γ and η are set to 0.5, 0.1 and 0.1 respectively.

[0056] Noise Augmented Alignment Learning (NAAL): In each batch, the most difficult original sample is selected and a noise vector based on its mean and variance is added.

[0057] Similarity Distribution Matching (SDM) loss is applied to optimize the global similarity distribution of image and text embeddings.

[0058] Noise-enhanced representation construction: Given the global embedding {v cls ,t cls}, first select an original image to embed And determine the hardest positive sample image associated with the embedding. Then, this embodiment calculates the mean μ and standard deviation σ of the hardest positive sample. Gaussian noise generated according to these parameters is then added to the instance-normalized image embedding. This process can be expressed as:

[0059]

[0060] in, is the noise-enhanced representation, represents the noise vector.

[0061] Noise-enhanced alignment: To maintain intra-class consistency in the presence of noise interference, this embodiment uses Similarity Distribution Matching (SDM) loss, which incorporates the cosine similarity distribution of image-text pair embeddings into the Kullback-Leibler (KL) divergence to correlate representations between different modalities. For a batch of image-text pairs, this embodiment creates a set of K pairs Where (y ij =1) indicates v cls,i and t cls,j is the original embedding. Then, this embodiment uses the softmax function to calculate the probability of matching pairs:

[0062]

[0063] Where τ is the temperature hyperparameter and sim() represents the cosine similarity between the two embeddings. The SDM loss for image-to-text can be calculated by the following formula:

[0064]

[0065] in, Is the true original probability distribution, ∈ is a small constant added for numerical stability. The SDM loss of text to image is calculated in a symmetric way, and the overall SDM loss is defined as:

[0066]

[0067] In order to further guide the model to minimize the interference caused by noise, this embodiment adopts a dynamic method to adjust the sensitivity of the loss function in the noise-enhanced image-text alignment process. This embodiment adjusts the sensitivity of the loss function according to the degree of noise interference (denoted as σ 2 )right Normalize and emphasize the penalty for high interference levels. The process can be expressed as:

[0068]

[0069] Overall, noise-augmented alignment learning is defined as:

[0070]

[0071] Multi-label contrastive learning (MLCL):

[0072] Use two queues to maintain the most recent image and text representations respectively, expand the batch size and include more samples of different categories.

[0073] The InfoNCE loss function is adopted to perform comparative learning of global and local views to ensure the embedding separation of different tampering categories.

[0074] Based on noise-enhanced raw alignment learning for binary classification, this embodiment focuses on more detailed multi-label classification challenges. Take a specific tampering category (e.g., face swapping) as an example. In this case, since the text accompanying the tampered image remains unchanged, there is a clear semantic difference between the two modalities. The cross encoder exploits this difference during the fusion process to detect tampering. Therefore, the main goal of this embodiment is to amplify this difference and group similar tampering categories into unique clusters before fusion.

[0075] Following MoCo and HAMMER++, this embodiment utilizes two queues to maintain the most recent M image-text representations from a unimodal encoder. The introduction of queues provides larger batch sizes and more samples of different categories. In addition, HAMMER++ introduces contrastive learning on global and local views. This embodiment also uses this loss to capture subtle tampering traces that may only affect a few local patches. Specifically, this embodiment uses the InfoNCE loss, which is formulated as:

[0076]

[0077] Where Q is the maintained queue and τ is the temperature parameter.

[0078] Different from traditional cross-modal contrastive learning, this embodiment focuses on unimodal alignment, aiming to cluster embeddings of four basic categories. The global view of CL can be expressed as:

[0079]

[0080] The local view of CL is realized by the following formula:

[0081]

[0082] Where M and N represent the number of image patches and text tags, respectively.

[0083] The overall multi-label contrast loss is calculated as:

[0084]

[0085] Prototype-guided Contrastive Learning (PCL):

[0086] Initialize the prototype vector of each tampering category to the mean of the embedding of the corresponding category samples.

[0087] Use a cross encoder to fuse image, text, and prototype embeddings and update the prototype vector.

[0088] The sample embeddings are aligned with the corresponding prototype vectors via a contrastive loss function.

[0089] The core idea of ​​prototype learning is to learn a set of representative prototypes that compress the core features of each category in the data distribution. Compared with image-text contrastive learning, prototype learning provides a more intuitive way to align instance-level samples with their corresponding category prototypes.

[0090] Prototype initialization and update:

[0091] Given a batch of global embeddings This embodiment initializes the prototype P of each category by calculating the mean vector of the corresponding feature. In particular, for the original sample, this embodiment will and The process can be described as follows:

[0092]

[0093] Among them, p c represents the prototype of category c, x cls represents the embedding of the corresponding category c, B c represents the set of samples belonging to category c.

[0094] Following BLIP, this embodiment inputs the query, image and text into the cross encoder to obtain the updated prototype P'={p'1,p'2,...,p'5} and the aggregate representation M={m cls ,m1,…,m M}, where M represents the number of image patches. The built-in attention mechanism in the cross encoder detects cross-modal differences by comparing image and text features with predefined category-specific queries. This process can be expressed as:

[0095] {P',M}=Transformer(Concat[V,P],T,T)

[0096] This embodiment also introduces the to help identify semantic differences.

[0097] To further smooth the changes in prototype features, this embodiment uses the sample features of the current mini-batch to update the prototype of each category. The update process can be expressed by the following equation:

[0098] p′ i ←αp′ i +(1-α)x i

[0099] This embodiment uses single-modal alignment for optimization. Prototype-guided contrastive learning can be expressed as:

[0100]

[0101] The losses of other categories are calculated symmetrically by replacing the corresponding features respectively. The overall goal of prototype-guided contrastive learning can be expressed as:

[0102]

[0103] Table 1. In DGM 4 Performance comparison with state-of-the-art methods on the dataset

[0104]

[0105] This example compares the proposed MPCL method with recent state-of-the-art methods, as shown in Table 1. First, this example compares with the most influential works, HAMMER and HAMMER++. These methods further separate tampered features by aligning original image-text pairs, leading to semantic inconsistencies when interacting with tampered features. As previously analyzed in this example, this approach inevitably leads to a dispersed distribution of tampered features and confounding noise interference. This example is motivated to address these issues. By leveraging multi-view contrastive learning, MPCL increases the differences between different tampered categories while reducing the distance within the same category, further amplifying semantic inconsistencies and achieving significant improvements across all metrics. Next, this example evaluates this method against another method, UFAFormer, which is inspired by the success of frequency domain information in deepfake detection, where tampering often leaves detectable artifacts in the frequency domain. UFAFormer integrates frequency domain features, image, and text into a cross-encoder designed to capture spatial and frequency inconsistencies, thereby improving detection performance. Notably, their multi-label classification metrics only show modest improvements. This limited improvement is mainly due to the fact that their method only supplements information without effectively distinguishing different types of tampering. Finally, this embodiment compares the method of this embodiment with CrUr, which captures semantic inconsistencies through bidirectional reconstruction of image and text features. Although CrUr performs competitively in text localization, especially in precision, recall and F1 value of 80.13%, 77.71% and 78.90% respectively, this comes at the cost of extremely high model complexity due to the stacking of multiple Transformer blocks. In contrast, the MP CL method of this embodiment surpasses CrUr in most key indicators while maintaining low model complexity by introducing a small number of prototype embeddings into VLP. Therefore, MPCL shows obvious advantages in overall performance and efficiency.

[0106] Ablation studies:

[0107] Table 2. In DGM 4 Ablation study of the components proposed in this embodiment on the dataset

[0108]

[0109] In the 9th row corresponding to the method in Table 2, the four icons from left to right represent noise-enhanced alignment learning, multi-label contrastive learning, masked language modeling, and prototype-guided contrastive learning, respectively.

[0110] This example analyzes the effectiveness of the optimization objective in the MPCL framework. Here, this example uses the image-text contrast loss (ITC) as the baseline. Starting with No. 1, this example replaces ITC with similarity distribution matching (SDM) in all subsequent experiments.

[0111] Baseline comparison (ITC vs. SDM): As shown in Table 2, replacing ITC with SDM (No. 0 vs. No. 1) provides significant improvements on all metrics, with the EER significantly reduced from 17.66% to 15.57%. This indicates that SDM provides more effective cross-modal feature alignment, enabling a clearer distinction between original and manipulated content, thus laying a more solid foundation for the integration of subsequent components.

[0112] Effectiveness of NAAL: The introduction of NAAL (No. 2) further improved the performance of No. 1, especially in binary classification and text localization, where the ACC increased from 84.02% to 86.00% (+1.98%). This indicates that NAAL, by simulating noise and optimizing the global similarity distribution, helps the model better resist noise interference, thereby achieving more accurate classification and localization.

[0113] Effectiveness of MLCL: The introduction of multi-label contrastive learning (No. 3) significantly improves the performance of multi-label classification tasks. Specifically, mAP is improved from 83.49% of the baseline SDM to 86.60%, CF1 is improved from 76.38% to 80.47%, and OF1 is improved from 76.44% to 80.81%. These improvements highlight the effectiveness of MLCL in distinguishing DGMs. 4 The effectiveness of MLCL in distinguishing multiple tampering categories in the dataset. By clustering similar tampering types into different clusters, MLCL ensures that the characteristics of different tampering categories are more clear, reducing overlap and ambiguity in classification. The role of MLCL was further verified in No. 5. Combining MLCL with NAAL brought even greater improvements, further strengthening the role of MLCL in enhancing feature differentiation. In addition, the improvement in CF1 and OF1 demonstrates MLCL's ability to maintain a balance between precision and recall, ensuring that the model not only accurately identifies multiple tampering labels, but also captures true positive examples in more categories.

[0114] Effectiveness of MLM: Inspired by the proven effectiveness of Masked Language Modeling (MLM) in previous research, this example introduces MLM into the fusion process within the MPCL framework, resulting in significant improvements in multiple metrics. By masking certain markers during cross-modal interaction, MLM forces the model to predict missing information, thereby deepening semantic understanding and improving alignment between image and text modalities.

[0115] Effectiveness of PCL: In initial experiments, this example introduced prototypes into the cross-encoder but did not provide a dedicated loss function for their learning (No. 6). This example observed a slight decrease in performance across several metrics. Specifically, mAP dropped from 87.08% (No. 5) to 85.12%, and CF1 and OF1 dropped from 77.83% and 76.97% to 76.83% and 76.12%, respectively. This decrease suggests that while introducing prototypes into the cross-encoder provides additional semantic anchors, the lack of a specific optimization objective limits their effective contribution to the model learning process. However, when this example introduced prototype-guided contrastive learning (PCL) and provided an explicit loss function for optimizing these prototypes (No. 7), performance improved significantly, with mAP increasing to 88.35%, and CF1 and OF1 increasing to 84.68% and 84.66%, respectively. These results highlight the importance of setting clear objectives for prototype learning. By explicitly guiding the learning of prototypes, PCL is able to better align features with their corresponding tampering categories, resulting in more distinct and meaningful clusters. In addition, the improvement in image localization metrics, with mean IoU increased from 75.53% to 77.52%, further confirms that PCL not only improves classification tasks but also improves the accuracy of localizing tampered images. This shows that PCL effectively enhances the semantic representation of the prototype, making it a more powerful anchor point in cross-modal interaction, thereby improving the overall performance of the model.

[0116] t-SNE visualization: Figure 2 As the present embodiment goes through each epoch (Epoch0 to EpochBest) in the MPCL model, the present embodiment observes the following key improvements:

[0117] Prototype-guided clustering: In successive epochs, the learnable prototype vectors effectively attract features of the same category, causing the clusters of each tampering category to become more compact and unique. As the clusters gradually tighten, this example clearly shows the process of prototype-guided feature convergence.

[0118] Multi-label contrastive learning: The MLCL component plays a crucial role in separating different tampering categories. As training progresses, the clusters between different tampering categories become more distinct, and the overlap between categories decreases significantly. This separation ensures that features from different tampering types do not interfere with each other, thus maintaining a clear decision boundary.

[0119] Noise-augmented alignment learning: A notable effect of NAAL is observed when processing the original samples. Initially, the original features are mixed with the tampered features (as seen in the early epochs). However, as NAAL is applied, the original features gradually align closer to their correct clusters, thereby reducing the confusion with the tampered features.

[0120] The MPCL method proposed in this example demonstrates comprehensive performance improvements in image-text pair tampering detection and localization compared to traditional models such as HAMMER, especially when dealing with three types of high-difficulty challenges.

[0121] When it comes to perceiving subtle tampering, traditional methods have difficulty identifying tiny substitutions hidden in complex backgrounds or image noise. MPCL, on the other hand, uses reconstruction and detail analysis techniques to accurately capture these hidden visual differences.

[0122] In terms of deep semantic understanding, when there is a semantic conflict between text and images, traditional methods are easily misled by surface concepts. MPCL, on the other hand, can grasp the core semantic elements through deep context understanding and accurately identify the logical inconsistencies of text replacement.

[0123] In terms of local change detection, traditional methods generally have the problem of missing detection when editing image attributes in small areas. However, MPCL can reliably identify these subtle modifications through precise local feature analysis, demonstrating its powerful capabilities in refined detection tasks.

[0124] Overall, these examples show that MPCL significantly outperforms HAMMER in detecting and localizing tampered text. By leveraging multi-label contrastive learning and prototype-guided contrastive learning, MPCL improves its ability to distinguish authentic from tampered content and reduces missed detections and false positives, especially in semantically complex text.

[0125] Hyperparameter sensitivity analysis:

[0126] Table 3 in DGM 4 The impact of different hyperparameter settings on MPCL performance on the dataset

[0127]

[0128] The proposed MPCL framework uses multiple loss functions to optimize the DGM 4 Each subtask of the dataset. For the loss function inherited from ALBEF or HAMMER (such as L mlm and L DGM4 ), this embodiment directly adopts their hyperparameter settings to ensure a fair comparison. Therefore, this embodiment only performs sensitivity analysis on the weights of NAAL, MLC L, and PCL losses, which are unique loss functions in the method of this embodiment.

[0129] As shown in Table 3, the performance metrics for all subtasks vary little across different hyperparameter settings. This demonstrates that the proposed MPCL framework is robust to variations in the weights of these loss functions. Whether in binary classification, multi-label classification, or localization, MPCL maintains consistent performance, further demonstrating its robustness in multimodal manipulation detection tasks.

[0130] Single field performance comparison:

[0131] Table 4DGM 4 Comparison results of deepfake detection methods on datasets

[0132]

[0133] Table 5 Sequence annotation methods used in DGM 4 Comparison results of forged text detection on datasets

[0134]

[0135] This embodiment comprehensively compares the MPCL method with the existing single-modal model. Through MPCL learning, the single-modal encoder of this embodiment has the ability to effectively cluster different tampering features. This allows the encoder to be able to perform well on the DGM even when only a simple prediction head is used for fine-tuning. 4 As shown in Tables 4 and 5, MPCL not only performs well in the binary classification task, but also in the tampering localization task. Specifically, MPCL improves the semantic inconsistency between different tampering categories, allowing the encoder to better capture and distinguish these subtle differences.

[0136] These results demonstrate that MPCL has a stronger ability in accurately locating tampered content, further demonstrating its broad applicability in multimodal data processing.

[0137] Cross-domain performance comparison:

[0138] Table 6 Comparison of cross-domain face forgery detection generalization ability

[0139]

[0140] The results in Table 6 highlight the robustness and effectiveness of the MPCL approach, particularly in the challenging scenario of unseen manipulations, where the model is tested directly on a diverse set of forged content without prior exposure to this data. MPCL significantly outperforms other methods, achieving the highest AUC of 66.80% for distinguishing between real and fake content under these conditions. Furthermore, MPCL exhibits the lowest EER (36.20%), demonstrating its superior ability to balance false positives and false negatives, an important factor in ensuring model reliability, especially when dealing with unseen manipulations. An accuracy of 65.67% further demonstrates MPCL's strong generalization capabilities.

[0141] MPCL's significant improvement is attributed to its multi-view contrastive learning mechanism, which effectively captures and clusters distinct forgery features, even when faced with unseen data. This mechanism enhances the model's ability to discern subtle semantic inconsistencies between forgery types, improving detection accuracy across diverse datasets. These results demonstrate that MPCL not only excels on standard deepfake detection tasks but also demonstrates remarkable robustness and accuracy when applied to unseen manipulations, bringing significant progress to the field.

[0142] A multimodal media tampering detection system based on multi-view contrastive learning, comprising:

[0143] A data acquisition module is used to obtain a training data set, wherein the training data set includes training image-training text pairs and corresponding tampering category labels;

[0144] A model construction and training module is configured to introduce a cross encoder based on the vision-language model, set up several multi-layer perceptron head structures, and design three types of contrastive learning: noise enhancement, prototype-based, and multi-label tampering classification, to obtain an initial multi-view contrastive learning framework; and train the initial multi-view contrastive learning framework based on the training dataset to obtain a trained multi-view contrastive learning framework.

[0145] The model application module is used to perform tampering detection tasks on image-text pairs based on the trained multi-view contrastive learning framework.

[0146] An electronic device includes a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform a multimodal media tampering detection method based on multi-view contrast learning.

[0147] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a multimodal media tampering detection method based on multi-view contrast learning.

[0148] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal media tampering detection method based on multi-view contrastive learning, characterized in that: include: Obtaining a training dataset, wherein the training dataset includes training image-training text pairs and corresponding tampering category labels; Based on the vision-language model, we introduce a cross encoder, set up several multi-layer perceptron head structures, and design three types of contrastive learning: noise enhancement, prototype-based, and multi-label tampering classification, to obtain the initial multi-view contrastive learning framework. The initial multi-view contrast learning framework is trained based on the training data set to obtain a trained multi-view contrast learning framework; and a tampering detection task of the image-text pair data to be detected is performed based on the trained multi-view contrast learning framework.

2. The multimodal media tampering detection method based on multi-view contrastive learning according to claim 1 is characterized in that: The training of the initial multi-view contrastive learning framework based on the training data set specifically includes: The training image is input into the image encoder in the initial multi-view contrastive learning framework to extract image embedding, and the training text is input into the text encoder in the initial multi-view contrastive learning framework to extract global embedding and local embedding; the outputs of the image encoder and the text encoder are input into the cross encoder for aggregation processing, and the detection and positioning task is performed based on the aggregated data, and training is performed according to the target loss function to obtain a trained multi-view contrastive learning framework; wherein the target loss function includes noise enhancement alignment learning loss, multi-label contrastive learning loss, prototype-guided contrastive learning loss, binary classification loss, image tampering positioning loss, multi-label classification loss and text tampering positioning loss.

3. The multimodal media tampering detection method based on multi-view contrastive learning according to claim 2, characterized in that: The process of training according to the noise-augmented alignment learning loss specifically includes: Determining a hard-to-distinguish positive sample image corresponding to the image embedding, calculating a mean and a standard deviation of the hard-to-distinguish positive sample image, generating a Gaussian noise vector based on the calculated mean and standard deviation, and adding the Gaussian noise vector to the image embedding to obtain a noisy image embedding; The similarity distribution matching loss between the image embedding and the text embedding after adding noise is calculated based on cosine similarity. The similarity distribution matching loss is normalized according to the degree of noise interference to obtain the noise-enhanced alignment learning loss. The noise-enhanced alignment learning loss is used to guide the model to minimize the interference caused by noise and improve the accuracy of alignment.

4. The multimodal media tampering detection method based on multi-view contrastive learning according to claim 2, characterized in that: The process of training according to the multi-label contrastive learning loss specifically includes: Based on the momentum contrast framework, two queues are used to maintain the embeddings input into the initial multi-view contrastive learning framework. According to the InfoNCE loss function, the initial multi-view contrastive learning framework performs contrastive learning of global and local views, and obtains the global view contrast loss and local view contrast loss; Combining the global view contrast loss and the local view contrast loss, a multi-label contrast loss is obtained; the multi-label contrast loss is used to combine information from the global view and the local view to improve the detection and positioning performance of the initial multi-view contrast learning framework.

5. The multimodal media tampering detection method based on multi-view contrastive learning according to claim 2, characterized in that: The process of training with contrastive learning loss guided by the prototype specifically includes: Initialize the prototype vector of each tampering category to the mean of the corresponding category sample embedding; Use a cross encoder to fuse image data, text data, and prototype embedding to update the prototype vector; Based on the contrast loss function, the alignment effects between image data and text data and prototype embeddings are optimized respectively, so that the embeddings of samples of the same category are close to the corresponding prototype vectors, and the embeddings of samples of different categories are far away from the prototype vectors of other categories; for other categories, their contrast losses are calculated symmetrically by replacing the corresponding features respectively.

6. A multimodal media tampering detection system based on multi-view contrastive learning, characterized in that: include: A data acquisition module is used to obtain a training data set, wherein the training data set includes training image-training text pairs and corresponding tampering category labels; The model construction training module introduces a cross encoder based on the vision-language model, sets up several multi-layer perceptron head structures, and designs three types of contrastive learning: noise enhancement, prototype-based, and multi-label tampering classification, to obtain the initial multi-view contrastive learning framework; Training the initial multi-view contrastive learning framework based on the training data set to obtain a trained multi-view contrastive learning framework; The model application module is used to perform tampering detection tasks on image-text pairs based on the trained multi-view contrastive learning framework.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform a multimodal media tampering detection method based on multi-view contrast learning according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The device stores a computer program, which, when executed by a processor, implements a multimodal media tampering detection method based on multi-view contrast learning as described in any one of claims 1 to 5.

Images