Image tampering detection method and device based on cross-modal ViT architecture

Abstract

The invention provides an image tampering detection method and device based on a cross-modal ViT architecture. The method and device are particularly suitable for positioning digital image tampering operations such as copying-moving, splicing and erasing. The method comprises the following steps: acquiring a to-be-detected image, and inputting the to-be-detected image into a pre-established cross-modal double-flow ViT architecture; the cross-modal double-flow ViT architecture comprises a space domain ViT encoder layer, a frequency domain ViT encoder layer and a deformable cross attention fusion layer; extracting spatial domain feature data corresponding to the to-be-detected image through the spatial domain ViT encoder layer; extracting frequency domain feature data corresponding to the to-be-detected image through a frequency domain ViT encoder layer; and performing dynamic gating fusion on the spatial domain feature data and the frequency domain feature data through a deformable cross attention fusion layer to obtain fusion features, and performing image tampering detection based on the fusion features. According to the method, the problem of long-distance dependence is solved based on a self-attention mechanism of Vision Transform, and picture tampering detection is realized through spatial domain and frequency domain cross-modal conjoint analysis. Technical capability is provided for application scenes such as digital forensics and content auditing, and the material authenticity detection level is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and digital image security technology, and in particular to an image tampering detection method and apparatus based on a cross-modal ViT architecture. Background Technology

[0002] The rapid upgrades and iterations of digital image editing tools and the development of AI technology have created fertile ground for digital image tampering, making such manipulation easier and more difficult to detect. When tampered images circulate in society, they can easily disrupt social order, undermine the credit system, infringe on the rights of others, and trigger rumors, economic losses, and legal concerns. In the financial sector, false image data can affect the security of financial transactions, interfere with financial supervision, and damage the reputation of financial institutions.

[0003] Traditional CNN methods not only rely on local features, making it difficult to capture long-distance tampering traces across regions, such as repeated regions. Single-domain analysis is insufficient: relying solely on spatial pixel or frequency domain statistical features, it cannot handle complex tampering, such as manual retouching after double JPEG compression. Furthermore, its generalization ability is poor: its performance drops sharply when detecting unknown tampering types, such as the novel DeepFake algorithm. Summary of the Invention

[0004] In view of the above problems, the present invention proposes an image tampering detection method and apparatus based on a cross-modal ViT architecture to overcome or at least partially solve the above problems, and is particularly suitable for digital image tampering operations such as locating copy-move, splicing, and erasing.

[0005] According to one aspect of the present invention, an image tampering detection method based on a cross-modal ViT architecture is provided, comprising:

[0006] The image to be detected is acquired and input into a pre-established cross-modal dual-stream ViT architecture; the cross-modal dual-stream ViT architecture includes a spatial domain ViT encoder layer, a frequency domain ViT encoder layer, and a deformable cross-attention fusion layer;

[0007] The spatial domain feature data corresponding to the image to be detected is extracted through the spatial domain ViT encoder layer; the frequency domain feature data corresponding to the image to be detected is extracted through the frequency domain ViT encoder layer.

[0008] The spatial domain feature data and frequency domain feature data are dynamically gated and fused through the deformable cross-attention fusion layer to obtain fused features, and image tampering detection is performed based on the fused features.

[0009] Optionally, the dynamic gating fusion of the spatial domain feature data and the frequency domain feature data through the deformable cross-attention fusion layer includes:

[0010] Spatial domain feature data is used as the Query, and frequency domain feature data is used as the Key and Value, which are then input into the deformable cross-attention fusion layer.

[0011] The deformable cross-attention fusion layer dynamically adjusts the attention weights through dynamic gating, enabling spatial feature data to be aligned with the relevant parts of frequency domain feature data at different locations, thereby outputting fused feature data;

[0012] The fused feature data includes a set tag.

[0013] Optionally, the deformable cross-attention fusion layer dynamically adjusts the attention weights via dynamic gating, including:

[0014] The deformable cross-attention fusion layer forces the spatial and frequency domain features of the normal region to align through the cross-modal InfoNCE loss function, while amplifying the feature differences in the tampered region.

[0015] Combining dynamic gating weighted optimization, the mathematical formula is as follows:

[0016]

[0017] in, and These are the spatial and frequency domain characteristics of a weighted normal region at the same location. It is the weighted frequency domain feature of other locations, sim(·) is the cosine similarity, τ is the temperature parameter, which controls the distribution smoothness, and L Gate-InfoNCE denoted as the cross-modal contrastive loss value after dynamic gating weighting, N represents the total number of positive sample pairs in the normal region, representing the total number of effective samples that the model needs to strengthen alignment, and M represents the total number of negative samples, representing the total number of interference samples that the model needs to distinguish.

[0018] Optionally, the cross-modal dual-stream ViT architecture further includes a multi-task joint decision layer, which includes a global classification head and a pixel classification head; image tampering detection based on the fused features includes:

[0019] The global classification head effectively fuses features associated with a set label through a fusion strategy, and then uses the fused feature information to output a judgment result on whether the image has been tampered with or not through a preset classification algorithm.

[0020] The pixel localization head generates a pixel-level tampering heatmap through a feature pyramid decoder to show the tampered areas that may exist in the image to be detected.

[0021] Optionally, the multi-task joint decision layer further includes a frequency domain anomaly auxiliary head for predicting the anomaly score of the DCT coefficient distribution, thereby enhancing the interpretability of the model.

[0022] Optionally, extracting the spatial feature data corresponding to the image to be detected through the spatial ViT encoder layer includes:

[0023] The image to be detected is divided into blocks, and the spatial feature data corresponding to the block image to be detected is extracted by the spatial domain ViT encoder layer.

[0024] The spatial domain ViT encoder layer employs a hierarchical vision transformer ViT.

[0025] Optionally, extracting the frequency domain feature data corresponding to the image to be detected through the frequency domain ViT encoder layer includes:

[0026] The image to be detected is divided into blocks and frequency domain transformation is performed. The frequency domain feature data corresponding to the image to be detected after frequency domain transformation is extracted through the frequency domain ViT encoder layer.

[0027] The frequency domain ViT encoder layer employs a lightweight vision transformer ViT; the frequency domain transformation includes Discrete Cosine Transform (DCT) or Fourier Transform (FFT).

[0028] According to another aspect of the present invention, an image tampering detection device based on a cross-modal ViT architecture is also provided, comprising:

[0029] The image acquisition module is used to acquire the image to be detected and input the image to be detected into a pre-established cross-modal dual-stream ViT architecture; the cross-modal dual-stream ViT architecture includes a spatial domain ViT encoder layer, a frequency domain ViT encoder layer, and a cross-attention fusion layer;

[0030] The cross-modal dual-stream ViT architecture extracts spatial feature data corresponding to the image to be detected through the spatial ViT encoder layer; extracts frequency feature data corresponding to the image to be detected through the frequency ViT encoder layer; and fuses the spatial feature data and frequency feature data through a cross-attention fusion layer to obtain fused features, and performs image tampering detection based on the fused features.

[0031] According to another aspect of the present invention, a computer-readable storage medium is also provided for storing program code, which, when executed by a processor, is used to implement the image tampering detection method based on the cross-modal ViT architecture described in any of the preceding claims.

[0032] According to another aspect of the present invention, a computing device is also provided, characterized in that the computing device includes a processor and a memory:

[0033] The memory is used to store program code and transmit the program code to the processor;

[0034] The processor is used to execute the image tampering detection method based on the cross-modal ViT architecture described above according to the instructions in the program code.

[0035] The image tampering detection method and apparatus based on the cross-modal ViT architecture of the present invention solves the long-distance dependency problem based on the self-attention mechanism of VisionTransformer, and achieves image tampering detection through cross-modal joint analysis in the spatial and frequency domains. The image tampering detection method and apparatus based on the cross-modal ViT architecture of the present invention have the following beneficial effects:

[0036] (1) Multimodal complementarity detection: By adopting a joint analysis of frequency and spatial domains, and with the help of the dual-stream architecture of VisionTransformer (ViT) and cross-modal attention mechanism, the accuracy and robustness of image tampering detection are effectively improved.

[0037] (2) Dynamic Gated Fusion Mechanism: Breaking through the traditional static weighted fusion, it pioneers dynamic modality selection based on regional tampering confidence, realizing on-demand weighting of spatial and frequency domain features. This avoids using uniform weights for all regions, thereby improving detection performance and enhancing the generalization ability to unknown tampering methods.

[0038] (3) Cross-regional joint analysis: ViT self-attention mechanism effectively identifies non-local repetitive regions and has excellent detection performance for copy-move tampering type.

[0039] (4) Multi-task joint decision-making mechanism: Global classification head: Uses [CLS] token fusion features to perform image-level authenticity determination. Pixel localization head: Generates pixel-level tampering heatmaps through feature pyramid decoder. Frequency domain anomaly auxiliary head: Predicts DCT coefficient distribution anomaly scores to enhance model interpretability.

[0040] (5) Resistance to attack: The tamper detection technology has strong resistance to attack when facing a variety of common post-processing interference methods (such as spatial ambiguity, noise interference, etc.). Combined with the dynamic gating mechanism, the resistance to attack is further enhanced. The dynamic gating will automatically increase the weight of frequency domain features (because spatial features are easily destroyed), so that the contrastive learning can still maintain the detection accuracy based on frequency domain anomalies (such as resampling traces).

[0041] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below.

[0042] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0044] Figure 1 A schematic flowchart of an image tampering detection method based on a cross-modal ViT architecture according to an embodiment of the present invention is shown;

[0045] Figure 2 A schematic flowchart of another embodiment of the image tampering detection method based on the cross-modal ViT architecture of the present invention is shown. Detailed Implementation

[0046] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0047] This invention provides an image tampering detection method based on a cross-modal ViT architecture, such as... Figure 1 As shown, the image tampering detection method based on the cross-modal ViT architecture in this embodiment of the invention includes the following steps S1 to S3.

[0048] S1, acquire the image to be detected, and input the image to be detected into a pre-established cross-modal dual-stream ViT architecture; the cross-modal dual-stream ViT architecture includes a spatial domain ViT encoder layer, a frequency domain ViT encoder layer, and a deformable cross-attention fusion layer.

[0049] S2, extract the spatial feature data of the image to be detected through the spatial domain ViT encoder layer; extract the frequency domain feature data of the image to be detected through the frequency domain ViT encoder layer.

[0050] S3 uses a deformable cross-attention fusion layer to dynamically gated and fuse spatial and frequency domain feature data to obtain fused features, and then performs image tampering detection based on these fused features.

[0051] The image tampering detection method of this invention addresses the long-distance dependency problem based on the self-attention mechanism of Vision Transformer. Through cross-modal joint analysis in the spatial and frequency domains, and a collaborative analysis and comparative learning mechanism of spatial-frequency features, it achieves high-precision image tampering detection and localization. This provides technical capabilities for applications such as digital forensics and content moderation, improving the level of material authenticity detection. It is particularly suitable for locating digital image tampering operations such as copying, moving, splicing, and erasing.

[0052] The following sections will provide a detailed explanation of each step.

[0053] S1, acquire the image to be detected, and input the image to be detected into a pre-established cross-modal dual-stream ViT architecture. This embodiment of the invention performs image tampering analysis using a cross-modal dual-stream ViT architecture. ViT (Vision Transformer) is a model that applies the Transformer architecture to computer vision tasks.

[0054] The cross-modal dual-stream ViT architecture in this embodiment includes a spatial ViT encoder layer, a frequency-domain ViT encoder layer, and a deformable cross-attention fusion layer.

[0055] I. Spatial Domain ViT Encoder Layer

[0056] The spatial domain ViT encoder layer employs a hierarchical ViT (Swin Transformer) to extract multi-scale spatial features while preserving texture details. Spatial feature analysis involves inputting the RGB pixel values ​​of the original image to be detected, after being divided into blocks, into ViT and extracting local texture and edge features through the hierarchical ViT (Swin Transformer).

[0057] II. Frequency Domain ViT Encoder Layer

[0058] The frequency domain ViT encoder layer employs lightweight ViT processing to handle DCT / FFT frequency domain features, capturing compression artifacts and resampling traces. Frequency domain feature analysis involves extracting the image's frequency domain feature map and then analyzing the image's frequency components. This primarily involves transforming the image's spatial domain to the frequency domain and extracting DCT coefficients, Fourier spectra, etc.

[0059] III. Deformable Cross-Attention Fusion Layer

[0060] Spatial domain feature data (RGB pixel values) and frequency domain feature data (DCT / FFT coefficients) of the same image are treated as two independent feature representation spaces (modalities). Deformable cross-attention fusion dynamically aligns the spatial position offsets of spatial and frequency domain features through dynamic gating. A contrastive learning algorithm is used to enhance the cross-modal consistency of normal regions while amplifying the cross-modal differences between tampered and normal regions.

[0061] S2 extracts the spatial feature data corresponding to the image to be detected through the spatial domain ViT encoder layer; and extracts the frequency domain feature data corresponding to the image to be detected through the frequency domain ViT encoder layer.

[0062] The input to the spatial branch is the RGB pixel values ​​of the original image blocks of the image to be detected. The ViT configuration uses a Swing Transformer to extract local texture and edge features. Extracting the corresponding spatial feature data of the image to be detected through the spatial ViT encoder layer involves dividing the image into blocks and extracting the corresponding spatial feature data of the blocks using the spatial ViT encoder layer. The spatial ViT encoder layer employs a hierarchical visual transformer, ViT.

[0063] The input to the frequency domain branch is either a DCT low-frequency coefficient map or a Fourier amplitude spectrum. Extracting the frequency domain feature data corresponding to the image to be detected through the frequency domain ViT encoder layer involves: dividing the image to be detected into blocks and performing a frequency domain transformation; then extracting the frequency domain feature data corresponding to the transformed image through the frequency domain ViT encoder layer. The frequency domain ViT encoder layer uses the lightweight ViT visual transformer; the frequency domain transformation includes Discrete Cosine Transform (DCT) or Fourier Transform (FFT).

[0064] In an optional embodiment of the present invention, extracting spatial feature data corresponding to the image to be detected through the spatial domain ViT encoder layer may include: dividing the image to be detected into blocks and performing frequency domain transformation, and extracting the spatial feature data corresponding to the image to be detected after frequency domain transformation through the spatial domain ViT encoder layer; the spatial domain ViT encoder layer adopts a lightweight visual transformer ViT; the frequency domain transformation includes discrete cosine transform (DCT) or Fourier transform (FFT).

[0065] Discrete Cosine Transform (DCT): DCT transforms an image from the spatial domain to the frequency domain. First, the image is divided into blocks, such as 8×8 blocks. Then, the DCT operation decomposes the image into cosine components of different frequencies. The upper left 3×3 region is used as the low-frequency component and downsampled to the original value. Figure 1 A 5x8 resolution area is used to capture blurred / smoothed traces in tampered areas; the lower right 5x5 area is used as the high-frequency component to calculate the mean square value of high-frequency coefficients, used to detect edge discontinuities and noise anomalies. When an image has been tampered with, the DCT coefficient distribution corresponding to the tampered area often differs from that of the untampered area. For example, in the case of double JPEG compression, the DCT coefficients in the corresponding area will show anomalies, which can serve as an important clue to determine whether the image has been tampered with and the location of the tampered area.

[0066] Fourier Transform (FFT): This method efficiently implements the Fast Fourier Transform (FFT) to extract the amplitude and phase spectra, utilizing the two-dimensional Discrete Fourier Transform (DFT). The input image to be detected is converted to grayscale, and the Cooley-Tukey algorithm (a concrete implementation of FFT) is used to calculate the amplitude and phase spectra. After logarithmic transformation, the low-frequency strong energy is compressed, while the high-frequency weak energy is amplified, making high-frequency details (such as edge and texture features) easier to observe. The high-frequency energy distribution is relatively uniform in normal image areas, while local energy peaks appear in tampered areas. When an image undergoes tampering operations such as copying or moving, periodic pattern features are usually introduced in the frequency domain, resulting in symmetrical peak pairs in the amplitude spectrum. When an image undergoes splicing-type tampering operations, the amplitude spectrum in normal areas is isotropic, while directional stripes appear in spliced ​​areas.

[0067] After phase spectrum data extraction, phase consistency and symmetry analysis are performed. Regarding consistency, the edges, corners, and other structures in the normal image region correspond to consistent peaks in the phase spectrum; splicing or tampering with the region will disrupt the continuity of these peaks. Regarding symmetry, the phase spectrum of the normal image region exhibits symmetry, while copying, moving, or tampering with the region may disrupt this symmetry (e.g., through rotation or scaling).

[0068] S3 uses a deformable cross-attention fusion layer to dynamically gated and fuse spatial and frequency domain feature data to obtain fused features, and then performs image tampering detection based on these fused features.

[0069] In an optional embodiment of the present invention, step S3 above, which performs dynamic gating fusion of spatial domain feature data and frequency domain feature data through a deformable cross-attention fusion layer, may include: inputting spatial domain feature data as a Query and frequency domain feature data as a Key and Value into the deformable cross-attention fusion layer; the deformable cross-attention fusion layer dynamically adjusts the attention weights through dynamic gating, so that the spatial domain feature data can be aligned with the relevant parts of the frequency domain feature data at different positions, thereby outputting fused feature data; the fused feature data includes a set token, which can be "[CLS]token".

[0070] In other words, the spatial feature data (multi-scale features) output by the spatial ViT encoder layer is used as the Query, and the frequency feature data (feature map) output by the frequency ViT encoder layer is used as the Key and Value. These are then input into the deformable cross-attention fusion layer, which performs dynamic gating fusion, calculates cross-domain correlation, and adjusts the attention weights by dynamic gating.

[0071] For example, when processing an image containing a person and a background, the deformable cross-attention fusion layer dynamically adjusts the attention weights based on the person's contour in the spatial and frequency domains. This ensures precise alignment between local textures and other features of the person in the spatial domain and corresponding low-frequency features such as the contour in the frequency domain, achieving effective cross-modal association. The dynamic gating automatically learns a set of weight matrices. These weight matrices dynamically assign different weights to features of each modality based on the specific input image and the extracted spatial and frequency domain features. Then, the spatial and frequency domain features are fused according to these weights to generate a fused feature representation.

[0072] The fusion feature data output by the deformable cross-attention fusion layer includes information such as the local and overall structure and texture of the image reflected in the spatial domain, and the frequency distribution and low-frequency contours reflected in the frequency domain.

[0073] The cross-modal contrastive learning enhancement mechanism of this invention treats the spatial domain (RGB pixels) and frequency domain (DCT / FFT features) of the same image as two independent modalities. Through contrastive learning, it brings the cross-modal features (positive sample pairs) of normal regions at the same location closer together and pushes the cross-modal features (negative sample pairs) of tampered regions further apart from normal regions. Bringing positive samples closer means that positive samples formed by the spatial and frequency domain features of the same untampered region should be matched; pushing them further apart means that negative samples formed by the spatial domain of the tampered region and the frequency domain of the normal region (or vice versa) should not be matched.

[0074] Specifically, the deformable cross-attention fusion layer dynamically adjusts the attention weights through dynamic gating, including: the deformable cross-attention fusion layer forces the spatial and frequency domain features of the normal region to align through the cross-modal InfoNCE loss function, while amplifying the feature differences of the tampered region; combined with dynamic gating weighted optimization.

[0075] In this embodiment, cross-modal contrastive learning, through the design of a cross-modal InfoNCE loss function, forces the spatial domain (RGB pixels) and frequency domain (DCT / FFT features) features of normal regions to align, while simultaneously amplifying the feature differences of tampered regions. Combined with dynamic gating weighted optimization, the weighted features are used when calculating the similarity of sample pairs, making the alignment of normal regions more accurate and the differences of tampered regions more significant. The mathematical formula is as follows:

[0076]

[0077] in, and These are the spatial and frequency domain characteristics of a weighted normal region at the same location. It is the weighted frequency domain feature of other locations, sim(·) is the cosine similarity, τ is the temperature parameter, which controls the distribution smoothness, and L Gate-InfoNCEThe value represents the cross-modal contrast loss after dynamic gating weighting. N represents the total number of positive sample pairs in the normal region, which represents the total number of effective samples that the model needs to strengthen alignment. M represents the total number of negative samples, which represents the total number of interference samples that the model needs to distinguish. The detection robustness is improved by increasing the difference between positive and negative samples.

[0078] Quantifying the modal alignment of normal regions: By minimizing the loss, the spatial and frequency features of normal regions are forced to cluster tightly in the embedding space (maximizing similarity); quantifying the modal differences of tampered regions: By maximizing the negative sample terms in the loss, the spatial and frequency features of tampered regions are forced to deviate from the normal distribution (minimizing similarity); the guiding role of dynamic weights: Combined with a dynamic gating mechanism, the loss calculation focuses more on "effective modalities" (such as the spatial texture of normal regions and the frequency anomalies of tampered regions), improving the targeting of contrastive learning. The smaller the loss value, the better the model's cross-modal alignment of normal regions and the clearer the distinction of cross-modal differences in tampered regions, resulting in better model performance.

[0079] Normal Region Alignment: For unaltered regions, spatial texture (e.g., skin texture) and frequency energy distribution (e.g., DCT low-frequency coefficients) should be highly consistent. Through InfoNCE loss, the model learns to maximize the similarity of these feature pairs. For example, the cosine similarity between an RGB block and its corresponding DCT low-frequency block in the embedding space should be close to 1. Altered Region Difference Expansion: Alteration operations (e.g., splicing, copying, moving) can lead to contradictions between spatial and frequency domain features. For example, the spatial edges of a spliced ​​boundary may be blurred (smoothed features), but the high-frequency energy in the frequency domain may be abnormally increased (due to resampling or JPEG compression). By treating the frequency domain features of the altered region as negative samples, the model learns to minimize their similarity to normal spatial features.

[0080] In an optional embodiment of the present invention, the cross-modal dual-stream ViT architecture further includes a multi-task joint decision layer, which includes a global classification head and a pixel classification head; image tampering detection based on fused features includes: the global classification head effectively fusing features associated with a set marker through a fusion strategy, and then using the fused feature information to output a determination result of whether the image has been tampered with or not through a preset classification algorithm; the pixel localization head generates a pixel-level tampering heatmap through a feature pyramid decoder to show the tampering areas that may exist in the image to be detected.

[0081] The global classification head primarily uses the [CLS] token to determine the overall authenticity of an image. Specifically, after initial feature extraction and other processing, an image generates a feature representation that includes the [CLS] token. This global classification head effectively fuses the features associated with the [CLS] token using a specific fusion strategy. Then, using the fused feature information and a pre-defined classification algorithm (such as a deep learning-based classifier trained on a large number of samples to distinguish between tampered and untampered image features), it outputs a determination of whether the image has been tampered with or not, thus achieving overall image authenticity assessment.

[0082] The structure of the global classification head is: MLP (Linear + Activation + Linear) + Sigmoid. MLP is a multilayer perceptron mechanism, and Sigmoid represents binary classification, i.e., true or false.

[0083] Input: Feature vector of [CLS] token (dimension is usually d_model, such as 768 or 1024).

[0084] Output: A probability value (0-1) representing the likelihood that the image has been tampered with (binary classification).

[0085] Example:

[0086] Input a 224x224 image:

[0087] 1. After processing by the ViT encoder and cross-attention fusion, the [CLS] token is obtained.

[0088] 2. Feed the output vector of [CLS]token into the global classification head.

[0089] 3. Classification Head Structure: Linear(768->256)->ReLU->Linear(256->1)->Sigmoid

[0090] 4. The output result, such as 0.87, indicates that there is an 87% probability that the image has been tampered with.

[0091] A pixel-level tampering heatmap is used to generate a pixel-level tampering heatmap via a feature pyramid decoder, enabling precise location of potentially tampered areas in an image. After acquiring appropriately processed image features (which carry information at different levels and scales), the feature pyramid decoder, based on its multi-level structure and feature fusion mechanism, progressively decodes and upsamples the input features, continuously integrating feature information from each level to ultimately generate a pixel-level tampering heatmap. In this heatmap, the value of each pixel represents the probability of tampering at that location in the image; for example, pixels with higher values ​​have a greater probability of being tampered with, thus achieving precise location of tampered areas. Constructing a pixel-level tampering heatmap using a feature pyramid decoder fully utilizes multi-scale image features, effectively capturing tampered areas of different sizes and locations. This overcomes the inaccuracy of traditional location methods for small or complex-shaped tampered areas, providing high-precision tampering location information for subsequent detailed analysis and practical applications (such as business data authenticity verification, image copyright protection, and forensic evidence identification).

[0092] Pixel positioning head structure: FPN decoder + 1x1 Conv + Sigmoid.

[0093] Input: Multi-scale features of the ViT encoder, typically represented using a Feature Pyramid Network (FPN).

[0094] Output: A heatmap (also called a binary image) with the same size as the input image, where each pixel value represents the probability that the location has been tampered with.

[0095] Example:

[0096] 1. FPN Decoder: Deep features -> Upsampling -> Fusion with mid-level features -> Convolution -> Final upsampling to 224x224.

[0097] 2.1x1 Conv+Sigmoid: Reduces the number of channels to 1 using 1x1 Conv and outputs the tampering probability (0~1) of each pixel using Sigmoid.

[0098] 3. Output heatmap: A 224x224x1 heatmap, in which tampered areas (such as copied or moved objects) are highlighted.

[0099] In an optional embodiment of the present invention, the multi-task joint decision layer further includes a frequency domain anomaly auxiliary head for predicting the anomaly score of the DCT coefficient distribution, thereby enhancing the interpretability of the model.

[0100] Figure 2 A schematic flowchart of another embodiment of the image tampering detection method based on the cross-modal ViT architecture of the present invention is shown, as follows: Figure 2 As shown, the complete image tampering detection process in this embodiment is as follows:

[0101] 1. Input Processing. The spatial domain analysis branch segments the RGB image into 16×16 pixel blocks and linearly embeds them into 384-dimensional vectors. The frequency domain analysis branch calculates 8×8 block DCT coefficients, extracts low-frequency components to construct 64×64 feature maps, and embeds them into 192-dimensional vectors to represent spatial information such as local texture and edges of the image.

[0102] 2. Feature Extraction. Spatial domain ViT uses a four-stage Swing Transformer to output 14×14 resolution multi-scale features, while frequency domain ViT uses a two-layer standard Transformer to output 14×14 resolution feature maps; low-frequency components: overall image contour and smooth regions; high-frequency components: detail features and noise components.

[0103] 3. Cross-modal fusion. Through a deformable cross-attention mechanism, the attention weights are dynamically adjusted according to the actual situation of the image content, so that spatial features can be accurately aligned with the relevant parts of frequency domain features at different locations. The dynamic gating feature fusion mechanism intelligently adjusts the proportion of different modal features in the fusion process, providing comprehensive and accurate feature input for subsequent output decisions.

[0104] The multi-scale features output from the spatial domain ViT are used as the Query (providing spatial detail guidance signals), and the feature maps output from the frequency domain ViT are used as the Key / Value (carrying frequency domain structural information), both input into the deformable cross-attention fusion layer. For example, when processing an image containing a person and a background, the deformable cross-attention fusion layer dynamically adjusts the attention weights based on the appearance of the person's contour in the spatial and frequency domains, ensuring accurate alignment between the local texture and other features of the person in the spatial domain and the corresponding low-frequency features such as the contour in the frequency domain, achieving effective cross-modal association. The features processed by the deformable cross-attention fusion layer then enter the dynamic gating feature fusion part. The system learns an appropriate weight matrix based on the specific characteristics of the spatial and frequency domain features in the image. For example, for the main body of the person in the image, it may focus more on the texture details of the spatial domain features, while for the background, it may focus more on fusing the low-frequency structural information of the frequency domain features, generating a comprehensive feature representation through weighted fusion.

[0105] 4. Cross-modal fusion. Deformable attention: Spatial domain figure contour edge features (Query) detect high-frequency energy anomalies (Key / Value) at corresponding locations in the frequency domain, dynamically reducing alignment weights (weight values ​​from 0.8 to 0.3). Dynamic gating fusion: Learns spatial domain edge feature weights W. s = [0.9, 0.1, ...] and frequency domain energy feature weight W f= [0.1, 0.9, ...], the fused features highlight the differences (e.g., the high-frequency dimension value changes from 0.2 to 0.7).

[0106] Output decision. The classification head takes the cross-modal fusion output features as input and outputs the image tampering probability value through a fully connected layer. The localization head decoder generates a pixel-level mask to show the tampered area.

[0107] The following explanation uses a 256×256 tampered image as an example.

[0108] 1. Input Processing. The spatial domain analysis branch segments the RGB image into 16×16 pixel blocks and linearly embeds them into 384-dimensional vectors. The frequency domain analysis branch calculates 8×8 block DCT coefficients, extracts low-frequency components to construct 64×64 feature maps, and embeds them into 192-dimensional vectors.

[0109] 2. Feature Extraction. The spatial domain ViT uses a 4-stage Swing Transformer to output multi-scale features with a resolution of 14×14, while the frequency domain ViT uses a 2-layer standard Transformer to output a feature map with a resolution of 14×14.

[0110] 3. Cross-modal fusion.

[0111] Input features:

[0112] Spatial ViT output: 14×14 resolution feature map (384 dimensions), corresponding to the global semantics of a 16×16 pixel block in the original image (such as the texture of the edge of a person);

[0113] Frequency domain ViT output: 14×14 resolution feature map (dimension 192), corresponding to 8×8 blocks of DCT low-frequency components (such as the frequency domain energy distribution of the contour region).

[0114] 1) Deformable cross-attention mechanism:

[0115] a. Feature reshaping and Query / Key / Value definition:

[0116] Spatial features as a query: reshape 14×14×384 to 196×384, with each element representing the semantics of a 16×16 block in the original image (e.g., "eye region edge").

[0117] Frequency domain features as Key / Value: Reshape 14×14×192 to 196×192, with each element representing the energy feature of the corresponding frequency domain block (e.g., "high-frequency energy missing").

[0118] b. Dynamic position alignment process:

[0119] Case scenario: The left eye area of ​​a person in the original image has been altered (stitched from another image). Its spatial domain feature is "blue pupil edge", but its frequency domain feature shows "high-frequency energy smoothing" due to resampling.

[0120] Attention weight calculation: The spatial domain query (pupil edge) is automatically focused on the corresponding position in the frequency domain (theoretically corresponding to the high-frequency energy region), but the energy value of the frequency domain key at this point is abnormally low (weight is reduced), indicating cross-modal inconsistency; for the tampered region, the attention mechanism dynamically adjusts the weights so that the non-corresponding positions of the spatial domain features and frequency domain features have high weights (such as the spatial domain features of the pupil edge being incorrectly aligned to the low-frequency region of the frequency domain).

[0121] c. Output results: Generate aligned features of 196×384, where the cross-modal correlation of the tampered region is significantly lower than that of the normal region (e.g., the correlation value of the pupil edge is 0.3, while that of the normal region is 0.8).

[0122] 2) Gating feature fusion mechanism:

[0123] a. Weight matrix learning:

[0124] Input: Aligned spatial domain features (196×384) and frequency domain features (196×192);

[0125] Learning process: A weight matrix W is generated through a two-layer MLP. s ∈R384 and W f ∈R192, where: W s Weighting of spatial features across dimensions (e.g., enhancing the weights of edge feature dimensions); W f Weighting of each dimension of frequency domain features (e.g., increasing the weight of the high-frequency energy anomaly dimension).

[0126] b. Weighted fusion calculation: for the feature vector of the tampered region (e.g., pupil): The spatial domain feature vector is "blue pupil edge" (values ​​[0.7, 0.2, ...]), and the frequency domain feature vector is "high-frequency smoothing" (values ​​[0.1, 0.8, ...]). The weighted fusion features highlight the differences between the two (e.g., the high-frequency correlation dimension in the vector is 0.9).

[0127] c. Output: Generate a 196×384 fusion feature map, where each element contains weighted information of spatial semantics and frequency energy (e.g., in the fusion features of tampered regions, the cross-modal difference dimension is enhanced).

[0128] 4. Output Decision. The classification head takes the cross-modal fusion output features as input and outputs the image tampering probability value through a fully connected layer. The localization head decoder generates a pixel-level mask to show the tampered area.

[0129] Based on the same inventive concept, embodiments of the present invention also provide an image tampering detection device based on a cross-modal ViT architecture, the system comprising:

[0130] The image acquisition module is used to acquire the image to be detected and input the image to be detected into a pre-established cross-modal dual-stream ViT architecture; the cross-modal dual-stream ViT architecture includes a spatial domain ViT encoder layer, a frequency domain ViT encoder layer, and a cross-attention fusion layer;

[0131] The cross-modal dual-stream ViT architecture extracts spatial feature data corresponding to the image to be detected through a spatial ViT encoder layer; extracts frequency feature data corresponding to the image to be detected through a frequency ViT encoder layer; and fuses the spatial and frequency feature data through a cross-attention fusion layer to obtain fused features, which are then used for image tampering detection.

[0132] In an optional embodiment of the present invention, the cross-modal dual-stream ViT architecture is used to input spatial domain feature data as a query and frequency domain feature data as a key and value into a deformable cross-attention fusion layer; the deformable cross-attention fusion layer dynamically adjusts the attention weights through dynamic gating, so that the spatial domain feature data can be aligned with the relevant parts of the frequency domain feature data at different positions, thereby outputting fused feature data; the fused feature data includes a set marker.

[0133] In an optional embodiment of the present invention, the deformable cross-attention fusion layer of the cross-modal dual-stream ViT architecture forces the spatial and frequency domain features of the normal region to align through the cross-modal InfoNCE loss function, while amplifying the feature differences of the tampered region; combined with dynamic gating weighted optimization, the mathematical formula is as follows:

[0134]

[0135] in, and These are the spatial and frequency domain characteristics of a weighted normal region at the same location. It is the weighted frequency domain feature of other locations, sim(·) is the cosine similarity, τ is the temperature parameter, which controls the distribution smoothness, and L Gate-InfoNCE The value represents the cross-modal contrast loss after dynamic gating weighting. N represents the total number of positive sample pairs in the normal region, which represents the total number of effective samples that the model needs to strengthen alignment. M represents the total number of negative samples, which represents the total number of interference samples that the model needs to distinguish. The detection robustness is improved by increasing the difference between positive and negative samples.

[0136] In an optional embodiment of the present invention, the cross-modal dual-stream ViT architecture further includes a multi-task joint decision layer, which includes a global classification head and a pixel classification head.

[0137] The global classification head effectively fuses features associated with a set marker through a fusion strategy, and then uses the fused feature information to output a judgment result of whether the image has been tampered with or not through a preset classification algorithm; the pixel localization head generates a pixel-level tampering heatmap through a feature pyramid decoder to show the possible tampering areas in the image to be detected.

[0138] In an optional embodiment of the present invention, the multi-task joint decision layer further includes a frequency domain anomaly auxiliary head for predicting the anomaly score of the DCT coefficient distribution, thereby enhancing the interpretability of the model.

[0139] In an optional embodiment of the present invention, the cross-modal dual-stream ViT architecture is further used to divide the image to be detected into blocks, and extract the corresponding spatial feature data of the block-based image to be detected through the spatial ViT encoder layer; the spatial ViT encoder layer adopts a hierarchical visual transformer ViT.

[0140] In an optional embodiment of the present invention, the cross-modal dual-stream ViT architecture is further used to divide the image to be detected into blocks and perform frequency domain transformation, and extract the frequency domain feature data corresponding to the image to be detected after frequency domain transformation through the frequency domain ViT encoder layer; the frequency domain ViT encoder layer adopts a lightweight visual transformer ViT; the frequency domain transformation includes discrete cosine transform DCT or Fourier transform FFT.

[0141] This invention also provides a computer-readable storage medium for storing program code, which, when executed by a processor, is used to implement the image tampering detection method based on the cross-modal ViT architecture described above.

[0142] This invention also provides a computing device, which includes a processor and a memory: the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the image tampering detection method based on the cross-modal ViT architecture described above according to the instructions in the program code.

[0143] Those skilled in the art will clearly understand that the specific working process of the systems, devices, modules and units described above can be referred to the corresponding process in the foregoing method embodiments. For the sake of brevity, it will not be repeated here.

[0144] Furthermore, the functional units in the various embodiments of the present invention can be physically independent of each other, or two or more functional units can be integrated together, or all functional units can be integrated into one processing unit. The integrated functional units described above can be implemented in hardware, or in software or firmware.

[0145] Those skilled in the art will understand that integrated functional units, if implemented in software and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or all or part of it, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computing device (e.g., a personal computer, server, or network device) to execute all or part of the steps of the methods of the various embodiments of this invention when running the instructions. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] Alternatively, all or part of the steps of the foregoing method embodiments can be implemented by hardware (such as a computing device, personal computer, server, or network device) related to the program instructions. The program instructions can be stored in a computer-readable storage medium. When the program instructions are executed by the processor of the computing device, the computing device executes all or part of the steps of the methods of the various embodiments of the present invention.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that within the spirit and principles of the present invention, modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the corresponding technical solutions to depart from the protection scope of the present invention.

Claims

1. An image tampering detection method based on a cross-modal ViT architecture, characterized in that, include: Acquire the image to be detected and input the image to be detected into a pre-established cross-modal dual-stream ViT architecture; The cross-modal dual-stream ViT architecture includes a spatial ViT encoder layer, a frequency ViT encoder layer, and a deformable cross-attention fusion layer. The spatial feature data corresponding to the image to be detected is extracted through the spatial ViT encoder layer. The frequency domain feature data corresponding to the image to be detected is extracted through the frequency domain ViT encoder layer; The spatial domain feature data and frequency domain feature data are dynamically gated and fused through the deformable cross-attention fusion layer to obtain fused features, and image tampering detection is performed based on the fused features.

2. The method according to claim 1, characterized in that, The dynamic gating fusion of the spatial domain feature data and the frequency domain feature data through the deformable cross-attention fusion layer includes: Spatial domain feature data is used as the Query, and frequency domain feature data is used as the Key and Value, which are then input into the deformable cross-attention fusion layer. The deformable cross-attention fusion layer dynamically adjusts the attention weights through dynamic gating, enabling spatial feature data to be aligned with the relevant parts of frequency domain feature data at different locations, thereby outputting fused feature data; the fused feature data includes a set label.

3. The method according to claim 2, characterized in that, The deformable cross-attention fusion layer dynamically adjusts the attention weights via dynamic gating, including: The deformable cross-attention fusion layer forces the spatial and frequency domain features of the normal region to align through the cross-modal InfoNCE loss function, while amplifying the feature differences in the tampered region. Combining dynamic gating weighted optimization, the mathematical formula is as follows: in,, and These are the spatial and frequency domain characteristics of a weighted normal region at the same location. It is the weighted frequency domain feature of other locations, sim(·) is the cosine similarity, τ is the temperature parameter, which controls the distribution smoothness, and L Gate-InfoNCE denoted as the cross-modal contrastive loss value after dynamic gating weighting, N represents the total number of positive sample pairs in the normal region, representing the total number of effective samples that the model needs to strengthen alignment, and M represents the total number of negative samples, representing the total number of interference samples that the model needs to distinguish.

4. The method according to claim 2, characterized in that, The cross-modal dual-stream ViT architecture further includes a multi-task joint decision layer, which comprises a global classification head and a pixel classification head; image tampering detection based on the fused features includes: The global classification head effectively fuses features associated with a set label through a fusion strategy, and then uses the fused feature information to output a judgment result on whether the image has been tampered with or not through a preset classification algorithm. The pixel localization head generates a pixel-level tampering heatmap through a feature pyramid decoder to show the tampered areas that may exist in the image to be detected.

5. The method according to claim 2, characterized in that, The multi-task joint decision layer also includes a frequency domain anomaly auxiliary head, which is used to predict the anomaly score of the DCT coefficient distribution and enhance the interpretability of the model.

6. The method according to any one of claims 1-5, characterized in that, Extracting spatial feature data corresponding to the image to be detected through the spatial ViT encoder layer includes: The image to be detected is divided into blocks, and the spatial feature data corresponding to the block image to be detected is extracted by the spatial domain ViT encoder layer. The spatial domain ViT encoder layer employs a hierarchical vision transformer ViT.

7. The method according to any one of claims 1-5, characterized in that, Extracting frequency domain feature data corresponding to the image to be detected through the frequency domain ViT encoder layer includes: The image to be detected is divided into blocks and frequency domain transformation is performed. The frequency domain feature data corresponding to the image to be detected after frequency domain transformation is extracted through the frequency domain ViT encoder layer. The frequency domain ViT encoder layer employs a lightweight vision transformer ViT; the frequency domain transformation includes Discrete Cosine Transform (DCT) or Fourier Transform (FFT).

8. An image tampering detection device based on a cross-modal ViT architecture, characterized in that, include: The image acquisition module is used to acquire the image to be detected and input the image to be detected into a pre-established cross-modal dual-stream ViT architecture; The cross-modal dual-stream ViT architecture includes a spatial ViT encoder layer, a frequency ViT encoder layer, and a cross-attention fusion layer. A cross-modal dual-stream ViT architecture extracts spatial feature data corresponding to the image to be detected through the spatial ViT encoder layer. The frequency domain feature data corresponding to the image to be detected is extracted through the frequency domain ViT encoder layer; The spatial domain feature data and frequency domain feature data are fused by a cross-attention fusion layer to obtain fused features, and image tampering detection is performed based on the fused features.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code, which, when executed by a processor, is used to implement the image tampering detection method based on the cross-modal ViT architecture as described in any one of claims 1-7.

10. A computing device, characterized in that, The computing device includes a processor and memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute, according to the instructions in the program code, the image tampering detection method based on the cross-modal ViT architecture as described in any one of claims 1-7.

Images