A dual-stream image tampering detection method based on transformer and dynamic fusion

Abstract

The application discloses a double-flow image tampering detection method based on a Transformer and dynamic fusion, relates to the fields of computer vision and information security, and specifically comprises the following steps: obtaining a public tampered image dataset, labeling each tampered image in the tampered image dataset, obtaining tampering type labels and tampering region labels corresponding to each tampered image, and further preprocessing the tampered image dataset; training a double-flow deep feature fusion network model by using the preprocessed tampered image dataset; and testing a to-be-tested image by using the trained double-flow deep feature fusion network model to obtain a final tampering detection result. The method can more accurately capture details and abnormal changes in an image, and improves the positioning accuracy and detection accuracy under complex scenes and subtle tampering traces.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and information security, specifically to image tampering detection and digital image content verification. Background Technology

[0002] With the rapid development of image editing technology and the widespread availability of user-friendly software, image tampering has become easily accessible and low-cost, enabling precise and covert modification of images to achieve the purposes of forging evidence, concealing facts, and spreading false information. The authenticity and integrity of images are crucial for information security, digital forensics, and media verification; therefore, effectively detecting tampered areas in images has become an urgent problem to be solved in the field of image processing.

[0003] Currently, image tampering detection techniques are mainly divided into traditional method-based detection techniques and deep learning-based detection techniques. Traditional image tampering detection methods rely on low-level image features, analyzing discontinuities, color changes, or texture differences to locate potential tampering areas. However, traditional methods only perform feature analysis in the spatial domain, lacking sensitivity to frequency domain information, making it difficult to capture subtle changes. Furthermore, traditional methods depend on manually designed features, making them ill-suited for diverse tampering patterns and unable to accurately detect subtle adjustments and highly concealed tampering.

[0004] Deep learning-based image tampering detection methods reduce reliance on hand-crafted features by automatically learning image features. However, existing deep learning-based image tampering detection methods are limited by their receptive field when capturing long-distance dependencies and cross-regional correlations, resulting in insufficient ability to capture global information and difficulty in handling complex tampering in multiple regions. Furthermore, deep learning-based detection techniques cannot effectively utilize hidden frequency domain features in tampered regions, making it difficult to accurately detect complex and highly concealed tampered images. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a dual-stream image tampering detection method based on Transformer and dynamic fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A dual-stream image tampering detection method based on Transformer and dynamic fusion is proposed, which performs tampering detection on the input image according to steps S1 to S3 to obtain the tampering detection result of the input image:

[0008] Step S1: Obtain the publicly available tampered image dataset, and label each tampered image in the tampered image dataset to obtain the tampering type label and tampering region label corresponding to each tampered image. Further preprocess the tampered image dataset by horizontal flipping and color perturbation.

[0009] Step S2: Train the dual-stream deep feature fusion network model using the preprocessed tampered image dataset. The dual-stream deep feature fusion network includes a deep spatial feature extraction module, a deep spectral feature extraction module, a feature fusion module, and an output module.

[0010] Step S3: The trained dual-stream deep feature fusion network model is used to test the image under test to obtain the final tamper detection result.

[0011] Furthermore, the input terminals of the deep spatial feature extraction module and the deep spectral feature extraction module constitute the input terminal of the dual-stream deep feature fusion network; the input terminal of the feature fusion module is connected to the output terminals of the deep spatial feature extraction module and the deep spectral feature extraction module; the input terminal of the output module is connected to the output terminal of the feature fusion module; and the output terminal of the output module constitutes the output terminal of the dual-stream deep feature fusion network.

[0012] The deep spatial feature extraction module is used to extract and output the global spatial features of the input image; the deep spectral feature extraction module is used to extract and output the depth frequency domain features of the input image; the feature fusion module is used to obtain the fused features of the input image based on the global spatial features and the depth frequency domain features; the output module is used to generate the classification result of the image tampering type and the probability mask map of the tampering region based on the fused features of the input image.

[0013] Furthermore, the deep spatial feature extraction module includes a convolution module and an extended spatial feature extraction module, and the extended spatial feature extraction module includes an embedding vector module, a position encoding module, and a Transformer encoder module;

[0014] The input of the convolution module constitutes the input of the deep spatial feature extraction module, and the output of the convolution module is connected to the input of the extended spatial feature extraction module; the input of the embedding vector module constitutes the input of the extended spatial feature extraction module, the input of the position encoding module is connected to the output of the embedding vector module, the input of the Transformer encoder module is connected to the output of the position encoding module, and the output of the Transformer encoder constitutes the output of the extended spatial feature extraction module.

[0015] The input image is received by the convolutional module, which extracts local spatial features of the input image using a trained ResNet50 and outputs these local spatial features to the embedding vector module. The embedding vector module divides the local spatial features into segments based on a preset size using patching, obtains the embedding vectors for each segment using linear projection, and outputs the embedding vectors for each segment to the position encoding module. The position encoding module generates position codes for each segment using sine and cosine functions, and outputs the embedding vectors and position codes for each segment to the Transformer encoder. The Transformer encoder extracts and outputs global spatial features through multi-head self-attention and a feedforward network.

[0016] Furthermore, the deep spectral feature extraction module includes a discrete cosine transform module, an input feature embedding module, an adaptive multi-head attention module, and a feedforward network module;

[0017] The input of the discrete cosine transform module constitutes the input of the deep spectrum feature extraction module. The input of the input feature embedding module is connected to the output of the discrete cosine transform module. The input of the adaptive multi-head attention module is connected to the output of the input feature embedding module. The input of the feedforward network module is connected to the output of the adaptive multi-head attention module. The output of the feedforward network module constitutes the output of the deep spectrum feature extraction module.

[0018] The input image is received by the Discrete Cosine Transform (DCT) module, which extracts the local frequency domain features of the input image and outputs these features to the input feature embedding module. The input feature embedding module then uses a flattening method to transform the local frequency domain features into vector form and outputs them to the adaptive multi-head attention module. The adaptive multi-head attention module uses an adaptive multi-head attention mechanism and a multi-head attention concatenation operation to extract the global frequency domain features and outputs them to the feedforward network module. Finally, the feedforward network module uses nonlinear activation processing to extract and output the depth frequency domain features.

[0019] Furthermore, the feature fusion module dynamically generates feature fusion weights based on the deep spatial features and depth frequency domain features of the input image through global average pooling and a two-layer perceptron, and obtains the fused features of the input image by weighted merging.

[0020] Furthermore, the output module includes a classification module and a mask generation module; wherein, the classification module is used to process the fusion features of the input image using a fully connected layer to generate a classification result of the image tampering type; the mask generation module is used to upsample the fusion features of the input image and generate a probability mask map of the tampered region through a convolutional decoder.

[0021] Furthermore, during model training of the dual-stream deep feature fusion network, joint optimization is performed using a summative function that includes a contrastive learning loss function, a cross-entropy loss function, and a binary mask loss function. The contrastive learning loss function, cross-entropy loss function, binary mask loss function, and summative function are shown below:

[0022]

[0023] in, To compare the loss functions, N is the total number of sample pairs, that is, the total number of positive and negative sample pairs generated during training, f i Let f represent the features of the i-th sample. i + To be with f i Similar positive samples, i.e., tampered regions in the same image, f i - To be with f i Dissimilar negative samples, i.e., unaltered regions in the image, d(f a ,f b ) represents the eigenvector f a and f b Distance metric between them, t i This represents the label of a positive sample, where m is a preset distance threshold.

[0024]

[0025] in, Let y be the cross-entropy loss function, where C represents the number of classes, and y is the cross-entropy loss function. c It's a category label. It is the probability predicted by the model;

[0026]

[0027] in, Let M be the binary mask loss function, and M be the binary mask of the input image. This is the probability mask for the model's predictions.

[0028]

[0029] in, For the total function, To compare loss functions, Let cross-entropy be the loss function. Let be the binary mask loss function, α be the weighting coefficient of the contrastive learning loss, β be the weighting coefficient of the cross-entropy loss, and γ be the weighting coefficient of the binary mask loss.

[0030] The beneficial effects of adopting the above technical solution are as follows:

[0031] (1) The present invention uses the CNN+Transformer architecture to obtain global context information and combines spatial domain and frequency domain features to improve the depth of feature extraction and global perception capability. It can capture details and abnormal changes in images more accurately, improve the accuracy of detection and the precision of localization, and solve the problem that the existing technology relies only on spatial domain feature extraction and cannot effectively capture subtle changes in the frequency domain.

[0032] (2) The present invention adopts an adaptive multi-head attention mechanism and a dynamic feature fusion mechanism to capture global and local information of the image, thereby effectively improving the accuracy and robustness of image tampering detection and solving the problem of poor detection accuracy and reliability when detecting complex scenes and subtle tampering traces in the prior art. Attached Figure Description

[0033] Figure 1 This is a flowchart of the present invention;

[0034] Figure 2 This is the network architecture of the dual-stream deep feature fusion network of the present invention;

[0035] Figure 3 This is a diagram of the feature fusion module architecture of the present invention;

[0036] Figure 4 This is a diagram of the output module architecture of the present invention;

[0037] Figure 5 The graph shows the comparison results of application examples of this invention. Detailed Implementation

[0038] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0039] refer to Figure 1 and Figure 2 A dual-stream image tampering detection method based on Transformer and dynamic fusion is proposed, which performs tampering detection on the input image according to steps S1 to S3 to obtain the tampering detection result of the input image:

[0040] Step S1: Obtain the publicly available tampered image dataset, and label each tampered image in the tampered image dataset to obtain the tampering type label and tampering region label corresponding to each tampered image, and further preprocess the tampered image dataset;

[0041] Step S2: Train the dual-stream deep feature fusion network model using the preprocessed tampered image dataset. The dual-stream deep feature fusion network includes a deep spatial feature extraction module, a deep spectral feature extraction module, a feature fusion module, and an output module.

[0042] Furthermore, the input terminals of the deep spatial feature extraction module and the deep spectral feature extraction module constitute the input terminal of the dual-stream deep feature fusion network; the input terminal of the feature fusion module is connected to the output terminals of the deep spatial feature extraction module and the deep spectral feature extraction module; the input terminal of the output module is connected to the output terminal of the feature fusion module; the output terminal of the output module constitutes the output terminal of the dual-stream deep feature fusion network; specifically, the deep spatial feature extraction module is used to extract and output the global spatial features of the input image; the deep spectral feature extraction module is used to extract and output the depth frequency domain features of the input image; the feature fusion module is used to obtain the fused features of the input image based on the global spatial features and the depth frequency domain features; the output module is used to generate a classification result of the image tampering type and a probability mask map of the tampered region based on the fused features of the input image.

[0043] Furthermore, the deep spatial feature extraction module includes a convolution module and an extended spatial feature extraction module, and the extended spatial feature extraction module includes an embedding vector module, a position encoding module, and a Transformer encoder module;

[0044] The input of the convolution module constitutes the input of the deep spatial feature extraction module, and the output of the convolution module is connected to the input of the extended spatial feature extraction module; the input of the embedding vector module constitutes the input of the extended spatial feature extraction module, the input of the position encoding module is connected to the output of the embedding vector module, the input of the Transformer encoder module is connected to the output of the position encoding module, and the output of the Transformer encoder constitutes the output of the extended spatial feature extraction module.

[0045] Specifically, the convolutional module receives the input image, extracts local spatial features of the input image using a trained ResNet50, and outputs the local spatial features to the embedding vector module. The embedding vector module divides the local spatial features into segments based on a preset size using patching, obtains the embedding vectors corresponding to each block using linear projection, and outputs the embedding vectors corresponding to each block to the position encoding module. The position encoding module generates the position codes corresponding to each block using sine and cosine functions, and outputs the embedding vectors and position codes corresponding to each block to the Transformer encoder. The Transformer encoder extracts and outputs global spatial features through multi-head self-attention and a feedforward network.

[0046] Furthermore, the deep spectral feature extraction module includes a discrete cosine transform module, an input feature embedding module, an adaptive multi-head attention module, and a feedforward network module;

[0047] The input of the discrete cosine transform module constitutes the input of the deep spectrum feature extraction module. The input of the input feature embedding module is connected to the output of the discrete cosine transform module. The input of the adaptive multi-head attention module is connected to the output of the input feature embedding module. The input of the feedforward network module is connected to the output of the adaptive multi-head attention module. The output of the feedforward network module constitutes the output of the deep spectrum feature extraction module.

[0048] Specifically, the discrete cosine transform module receives the input image, extracts the local frequency domain features of the input image, and outputs the local frequency domain features to the input feature embedding module; the input feature embedding module uses a flattening method to transform the local frequency domain features into vector form and outputs them to the adaptive multi-head attention module; the adaptive multi-head attention module uses an adaptive multi-head attention mechanism and multi-head attention concatenation operation to extract global frequency domain features and outputs them to the feedforward network module; the feedforward network module uses nonlinear activation processing to extract and output depth frequency domain features.

[0049] Furthermore, such as Figure 3 As shown, the feature fusion module dynamically generates feature fusion weights based on the deep spatial features and depth frequency domain features of the input image through global average pooling and a two-layer perceptron, and obtains the fused features of the input image by weighted merging.

[0050] Furthermore, such as Figure 4 As shown, the output module includes a classification module and a mask generation module; wherein, the classification module is used to process the fusion features of the input image using a fully connected layer to generate a classification result of the image tampering type; the mask generation module is used to upsample the fusion features of the input image and generate a probability mask map of the tampered region through a convolutional decoder.

[0051] Step S3: The trained dual-stream deep feature fusion network model is used to test the image under test to obtain the final tamper detection result.

[0052] Furthermore, the process for training the model of the two-stream deep feature fusion network is as follows:

[0053] (1) Input the preprocessed tampered image dataset into a two-stream deep feature fusion network;

[0054] (2) Simultaneous training of the deep spatial feature extraction module and the deep spectral feature extraction module: For the deep spatial feature extraction module, the input image is processed by ResNet50 to obtain local spatial features X. conv Local spatial features X conv After patch cutting, linear projection, position encoding, and Transformer encoder, the deep spatial feature F is finally obtained.rgb For the depth spectral feature extraction module, the input image undergoes discrete cosine transform, flattening and renormalization, adaptive multi-head attention mechanism and multi-head attention stitching operation, as well as nonlinear activation of the feedforward network, to finally obtain the depth frequency domain feature F. fre ;

[0055] (3) F rgb and F fre The input image is fed into the feature fusion module, where global average pooling and a two-layer perceptron dynamically generate feature fusion weights. A weighted merging method is then used to obtain the fused features of the input image, ultimately yielding the fused feature F. fusion ;

[0056] (4) Synchronous training of the classification module and the mask generation module: For the classification module, the fused feature F fusion The input classification module processes the data through global average pooling, a fully connected layer, and a softmax activation function, ultimately generating a classification result for image tampering types. This outputs a probability distribution of length c (tampering type), where each value represents the probability of the input image corresponding to each tampering type. The tampering type corresponding to the highest probability value is selected as the classification result for that image tampering type. For the mask generation module, the fused feature F... fusion The input mask generation module, after upsampling, convolutional decoder, and sigmoid activation function, finally generates a mask image of the tampered region. That is, after passing through the sigmoid function, each pixel value can be interpreted as the probability value of the pixel belonging to the tampered region. Combined with a preset probability threshold, it is determined whether each pixel of the input image has been tampered with, and thus the mask image of the tampered region is obtained.

[0057] Furthermore, when training the two-stream deep feature fusion network, a joint optimization is performed using a summative function that includes the contrastive learning loss function, the cross-entropy loss function, and the binary mask loss function. The contrastive learning loss function, the cross-entropy loss function, the binary mask loss function, and the summative function are shown below:

[0058]

[0059] in, To compare the loss functions, N is the total number of sample pairs, that is, the total number of positive and negative sample pairs generated during training, f i Let f represent the features of the i-th sample. i + To be with f i Similar positive samples, i.e., tampered regions in the same image, f i - To be with f i Dissimilar negative samples, i.e., unaltered regions in the image, d(f a ,fb ) represents the eigenvector f a and f b Distance metric between them, t i This represents the label of a positive sample, where m is a preset distance threshold.

[0060]

[0061] in, Let y be the cross-entropy loss function, where C represents the number of classes, and y is the cross-entropy loss function. c It's a category label. It is the probability predicted by the model;

[0062]

[0063] in, Let M be the binary mask loss function, and M be the binary mask of the input image. This is the probability mask for the model's predictions.

[0064]

[0065] in, For the total function, To compare loss functions, Let cross-entropy be the loss function. Let be the binary mask loss function, α be the weighting coefficient of the contrastive learning loss, β be the weighting coefficient of the cross-entropy loss, and γ be the weighting coefficient of the binary mask loss.

[0066] Furthermore, to verify the effectiveness of this invention, based on the CASIAv1 dataset, it was compared with Mantra-Net, CR-CNN, SPAN, MVSS-Net, and MVSS-Net++. Specifically, different intensities of JPEG compression and Gaussian blur attacks were applied to various tampered images in the CASIAv1 dataset. The quality factor of JPEG compression was reduced from 100 to 50, with lower quality indicating stronger compression. The kernel size of Gaussian blur was increased from 1 to 29 to simulate different levels of blurring. A fixed-threshold F1 score was used to measure the balance between precision and recall of each model in the tampered region; a higher score indicates a more effective model in detecting the tampered region. The test comparison results are as follows: Figure 5 As shown. From Figure 5The data shows that, as the compression quality decreases from 100 to 50, the F1 score of this invention remains high at all quality levels. In particular, at compression quality of 50, the F1 score still reaches above 0.5, which is significantly better than ManTra-Net, CR-CNN, SPAN, MVSS-Net, and MVSS-Net++. This indicates that this invention has strong robustness against JPEG compression attacks and can accurately locate the tampered area even when image details are damaged. In addition, as the Gaussian blur kernel size increases, the F1 score of this invention decreases, but it still maintains a high detection capability. Especially at small kernel sizes (5 to 11), the F1 score is always higher than the comparison method. Even when the Gaussian blur kernel size is 29, the detection effect of this invention is still better than the comparison method, indicating that this invention has strong adaptability and stability under image blur interference.

[0067] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A dual-flow image tampering detection method based on Transformer and dynamic fusion, characterized in that, The input image is detected according to steps S1 to S3 to obtain a tamper detection result of the input image: Step S1, obtaining a public tampered image dataset, and labeling each tampered image in the tampered image dataset to obtain a tamper type label and a tamper region label corresponding to each tampered image, and further preprocessing the tampered image dataset; Step S2, using the preprocessed tampered image dataset to train a dual-flow deep feature fusion network, the dual-flow deep feature fusion network comprising a deep spatial feature extraction module, a deep frequency spectrum feature extraction module, a feature fusion module, and an output module; Step S3, using the trained dual-flow deep feature fusion network model to test the to-be-tested image to obtain a final tamper detection result; Specifically, the deep spatial feature extraction module comprises a convolution module and an extended spatial feature extraction module, and the extended spatial feature extraction module comprises an embedding vector module, a position encoding module, and a Transformer encoder module; The input end of the convolution module constitutes the input end of the deep spatial feature extraction module, and the output end of the convolution module is connected with the input end of the extended spatial feature extraction module; the input end of the embedding vector module constitutes the input end of the extended spatial feature extraction module, the input end of the position encoding module is connected with the output end of the embedding vector module, the input end of the Transformer encoder module is connected with the output end of the position encoding module, and the output end of the Transformer encoder constitutes the output end of the extended spatial feature extraction module; The input picture is received by the convolution module, the local spatial features of the input picture are extracted using the trained ResNet50, and the local spatial features are output to the embedding vector module; the local spatial features are divided using Patch cutting based on a preset size by the embedding vector module, the embedding vectors corresponding to each block are obtained using linear projection, and the embedding vectors corresponding to each block are output to the position encoding module; the position encoding corresponding to each block is generated using a sine and cosine function by the position encoding module, and the embedding vectors and the position encoding corresponding to each block are output to the Transformer encoder; the global spatial features are extracted and output by the Transformer encoder through multi-head self-attention and a feedforward network; The deep frequency spectrum feature extraction module comprises a discrete cosine transform module, an input feature embedding module, an adaptive multi-head attention module, and a feedforward network module; The input end of the discrete cosine transform module constitutes the input end of the deep frequency spectrum feature extraction module, the input end of the input feature embedding module is connected with the output end of the discrete cosine transform module, the input end of the adaptive multi-head attention module is connected with the output end of the input feature embedding module, the input end of the feedforward network module is connected with the output end of the adaptive multi-head attention module, and the output end of the feedforward network module constitutes the output end of the deep frequency spectrum feature extraction module; The input picture is received by the discrete cosine transform module, the local frequency domain features of the input image are extracted, and the local frequency domain features are output to the input feature embedding module; the input feature embedding module converts the local frequency domain features into vector form by using a flattening method, and outputs them to the adaptive multi-head attention module; the adaptive multi-head attention module extracts global frequency domain features by using an adaptive multi-head attention mechanism and a multi-head attention splicing operation, and outputs them to the feedforward network module; The feedforward network module extracts and outputs deep frequency domain features by using nonlinear activation processing.

2. The dual-stream image tampering detection method based on Transformer and dynamic fusion according to claim 1, characterized in that, In step S1, the tampered image dataset is preprocessed by using horizontal flipping and color disturbance.

3. The dual-stream image tampering detection method based on Transformer and dynamic fusion according to claim 1, characterized in that, The input end of the deep spatial feature extraction module and the input end of the deep frequency spectrum feature extraction module constitute the input end of the dual-flow deep feature fusion network; the input end of the feature fusion module is connected with the output end of the deep spatial feature extraction module and the output end of the deep frequency spectrum feature extraction module; the input end of the output module is connected with the output end of the feature fusion module; and the output end of the output module constitutes the output end of the dual-flow deep feature fusion network. The deep spatial feature extraction module is configured to extract and output global spatial features of an input image; the deep frequency spectrum feature extraction module is configured to extract and output deep frequency domain features of the input image; and the feature fusion module is configured to obtain fusion features of the input image based on the global spatial features and the deep frequency domain features of the input image. The output module is configured to generate a classification result of an image tampering type and a region probability mask map based on the fusion features of the input image.

4. The dual-stream image tampering detection method based on Transformer and dynamic fusion according to claim 3, characterized in that, The feature fusion module dynamically generates fusion weights based on deep spatial features and deep frequency domain features of an input image by using global average pooling and a double-layer perception machine, and obtains fusion features of the input image by using a weighted merging manner.

5. The dual-stream image tampering detection method based on Transformer and dynamic fusion according to claim 3, characterized in that, The output module includes a classification module and a mask generation module. The classification module is configured to perform full connection layer processing on the fusion features of the input image to generate a classification result of an image tampering type; and the mask generation module is configured to perform up-sampling on the fusion features of the input image and generate a probability mask map of a tampered region by using a convolution decoder.

6. The dual-stream image tampering detection method based on Transformer and dynamic fusion according to claim 1, characterized in that, When the dual-flow deep feature fusion network is trained, the model is jointly optimized by using a total function including a contrastive learning loss function, a cross-entropy loss function, and a binary mask loss function, and the contrastive learning loss function, the cross-entropy loss function, the binary mask loss function, and the total function are as follows: ； wherein, is a contrastive loss function, is the total number of sample pairs, i.e., the total number of positive sample, negative sample pairs generated in the training process, represents the feature of the i-th sample, is a similar positive sample, i.e., a tampered region in the same image, is a dissimilar negative sample, i.e., a non-tampered region in the image, represents a distance measure between the feature vectors and represents a positive sample label, is a preset distance threshold value;​​​​ ； wherein, is a cross-entropy loss function, denotes the number of classes, is a class label, is a probability predicted by the model; ； wherein, is a binary mask loss function, is a binary mask of the input image, is a probability mask predicted by the model; ； wherein, is a total function, is a contrastive loss function, is a cross-entropy loss function, is a binary mask loss function, is a weighting coefficient for the contrastive learning loss, is a weighting coefficient for the cross-entropy loss, is a weighting coefficient for the binary mask loss.

Images