Multi-modal CNN-Transform fused image tampering detection method

Abstract

The invention belongs to the field of computer vision, and particularly relates to a multi-mode CNN-Transform fused image tampering detection method, which designs a CNN and Transform double-flow parallel feature extraction structure, effectively combines the advantage of CNN at capturing fine local features and the advantage of Transform in capturing long-distance dependency relationship and global semantic information, and improves the accuracy of image tampering detection. Local details and global tampering features in the image can be sensitively perceived at the same time, so that the detection capability and stability of image tampering are effectively improved. Processing the noise domain image filtered by the SRM through a CNN (Convolutional Neural Network), and capturing local texture and noise artifact features; meanwhile, a Transform branch processes an RGB spatial domain image, extracts rich global semantic information, and realizes deep interaction and advantage complementation of two types of modal information through a feature fusion module. The BAFM module provided by the invention can deeply mine feature information of different scales and different spatial directions, and more accurate spatial attention features are generated, so that the sensitivity and expression ability of the network to tampered regions are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an image tampering detection method using a multimodal CNN-Transformer fusion. Background Technology

[0002] With the rapid development of digital technology and image editing software, the tampering of digital images has become increasingly easy, and image forgery has become rampant. Such tampering, including copying and pasting, splicing, and removing, poses a serious threat to the authenticity of news media, the reliability of forensic identification, the security of online dissemination, and the social credit system. Therefore, efficient, accurate, and convenient digital image forgery detection and localization technologies have become a research hotspot in the fields of computer vision, image processing, and information security.

[0003] Methods for detecting image tampering can be broadly categorized into two types: traditional methods and deep learning methods. Traditional methods rely on hand-designed features, such as ELA and DCT. While these methods have some effectiveness, they suffer from poor generalization and limited performance when dealing with complex tampering scenarios or subtle tampering traces, making them unsuitable for practical applications.

[0004] In recent years, deep learning methods have gradually become the mainstream in the field of image tampering detection. The rise of Convolutional Neural Networks (CNNs), in particular, has greatly improved the ability of automatic feature extraction. CNN-based methods can effectively capture local details, texture information, and subtle forgery traces caused by local tampering. However, CNN networks themselves are limited by the receptive field, resulting in limited ability to capture long-distance spatial dependencies and difficulty in effectively capturing global information in large-scale or complex splicing tampering operations. With the successful application of the Transformer architecture in Natural Language Processing (NLP) and Computer Vision (CV), more and more image tampering detection methods have also adopted the Transformer architecture. Due to its self-attention mechanism and global feature modeling capabilities, the Transformer can effectively capture global contextual information and long-distance dependencies in images, improving the overall understanding ability of the model. However, the Transformer architecture typically processes image data in a serialized manner, ignoring the inherent local spatial structure information and texture details of the image, leading to insufficient capture of some subtle local tampering traces.

[0005] Most existing image tampering detection methods only utilize image features in the spatial domain (RGB), often neglecting the unique traces left by forgery operations in the noise domain. Studies have demonstrated that tampering creates specific and stable forgery traces in the noise domain, and noise information can more effectively reflect these subtle changes. However, most current methods focus only on the single RGB domain during feature extraction, making it difficult to effectively fuse and complement forgery traces from different domains. Summary of the Invention

[0006] This invention proposes a multimodal feature fusion network (CTF-Net) that integrates CNN and Transformer architectures. Addressing the shortcomings of existing image tampering detection methods, such as incomplete information capture, insufficient single-modal features, inadequate multimodal information fusion, limited feature interaction mechanisms, and difficulty in simultaneously identifying tampering types and locating tampered regions, this invention proposes a CNN-Transformer Fusion Network for Image Manipulation Detection and Localization (CTF-Net). This method fully leverages the advantages of multimodal features in both the RGB spatial domain and the noise domain, constructs a dual-stream parallel architecture of CNN and Transformer, and designs a bidirectional attention fusion module (BAFM) to achieve deep fusion of multi-scale and multimodal features, thereby simultaneously improving the accuracy of tampering type classification and the accuracy of tampered region localization.

[0007] The technical solution adopted by the method of the present invention is as follows: A multimodal CNN-Transformer fusion image tampering detection method, comprising the following steps:

[0008] Step 1: Input the RGB image to be detected, and proceed to step 2 for data preprocessing;

[0009] Step 2: Perform preprocessing on the input image from both the RGB domain and the noise domain branches, such as noise domain transformation and patching. The features of both branches are transformed into feature representations suitable for subsequent network modules, providing input for subsequent feature extraction. Proceed to Step 3.

[0010] Step 3: Perform feature extraction and processing on the two branches respectively. The RGB stream extracts global semantic features step by step based on the Transformer Block, while the noise stream extracts local noise domain features step by step based on the convolution module. The two streams run in parallel step by step, and feature fusion is performed through the corresponding fusion module at the end of each stage to obtain fused features at different scales, and then proceed to step 4.

[0011] Step 4: Feed the different scale fusion features generated in Step 3 into the bidirectional attention fusion module BAFM to further fuse feature information of different modes and scales, output the final fusion features, and proceed to Step 5;

[0012] Step 5: Finally, the fused features are processed by a classifier to obtain the tampering type classification result, and the tampering region mask map is obtained by a forgery detection decoder, thereby realizing the dual task output of tampering detection and localization.

[0013] Furthermore, as a preferred embodiment of the present invention, the input in step 1 is an RGB format image to be detected. It is composed of three primary colors: red, green, and blue. In a specific implementation of this invention, it is set as follows: .

[0014] Furthermore, as a preferred embodiment of the present invention, a dual-modal preprocessing strategy is designed in step 2 of the data preprocessing stage, specifically including two sub-steps: noise domain processing and RGB spatial domain processing.

[0015] S201: Noise domain data preprocessing.

[0016] This invention utilizes SRM to extract noise domain features from images. The SRM filter can effectively suppress image content information and enhance noise domain forgery traces in tampered areas, thereby providing high-quality input feature maps for the CNN feature extraction branch. Specifically: the input image is... The three basic kernel weights of SRM are shown in Equation (1).

[0017]

[0018] Using the center of each filter kernel as the origin, rotate clockwise by 90°, 180°, and 270° respectively to better capture differences between adjacent pixels in different directions, detect whether the image has undergone geometric transformation, and effectively improve the detection accuracy of tampered images. The kernel of the filter layer becomes... Then, a noise domain image is generated. , represented as:

[0019]

[0020] in, This indicates an SRM filtering operation. This indicates the number of noise feature channels obtained after filtering; in this invention, it is 9.

[0021] S202: RGB domain data preprocessing.

[0022] To leverage the feature extraction advantages of Transformer networks, this invention will input the image... Divide into several sizes The image patch. In this invention, the image patch size is... , No. Each patch is represented as:

[0023]

[0024] These image patches are then embedded using linear mapping (Patch Embedding).

[0025]

[0026] in, Indicates the flattening operation. , These represent the embedding weights and biases, respectively. Indicates the embedding dimension.

[0027] To preserve the spatial location information of image patches, this invention further incorporates learnable location embedding:

[0028]

[0029] The above The location embedding function can be learned through network training.

[0030] The embedded feature sequences of the RGB domain patches are better suited for global feature extraction using the Transformer architecture, while the noise domain data, after processing with the SRM filter, effectively highlights subtle noise features in locally tampered areas of the image, making it more suitable for subsequent CNN modules to extract local detail features. This achieves an effective representation of both the RGB spatial domain and noise domain information of the image to be detected.

[0031] Furthermore, as a preferred technical solution of the present invention, the present invention designs a feature extraction structure based on dual-stream parallelism to effectively capture forgery traces generated by image tampering in different domains. The CNN branch focuses on extracting local features in the noise domain, while the Transformer branch focuses on modeling global semantic features. Simultaneously, a Fusion Block fusion module is designed to achieve in-depth interaction and fusion of dual-stream features. The detailed steps of step 3 are as follows:

[0032] S301: Noise flow feature extraction.

[0033] The noise flow feature extraction branch employs a ConvNeXt-based convolutional network structure for feature extraction. The ConvNeXt network, drawing inspiration from the Transformer architecture, boasts a larger receptive field and superior feature representation capabilities, effectively increasing the network's receptive field and extracting more refined local features. The noisy image passes through four consecutively stacked Conv Blocks. After each convolutional module, the noise flow features are further extracted to reveal deeper levels of local texture and tamper-related noise features, ultimately outputting noise domain feature representations at different scales, denoted as:

[0034]

[0035] in, , representing the initial input features. The features output by each convolutional module are denoted as . Its specific feature size and number of channels gradually change with the network depth in order to capture local forgery features at different scales.

[0036] S302: RGB stream feature extraction.

[0037] RGB stream feature extraction is performed using a Transformer architecture, leveraging the Transformer network's powerful global information capture capabilities. The input is a feature sequence processed by patch embedding and position embedding. The process proceeds sequentially through four Transformer modules. Each Transformer module includes Layer Normalization (LN), Multi-head Self-Attention (MHSA), and Feed Forward Network (FFN), as shown in the following formula:

[0038]

[0039]

[0040] in, This represents the RGB domain characteristics of the output of the i-th stage of the Transformer.

[0041] S303: Phased feature fusion.

[0042] At the end of each feature extraction stage, noise flow features Features of RGB Stream Transformer Output Enter the corresponding feature fusion module. The specific fusion process is as follows:

[0043] Given the features of the i-th stage Transformer and corresponding CNN noise features The fusion module first processes the Transformer features. Perform max pooling and average pooling operations respectively:

[0044]

[0045]

[0046] Subsequently, the features obtained from the two pooling methods are concatenated, and then a 1×1 convolution is used to adjust the channel dimension and integrate the features to obtain the fused weighted features. Specifically, it is expressed as:

[0047]

[0048] Here, [·] represents the concatenation operation of the channel dimension.

[0049] Finally, the CNN noise stream features are extracted using element-wise multiplication. With fusion weight features The fusion is performed to obtain the fusion features of the i-th stage. :

[0050]

[0051] Among them, symbols This indicates the element-wise multiplication operation.

[0052] The network effectively fuses global features in the RGB domain with local features in the noise domain at each stage, fully leveraging the advantages of CNN and Transformer to achieve complementary advantages of features from different modalities and scales, and generate fused features. This is for further in-depth integration of subsequent features.

[0053] Furthermore, as a preferred technical solution of the present invention, in order to further improve the model's sensitivity to tampered region features and fully explore the correlation and complementarity between multi-scale and multi-modal fused features, the present invention proposes a Bi-directional Attention Fusion Module (BAFM) to achieve deeper feature interaction between fused features at different scales. Step 4 involves the four fused features at different scales generated in step 3. First, a splicing operation is performed at the channel dimension to form a fused feature. :

[0054]

[0055] Subsequently, the fusion features Vertical Max Pooling (VMP) and Horizontal Max Pooling (HMP) operations are used to capture salient features in the vertical and horizontal spatial directions of the fused features, respectively. The output features of these two pooling operations are then concatenated, and spatial attention weights are calculated using a batch normalization operation and a sigmoid activation function. The specific formula is as follows:

[0056]

[0057] in, This indicates a batch normalization operation. This represents the Sigmoid activation function.

[0058] The obtained features Furthermore, two parallel 1×1 convolutional layers are used, each followed by a sigmoid activation function to generate two attention weight maps, which are used to spatially weight the fused features. Generate attention weights:

[0059]

[0060]

[0061] The above operations generate two attention weight feature maps in the vertical and horizontal directions, respectively. , .

[0062] Finally, the two attention weights are combined with the original fused features. Element-wise multiplication is performed to obtain the final fusion feature enhanced by the bidirectional attention mechanism. :

[0063] .

[0064] Furthermore, as a preferred embodiment of the present invention, step 5 is the feature fusion and output stage. Specifically, the final fused features output by the bidirectional attention fusion module BAFM designed in this invention... This not only efficiently integrates multimodal information from the RGB and noise domains, but also possesses rich spatial details and global semantic information. To simultaneously achieve dual-task detection and output of image tampering type classification and tampering region localization, this invention integrates the fused features... The data is fed into two parallel processing branches: the classifier and the forgery decoder.

[0065] S501: Tamper type classifier.

[0066] The role of the classifier is to analyze the fused features. Further feature refinement is performed to generate accurate tampering type classification results;

[0067] Specifically, the classifier first processes the fused features. Channel adjustment is performed using 1×1 convolutional layers to achieve dimensionality reduction and feature optimization:

[0068]

[0069] Subsequently, the features Global Average Pooling (GAP) is performed to extract the global semantic feature vector, as follows:

[0070]

[0071] Finally, after mapping through a fully connected layer (FC) and applying the softmax activation function, the final probability of the tamper type classification result is output:

[0072]

[0073] in, This represents the operation of a fully connected layer, and the output probability vector. Each element represents the probability of a specific type of alteration, such as copying and pasting, splicing, or removing.

[0074] S502: Forgery detection decoder.

[0075] The forgery detection decoder designed in this invention can achieve precise location of tampered areas. Specifically, the forgery detection decoder uses fused features... As input, multiple convolutional layers and a progressive upsampling strategy are employed to continuously increase the spatial size of features and restore fine spatial details, ultimately outputting a tamper mask image of the same size as the input image. The generated mask image is defined as follows:

[0076]

[0077] In this mask image, each element represents the probability that a pixel at a corresponding spatial location has been tampered with; a higher probability value indicates a greater likelihood of tampering. The final mask image can be thresholded to accurately pinpoint the tampered area.

[0078] Finally, the model outputs the predicted type of tampering and the location of the tampered area.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] (1) Dual-stream structure enhances the complementarity of local and global features. This invention designs a dual-stream parallel feature extraction structure of CNN and Transformer, which effectively combines the advantages of CNN in capturing subtle local features and the advantages of Transformer in capturing long-distance dependencies and global semantic information. It can simultaneously and keenly perceive local details and global tampering features in the image, thereby effectively improving the detection capability and stability of image tampering.

[0081] (2) Multimodal feature fusion improves forgery detection performance. This invention uses CNN to process the noise domain image after SRM filtering to capture local texture and noise artifact features; at the same time, the Transformer branch processes the RGB spatial domain image to extract rich global semantic information, and realizes deep interaction and complementary advantages of the two modal information through the feature fusion module. Compared with the single RGB modal method, it can significantly improve experimental accuracy.

[0082] (3) The bidirectional attention mechanism effectively improves the accuracy of tampered region localization. The BAFM module proposed in this invention can deeply mine feature information at different scales and in different spatial directions, generating more accurate spatial attention features, thereby improving the network's sensitivity and expressive ability to tampered regions. The model simultaneously optimizes the tampering type classification task and the tampered region mask localization task through a joint loss function, realizing mutual promotion between the two tasks and improving the overall precision of detection and localization. Attached Figure Description

[0083] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0084] Figure 2 This is a diagram of the overall architecture of CTF-Net in this invention;

[0085] Figure 3 This is a flowchart of the feature processing of the present invention;

[0086] Figure 4 This is a diagram of the Fusion Block architecture of the present invention;

[0087] Figure 5 This is a schematic diagram of the BAFM architecture of the bidirectional attention fusion module of the present invention;

[0088] Figure 6 This is a schematic diagram of the Classifier structure of the present invention. Detailed Implementation

[0089] The present invention will be further explained in detail below with reference to the accompanying drawings, so that those skilled in the art can better understand and implement the present invention. However, the following examples are only used to explain the present invention and are not intended to limit the present invention.

[0090] This invention proposes a multimodal feature fusion method (CTF-Net) for image tampering detection and localization, integrating convolutional neural networks (CNNs) and Transformers. It aims to address the shortcomings of existing deep learning methods, such as insufficient utilization of multimodal features, performance limitations of single-structure CNNs and Transformers, and inadequate feature fusion. By combining the advantages of CNNs and Transformers, this invention achieves effective fusion of RGB spatial domain features and noise domain features, and utilizes a bidirectional attention mechanism to deeply explore the complementarity between features, thereby significantly improving the accuracy of image tampering type classification and tampering region localization. The overall flowchart is as follows: Figure 1 As shown, the network architecture is as follows Figure 2The diagram shows the overall architecture of the CNN-Transformer fusion network (CTF-Net) proposed in this invention. The network includes a CNN noise branch and a Transformer RGB branch. The CNN noise branch extracts noise domain features using an SRM filter, and then extracts local texture details layer by layer through four ConvNeXt-based convolutional modules. The RGB branch divides the image into several patches, processes them with patch embedding and position embedding, and then feeds them into four Transformer modules to progressively extract global semantic information. Subsequently, the features from the two branches at each stage are fused through a feature fusion module to obtain multi-scale fused features. - The data are then fed into a bidirectional attention fusion module (BAFM) for deep fusion. The final fused features are then input into a classifier for tamper type classification and into a forgery detection decoder for mask generation to locate the tampered region.

[0091] The specific implementation process of this invention includes the following steps:

[0092] Step 1: Input the RGB image to be detected, and proceed to step 2 for data preprocessing.

[0093] Step 2: The input image is preprocessed in both the RGB domain and the noise domain, including noise domain transformation and patching. The features of both branches are transformed into feature representations suitable for subsequent network modules, providing input for feature extraction. Then proceed to Step 3.

[0094] Step 3: Feature extraction and processing are performed on the two branches separately. The RGB stream extracts global semantic features stage by stage based on the Transformer Block, while the noise stream extracts local noise domain features stage by stage based on the convolution module. The two streams run in parallel stage by stage, and feature fusion is performed at the end of each stage through the corresponding fusion module to obtain fused features at different scales. Then proceed to Step 4.

[0095] Step 4: Input the different scale fusion features generated in Step 3 into the bidirectional attention fusion module (BAFM) to further fuse feature information of different modes and scales, output the final fusion features, and proceed to Step 5.

[0096] Step 5: Finally, the fused features are processed by a classifier to obtain the tampering type classification result, and the tampering region mask map is obtained by a forgery detection decoder, thereby realizing the dual task output of tampering detection and localization.

[0097] Step 6: End.

[0098] Specifically, the input for step 1 is the image to be detected in RGB format. It is composed of three primary colors: red, green, and blue. In a specific implementation of this invention, it is set as follows: .

[0099] As a preferred approach, a dual-modal preprocessing strategy is designed for the data preprocessing stage in step 2, specifically including two sub-steps: noise domain processing and RGB spatial domain processing.

[0100] S201: Noise domain data preprocessing.

[0101] This invention utilizes SRM to extract noise domain features from images. The SRM filter can effectively suppress image content information and enhance noise domain forgery traces in tampered areas, thereby providing high-quality input feature maps for the CNN feature extraction branch. Specifically: the input image is... The three basic kernel weights of SRM are shown in Equation (1).

[0102]

[0103] Using the center of each filter kernel as the origin, rotate clockwise by 90°, 180°, and 270° respectively to better capture differences between adjacent pixels in different directions, detect whether the image has undergone geometric transformation, and effectively improve the detection accuracy of tampered images. The kernel of the filter layer becomes... Then, a noise domain image is generated. , represented as:

[0104]

[0105] in, This indicates an SRM filtering operation. This indicates the number of noise feature channels obtained after filtering; in this invention, it is 9.

[0106] S202: RGB domain data preprocessing.

[0107] To leverage the feature extraction advantages of Transformer networks, this invention will input the image... Divide into several sizes The image patch. In this invention, the image patch size is... , No. Each patch is represented as:

[0108]

[0109] These image patches are then embedded using linear mapping (Patch Embedding).

[0110]

[0111] in, Indicates the flattening operation. , These represent the embedding weights and biases, respectively. Indicates the embedding dimension.

[0112] To preserve the spatial location information of image patches, this invention further incorporates learnable location embedding:

[0113]

[0114] The above The location embedding function can be learned through network training.

[0115] The embedded feature sequences of the RGB domain patches are better suited for global feature extraction using the Transformer architecture, while the noise domain data, after processing with the SRM filter, effectively highlights subtle noise features in locally tampered areas of the image, making it more suitable for subsequent CNN modules to extract local detail features. This achieves an effective representation of both the RGB spatial domain and noise domain information of the image to be detected.

[0116] This invention designs a dual-stream parallel feature extraction structure to effectively capture forgery traces generated by image tampering in different domains. The CNN branch focuses on extracting local features in the noise domain, while the Transformer branch focuses on modeling global semantic features. A Fusion Block module is also designed to achieve deep interaction and fusion of the two-stream features. The flowchart for step 3 is as follows: Figure 3 As shown, the detailed steps are as follows:

[0117] S301: Noise flow feature extraction.

[0118] The noise flow feature extraction branch employs a ConvNeXt-based convolutional network structure for feature extraction. The ConvNeXt network, drawing inspiration from the Transformer architecture, boasts a larger receptive field and superior feature representation capabilities, effectively increasing the network's receptive field and extracting more refined local features. The noisy image passes through four consecutively stacked Conv Blocks. After each convolutional module, the noise flow features are further extracted to reveal deeper levels of local texture and tamper-related noise features, ultimately outputting noise domain feature representations at different scales, denoted as:

[0119]

[0120] in, , representing the initial input features. The features output by each convolutional module are denoted as . Its specific feature size and number of channels gradually change with the network depth in order to capture local forgery features at different scales.

[0121] S302: RGB stream feature extraction.

[0122] RGB stream feature extraction is performed using a Transformer architecture, leveraging the Transformer network's powerful global information capture capabilities. The input is a feature sequence processed by patch embedding and position embedding. The process proceeds sequentially through four Transformer modules. Each Transformer module includes Layer Normalization (LN), Multi-head Self-Attention (MHSA), and Feed Forward Network (FFN), as shown in the following formula:

[0123]

[0124]

[0125] in, This represents the RGB domain characteristics of the output of the i-th stage of the Transformer.

[0126] S303: Phased feature fusion.

[0127] At the end of each feature extraction stage, noise flow features Features of RGB Stream Transformer Output Entering the corresponding feature fusion module, the architecture of Fusion Block is as follows: Figure 4 As shown: Features from the Transformer branch After vertical max pooling and average pooling, the features are concatenated, then passed through a 1×1 convolutional layer for feature dimensionality reduction and channel adjustment to generate weighted features. Subsequently, these weighted features are combined with the features output from the CNN branch. Element-wise multiplication is performed to obtain the fused multimodal features. .

[0128] The specific integration process is as follows:

[0129] Given the features of the i-th stage Transformer and corresponding CNN noise features The fusion module first processes the Transformer features. Perform max pooling and average pooling operations respectively:

[0130]

[0131]

[0132] Subsequently, the features obtained from the two pooling methods are concatenated, and then a 1×1 convolution is used to adjust the channel dimension and integrate the features to obtain the fused weighted features. Specifically, it is expressed as:

[0133]

[0134] Here, [·] represents the concatenation operation of the channel dimension.

[0135] Finally, the CNN noise stream features are extracted using element-wise multiplication. With fusion weight features The fusion is performed to obtain the fusion features of the i-th stage. :

[0136]

[0137] Among them, symbols This indicates the element-wise multiplication operation.

[0138] The network effectively fuses global features in the RGB domain with local features in the noise domain at each stage, fully leveraging the advantages of CNN and Transformer to achieve complementary advantages of features from different modalities and scales, and generate fused features. This is for further in-depth integration of subsequent features.

[0139] To further enhance the model's sensitivity to tampered region features and fully explore the correlations and complementarities between multi-scale and multi-modal fused features, this invention proposes a Bi-directional Attention Fusion Module (BAFM) to achieve deeper feature interaction between fused features at different scales. The architecture diagram is shown below. Figure 5 As shown: First, the fusion features at different scales are... , , , Features are formed by splicing together along the channel dimension. Subsequently, vertical max pooling (VMP) and horizontal max pooling (HMP) are performed separately. The pooled features are then concatenated, batch normalized, and activated with a sigmoid function to calculate spatial attention weights. Finally, two 1×1 convolutional layers and a sigmoid activation function are used to generate vertical and horizontal attention feature maps, which are then fused with the original features. Perform element-wise multiplication to obtain the final fusion feature. Step 4: The fusion features generated in Step 3 at four different scales. First, a splicing operation is performed at the channel dimension to form a fused feature. :

[0140]

[0141] Subsequently, the fusion features Vertical Max Pooling (VMP) and Horizontal Max Pooling (HMP) operations are used to capture salient features in the vertical and horizontal spatial directions of the fused features, respectively. The output features of these two pooling operations are then concatenated, and spatial attention weights are calculated using a batch normalization operation and a sigmoid activation function. The specific formula is as follows:

[0142]

[0143] in, This indicates a batch normalization operation. This represents the Sigmoid activation function.

[0144] The obtained features Furthermore, two parallel 1×1 convolutional layers are used, each followed by a sigmoid activation function to generate two attention weight maps, which are used to spatially weight the fused features. Generate attention weights:

[0145]

[0146]

[0147] The above operations generate two attention weight feature maps in the vertical and horizontal directions, respectively. , .

[0148] Finally, the two attention weights are combined with the original fused features. Element-wise multiplication is performed to obtain the final fusion feature enhanced by the bidirectional attention mechanism. :

[0149]

[0150] Step 5 is the feature fusion and output stage. Specifically, the final fused features output by the bidirectional attention fusion module BAFM designed in this invention... This not only efficiently integrates multimodal information from the RGB and noise domains, but also possesses rich spatial details and global semantic information. To simultaneously achieve dual-task detection and output of image tampering type classification and tampering region localization, this invention integrates the fused features... The data is fed into two parallel processing branches: the classifier and the forgery decoder.

[0151] S501: Tamper type classifier.

[0152] The role of the classifier is to analyze the fused features. Further feature refinement is performed, and accurate tamper type classification results are generated, with the architecture as follows: Figure 6 As shown: The fused features output by the Bidirectional Attention Fusion Module (BAFM) First, the features are reduced and optimized using a 1×1 convolutional layer. Then, global semantic features are extracted using global average pooling. Finally, the features are mapped to the tampering type classification space through a fully connected layer, and the classification results are output, thus achieving accurate identification of different image tampering types such as copy-paste, splicing, and removal.

[0153] Specifically, the classifier first processes the fused features. Channel adjustment is performed using 1×1 convolutional layers to achieve dimensionality reduction and feature optimization:

[0154]

[0155] Subsequently, the features Global Average Pooling (GAP) is performed to extract the global semantic feature vector, as follows:

[0156]

[0157] Finally, after mapping through a fully connected layer (FC) and applying the softmax activation function, the final probability of the tamper type classification result is output:

[0158]

[0159] in, This represents the operation of a fully connected layer, and the output probability vector. Each element represents the probability of a specific type of alteration, such as copying and pasting, splicing, or removing.

[0160] S502: Forgery detection decoder.

[0161] The forgery detection decoder designed in this invention can achieve precise location of tampered areas. Specifically, the forgery detection decoder uses fused features... As input, multiple convolutional layers and a progressive upsampling strategy are employed to continuously increase the spatial size of features and restore fine spatial details, ultimately outputting a tamper mask image of the same size as the input image. The generated mask image is defined as follows:

[0162]

[0163] In this mask image, each element represents the probability that a pixel at a corresponding spatial location has been tampered with; a higher probability value indicates a greater likelihood of tampering. The final mask image can be thresholded to accurately pinpoint the tampered area.

[0164] Finally, the model outputs the predicted type of tampering and the location of the tampered area.

[0165] This invention proposes a new image tampering detection and localization method (CTF-Net) that integrates CNN and Transformer, aiming to address the problems of existing methods, such as single network structure, insufficient utilization of single-modal features, imperfect multimodal feature fusion, and difficulty in simultaneously classifying tampering types and locating regions. Through dual-stream feature extraction, cross-modal fusion, and a bidirectional attention mechanism (BAFM), this invention improves the accuracy and robustness of tampering detection, enabling it to adapt to various application scenarios, such as image authenticity verification in news media, digital forensic evidence collection, social network content supervision, e-commerce platform product verification, and digital copyright protection.

[0166] Example 1: Verification of the authenticity of images in online news media

[0167] Ensuring the authenticity of images in news reporting and media dissemination is crucial for maintaining public trust. However, maliciously altered news images are commonplace, such as fabricating news events through splicing, copying, and pasting, thus misleading public opinion. Traditional manual review methods rely on the experience of professionals, resulting in low detection efficiency and difficulty in detecting subtle signs of alteration.

[0168] The CTF-Net proposed in this invention can be used for image authenticity verification by news media. News organizations can input news images to be published into the model of this invention. CTF-Net extracts multimodal features of the image through a CNN-Transformer two-stream structure and fuses local noise and global information through a BAFM module to accurately determine whether the image has been tampered with and automatically marks possible tampered areas. Compared with traditional manual verification methods, this invention can significantly improve the detection speed and accuracy in large-scale news image verification, helping news organizations quickly screen suspicious content and reduce the risk of spreading fake news.

[0169] Example 2: Digital Forensic Evidence Collection and Authentication

[0170] In judicial proceedings, digital images are frequently used as crucial evidence. However, some cases involve altered or forged image evidence, such as tampering with surveillance video screenshots or modifying photos of traffic accident scenes, complicating the investigation. Existing image forensics methods are typically based on manual feature analysis, which often struggles to provide effective technical support against emerging technologies like deepfakes.

[0171] This invention, CTF-Net, can be used in digital judicial evidence collection scenarios. Forensic experts can input case-related image evidence into the system, and the model automatically identifies image tampering behavior, providing tampering type classifications and tampered area mask images to assist judges or experts in determining the authenticity of evidence. Through the multi-scale feature fusion and bidirectional attention mechanism of this invention, complex tampering methods, such as document forgery and fingerprint image tampering, can be effectively detected, providing more accurate technical support for judicial evidence collection.

[0172] Example 3: Social Network Content Regulation

[0173] Social networking platforms generate massive amounts of image content daily, some of which may be manipulated to spread rumors, mislead the public, or commit malicious fraud. For example, on social media, users may spread false information by altering celebrity photos or forge contracts and documents for fraud. Existing social platforms typically rely on manual review or simple rule checks, making it difficult to process the large volume of user-uploaded content efficiently in real time.

[0174] This invention, CTF-Net, can be embedded into the content moderation system of social media platforms to achieve real-time monitoring of user-uploaded images. The system receives user-uploaded images and automatically determines whether they have been tampered with, while providing visual annotations of tampered areas to assist platform administrators in making quick decisions. CTF-Net's multimodal fusion strategy enables it to effectively handle various complex tampering techniques, including local modifications, object removal, and cross-image stitching, helping to improve the efficiency of content moderation on social media platforms and reduce the spread of misinformation.

[0175] Example 4: Authenticity Detection of Product Images on E-commerce Platforms

[0176] On e-commerce platforms, product images uploaded by merchants directly influence consumers' purchasing decisions. However, some unscrupulous merchants may mislead consumers by tampering with product images, such as adjusting colors, removing blemishes, and making false claims about product performance. Existing e-commerce platforms mainly rely on manual review and simple watermark detection technology, which is insufficient to effectively prevent sophisticated tampering.

[0177] This invention, CTF-Net, can be used for product image authenticity detection on e-commerce platforms. The platform can automatically review product images uploaded by merchants, and the model can identify whether product images have been tampered with, providing detailed visualizations of the tampered areas. Combining the feature extraction capabilities of CNN and Transformer, CTF-Net can accurately identify tampering traces in product images, such as the removal of product defects or the addition of false product labels, thereby improving the platform's transaction transparency and consumer trust.

[0178] Example 5: Digital Copyright Protection and Intellectual Property Authentication

[0179] In fields such as digital art, photography, and advertising design, copyright protection for original works is crucial. However, malicious tampering and misappropriation are rampant, such as altering signatures on artworks or modifying image details to evade copyright tracking. Traditional copyright authentication methods primarily rely on digital watermarks and hash verification, but their ability to detect content-level tampering is relatively weak.

[0180] This invention, CTF-Net, can be applied to digital copyright protection, helping copyright agencies or original authors to verify the authenticity of digital works. CTF-Net can accurately detect whether an image has been tampered with and provides a visualization of the tampered area, assisting original authors or legal institutions in collecting evidence of infringement. Compared to traditional watermarking technology, this invention can accurately identify image forgery even without a watermark, providing a more robust copyright protection solution.

[0181] This invention proposes a new image tampering detection and localization method (CTF-Net) that integrates CNN and Transformer, effectively addressing the limitations of existing methods in single-network feature extraction, insufficient single-modal features, inadequate multimodal feature fusion, and the difficulty in simultaneously classifying tampering types and locating tampered regions. Specifically, this invention achieves fine-grained capture of local forgery details in the image noise domain by CNN and effective modeling of global information in the RGB spatial domain by Transformer through a dual-branch structure and a multi-stage fusion mechanism. This significantly improves the model's recognition accuracy and localization precision in complex tampering scenarios, demonstrating high application value and potential for widespread adoption.

[0182] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multi-modal CNN-Transformer fusion image tampering detection method, characterized in that, Comprising the following steps: Step 1: input the RGB image to be detected, enter step 2 data preprocessing; Step 2: the input image is preprocessed from the RGB domain and the noise domain two branches respectively, and the features of the two branches are converted into feature expression forms suitable for subsequent network module processing, providing input for subsequent feature extraction, entering step 3; Step 3: feature extraction and processing are performed on the two branches respectively, and the RGB stream extracts global semantic features based on the Transformer Block stage by stage, and the noise stream extracts local noise domain features based on the convolution module stage by stage; The two streams run in parallel stage by stage, and at the end of each stage, the corresponding fusion module is used for feature fusion to obtain different scale fusion features, which enter step 4; Step 4: the different scale fusion features generated in step 3 are sent to the bidirectional attention fusion module BAFM for further fusion of feature information of different modules and different scales, and the final fusion features are output, which enter step 5; Step 5: the final fusion features pass through the classifier to obtain the tampering type classification result, and the tampering area mask map is obtained through the forgery detection decoder, so as to realize the dual task output of tampering detection and positioning.

2. The multi-modal CNN-Transformer fusion image tampering detection method according to claim 1, characterized in that, Step 1 input is the image to be detected in RGB format consists of three primary colors red, green and blue.

3. The multi-modal CNN-Transformer fusion image tampering detection method according to claim 2, characterized in that, In step 2, a double-mode preprocessing strategy is designed in the data preprocessing stage, which specifically includes two sub-steps of noise domain processing and RGB spatial domain processing: Step 201, noise domain data preprocessing: the noise domain features of the image are extracted by using SRM; the input image is The weights of the three basic kernels of SRM are shown in formula (1): ； With the center of each filter kernel as the origin, rotating 90°, 180° and 270° clockwise, respectively, in order to capture the difference of adjacent pixels in different directions, the kernel of the filter layer becomes , and then a noise domain image is generated, denoted as: ； wherein, denotes an SRM filtering operation, denotes the number of noise feature channels obtained after filtering. Step 202, RGB domain data preprocessing; To take advantage of the feature extraction of the Transformer network, the input image is divided into several image blocks (patches) of size The image block size is The th patch is represented as:​ ； The image block patch is then embedded through linear mapping: ； wherein, denotes a flattening operation, , denote the embedded weights and bias terms, respectively, denotes the embedding dimension; In order to retain the spatial position information of the image patch, a learnable position embedding is added: ； The above The position embedding function is learned by network training.

4. The multi-modal CNN-Transformer fusion image tampering detection method according to claim 3, characterized in that, Said step 3 specifically includes: Step 301, noise stream feature extraction: The noise stream feature extraction branch uses a convolutional network structure based on ConvNeXt for feature extraction; the noise image passes through 4 Conv Blocks stacked in succession, and after each convolution module, the noise stream features are further extracted deeper local texture and tampering-related noise feature expressions, and finally output different scale noise domain feature representations as: ； wherein, , denotes the initial input feature; the feature output by each convolutional module is denoted as , the specific feature size and the number of channels gradually change with the network depth to capture local fake features of different scales; Step 302, RGB stream feature extraction: RGB stream feature extraction is performed using a Transformer architecture, which has strong global information capturing capability. The input feature sequence is processed by patch embedding and position embedding The RGB stream is sequentially passed through four Transformer modules, each of which includes layer normalization, multi-head self-attention mechanism, and feedforward neural network, which can be expressed by the following formula: ； ； wherein, represents the RGB domain feature output by the i-th stage of the Transformer. Step 303, periodic feature fusion: at the end of each feature extraction stage, the noise flow feature and the RGB flow Transformer output feature into the corresponding feature fusion module, and the specific fusion process is as follows: Given the i-th stage Transformer feature and the corresponding stage's CNN noise feature , the fusion module first performs max-pooling and average-pooling operations on the Transformer feature respectively: ； ； Subsequently, the two pooled features are spliced, and a 1x1 convolution is performed to adjust the channel dimension and integrate the features to obtain the fusion weight feature , which is specifically represented as: ； Where [·] represents the channel dimension splicing operation; Finally, the CNN noise stream features are fused with the fusion weight features by element-wise multiplication, resulting in the fusion features at the i-th stage : ； where the symbol represents an element-wise multiplication operation.

5. The multi-modal CNN-Transformer fusion image tampering detection method of claim 4, wherein Step 4: fusing features of 4 different scales generated in Step 3 The concatenation operation is first performed in the channel dimension to form fused features : ； Subsequently, fusing features The vertical and horizontal spatial directions of the fusing features are captured respectively by vertical maximum pooling (VMP) and horizontal maximum pooling (HMP) operations. The output features of the above two kinds of pooling operations are spliced again, and then a batch normalization Batch Normalization operation and a sigmoid activation function are used to calculate the spatial attention weight, which is specifically expressed as: ； wherein, denotes a batch normalization operation, denotes a Sigmoid activation function; The obtained features Further, two attention weight maps are generated respectively by two parallel 1x1 convolution layers, and each is further subjected to a Sigmoid activation function, for fusing features in the spatial direction Generate attention weight: ； ； The above operations respectively generate two attention weight feature maps in vertical and horizontal directions , ; Finally, the two attention weights are multiplied with the original fusion feature element by element to obtain the final fusion feature reinforced by the bidirectional attention mechanism : 。 6. The multi-modal CNN-Transformer fusion image tampering detection method according to claim 5, characterized in that, Said step 5 specifically includes: S501: tampering type classifier; The role of the classifier is to classify the fused features to further refine the features and generate accurate tampering type classification results; Specifically, the classifier first performs channel adjustment on the fusion features dimension reduction and feature optimization through a 1x1 convolutional layer: ； Subsequently, the features Global average pooling (GAP) is performed to extract a global semantic feature vector, denoted as: ； Finally, after mapping through the fully connected layer FC and passing through the Softmax activation function, the final tampering type classification result probability is output: ； wherein, denotes a fully connected layer operation, the output probability vector each element in the vector represents the probability of a specific tampering type such as copy-paste, splice or removal, respectively. S502: forgery detection decoder; Forgery detection decoder to fuse features For input, multiple convolutional layers and step-up sampling strategy are adopted to increase the spatial size of features and restore fine spatial details, and finally output a tampering mask map with the same size as the input image The generated mask map is defined as: ； Wherein each element in the mask map represents the probability that the corresponding spatial position pixel is tampered with, and the higher the probability value, the greater the possibility of being tampered with; the final mask map can be accurately positioned by thresholding operation to locate the tampered area; Finally, the model outputs the predicted tampering type and tampering area positioning.

Images