Improved image tampering detection network based on AEA and SR modules
By introducing an improved image tampering detection network with AEA and SR modules, and combining edge perception and spatial feature enhancement techniques, the problem of insufficient robustness of existing methods in complex tampering operations is solved, and high-precision tampering region detection and localization are achieved.
Patent Information
- Application Number
- CN202411102618.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-12
AI Technical Summary
Existing image tampering detection methods lack robustness in complex tampering operations. Traditional methods require manual parameter selection and cannot effectively capture edge artifacts and weak, general tampering trace signals. Noise view methods have low effectiveness in copy-movement type tampering.
An improved image tampering detection network based on AEA and SR modules is adopted. By combining edge perceptron, AEA module and SR module, and using a dual-supervised structure model, RGB stream features and weak general tampering features are extracted. A dual attention module is used for feature selection and fusion to generate a pixel-level tampering region prediction map.
It significantly improves the localization accuracy and detection accuracy of tampered areas, effectively captures edge artifacts and weak tampering traces, suppresses noise interference, and improves the robustness of the model.
Smart Images

Figure CN118941935A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a deep learning image tampering detection technology, and in particular to an improved image tampering detection network based on AEA and SR modules. Background Art
[0002] In recent years, image-based tampering detection technology has attracted widespread attention in the field of computer vision. The goal of image tampering detection is to identify and locate tampering or modification in an image to ensure the integrity and authenticity of the image. This technology has important applications in many fields, including legal evidence collection, news media credibility verification, digital content security, etc.
[0003] Traditional image tampering detection methods usually rely on manually designed feature extractors, such as the local invariant feature transform (SIFT), histogram of oriented gradients (HOG), etc. Although these methods perform well in specific scenarios, they usually require manual selection and adjustment of parameters, and may lack robustness for complex tampering operations. In recent years, with the development of deep learning technology, image tampering detection methods based on deep learning have significantly improved the accuracy and robustness of detection. Deep learning models can automatically learn feature representations in images, and there is no need to manually design feature extractors. In particular, convolutional neural networks (CNNs) have made breakthrough progress in image processing tasks, such as being widely used in image classification, object detection, and segmentation, and have gradually been introduced into image tampering detection.
[0004] This paper aims to combine the attention mechanism of deep learning and the spatial feature enhancement technology, and propose an improved image tampering recognition network by using the deep learning model and advanced network structure. Summary of the invention
[0005] The present invention aims to overcome the limitations of the prior art and proposes an improved image tampering detection network based on AEA and SR modules. This method can effectively capture edge artifacts, weak and common tampering trace signals, and information of high-level semantic objects. In order to significantly improve the detection accuracy and the positioning accuracy of the tampered area, a new dual-supervision structure model that deeply mines the correlation between edge features and RGB features is adopted. To achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0006] Get the RGB image to be detected;
[0007] The RGB image is input into the trained image tampering detection network based on the AEA and SR modules, which outputs the tampered / true binary classification prediction results and the segmentation prediction map of the tampered area;
[0008] The image tampering detection network improved based on AEA and SR modules includes:
[0009] An edge perceptron is used to generate an edge prediction map based on the RGB image and extract RGB flow features from the RGB image;
[0010] AEA module, which is used to optimize the synergy between edge perception and tampering segmentation to accurately locate the tampered area;
[0011] The SR module is used to extract weak universal tampering features that are unrelated to visual information from RGB images and output features with the same dimension as the RGB features;
[0012] The dual attention module is composed of the Shift method and the spatial attention mechanism. It is used to splice the RGB features and the irrelevant visual information tampering features processed by the SR module along the channel dimension, select and fuse the features, and finally generate a pixel-level tampering area prediction map;
[0013] The edge perceptron uses four ResNet50 feature (R4) extraction modules to extract RGB stream features from the RGB image. The RGB stream features extracted by each level of ResNet50 block are respectively input into the Sobel layer and the AEA module to extract edge features, and they are input in series into the subsequent attention mechanism as the final edge prediction features, and then the edge prediction map is obtained by converting the output results using the Sigmoid layer.
[0014] The AEA module takes the feature map of the edge features extracted by R4 through 1×1 convolution and each level of ResNet50 feature extraction module as input, extracts multi-scale features in the edge, performs matrix multiplication with the features extracted by R4 through 1×1 convolution, uses the sigmoid function to constrain to between 0 and 1 and acts as a weight on the RGB stream feature map extracted by R4 through 1×1 convolution in the edge perceptron.
[0015] The edge perceptron and AEA module are important components of the upper branch (forged edge enhancement branch), the SR module described below is a component of the lower branch (interference factor weakening branch), and the dual-attention module is used to communicate the upper and lower branches, which can capture the global context in the feature map. In addition, both the upper and lower branches use ResNet50 as the backbone, and the baseline model used is MVSS-Net++.
[0016] The SR module takes the RGB image as input, and performs a subtraction operation on the feature map obtained by a set of resolution reduction blocks and the feature map reconstructed by a set of resolution increase blocks, so that the obtained view reduces the difference between the feature map obtained by the downsampling operation and the feature map reconstructed by the upsampling operation through a supervised learning strategy. Subsequently, the view is subjected to feature extraction by four SR feature extraction modules and outputs a weak universal tampering feature with the same dimension as the RGB feature, which enhances the semantic expression of local features by introducing statistical information of global features and suppresses noise and interference.
[0017] The quasi-dual attention module splices the RGB features of the edge perceptron and the weak tampering features enhanced by the SR module along the channel dimension, performs feature selection and fusion, and generates a pixel-level tampering area prediction map. After global average pooling, the pixel-level tampering area prediction map finally obtains the image-level prediction score and area under the curve.
[0018] The Shift method in the dual attention module does not contain any parameters or arithmetic calculations, and only involves exchanging a small number of channels between adjacent features. It is not a module in deep learning, but just a simple and effective python function.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] Traditional edge detection methods fail to effectively establish the connection between RGB features and edge features, resulting in insufficient utilization of spatial context information and failure to effectively mine unique and effective information at each scale. The present invention introduces a new AEA module, which enables the model to effectively focus on the edges of the tampered area, thereby further improving the ability to locate the tampered area.
[0021] In addition, the noise view method commonly used in existing image tampering detection networks cannot effectively detect tampering performed only on the target image, such as copy-move type. Since copy-move does not introduce new elements, the noise introduced in the original image is almost negligible. Therefore, in this image tampering scenario, the effectiveness of the noise view method is low. The present invention introduces a new SR module, which can effectively eliminate the interference of visual semantic information in the image, while effectively highlighting the weak but ubiquitous tampering trace signals of the image and suppressing possible noise. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 is a step diagram of the image tampering detection of the present invention;
[0024] Figure 2 It is a structural diagram of the AEA module proposed by the present invention for improving the image tampering detection performance;
[0025] Figure 3 It is a structural diagram of the SR module proposed by the present invention for improving the image tampering detection performance;
[0026] Figure 4 It is a diagram of the overall architecture of the model proposed by the present invention for improving the image tampering detection performance;
[0027] Figure 5 It is a comparison of the tampering graph, GroundTruth and the predicted result graph detected in the embodiment of the present invention. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0029] like Figure 1 As shown, the improved image tampering detection network based on the AEA and SR modules in this embodiment specifically includes the following steps:
[0030] S1. Get the RGB image to be detected and extract image features:
[0031] CASIAv2 is used as the training set for the model. It contains two tampering formats, splicing and copy-move, with more than 5,000 images and ground-truth. The RGB image is fed into two pre-trained ResNet50 networks to extract the RGB stream features of the image from CASIAv2. The features extracted by the two trunk branch networks are used for the edge perceptron and the SR module respectively.
[0032] S2. Build AEA and SR modules and integrate them into the network:
[0033] like Figure 2As shown in the figure, the structure of the AEA module is composed of ERB (edge residual block) and x-volution (convolution combined with self-attention). The RGB image is first input into the backbone network ResNet50 to extract the RGB stream features R1, R2, R3 and R4. The RGB features extracted by each feature extraction module are further processed by the Sobel layer and the AEA module to extract edge features. All these edge features are connected in series and used as the final edge prediction features after a series of processing, and finally the edge prediction map is generated through the Sigmoid layer.
[0034] The AEA module receives as input the feature map of the edge features extracted by each ResNet50 feature extraction module, and receives the original image data at the same time, which has a shape of C×H×W, where C represents the number of channels, and H and W represent the height and width of the image. After a series of operations, the image data is converted into a C×1×1 form to reduce the amount of computation and retain spatial information. Next, the data is passed through three convolutional layers to extract image features, and each convolutional layer is followed by a BN layer to accelerate training and improve model performance. The ReLU activation function is applied after each convolutional layer and BN layer to help the network learn more complex features. Finally, after being processed by the two parallel branches of PSSA and convolutional of x-volution, the data is flattened again into a C×H×W form as the final output for subsequent task processing or classification. Among them, the upper branch contains pixel offset self-attention (PSSA), which provides approximate global self-attention features, and the lower branch consists of cascaded convolution and batch normalization, integrating convolution capabilities.
[0035] In the edge perceptron, the RGB stream features processed by the AEA module are matrix multiplied with the original RGB features R4 extracted by 1×1 convolution, and the result is input into the subsequent feature extraction module as the final edge prediction map.
[0036] The upper branch process through the AEA module can be abstracted into the following formula:
[0037]
[0038] Among them, PSSA(X) is the feature map after pixel offset self-attention processing, and W is the final convolution kernel weight.
[0039] like Figure 3As shown in the figure, the SR module is obtained by integrating the SGE block into the ResNet block. The core idea of the SR module is to enhance the semantic expression of local features and suppress noise and interference by introducing statistical information of global features. Specifically, the SGE module first uses global features to approximate the learned semantic vector through a spatial average function. Then, SGE measures the similarity between global semantic features and local features through a simple dot product to obtain the importance coefficient of each feature point. Then, two parameters γ and β of the SGE module are introduced, which are used to adjust the enhancement effect. Finally, each local feature is enhanced by the sigmoid function. After the enhanced feature output, it is then operated in the ResNet process.
[0040] like Figure 4 As shown in the figure, the dual attention module has two serial mechanisms: shift operation (SO) and spatial attention (PA). First, PA selectively updates the features of each position by the weighted sum of all position features. Then SO shifts a small portion of the channel along the four spatial directions, i.e., left, right, up, and down. Pixels out of range are discarded, and the remaining empty pixels are filled with zeros. The outputs of SO and PA are added and convolved to obtain the required feature map.
[0041] The core process of the SR module has the following formula:
[0042] x′ i =x i ·σ(γ·a i +β)
[0043] Among them, x i is a local feature, a i is the importance coefficient of each feature point.
[0044] S3. Use the dataset to train the image tampering detection model:
[0045] According to the preset, the upper branch and the lower branch are input respectively. After a series of feature extraction and feature fusion are performed on the two branches, the final training results are obtained. The image tampering detection network proposed in this paper is built based on PyTorch-1.10.0, Cuda-11.3, and uses the Adam optimizer with a BatchSize of 8. All training processes are run on NVIDIA RTX3090. This invention selects Resnet50 as the backbone and uses ImageNet to initialize its pre-training parameters. The input image is uniformly set to 256×256, and the initial learning rate of the model is set to 1e-4, and the epoch is 150 rounds. After the training is completed, save the weight file of the corresponding epoch number for subsequent testing.
[0046] S4. Use the model to detect and test the tampered area:
[0047] When the edge features and weak universal tampering features of the image are extracted using the well-trained backbone network, the edge information segmentation map of the tampered area is obtained through the edge extractor of the upstream branch. These edge information segmentation maps are fed into the AEA module to guide the detection of the tampered area. At the same time, the RGB stream features of the tampered area boundary are processed by the SR module of the downstream branch, and the weak universal tampering features are spliced along the channel dimension. Then, feature selection and fusion are performed through the dual-attention module to generate a pixel-level tampered area prediction image. Finally, each pixel value of the edge segmentation feature map and the tampered area segmentation map is processed by the sigmoid activation function so that it is between 0 and 1.
[0048] The present invention uses 0.5 as the threshold, and pixels with a value greater than 0.5 are considered tampered pixels and assigned a value of 1, and pixels with a value less than 0.5 are considered non-tampered pixels and assigned a value of 0. Finally, the pixel-level F1 score and the area under the curve AUC score are calculated. The comparison of the detected tampering graph, GroundTruth and predicted result graph obtained during the test is shown in the figure below. Figure 5 shown.
[0049] S5. Evaluate and adjust the improved model:
[0050] First, evaluate the performance of the model on an independent test set to verify its performance on unseen data. Analyze the performance of the model on different categories (such as different types of tampering in images) to identify its strengths and limitations. Then analyze the evaluation results to identify where the model performs well and where it needs improvement. Adjust the model based on the analysis results, which may require adjusting model parameters, adding training data, improving data augmentation methods, etc. Finally, compare the improved model with other state-of-the-art models in the current field to understand its strengths and weaknesses in performance and efficiency.
[0051] Finally, it is noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An improved image tampering detection network based on AEA and SR modules, characterized by: A. Get the RGB image to be detected; B. Input the RGB image into the trained image tampering detection network based on the AEA and SR modules, which outputs the tampered / true binary classification prediction results and the segmentation prediction map of the tampered area; The image tampering detection network improved based on AEA and SR modules includes: An edge perceptron is used to generate an edge prediction map based on the RGB image and extract RGB flow features from the RGB image; AEA module, which is used to optimize the synergy between edge perception and tampering segmentation to accurately locate the tampered area; The SR module is used to extract weak universal tampering features that are unrelated to visual information from RGB images and output features with the same dimension as the RGB features; A dual attention module, which is composed of the Shift method and the spatial attention mechanism, is used to splice the RGB features and the irrelevant visual information tampering features processed by the SR module along the channel dimension, select and fuse the features, and finally generate a pixel-level tampering area prediction map.
2. The image tampering detection network based on the improvement of AEA and SR modules according to claim 1 is characterized in that: The edge perceptron uses four ResNet50 feature (R4) extraction modules to extract RGB stream features from the RGB image. The RGB stream features extracted by each level of ResNet50 block are respectively input into the Sobel layer and the AEA module to extract edge features, and they are input in series into the subsequent attention mechanism as the final edge prediction features, and then the edge prediction map is obtained by converting the output results using the Sigmoid layer.
3. The image tampering detection network based on the improvement of AEA and SR modules according to claim 1 is characterized in that: The AEA module takes the feature map of the edge features extracted by R4 through 1×1 convolution and each level of ResNet50 feature extraction module as input, extracts multi-scale features in the edge, performs matrix multiplication with the features extracted by R4 through 1×1 convolution, uses the sigmoid function to constrain to between 0 and 1 and acts as a weight on the RGB stream feature map extracted by R4 through 1×1 convolution in the edge perceptron.
4. The image tampering detection network based on the improvement of AEA and SR modules according to claim 1 is characterized in that: The edge perceptron and AEA module are important components of the upper branch (forged edge enhancement branch), the SR module described below is a component of the lower branch (interference factor weakening branch), and the dual-attention module is used to communicate the upper and lower branches, which can capture the global context in the feature map. In addition, both the upper and lower branches use ResNet50 as the backbone, and the baseline model used is MVSS-Net++.
5. The image tampering detection network based on the improvement of AEA and SR modules according to claim 1 is characterized in that: The SR module takes the RGB image as input, and performs a subtraction operation on the feature map obtained by a set of resolution reduction blocks and the feature map reconstructed by a set of resolution increase blocks, so that the obtained view reduces the difference between the feature map obtained by the downsampling operation and the feature map reconstructed by the upsampling operation through a supervised learning strategy. Subsequently, the view is subjected to feature extraction by four SR feature extraction modules and outputs a weak universal tampering feature with the same dimension as the RGB feature, which enhances the semantic expression of local features by introducing statistical information of global features and suppresses noise and interference.
6. The image tampering detection network based on the improvement of AEA and SR modules according to claim 1 is characterized in that: The quasi-dual attention module splices the RGB features of the edge perceptron and the weak tampering features enhanced by the SR module along the channel dimension, performs feature selection and fusion, and generates a pixel-level tampering area prediction map. After global average pooling, the pixel-level tampering area prediction map finally obtains the image-level prediction score and area under the curve.
7. The image tampering detection network based on the improvement of AEA and SR modules according to claim 1 is characterized in that: The Shift method in the dual attention module does not contain any parameters or arithmetic calculations, and only involves exchanging a small number of channels between adjacent features. It is not a module in deep learning, but just a simple and effective python function.
Citation Information
Patent Citations
Multi-view multi-scale image tampering detection method
CN113989245A
Image tampering detection method based on shallow feature enhancement and attention mechanism
CN115357944A
Image tampering detection method based on feature fusion and local anomaly
CN117011255A
Image tampering detection method based on irrelevant visual information suppression
CN117315284A
Image tampering detection and positioning method based on multi-scale supervised contrast learning
CN117541571A
