An image tamper detection method based on adaptive spatial feature fusion
By improving the YOLOX network, combining the CBAM attention mechanism and the adaptive spatial feature fusion module, and optimizing the loss function, the accuracy and speed problems of existing image tampering detection methods are solved, and more efficient tampering region localization is achieved.
Patent Information
- Application Number
- CN202210784561.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-05
- Publication Date
- 2026-02-24
- Estimated Expiration
- 2042-07-05
AI Technical Summary
Existing image tampering detection methods suffer from low detection accuracy, inaccurate localization of tampered areas, and slow speed, especially when faced with complex tampering features, they cannot be effectively extracted.
The image tampering detection method based on adaptive spatial feature fusion improves the YOLOX network by combining the CBAM attention mechanism and the adaptive spatial feature fusion module, optimizes the loss function, enhances feature extraction and fusion capabilities, and replaces the IOU loss function with the EIOU loss function to enhance the detection of tampered regions.
It improves the accuracy and speed of image tampering detection, and can more accurately locate tampered areas, especially showing better detection results in complex tampering features.
Smart Images
Figure CN115311204B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image tampering detection, specifically relating to an image tampering detection method based on adaptive spatial feature fusion. Background Technology
[0002] With the development of the times, the transmission of various kinds of information has become increasingly easy, evolving from text to images and videos. Compared to text, people are more willing to believe the content directly conveyed by images and videos. However, due to the widespread use of image processing software, more and more fake images are being widely disseminated online and in society, causing serious negative impacts on the country. Therefore, research on tampering detection technology is of great significance.
[0003] Traditional image tampering detection algorithms often rely on their own characteristics, detecting tampering based on the features generated. Ferrara et al. proposed a method using a Color Filter Array (CFA) algorithm to obtain the correlation between the RGB three channels of an image, capturing differences between images from different sources. Bianchi et al. proposed a method that compares compression traces generated after JPEG image compression with the initial compression features to detect and locate tampered regions. In recent years, with the rapid development of deep learning, many researchers have attempted to apply deep learning to image tampering detection. Zhou et al. proposed a two-stream network method based on Faster R-CNN, using an SRM filter to fuse RGB and noise stream features, effectively improving detection performance.
[0004] However, in practical applications, these methods generally suffer from technical drawbacks such as low accuracy in detecting tampered areas, inaccurate localization, and slow speed. Furthermore, they are unable to effectively extract complex tampering characteristics. Summary of the Invention
[0005] The purpose of this invention is to address the technical problems of large errors and inaccurate localization of tampered areas in existing image tampering detection methods, and to provide an image tampering detection method based on adaptive spatial feature fusion.
[0006] An image tampering detection method based on adaptive spatial feature fusion includes the following steps:
[0007] Step 1: Construct and organize the dataset;
[0008] Step 2: Train the tamper detection network;
[0009] Step 3: Conduct a test on the tamper detection network;
[0010] Step 4: Detect and locate the tampered image;
[0011] In step 1, a batch of synthetic tampered datasets are generated, which include a certain proportion of training and test sets;
[0012] In step 2, since the sample size of the currently available image tampering datasets is generally small, it is not possible to fully train the neural network. Therefore, the improved YOLOX tampering detection network is first pre-trained using a synthetic tampering dataset to obtain a pre-trained model. Based on the pre-trained model, the publicly available tampering dataset is then trained.
[0013] In step 3, the trained network is tested using a pre-defined test set to obtain test results;
[0014] In step 4, the image to be detected is input into the trained network for detection, and the tampered area is output.
[0015] The above steps are used to obtain the tampered area in the image.
[0016] The improved YOLOX tamper detection network structure is as follows:
[0017] Input Layer → Focus Module → First Convolutional Module → First CSP Module → Second Convolutional Module → Second CSP Module → Third Convolutional Module → Third CSP Module → Fourth Convolutional Module → SPP Module → Fourth CSP Module → Fifth Convolutional Module → First Upsampling → First CBAM Attention Module → Channel Fusion of Feature Maps Processed by the First CBAM Attention Module and the Third CSP Module → Fifth CSP Module → Sixth Convolutional Module → Second Upsampling → Second CBAM Attention Module → Channel Fusion of Feature Maps Processed by the Second CBAM Attention Module and the Second CSP Module → Sixth CSP Module → Seventh Convolutional Module → First Downsampling → Third CBAM Attention Module → Channel Fusion of Feature Maps Processed by the Third CBAM Attention Module and the Sixth Convolutional Module → Seventh CSP Module → Eighth Convolutional Module → Second Downsampling → Fourth CBAM Attention Module → Channel Fusion of Feature Maps Processed by the Fourth CBAM Attention Module and the Fifth Convolutional Module → Eighth CSP Module.
[0018] The structure of the CBAM attention module is as follows:
[0019] Input feature map → Channel attention module → The feature map obtained after processing by the channel attention module is multiplied by the input feature map → Channel feature map → Spatial attention module → The feature map obtained after processing by the spatial attention module is multiplied by the channel feature map → Attention feature map.
[0020] The structure of the adaptive spatial feature fusion module is as follows:
[0021] The network outputs three feature maps of different sizes → adaptively fuses feature maps of different levels → predicts the image.
[0022] In step 2, the training of the tamper detection network includes the following sub-steps:
[0023] Step 2-1) Extract the tampering features from the tampered image;
[0024] The tampered image to be detected is input into the backbone network for feature extraction. The backbone network uses CSPDarknet. The image to be detected is input into CSPDarknet (CSP is a module in CSPDarknet). The Focus module is used to segment the input feature map to expand the feature map dimension. Then, it undergoes three convolution operations and CSP module processing to increase the network depth and improve the network's ability to learn target features in the feature map. The feature map after the above processing is then subjected to another convolution operation and SPP spatial pyramid pooling to convert the feature map into a fixed-size feature vector, thereby increasing the network's receptive field. Finally, it is processed by the CSP module again to complete the extraction of tampered features.
[0025] Step 2-2) Fuse the extracted tampering features;
[0026] The extracted tampering features are fed into the PANet neck network for feature fusion, as follows: PANet adds a bottom-up path on top of FPN, which can make fuller use of the feature information of the lower layers. In the bottom-up path, the extracted tampering features are first subjected to a convolution operation, and then upsampled. The feature map obtained after sampling is processed by the CBAM attention mechanism. The processed feature map is then fused with the feature map obtained after processing by the third CSP module in step 2-1) through a channel fusion operation. The resulting feature map is then processed by the CSP module. The above process is repeated. The feature map processed by the second CSP module is fed into the top-down path, subjected to a convolution operation, and then downsampled. The resulting feature map is then processed by the CBAM attention mechanism. The processed feature map is then fused with the feature map obtained after the second convolution operation in the bottom-up path through a channel fusion operation. The resulting feature map is then processed by the CSP module. The above process is then repeated. In the top-down path, the feature maps obtained after processing by the three CSP modules are processed by the adaptive feature fusion module to obtain three feature maps respectively. The three feature maps are then fused to obtain the final prediction map.
[0027] Step 3 includes the following sub-steps:
[0028] Step 3-1) Perform ablation experiments on the pre-trained model;
[0029] Step 3-2) Conduct experiments to compare the results with existing tampering detection algorithms and the YOLOX network on a standard image tampering dataset;
[0030] Step 3-3) Conduct data augmentation experiments.
[0031] Compared with the prior art, the present invention has the following technical effects:
[0032] This invention constructs an image tampering detection network based on YOLOX, combines the CBAM attention mechanism to process feature maps, enhances the extraction of tampering features, and introduces the ASFF adaptive feature fusion mechanism to more fully fuse the extracted multi-scale tampering features. Finally, the loss function in the network is optimized by replacing IOU_loss with EIOU, enabling the network to detect tampered regions more quickly and accurately. Attached Figure Description
[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0034] Figure 1 This is a flowchart of the present invention;
[0035] Figure 2 This is a structural diagram of the CBAM attention module in this invention;
[0036] Figure 3 This is a structural diagram of the adaptive spatial feature fusion module in this invention;
[0037] Figure 4 This is a schematic diagram of the improved YOLOX network structure in this invention;
[0038] Figure 5 This is a diagram illustrating the effect of the present invention in detecting and locating tampered areas. Detailed Implementation
[0039] like Figure 1 As shown, an image tampering detection method based on adaptive spatial feature fusion includes the following steps:
[0040] Step 1: Construct and organize the dataset;
[0041] Step 2: Train the tamper detection network;
[0042] Step 3: Conduct a test on the tamper detection network;
[0043] Step 4: Detect and locate the tampered image;
[0044] In step 1, a batch of synthetic tampered datasets are generated, which include a certain proportion of training and test sets;
[0045] In step 2, since the sample size of the currently available image tampering datasets is generally small, it is not possible to fully train the neural network. Therefore, the improved YOLOX tampering detection network is first pre-trained using a synthetic tampering dataset to obtain a pre-trained model. Based on the pre-trained model, the publicly available tampering dataset is then trained.
[0046] In step 2, "pre-training" is a common technique in this field, such as the "pre-training" in the literature Zhou P, Han X, Morariu VI, et al. Learning Rich Features for Image Manipulation Detection[C] / / 2018IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR).IEEE,2018.
[0047] In step 3, the trained network is tested using a pre-defined test set to obtain test results;
[0048] In step 4, the image to be detected is input into the trained network for detection, and the tampered area is output.
[0049] The above steps are used to obtain the tampered area in the image.
[0050] The improved YOLOX tamper detection network structure described above is as follows:
[0051] Input layer → Focus module → Convolutional module → First CSP module → Convolutional module → Second CSP module → Convolutional module → Third CSP module → Convolutional module → SPP module → Fourth CSP module → Convolutional module → First upsampling → First CBAM attention module → Channel fusion of the feature map obtained from the first CBAM attention module and the feature map obtained from the third CSP module → Fifth CSP module → Convolutional module → Second upsampling → Second CBAM attention module → Channel fusion of the feature map obtained from the second CBAM attention module and the feature map obtained from the second CSP module → Sixth CSP module, which also includes the information flow: after adaptive feature processing... The fusion module processes the data to obtain an 80×80 feature map → convolution module → first downsampling → third CBAM attention module → channel fusion of the feature map obtained after processing by the third CBAM attention module and the feature map obtained after convolution → seventh CSP module, which also includes an information flow: after processing by the adaptive feature fusion module, a 40×40 feature map is obtained → convolution module → second downsampling → fourth CBAM attention module → channel fusion of the feature map obtained after processing by the fourth CBAM attention module and the feature map obtained after convolution → eighth CSP module, which also includes an information flow: after processing by the adaptive feature fusion module, a 20×20 feature map is obtained → prediction map.
[0052] Step 2 includes the following sub-steps:
[0053] Step 2-1) Extract tampering features from the tampered image; input the tampered image to be detected into the backbone network for feature extraction; the backbone network uses CSPDarknet. Input the image to be detected into CSPDarknet, use the Focus module to cut the input feature map to expand the feature map dimension, then perform three convolution operations and CSP module processing to increase the network depth and improve the network's ability to learn target features in the feature map; perform another convolution operation on the feature map after the above processing, and then perform SPP spatial pyramid pooling to convert the feature map into a fixed-size feature vector to improve the network's receptive field; finally, perform CSP module processing again to complete the extraction of tampering features.
[0054] Step 2-2) Fusing the extracted tampering features: The extracted tampering features are fed into the PANet neck network for feature fusion. PANet adds a bottom-up path on top of FPN, which can make fuller use of the feature information of the lower layers. In the bottom-up path, the extracted features are first subjected to a convolution operation, then upsampled, and the resulting feature map is processed by the CBAM attention mechanism. The processed feature map is then fused with the feature map obtained after processing by the third CSP module in Step 2-1), and the resulting feature map is processed by the CSP module. The above process is repeated, and the feature map after the second CSP module processing is fed into the top-down path. After a convolution operation, it is downsampled, and the resulting feature map is then processed by the CBAM attention mechanism. The processed feature map is then fused with the feature map obtained after the second convolution operation in the bottom-up path, and the resulting feature map is processed by the CSP module. The above process is repeated. In the top-down path, the feature maps obtained after processing by the three CSP modules are processed by the adaptive feature fusion module to obtain feature maps of sizes 80×80, 40×40, and 20×20, respectively. The three feature maps are then fused to obtain the final prediction map.
[0055] In step 2, the YOLOX network was optimized as follows:
[0056] (1) Adding a CBAM attention module after upsampling and downsampling in the neck network makes the network pay more attention to abnormal information in the image, improves the ability to extract tampering features in the image, and avoids a significant increase in computational cost. The CBAM attention module is as follows: Figure 2 As shown.
[0057] The CBAM attention module consists of a channel attention module and a spatial attention module. The channel attention module identifies and focuses on important content in each feature map by leveraging the correlations between different feature maps, thereby strengthening important features and suppressing unimportant ones. The spatial attention module compresses the spatial feature map to obtain a one-dimensional channel feature vector before further processing.
[0058] In the channel attention module, average pooling and max pooling are first used to extract features from the feature map. The extracted features are then fed into a shared fully connected layer and summed. Finally, the features are processed by a sigmoid activation function to generate the channel attention feature map. The formula for calculating channel attention is:
[0059] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))
[0060] Where F is the feature map input to the channel attention module, σ is the Sigmoid activation function, and MLP is a multilayer perceptron.
[0061] The role of the spatial attention module is to enhance the feature representation ability of key regions. By generating masks for key locations in the image, it enhances attention to the target region and reduces the perception of irrelevant regions.
[0062] The spatial attention module receives the feature map generated by multiplying the channel attention feature map and the input feature map. It then performs max pooling and average pooling based on the channel dimension, concatenates the results, and finally applies a regular convolution and sigmoid activation function to obtain the spatial attention feature map. The formula for calculating spatial attention is:
[0063] M s (F1)=σ(f 7×7 ([AvgPool(F1);MaxPool(F1)]))
[0064] Where σ is the sigmoid activation function, f 7×7 This refers to a convolutional layer with a kernel size of 7×7.
[0065] The CBAM module combines channel attention and spatial attention and performs weighted fusion of input features, achieving better results than single channel attention. The formula for CBAM is as follows:
[0066]
[0067] In the formula, F is the feature map input to the channel attention module, F1 is the feature map obtained after channel attention, and F2 is the feature map obtained after spatial attention.
[0068] (2) An adaptive spatial feature fusion module is introduced in the feature fusion part. In YOLOX, PANet is used to implement feature fusion, but this method cannot fully utilize the extracted tampered features at different scales; it can only convert the feature maps to the same size and add them together. Therefore, this paper introduces an adaptive spatial feature fusion module, which allows the network to directly filter features from other levels and retain the required tampered features, thus improving the scale invariance of the features. The adaptive spatial feature fusion module is as follows: Figure 3 As shown.
[0069] Multiply the three feature maps at different scales output by YOLOX by their respective weight parameters and sum them to obtain the fused feature maps at three corresponding scales. This can be represented by the following formula:
[0070]
[0071] in, This is the feature map output after processing by ASFF. The weighting coefficients used. These are the features output for Level 1, Level 2, and Level 3, respectively.
[0072] The three feature maps are processed to have the same number of channels using a 1x1 convolution kernel, then upsampled to obtain the same dimension, and finally added together.
[0073] In image tampering detection tasks, some tampered areas are relatively small. For these smaller targets, it is necessary to extract subtle features from the underlying features for identification. By introducing an adaptive spatial feature fusion module, irrelevant features from other feature layers can be removed, while retaining the required tampering features.
[0074] (3) Loss function optimization. The loss function of YOLOX is:
[0075] L = L cls +L obj +L reg
[0076] Among them, L cls For the classification loss function, L obj For the confidence loss function, the binary crossover loss function was used in all cases, L. reg The bounding box loss function used is the IOU loss function. The IOU loss function calculates the ratio of 1 to the intersection and union of the predicted and ground truth bounding boxes, and its formula is as follows:
[0077]
[0078] In the formula, B is the area of the prediction box. i This represents the area of the ground truth bounding box. The IOU loss function typically performs well in representing the detection results of the ground truth and predicted bounding boxes, but it cannot determine the positional relationship between them. When the IOU values are the same, it cannot optimize bounding boxes with different intersection conditions. Therefore, this invention uses the EIOU loss function to replace the original IOU loss function. The EIOU loss function adds the distance between the center points of the two boxes and the minimum bounding box covering both the predicted and ground truth boxes to the IOU loss function, and also calculates the width and height of the target box. The EIOU loss function consists of three parts: overlap loss, center distance loss, and width / height loss. The EIOU loss function formula is as follows:
[0079]
[0080] In the formula, b and b gtρ and h are the center coordinates of the predicted bounding box and the ground truth bounding box, respectively; ρ is the distance between the center coordinates; d is the diagonal distance of the bounding box; ω and h are the width and length of the predicted bounding box, respectively. gt and h gt C represents the width and length of the actual bounding box. ω and C h The bounding box width and length are given. Using the EIOU loss function effectively improves the network's convergence speed and accuracy. The improved YOLOX network structure is as follows: Figure 4 As shown.
[0081] In step 3, the tamper detection network test results are as follows:
[0082] Step 3.1: Perform ablation experiments on the pre-trained model. Average precision (AP) was used as the evaluation metric. The experimental results are shown in Table 1.
[0083] Table 1
[0084]
[0085] Step 3.2: Experiments were conducted on three standard image tampering datasets to compare the results with existing tampering detection algorithms and the YOLOX network, using the F1 score as the evaluation metric. The results are shown in Table 2.
[0086] Table 2
[0087]
[0088] As can be seen from Tables 1 and 2, because the present invention combines an adaptive feature fusion mechanism, makes full use of tampering features at different scales, and introduces an attention mechanism and an EIOU loss function, the network pays more attention to the tampering information in the image, and achieves better detection results on the pre-training dataset and three standard image tampering detection datasets.
[0089] Step 3.3, Data Augmentation Experiments. This invention conducted data augmentation experiments on three standard datasets, using the F1 score as the evaluation metric, and compared three data augmentation operations: horizontal flipping, adding noise, and HVS augmentation. According to the results shown in Table 3, the horizontal flipping data augmentation method achieves the best results.
[0090] Table 3
[0091]
[0092] In step 4, the present invention randomly selects several tampered images that have been modified through copy-move, splicing, and removal methods to locate and detect the tampered areas. The test results are as follows: Figure 5 As shown.
Claims
1. An image tampering detection method based on adaptive spatial feature fusion, characterized in that, It includes the following steps: Step 1: Construct and organize the dataset; Step 2: Train the tamper detection network; Step 3: Conduct a test on the tamper detection network; Step 4: Detect and locate the tampered image; In step 1, a batch of synthetic tampered datasets are generated, which include training and test sets; In step 2, the improved YOLOX tamper detection network is first pre-trained using a synthetic tamper dataset to obtain a pre-trained model. Based on the pre-trained model, the network is then trained on a publicly available tamper dataset. In step 3, the trained network is tested using a pre-defined test set to obtain test results; In step 4, the image to be detected is input into the trained network for detection, and the tampered area is output. The above steps are used to obtain the tampered area in the image; The improved YOLOX tamper detection network structure is as follows: Input Layer → Focus Module → First Convolutional Module → First CSP Module → Second Convolutional Module → Second CSP Module → Third Convolutional Module → Third CSP Module → Fourth Convolutional Module → SPP Module → Fourth CSP Module → Fifth Convolutional Module → First Upsampling → First CBAM Attention Module → Channel Fusion of Feature Maps Processed by the First CBAM Attention Module and the Third CSP Module → Fifth CSP Module → Sixth Convolutional Module → Second Upsampling → Second CBAM Attention Module → Channel Fusion of Feature Maps Processed by the Second CBAM Attention Module and the Second CSP Module → Sixth CSP Module → Seventh Convolutional Module → First Downsampling → Third CBAM Attention Module → Channel Fusion of Feature Maps Processed by the Third CBAM Attention Module and the Sixth Convolutional Module → Seventh CSP Module → Eighth Convolutional Module → Second Downsampling → Fourth CBAM Attention Module → Channel Fusion of Feature Maps Processed by the Fourth CBAM Attention Module and the Fifth Convolutional Module → Eighth CSP Module.
2. The method according to claim 1, characterized in that, The structure of the CBAM attention module is as follows: Input feature map → Channel attention module → The feature map obtained after processing by the channel attention module is multiplied by the input feature map → Channel feature map → Spatial attention module → The feature map obtained after processing by the spatial attention module is multiplied by the channel feature map → Attention feature map.
3. The method according to claim 1, characterized in that, The structure of the adaptive spatial feature fusion module is as follows: three feature maps of different sizes output from the network → adaptive fusion of feature maps at different levels → prediction map.
4. The method according to claim 1, characterized in that, In step 2, the training of the tamper detection network includes the following sub-steps: Step 2-1) Extract the tampering features from the tampered image; The tampered image to be detected is input into the backbone network for feature extraction. The backbone network uses CSPDarknet. The image to be detected is input into CSPDarknet, and the Focus module is used to segment the input feature map to expand the feature map dimension. Then, it undergoes three convolution operations and CSP module processing to increase the network depth and improve the network's ability to learn target features in the feature map. The feature map after the above processing is then subjected to another convolution operation and SPP spatial pyramid pooling to convert the feature map into a fixed-size feature vector, thereby increasing the network's receptive field. Finally, it is processed by the CSP module again to complete the extraction of tampered features. Step 2-2) Fuse the extracted tampering features; The extracted tampering features are fed into the PANet neck network for feature fusion, as follows: PANet adds a bottom-up path on top of FPN, which can make fuller use of the feature information of the lower layers. In the bottom-up path, the extracted tampering features are first subjected to a convolution operation, and then upsampled. The feature map obtained after sampling is processed by the CBAM attention mechanism. The processed feature map is then fused with the feature map obtained after processing by the third CSP module in step 2-1) through a channel fusion operation. The resulting feature map is then processed by the CSP module. The above process is repeated. The feature map processed by the second CSP module is fed into the top-down path, subjected to a convolution operation, and then downsampled. The resulting feature map is then processed by the CBAM attention mechanism. The processed feature map is then fused with the feature map obtained after the second convolution operation in the bottom-up path through a channel fusion operation. The resulting feature map is then processed by the CSP module. The above process is then repeated.
5. The method according to claim 4, characterized in that, In the top-down path, the feature maps obtained after processing by the three CSP modules are processed by the adaptive feature fusion module to obtain three feature maps respectively. The three feature maps are then fused to obtain the final prediction map.
6. The method according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3-1) Perform ablation experiments on the pre-trained model; Step 3-2) Conduct experiments comparing the algorithm with existing tampering detection algorithms and the YOLOX network on a standard image tampering dataset; Step 3-3) Conduct data augmentation experiments.