A method for image tamper detection based on PID-Net

Abstract

The application discloses a kind of image tampering detection methods based on PID-Net, including the following steps: Step1, the preparation and processing of data set;Step2, the construction of tampering model, based on PID-Net network structure, it is applied to image tampering, on this basis, join transformer, and constraint convolution;Step3, the training of tampering model;Step4, tampering model test and evaluation.For the problem that the accuracy is not good at present, traditional algorithm can only be aimed at a certain specific tampering type, application exists limit, and the problem that space feature information is not perfect, there is low-level space information loss, boundary information loss, cause model learning feature difficult, poor universality etc.;Can effectively obtain the area of image anomaly, improve detection performance and accuracy;PID-Net network three branches are used to extract feature mechanism, obtain deep complete spatial feature information and boundary feature information.Lastly, transformer mechanism and constraint convolution are added to enhance the model attention abnormal area feature, strengthen the learning ability of model, improve the accuracy of model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image tampering detection method based on PID-Net. BACKGROUND

[0002] In recent years, the development of various software such as Meitu, Xingtu and Photoshop makes image editing ubiquitous. In daily life, people modify images for the purpose of beautification and entertainment, which does not cause adverse effects, but it cannot be avoided that someone maliciously tampers with images to distort facts or seek benefits, which will affect people's judgment of objective things and bring serious negative effects to the society and the country. Image tampering technology mainly includes copy-move and splicing and removal. Copy-move is to copy a part of an image and paste it in a certain area of the same image; image splicing is to copy a region from a real image and paste it into other images; image removal is to erase a certain area in the image and repair it according to the surrounding pixels. In addition, in order to eliminate the traces left after tampering, related post-processing operations are usually performed to eliminate these differences, such as compression.

[0003] In the field of computer vision, image manipulation detection using deep learning method is a new research direction. Most scholars use image detection and segmentation, and many excellent deep learning frameworks have emerged. Rao et al. use the high-pass filter in the SRM (steganalysis rich model) model to initialize the CNN network, perform steganalysis on the image, effectively suppress the influence of complex image content on manipulation detection and accelerate the fusion speed of the network. Since then, many scholars have used CNN network to realize image manipulation detection and positioning of the tampered area. Zhou et al. use RGB stream and noise stream to extract more rich features, not only use CNN network, but also create the idea of double stream, improve the application effect of the model on image manipulation detection (Zhou P, Han X, Morariu V I, Davis L S. Learning Rich Features for Image Manipulation Detection [C] / / Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018: 1053-1061.). Bayar et al. first use constrained convolution to extract manipulation features, learn boundary differences, effectively suppress irrelevant image semantic information, and focus on manipulation-related information (BAYAR B, STAMM M C. A deep learning approach to universal image manipulation detection using a new convolutional layer [C] / / Proceedings of the 4th ACM Workshop on Information Hiding and Multimedia Security, Vigo, Jun 20-22, 2016. New York: ACM, 2016: 5-10.).Chen et al. use the edge supervision branch to capture the subtle differences between the tampered and non-tampered regions at the boundary; the noise stream branch captures the noise inconsistency between the tampered and non-tampered regions, and combines the boundary features and noise features of the tampered region using the dual-flow idea, learning more general semantic-independent information (Xinru Chen, Chengbo Dong, Jiaqi Ji, Juan Cao, Xirong Li: Image Manipulation Detection by Multi-View Multi-Scale Supervision[J] / / International Conference on Computer Vision (ICCV), 2021.).

[0004] Deep learning-based digital image tamper detection algorithms have made great progress over traditional tamper detection, but have high computational complexity, poor tampered region detection performance, and continuous sampling that can cause tamper feature loss and accuracy degradation. Second, tamper detection is not specific to a particular region of an image, and the model learning complexity is high. Consider giving priority to suspicious areas to improve model efficiency and accuracy. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an image tamper detection method based on PID-Net, which can effectively obtain the abnormal region of the image, improve the detection performance and accuracy, and use the PID-Net network three-branch feature extraction mechanism to obtain deep and complete spatial feature information and boundary feature information. Finally, the transformer mechanism and constraint convolution are added to enhance the model's attention to abnormal region features and improve the model's learning ability and accuracy.

[0006] To solve the above technical problems, the technical solution adopted by the present application is:

[0007] An image tamper detection method based on PID-Net, comprising the following steps:

[0008] Step 1, preparation and processing of data set;

[0009] Step 2, construction of tamper model, based on PID-Net network structure, applied to image tampering, and transformer and constraint convolution are added on this basis;

[0010] Step 3: Tamper with the model's training;

[0011] Step 4: Model tampering testing and evaluation.

[0012] The datasets in Step 1 above include the CASIA dataset, NIST2016 dataset, Columbia dataset, and COVERAGE dataset; the image size is uniformly scaled to 256x256, and the tamper mask images of each dataset are formatted in a uniform way.

[0013] The steps for constructing the tampered model in Step 2 above are as follows:

[0014] Step 2.1: Construct a network structure based on PID-Net. The PID-Net network is a three-branch network structure. The three branches are responsible for obtaining detailed information, boundary information, and context information, respectively.

[0015] Step 2.2: Integrating transformer and adopting Multi-Attention can enhance the tampering information, which is beneficial for the model to learn the tampering features and ignore semantic information that is not related to the tampering.

[0016] Step 2.3: Using constrained convolution, learn to modify relevant semantic information while suppressing irrelevant semantic information, learn to modify features, and suppress the semantic information of the image that is of interest.

[0017] In Step 2.1 above, the PID-Net network is built on a three-branch system. One branch acquires detailed information from the image feature map, another branch acquires contextual information, and the last branch predicts the boundary region. The three branches operate in parallel, and their feature maps need to be fused. After an image is input, a convolution operation is performed first, with a kernel size of 3×2. Each convolution is followed by a BatchNorm2d and ReLU operation. Next are the three branches, each employing K... p e(t), p, i, d represent three branches, K p K represents the proportionality coefficient. i K represents the integration time. d The differential time is represented by e(t), which represents the difference between the input and output values. The three branches perform proportional, integral, and differential operations respectively. Finally, the outputs of the three branches are fused together to obtain a more comprehensive image feature map.

[0018] The multi-head attention mechanism of the Multi-Attention module in Step 2.2 above is composed of multiple self-Attention mechanisms. The multi-head attention mechanism has multiple Query, Keys, and Values vectors, and each Z matrix is calculated respectively. Finally, the Z matrices are spliced. The multi-head attention mechanism can prevent model overfitting and learn different features in multiple aspects, facilitating model learning of tampering features:

[0019]

[0020] MultiHead(Q,K,V)=Concat(head1,…,head h )W Q (2)

[0021] wherehead i =Attention(QW i Q ,kW i k ,VW i V Y (3)

[0022] The process of the multi-head attention mechanism is as follows: first, input an image and divide the image into multiple individual blocks; second, multiply by the weight W i to obtain the input vector, forming Q i , K i , and V i ; then calculate the Attention weight matrix; and finally, splice the matrix.

[0023] The constraint convolution kernel in Step 2.3 above is set as the kernel center weight being -1, the sum of the weights of the remaining positions being normalized to 1, and the sum of the entire kernel parameter values being 0. The constraint convolution kernel is set as follows:

[0024]

[0025] where w k is the kth convolution kernel of the constraint convolution layer, and (0,0) is the center position coordinate of w k . The PID-Net three branches are added based on the constraint convolution, and a single image is subjected to convolution operation and then sent to the constraint convolution and then to the PID-Net three branches.

[0026] The specific process of training the tampering model in Step 3 above is as follows:

[0027] Load the training and validation data sets, set the hyperparameters of the experiment batchsize, learning rate, whether to use GPU, and the learning rate decay method, loss function, optimizer selection, and training period;Batchsize is set to 6, the learning rate is 0.0001, the GPU is set to TRUE, the learning rate is 10 times the learning rate when the training period is 100, the loss function is cross-entropy loss, the optimizer is selected as adam, and the training period is set to 160;The best weight information in the training process is saved for model effect evaluation and prediction.

[0028] The effect test and result evaluation of the tampering model in the above Step4 are as follows:

[0029] Step4.1, by loading the model weight file, test on the test set, use F1, AUC as model evaluation index, evaluation index formula as follows:

[0030]

[0031]

[0032]

[0033]

[0034] M represents the number of positive samples, N represents the number of negative samples, P represents the probability;TP is actually positive, and the prediction is also positive, FP is actually negative, and the prediction is positive, TN is actually negative, and the prediction is negative, FN is actually positive, and the prediction is negative;

[0035] Step4.2, the robustness experiment of the model, the robustness experiment makes the picture blurred by JPEG compression and Gaussian blur, reduces the definition;JPEG compression processing, four quality factors are set, which are 20, 40, 60 and 80, the smaller the quality factor, the lower the image definition;Gaussian blur sets three different size Gaussian kernels, which are 1, 5 and 7, the larger the size, the worse the image definition;Load the model of images with different size quality factors and different size Gaussian kernels, and judge whether the model has good robustness by whether the evaluation index is stable.

[0036] The image tampering detection method based on PID-Net provided by the application has the following beneficial effects:

[0037] (1) Based on the three-branch structure of PID-Net, the model can effectively obtain boundary information, avoid spatial information loss, and learn tampering features.

[0038] (2) The traditional tampering detection is limited to a certain specific type, and the model of the application can detect three types of tampered images, namely splicing, copy-paste and deletion.

[0039] (3) Based on Multi-Attention and constraint convolution, the tampering semantic information is enhanced, the image semantic information irrelevant to tampering is inhibited, the image tampering feature is enhanced, the model pays more attention to the tampering feature and ignores irrelevant semantic information, and the precision and accuracy of the model are improved. BRIEF DESCRIPTION OF DRAWINGS

[0040] The application will be further described below in combination with the drawings and examples:

[0041] Figure 1 The flowchart of the application;

[0042] Figure 2 The PID-Net three-flow branch structure diagram of the application;

[0043] Figure 3 The Multi-Attention structure diagram of the application. DETAILED DESCRIPTION

[0044] The technical solutions of the application will be described in detail below in combination with the drawings and examples.

[0045] As shown in the Figure 1 The flow of the image tampering detection method based on PID-Net is as follows:

[0046] Step 1: preparation and processing of the data set;

[0047] Step 2: construction of the tampering model, based on the PID-Net network structure, which is applied to image tampering, and then transformer and constraint convolution are added on this basis;

[0048] Step 3: training of the tampering model;

[0049] Step 4: testing and evaluation of the tampering model.

[0050] In step 1: preparation and processing of the data set: the tampering field generally uses public data sets at present, including CASIA data set, NIST2016 data set, Columbia data set and COVERAGE data set; first, the tampered areas of the masks of various data sets are white or black, which are not unified and need to be unified; second, the picture size is scaled to 256x256, and horizontal flip, standardization processing and normalization are performed; finally, the data set needs to be divided into test data set, verification data set and training data set.

[0051] In step 2: the construction of the tampering model is specifically operated as follows:

[0052] 2.1 Based on the construction of the three branches of PID-Net; one branch obtains detailed information in the image feature map, one branch obtains context information, and the last branch predicts the boundary region; the three branches are parallel, and the feature maps of the three branches need to be fused; after inputting an image, first perform convolution operation, the size of the convolution kernel is 3x2, and after each convolution, there is a BatchNorm2d and ReLu operation, followed by three branches, three branches respectively adopt K p e(t), p, i, d represent three branches, K p represents the proportional coefficient, K i represents the integral time, and K d represents the differential time, and e(t) represents the difference value between the input and the output value, three branches respectively perform proportional, integral and differential operations; finally, the output results of the three branches are fused, and the characteristic information of the three branches is fused with each other to obtain a more comprehensive image feature map information.

[0053] 2.2 Multi-Attention module; Multi-Attention multi-head attention mechanism is composed of multiple self-Attention self-attention mechanisms, and the multi-head attention mechanism has multiple Query, Keys, Values vectors, and then calculates the respective Z matrix, and finally the Z matrix is spliced; by adding the multi-head attention mechanism, the overfitting of the model can be prevented, and since the amount of data set disclosed by the present application is small, a large amount of training will cause overfitting, and the multi-head attention mechanism has multiple Attention to perform the calculation task, which is an integrated left and right, which can effectively avoid overfitting; secondly, the Dropout in the multi-head attention mechanism can randomly discard neurons by setting the value, and the multi-head attention can learn different features in multiple aspects, which is conducive to the model learning tampering features:

[0054]

[0055] MultiHead(Q,K,V)=Concat(head1,…,head h )W Q (2)

[0056] wherehead i =Attention(QW i Q ,kW i k ,VW i VY (3)

[0057] The process of the multi-head attention mechanism is as follows: first, an image is input, and the image is divided into a plurality of patches; second, the image is multiplied with a weight W i to obtain an input vector to form Q i , K i , V i ; then, an attention weight matrix is calculated, and finally, the matrix is spliced; the feature map after the PID-Net three-branch fusion is sent into the Multi-Attention module, so as to obtain an enhanced feature map.

[0058] 2.3 Constraint convolution module; generally, the ordinary convolution operation acts on the semantic information of the image, and the tampered semantic information is different from the image semantic information; in some cases, the original semantic information of the image will interfere with the tampered semantic information, so that the expression of the image semantic information is suppressed, so that the model learns the features related to tampering; the constraint convolution is different from the ordinary convolution, that is, the constraint convolution prevents the convolutional neural network from becoming an arbitrary filter by applying a constraint in the weight iteration process, so that the filter becomes a filter that suppresses the content information irrelevant to image tampering; the setting of the constraint convolution kernel is generally that the center weight of the kernel is -1, the sum of the weights of the other positions is normalized to 1, and the sum of the entire kernel parameter values is 0; the constraint convolution kernel setting mode is as follows:

[0059]

[0060] wherein w k is the kth convolution kernel of the constraint convolution layer, and (0, 0) is the center position coordinate of w k ; the constraint convolution is added based on the PID-Net three-branch respectively, and a convolution operation is performed on an image, and then the image is sent into the constraint convolution and then into the PID-Net three-branch.

[0061] In step 3: training of the tampering model; the training and verification data sets are loaded, and the experimental hyperparameters such as batchsize, learning rate, whether to use GPU, and the learning rate decay method, loss function, selection of the optimizer, and training period are set; the batchsize is set to 6, the learning rate is 0.0001, the use of GPU is set to TRUE, the learning rate is reduced by 10 times when the training period is 100, the loss function is cross-entropy loss, the optimizer is selected as adam, and the training period is set to 160; the best weight information in the training process is saved for the evaluation and prediction of the model effect.

[0062] In step 4: the effect test and result evaluation steps of the tampering model are as follows:

[0063] 4.1 Through loading model weight file, test on test set, the application uses F1, AUC as model evaluation index, and the evaluation index formula is as follows:

[0064]

[0065]

[0066]

[0067]

[0068] M represents the number of positive samples, N represents the number of negative samples, P represents the probability; TP is actually positive, and the prediction is also positive, FP is actually negative, and the prediction is positive, TN is actually negative, and the prediction is negative, FN is actually positive, and the prediction is negative;

[0069] 4.2 Robustness experiment is performed on the model of the application. The robustness experiment generally makes the picture blurred by JPEG compression and Gaussian blur to reduce the definition. The JPEG compression processing is set with four quality factors, which are 20, 40, 60 and 80. The smaller the quality factor, the lower the image definition. The Gaussian blur is set with three different sizes of Gaussian kernels, which are 1, 5 and 7. The larger the size, the worse the image definition. The images with different sizes of quality factors and different sizes of Gaussian kernels are loaded into the model of the application, and whether the evaluation index is stable is used to judge whether the model of the application has good robustness.

[0070] Embodiment:

[0071] The specific steps of the input data set NIST2016 are as follows:

[0072] Step 1: The development program language of the application is python language, and pytorch is used to realize the specific algorithm;

[0073] Step 2: Process the NIST2016 data set: 1) The NIST2016 data set has 564 pictures, and the picture size is different. The mask picture is white area for tampering area; uniformly scale the data set to 256x256, and convert the mask picture to black for tampering area, so as to achieve consistency with other data sets; 2) Finally, divide the data set into training set, verification set and test data set according to the ratio of 8:1:1; 2) horizontally flip the data set to increase the number of data sets, avoid overfitting, standardize and normalize the image number to tensor format, so as to facilitate subsequent operation;

[0074] Step 3: After the image processing is completed, it is sent into the model, 1) the sent image is first sequentially subjected to convolution, BatchNorm2d, Relu, convolution, BatchNorm2d and downsampling operation, and the generated feature map enters three parallel branches; 2) the feature maps sent into the three branches are subjected to branch acquisition of detailed information, edge information and context information, and instead of simple convolution operation, constrained convolution is adopted to suppress the expression of image semantic information, to obtain tampering-related feature maps, and the three branches respectively obtain tampering detailed information, tampering edge information and tampering context information, and the three analysis features obtained are fused; 3) finally, the fused feature map is input into the Multi-Attention module, an enhanced attention feature map is output, and the result is output by the prediction head part.

[0075] Step 4: Set the experimental hyperparameters such as batchsize, learning rate, whether to use GPU, and learning rate decay method, loss function, optimizer selection, and training period; the batchsize is set to 6, the learning rate is 0.0001, the use of GPU is set to TRUE, the learning rate is 10 times the learning rate decay when the training period is 100, the loss function is cross-entropy loss, the optimizer is selected as adam, and the training period is set to 160.

[0076] Step 5: Use the weight information saved during the training process in step 3 to predict and evaluate the model; 1) first load the prediction data set, send it into the network, then load the model weight information, compare the output mask with the real mask, evaluate the effect of the model, and output the evaluation result; 2) in order to verify the robustness of the model, the test data set is subjected to JPEG compression and Gaussian blur processing, and then sent into the model for prediction and evaluation.

[0077] The results on the input NIST2016 data set show that the model of the present patent has good effect, the F1 of NIST2016 data set in ManTra-Net model is 0.632, and the model of the present patent has 0.735, which is about 0.1 higher than that; in addition, the NIST2016 data set in RGB-N model has 0.663, and the model of the present patent is also higher than that, which shows that the model of the present patent has obvious advantages.

[0078] The patent proposes an image tampering detection method model based on PID-Net, which can accurately identify the tampering technology and accurately mark the tampered area; through the PID-Net three-branch structure, Multi-Attention and constraint convolution, the model is no longer limited to a certain tampering type, has good generalization ability, obtains the boundary information of the tampered area and the non-tampered area, makes the tampering boundary more accurate, and then suppresses the semantic information irrelevant to the tampering and enhances the tampering feature; in the future, more accurate methods can be used to mark the tampered area, and the universality of the model can be verified on more tampering data sets.

Claims

1. An image tampering detection method based on PID-Net, characterized in that, Includes the following steps: Step 1: Preparation and processing of the dataset; Step 2: Construction of the tampering model. Based on the PID-Net network structure, it is applied to image tampering. On this basis, a transformer and constrained convolution are added. The steps for constructing the tampering model are as follows: Step 2.1: Construct a network structure based on PID-Net. The PID-Net network is a three-branch network structure. The three branches are responsible for obtaining detailed information, boundary information, and context information, respectively. The PID-Net network is built on a three-branch architecture. One branch acquires detailed information from the image feature map, another branch acquires contextual information, and the last branch predicts boundary regions. These three branches operate in parallel, requiring the feature maps from all three branches to be fused. After an image is input, convolution operations are performed first, with a kernel size of 3×2. Each convolution is followed by a BatchNorm2d and ReLU operation. Next come the three branches, each employing K... p e(t), , p, i, d represent three branches, K p This represents the proportionality coefficient. , e(t) represents the difference between the input and output values. The three branches perform proportional, integral, and differential operations respectively. Finally, the outputs of the three branches are fused together to obtain a comprehensive image feature map. Step 2.2: Integrating transformer and adopting Multi-Attention can enhance the tampering information, which is beneficial for the model to learn the tampering features and ignore semantic information that is not related to the tampering. Step 2.3: Using constrained convolution, learn to modify relevant semantic information while suppressing irrelevant semantic information, learn to modify features, and suppress the semantic information of the image that is of interest. Step 3: Tamper with the model's training; Step 4: Model tampering testing and evaluation.

2. The image tampering detection method based on PID-Net according to claim 1, characterized in that, The datasets in Step 1 include the CASIA dataset, NIST2016 dataset, Columbia dataset, and COVERAGE dataset; the image size is uniformly scaled to 256x256, and the tamper mask images of each dataset are formatted in a uniform way.

3. The image tampering detection method based on PID-Net according to claim 2, characterized in that, The multi-attention mechanism in Step 2.2 consists of multiple self-attention mechanisms. Each multi-attention mechanism has multiple query, key, and value vectors, and each calculates its own Z matrix. Finally, the Z matrices are concatenated. This multi-attention mechanism can prevent overfitting and learn multiple different features, making it easier for the model to learn modified features. (2) (3) The multi-head attention mechanism works as follows: First, an image is input and divided into multiple individual blocks; then, it is connected to weights W. i Multiplying them yields the input vector, forming Q. i K i V i Next, the Attention weight matrix is ​​calculated, and finally the matrices are concatenated.

4. The image tampering detection method based on PID-Net according to claim 3, characterized in that, In Step 2.3, the constrained convolution kernel is set with a center weight of -1, and the sum of the weights at the remaining positions is normalized to 1, resulting in a total kernel parameter sum of 0. The convolution kernel setting method for constrained convolution is as follows: ；(4) in, The k-th convolutional kernel of the constrained convolutional layer is (0,0). The center coordinates are determined by adding constrained convolution based on the three branches of PID-Net. An image is convolved, then fed into constrained convolution, and then into the three branches of PID-Net.

5. The image tampering detection method based on PID-Net according to claim 1, characterized in that, The specific process of tampering with the model training in Step 3 is as follows: Load the training and validation datasets, and set the hyperparameters for the experiment: batch size, learning rate, whether to use GPU, learning rate decay method, loss function, optimizer selection, and training cycle. Set the batch size to 6, the learning rate to 0.0001, set GPU use to TRUE, the learning rate to decay by 10 times when the training cycle is 100, the loss function to cross-entropy loss, the optimizer to Adam, and the training cycle to 160. Save the weight information with the best performance during training for model performance evaluation and prediction.

6. The image tampering detection method based on PID-Net according to claim 5, characterized in that, The specific steps for testing and evaluating the effect of tampering with the model in Step 4 are as follows: Step 4.1: Load the model weight file and test it on the test set. Use F1 and AUC as model evaluation metrics. The evaluation metric formulas are as follows: M represents the number of positive samples, N represents the number of negative samples, and P represents the probability; TP means the actual result is positive and the prediction is also positive, FP means the actual result is negative and the prediction is positive, TN means the actual result is negative and the prediction is negative, and FN means the actual result is positive and the prediction is negative. Step 4.2: Robustness experiments were conducted on the model. The robustness experiments involved blurring the images using JPEG compression and Gaussian blurring to reduce their sharpness. JPEG compression was performed with four quality factors: 20, 40, 60, and 80; the smaller the quality factor, the lower the image sharpness. Gaussian blurring was performed with three different Gaussian kernel sizes: 1, 5, and 7; the larger the size, the worse the image sharpness. Images with different quality factors and Gaussian kernel sizes were loaded into the model, and the stability of the evaluation metrics was used to determine whether the model had good robustness.

Images