An image tampering detection method based on cross-window self-attention correlation network

Abstract

The invention discloses an image tampering detection method based on a cross-window self-attention correlation network, and relates to the technical field of image recognition. S1: prepare a data set: tampered images and real images used for model pre-training are selected from the MS COCO data set, and the data sets used for fine-tuning the model are CASIAv2.0, Columbia and NIST16 data sets; S2: design a network structure based on cross-window self-attention correlation, including local and global feature extraction modules and prediction modules; S3: design a loss function, the local, global loss function and BCE loss function in the network structure mentioned in step S2, the loss functions on four scales are added and averaged to obtain a final loss function; S4: input the training image and the mask generated by it into the network in S2 for training, generate an image tampering detection pre-training and fine-tuning model and save it; S5: input an image to be detected, use the image tampering detection model trained in step S4 for prediction, and return the result of image tampering detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to an image tampering detection method based on a cross-window self-attention correlation network. Background Art

[0002] With the rapid development of modern science and technology, everyone can use image editing software such as Photoshop to tamper with images according to their own ideas. It is often difficult for people to tell whether the tampered images are tampered images by human eyes. Although the development of image editing technology has facilitated people's daily lives in some aspects, it has also brought many serious negative consequences at the same time, such as tampering with images of important materials such as certificates, bills, contracts, and transaction records. These tampering of images may cause serious consequences. In order to solve this problem, a superior image tampering detection technology is needed.

[0003] In order to cope with the increasingly serious situation of malicious image tampering, many research methods on image tampering detection have emerged. In the early stage, the method proposed by Mahdian et al. had great limitations and low detection accuracy. In recent years, Zhuang et al. proposed advanced tampering detection and positioning methods, which can not only detect a variety of tampering types, but also locate the tampered area at the pixel level. The superiority of the models proposed by these methods is also particularly prominent. Although the existing models are relatively excellent, there are still two problems in this field:

[0004] 1) Low utilization of local and global features: Because the sizes of tampered regions in tampered images are often different, it is difficult to detect the sizes of different tampered regions during the detection process. The importance of scale changes has been ignored in many previous works. The methods proposed by Zhou and Wu et al. are difficult to utilize local and global features at the same time, so they are limited to processing limited scale changes.

[0005] 2) Attention mechanism: Attention mechanisms applied to the field of image tampering detection include channel attention, spatial attention, mixed domain attention, etc. These attention mechanisms often have problems such as high computational complexity, large number of parameters, and limited spatial coverage of calculations, which results in poor feature extraction. It is an urgent problem to reduce the overall computational complexity while expanding the receptive field.

[0006] Therefore, it is necessary to provide an image tampering detection method based on a cross-window self-attention correlation network to solve the above problems. Summary of the invention

[0007] The purpose of the present invention is to provide an image tampering detection method based on a cross-window self-attention correlation network to solve the problem of identifying a large number of forged images currently appearing on the Internet and in the media.

[0008] To achieve the above object, the present invention provides an image tampering detection method based on a cross-window self-attention correlation network, comprising the following steps:

[0009] S1: Prepare pre-training datasets and datasets for fine-tuning the model. The pre-training datasets provided for model training are selected from the MS COCO dataset, which contain tampered images and real images. The datasets used for fine-tuning the model are CASIAv2.0, Columbia, and NIST16 datasets.

[0010] S2: Design a network structure based on cross-window self-attention, including local and global feature extraction modules and prediction modules;

[0011] S3: Design the loss function, the local loss function, the global loss function and the BCE loss function in the network structure mentioned in step S2, add and average the above local loss function, global loss function and BCE loss function to get the final loss function to optimize the model;

[0012] S4: Input the training image and its generated mask into the network in S2 for training, generate and save the image tampering detection pre-trained model and fine-tuning model;

[0013] S5: Input the tampered image to be detected, use the image tampering detection model trained in step S4 to make predictions, and return the result of the image tampering detection.

[0014] Preferably, in step S1, the pre-training data set includes 376,742 tampered images and real images in total, and the tampered image types include splicing, copy-move and deletion.

[0015] Preferably, in step S1, the CASIAv2.0 dataset includes 5123 tampered images, and the tampered image types include splicing and copy-move; Columbia includes 180 tampered images, and the tampered image types include splicing and copy-move; NIST16 dataset includes 564 tampered images, and the tampered image types include splicing, copy-move and deletion. The CASIAv2.0 dataset, Columbia and NIST16 datasets are divided into training sets and test sets at a ratio of 9:1. All training sets in the CASIAv2.0 dataset, Columbia and NIST16 dataset constitute a training set set, and all test sets in the CASIAv2.0 dataset, Columbia and NIST16 dataset constitute a test set set. All tampered images in the training set set and the test set set are input into the network in S2, so that each image generates a corresponding Mask. Preferably, in step S2, the network structure based on cross-window self-attention correlation includes two branches, namely, a local and global feature extraction module and a prediction module; specifically including:

[0016] S21: In the local and global feature extraction module, local and global features of the tampered image in the training set are extracted, and the encoder of the backbone network extracts local and global features from the tampered image in the input training set for subsequent use;

[0017] S22: In the prediction module, it is detected whether the image input in S21 is a tampered image, and four masks are predicted according to the local and global features extracted in step S21. The masks have different scales from small to large. In order to obtain more accurate tampering positioning, each mask is set as a priori for the next scale prediction;

[0018] S23: In the prediction module, in order to obtain a better predicted mask, a cross-window self-attention CWSA module is added before the mask prediction of each scale. Self-attention calculations are performed in the local window and the translated local window respectively, and the information flow between windows is increased by window interaction. The calculation of the cross-window self-attention module is:

[0019]

[0020]

[0021] In the formula, and They represent the output features of the window multi-head attention module and the cross-window multi-head attention module after the LN layer, R s and R s+1 They represent the output features of the MLP module after the LN layer, and W-MSA and CW-MSA represent the multi-head self-attention modules.

[0022] Preferably, in step S3, the loss function is expressed as:

[0023] BCELoss=-{y*log(p)+(1-y)*log(1-p)}

[0024] Among them, p is the predicted probability, y is the true value, when y = 1, it means that the true value is predicted as a positive sample, and when y = 0, it means that the true value is predicted as a negative sample.

[0025] Preferably, in step S4, the pre-training data set and the data set for fine-tuning the model are respectively input into the network structure in step S2, the number of training iterations is specified, the training is performed in batches, and the optimal model result is saved.

[0026] Preferably, in the prediction stage of step S5, the tampered image is detected using the optimal image tampering detection model obtained in step S4 to obtain a predicted binary mask image.

[0027] Therefore, the present invention adopts the above-mentioned image tampering detection method based on cross-window self-attention correlation network, which has the following beneficial effects:

[0028] (1) The present invention provides an image tampering detection method based on a cross-window self-attention correlation network, which combines local and global feature extraction modules so that the model can learn better feature information and enhance the feature expression ability of the network.

[0029] (2) The present invention adds a cross-window self-attention (CWSA) module, which enables the model to focus on the effective features of the tampered area. The ability to extract effective features is further improved, laying a solid foundation for the model to make more accurate predictions.

[0030] (3) The present invention uses window interaction to increase the information flow between windows, thereby increasing the spatial coverage of attention calculation, expanding the receptive field, providing more useful information for subsequent prediction tasks, and further helping the model to better perform detection and positioning tasks.

[0031] (4) The present invention achieves better performance on three standard datasets (CASIA2.0, NIST2016, COLUMBIA). In addition, it has good robustness under JPEG compression attack and noise destruction attack.

[0032] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1It is a flow chart of an image tampering detection method based on a cross-window self-attention correlation network of the present invention;

[0034] Figure 2 It is a network structure diagram of an image tampering detection method based on a cross-window self-attention correlation network of the present invention;

[0035] Figure 3 It is a device module block diagram of an image tampering detection method based on a cross-window self-attention correlation network of the present invention;

[0036] Figure 4 This is a result diagram of an image tampering detection method based on a cross-window self-attention correlation network of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0038] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0039] The words "include" or "comprises" and the like used in the present invention mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of also including other elements. The orientation or position relationship indicated by the terms "inside", "outside", "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. When the absolute position of the described object changes, the relative position relationship may also change accordingly. In the present invention, unless otherwise clearly specified and limited, the terms "attachment" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral body; it can be directly connected, or indirectly connected through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0040] Embodiment 1

[0041] like Figure 1 As shown, the present invention provides an image tampering detection method based on a cross-window self-attention correlation network, comprising the following steps:

[0042] S1: prepare a pre-training dataset and a dataset for fine-tuning the model. The pre-training dataset provided for model training is selected from the MS COCO dataset, which contains tampered images and real images. The datasets used for fine-tuning the model are CASIAv2.0, Columbia, and NIST16 datasets. In step S1, the pre-training dataset includes tampered images and real images, totaling 376,742 images. The types of tampered images include splicing, copy-move, and deletion.

[0043] In step S1, the CASIAv2.0 dataset includes 5123 tampered images, and the tampered image types include splicing and copy-move; the Columbia dataset includes 180 tampered images, and the tampered image types include splicing and copy-move; the NIST16 dataset includes 564 tampered images, and the tampered image types include splicing, copy-move and deletion. The CASIAv2.0 dataset, Columbia and NIST16 datasets are divided into training sets and test sets at a ratio of 9:1. All training sets in the CASIAv2.0 dataset, Columbia and NIST16 dataset constitute a training set set, and all test sets in the CASIAv2.0 dataset, Columbia and NIST16 dataset constitute a test set set. All tampered images in the training set set and the test set set are input into the network in S2, so that each image generates a corresponding Mask.

[0044] S2: Design a network structure based on cross-window self-attention, including local and global feature extraction modules and prediction modules; the real image obtains useful image features after local and global feature extraction modules, and then inputs them into the prediction module for processing. The prediction module makes predictions after receiving the features and finally outputs the final binary mask image.

[0045] In step S2, the network structure based on cross-window self-attention includes two branches, namely the local and global feature extraction module and the prediction module, as shown in Figure 2 As shown; specifically including:

[0046] S21: In the local and global feature extraction module, local and global features of the tampered image in the training set are extracted, and the encoder of the backbone network extracts local and global features from the tampered image in the input training set for subsequent use;

[0047] S22: In the prediction module, it is detected whether the image input in S21 is a tampered image, and four masks are predicted according to the local and global features extracted in step S21. The masks have different scales from small to large. In order to obtain more accurate tampering positioning, each mask is set as a priori for the next scale prediction;

[0048] S23: In the prediction module, in order to obtain a better predicted mask, a cross-window self-attention CWSA module is added before the mask prediction of each scale. Self-attention calculations are performed in the local window and the translated local window respectively, and the information flow between windows is increased by window interaction. The calculation of the cross-window self-attention module is:

[0049]

[0050] In the formula, and They represent the output features of the window multi-head attention module and the cross-window multi-head attention module after the LN layer, R s and R s+1 They represent the output features of the MLP module after the LN layer, and W-MSA and CW-MSA represent the multi-head self-attention modules.

[0051] S3: Design the loss function. The local loss function, global loss function and BCE loss function in the network structure mentioned in step S2 are added and averaged to obtain the final loss function to optimize the model. In step S3, the loss function is expressed as:

[0052] BCELoss=-{y*log(p)+(1-y)*log(1-p)}

[0053] Among them, p is the predicted probability, y is the true value, when y = 1, it means that the true value is predicted as a positive sample, and when y = 0, it means that the true value is predicted as a negative sample.

[0054] S4: The training image and its generated mask Mask are input together into the network in S2 for training, and the image tampering detection pre-trained model and fine-tuning model are generated and saved; in step S4, the pre-training data set and the data set for fine-tuning the model are respectively input into the network structure in step S2, the number of training iterations is specified, the training is performed in batches, and the optimal model result is saved.

[0055] During training, the Adam training optimizer was used. The batch size was 4 and the initial learning rate was 2e-4. The learning rate was halved every 5 times, and the total number of epochs was 25. All training processes were performed on NVIDIA's GP104 [GeForce GTX1070] GPU.

[0056] S5: Input the tampered image to be detected, use the image tampering detection model trained in step S4 to make predictions, and return the result of image tampering detection. In the prediction stage of step S5, the tampered image is detected using the optimal image tampering detection model obtained in step S4 to obtain a predicted binary mask image.

[0057] Test Example

[0058] Two metrics are used: AUC and F1 score to test this scheme on three datasets: CASIAv2.0, Columbia and NIST16;

[0059] Experiments show that this method obtains the best scores and the best positioning performance. Compared with PSCC-Net, the AUC indicators on the three datasets are improved by 0.8%, 1.1% and 1.2% respectively. It is also higher than the AUC of other methods in the table. As shown in Table 1, this solution is the most advanced in terms of model generalization ability. As shown in Table 2, on the fine-tuning model, this method uses the network weights of the pre-trained model to initialize the fine-tuning model trained on the training splits of the two datasets CASIA v2.0 and NIST16.

[0060] Table 1

[0061] Method CASIAv2.0 Columbia NIST16 ManTra-Net 81.7 82.4 79.5 SPAN 79.7 93.6 84.0 PSCC-Net 81.9 97.4 84.2 The present invention 82.7 98.5 85.4

[0062] Compared with PSCC-Net, the AUC index on the three datasets increased by 0.8%, 1.1% and 1.2% respectively. It is also higher than the AUC of other methods in the table. In terms of model generalization ability, the method proposed in this paper is the most advanced.

[0063] Table 2

[0064] Method CASIAv2.0 NIST16 J-LSTM - / - 76.4 / - H-LSTM - / - 79.4 / - RGB-N 79.5 / 40.8 93.7 / 72.2 SPAN 83.8 / 38.2 96.1 / 58.2 PSCC-Net 86.7 / 54.3 98.5 / 73.4 NCL 86.4 / 59.8 91.2 / 83.1 The present invention 87.8 / 60.7 98.7 / 79.7

[0065] The results of this solution under the fine-tuned model are represented by the AUC / F1 score. The NCL model is not pre-trained and fine-tuned.

[0066] In the fine-tuning model, this scheme uses the network weights of the pre-trained model to initialize the fine-tuning model trained on the training splits of the CASIA v2.0 and NIST16 datasets. As shown in Table 2, in different model comparisons, the model proposed by this scheme is better than other models in terms of AUC indicators. In terms of F1 score, the model proposed by this scheme is also ahead of other compared model performances, only slightly lower than the NCL model on the NIST16 dataset.

[0067] Therefore, the present invention adopts the above-mentioned image tampering detection method based on cross-window self-attention correlation network, and achieves better performance on three standard data sets (CASIA2.0, NIST2016, COLUMBIA). In addition, the algorithm has good robustness under JPEG compression attack and noise destruction attack.

[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An image tampering detection method based on a cross-window self-attention correlation network, characterized by: The following steps are involved: S1: Prepare pre-training datasets and datasets for fine-tuning the model. The pre-training datasets provided for model training are selected from the MS COCO dataset, which contain tampered images and real images. The datasets used for fine-tuning the model are CASIAv2.0, Columbia, and NIST16 datasets. S2: Design a network structure based on cross-window self-attention, including local and global feature extraction modules and prediction modules; In step S2, the network structure based on cross-window self-attention includes two branches, namely, a local and global feature extraction module and a prediction module; specifically, it includes: S21: In the local and global feature extraction module, local and global features of the tampered images in the training set are extracted, and the encoder of the backbone network extracts local and global features from the tampered images in the training set for subsequent use; S22: In the prediction module, it is detected whether the image input in S21 is a tampered image, and four masks are predicted according to the local and global features extracted in step S21. The masks have different scales from small to large. In order to obtain more accurate tampering positioning, each mask is set as a priori for the next scale prediction; S23: In the prediction module, in order to obtain a better predicted mask, a cross-window self-attention CWSA module is added before the mask prediction of each scale. Self-attention calculations are performed in the local window and the translated local window respectively, and the information flow between windows is increased by window interaction. The calculation of the cross-window self-attention module is: In the formula, and They represent the output features of the window multi-head attention module and the cross-window multi-head attention module after the LN layer, R s and R s+1 They represent the output features of the MLP module after the LN layer, W-MSA and CW-MSA represent the multi-head self-attention module; S3: Design the loss function, the local loss function, the global loss function and the BCE loss function in the network structure mentioned in step S2, add and average the above local loss function, global loss function and BCE loss function to get the final loss function to optimize the model; S4: Input the training image and its generated mask into the network in S2 for training, generate and save the image tampering detection pre-trained model and fine-tuning model; S5: Input the tampered image to be detected, use the image tampering detection model trained in step S4 to make predictions, and return the result of the image tampering detection.

2. According to the image tampering detection method based on cross-window self-attention correlation network described in claim 1, it is characterized by: In step S1, the pre-training data set includes 376,742 tampered images and real images, and the tampered image types include splicing, copy-move, and deletion.

3. According to the image tampering detection method based on cross-window self-attention correlation network described in claim 1, it is characterized by: In step S1, the CASIAv2.0 dataset includes 5123 tampered images, and the tampered image types include splicing and copy-move; the Columbia dataset includes 180 tampered images, and the tampered image types include splicing and copy-move; the NIST16 dataset includes 564 tampered images, and the tampered image types include splicing, copy-move and deletion. The CASIAv2.0 dataset, Columbia and NIST16 datasets are divided into training sets and test sets at a ratio of 9:

1. All training sets in the CASIAv2.0 dataset, Columbia and NIST16 dataset constitute a training set set, and all test sets in the CASIAv2.0 dataset, Columbia and NIST16 dataset constitute a test set set. All tampered images in the training set set and the test set set are input into the network in S2, so that each image generates a corresponding Mask.

4. According to the image tampering detection method based on cross-window self-attention correlation network described in claim 1, it is characterized by: In step S3, the loss function is expressed as: BCELoss=-{y*log(p)+(1-y)*log(1-p)} Among them, p is the predicted probability, y is the true value, when y = 1, it means that the true value is predicted as a positive sample, and when y = 0, it means that the true value is predicted as a negative sample.

5. According to the image tampering detection method based on cross-window self-attention correlation network as described in claim 1, it is characterized by: In step S4, the pre-training data set and the data set for fine-tuning the model are respectively input into the network structure in step S2, the number of training iterations is specified, the training is performed in batches, and the optimal model result is saved.

6. According to the image tampering detection method based on cross-window self-attention correlation network as described in claim 1, it is characterized by: In the prediction stage of step S5, the tampered image is detected using the optimal image tampering detection model obtained in step S4 to obtain a predicted binary mask image.

Images