A deep feature fusion image tampering detection method based on JPEG fingerprint

Abstract

The present invention claims protection for a method for detecting image tampering based on deep feature fusion of JPEG fingerprint, which relates to the technical fields of digital image processing, computer vision, deep learning, etc. The specific steps are: 1) Collecting and organizing public uncompressed image samples; 2) Tampering the image samples in two ways and annotating them to obtain tampered image samples and labels to complete the construction of the image tampering data set; 3) Using the prepared image tampering data set to train the deep feature fusion network; 4) Using the trained model to test the two tampered images to obtain the final effect. This method uses the model trained by the feature fusion convolutional neural network to detect tampered images in reality, which has practical significance and achieves good detection accuracy.

Description

Technical Field

[0001] The present invention relates to an image tampering detection method based on deep feature fusion convolutional neural network, and belongs to the technical fields of digital image processing, computer vision, deep learning, etc. Background Art

[0002] In the past, people believed that all digital images posted online were authentic. However, with the development of digital software and smartphone applications, images can be easily manipulated or tampered with. These forged photos spread distorted information and may cause various social impacts. Image forgeries can be roughly divided into two categories: global forgeries and local forgeries, depending on whether the semantic content of the image is changed. Global forgeries are considered less harmful because they only change the visual effect of the image (through techniques such as image compression, image blurring, and image contrast enhancement) without changing the semantic content. In contrast, local forgeries change the semantic content of the image. Because it can spread false or misleading information through tampered content, it is more harmful than global forgeries.

[0003] Inspired by the great success of convolutional neural networks in computer vision, the application of convolutional neural networks in image forgery detection has become increasingly widespread. The powerful adaptive feature learning ability of convolutional neural networks makes up for the poor applicability of traditional image forgery detection techniques based on feature extraction. With the help of convolutional neural networks, the detection of various image forgeries has been greatly improved. However, these detection methods still have the following problems: (1) After the image is tampered, if some JPEG compression post-processing is performed accordingly, the detection method will become invalid; (2) It is only applicable to a certain type of tampering and has poor generalization ability.

[0004] After searching, the application publication number CN110349136A, a method for detecting tampered images based on deep learning, relates to the field of passive image forensics. This method is a multi-stage method, which draws on the idea of ​​target detection. It uses three networks to extract image features, generate candidate frames, and segment tampered areas. The detection process is relatively complicated. The present invention is an end-to-end single-stage method that only requires one network to segment tampered areas from input images, and ensures good accuracy and generalization ability. Summary of the invention

[0005] The present invention aims to solve the above problems of the prior art. A method for detecting image tampering based on deep feature fusion of JPEG fingerprint is proposed. The technical solution of the present invention is as follows:

[0006] A method for detecting image tampering based on deep feature fusion of JPEG fingerprint, comprising the following steps:

[0007] (1) Obtain a sample of publicly available uncompressed images;

[0008] (2) Use Photoshop to tamper with the collected tampered image samples by splicing and copying and moving, and mark the tampered areas to obtain tampered image samples and labels of tampered areas, completing the construction of the tampered image dataset;

[0009] (3) using the tampered image data set obtained after processing in step (2) to perform model training on the feature fusion network, the feature fusion convolutional neural network includes an image noise flow module, an image gradient flow module and a dense connection module; the image noise flow module uses a spatial rich model to extract the noise feature vector of the input image; the image gradient flow module uses a discrete wavelet transform to extract the gradient feature vector of the input image; the convolution in the dense connection module uses a dense connection method to increase the reuse rate of the feature;

[0010] (4) Use the model trained in step (3) to test the input image to obtain the final detection result.

[0011] Furthermore, the image noise flow module uses a spatial rich model high-pass filter to extract a noise feature vector of an input image, specifically comprising: the image noise flow module uses a spatial rich model high-pass filter to extract a JPEG fingerprint on the input image, and then performs a convolution operation with a convolution kernel size of 1×1, and then performs a maximum pooling operation with a size of 2×2 and a step size of 2, and finally obtains a JPEG fingerprint feature vector with 30 feature maps.

[0012] Furthermore, the image gradient flow module uses discrete wavelet transform to extract the gradient feature vector of the input image, specifically including: the image gradient flow module performs a third-order Haar discrete wavelet transform on each color channel of the input image, and each order of wavelet decomposition obtains 12 gradient feature vectors.

[0013] Furthermore, the dense connection module will fuse 30 feature maps of the image noise stream and 12 feature maps of each order of the image gradient stream into 42 JPEG fingerprint features in the first dense connection module, and then perform three convolution operations with a convolution kernel size of 3×3, and the convolution connection method is dense connection; then this operation will be repeated twice, and finally the final 30 JPEG fingerprint feature maps will be obtained.

[0014] Furthermore, the step (3) of performing model training on the deep feature fusion convolutional neural network specifically includes:

[0015] 1) Send the training data set as input to the deep feature fusion convolutional neural network;

[0016] 2) For the image noise flow, the input image is convolved by a spatially rich model high-pass filter to obtain 5 noise feature maps, and then passes through a 1×1 convolution layer and a 2×2 maximum pooling layer to finally obtain a noise feature vector F with 30 feature maps. n ;

[0017] 3) For the image gradient flow, each color channel of the input image is transformed by a third-order Haar discrete wavelet transform to obtain three gradient feature vectors F with 12 feature maps. g1 ,F g2 ,F g3 ;

[0018] 4) The noise feature vector F n and the gradient eigenvector F g1 Spliced ​​along the channel direction into a feature vector F with 42 feature maps n+g1 , then F n+g1 Input into a densely connected module to obtain a feature vector F with 30 feature maps j2 ; then put F j2 After the 2×2 maximum pooling operation, g2 Splicing along the channel direction to obtain 42 feature vectors F with global features j2+g2 , then F j2+g2 Input into a densely connected module to obtain a feature vector F with 30 feature maps j3 ; then put F j3 After the 2×2 maximum pooling operation, g3 Splicing along the channel direction to obtain 42 feature vectors F with global features j3+g3 Finally, F j3+g3 Input into a densely connected module to obtain a JPEG fingerprint feature vector F with 30 feature maps j4

[0019] 6) Put F j4 Decode it through the decoder of the SegNet network to get the final prediction result.

[0020] Furthermore, the feature extraction of the noise stream in step 2) and the gradient stream in step 3) are trained synchronously.

[0021] Furthermore, the entire deep feature fusion convolutional neural network optimizes the prediction results by minimizing the cross entropy loss function. The cross entropy loss function is as follows:

[0022]

[0023] where p i and m iThey are the predicted result and actual label of the i-th input image of the network, and N represents the total number of samples.

[0024] The advantages and beneficial effects of the present invention are as follows:

[0025] The present invention uses digital image processing, computer vision, deep learning and other technologies to achieve the task of detecting tampered images. The present invention is an end-to-end method that can obtain the tampered area by inputting a tampered image without any pre-processing and post-processing operations. The present invention has the following advantages:

[0026] (1) The present invention is an end-to-end network, which only passes through one network from inputting an image to obtaining a result, without requiring other operations;

[0027] (2) The present invention performs tampering detection based on the commonality of JPEG images, such as JPEG fingerprints. As long as the tampered image is in JPEG format, it can be detected regardless of the tampering method.

[0028] (3) The present invention utilizes a spatial rich model high-pass filter to extract image noise features and utilizes wavelet transform to extract image gradient features. These two feature vectors work together to form JPEG fingerprint features, thereby enhancing the information learned by the network and improving the accuracy and generalization ability of detection.

[0029] (4) The dense connection module proposed in the present invention can enhance the feature reuse capability of the network and reduce the information loss of features in the network, so it is also sensitive to some slight tampering traces. At the same time, the clever design of the dense connection module reduces a large number of parameters while increasing the detection effect of the network.

[0030] (5) The present invention has a high accuracy rate, and the test accuracy rate is about 90%; BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The present invention provides a preferred embodiment, which is a system flow chart of the present invention;

[0032] Figure 2 is the cut tampered image 1 and its label;

[0033] Figure 3 yes Figure 2 The prediction result image obtained by deep feature fusion convolutional neural network;

[0034] Figure 4 is to copy and paste the tampered image 2 and its label;

[0035] Figure 5 yes Figure 4 The prediction result image obtained by deep feature fusion convolutional neural network. DETAILED DESCRIPTION

[0036] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0037] The technical solution of the present invention to solve the above technical problems is:

[0038] System flow chart as follows Figure 1 As shown, a deep feature fusion image tampering detection method based on JPEG fingerprint includes the following steps:

[0039] Step 1: Collect and organize public uncompressed image samples;

[0040] Step 2: Use Photoshop to tamper with the collected tampered image samples in two ways and mark the tampered areas, obtain the tampered image samples and the labels of the tampered areas, and complete the construction of the tampered image dataset, so as to train the deep feature fusion convolutional neural network;

[0041] Step 3: Use the constructed dataset to train the deep feature fusion convolutional neural network; this network mainly consists of three parts: image noise flow module, image gradient flow module and dense connection module.

[0042] In the device fingerprint stream, the present invention makes the input image pass through a convolution composed of a spatial rich model high-pass filter to obtain 5 noise feature maps, and then pass through a layer of 1×1 convolution and a layer of 2×2 maximum pooling layer, and finally obtain a noise feature vector F with 30 feature maps. n ;

[0043] In the operation chain fingerprint flow, each color channel of the input image is transformed by the third-order Haar discrete wavelet transform to obtain three gradient feature vectors F with 12 feature maps. g1 ,F g2 ,F g3 Specifically, a third-order Haar discrete wavelet transform is performed on each color channel of the input image, and 36 feature maps (12 (3×4) feature maps for each order) are obtained through third-order wavelet decomposition.

[0044] In the dense connection module, the present invention converts the noise feature vector F n and the gradient eigenvector F g1 Spliced ​​along the channel direction into a feature vector F with 42 feature maps n+g1 , then F n+g1 Input into a densely connected module to obtain a feature vector F with 30 feature maps j2 ; Then put F j2 With F g2Splicing along the channel direction to obtain 42 feature vectors F with global features j2+g2 , then F j2+g2 Input into a densely connected module to obtain a feature vector F with 30 feature maps j3 ; Then put F j3 With F g3 Splicing along the channel direction to obtain 42 feature vectors F with global features j3+g3 Finally, F j3+g3 Input into a densely connected module to obtain a JPEG fingerprint feature vector F with 30 feature maps j4

[0045] Step 4: Use the trained network model to test other tampered images (images not used in the training process) to obtain the final detection effect, with an accuracy rate of about 90%.

[0046] The specific training process of the deep feature fusion convolutional neural network is as follows:

[0047] 1) Send the training data set obtained in step 2) as input to the deep feature fusion convolutional neural network;

[0048] 2) For the image noise flow, the input image is convolved by a spatially rich model high-pass filter to obtain 5 noise feature maps, and then passes through a 1×1 convolution layer and a 2×2 maximum pooling layer to finally obtain a noise feature vector F with 30 feature maps. n ;

[0049] 3) For the image gradient stream, each color channel of the input image is transformed by the third-order Haar discrete wavelet transform to obtain 36 feature maps. It is worth noting that the device fingerprint stream and the operation chain fingerprint stream are trained synchronously;

[0050] 4) Based on step 3), we transform the noise feature vector F n and the gradient eigenvector F g1 Spliced ​​along the channel direction into a feature vector F with 42 feature maps n+g1 , then F n+g1 Input into a densely connected module to obtain a feature vector F with 30 feature maps j2 ; Then put F j2 With F g2 Splicing along the channel direction to obtain 42 feature vectors F with global features j2+g2 , then F j2+g2 Input into a densely connected module to obtain a feature vector F with 30 feature maps j3 ; Then put F j3 With F g3Splicing along the channel direction to obtain 42 feature vectors F with global features j3+g3 Finally, F j3+g3 Input into a densely connected module to obtain a JPEG fingerprint feature vector F with 30 feature maps j4 .

[0051] 5) Based on step 4), we finally transform the feature map F with global features j4 The final prediction result is obtained by decoding through the decoder of the SegNet network. The entire deep feature fusion convolutional neural network optimizes the prediction result by minimizing the cross entropy loss function. The cross entropy loss function is as follows:

[0052]

[0053] where p i and m i They are the predicted result and actual label of the i-th input image of the network, and N represents the total number of samples.

[0054] In order to verify the effect of the present invention, the following experiments were carried out:

[0055] The verification experiment was carried out on a computer with an Intel i9-9900 eight-core processor, 32GB of memory, a graphics card NVIDIA GeForce RTX 2080Ti (11GB), and the platforms were PyCharm and Photoshop CC 2019.

[0056] Experimental methods:

[0057] During this experiment, we collected and organized a variety of public uncompressed images to create a dataset.

[0058] Step 1: Use Photoshop CC 2019 software to make 4,000 of the collected data into training sets to train the deep feature fusion convolutional neural network, and make 80 of them into test sets to test the detection accuracy of the deep feature fusion convolutional neural network. At the same time, we made labels for each image, that is, for each tampered image, we made a black and white image representing its tampered area, where the white part represents the tampered area and the black part represents the non-tampered area.

[0059] Step 2: Use PyCharm to run the program, input the training set images and corresponding labels into the deep feature fusion convolutional neural network, and after 10 iterative training, the final trained model is obtained.

[0060] Step 3: Use the trained model to test the images in the test set and calculate the detection accuracy based on the corresponding labels. Specifically, input the tampered image and tampered area (such as Figure 2 , as shown in 4), after the trained model is tested, the test results are as follows Figure 3 , as shown in Figure 5, the predicted tampered area will be marked in white and the non-tampered area will be marked in black.

[0061] It can be seen that after training, the deep feature fusion convolutional neural network can effectively detect the tampered areas of the tampered images, and the detection accuracy can reach about 90%.

[0062] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0063] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0064] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A deep feature fusion image tampering detection method based on JPEG fingerprint, characterized in that: The following steps are involved: (1) Obtain a sample of publicly available uncompressed and untampered images; (2) Use Photoshop to tamper with the collected tampered image samples by splicing and copying and moving, and mark the tampered areas to obtain tampered image samples and labels of tampered areas, completing the construction of the tampered image dataset; (3) using the tampered image data set obtained after processing in step (2) to perform model training on the feature fusion network, the feature fusion convolutional neural network includes an image noise flow module, an image gradient flow module and a dense connection module; the image noise flow module uses a spatial rich model to extract the noise feature vector of the input image; the image gradient flow module uses a discrete wavelet transform to extract the gradient feature vector of the input image; the convolution in the dense connection module uses a dense connection method to increase the reuse rate of the feature; (4) Using the model trained in step (3) to test the input image, and obtain the final detection result; The image noise flow module uses a spatial rich model high-pass filter to extract a noise feature vector of an input image, specifically comprising: the image noise flow module uses a spatial rich model high-pass filter to extract a JPEG fingerprint on the input image, then performs a convolution operation with a convolution kernel size of 1×1, and then performs a maximum pooling operation with a size of 2×2 and a step size of 2, and finally obtains a JPEG fingerprint feature vector with 30 feature maps; The image gradient flow module uses discrete wavelet transform to extract the gradient feature vector of the input image, specifically comprising: the image gradient flow module performs a third-order Haar discrete wavelet transform on each color channel of the input image, and each order of wavelet decomposition obtains 12 gradient feature vectors respectively; There are three dense connection modules. In the first dense connection module, the 30 feature maps of the image noise stream and the 12 feature maps of each order of the image gradient stream are fused into 42 JPEG fingerprint features, and then three convolution operations with a convolution kernel size of 3×3 are performed, and the convolution connection method is dense connection; then this operation is repeated twice, and finally the final 30 JPEG fingerprint feature maps are obtained.

2. According to claim 1, a method for detecting image tampering based on deep feature fusion of JPEG fingerprint, characterized in that: The step (3) of performing model training on the deep feature fusion convolutional neural network specifically includes: 1) Send the training data set as input to the deep feature fusion convolutional neural network; 2) For the image noise flow, the input image is convolved by a spatially rich model high-pass filter to obtain 5 noise feature maps, and then passes through a 1×1 convolution layer and a 2×2 maximum pooling layer to finally obtain a noise feature vector F with 30 feature maps. n ; 3) For the image gradient flow, each color channel of the input image is transformed by a third-order Haar discrete wavelet transform to obtain three gradient feature vectors F with 12 feature maps. g1 ,F g2 ,F g3 ; 4) The noise feature vector F n and the gradient eigenvector F g1 Spliced ​​along the channel direction into a feature vector F with 42 feature maps n+g1 , then F n+g1 Input into a densely connected module to obtain a feature vector F with 30 feature maps j2 ; then put F j2 After the 2×2 maximum pooling operation, g2 Splicing along the channel direction to obtain 42 feature vectors F with global features j2+g2 , then F j2+g2 Input into a densely connected module to obtain a feature vector F with 30 feature maps j3 ; then put F j3 After the 2×2 maximum pooling operation, g3 Splicing along the channel direction to obtain 42 feature vectors F with global features j3+g3 Finally, F j3+g3 Input into a densely connected module to obtain a JPEG fingerprint feature vector F with 30 feature maps j4 6) Put F j4 Decode it through the decoder of the SegNet network to get the final prediction result.

3. The method for detecting image tampering based on deep feature fusion of JPEG fingerprint according to claim 2, characterized in that: The feature extraction of the noise stream in step 2) and the gradient stream in step 3) are trained synchronously.

4. The method for detecting image tampering based on deep feature fusion of JPEG fingerprint according to claim 2, characterized in that: The entire deep feature fusion convolutional neural network optimizes the prediction results by minimizing the cross entropy loss function. The cross entropy loss function is as follows: where p i and m i They are the predicted result and actual label of the i-th input image of the network, and N represents the total number of samples.

Images