A method for image tampering detection based on convolutional neural network

Abstract

The present invention discloses a method for detecting image tampering based on a convolutional neural network. The method first collects and organizes publicly available image tampering datasets, and performs data enhancement on the collected tampered image datasets through a Python script; then, the data-enhanced tampered image dataset is input into a multi-scale integrated attention convolutional network for network model training. The multi-scale integrated attention convolutional network comprises a U-net network framework, multi-scale feature map processing, and a position and channel attention mechanism. The trained model is used to test the input image to obtain the final tampering location result. The present invention adopts an end-to-end method, and a tampered area can be obtained by inputting a tampered image without any pre-processing or post-processing operations, which is convenient and fast. The present invention uses self-attention to expand the difference between the tampered area and the non-tampered area, thereby achieving more precise detection of tampered targets, and the training and testing costs are low.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, in particular to the field of computer vision and digital image processing technology, and specifically relates to an image tampering detection method based on a convolutional neural network. Background Art

[0002] Images, as a vehicle for disseminating information, are increasingly used in our daily lives. As people rely on images as a crucial source of information, their authenticity becomes increasingly crucial. Images are often viewed as crucial and conclusive evidence in areas such as public opinion reporting and legal evidence collection. However, with the rapid development of image editing software, the authenticity of images is no longer indestructible, making them easily tampered with by non-professionals. Furthermore, because image tampering is low-cost, highly concealed, and difficult to detect with the naked eye, malicious tampering to spread false information is becoming increasingly rampant, posing significant security risks to both society and individuals.

[0003] With the development of computer deep learning technology, neural networks have made many outstanding achievements in the field of image processing in recent years, with outstanding performance in image classification, semantic segmentation, and target detection. Image tampering detection is also a binary semantic segmentation to a certain extent, so whether deep learning algorithms can be used to detect digital image tampering has naturally become a topic worth exploring. At present, tampering detection based on deep learning algorithms has made some progress, but there are still some problems: (1) tampered images are incomplete and edges are blurred; (2) small tampered targets are difficult to detect; (3) after the image is tampered, if it is post-processed, the detection effect will easily fail. Summary of the Invention

[0004] The purpose of the present invention is to provide an image tampering detection method based on convolutional neural network to address the shortcomings of the existing technology.

[0005] The method of the present invention specifically is:

[0006] Step (1) collects and organizes public image tampering datasets; searches for tampering image datasets in the field of tampering detection through open source communities, wherein each tampering image dataset includes a tampered image and a tampered label image (ground-truth image), which are placed in two folders respectively. For datasets with original images, an original image folder is established to store the original image that has not been tampered with. The tampering image datasets include the CASIA dataset, the Columbia dataset, and the NIST16 dataset.

[0007] Step (2) performs data enhancement on the collected tampered image dataset through a Python script; first, the tampered area in the tampered image is extracted, and then the tampered area is scaled, rotated and pasted to the original image without tampering.

[0008] Step (3) inputs the data-enhanced tampered image dataset into the multi-scale integrated attention convolutional network for network model training.

[0009] The multi-scale integrated attention convolutional network consists of a U-net network framework, multi-scale feature map processing, position and channel attention mechanisms;

[0010] The U-net framework described above downsamples an input 256×256 image five times to produce 1024 8×8 feature maps. It then upsamples the image five times to produce the original 256×256 output image. Specifically, in the basic U-net framework, the input image is first encoded, passed through two 3×3 convolution kernels, and then downsampled by a 4×4 convolution kernel with a stride of 2 to produce a 64-channel feature map. The next layer produces a 128-channel feature map. In the following three layers, the input feature map is first split into multiple tiles and fed into the attention mechanism. Convolution and downsampling are then performed. After the five encoder layers, decoding is performed. The network's decoder consists of four pairs of upsampling and convolution layers. The upsampling layers upsample the feature maps using bilinear interpolation. After the upsampling layers, the feature maps are doubled in width and height. The upsampling layer is followed by a convolutional layer with a kernel size of 1×1 and a stride of 1, followed by a batch normalization layer and a ReLU layer to process the upsampled feature maps. Skip connections are established between the same layers of the encoder and decoder, concatenating the encoder's feature maps to the decoder's feature maps, doubling the number of channels and compensating for the decoder's information loss.

[0011] The multi-scale feature map processing is specifically performed as follows: the feature map is divided into blocks using an odd sequence, where the third layer is 5×5, the fourth layer is 3×3, and the fifth layer is 1×1.

[0012] The position and channel attention mechanism is designed from the perspectives of channel and position, and uses the self-attention algorithm to expand the difference between the tampered area and the non-tampered area.

[0013] The specific training process of the multi-scale integrated attention convolutional network is as follows:

[0014] ① Input the data-enhanced tampered image dataset into the multi-scale integrated attention convolutional network.

[0015] ②The first two layers of the encoder perform the following processing:

[0016] For convolutional neural networks, the input image passes through multiple components consisting of convolutional layers, normalization layers, and activation layers, and finally obtains 1024 feature maps.

[0017] The convolution layer uses k convolution kernels to convolve the image and generate k new feature maps; for the j-th output feature map of the n-th layer The specific operations are: in, and denote the convolution kernel and bias respectively, is the input feature map, is the output feature map.

[0018] The convolution layer is followed by a normalization layer. The batch normalization layer is used to transform the data into a distribution with a mean of 0 and a variance of 1. The data batch B={x1,…,x i ,…,x n} for each x i Convert to y i The calculation update of x i Represents the i-th layer channel input from the previous layer, y i Represents the data of the i-th layer channel after batch normalization operation.

[0019] Among them, μ B and denote the mean and variance of batch B respectively, and ε is a small positive number used to avoid division by 0.

[0020] The activation layer is after the normalization layer. The activation layer transforms the input feature map through nonlinear mapping and uses the Relu activation function. The data result f(z i )=max(0,z i ), z i Is the result of the convolution operation, if z i <0,f(z i )=0, if z i >0,f(z i )=z i .

[0021] ③The last three layers of the encoder are processed as follows:

[0022] Enter F k,x Through the parallel processing of position and channel attention mechanism, the complete result is obtained by fusion, k = 3, 4, 5, indicating the number of layers: First, the input F k,x Divide into 5×5 sub-regions f k,x; For each sub-region, input it into the position attention mechanism for calculation to obtain the sub-block f k,y , each sub-block f k,y Then combine it back to its original size and get F k,y ; Then comes the input channel attention mechanism, F k,x After inputting the channel self-attention, it is no longer split, and the channel self-attention calculation is performed directly to obtain the output F k,z ; Finally, F k,y and F k,z Add them together to get a 1024-layer feature vector.

[0023] ④ The feature map with global features is decoded through the decoder to obtain the final prediction result. The entire convolutional network uses the stochastic gradient descent optimization algorithm to minimize the cross entropy loss function to optimize the prediction result. The cross entropy loss function formula is as follows: where p (w,h) is the probability of a pixel pair at coordinate (w,h) in the stitching area, y (w,h) is the label of the true value mask of the current pixel, W is the width of the image, H is the height of the image, w = 1, ..., W, h = 1, ..., H.

[0024] Step (4) uses the model trained in the previous step to test the input image to obtain the final tampering positioning result.

[0025] The beneficial effects of the present invention include:

[0026] (1) The method of the present invention is an end-to-end method. By inputting a tampered image, the tampered area can be obtained without any pre-processing or post-processing operations, which is convenient and fast.

[0027] (2) The method of the present invention uses PyCharm software for code writing, and the training and testing costs are low;

[0028] (3) The method of the present invention expands the difference between the tampered area and the non-tampered area through self-attention, achieving more precise detection of tampered targets;

[0029] (4) The method of the present invention is also effective against small-scale tampering through multi-scale design. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a system flow chart of the present invention;

[0031] Figure 2 Schematic diagram of tampered images in the dataset;

[0032] Figure 3 for Figure 2 Schematic diagram of the corresponding tampered label image;

[0033] Figure 4 For Figure 2 Schematic diagram of detection results of tampered images. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below through specific embodiments.

[0035] In order to verify the effectiveness of this method, the following verification is carried out. The verification is carried out on a computer in a Linux environment, and the graphics card is a 12G Nvidia GeForce RTX 3060 graphics card. The system flow chart is as follows Figure 1 shown.

[0036] We collected and organized a variety of public image tampering datasets, including the CASIA dataset, the NIST16 dataset, and the Columbia dataset. Each tampered image dataset includes tampered images and tampered label images, which are placed in two folders respectively. For datasets with original images, we created an original image folder to store the original images that have not been tampered with. Figure 2 As shown, the tampered label image is Figure 3 As shown, the white part represents the tampered area, and the black part represents the non-tampered area.

[0037] A Python script was used to perform image augmentation on each of the three datasets. The tampered regions were first extracted from the tampered images. The tampered regions were then scaled, rotated, and pasted onto the original, untampered images. 80% of each dataset was used as the training set to train a multi-scale attention convolutional neural network, and 20% of the dataset was used as the test set to test the detection accuracy of the multi-scale attention convolutional neural network. Through training and testing, the model's performance on different datasets was analyzed.

[0038] Run the program on PyCharm, input the training set images and corresponding labels into the multi-scale attention convolutional neural network, and after 150 iterative training, obtain the final trained model.

[0039] The composition of the multi-scale integrated attention convolutional network includes the U-net network framework, multi-scale feature map processing, position and channel attention mechanism;

[0040] The U-net framework described above downsamples an input 256×256 image five times to produce 1024 8×8 feature maps. It then upsamples the image five times to produce the original 256×256 output image. Specifically, in the basic U-net framework, the input image is first encoded, passed through two 3×3 convolution kernels, and then downsampled by a 4×4 convolution kernel with a stride of 2 to produce a 64-channel feature map. The next layer produces a 128-channel feature map. In the following three layers, the input feature map is first split into multiple tiles and fed into the attention mechanism. Convolution and downsampling are then performed. After the five encoder layers, decoding is performed. The network's decoder consists of four pairs of upsampling and convolution layers. The upsampling layers upsample the feature maps using bilinear interpolation. After the upsampling layers, the feature maps are doubled in width and height. The upsampling layer is followed by a convolutional layer with a kernel size of 1×1 and a stride of 1, followed by a batch normalization layer and a ReLU layer to process the upsampled feature maps. To compensate for the feature loss caused by upsampling, skip connections are established between the same layers of the encoder and decoder, concatenating the encoder's feature maps to the decoder's feature maps, doubling the number of channels and compensating for the decoder's information loss.

[0041] The multi-scale feature map processing described above involves multi-scale processing of the feature map by the deep layers of the encoder while processing the high-level semantics of the image. This splits the feature map, thereby highlighting small tampering targets and improving detection effectiveness. Specifically, the feature map is divided into tiles using an odd-numbered sequence: the third layer is 5×5, the fourth layer is 3×3, and the fifth layer is 1×1. This tile division allows small-scale tiles to highlight various parts of the feature map, thereby detecting small-scale tampering targets.

[0042] The position and channel attention mechanism uses self-attention to increase the difference between tampered and non-tampered areas, improving detection effectiveness. Specifically, it is designed from two perspectives: channel and position. Using the self-attention algorithm, it increases the difference between tampered and non-tampered areas, thereby better identifying tampered areas.

[0043] The specific training process is as follows:

[0044] (1) The data-enhanced tampered image dataset is input into the multi-scale integrated attention convolutional network.

[0045] (2) The first two layers of the encoder perform the following processing:

[0046] For convolutional neural networks, the input image passes through multiple components consisting of convolutional layers, normalization layers, and activation layers, and finally obtains 1024 feature maps.

[0047] The convolution layer uses k convolution kernels to convolve the image and generate k new feature maps; for the j-th output feature map of the n-th layer The specific operations are: in, and denote the convolution kernel and bias respectively, is the input feature map, is the output feature map;

[0048] The batch normalization layer is used to transform the data into a distribution with a mean of 0 and a variance of 1; the data batch B={x1,…,x i ,…,x n} for each x i Convert to y i The calculation update of x i Represents the i-th layer channel input from the previous layer, y i Represents the data of the i-th layer channel after batch normalization operation; Among them, μ B and Represent the mean and variance in batch B respectively, and ε is a small positive number used to avoid division by 0;

[0049] The activation layer is after the normalization layer. The activation layer transforms the input feature map through nonlinear mapping and uses the Relu activation function. The data result f(z i )=max(0,z i ), z i Is the result of the convolution operation, if z i <0,f(z i )=0, if z i >0,f(z i )=z i .

[0050] (3) The last three layers of the encoder are processed as follows:

[0051] Enter F k,x Through the parallel processing of position and channel attention mechanism, the complete result is obtained by fusion, k = 3, 4, 5, indicating the number of layers: First, the input F k,x Divide into 5×5 sub-regions f k,x ; For each sub-region, input it into the position attention mechanism for calculation to obtain the sub-block f k,y , each sub-block f k,y Then combine it back to its original size and get F k,y ; Then comes the input channel attention mechanism, F k,xAfter inputting the channel self-attention, it is no longer split, and the channel self-attention calculation is performed directly to obtain the output F k,z ; Finally, F k,y and F k,z Add them together to get a 1024-layer feature vector.

[0052] (4) The feature map with global features is decoded by the decoder to obtain the final prediction result; the entire convolutional network is optimized by the stochastic gradient descent algorithm to minimize the cross entropy loss function, the cross entropy loss function p (w,h) is the probability of a pixel pair at coordinate (w,h) in the stitching area, y (w,h) is the label of the true value mask of the current pixel, w = 1, ..., W, h = 1, ..., H, W is the width of the image, and H is the height of the image.

[0053] Use the trained model to test the images in the test set and calculate the detection accuracy based on the corresponding tampered area labels. Specifically, input the tampered image and the corresponding tampered area respectively, and after the trained model is tested, the detection results will be obtained, such as Figure 4 As shown, the predicted tampered areas are marked in white and the non-tampered areas are marked in black.

[0054] This shows that after training, the multi-scale integrated attention convolutional neural network can effectively detect tampered areas in tampered images. The IOU accuracy rate of the CASIA dataset can reach 60.8%, and the IOU accuracy rate of the Columbia dataset can reach 91%.

[0055] The above embodiments should be understood as merely illustrative of the present invention and not intended to limit the scope of protection of the present invention. Any simple modifications, equivalent variations, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for detecting image tampering based on a convolutional neural network, characterized by: Step (1) collecting and organizing public image tampering datasets; searching for tampered image datasets in the field of tampering detection through open source communities, wherein each tampered image dataset includes tampered images and tampered label images, which are placed in two folders respectively. For datasets with original images, an original image folder is created to store the original images that have not been tampered with; Step (2) performs data enhancement on the collected tampered image dataset through a Python script; first, the tampered area in the tampered image is extracted, and then the tampered area is scaled, rotated and pasted to the original image without tampering; Step (3) inputting the data-enhanced tampered image dataset into the multi-scale integrated attention convolutional network for network model training; The multi-scale integrated attention convolutional network comprises a U-net network framework, multi-scale feature map processing, and position and channel attention mechanisms. The U-net network framework specifically encodes the input image, and then decodes it after undergoing five layers of encoder operations. The network's decoder consists of four pairs of upsampling layers and convolutional layers. Skip connections are established between the same layers of the encoder and decoder to concatenate the encoder's feature map to the decoder's feature map. The specific training process of the multi-scale integrated attention convolutional network is as follows: ① Input the data-enhanced tampered image dataset into the multi-scale integrated attention convolutional network; ②The first two layers of the encoder perform the following processing: For convolutional neural networks, the input image passes through multiple components consisting of convolutional layers, normalization layers, and activation layers, and finally obtains 1024 feature maps; The convolution layer uses k convolution kernels to convolve the image and generate k new feature maps; for the j-th output feature map of the n-th layer The specific operations are: in, and denote the convolution kernel and bias respectively, is the input feature map, is the output feature map; The batch normalization layer is used to transform the data into a distribution with a mean of 0 and a variance of 1; the data batch B={x1,…,x i ,…,x n } for each x i Convert to y i The calculation update of x i Represents the i-th layer channel input from the previous layer, y i Represents the data of the i-th layer channel after batch normalization operation; Among them, μ B and Represent the mean and variance in batch B respectively, and ε is a small positive number used to avoid division by 0; The activation layer is after the normalization layer. The activation layer transforms the input feature map through nonlinear mapping and uses the Relu activation function. The data result f(z i )=max(0,z i ), z i Is the result of the convolution operation, if z i <0,f(z i )=0, if z i >0,f(z i )=z i ; ③The last three layers of the encoder are processed as follows: Enter F k,x Through the parallel processing of position and channel attention mechanism, the complete result is obtained by fusion, k = 3, 4, 5, indicating the number of layers: First, the input F k,x Divide into 5×5 sub-regions f k,x ; For each sub-region, input it into the position attention mechanism for calculation to obtain the sub-block f k,y , each sub-block f k,y Then combine it back to its original size and get F k,y ; Then comes the input channel attention mechanism, F k,x After inputting the channel self-attention, it is no longer split, and the channel self-attention calculation is performed directly to obtain the output F k,z ; Finally, F k,y and F k,z Add them together to get a 1024-layer feature vector; ④ The feature map with global features is decoded through the decoder to obtain the final prediction result; the entire convolutional network is optimized by the stochastic gradient descent algorithm to minimize the cross entropy loss function, the cross entropy loss function p (w,h) is the probability of a pixel pair at coordinate (w,h) in the stitching area, y (w,h) is the label of the ground truth mask of the current pixel, w = 1, ..., W, h = 1, ..., H, W is the width of the image, H is the height of the image; Step (4) uses the trained model to test the input image to obtain the final tampering positioning result.

2. The image tampering detection method based on convolutional neural network according to claim 1, characterized in that: The tampered image datasets include the CASIA dataset, the Columbia dataset, and the NIST16 dataset.

3. The image tampering detection method based on convolutional neural network according to claim 1, characterized in that: The U-net network framework performs five downsampling operations on the input 256×256 image to obtain 1024 8×8 feature maps; then it performs five upsampling operations to output the original size 256×256 image; the specific operation is as follows: in the basic U-net framework, the input image is first encoded, passed through two 3×3 convolution kernels, and then downsampled by a 4×4 convolution kernel with a stride of 2 to obtain a 64-channel feature map, and the next layer obtains a 128-channel feature map; in the next three layers, the input feature map is first divided into multiple tiles and fed into the attention mechanism respectively; then convolution and downsampling operations are performed; after the five layers of encoder operations, decoding operations are performed; the network decoder consists of four pairs of upsampling layers and convolution layers; the upsampling layer uses bilinear interpolation to upsample the feature map; after the upsampling layer, the feature map is doubled in width and height; The upsampling layer is followed by a convolution layer with a kernel size of 1×1 and a stride of 1, followed by a BN layer and a ReLU layer to process the upsampled feature maps.

4. The image tampering detection method based on convolutional neural network according to claim 1, characterized in that: The multi-scale feature map processing is specifically performed as follows: the feature map is divided into blocks using an odd sequence, where the third layer is 5×5, the fourth layer is 3×3, and the fifth layer is 1×1.

5. The image tampering detection method based on convolutional neural network according to claim 1, characterized in that: The position and channel attention mechanism is designed from the perspectives of channel and position, and uses the self-attention algorithm to expand the difference between the tampered area and the non-tampered area.

Images