A Recoloring Image Forensics Method Based on Spatial Correlation
By constructing a recolored image evidence method based on spatial correlation, using the ResNet18 network to extract the spatial correlation characteristics of the image, the problem of difficult to detect image recoloring tampering in the prior art is solved, and the precise detection and security application of images are achieved. promote.
Patent Information
- Application Number
- CN202111392793.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-23
AI Technical Summary
Existing image forensics technologies are difficult to effectively detect whether images have been re-colored and tampered, especially in deep learning scenarios, which lacks specialized processing methods.
A recolored image evidence forensics method based on spatial correlation is constructed. By constructing training sets and test sets, the spatial correlation features of images are extracted, and feature learning is used for ResNet18 network, and detection models are trained to distinguish between natural images and recolored images.
It realizes accurate detection of whether the image has been re-colored and tampered, which improves the performance of image security applications, especially in deep learning scenarios.
Smart Images

Figure CN114202592B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of image processing and information security, and in particular to a recoloring image forensics method based on spatial correlation. Background Art
[0002] As an important information medium, images are widely used to publish news and record events. For a long time, people are accustomed to seeing is believing. However, with the rapid popularization of image editing software such as Adobe Photoshop and GIMP, anyone can easily edit digital images at a very low cost, and the generated content is quite difficult for humans to distinguish. In addition to the most common editing operations such as copy-paste, splicing, repair, etc., a new type of image editing technology - image recoloring technology came into being.
[0003] Unlike the common operation of changing the content of an image by adding or removing regions of interest, the purpose of image recoloring is to change the theme or style of an image by tampering with color values without losing detail. This visually plausible tampering operation makes it difficult for the visual system to perceive. This may cause misjudgment when specific objects and scenes must be identified or tracked.
[0004] Fortunately, image forensics has flourished over the past few decades. Image forgery detection methods can be divided into two categories: active forensics and passive forensics, based on their working mechanisms and principles. Active forensics, such as digital watermarking, embeds identity information into images in advance. During the forensics phase, if an uncorrupted version of the information cannot be extracted, it can be assumed that the image has been tampered with. However, it requires embedding a watermark in the image before it can be distributed, which limits its practical application. Passive forensics, also known as blind forensics, does not rely on any prior information. It relies entirely on the multimedia content being analyzed and attempts to reveal anomalies that may indicate tampering. For example, camera source identification utilizes features such as lens distortion, color filter array (CFA), and pattern noise; forgery detection utilizes specific artifacts left by tampering methods such as JPEG compression, contrast enhancement, resampling, copy-paste, splicing, and repair. Overall, existing forensics methods have achieved good performance in detecting traditional image processing such as copy-paste and splicing. However, these methods cannot be used for heavy color image detection due to different tampering mechanisms. Although changing the color of an image is one of the most common tasks in image processing, there are currently few forensic works specifically targeting image recoloring. Yan et al. [Yan, Yanyang, Wenqi Ren, and Xiaochun Cao. "Recolored image detection via a deep discriminative model." IEEE Transactions on Information Forensics and Security 14.1 (2018): 5-17.] first tried to use two consistencies (inter-channel correlation and illumination consistency) as well as the original input image to distinguish whether the image has been recolored. However, the hard-coded operations of the difference image (such as RG) are not necessarily the best correlation representation, and the method has only verified its effectiveness on traditional image recoloring methods. In addition, there is currently no work specifically dealing with deep learning-based recoloring scenarios. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a re-coloring image forensics method based on spatial correlation, train a preset network model to obtain a trained re-coloring image detection model, and use the re-coloring image detection model to accurately detect whether a picture has been tampered with by re-coloring, which can play an important role in various practical application scenarios involving image security.
[0006] In order to solve the above technical problems, the present invention provides a recoloring image forensics method based on spatial correlation, comprising the following steps:
[0007] Step 1: Use images processed by various recolorization algorithms and corresponding natural images to construct training sets and test sets;
[0008] Step 2: Extract spatial correlation features. Calculate the co-occurrence matrix in four directions (horizontal, vertical, diagonal and anti-diagonal) for each color channel of the image;
[0009] Step 3: Construct a feature learning network. The network is based on the ResNet18 network and contains the following modules: convolution layer, maximum pooling (Maxpool) layer, four residual modules (Resnet block), average pooling layer, and fully connected layer. Each convolution layer is followed by a batch normalization (BatchNorm) layer and a Relu activation function. Each residual module includes: two convolution layers, and a jump connection before the Relu operation of the second convolution layer. The output category of the fully connected layer is 2, that is, the output probability of the image being a natural image and a recolored image is respectively output;
[0010] Step 4: Use the constructed training set and the corresponding label set to train the designed model;
[0011] Step 5: Use the saved best model weights to make predictions on the images in the test set to distinguish between natural images and recolored images.
[0012] Preferably, in step 1, the training set D1 includes three subsets. The first two subsets are generated using the ImageNet validation dataset using two traditional recoloring methods, respectively. Both sub-training sets contain about 19,000 pairs of natural images and corresponding recolored image pairs, and it is ensured that there are no overlapping images in the first and second sub-training sets. The third sub-training set consists of about 16,000 image pairs, using a deep learning-based recoloring method, while also considering some semantic segmentation information of images in the COCO validation dataset. Therefore, the generated images include not only globally recolored images, but also recolored images for specific objects. The test set is divided into deep learning and traditional recoloring scenarios. In the deep learning-based scenario, four deep learning-based recoloring methods are used to generate a benchmark test set D2 containing 240 natural images and corresponding recolored images. Traditional image recoloring uses two public recoloring test sets, denoted as D3 and D4.
[0013] Preferably, in step 2, for each color channel (R, G or B) of the image, the co-occurrence matrix in four directions (horizontal, vertical, diagonal and anti-diagonal) is calculated to analyze the spatial correlation between adjacent pixels to distinguish between natural images and recolored images. The co-occurrence matrix is calculated for any image V using the following formula:
[0014]
[0015] where I{·} is the indicator function, n is the normalization factor, θ1,θ2,…,θ d is the index of the co-occurrence matrix, Δx, Δy is the offset between two adjacent pixels. Set the parameter d to 2 to obtain a two-dimensional co-occurrence matrix, which represents the two-dimensional histogram of the values of pixel pairs in adjacent regions.
[0016] Preferably, in step 3, the original input and output shapes in the ResNet18 network are modified according to the task. The number of the first convolution kernel is changed from 3 to 12. Because after calculating the co-occurrence matrix, 12 matrices are stacked into a tensor of size 256×256×12. The output size of the last fully connected layer of the network is also changed to 2. This output is used to determine whether a given image is a natural image or a recolored image.
[0017] Preferably, in step 4, the constructed training set D1 is used to train the designed model. The network is implemented using the Pytorch deep learning framework. The Adam optimizer is used for optimization, and the initial learning rate is set to 1×10 -4 The CosineAnnealingLR method is used to adjust the learning rate, and the number of iterations of a learning rate cycle is set to 64. L2 regularization is also used, and the weight decay is set to 1×10 -3 . Since images of any size are calculated to obtain a 256×256 co-occurrence matrix, the image size is not adjusted and the batch size is set to 128. The kernel weights are initialized using the He normal initialization method. During the training process, 80% of the training set is used to learn and update the network parameters, and the remaining 20% is used for validation, and the cross entropy loss is used as the loss function of the network. The entire network is trained for 20 cycles, and the data is shuffled at the beginning of each cycle. By observing the accuracy of the validation data, an early stopping strategy is adopted. If the accuracy cannot be improved for 5 consecutive rounds, the training is stopped, and the model with the highest validation accuracy is saved as the final model.
[0018] Preferably, in step 5, the best model weights saved are used to predict each image in the test sets D2, D3 and D4 respectively, and the natural images and recolored images in the test sets are distinguished. In order to provide better visual explanation for the recolored image inspection model of the present invention, grad-cam++ is used to display the heat map of the last convolutional layer in the network. In order to verify whether the spatial correlation feature proposed in the present invention can improve the final recoloring detection performance, t-SNE is used to visualize the feature space of the penultimate layer in the network proposed in the present invention.
[0019] The beneficial effects of the present invention are as follows: the present invention generates classification probabilities corresponding to pictures in a training set according to pictures through a preset network model, corrects model parameters according to the classification probabilities of each picture and the true labels of each picture, and continues to execute the step of training the computational network model until the trained re-colorized image detection model is obtained; the trained re-colorized image detection model is then used to detect images in a test set to determine whether the images are natural images or re-colorized images; the preset network model is trained to obtain the trained re-colorized image detection model, and the re-colorized image detection model can be used to accurately detect whether a picture has been tampered with by re-colorization, and therefore can play an important role in various practical applications involving image security. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 It is a structural schematic diagram of the recoloring image detection model of the present invention.
[0021] FIG2( a ) is a natural image of the present invention and a thermal map of the last convolutional layer in the network.
[0022] FIG2( b ) is a natural image of the present invention and its corresponding recolored image and its heat map at the last convolutional layer in the network.
[0023] FIG3( a ) is a schematic diagram of the feature space of the penultimate layer in the network when the input of the present invention is a spatial correlation feature, visualized using t-SNE.
[0024] FIG3( b ) is a schematic diagram of the feature space of the penultimate layer in the network visualized using t-SNE when the input of the present invention is the original RGB image. DETAILED DESCRIPTION
[0025] A recoloring image forensics method based on spatial correlation comprises the following steps:
[0026] Step 1: Construct training and test sets. Before constructing the training set, we divide the ImageNet validation dataset into 1,000 categories by category, and then randomly divide the images in each category into two parts, as style images and content images. Since the content correlation of images in the same category is stronger, this process avoids excessive artifacts in the dataset due to the irrelevance or weak correlation between style images and content images. The training set D1 includes three subsets. First, we use the traditional recoloring method [E.Reinhard, M.Adhikhmin, B.Gooch, and P.Shirley, “Color transfer between images,” IEEE Comput.Graph.Appl., vol.21, no.5, pp.34–41, 2001.] to generate the first sub-training set, which contains about 19,000 pairs of natural images and corresponding recolored image pairs. The above generation process randomly selects images from style images and content images each time, and ensures that the images used are not reused. We then used another traditional recoloring method [F.Piti′e, A Kokaram, and R.Dahyot, “Automated colour grading using colour distribution transfer,” Comput.Vis.andImage Understand., vol.107, no.1-2, pp.123–137, 2007.] to obtain the second sub-training set through a similar generation process, whose capacity is the same as the first sub-training set. The above process ensures that there are no overlapping images in the first and second sub-training sets by exchanging style images and content images. The third sub-training set consists of about 16,000 image pairs. Different from the above process, a deep learning-based recoloring method is used [J.Yoo, Y.Uh, S.Chun, B.Kang, and J.-W.Ha, "Photorealistic style transfer via wavelet transforms," in Proc. of the IEEE / CVF Int.Conf. on Comput.Vis., 2019, pp.9036–9045.], and some semantic segmentation information of images in the COCO validation dataset is also considered. Therefore, the generated images include not only globally recolored images, but also recolored images for specific objects.
[0027] The test sets are divided into deep learning and traditional recolorization scenarios. In the deep learning-based scenario, we use four deep learning-based recolorization methods to generate a benchmark test set D2 containing 240 natural images and corresponding recolorization images. We first select the original photos in the COCO validation dataset, the ImageNet validation dataset, and the images scraped from the website, and then randomly recolor 60 natural images using the four methods respectively. The generation process ensures that the natural images used by each method are not repeated, and the images between the test set and the training set do not overlap. Traditional image recolorization uses two public recolorization test sets denoted as D3 and D4. D3 includes 100 real photos (scraped from the website) and recolorization images generated by various traditional recolorization methods. D4 contains 80 manually edited recolorization photos. These photos are either made by mobile phone applications or downloaded from websites such as Photoshop tutorial websites.
[0028] Step 2: Extract spatial correlation features. Figure 1 As shown in the "Preprocessing" section, we calculate the co-occurrence matrix in four directions (horizontal, vertical, diagonal, and anti-diagonal) for each color channel (R, G, or B) of the image to analyze the spatial correlation between adjacent pixels to distinguish natural images from recolored images. For any image V, the co-occurrence matrix is calculated using the following formula:
[0029]
[0030] where I{·} is the indicator function, n is the normalization factor, θ1,θ2,…,θ d is the index of the co-occurrence matrix, and Δx, Δy are the offsets of two adjacent pixels. In our actual implementation, we set the parameter d to 2 to obtain a two-dimensional co-occurrence matrix that represents a two-dimensional histogram of pixel pairs in adjacent areas. The vertical axis of the histogram represents the first value, and the horizontal axis represents the second value. The advantage of doing this is that it avoids the need to truncate the obtained co-occurrence matrix to reduce the complexity of the operation. For images of any size, assuming that the image has an 8-bit pixel depth, this will always produce a co-occurrence matrix of size 256×256, avoiding the need to resize the image when training and testing the network. In addition, for any pixel that is not on the edge, although there are 8 possible neighbors, we only consider four of them (horizontal, vertical, diagonal, and anti-diagonal) because the other four directions are redundant.
[0031] Step three, use CNN to learn features. For any RGB image, we calculate the co-occurrence matrix of each color channel in four directions and obtain 12 matrices of size 256×256. Previous methods mainly used the co-occurrence matrix to train the support vector machine classifier, but directly using these manually constructed features and using machine learning classifiers may not be the best solution. In order to distinguish between natural images and recolored images, it is necessary to collect effective discriminative features from the co-occurrence matrix. Since CNN can automatically learn feature representations and is widely used in computer vision tasks such as image classification, we constructed a CNN-based feature extraction module. Combined with Figure 1 As shown in the "Recoloring detection" section of the present invention, the feature extraction module of the present invention is based on ResNet18, and includes the following modules in total: convolution layer, maximum pooling (Maxpool) layer, four residual modules (Resnet block), average pooling layer, and fully connected layer. Each convolution layer is followed by a batch normalization (BatchNorm) layer and a Relu activation function. Each residual module includes: two convolution layers, and a jump connection before the Relu operation of the second convolution layer, and the input can be propagated forward faster through the jump connection across layers. We modify the original input and output shapes in the ResNet18 network according to the task. We change the number of the first convolution kernel from 3 to 12. Because after calculating the co-occurrence matrix, we will get 12 matrices stacked into a tensor of size 256×256×12. The output size of the last fully connected layer of the network is also changed to 2. This output is used to determine whether a given picture is a natural image or a recolored image.
[0032] Step 4: Use the constructed training set D1 to train the designed model. The network is implemented using the Pytorch deep learning framework. We use the Adam optimizer for optimization and set the initial learning rate to 1×10 +4 The CosineAnnealingLR method is used to adjust the learning rate, and the number of iterations of one learning rate cycle is set to 64. We also use L2 regularization and set the weight decay to 1×10 -3. Since images of any size are calculated to obtain a 256×256 co-occurrence matrix, we do not resize the image and set the batch size to 128. The kernel weights are initialized using the He normal initialization method. During the training process, 80% of the training set is used to learn and update the network parameters, and the remaining 20% is used for validation, and the cross entropy loss is used as the loss function of the network. The entire network is trained for 20 cycles, and the data is shuffled at the beginning of each cycle. By observing the accuracy of the validation data, an early stopping strategy is adopted. If the accuracy cannot be improved for 5 consecutive rounds, the training is stopped, and the model with the highest validation accuracy is saved as the final model. Training and testing are performed on a computer equipped with an NVIDIA RTX 2080Ti GPU.
[0033] Step five, use the saved best model weights to predict each image in the test sets D2, D3 and D4 respectively, and distinguish the natural images and recolored images in the test sets. The specific detection results are shown in Table 1. In order to provide better visual interpretability for the recolored image inspection model of the present invention, we use grad-cam++ to display the heat map of the last convolutional layer in the network. As shown in Figure 2(a) and Figure 2(b), although natural images and their corresponding recolored images look similar, their heat maps are quite different. The upper left corner area of the heat map of the natural image has a higher negative correlation response (the gray value at the center of the contour line is lower); the lower right corner area of the heat map of the corresponding recolored image has a higher negative correlation response. The visualization results of the heat map show that the present invention can further learn from the extracted spatial correlation feature set and can distinguish natural images from recolored images.
[0034] Table 1 Test results
[0035]
[0036] In order to verify whether the spatial correlation feature proposed in the present invention can improve the final recoloring detection performance, we use t-SNE visualization to display the feature space of the penultimate layer in the network proposed in the present invention. As shown in Figure 3(a), the input to the network is a feature set for calculating spatial correlation. The solid circles (marked as ● in the figure) represent natural images, and the solid triangles (marked as ▲ in the figure) represent recolored images. It can be seen that the solid circles and solid triangles are clustered into two groups with clear boundaries. In contrast, the input of Figure 3(b) is the original RGB image. It can be seen that the solid circles and solid triangles are mixed together, and the network cannot distinguish them well. The visualization results show that spatial correlation is indeed a significant feature of natural images and recolored images. The present invention can effectively detect recolored images by using this feature.
[0037] The present invention extracts the spatial correlation features of neighboring pixels of the image and uses the powerful learning ability of CNN to learn discriminative features. A large number of experimental results in two recolorization scenarios show that the spatial correlation features have strong discriminative ability and achieve state-of-the-art performance in detecting recolorization images generated by various methods, and have good generalization ability when dealing with unknown types of recolorization methods.
Claims
1. A recoloring image forensics method based on spatial correlation, characterized in that: The steps include: Step 1: Use images processed by various recolorization algorithms and corresponding natural images to construct training sets and test sets; Step 2: Extract spatial correlation features. Calculate the co-occurrence matrix in four directions for each color channel of the image. The four directions are horizontal, vertical, diagonal and anti-diagonal. Calculate the co-occurrence matrix in four directions for each color channel of the image to analyze the spatial correlation between adjacent pixels to distinguish natural images from recolored images. For any image V, use the following formula to calculate the co-occurrence matrix: where I{·} is the indicator function, n is the normalization factor, θ1,θ2,…,θ d is the index of the co-occurrence matrix, Δx, Δy are the offsets of two adjacent pixels, and the parameter d is set to 2 to obtain a two-dimensional co-occurrence matrix, which represents the two-dimensional histogram of the values of pixel pairs in adjacent regions; Step 3: Construct a feature learning network; the network is based on the ResNet18 network and includes: a convolutional layer, a maximum pooling layer, four residual modules, an average pooling layer, and a fully connected layer; each convolutional layer is followed by a batch normalization layer and a Relu activation function; each residual module includes: two convolutional layers, and a jump connection before the Relu operation of the second convolutional layer; the output category of the fully connected layer is 2, that is, the output probability of the image being a natural image and a recolored image is respectively output; Step 4: Use the constructed training set and the corresponding label set to train the designed model; Step 5: Use the saved best model weights to make predictions on the images in the test set to distinguish between natural images and recolored images.
2. The method for recoloring image forensics based on spatial correlation as claimed in claim 1, characterized in that: In step 1, the training set D1 includes three subsets, the first two of which are generated using the ImageNet validation dataset using two traditional recolorization methods respectively. Both subsets contain about 19,000 pairs of natural images and corresponding recolored image pairs, and ensure that there are no overlapping images in the first and second subsets. The third subset consists of about 16,000 image pairs, using a deep learning-based recolorization method, while also considering some semantic segmentation information of images in the COCO validation dataset; the test set is divided into deep learning and traditional recolorization scenarios. In the deep learning-based scenario, four deep learning-based recolorization methods are used to generate a benchmark test set D2 containing 240 natural images and corresponding recolored images. The traditional image recolorization uses two public recolorization test sets, denoted as D3 and D4.
3. The method for recoloring image forensics based on spatial correlation as claimed in claim 1, characterized in that: In step 3, the original input and output shapes in the ResNet18 network are modified according to the task; the number of the first convolution kernels is changed from 3 to 12, and the 12 matrices are stacked into a tensor of size 256×256×12; the output size of the last fully connected layer of the network is also changed to 2. This output is used to determine whether a given image is a natural image or a recolored image.
4. The method for recoloring image forensics based on spatial correlation as claimed in claim 2, characterized in that: In step 4, the constructed training set D1 is used to train the designed model. The network is implemented using the Pytorch deep learning framework, optimized using the Adam optimizer, and the initial learning rate is set to 1×10 )* , the CosineAnnealingLR method is used to adjust the learning rate, the number of iterations of a learning rate cycle is set to 64, L2 regularization is used, and the weight decay is set to 1×10 )+ , the batch size is set to 128, and the kernel weights are initialized using the He normal initialization method; during the training process, 80% of the training set is used to learn and update the network parameters, and the remaining 20% is used for verification, and the cross entropy loss is used as the loss function of the network. The entire network is trained for 20 cycles, and data is shuffled at the beginning of each cycle. By observing the accuracy of the verification data, an early stopping strategy is adopted. If the accuracy cannot be improved for 5 consecutive rounds, the training is stopped, and the model with the highest verification accuracy is saved as the final model.
5. The method for recoloring image forensics based on spatial correlation as claimed in claim 2, characterized in that: In step 5, the saved best model weights are used to predict each image in the test sets D2, D3, and D4 respectively, and the natural images and recolored images in the test sets are distinguished. The heat map of the last convolutional layer in the network is displayed using grad-cam++, and the feature space of the second to last layer in the network is visualized using t-SNE.
