An image tamper detection method based on image reconstruction

Abstract

The application discloses an image tampering detection method based on image reconstruction. The application uses a reconstruction network to model the distribution of real images, specifically, applies metric learning on the decoder to make the features of real images close to each other in the feature space, and the features of real images and tampered images are far away from each other, and the learned features are easier to use to detect unknown tampering traces; further, a multi-scale graph reasoning module is used to further mine the representation with distinguishing ability; finally, a reconstruction guided attention module is used to focus on possible fake areas, and the learned image distribution difference can make the tampered image with unknown tampering mode easier to be identified.

Description

Technical Field

[0001] This invention relates to an image tampering detection method, specifically an image tampering detection method based on image reconstruction. Background Technology

[0002] Digital images, as easily understood information carriers, are ubiquitous in daily life and a primary means of obtaining information. However, with the development of digital image processing technology, people can use more methods to tamper with images. The development of commonly used image editing software, such as Photoshop and beauty camera apps, has made image tampering very common. Ordinary people can easily modify images, and maliciously tampered images, when disseminated through online media, can affect people's judgment of facts, allowing malicious users to easily manipulate images and cause adverse effects.

[0003] Image tampering detection falls within the technical fields of computer vision and image processing. It involves designing algorithms to identify and verify whether images have been tampered with or modified. Research in this field focuses on developing automated and reliable techniques to detect and identify traces of tampering, erasure, and splicing in images. These techniques are typically based on deep learning, image feature extraction, and other methods to analyze visual inconsistencies, discontinuities, compression marks, and editing marks in images to determine whether they have been tampered with. Image tampering detection technology has wide applications, including digital forensics, news verification, and copyright protection. With the continuous development of image processing technology and the evolution of tampering techniques, research and improvement of image tampering detection technology are ongoing.

[0004] Traditional image detection methods detect image tampering by leveraging specific low-level tampering traces, such as Color Filter Array (CFA) analysis, local noise analysis, and double JPEG compression. These methods are typically based on statistical analysis and feature extraction of images to detect tampering traces. Additionally, some research utilizes image consistency, most of which uses deep learning Siamese networks to determine whether two input image patches contain the same tampering features, such as EXIF ​​metadata or camera model features.

[0005] The following image tampering detection methods were also disclosed:

[0006] Multiple network branches are used for tamper detection. One network stream extracts RGB local texture features to capture visual artifacts, while another stream uses noise features to simulate the inconsistency between the tampered area and the original area. Finally, the two features are fused for tamper detection.

[0007] A top-down approach is used to extract hierarchical features at different scales, and then these features are fused to detect whether the input image has been tampered with. Furthermore, data augmentation is used to obtain features from different perspectives, enabling the learning of more generalizable features that are independent of semantics.

[0008] Design a deep learning neural network to estimate data uncertainty and model uncertainty. By assigning high uncertainty to misclassified regions and low uncertainty to correctly classified regions, guide the model's attention and improve the accuracy of tampered predictions.

[0009] Existing image tampering detection methods still have the following problems:

[0010] Limited generalization performance: Due to the complexity and diversity of tampering methods used in the real world, the data distribution of actual tampered images often differs from the data distribution used to train the model. This makes it difficult for the model to generalize to untrained data, resulting in a significant performance drop. Furthermore, finding universally applicable tampering traces remains challenging. Different image tampering detection schemes often utilize different features, such as local texture, noise, or frequency domain distribution. This is the most concerning and also the most difficult problem to solve.

[0011] Insufficient datasets. Deep learning is often trained on large datasets, which is the first step in solving problems using deep learning. Because image tampering detection is a relatively small branch of the image field, it has not received enough attention, and a complete dataset has not been established, leading to overfitting in the model.

[0012] Poor robustness: When tampered images are spread on real-world social media platforms, they are often subjected to image processing techniques such as compression, blurring, brightness adjustment, and scaling, which can cover up the original tampering traces and pose a greater challenge to the detection capabilities of the model. Summary of the Invention

[0013] To address the needs and problems in the background technology, this invention provides an image tampering detection method based on image reconstruction. This invention utilizes a reconstruction network to model the distribution of real images. Specifically, metric learning is applied to the decoder to make the features of real images close to each other in the feature space, while keeping the features of real images and tampered images far apart. The learned features are more easily used to detect unknown tampering traces. A multi-scale graph inference module is then used to further mine discriminative representations. Finally, a Reconstruction-Guided Attention (RGA) module is used to focus on possible forged regions, ensuring the learned image distribution differences and making it easier to identify tampered images with unknown tampering patterns.

[0014] The technical solution of the present invention is as follows:

[0015] I. An Image Tampering Detection Method Based on Image Reconstruction

[0016] First, the original image X is input into the reconstruction network for reconstruction learning to obtain the output features of the encoder, the output features of the decoder, and the reconstructed image X'.

[0017] Next, the output features of the encoder and the output features of the decoder are input into the multi-scale graph inference module for multi-scale aggregation of information to obtain the initial enhanced feature map F. enh ;

[0018] Then, the original image X, the reconstructed image X', and the initial enhanced feature map F are combined. enh The images are input together into the reconstruction-guided attention module for attention weighting to obtain the attention-weighted output features. Finally, the images are classified based on the attention-weighted output features to obtain the image tampering detection results.

[0019] The reconstruction network includes an encoder, a decoder, and a global average pooling layer. The encoder and decoder are connected. The output features of the encoder and the multi-scale output features of the decoder are processed by the global average pooling layer and the output is denoted as the reconstructed image X'.

[0020] The loss function used during training of the reconstructed network includes a measure of learning loss, as shown in the following formula:

[0021]

[0022] in, Let R and F represent the set of real image samples and the set of tampered image samples, respectively, and N represent the learning loss value. RR N is the total number of real images and real image pairs. RF It represents the total number of pairs of real and manipulated images, and d() is a pairwise distance function based on cosine distance. and Let represent the features of image sample i and image sample j after global pooling, and |||2 represent the Euclidean distance.

[0023] The encoder's output features and the decoder's output features are input into the multi-scale graph inference module for multi-scale aggregation of information to obtain the initial enhanced feature map F. enh Specifically:

[0024] First, the output features of the encoder and the output features of different blocks in the decoder are aggregated in a multi-scale manner to obtain aggregated features {v} at different scales. i agg Then, the encoder's output features are combined with the aggregated features v at each scale. i aggAfter concatenation, the vector is passed through a sigmoid function and two fully connected layers to obtain the enhanced feature vector v at the current scale. i enh Finally, the enhanced feature vectors {v} at different scales were analyzed. i enh After spatial combination, the initial enhanced feature map F is obtained. enh .

[0025] In the multi-scale graph inference module, for the output features of each block in the decoder, firstly, the output features of the encoder and the current-scale output features of the decoder are modeled as corresponding vertex sets to obtain the encoder vertex set V. enc and the current scale decoder vertex set V dec ,satisfy and h1 and w1 are the dimensions of the encoder's output feature map, and h2 and w2 are the dimensions of the decoder's output feature map at the current scale, thus constructing the decoder vertex set V at the current scale. dec Vertex v in the encoder vertex set i enc Connected vertices A set, denoted as the set of connected vertices N(v i enc ),satisfy N represents the number of vertices. For the current scale decoder vertex set V dec The vertices in;

[0026] Next, using the encoder vertex set V enc and the set of connected vertices N(v) i enc After weighted aggregation, the aggregated feature v at the current scale is obtained. i agg The formula is as follows:

[0027]

[0028] in, and These are respectively the vertices obtained through neural networks. and vertex The projected vertex, a, obtained after projecting onto a shared embedding space. j Here, represents the weighting coefficient, and || denotes the join operation. This represents a single convolutional layer, and ψ() represents the sigmoid function. This represents element-wise multiplication, and l represents the set of connected vertices N(v). i enc The index of the vertex in ).

[0029] In the reconstruction-guided attention module, firstly, the difference between the original image X and the reconstructed image X' is calculated at the pixel level to obtain the difference mask m; then, the difference mask m and the initial enhanced feature map F are respectively processed. enh After convolution, the difference mask convolution features and the enhancement convolution features are obtained separately. The difference mask convolution features and the enhancement convolution features are multiplied together, and then sigmoid activation is applied to obtain the enhancement attention feature map. Finally, the enhancement attention feature map and the initial enhancement feature map F are combined. enh After summing, we obtain the attention-weighted output features.

[0030] II. A computer device

[0031] The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method.

[0032] III. A computer-readable storage medium

[0033] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method.

[0034] IV. A computer program product

[0035] The computer program product includes a computer program / instruction that, when executed by a processor, implements the steps of the method.

[0036] The beneficial effects of this invention are:

[0037] 1. The detection method proposed in this invention focuses more on learning from real images and will not overfit to certain forgery traces, thereby achieving better generalization and robustness.

[0038] 2. Compared to rare and difficult-to-obtain tampered images, this invention relies more on easily obtainable real images for training.

[0039] 3. The multi-scale graph reasoning module proposed in this invention can better uncover the subtle differences between real and fake images. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0042] Because real-world tampering methods are complex and diverse, while the types of image tampering data accessible to models are limited, it is possible to distinguish tampered images by focusing the model on learning the distribution of real images and simultaneously widening the difference in distribution between tampered image features and real image features. Therefore, this invention proposes a reconstruction learning network specifically designed to model the distribution of real images. Simultaneously, metric learning is applied to the decoder to ensure that features of real images are close to each other in the feature space, while features of real images and tampered images are far apart. After training, real images reconstruct well, while tampered images reconstruct poorly. This ensures that tampered images with unknown tampering patterns are more easily identified due to the learned distribution differences.

[0043] like Figure 1 As shown, the present invention includes the following steps:

[0044] First, the original image X is input into the reconstruction network for reconstruction learning to obtain the output features of the encoder, the multi-scale output features of the decoder, and the reconstructed image X'.

[0045] Since image tampering methods are often diverse, exploring common features of real images is considered more appropriate than overfitting to specific tampering traces in the training set. Therefore, this invention proposes reconstruction learning to recover only real images. The reconstruction network is designed to model only the distribution of real facial images. Therefore, the learned features are more easily used to detect unknown tampering traces. The reconstruction network includes an encoder, a decoder, and a global average pooling layer. The encoder and decoder are connected, and the output features of the encoder and the multi-scale output features of the decoder are combined and then processed by the global average pooling layer to output the reconstructed image X'.

[0046] The loss function used during network training includes a measure of learning loss that makes real images closer in the embedding space, while keeping real and manipulated images further apart, as shown in the following formula:

[0047]

[0048] in, Let R and F represent the set of real image samples and the set of tampered image samples, respectively, and N represent the learning loss value. RR N is the total number of real images and real image pairs. RF It is the total number of pairs of real images and tampered images. It is a pairwise distance function based on cosine distance. and Let represent the features of image sample i and image sample j after global pooling, and |||2 represent the Euclidean distance.

[0049] Next, the output features of the encoder and decoder are input into the multi-scale graph inference (MGR) module for multi-scale aggregation of information to obtain the initial enhanced feature map F for subsequent reconstruction and attention guidance. enh ;

[0050] The output features of the encoder and decoder are input into the multi-scale graph inference (MGR) module for multi-scale aggregation of information, resulting in an initial enhanced feature map F for subsequent reconstruction and attention guidance. enh Specifically:

[0051] When applying metric learning loss to the decoder, useful information for distinguishing between real and tampered images is also embedded in the decoder. To effectively utilize the tampering cues captured by the decoder features for final classification, this invention proposes a multi-scale approach to mine tampering information to obtain a comprehensive representation, i.e., further mining discriminative representations, since different tampering techniques lead to tampering traces at different scales. The multi-scale graph inference scheme aggregates the difference information between real and tampered faces captured by the encoder and decoder of the reconstruction network in a multi-scale manner. The multi-scale graph inference module combines the latent features of the decoder block and the encoder output into a bipartite graph for comprehensive inference of tampering cues.

[0052] First, the output features of the encoder and the output features of different blocks in the decoder (i.e., output features at different scales) are aggregated in a multi-scale manner to obtain aggregated features {v} at different scales. i agg Then, the encoder's output features are combined with the aggregated features v at each scale. i agg After concatenation, the image is passed through a sigmoid function and two fully connected layers to obtain the image at the current scale and v. i enc Enhanced feature vectors v with the same channel dimension i enh Finally, the enhanced feature vectors {v} at different scales were analyzed. i enh After spatial combination, an initial enhanced feature map F is obtained for subsequent reconstruction to guide attention. enh .

[0053] Graph reasoning from N(v) i enc Aggregate information in v to enrich v i encThe feature representation is used to better infer tampering clues. When aggregating information from the decoder to the encoder, this invention maintains spatial correspondences to model local relationships, as tampering traces often exist in continuous local regions. In the multi-scale graph inference module, for the output features of each block in the decoder (output features at each scale), firstly, the encoder's output features and the decoder's current scale i output features are modeled as corresponding vertex sets, respectively, to obtain the encoder vertex set V. enc and the current scale decoder vertex set V dec ,satisfy and h1 and w1 are the width and height of the encoder's output feature map, and h2 and w2 are the width and height of the decoder's output feature map at the current scale, respectively. This allows us to construct the decoder vertex set V at the current scale i. dec Vertex v in the encoder vertex set i enc Connected vertices A set, denoted as the set of connected vertices N(v i enc ),satisfy N represents the number of vertices. For the current scale decoder vertex set V dec The vertices in;

[0054] Next, using the encoder vertex set V enc and the set of connected vertices N(v) i enc After weighted aggregation, the aggregated feature v at the current scale is obtained. i agg The formula is as follows:

[0055]

[0056] in, and The vertices are respectively processed by the neural network g(·). and vertex The projected vertex, a, obtained after projecting onto a shared embedding space. j These are weighting coefficients used to represent v. i,j dec For v i enc The importance of . ‖ indicates a join operation. This represents a single convolutional layer, and ψ() represents the sigmoid function. This represents element-wise multiplication, and l represents the set of connected vertices N(v). i enc The index of the vertex in ). This invention uses a nonlinear transformation function ψ(·), based on vi enc Compute a vector of values ​​in the range [0,1] to generate pairs of values ​​for v. i enc 'Feature richness measurement at the channel level. During information aggregation, when v i enc This invention is particularly enhanced when the weight of the corresponding channel is relatively small. The passage.

[0057] Then, the original image X, the reconstructed image X', and the initial enhanced feature map F are combined. enh The inputs are combined and fed into the reconstruction-guided attention module for attention weighting to obtain the attention-weighted output features;

[0058] Constrained by the reconstruction network, the reconstructed tampered image visually differs significantly from the input tampered image. To address this, this invention proposes a Reconstruction-Guided Attention (RGA) module, which focuses more on potentially forged regions. This module guides the classification network to pay closer attention to tampering traces for subsequent classification tasks. During training, both the tampered and real images are reconstructed, resulting in corresponding reconstructed images. The graph inference module also processes the features corresponding to both the tampered and real images.

[0059] In the reconstruction-guided attention module, firstly, the difference between the original image X and the reconstructed image X' is calculated at the pixel level to obtain the difference mask m, which satisfies m = |X – X'|, where || represents the absolute value function; then, the difference mask m and the initial enhanced feature map F are respectively processed. enh After convolution, the difference mask convolution features and the enhancement convolution features are obtained separately. The difference mask convolution features and the enhancement convolution features are multiplied together, and then sigmoid activation is applied to obtain the enhancement attention feature map. Finally, the enhancement attention feature map and the initial enhancement feature map F are combined. enh After summing, we obtain the attention-weighted output features, as shown in the following formula:

[0060]

[0061] F att =F′ enh +F enh ,

[0062] Where f1 and f2 represent convolution operations, and σ represents the sigmoid function. This represents element-wise multiplication. For simplicity, the space size of these tensors is omitted, and bilinear interpolation is used to appropriately preserve the space size in the above multiplication and addition operations.

[0063] Finally, image tampering detection results are obtained by classifying the output features based on attention weighting.

[0064] To verify this invention, the publicly available license plate dataset is used as the test dataset. In this embodiment, 100,000 images from the real image dataset CCPD-base plus 80,000 self-created license plate tampering images are used, with data augmentation such as blurring and color balancing added. CCPD contains vehicle images from different cities, times, and weather conditions within China. Each image contains a license plate and provides corresponding annotation information, including the license plate's location, color, characters, font, and other recognition-related information. The license plates in the CCPD dataset cover various special cases and variations of Chinese-style license plates, providing significant reference value for research on license plate recognition algorithms. Table 1 shows the license plate tampering detection results of the proposed method (RECCE) and existing methods on the CCPD dataset. As can be seen from Table 1, the proposed method achieves the highest ACC and AUC.

[0065] Table 1 shows the license plate tampering detection results of existing methods and the method proposed in this invention on the CCPD dataset.

[0066]

[0067] In addition to license plate tampering scenarios, the present invention also achieves extremely high accuracy in text image tampering scenarios, as shown in Table 2, indicating that the present invention has a relatively high generalization ability in multiple scenarios.

[0068] Table 2 shows the tampering detection results of the method proposed in this invention in the context of text image tampering.

[0069]

[0070]

[0071] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image tampering detection method based on image reconstruction.

[0072] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the image tampering detection method based on image reconstruction.

[0073] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the image tampering detection method based on image reconstruction.

[0074] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A method for detecting image tampering based on image reconstruction, characterized in that, Includes the following steps: First, the original image X is input into the reconstruction network for reconstruction learning to obtain the output features of the encoder, the output features of the decoder, and the reconstructed image X'. Next, the output features of the encoder and the output features of the decoder are input into the multi-scale graph inference module for multi-scale aggregation of information to obtain the initial enhanced feature map F. enh ; The encoder's output features and the decoder's output features are input into the multi-scale graph inference module for multi-scale aggregation of information to obtain the initial enhanced feature map F. enh Specifically: First, the output features of the encoder and the output features of different blocks in the decoder are aggregated in a multi-scale manner to obtain aggregated features {v} at different scales. i agg Then, the encoder's output features are combined with the aggregated features v at each scale. i agg After concatenation, the vector is passed through a sigmoid function and two fully connected layers to obtain the enhanced feature vector v at the current scale. i enh Finally, the enhanced feature vectors {v} at different scales were analyzed. i enh After spatial combination, the initial enhanced feature map F is obtained. enh ; Then, the original image X, the reconstructed image X', and the initial enhanced feature map F are combined. enh The images are input together into the reconstruction-guided attention module for attention weighting to obtain the attention-weighted output features. Finally, the images are classified based on the attention-weighted output features to obtain the image tampering detection results.

2. The image tampering detection method based on image reconstruction according to claim 1, characterized in that, The reconstruction network includes an encoder, a decoder, and a global average pooling layer. The encoder and decoder are connected. The output features of the encoder and the multi-scale output features of the decoder are processed by the global average pooling layer and the output is denoted as the reconstructed image X'.

3. The image tampering detection method based on image reconstruction according to claim 1, characterized in that, The loss function used during training of the reconstructed network includes a measure of learning loss, as shown in the following formula: in, Let R and F represent the set of real image samples and the set of tampered image samples, respectively, and N represent the learning loss value. RR N is the total number of real images and real image pairs. RF It is the total number of pairs of real images and tampered images. It is a pairwise distance function based on cosine distance. and This represents the features of image sample i and image sample j after global pooling. This indicates Euclidean distance.

4. The image tampering detection method based on image reconstruction according to claim 1, characterized in that, In the multi-scale graph inference module, for the output features of each block in the decoder, firstly, the output features of the encoder and the current-scale output features of the decoder are modeled as corresponding vertex sets to obtain the encoder vertex set V. enc and the current scale decoder vertex set V dec Satisfying V enc and V dec = h1 and w1 are the dimensions of the encoder output feature map, and h2 and w2 are the dimensions of the decoder output feature map at the current scale, respectively. This allows the construction of the decoder vertex set V at the current scale. dec Vertex v in the encoder vertex set i enc Connected vertices A set, denoted as the set of connected vertices N(v i enc ), satisfying N(v i enc )= N represents the number of vertices. For the current scale decoder vertex set V dec The vertices in; Next, using the encoder vertex set V enc and the set of connected vertices N(v i enc After weighted aggregation, the aggregated feature v at the current scale is obtained. i agg The formula is as follows: in, and These are respectively the vertices obtained through neural networks. and vertex The projected vertex, a, obtained after projecting into a shared embedding space. j Here, represents the weighting coefficient, and || denotes the join operation. Represents a single convolutional layer. This represents the sigmoid function. This represents element-wise multiplication, and l represents the set of connected vertices N(v). i enc The index of the vertex in ).

5. The image tampering detection method based on image reconstruction according to claim 1, characterized in that, In the reconstruction-guided attention module, firstly, the difference between the original image X and the reconstructed image X' is calculated at the pixel level to obtain the difference mask m; then, the difference mask m and the initial enhanced feature map F are respectively processed. enh After convolution, the difference mask convolution features and the enhancement convolution features are obtained separately. The difference mask convolution features and the enhancement convolution features are multiplied together, and then sigmoid activation is applied to obtain the enhancement attention feature map. Finally, the enhancement attention feature map and the initial enhancement feature map F are combined. enh After summing, we obtain the attention-weighted output features.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 5.

Images