Image tampering detection method based on image reconstruction

By using image reconstruction network and multi-scale graph inference module in image tamper detection, combined with reconstruction to guide attention, the problems of limited generalization performance and poor robustness in the existing technology are solved, and more efficient image tamper detection is achieved .

CN118918452AActive Publication Date: 2024-11-08INSTITUTE OF COMPUTING INNOVATION ZHEJIANG UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202410927791.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-08
Estimated Expiration
2044-07-11

AI Technical Summary

Technical Problem

Existing image tamper detection methods have problems such as limited generalization performance, insufficient data sets and poor robustness. Especially when facing complex real-world tampering methods and image processing operations, it is difficult to effectively detect untrained data and mask tampering. trace.

Method used

Using the detection method based on image reconstruction, the reconstructed network applies metric learning on the decoder, so that the real image features are close to the feature space while the tampered image features are far away, and the multi-scale graph inference module and the reconstruction guide attention module are used. , mining the representation of distinguishing capabilities and focusing on fake areas to ensure the generalization and robustness of detection.

Benefits of technology

It improves the generalization and robustness of image tamper detection, reduces overfitting of specific tamper traces, can better identify unknown tamper patterns, and improves the accuracy and stability of detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918452A_ABST
    Figure CN118918452A_ABST
Patent Text Reader

Abstract

The invention discloses an image tampering detection method based on image reconstruction. According to the method, distribution of real images is modeled by using a reconstruction network, specifically, metric learning is applied to a decoder, so that features of the real images are close to each other in a feature space, the features of the real images are far away from features of tampered images, and the learned features are easier to detect unknown tampering traces; a multi-scale image reasoning module is used for further mining expressions with distinguishing capacity; and finally, a reconstruction guiding attention module is utilized to pay attention to a possible forged area, so that the learned image distribution difference is ensured, and a tampered image with an unknown tampering mode can be recognized more easily.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an image tampering detection method, and in particular to an image tampering detection method based on image reconstruction. Background Art

[0002] As an easy-to-understand information carrier, digital images are commonly seen in daily life and are the main way to obtain information in daily life. However, with the development of digital image processing technology, people can use more means to tamper with images. The development of some commonly used image editing software, such as PhotoShop and Beauty Camera, has made image tampering a very common thing. Ordinary people can easily tamper with images accordingly. The spread of maliciously tampered images through network media will affect people's judgment of the facts, allowing malicious users to easily tamper with images and cause adverse effects.

[0003] Image tampering detection is in the technical field of computer vision and image processing. It designs algorithms to identify and verify whether there is tampering or modification in the image. Research in this field focuses on developing automated and reliable technologies to detect and identify tampering traces such as tampering, erasure and splicing in images. These technologies are usually based on deep learning, image feature extraction and other methods to analyze features such as visual contradictions, discontinuities, compression traces and editing traces in the image to determine whether the image has been tampered with. Image tampering detection technology has a wide range of applications, including digital forensics, news verification, copyright protection and other fields. With the continuous development of image processing technology and the continuous evolution of tampering technology, research and improvement of image tampering detection technology is still ongoing.

[0004] Traditional image detection methods detect image tampering by exploiting specific low-level tampering traces, such as color filter array (CFA) analysis, local noise analysis, and double JPEG compression. These methods are usually based on statistical analysis and feature extraction of images to detect tampering traces in images. There are also some studies that exploit image consistency, most of which use deep learning twin networks to determine whether two input image blocks contain the same tampering features, such as EXIF ​​metadata or camera model features.

[0005] The following image tampering detection method is also disclosed:

[0006] Multiple network branches are used for tampering detection. For example, one network stream extracts RGB local texture features to capture visual artifacts, and the other stream uses noise features to simulate the inconsistency between the tampered area and the original area. Finally, the two features are fused for tampering detection.

[0007] A top-down path is used to extract hierarchical features of different scales, and then features of different scales are fused to detect whether the input image has been tampered with. In addition, features of different perspectives are obtained through data enhancement to learn semantically independent and more generalized features.

[0008] A deep learning neural network is designed to estimate data uncertainty and model uncertainty, which guides the model attention and improves the tampering prediction results by assigning high uncertainty to misclassified regions and low uncertainty to correctly classified regions.

[0009] The existing image tampering detection solutions still have the following problems:

[0010] Limited generalization performance: Due to the complexity and diversity of tampering methods used in the real world, the data distribution of actual tampered images is often different from the data distribution used to train the model, making it difficult for the model to generalize to data that has not been trained, and performance is usually severely degraded. At present, it is still difficult to find universal tampering traces. Different image tampering detection schemes often use different features, such as local texture, noise, or frequency domain distribution. This is the most concerned and difficult problem to solve.

[0011] Insufficient data sets. Deep learning is often trained through large data sets. Data sets are the first step in using deep learning to solve problems. Since the image tampering detection task is a relatively small branch in the image field, it has not received enough attention and a complete data set has not been established, resulting in overfitting of the model.

[0012] Poor robustness: When tampered images are spread on real social platforms, they are often subjected to image processing methods such as compression, blurring, brightness adjustment, and scaling, which in turn conceals the original traces of tampering, posing a greater challenge to the model's detection capabilities. Summary of the invention

[0013] In order to solve the needs and problems in the background technology, the present invention provides an image tampering detection method based on image reconstruction. The present invention uses a reconstruction network to model the distribution of real images, specifically, metric learning is applied on the decoder to make the features of the real image close to each other in the feature space, while the real image features and the tampered image features are far away from each other, and the learned features are easier to detect unknown tampering traces; then the multi-scale graph reasoning module is used to further mine the representation with distinguishing ability; finally, the reconstruction guided attention (RGA) module is used to focus on possible forgery areas, ensuring the difference in the learned image distribution, so that tampered images with unknown tampering patterns can be more easily identified.

[0014] The technical solution of the present invention is as follows:

[0015] 1. An image tampering detection method based on image reconstruction

[0016] First, the original image X is input into the reconstruction network for reconstruction learning to obtain the output features of the encoder, the output features of the decoder, and the reconstructed image X';

[0017] Next, the output features of the encoder and the decoder are input into the multi-scale graph reasoning module for multi-scale aggregation of information to obtain the initial enhanced feature map F enh ;

[0018] Then, the original image X, the reconstructed image X' and the initial enhanced feature map F enh They are input together into the reconstruction-guided attention module for attention weighting to obtain the attention-weighted output features; finally, the image is classified based on the attention-weighted output features to obtain the image tampering detection results.

[0019] The reconstruction network includes an encoder, a decoder and a global average pooling layer. The encoder and the decoder are connected. The output features of the encoder and the multi-scale output features of the decoder after the global average pooling layer are output as the reconstructed image X'.

[0020] The loss function used in the training of the reconstruction network includes metric learning loss, and the formula is as follows:

[0021]

[0022] in, represents the metric learning loss value, R and F represent the set of real image samples and the set of tampered image samples respectively, and N RR is the total number of real image and real image pairs, N RF is the total number of pairs of real and tampered images, d() is the pairwise distance function based on cosine distance, and represents the features of image sample i and image sample j after global pooling, and ‖‖2 represents the Euclidean distance.

[0023] The output features of the encoder and the decoder are input into the multi-scale graph reasoning module for multi-scale aggregation of information to obtain the initial enhanced feature map F enh , specifically:

[0024] First, the output features of the encoder and the output features of different blocks in the decoder are aggregated in a multi-scale manner to obtain aggregated features {v i agg}; Then the output features of the encoder are combined with the aggregate features v at each scale i aggAfter concatenation, it passes through the sigmoid function and two fully connected layers in turn to obtain the enhanced feature vector v of the current scale i enh ; Finally, the enhanced feature vectors {v i enh After combining them spatially, we obtain the initial enhanced feature map F enh .

[0025] In the multi-scale graph reasoning module, for the output features of each block in the decoder, first, the output features of the encoder and the current scale output features of the decoder are modeled as corresponding vertex sets, respectively, to obtain the encoder vertex set V enc and the decoder vertex set V of the current scale dec ,satisfy and h1 and w1 are the sizes of the encoder output feature map, h2 and w2 are the sizes of the decoder output feature map at the current scale, and then the decoder vertex set V of the current scale is constructed. dec The vertex v in the encoder vertex set i enc Connected Vertices Set, denoted as the connected vertex set N(v i enc ),satisfy N represents the number of vertices, is the decoder vertex set V of the current scale dec The vertices in

[0026] Next, using the encoder vertex set V enc and the connected vertex set N(v i enc ) After weighted aggregation, the aggregated feature v at the current scale is obtained i agg , the formula is as follows:

[0027]

[0028] in, and The vertices are respectively and vertices The projected vertices obtained after projection into a shared embedding space, a j is the weight coefficient, ‖ represents the connection operation, represents a single convolutional layer, ψ() represents the sigmoid function, represents element-by-element multiplication, l represents the connected vertex set N(v i enc )

[0029] In the reconstruction-guided attention module, first, the difference between the original image X and the reconstructed image X' is calculated at the pixel level to obtain the difference mask m; then, the difference mask m and the initial enhanced feature map F are respectively enh After convolution, the difference mask convolution feature and the enhanced convolution feature are obtained respectively. After multiplying the difference mask convolution feature and the enhanced convolution feature and performing sigmoid activation, the enhanced attention feature map is obtained. Finally, the enhanced attention feature map and the initial enhanced feature map F enh After addition, the attention-weighted output features are obtained.

[0030] 2. A computer device

[0031] The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0032] 3. A computer-readable storage medium

[0033] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method are implemented.

[0034] IV. A COMPUTER PROGRAM PRODUCT

[0035] The computer program product comprises a computer program / instructions which, when executed by a processor, implement the steps of the method.

[0036] Beneficial effects of the present invention:

[0037] 1. The detection method proposed in the present invention focuses more on learning real images and will not overfit on certain forgery traces, thereby achieving better generalization and robustness.

[0038] 2. Compared with tampered images that are rare and difficult to obtain, the present invention relies more on easily available real images for training.

[0039] 3. The multi-scale graph reasoning module proposed in this invention can better explore the subtle differences between true and false images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0042] Since the tampering methods used in the real world are complex and diverse, the types of image tampering data that the model can access are limited. Therefore, if the model can focus on learning the distribution of real images and at the same time widen the distribution difference between the tampered image features and the real image features, the tampered images can be distinguished. Therefore, the present invention proposes to use reconstruction learning to design a network specifically for modeling the distribution of real images. At the same time, metric learning is applied to the decoder so that the features of the real image are close to each other in the feature space, while the real image features and the tampered image features are far away from each other. After training, the real image has a good reconstruction effect, while the tampered image has a poor reconstruction effect, which ensures that due to the learned distribution difference, tampered images with unknown tampering patterns are easier to identify.

[0043] like Figure 1 As shown, the present invention comprises the following steps:

[0044] First, the original image X is input into the reconstruction network for reconstruction learning to obtain the output features of the encoder, the multi-scale output features of the decoder, and the reconstructed image X';

[0045] Since image tampering methods are usually diverse, it is considered more appropriate to explore the common features of real images than to overfit specific tampering traces in the training set. Therefore, the present invention proposes to perform reconstruction learning to restore only real images. The reconstruction network is intended to model only the distribution of real facial images. Therefore, the learned features are more easily used to detect unknown tampering traces. The reconstruction network includes an encoder, a decoder, and a global average pooling layer. The encoder and the decoder are connected, and the output features of the encoder and the multi-scale output features of the decoder after the global average pooling layer are recorded as the reconstructed image X'.

[0046] The loss function used by the reconstruction network during training includes metric learning loss, which makes the real image close in the embedding space, while the real image and the tampered image are far away. The formula is as follows:

[0047]

[0048] in, represents the metric learning loss value, R and F represent the set of real image samples and the set of tampered image samples respectively, and N RR is the total number of real image and real image pairs, N RF is the total number of real image and tampered image pairs, is a pairwise distance function based on cosine distance, and represents the features of image sample i and image sample j after global pooling, and ‖‖2 represents the Euclidean distance.

[0049] Next, the output features of the encoder and the decoder are input into the multi-scale graph reasoning (MGR) module for multi-scale aggregation of information to obtain the initial enhanced feature map F for subsequent reconstruction and guidance of attention. enh ;

[0050] The output features of the encoder and the decoder are input into the multi-scale graph reasoning (MGR) module for multi-scale aggregation of information to obtain the initial enhanced feature map F for subsequent reconstruction and guidance of attention. enh , specifically:

[0051] When the metric learning loss is applied to the decoder, useful information for distinguishing between real images and tampered images is also embedded in the decoder. In order to effectively use the tampering clues captured by the decoder features for the final classification, since different tampering techniques will cause tampering traces at different scales, the present invention proposes to mine the tampering information in a multi-scale manner to obtain a comprehensive representation, that is, to further mine the representation with distinguishing ability. The multi-scale graph reasoning scheme aggregates the difference information between the real face and the tampered face captured between the encoder and decoder of the reconstruction network in a multi-scale manner. The multi-scale graph reasoning module combines the latent features of the decoder block and the encoder output into a bipartite graph to comprehensively reason about the tampering clues.

[0052] First, the output features of the encoder and the output features of different blocks in the decoder (i.e., output features of different scales) are aggregated in a multi-scale manner to obtain aggregated features {v i agg}; Then the output features of the encoder are combined with the aggregate features v at each scale i agg After concatenation, it passes through the sigmoid function and two fully connected layers in turn to obtain the current scale and v i enc The augmented feature vector v with the same channel dimension i enh ; Finally, the enhanced feature vectors {v i enh After spatial combination, the initial enhanced feature map F is obtained for subsequent reconstruction to guide attention enh .

[0053] Graph reasoning from N(v i enc ) to enrich v i encThe feature representation of the decoder is used to better infer tampering clues. When aggregating information from the decoder to the encoder, the present invention maintains spatial correspondence to model local relationships, because tampering traces usually exist in continuous local areas. In the multi-scale graph reasoning module, for the output features of each block in the decoder (the output features of each scale), first, the output features of the encoder and the output features of the current scale i of the decoder are modeled as corresponding vertex sets, respectively, to obtain the encoder vertex set V enc and the decoder vertex set V of the current scale dec ,satisfy and h1 and w1 are the width and height of the encoder output feature map, h2 and w2 are the width and height of the decoder output feature map at the current scale, and then the decoder vertex set V of the current scale i is constructed. dec The vertex v in the encoder vertex set i enc Connected Vertices Set, denoted as the connected vertex set N(v i enc ),satisfy N represents the number of vertices, is the decoder vertex set V of the current scale dec The vertices in

[0054] Next, using the encoder vertex set V enc and the connected vertex set N(v i enc ) After weighted aggregation, the aggregated feature v at the current scale is obtained i agg , the formula is as follows:

[0055]

[0056] in, and The vertices are respectively and vertices The projected vertices obtained after projection into a shared embedding space, a j is the weight coefficient, used to represent v i,j dec v i enc ‖ represents the connection operation, represents a single convolutional layer, ψ() represents the sigmoid function, represents element-by-element multiplication, l represents the connected vertex set N(v i enc ) in the vertex index. The present invention uses a nonlinear transformation function ψ(·), based on vi enc Compute a vector of [0,1] values ​​to generate the pair v i enc 'Feature richness measurement at the channel level. In the information aggregation process, when v i enc 'When the weight of the corresponding channel is small, the present invention particularly enhances channel.

[0057] Then, the original image X, the reconstructed image X' and the initial enhanced feature map F enh They are input together into the reconstruction-guided attention module for attention weighting to obtain the attention-weighted output features;

[0058] Constrained by the reconstruction network, the reconstructed tampered image is visually different from the input tampered image. For this reason, the present invention proposes a reconstruction guided attention (RGA) module, which pays more attention to possible forgery areas. The reconstruction guided attention module guides the classification network to pay more attention to tampering traces for subsequent classification tasks. During training, both the tampered image and the real image are reconstructed to obtain the corresponding reconstructed image; the features corresponding to the tampered image and the real image are also processed in the graph reasoning module.

[0059] In the reconstruction-guided attention module, first, the difference between the original image X and the reconstructed image X' is calculated at the pixel level to obtain the difference mask m, which satisfies m = |X–X'|, where || represents the absolute value function; then, the difference mask m and the initial enhanced feature map F are respectively enh After convolution, the difference mask convolution feature and the enhanced convolution feature are obtained respectively. After multiplying the difference mask convolution feature and the enhanced convolution feature and performing sigmoid activation, the enhanced attention feature map is obtained. Finally, the enhanced attention feature map and the initial enhanced feature map F enh After adding, the attention-weighted output features are obtained, and the formula is as follows:

[0060]

[0061] F att =F′ enh +F enh ,

[0062] Among them, f1 and f2 represent convolution operations, σ represents the sigmoid function, represents element-wise multiplication. For simplicity, the spatial size of these tensors is omitted, and bilinear interpolation is used to maintain the spatial size appropriately in the above multiplication and addition operations.

[0063] Finally, after image classification based on the attention-weighted output features, the image tampering detection results are obtained.

[0064] In order to verify the present invention, the public license plate data set is used as a test data set. In this embodiment, 100,000 images in the real image data set CCPD-base + 80,000 self-built license plate tampering images are used, and data enhancements such as blur and color balance are added. CCPD contains vehicle pictures in different cities, different times, and different weather conditions in China. Each picture contains a license plate and provides corresponding annotation information, including the location, color, characters, fonts and other identification-related information of the license plate. The license plates in the CCPD data set cover various special cases and variations of Chinese license plates, which are of great reference value for studying algorithms such as license plate recognition. On this data set, the license plate tampering detection results of the method proposed in the present invention (RECCE) and the existing methods on the CCPD data set are shown in Table 1. It can be seen from Table 1 that the present invention achieves the highest ACC and AUC.

[0065] Table 1 shows the license plate tampering detection results of the existing method and the method proposed in this invention on the CCPD dataset

[0066]

[0067] In addition to the license plate tampering scenario, the present invention also achieves extremely high accuracy in the text image tampering scenario, as shown in Table 2, indicating that the present invention has relatively high generalization ability in multiple scenarios.

[0068] Table 2 shows the tampering detection results of the method proposed in this invention in the text image tampering scenario

[0069]

[0070]

[0071] The present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the image tampering detection method based on image reconstruction when executing the computer program.

[0072] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image tampering detection method based on image reconstruction are implemented.

[0073] The present invention also provides a computer program product, comprising a computer program / instruction, which implements the steps of the image tampering detection method based on image reconstruction when executed by a processor.

[0074] Finally, it should be noted that the above embodiments and explanations are only used to illustrate the technical solution of the present invention rather than to limit it. Those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope disclosed in the technical solution of the present invention, which should be included in the scope of protection of the claims of the present invention.

Claims

1. An image tampering detection method based on image reconstruction, characterized in that: The following steps are involved: First, the original image X is input into the reconstruction network for reconstruction learning to obtain the output features of the encoder, the output features of the decoder, and the reconstructed image X'; Next, the output features of the encoder and the decoder are input into the multi-scale graph reasoning module for multi-scale aggregation of information to obtain the initial enhanced feature map F enh ; Then, the original image X, the reconstructed image X' and the initial enhanced feature map F enh They are input together into the reconstruction-guided attention module for attention weighting to obtain the attention-weighted output features; finally, the image is classified based on the attention-weighted output features to obtain the image tampering detection results.

2. The image tampering detection method based on image reconstruction according to claim 1 is characterized in that: The reconstruction network includes an encoder, a decoder and a global average pooling layer. The encoder and the decoder are connected. The output features of the encoder and the multi-scale output features of the decoder after the global average pooling layer are output as the reconstructed image X'.

3. The image tampering detection method based on image reconstruction according to claim 1 is characterized in that: The loss function used in the training of the reconstruction network includes metric learning loss, and the formula is as follows: in, represents the metric learning loss value, R and F represent the set of real image samples and the set of tampered image samples respectively, and N RR is the total number of real image and real image pairs, N RF is the total number of pairs of real and tampered images, d() is the pairwise distance function based on cosine distance, and represents the features of image sample i and image sample j after global pooling, and ‖‖2 represents the Euclidean distance.

4. The image tampering detection method based on image reconstruction according to claim 1 is characterized in that: The output features of the encoder and the decoder are input into the multi-scale graph reasoning module for multi-scale aggregation of information to obtain the initial enhanced feature map F enh , specifically: First, the output features of the encoder and the output features of different blocks in the decoder are aggregated in a multi-scale manner to obtain aggregated features {v i agg }; Then the output features of the encoder are combined with the aggregate features v at each scale i agg After concatenation, it passes through the sigmoid function and two fully connected layers in turn to obtain the enhanced feature vector v of the current scale i enh ; Finally, the enhanced feature vectors {v i enh After combining them spatially, we obtain the initial enhanced feature map F enh .

5. The image tampering detection method based on image reconstruction according to claim 4 is characterized in that: In the multi-scale graph reasoning module, for the output features of each block in the decoder, first, the output features of the encoder and the current scale output features of the decoder are modeled as corresponding vertex sets, respectively, to obtain the encoder vertex set V enc and the decoder vertex set V of the current scale dec ,satisfy and h1 and w1 are the sizes of the encoder output feature map, h2 and w2 are the sizes of the decoder output feature map at the current scale, and then the decoder vertex set V of the current scale is constructed. dec The vertex v in the encoder vertex set i enc Connected Vertices Set, denoted as the connected vertex set N(v i enc ),satisfy N represents the number of vertices, is the decoder vertex set V of the current scale dec The vertices in Next, using the encoder vertex set V enc and the connected vertex set N(v i enc ) After weighted aggregation, the aggregated feature v at the current scale is obtained i agg , the formula is as follows: in, and The vertices are respectively and vertices The projected vertices obtained after projection into a shared embedding space, a j is the weight coefficient, ‖ represents the connection operation, represents a single convolutional layer, ψ() represents the sigmoid function, represents element-by-element multiplication, l represents the connected vertex set N(v i enc ) 6. The image tampering detection method based on image reconstruction according to claim 1 is characterized in that: In the reconstruction-guided attention module, first, the difference between the original image X and the reconstructed image X' is calculated at the pixel level to obtain the difference mask m; then, the difference mask m and the initial enhanced feature map F are respectively enh After convolution, the difference mask convolution feature and the enhanced convolution feature are obtained respectively. After multiplying the difference mask convolution feature and the enhanced convolution feature and performing sigmoid activation, the enhanced attention feature map is obtained. Finally, the enhanced attention feature map and the initial enhanced feature map F enh After addition, the attention-weighted output features are obtained.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Image tampering detection method based on shallow feature enhancement and attention mechanism

    CN115357944A

  • Image tampering detection method based on uncertainty guidance

    CN116778307A

  • Multi-scale image tampering detection and positioning method based on double-flow feature extraction

    CN118037641A

  • Variable bit rate generative compression method based on adversarial learning

    US11153566B1