Universal image tamper forensics method and system based on multi-scale feature fusion

Abstract

The application provides a general image tampering forensics method and system based on multi-scale feature fusion, which can not only detect various tampering technologies, but also has strong robustness, efficient and accurate extraction and identification of image features, and fast image authentication speed. The method comprises the following steps: step 1, preprocessing the image data to be tampered with to enhance the tampering effect; step 2, extracting the local information of the preprocessed image by using the rich model in steganalysis, and combining the multi-scale local information guided by the attention mechanism; step 3, extracting the overall semantic features of the image by using the image classification model; step 4, fusing the local information and the semantic features by using the bilinear pooling; step 5, inputting the fused features into the attention map of the spatial domain to learn the positioning of the image tampering area; and step 6, inputting the fused features into the full connection network of the classification to evaluate the image tampering forensics and determine whether the image is tampered with.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital image forensics, and particularly relates to a general image tampering forensics method and system based on multi-scale feature fusion. BACKGROUND

[0002] Digital image integrity forensics analysis mainly studies whether the content of a digital image is tampered with, and these tampering includes copy-paste, splicing, object deletion and other digital image post-processing operations. By analyzing the integrity of a digital image, researchers can identify fake images in news reports and fake evidence presented in court, thereby ensuring the authenticity and reliability of image content information.

[0003] Early forensics of tampered images often used tampered region positioning methods, which analyzed certain specific statistical features in combination with the characteristics of tampering to determine whether tampering existed. However, shallow methods are limited to specific types of tampering, and different methods require manual redesign of features, which are not universal, and do not have good forensics effects on deep learning-based tampering methods. Therefore, deep learning-based forensics methods have emerged.

[0004] The key to image forensics often lies in determining whether there are traces of tampering in the image, and this conclusion can be regarded as a binary classification problem. In recent years, deep learning has led the technological progress in the field of computer vision, among others. MFCN introduces a detection branch to learn the boundary information of the splicing region, which has shown great potential in image tampering detection tasks and effectively improves the positioning accuracy of tampered regions. Based on this idea, H-LSTM applies a J-LSTM-based image block comparison method to detect the boundaries of tampered regions and real regions, and further proposes a hybrid encoder-decoder structure to improve algorithm performance. A network based on adaptive attention and residual refinement fuses position and channel attention features and optimizes the coarse positioning results through a residual refinement module. Although the above methods can achieve good performance, most of these methods focus on a single tampering method.

[0005] Current tampered image detection methods are mainly classified into detection models based on time domain information and spatial domain information, and existing general image forensics methods mainly have the following defects:

[0006] 1) Most forensics methods are only applicable to a single tampering type: as the number of tampering types increases, multiple tampering types may occur in an image, such as diffusion model-generated overlays and multiple image mixing and superimposition. Therefore, how to detect multiple tampering types will be a research focus in the field of image forensics in the future;

[0007] 2) The generalization performance of image forensics is low: the end-to-end characteristics of the image classification network bring convenience to the designers of the system, but the cost is that it is extremely limited to the training data, and often powerless to images outside the data distribution, and even cannot achieve good detection effect on the tampering of the same type of tampering method on different types of images. SUMMARY

[0008] The present application is carried out in order to solve the above problems, and aims to provide a general image tampering forensics method and system based on multi-scale feature fusion, which can not only detect various tampering techniques, but also has strong robustness, efficient and accurate extraction and recognition of image features, and fast image authentication speed and other advantages.

[0009] In order to achieve the above purpose, the present application adopts the following scheme:

[0010] <Method>

[0011] As shown in Figure 1 The present application provides a general image tampering forensics method based on multi-scale feature fusion, comprising the following steps:

[0012] Step 1, pre-processing the image data to be tampered with for forensics, enhancing the tampering effect;

[0013] Step 2, extracting the local information of the pre-processed image using the rich model in steganalysis, and combining the multi-scale local information guided by the attention mechanism;

[0014] Step 3, using an image classification model to extract the overall semantic features of the pre-processed image;

[0015] Step 4, fusing the local information and semantic features through bilinear pooling;

[0016] Step 5, inputting the fused features into the attention map of the spatial domain to learn the positioning of the image tampering area;

[0017] Step 6, inputting the fused features into the classification fully connected network to evaluate the image tampering forensics and determine whether the image has been tampered with.

[0018] Preferably, the general image tampering forensics method based on multi-scale feature fusion provided by the present application comprises the following sub-steps in step 2:

[0019] Step 2.1, using the rich model filter in the steganalysis model to establish a noise residual extraction model of the image;

[0020] Step 2.2, dividing the pre-processed image into different regions, and giving different position encodings for feature representation learning for different regions;

[0021] Step 2.3, feature fusion is performed by giving different weights to the features of different regions of the image using the attention map.

[0022] Preferably, the universal image tampering forensics method based on multi-scale feature fusion provided by the present application adopts a VGG image classification model to extract the overall semantic features of the preprocessed image in step 3, and obtains the global semantic features of the image for tampering detection.

[0023] Preferably, the universal image tampering forensics method based on multi-scale feature fusion provided by the present application obtains the probability of the spatial position information of the corresponding image being tampered with according to the attention map in step 5, and then performs tampering positioning of the image.

[0024] In step 6, the fused features are input into the classification full connection network, the positioning of step 5 and the forensics performance of the full connection network are trained using the training data set, the image tampering forensics process is evaluated using the verification data set, and it is determined whether the image is tampered with.

[0025] <SYSTEM>

[0026] Further, the present application also provides a universal image tampering forensics system based on multi-scale feature fusion capable of automatically implementing the above-mentioned <method>, comprising:

[0027] A preprocessing unit for preprocessing image data to be subjected to tampering forensics to enhance the tampering effect.

[0028] A local feature extraction unit for extracting local information of the preprocessed tampered image using a rich model in steganalysis and combining multi-scale local information through an attention mechanism.

[0029] A semantic feature extraction unit for extracting overall semantic features of the preprocessed image using an image classification model.

[0030] A fusion unit for fusing local information and semantic features through bilinear pooling.

[0031] A tampering positioning unit for inputting the fused features into the attention map of the spatial domain to learn the positioning of the tampered region of the image.

[0032] A tampering judgment unit for inputting the fused features into the classification full connection network to evaluate the image tampering forensics process and determine whether the image is tampered with.

[0033] A control unit in communication with the preprocessing unit, the local feature extraction unit, the semantic feature extraction unit, the fusion unit, the tampering positioning unit, and the tampering judgment unit to control their operation.

[0034] The universal image tampering forensics system based on multi-scale feature fusion provided by the application can further comprise an input display unit connected in communication with the control unit, used for inputting operation instructions by a user and performing corresponding display.

[0035] Preferably, in the local feature extraction unit, a rich model filter in the steganalysis model is used to establish a noise residual extraction model of the image, and the preprocessed image is divided into different regions, different position encodings are given for different regions for feature representation learning, and different weight information is given to features of different regions of the image by using an attention map for feature fusion.

[0036] Preferably, in the local feature extraction unit, each image is equally divided into a plurality of different regions.

[0037] Preferably, in the semantic feature extraction unit, a VGG image classification model is used to extract overall semantic features of the preprocessed image, and global semantic features of the image are obtained for tampering detection.

[0038] Preferably, in the tampering positioning unit, according to the attention map, the probability that the spatial position information of the corresponding image is tampered is obtained, and then the tampering positioning of the image is performed.

[0039] In the tampering judgment unit, the fused features are input into a classified full connection network, the positioning of the tampering positioning unit and the forensics performance of the full connection network are trained by using a training data set, a verification data set is used to evaluate the image tampering forensics process, and it is determined whether the image is tampered.

[0040] Effects of the application

[0041] The application extracts local information of the preprocessed image by using a rich model in steganalysis, combines multi-scale local information by using an attention mechanism, extracts overall semantic features of the image by using an image classification model, fuses the local information and the overall semantic features together, trains by using the attention mechanism, positions the tampering region, and determines whether the image is tampered, thereby realizing accurate and rapid identification of various tampering technologies (tampering types), having strong robustness, and greatly improving the accuracy of image forgery identification forensics.

[0042] Compared with the prior art deep learning scheme for tampered image forensics, the application combines the destruction of specific statistical characteristics by image tampering to add prior knowledge and additional information to help the deep model for tampering detection: in the local tampering detection of the image, the application helps the classical rich model in steganalysis to model the noise residual of the image in multiple scales, and explores the abnormalities such as strong contrast difference and unnatural tampering boundary, in the semantic tampering detection of the image, the application first excavates the high-level semantic information of the image through the existing image classification model, and finally fuses the two branches through a bilinear pooling layer, and combines the successful effect of the bilinear pooling mechanism applied in the image fine-grained classification, since the image tampering detection and the fine-grained classification are consistent in the detection of subtle textures, therefore, the performance of feature combination can also be effectively improved, through experiments, it can be proved that the double-flow framework of separating and extracting the features of the two and then fusing them for forensic detection can effectively improve the forensic accuracy of the single feature detection rather than causing a large amount of information redundancy. In addition, the application can truly learn the inconsistency between the observed tampered pictures and the real pictures rather than overfitting to the influence of a certain specific tampering method on part of the statistical characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of a general image tampering forensics method based on multi-scale feature fusion related to the application;

[0044] Figure 2 A forensic model framework related to the embodiment of the application. DETAILED DESCRIPTION

[0045] The specific implementation of the general image tampering forensics method and system based on multi-scale feature fusion related to the application will be described in detail below in combination with the drawings.

[0046] <Embodiment I>

[0047] As shown in the figure, the general image tampering forensics method based on multi-scale feature fusion related to the embodiment I has the following specific steps: Figures 1-2

[0048] Step 1, based on the CASIA dataset and the Columnbia dataset, post-processing is performed on the tampered image data (the image to be tampered with for forensics) to enhance the tampering effect.

[0049] ​CASIA1.0 contains 921 tampered images, CASIA2.0 contains 5123 images, the tampering techniques include splicing and copy-paste, the tampered areas include animals, textures, natural scenes, etc., and the dataset has post-processed the tampered images to enhance the tampering effect. The Columnbia dataset contains 183 spliced images, and the tampered area is an indoor scene. This embodiment uses CASIA2.0 as the training dataset to train the method, and uses CASIA1.0 and Columnbia as the verification dataset to evaluate the effect of the model.

[0050] Image preprocessing aims to eliminate information irrelevant to classification, enhance pixel correlation, and maximize the simplification of input data, thereby improving the accuracy and robustness of feature extraction, matching, and recognition. Due to limitations such as network bandwidth and storage capacity, social platforms often compress pictures shared online, interfering with image feature extractor performance.

[0051] Step 2, use the rich model in steganalysis to extract the local information of the tampered picture preprocessed in step 1, and guide the combination of multi-scale local information through the Attention mechanism.

[0052] Step 2.1, establish an image noise residual extraction model with the help of the rich model filter commonly used in steganalysis models;

[0053] Step 2.2, divide the image into different regions, and give different position encodings for different regions for feature representation learning; in this embodiment, after the filter construction operation of each image 224*224, the image is equally divided into 8 parts (8 regions), and each part has a size of 28*28 pixels;

[0054] Step 2.3, use the attention map to give different weight information to the features of different regions of the image for feature fusion.

[0055] Step 3, use a classic image classification model such as VGG to extract the overall semantic features of the tampered picture preprocessed in step 1, and add the global semantic feature information to the tampering detection.

[0056] Step 4, fuse the local information and semantic features through bilinear pooling.

[0057] Bilinear pooling in the present application is mainly used for the fusion of features extracted by the double-flow network. For features x and features y extracted from the same sample, bilinear pooling is used to obtain a vector after the fusion of the two features, which is then used for model classification. The specific process is as follows:

[0058] For image I, the global and local features f A(I, u) e R T×M and f B (I, u) e R T×M The following operations are performed;

[0059]

[0060]

[0061]

[0062]

[0063]

[0064] Step 5: input the fusion features into the attention map of the spatial domain to learn the positioning of the image tampering region, and further improve the detection effect of the model.

[0065] The attention map is interpreted as the probability that the spatial position information of the corresponding image is tampered with, and the tampering positioning of the image is performed.

[0066] Step 6: input the fusion features into the classification fully connected network, train the forensic performance of the attention positioning module in step 5 and the fully connected network using the training data set, and evaluate the model using the validation data set.

[0067] In this embodiment, the network model is built and trained using the pytorch framework, the loss parameter λ of the attention map is finally determined to be set to 0.8 through comparative experiments, the batch size of the experiment input is 32, and Xception pre-trained on ImageNet is used as the dual-flow classification network. In order to obtain the mask corresponding to the forged image, the mask is generated by subtracting the corresponding true and false image frames in the supervised experiment. The parameters of the attention layer are initialized in a random manner. In the experiment, the Adam optimizer and the learning rate of 0.0002 are selected. In the cross-dataset test, the robustness is good, and the convergence speed is relatively fast during training.

[0068] [Example Two]

[0069] In this embodiment two, a general image tampering forensics system based on multi-scale feature fusion capable of automatically implementing the above method of the application is provided, which includes a preprocessing part, a local feature extraction part, a semantic feature extraction part, a fusion part, a tampering positioning part, a tampering judgment part, an input display part, and a control part.

[0070] The preprocessing part performs the content described in step 1 above, and pre-processes the image data to be tampered with for forensics, enhancing the tampering effect.

[0071] The local feature extraction unit performs the content described in step 2 above, extracts the local information of the preprocessed image using the rich model in steganalysis, and guides the combination of multi-scale local information through the attention mechanism.

[0072] The semantic feature extraction unit performs the content described in step 3 above, and extracts the overall semantic features of the preprocessed image using an image classification model.

[0073] The fusion unit performs the content described in step 4 above, and fuses the local information and semantic features through bilinear pooling.

[0074] The tampering positioning unit performs the content described in step 5 above, and inputs the fused features into the attention map in the spatial domain to learn the positioning of the image tampering region;

[0075] The tampering judgment unit performs the content described in step 6 above, and inputs the fused features into the classification fully connected network to evaluate the image tampering forensics process and determine whether the image has been tampered with.

[0076] The input display unit is used for the user to input operation instructions, and displays the data of the corresponding unit according to the operation instructions.

[0077] The control unit is in communication with the preprocessing unit, the local feature extraction unit, the semantic feature extraction unit, the fusion unit, the tampering positioning unit, the tampering judgment unit, and the input display unit, and controls the operation of them.

[0078] The above embodiments are only examples of the technical solutions of the present application. The general image tampering forensics method and system based on multi-scale feature fusion involved in the present application are not limited to the content described in the above embodiments, but are subject to the scope defined in the claims. Any modification, supplement or equivalent replacement made by the skilled in the art on the basis of the above embodiments is within the scope claimed by the claims of the present application.

Claims

1. A general image tampering forensics method based on multi-scale feature fusion, characterized in that, Includes the following steps: Step 1: Preprocess the image data to be tampered with for evidence collection to enhance the tampering effect; Step 2: Use rich models in steganalysis to extract local information from the preprocessed image, and guide the combination of multi-scale local information through an attention mechanism; Step 3: Use an image classification model to extract the overall semantic features of the preprocessed image; Step 4: Fuse local information with semantic features using bilinear pooling; Step 5: Input the fused features into the attention map in the spatial domain to learn the localization of the image tampering region; Step 6: Input the fused features into the fully connected network for classification to evaluate the image tampering evidence collection process and determine whether the image has been tampered with. Step 2 includes the following sub-steps: Step 2.1: Establish an image noise residual extraction model using the rich model filter in the steganalysis model; Step 2.2: Divide the preprocessed image into different regions, and provide different location codes for different regions for feature representation learning; Step 2.3: Use attention maps to assign different weights to the features of different regions of the image for feature fusion.

2. The general image tampering forensics method based on multi-scale feature fusion according to claim 1, characterized in that: in, In step 3, the VGG image classification model is used to extract the overall semantic features of the preprocessed image, and the global semantic features of the image are used for tamper detection.

3. The general image tampering forensics method based on multi-scale feature fusion according to claim 1, characterized in that: in, In step 5, the probability of the spatial location information of the corresponding image being tampered with is obtained based on the attention map, and then the tampering location of the image is performed. In step 6, the fused features are input into the fully connected network for classification. The forensic performance of the localization and fully connected network in step 5 is trained using the training dataset. The image tampering forensic process is evaluated using the verification dataset to determine whether the image has been tampered with.

4. A general image tampering evidence collection system based on multi-scale feature fusion, capable of automatically implementing the general image tampering evidence collection method based on multi-scale feature fusion as described in any one of claims 1 to 3, characterized in that, include: The preprocessing department preprocesses the image data to be tampered with for evidence collection to enhance the tampering effect; The local feature extraction unit uses a rich model in steganalysis to extract local information from the preprocessed image and guides the combination of multi-scale local information through an attention mechanism. The semantic feature extraction unit uses an image classification model to extract the overall semantic features of the preprocessed image; The fusion section uses bilinear pooling to fuse local information with semantic features. The localization unit modifies the image by inputting the fused features into the attention map in the spatial domain to learn the localization of the image tampering region; The tampering judgment unit inputs the fused features into the fully connected network for classification to evaluate the image tampering evidence collection process and determine whether the image has been tampered with. The control unit is communicatively connected to the preprocessing unit, local feature extraction unit, semantic feature extraction unit, fusion unit, tamper location unit, and tamper judgment unit, and controls their operation.

5. The general image tampering evidence collection system based on multi-scale feature fusion according to claim 4, characterized in that, Also includes: The input display unit is connected in communication with the control unit and is used to allow users to input operation commands and display the corresponding commands.

6. The general image tampering evidence collection system based on multi-scale feature fusion according to claim 4, characterized in that: in, In the local feature extraction section, a noise residual extraction model for the image is established using the rich model filter in the steganalysis model. The preprocessed image is divided into different regions, and different position codes are given for different regions for feature representation learning. Attention maps are used to give different weight information to the features of different regions of the image for feature fusion.

7. The general image tampering evidence collection system based on multi-scale feature fusion according to claim 4, characterized in that: in, The local feature extraction unit divides each image into multiple distinct regions.

8. The general image tampering evidence collection system based on multi-scale feature fusion according to claim 4, characterized in that, Also includes: In the semantic feature extraction section, the VGG image classification model is used to extract the overall semantic features of the preprocessed image, and the global semantic features of the image are used for tamper detection.

9. The general image tampering evidence collection system based on multi-scale feature fusion according to claim 4, characterized in that: in, In the tampering localization unit, the probability of the spatial location information of the corresponding image being tampered with is obtained based on the attention map, and then the tampering localization of the image is performed. In the tampering detection unit, the fused features are input into the fully connected network for classification. The localization of the tampering localization unit and the forensic performance of the fully connected network are trained using the training dataset. The image tampering forensic process is evaluated using the verification dataset to determine whether the image has been tampered with.

Images