A tampered image detection method based on multi-scale generative adversarial network
By building a high-quality tampered image data set based on multi-scale generation of adversarial networks, the problem of insufficient training data in the prior art is solved, and the performance and accuracy of image tampered detection are improved.
Patent Information
- Application Number
- CN202010883513.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-28
- Publication Date
- 2025-02-28
- Estimated Expiration
- 2040-08-28
AI Technical Summary
The existing deep learning-based image tamper detection technology lacks high-quality and rich training data, resulting in low model detection performance and prone to model overfitting, reducing detection accuracy.
The method of generating adversarial networks based on multi-scale is adopted. By synthesizing basic copy-paste images from the standard image dataset, a preliminary tampered image dataset is constructed, and an adversarial network model is generated through multi-scale, the generator loss function is optimized, the generator and discriminator are alternately trained, the tampered images at different scales are output, and a multi-level tampered image dataset is constructed, and the convolutional neural network is finally trained using this dataset.
By generating high-quality and rich tampered image datasets, the performance of image tampered detection is improved, the generalization ability of the model is enhanced, the model overfitting phenomenon is avoided, and the detection accuracy is improved.
Smart Images
Figure CN112102261B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image information security technology, and in particular to a tampered image detection method based on a multi-scale generative adversarial network. Background Art
[0002] With the development of image processing and image editing technology, more and more malicious tampered images have been produced, which has had a great negative impact on society and has also inspired a large number of scholars to study image tampering detection. Common image tampering includes three types: splicing, copy-move, and removal. Image splicing can copy part of an image and paste it into other images. Copy-move is to copy and paste areas in the same image. Removal will eliminate part of the image and then repair it. In order to increase the difficulty of detecting and identifying tampered images, post-processing operations such as smoothing and blurring are usually performed after the image is tampered.
[0003] The current passive image tampering detection based on deep learning can automatically find and learn abstract image features. The robustness and detection efficiency of the model have been improved to a certain extent, but there are still the following problems: lack of high-quality and rich training data, and low-quality and small amount of training data will affect the detection performance of the model, and it is easy to cause model overfitting, which reduces the detection accuracy. At present, there are usually three sources of training data: first, obtain standard tampered image data sets, but the number is small; second, use the information in the target detection data set to automatically synthesize pictures. This method does not post-process the image, which will reduce the quality of the tampered image; third, hire professional annotation personnel to create tampered images, but the cost of doing so is very high. Therefore, in order to improve the detection performance of image tampering, how to easily obtain high-quality and rich training data is crucial. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a tampered image detection method based on a multi-scale generative adversarial network, so as to improve the detection performance of tampered images by generating a high-quality and rich tampered image data set.
[0005] The purpose of the present invention can be achieved by the following technical solution: A method for detecting tampered images based on a multi-scale generative adversarial network, comprising the following steps:
[0006] S1. Synthesize basic copy-paste images from a standard image dataset and obtain the XML file corresponding to the synthesized image to construct a preliminary tampered image dataset;
[0007] S2. Build a model based on a multi-scale generative adversarial network, using a preliminary tampered image dataset as the input of the model to set model-related parameters;
[0008] S3, optimize the generator of the multi-scale generative adversarial network to learn the edge features of the tampered area in the image;
[0009] S4, alternately train the generator and the discriminator to output tampered images at different scales to construct a multi-level tampered image dataset;
[0010] S5, training convolutional neural network using multi-level tampered image dataset;
[0011] S6. Input the tampered image to be detected into the trained convolutional neural network to obtain the corresponding detection result.
[0012] Furthermore, the step S1 specifically includes the following steps:
[0013] S11, selecting a target image and a picture to be processed from a standard image dataset;
[0014] S12, using the target image as the background, pasting the portion of the tampered area mask in the image to be processed to a random position of the target image, and synthesizing to obtain a basic copy-and-paste image;
[0015] S13, reading the XML file corresponding to the image to be processed from the standard image data set to read out the bounding box position information corresponding to the tampered area mask, and then modifying the XML file corresponding to the target image, and using the modified XML file as the XML file corresponding to the copied and pasted image;
[0016] S14. According to the order of images in the standard image dataset, steps S11 to S13 are repeated in sequence to synthesize multiple basic copy-and-paste images and obtain corresponding xml files to construct a preliminary tampered image dataset.
[0017] Furthermore, the step S12 specifically includes the following steps:
[0018] S121, reading a mask image corresponding to the image to be processed, and reading out the regional masks of all objects in the mask image;
[0019] S122, randomly selecting a region mask of a type of object as a tampering region mask;
[0020] S123, finding an area in the image to be processed that corresponds to the tampered area mask, and intercepting the area from the image to be processed as the tampered source image;
[0021] S124, pasting the tampered source image to a random position of the target image to synthesize a basic copy-and-paste image.
[0022] Furthermore, the specific process of step S13 is: reading the xml file corresponding to the image to be processed from the annotation folder of the standard image data set, and reading out the bounding box position information corresponding to the tampered area mask selected in step S122;
[0023] According to the read bounding box position information and the position of the pasted area in the copy-paste image, the XML file corresponding to the target image is modified, the detection category is changed to "tamper", the bounding box position information in the target image is changed to the position information of the pasted area in the copy-paste image, and the modified XML file is used as the XML file corresponding to the copy-paste image.
[0024] Furthermore, the step S2 specifically includes the following steps:
[0025] S21. Construct a multi-scale generative adversarial network model including a discriminator network and a generator network;
[0026] S22. According to the preliminary tampered image data set, set relevant parameters of the model.
[0027] Furthermore, in step S21, the generator network includes multiple generators corresponding to different scales, and the multiple generators are in a pyramid structure, and for the generator with the smallest scale at the top layer, its input only contains spatial Gaussian noise;
[0028] For the generators of each scale below the top layer, the input contains spatial Gaussian noise and the upsampled image generated by the generator of the previous scale;
[0029] For the generators of each scale below the top layer, the output results are as follows: the upsampled image generated by the previous scale generator plus the spatial Gaussian noise image is convolved to output a residual image. The residual image is then added with the upsampled image generated by the previous scale generator, which is the output of the generator.
[0030] Furthermore, the step S3 specifically optimizes the loss function of the generator, including the following steps:
[0031] S31. Define reconstruction background loss to speed up training;
[0032] S32. Define edge loss to constrain the shape of the tampered area.
[0033] Furthermore, the background reconstruction loss function in step S31 is specifically:
[0034]
[0035] Among them, N bg is the total number of pixels in the background, m iis the i-th pixel in M, where M is the copy-paste image, mask i is the tampered area mask M at position i mask The pixel value in tar i M tar The i-th pixel in M tar is the target image;
[0036] The marginal loss function is specifically:
[0037]
[0038] Among them, N edge is the number of edge pixels, e i is the value of the edge mask at position i, tam i M tam The i-th pixel in M tam To tamper with the source image.
[0039] Furthermore, the step S4 specifically includes the following steps:
[0040] S41. Train the model in sequence from the smallest scale to the largest scale, where the training loss of each generator includes adversarial loss and gradient loss. Each generator has a corresponding Markov discriminator, and the loss of the discriminator only includes adversarial loss. The overall loss of the entire generator network is a compound of adversarial loss and gradient loss.
[0041] S42, training is performed according to a preset number of iterations, and the generated images at different numbers of iterations are jointly constructed into a multi-level tampered image dataset.
[0042] Furthermore, the overall loss of the entire generator network in step S41 is specifically:
[0043] L G =λ adv L adv +λ grad L grad
[0044]
[0045] Among them, L adv To combat the loss, L grad is the gradient loss, λ adv and λ grad L adv and L grad The hyperparameters, G x (M) is the gradient image of the copied and pasted image M in the horizontal direction, G y(M) is the gradient image of the copy-paste image M in the vertical direction, and f(M) is the fusion image of the copy-paste image M and the output of the multi-scale generative adversarial network f.
[0046] Compared with the prior art, the present invention has the following advantages:
[0047] 1. The present invention is based on a multi-scale generative adversarial network structure, which can learn rich features from images of different sizes. Since the scales of the generated tampered images are different, the presentation degree of the tampering traces is different. The multi-scale network structure can learn features at different scale levels, thereby enhancing the generalization ability of the model. It no longer relies on large-scale data sets, and can generate high-quality tampered pictures based on a single image. At the same time, it can expand the number of generated images, which is conducive to the subsequent high-quality training of the convolutional neural network, thereby improving the detection performance of tampered images.
[0048] 2. The present invention optimizes the loss function of the generator so that the generator can learn the characteristics of tampered edges and accelerate the training speed, thereby ensuring that the images generated by the generative adversarial network model are more natural, effectively improving the quality of tampered images, avoiding overfitting during tampered image detection, and improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0050] Figure 2 A schematic diagram of the process of constructing a multi-level tampered image data set in an embodiment;
[0051] Figure 3a Schematic diagram of tampering area mask in the embodiment;
[0052] Figure 3b This is a schematic diagram of tampering with a source image in an embodiment;
[0053] Figure 3c Schematic diagram of the target image in the embodiment;
[0054] Figure 3d A schematic diagram of a copy-and-paste image synthesized in the embodiment;
[0055] Figure 4 A schematic diagram of a multi-scale generative adversarial network structure in an embodiment;
[0056] Figure 5a An example of a multi-layer tampered image generated in an embodiment;
[0057] Figure 5b This is an example of a multi-layered tampered image generated in another embodiment. DETAILED DESCRIPTION
[0058] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0059] Example
[0060] like Figure 1 As shown, a method for detecting tampered images based on a multi-scale generative adversarial network includes the following steps:
[0061] S1. Synthesize basic copy-paste images from a standard image dataset and obtain the XML file corresponding to the synthesized image to construct a preliminary tampered image dataset;
[0062] S2. Build a model based on a multi-scale generative adversarial network, using a preliminary tampered image dataset as the input of the model to set model-related parameters;
[0063] S3, optimize the generator of the multi-scale generative adversarial network to learn the edge features of the tampered area in the image;
[0064] S4, alternately train the generator and the discriminator to output tampered images at different scales to construct a multi-level tampered image dataset;
[0065] S5, training convolutional neural network using multi-level tampered image dataset;
[0066] S6. Input the tampered image to be detected into the trained convolutional neural network to obtain the corresponding detection result.
[0067] The entire training and testing of this embodiment are run under the deep learning open source framework TensorFlow under the Ubuntu 16.04.6 system, the graphics card model is Nvidia GeForce RTX 2080TI, the CPU model is Intel Core i9-9900K, the CUDA version is CUDA 10.1, and the cuDNN version is cuDNN v7.5.0.
[0068] This embodiment uses the PASCAL VOC standard dataset as the standard image dataset. The dataset includes about 20,000 images and corresponding mask images, which provides a complete set of standardized and excellent datasets for image recognition and classification. The specific working process of this embodiment using the above method to construct a multi-level tampered image dataset is as follows: Figure 2 As shown, including:
[0069] Step 1: Synthesize basic copy-paste images from a standard image dataset:
[0070] The tampered region mask, tampered source image and target image (such as Figure 3a to Figure 3cAs shown), with the tampered source image as the foreground and the target image as the background, a simple copy-paste image is created (as shown Figure 3d As shown) as the input of the multi-scale adversarial generation network, specifically,
[0071] (a1) Sequentially extract an image from the PASCAL VOC dataset as the target image;
[0072] (a2) Randomly select a picture from the remaining pictures except the target image as the picture to be processed;
[0073] (a3) reading the mask image corresponding to the image to be processed, reading out the regional masks of all objects in the true mask image, and randomly selecting the regional masks of a class of targets as the tampered regional masks;
[0074] (a4) according to the tampered region mask selected in the mask image, finding the region in the image to be processed corresponding to the tampered region mask, cutting out the region from the image to be processed, and pasting the cutout portion to a random position of the target image to form a synthetic copy-and-paste image;
[0075] (a5) Read the XML file corresponding to the image to be processed from the Annotations folder (i.e., the image annotation folder) of the dataset, read out the bounding box position information corresponding to the tampered area mask selected in step (a3), the position information is the upper left corner coordinates and the lower right corner coordinates of the bounding box, and modify the XML file (script file, including the tampering type and tampered image coordinates) corresponding to the target image according to the read bounding box position information and the position of the pasted area in the copy-paste image, modify the detection category to "tamper" (i.e., tampering type), change the bounding box position information in the target image to the position information of the pasted area in the copy-paste image, and use the modified XML file as the XML file corresponding to the copy-paste image, thus completing a copy-paste image and its corresponding XML file;
[0076] (a6) Perform the operations of step (a2) to step (a5) on the next image in the PASCAL VOC dataset, and loop the operations to the last image to create a synthetic basic tampered image dataset.
[0077] Step 2: Build a model based on a multi-scale generative adversarial network, which includes a discriminator network and a generator network, and set the corresponding parameters. Specifically,
[0078] (b1) Figure 4 As shown, the multi-scale adversarial generation network contains a pyramid-structured generator {G0,...,G N}, the entire adversarial generation network starts from G N Start processing, G Nis the minimum scale, the scales are getting larger and larger, and G0 is the maximum scale, respectively for images For training, the image x n It is obtained by downsampling the image x (downsampling means scaling the image size to meet network requirements), and the sampling factor is r n , where r>1, each generator G n Responsible for generating the corresponding x n The model generates image samples from the smallest scale, and then sequentially passes through the generator to gradually generate image samples of the largest scale. The generator G corresponding to each scale n There is random noise Z n As input (0≤n≤N);
[0079] (b2) At the minimum scale, the generator G N The input contains only spatial Gaussian noise Z N , and maps it to
[0080]
[0081] (b3) For a larger scale, the generator G n (n<N) not only the noise Z n (n<N) as input, and also receives the upsampling of the previous scale generated image (Upsampling means expanding the image size to meet network requirements) as input:
[0082]
[0083] (b4) All generators have similar structures. When the generator is G n (n<N), the upsampling of the previous scale generated image Add the noisy image Z n , after the convolution operation, a residual image is output, and then Finally got The generator G n The output of the generator is G n The operation can be expressed as follows:
[0084]
[0085] (b5) According to the model and dataset, set the relevant parameters of the model, adjust the model input size to 256×256, in the multi-scale structure, the minimum scale is 25×25, the maximum scale is 250×250, the upsampling multiple is set to 10, and the learning rate is fixed to 2×10 -4 , the hyperparameter λ adv and λgrad Set to 5 and 20 respectively.
[0086] Step 3: Further optimize the edge traces of the tampered area to make the image generated by the model more natural. Specifically,
[0087] (c1) In order to further optimize the edge traces of the tampered area, the loss function of the generator is optimized to encourage the generator to learn the edge features of the tampered area. In order to speed up the training process, L1 loss is used to reconstruct the background:
[0088]
[0089] Among them, N bg is the total number of pixels in the background, m i is the i-th pixel in M, where M is the copy-paste image, mask i is the tampered area mask M at position i mask The pixel value in tar i M tar The i-th pixel in M tar is the target image;
[0090] (c2) In order to constrain the shape of the tampered area, the edge loss is defined:
[0091]
[0092] Among them, N edge is the number of edge pixels, e i is the value of the edge mask at position i, tam i M tam The i-th pixel in M tam To tamper with the source image.
[0093] Step 4: Alternately train the generator and the discriminator to output tampered images at different scales to construct a multi-level tampered image dataset. Specifically,
[0094] (d1) The model is trained sequentially from the smallest scale to the largest scale. When each generator is trained, it will be fixed. The training loss of the nth generator includes the adversarial loss L adv and the gradient loss L grad , such as the formula:
[0095]
[0096] (d2) During the training of the model, each generator G n There is a corresponding Markov discriminator D n To distinguish true from false, the discriminator D n The structure and generator Gn Similarly, in order to increase training stability, only the adversarial loss L of WGAN-GP is selected adv ;
[0097] (d3) In order to make the tampered area and the background in the image have similar color change trends, a custom gradient loss is proposed by referring to the idea of gradient field in Poisson fusion. For the fused image f(M) containing the input copy-paste image M and the output of the generative adversarial network f, the gradient loss is defined as the square root of the L1 norm of the gradient difference between the two:
[0098]
[0099] Among them, G x (M) represents the gradient image of M in the horizontal direction, G y (M) represents the gradient image of M in the vertical direction. In the model, the gradient loss plays a role in limiting the gradient change in the tampered area, ensuring that the inherent semantics of the tampered area remains unchanged. The square root form of the gradient loss focuses on the consistency of the gradient direction, mitigates the penalty for large gradient differences, and reserves a certain amount of space for the prediction of the brightness information of the tampered area in the generation stage;
[0100] (d4) The overall loss of the generator G is obtained by compounding the adversarial loss and the gradient loss:
[0101] L G =λ adv L adv +λ grad L grad
[0102] Among them, λ adv and λ grad It is an important parameter to control the corresponding loss term. Under the above constraints, the generator retains the edge texture information of the tampered area and fuses the tampered area with the background area to generate a high-quality tampered image;
[0103] (d5) As the number of model iterations increases, the generated images will look more and more realistic. To address this feature, we use the generated images at different iterations (Epochs) (e.g. Figure 5a As shown in Figure 2, a multi-level tampered image dataset was constructed.
[0104] It should be noted that, in practical applications, the images synthesized in step 1 can also be in a spliced form, that is, the images in the preliminary tampered image data set are all spliced images, and the remaining steps are consistent with the above embodiment. The multi-level tampered image data set generated by the final generative adversarial network is also in a spliced form (such as Figure 5b shown).
[0105] In summary, the method proposed in the present invention can generate rich, high-quality tampered images based on a multi-scale generative adversarial network, thereby ensuring the reliable training of subsequent convolutional neural networks, thereby improving the detection performance of tampered images.
Claims
1. A method for detecting tampered images based on a multi-scale generative adversarial network, characterized in that: The following steps are involved: S1. Synthesize basic copy-paste images from a standard image dataset and obtain the XML file corresponding to the synthesized image to construct a preliminary tampered image dataset; S2. Build a model based on a multi-scale generative adversarial network, using a preliminary tampered image dataset as the input of the model to set model-related parameters; S3, optimize the generator of the multi-scale generative adversarial network to learn the edge features of the tampered area in the image; S4, alternately train the generator and the discriminator to output tampered images at different scales to construct a multi-level tampered image dataset; S5, training convolutional neural network using multi-level tampered image dataset; S6, inputting the tampered image to be detected into the trained convolutional neural network to obtain the corresponding detection result; The step S1 specifically includes the following steps: S11, selecting a target image and a picture to be processed from a standard image dataset; S12, using the target image as the background, pasting the portion of the tampered area mask in the image to be processed to a random position of the target image, and synthesizing to obtain a basic copy-and-paste image; S13, reading the XML file corresponding to the image to be processed from the standard image data set to read out the bounding box position information corresponding to the tampered area mask, and then modifying the XML file corresponding to the target image, and using the modified XML file as the XML file corresponding to the copied and pasted image; S14, according to the order of images in the standard image dataset, repeat steps S11 to S13 in sequence, synthesize multiple basic copy-and-paste images, and obtain corresponding xml files to construct a preliminary tampered image dataset; The step S12 specifically includes the following steps: S121, reading a mask image corresponding to the image to be processed, and reading out the regional masks of all objects in the mask image; S122, randomly selecting a region mask of a type of object as a tampering region mask; S123, finding an area in the image to be processed that corresponds to the tampered area mask, and intercepting the area from the image to be processed as the tampered source image; S124, pasting the tampered source image to a random position of the target image to synthesize a basic copy-and-paste image; The step S3 specifically optimizes the loss function of the generator, including the following steps: S31. Define reconstruction background loss to speed up training; S32, defining edge loss to constrain the shape of the tampered area; The reconstruction background loss function is specifically: Among them, N bg is the total number of pixels in the background, m i is the i-th pixel in M, where M is the copy-paste image, mask i is the tampered area mask M at position i mask The pixel value in tar i M tar The i-th pixel in M tar is the target image; The marginal loss function is specifically: Among them, N edge is the number of edge pixels, e i is the value of the edge mask at position i, tam i M tam The i-th pixel in M tam To tamper with the source image; The step S4 specifically comprises the following steps: S41. Train the model in sequence from the smallest scale to the largest scale, where the training loss of each generator includes adversarial loss and gradient loss. Each generator has a corresponding Markov discriminator, and the loss of the discriminator only includes adversarial loss. The overall loss of the entire generator network is a compound of adversarial loss and gradient loss. S42, training is performed according to a preset number of iterations, and the generated images at different numbers of iterations are jointly constructed into a multi-level tampered image data set; In step S41, in order to make the tampered area and the background in the image have similar color change trends, a custom gradient loss L is proposed by referring to the idea of gradient field in Poisson fusion. grad , for a fused image f(M) containing the copy-paste input image M and the output of the generative adversarial network f, the gradient loss is defined as the square root of the L1 norm of the gradient difference between the two: Among them, G x (M) is the gradient image of the copied and pasted image M in the horizontal direction, G y (M) is the gradient image of the copied and pasted image M in the vertical direction, G x (f(M)) is the gradient image of the fused image f(M) in the horizontal direction, G y (f(M)) is the gradient image of the fused image f(M) in the vertical direction. In the model, the gradient loss plays a role in limiting the gradient change in the tampered area, ensuring that the inherent semantics of the tampered area remains unchanged. The square root form of the gradient loss focuses on the consistency of the gradient direction; The overall loss of the entire generator network is: L G =λ adv L adv +λ grad L grad Among them, L adv To combat the loss, L grad is the gradient loss, λ adv and λ grad L adv and L grad The generator retains the edge texture information of the tampered area and fuses the tampered area with the background area to generate a high-quality tampered image.
2. According to claim 1, a method for detecting tampered images based on a multi-scale generative adversarial network is characterized in that: The specific process of step S13 is: reading the XML file corresponding to the image to be processed from the annotation folder of the standard image data set, and reading the bounding box position information corresponding to the tampered area mask selected in step S122; According to the read bounding box position information and the position of the pasted area in the copy-paste image, the XML file corresponding to the target image is modified, the detection category is changed to "tamper", the bounding box position information in the target image is changed to the position information of the pasted area in the copy-paste image, and the modified XML file is used as the XML file corresponding to the copy-paste image.
3. The method for detecting tampered images based on a multi-scale generative adversarial network according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21. Construct a multi-scale generative adversarial network model including a discriminator network and a generator network; S22. According to the preliminary tampered image data set, set relevant parameters of the model.
4. The method for detecting tampered images based on a multi-scale generative adversarial network according to claim 3, characterized in that: In step S21, the generator network includes multiple generators corresponding to different scales, and the multiple generators are in a pyramid structure. For the generator with the smallest scale at the top layer, its input only contains spatial Gaussian noise; For the generators of each scale below the top layer, the input contains spatial Gaussian noise and the upsampled image generated by the generator of the previous scale; For the generators of each scale below the top layer, the output results are as follows: the upsampled image generated by the previous scale generator plus the spatial Gaussian noise image is convolved to output a residual image. The residual image is then added with the upsampled image generated by the previous scale generator, which is the output of the generator.
Citation Information
Patent Citations
Dual-channel network-based digital image tampering blind detection method
CN110992238A