Image Tampering Detection Method and System for Generative Model Image Editing
By using a batch image editing process and a method of comparing local frequency domain features, training data is generated and a detection model is trained, which solves the detection problem in generative model image editing and achieves efficient tamper detection of generative editing.
Patent Information
- Application Number
- CN202311576310.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-11-23
AI Technical Summary
Existing image tampering detection methods struggle to effectively detect subtle traces of end-to-end editing when faced with generative model image editing. Furthermore, the training datasets are concentrated on a limited number of generative models, and the editing objectives are mainly focused on deletion, resulting in poor performance of detection methods under unconstrained conditions.
A batch image editing workflow is adopted to generate large-scale training data. Editing intent is generated through zero-shot semantic segmentation and large language model. A detection method combining local frequency domain feature comparison is used to guide the detection model to learn real features using real image prototypes. The detection model is trained using cross-contrast loss function.
It improves the detection performance of generative image editing, effectively identifies tampering caused by generative editing, generates a large amount of high-quality training data, and enhances the generalization ability of the detection model.
Smart Images

Figure CN117690005B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image tampering detection technology, and in particular relates to an image tampering detection method and system for generative model image editing. Background Technology
[0002] Image editing technology offers greater possibilities for creating richer digital images. Editors can take existing images as a base and modify the original content according to their intentions to achieve various purposes and artistic expressions. Traditional image editing methods are mainly based on a series of complex operations, including cropping, moving and pasting, blurring, filtering, etc. These operations require specialized editing software and experienced experts to achieve precise control and the desired results.
[0003] Generative models offer a novel approach to image editing, enabling end-to-end image manipulation without the need for complex additional operations. In their early stages, generative models, particularly generative adversarial networks, had limited representational capabilities. Most relevant image editing work focused on specific domains, such as facial regions, achieving precise attribute editing through specific control mechanisms. However, in unconstrained and general cases, previous generative editing methods faced challenges in achieving desired results, and thus they have not completely replaced editing techniques driven by complex sequences of operations. The emergence of diffusion models has truly changed this landscape, making generative image editing more efficient and creative in general. Editors can achieve precise end-to-end control of image editing using more intuitive and user-friendly guidance, such as natural language prompts or examples of target images. Today, we can say, "Seeing is not believing."
[0004] The widespread adoption of diffusion models and the proliferation of these generative editing techniques mark the beginning of a generative AI revolution. While this development better meets the needs of image editing in daily life, it poses a serious threat to the authenticity and security of image content disseminated in media. Therefore, detecting and locating this type of editing and tampering has become crucial. However, two main bottlenecks need to be addressed: 1) Training data: Most tampering detection datasets are primarily created through manual or automated manipulation sequences. Some datasets consider generative-based tampering, but they typically use a limited range of generative models, with the editing objective mainly focused on deletion. 2) Detection methods: Most existing detection methods focus on altering patterns of defects and noise left by manual manipulation sequences. This approach faces challenges when detecting generative-based tampering because generative editing methods achieve end-to-end editing and produce more realistic effects, leaving less trace during the editing process. Summary of the Invention
[0005] To address the aforementioned problems, this invention proposes an image tampering detection method for generative model image editing, comprising: performing semantic segmentation on a real image to obtain at least one object in the real image; generating an edited image using a generative model based on the editing target of the object; training a detection model using the real image and the edited image as a training set; and detecting image tampering in the target image using the detection model.
[0006] The image tampering detection method of the present invention includes the following steps for training the detection model: dividing the input image x into R image blocks, using Fast Fourier Transform to transform the image blocks from the spatial domain to the frequency domain, and extracting the frequency domain feature sequence of the image blocks; the input image x includes the real image p and the edited image p'; based on the frequency domain feature sequence, obtaining the set S of local feature blocks of the edited region of p'. e And the set of local feature blocks S corresponding to the real region of p a To obtain the true regional category center h,
[0007]
[0008] By pulling h towards the real image prototype p, and S e By pushing h further away, the detection model is guided to learn real image features.
[0009] The image tampering detection method of the present invention uses a corresponding cross-comparison loss function. As the loss function of this detection model,
[0010]
[0011] The image tampering detection method of the present invention includes the following steps for generating the edited image: performing semantic segmentation on the real image p to obtain at least one object; randomly selecting an object; marking the object region of the object as an object mask; marking the rectangular background region surrounding the object as a random mask; generating the edit target based on the original content of the object mask; and performing end-to-end image editing on the object using a generative model based on the edit target to generate the edited image.
[0012] The image tampering detection method described in this invention uses ChatGPT as the generator for generating the edited content.
[0013] The image tampering detection method of the present invention includes a generative model comprising: MAT model, LaMa model, Stable Diffusion model, ControlNet model, or Paint by Example model.
[0014] This invention also proposes an image tampering detection system for generative model image editing, comprising: a data generation module for performing semantic segmentation on a real image to obtain at least one object from the real image; generating an edited image using a generative model based on the editing target of the object; a model training module for training the detection model using the real image and the edited image as training sets; and an image detection module for detecting image tampering in the target image using the detection model.
[0015] The image tampering detection system of the present invention includes a model training module comprising: a feature extraction module, used to segment the input image x into R image blocks, perform a fast Fourier transform on each image block to convert it from the spatial domain to the frequency domain, and extract the frequency domain feature sequence {s} of the image block. i}; The input image x includes the real image p and the edited image p'; The original image reference module is used to detect the edit position of p' through the reference p, obtain the real region category center h, and train the detection model; wherein, based on the frequency domain feature sequence {s i}, including the set of local feature blocks S of the edit region of p'. e And the set of local feature blocks S corresponding to the real region of p a To obtain the true regional category center h,
[0016]
[0017] By pulling h towards the real image prototype p, and S e By pushing h further away, the detection model is guided to learn real image features.
[0018] With the corresponding cross-comparison loss function As the loss function of this detection model,
[0019]
[0020] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that, when the computer-executable instructions are executed, the image tampering detection method for generative model image editing as described above is implemented.
[0021] The present invention also proposes a data processing apparatus, including a computer-readable storage medium as described above, wherein when the processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it realizes image tampering detection for generative model image editing. Attached Figure Description
[0022] Figure 1 This is a flowchart of the image tampering detection method of the present invention.
[0023] Figure 2 This is a schematic diagram of the generative image batch automatic generation method of the present invention.
[0024] Figure 3 This is a schematic diagram of the generative image detection method based on local frequency domain feature comparison of the present invention.
[0025] Figure 4 This is a schematic diagram of the data processing device of the present invention. Detailed Implementation
[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0027] The image tampering detection method for generative models of this invention comprises two core components: 1. First, a batch image editing process implemented using multiple large models enables the generation of large-scale training data. 2. Based on the existing training data, a generative image editing detection method based on local frequency domain feature comparison is trained with a specific design. The purpose of this invention is to propose an automated training data generation method for generative image tampering detection models, while simultaneously improving the model's detection performance for generative image editing.
[0028] Among them, the generative image batch automatic generation method can generate a large amount of high-quality training data for the detection model. The method first randomly selects the object region as the object to be edited based on the zero-shot semantic segmentation model, then uses the large language model ChatGPT to generate the editing intent in text form, and further enriches the expression of the editing intent. Finally, it uses a variety of image editing methods to perform end-to-end image editing on the original image according to the various forms of editing intent.
[0029] A generative image editing detection method based on local frequency domain feature contrast is developed. This method trains the image using batch-generated training data and improves detection performance for generative image editing by focusing on the contrast between local frequency domain features. The method uses a specially designed frequency domain transformation module to convert the image to be detected from the spatial domain to the frequency domain. Then, a real image prototype is introduced, and the inconsistency between the features of the edited region and the prototype guides the model to identify the edit location. Finally, a local region cross-attention mechanism guides the model's attention to the contrast of features at different locations, thus preventing the model from over-adapting to the absolute features of local locations.
[0030] like Figure 1 As shown, the present invention comprises three parts:
[0031] I. Batch Image Editing Methods
[0032] To ensure the logical rationality and semantic diversity of image editing scenarios and content, and to construct a large-scale generative model image editing dataset, this invention employs the following... Figure 2 The image batch editing workflow is shown. In designing this workflow, the following three key issues were primarily addressed:
[0033] (a) How to determine the location and precise area where the editing occurs.
[0034] Since editing can be performed anywhere in an image, this invention employs an advanced zero-shot semantic segmentation model (SAM) to obtain accurate region masks in order to simulate the human process of selecting appropriate objects for editing. By selecting random points, the diversity of semantic categories within the tampered regions of the dataset is ensured, which avoids semantic inconsistencies in the final training data. The randomly selected object region is labeled as the object mask. Furthermore, the rectangular background region surrounding the randomly selected object is labeled as a random mask.
[0035] (ii) How to determine the purpose or intention of editing.
[0036] This invention utilizes ChatGPT as a generator of editing objectives to simulate the process by which humans generate editing goals or intentions. After informing ChatGPT of the original content within the selected object area, it is instructed to provide what it considers a reasonable editing target. Specifically, the instruction input to ChatGPT is: "In a scene, there exists an object, source object: {description}. You wish to edit or replace this object, but still make it consistent with the scene. Please provide a description of the target object: target object: {description}", where the source prompt comes from Blip2's recognition of the original content within the selected object area. With the natural language description of the editing purpose or intention, Stable Diffusion is used to convert these descriptions into image examples.
[0037] (III) How to perform end-to-end image editing by following the instructions generated in the preceding steps.
[0038] In this era of burgeoning generative models, numerous different generative editing methods have emerged, each with its specific focus. This invention can select from a variety of generative editing methods, including but not limited to MAT, LaMa, StableDiffusion, ControlNet, and Paintby Example. For each original image, by collecting the aforementioned types of guidance information, the edited image can be obtained end-to-end without additional intervention or manipulation.
[0039] Ultimately, the aforementioned batch image editing workflow can be used to automatically generate large-scale image editing data while ensuring the diversity of edited areas and content. This large-scale dataset is then used to train subsequent detection models.
[0040] II. Training Method for Generative Editing Image Detection Model Based on Local Frequency Domain Feature Contrast
[0041] Considering that frequency domain features have been widely used for the detection of machine-generated content, this invention designs a generative edited image detection method based on local frequency domain feature comparison, the structural diagram of which is shown below. Figure 3 As shown, it mainly includes three core modules and functions: frequency domain transformation with preserved location information and real image prototype reference. The following sections will introduce these two modules in detail.
[0042] First, there's the frequency domain transformation that preserves location information. The image is segmented into equal-sized image patches, and then the patch counts are merged into the batch dimension. Given an input image x, it is divided into R image patches. These patches undergo Fast Fourier Transform (FFT) to transform from the spatial domain to the frequency domain, and are then processed by an encoder to obtain a patch-level frequency domain feature sequence. Besides explicitly preserving location information and facilitating the acquisition of local frequency domain features, this dimensionality transformation operation avoids the problem of increased receptive field due to increased network depth, which could lead to global information being fused into local features, making them difficult to separate.
[0043] Secondly, a real-world image prototype is used as a reference. Given the widespread and evolving nature of image editing technology, obtaining generalizable features in unknown contexts becomes impractical, especially when facing new editing techniques. However, real-world images possess inherent features that are easier to model. This inherent realism is easier to model. Therefore, a real-world image prototype is introduced to guide the detection model in detecting edit locations through inconsistencies between the features of the edited region and the prototype representing the real region. The input image x is divided into R image blocks, and a Fast Fourier Transform is used to transform each image block from the spatial domain to the frequency domain, extracting the frequency domain feature sequence {s} of the image block. i}; The input image x includes a trainable ground truth image p and an edited image p' corresponding to p; By referencing p, the edit position of p' is detected, the ground truth region category center h is obtained, and the detection model is trained; Wherein, based on the frequency domain feature sequence {s i}, including the set of local feature blocks S of the edit region of p'. e S e This includes the index of the edited patch and the set of local feature blocks S corresponding to the real region of p. a S aThis includes the index of feature blocks from unedited regions. Then, based on the ground truth local feature blocks, the ground truth region class center h can be obtained using the following formula:
[0044]
[0045] By pulling the ground truth class center h toward the ground truth image prototype p, while simultaneously pushing the edited local feature blocks further away from the ground truth region class center h, the model is guided to learn ground truth image features. The corresponding cross-contrast loss function using the prototype p is calculated as follows:
[0046]
[0047] Third, use the trained detection model to detect image tampering in the target image.
[0048] Figure 4 This is a schematic diagram of the data processing apparatus of the present invention. Figure 4 As shown, embodiments of the present invention also provide a computer-readable storage medium and a data processing apparatus. The computer-readable storage medium of the present invention stores computer-executable instructions. When these computer-executable instructions are executed by the processor of the data processing apparatus, the image tampering detection for generative model image editing described above is implemented. Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware (e.g., processor, FPGA, ASIC, etc.), and the program can be stored in a readable storage medium, such as a read-only memory, a disk, or an optical disk. All or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiments can be implemented in hardware, for example, by implementing its corresponding function through an integrated circuit, or it can be implemented in the form of a software functional module, for example, by a processor executing a program / instruction stored in memory to implement its corresponding function. Embodiments of the present invention are not limited to any particular combination of hardware and software.
[0049] This invention first generates a large amount of high-quality training data for the detection model through a batch automatic generation method of generative edited images; then, it trains the generative edited image detection method based on local frequency domain feature comparison using the batch-generated training data, and effectively improves the detection performance of the detection model for generative image editing by focusing on the comparison between local frequency domain features.
[0050] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.
Claims
1. A method for detecting image tampering in generative model image editing, characterized in that, include: Perform semantic segmentation on a real image p to obtain at least one object from the real image; Based on the editing goals for the object, an edited image p' is generated using a generative model; The detection model is trained using the real image p and the edited image p' as the training set; this includes: segmenting the input image x into R image blocks, performing a Fast Fourier Transform (FFT) to transform each image block from the spatial domain to the frequency domain, and extracting the frequency domain feature sequence of each image block; the input image x includes the real image p and the edited image p'; based on the frequency domain feature sequence, obtaining the set S of local feature blocks of the edited region of the edited image p'. e And the set of local feature blocks S corresponding to the real region of the real image p. a To obtain the true region category center h, the true region category center h is pulled towards the true image. And the set of local feature blocks S e The detection model is guided to learn real image features by moving away from the true region category center h; and the corresponding cross-contrast loss function is used. This serves as the loss function for the detection model; where... , ; This detection model is used to detect image tampering in target images.
2. The image tampering detection method as described in claim 1, characterized in that, The steps to generate this edited image include: Perform semantic segmentation on the real image p to obtain at least one object. Randomly select an object, mark the object region of the object as an object mask, and mark the rectangular background region around the object as a random mask. The editing target is generated based on the original content of the object mask; Based on the editing target, a generative model is used to perform end-to-end image editing on the object, generating the edited image.
3. The image tampering detection method as described in claim 2, characterized in that, ChatGPT was used as the generator for this editing purpose.
4. The image tampering detection method as described in claim 2, characterized in that, The generative models include: MAT model, LaMa model, Stable Diffusion model, ControlNet model, or Paint by Example model.
5. An image tampering detection system for generative model image editing, characterized in that, include: The data generation module is used to perform semantic segmentation on real images and obtain at least one object from the real image. Based on the editing goals for the object, a generative model is used to generate the edited image; The model training module is used to train the detection model using the real image and the edited image as training sets; The model training module includes: The feature extraction module is used to segment the input image x into R image blocks, use Fast Fourier Transform to transform the image blocks from the spatial domain to the frequency domain, and extract the frequency domain feature sequence of the image blocks; the input image x includes the real image p and the edited image p'; The original image reference module is used to detect the edit position of p' by referencing p, obtain the true region category center h, and train the detection model; wherein, based on the frequency domain feature sequence, the set of local feature blocks S of the edit region of p' is obtained. e And the set of local feature blocks S corresponding to the real region of p a To obtain the true region category center h, by pulling h towards the true image prototype. and S e By pushing h further away, the detection model is guided to learn real image features, using the corresponding cross-contrast loss function. As the loss function of the detection model; where , ; The image detection module is used to detect image tampering in the target image using the detection model.
6. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, the image tampering detection method for generative model image editing as described in any one of claims 1 to 4 is implemented.
7. A data processing apparatus comprising a computer-readable storage medium as claimed in claim 6, wherein when a processor of the data processing apparatus retrieves and executes computer-executable instructions in the computer-readable storage medium, it implements image tampering detection for generative model image editing.
Citation Information
Patent Citations
Active defense method for deep face tampering
CN112883874A
Deep fake face data identification method
CN113536990A
