Layered fine-grained image forgery detection method and system

Abstract

The invention discloses a layered fine-grained image forgery detection method and a layered fine-grained image forgery detection system. And the authenticity of the input image is preliminarily judged through the authenticity judgment module. Constructing a multi-branch pixel-level counterfeit positioning network, and obtaining multi-scale feature representation from low resolution to high resolution through a feature extraction backbone network; a high-resolution forged mask is obtained through pixel-level positioning network prediction; and the decision network constructs a coarse-to-fine reasoning path according to the multi-level label system, sequentially completes forgery property judgment, forgery type judgment, method-level traceability and platform-level traceability, and outputs a conditional probability of a corresponding-level label as a traceability result. The method has a unified multi-level label system, the systematicness and the accuracy of detection are remarkably improved, the source and the position of counterfeiting are visually presented through mask positioning and hierarchical classification, and the interpretability is high.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to an image forgery detection and localization method and system based on hierarchical fine-grained feature representation, which can be applied to scenarios such as digital media forensics, deepfake identification and generative model tracing. Background Technology

[0002] In recent years, the rapid development of generative artificial intelligence has enabled generative adversarial networks, diffusion models, convolutional neural network image editing tools, and other technologies to generate or tamper with image content with high fidelity. These forged images are highly realistic in terms of realism, semantic consistency, and cross-modal matching, posing a significant challenge to image forensics and forgery detection.

[0003] Research on image forgery detection mainly focuses on two directions. The first is global detection methods, which detect forgeries through full-image statistical features, frequency domain inconsistencies, or model residuals, but struggle to handle minor local tampering scenarios. The second is local localization methods, which rely on masks or pixel-level contrastive learning to segment forged regions, but lack hierarchical representation of forgery attributes. Furthermore, some studies have shown that different generative models contain unique source features in their frequency domain distribution, gradient response, and noise residuals, enabling forgery tracing; however, existing solutions often handle detection and attribution tasks independently, lacking a unified hierarchical structure. Therefore, a novel detection framework combining hierarchical label semantics, fine-grained forgery attribute classification, and pixel-level localization is urgently needed, capable of both forgery detection and attribution, while also ensuring interpretability and robustness across different forgery sources. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a hierarchical fine-grained image forgery detection method and system. By constructing a multi-level forgery label system, a multi-branch pixel-level forgery localization network, and a forgery mask, it jointly achieves forgery type identification, source model attribution, and local forgery region localization, thereby significantly improving the accuracy, generalization, and interpretability of forgery detection.

[0005] In a first aspect, embodiments of this application provide a layered, fine-grained image forgery detection method, comprising the following steps:

[0006] Step 1: Construct a multi-level labeling system and obtain the training dataset.

[0007] Step 2: Use the authenticity detection module to make a preliminary judgment on the authenticity of the input image.

[0008] Step 3: Construct a multi-branch pixel-level spoofing localization network, including a feature extraction backbone network, a pixel-level localization network, and a decision network.

[0009] Step 4: Obtain multi-scale feature representations from low resolution to high resolution through the multi-resolution feature extraction mechanism of the feature extraction backbone network.

[0010] Step 5: Based on the obtained multi-scale features, a high-resolution forgery mask is predicted through a pixel-level localization network, which is used to perform pixel-level localization of the forgery area.

[0011] Step 6: The decision network constructs a coarse-to-fine reasoning path based on a multi-level labeling system. Using multi-scale feature representations, it sequentially completes forgery determination, forgery type determination, method-level tracing, and platform-level tracing, outputting the conditional probabilities of the corresponding level labels as the tracing results. Each level of tracing result after forgery determination uses the previous level's tracing result as a conditional constraint, thus forming a hierarchical, constrained, step-by-step reasoning process.

[0012] Step 7: Train a multi-branch pixel-level fake localization network based on the acquired training dataset.

[0013] In one possible implementation, step 1 is specifically performed as follows:

[0014] Step 1.1: Multi-level fake labeling system. The top level first performs preliminary labeling of the authenticity of the training samples, which is used to divide the training samples into real images and images containing fake content.

[0015] Step 1.2: In the second-level label, after completing the authenticity labeling, for the forged sample, construct the forgery nature label according to the forgery nature, and divide the forgery nature into two major categories: tampering forgery and generation forgery.

[0016] Tampering-type forgery is based on real images and is achieved by editing, enhancing, repairing, or replacing local areas; generative forgery, on the other hand, relies entirely on the model to generate new images.

[0017] Step 1.3: In the third-level label, after clarifying the nature of forgery, the forgery samples are further subdivided along the technical characteristics of the forgery method to construct forgery type labels.

[0018] For tampering-type forgeries, they are further divided into two types: image enhancement and image editing. For generation-type forgeries, they are divided into GAN-based generation samples, diffusion model-based generation samples, Transformer-based generation samples, and hybrid generation samples that integrate multiple generation mechanisms, based on the technical route of their underlying generation framework.

[0019] Step 1.4: After completing the forgery type labeling, further construct method-level labels for the forged samples to identify their corresponding specific generation or tampering methods.

[0020] Step 1.5: Based on the method-level tags, construct the finest-grained platform-level tags to achieve the final source attribution labeling for the forged content.

[0021] By employing the above steps, training samples are assigned labels for real and fake, fake type, method-level, and platform-level, respectively, forming a training dataset containing multi-level fake semantic information.

[0022] In one possible implementation, the authenticity determination module includes:

[0023] 1) Multi-domain cue extraction unit: Normalizes and scales the input image to obtain the color domain baseline features as color domain cues. A multi-scale Laplacian operator is used to enhance the characterization of edge texture breaks and frequency domain anomalies in the input image at different scales, obtaining multi-scale high-frequency response feature maps as frequency domain cues. SRM (Spatial Rich Model) residual filtering is performed on the input image to obtain a noise residual map as noise residual domain cues.

[0024] 2) Authenticity screening unit: Construct multi-channel input features of color domain cues, frequency domain cues and noise residual domain cues into the same spatial scale, and input them into a lightweight feature extraction network for feature learning, and output authenticity confidence.

[0025] In one possible implementation, the feature extraction backbone network includes four resolution feature branches, each acquiring four types of resolution feature maps. When acquiring resolution feature maps, lower-resolution feature maps are obtained by downsampling the previous-level resolution feature map. Specifically, during resolution feature branch processing, the resolution feature map output from the previous-level resolution feature branch is used as input, and a convolution with a stride of 2 is used as a downsampling operator to reduce the spatial size of the feature map to half of its original size, thereby obtaining a lower-resolution feature map.

[0026] In one possible implementation, the pixel-level localization network is set on the highest resolution feature branch of the multi-branch pixel-level spoofing localization network. By performing convolutional mapping and spatial modeling on the highest resolution feature map, a spoofing probability map aligned with the input image space is generated. The spoofing probability map is further thresholded to obtain a high-resolution binary spoofing mask M.

[0027] In one possible implementation, the pixel-level localization network operates as follows:

[0028] The highest resolution feature map is input into the pixel-level localization network, and spatial alignment is achieved through convolutional mapping, outputting a pixel-level embedded feature map aligned with the input space.

[0029] Subsequently, the true feature center vector is obtained by averaging the pixel-level embedded feature map with the pixel embeddings of all real samples. And calculate the position of each pixel in the pixel-level embedded feature map. Embedded vector With the center vector distance :

[0030]

[0031] By setting the distance scale parameter R, Calculate the normalized distance between the embedding vector of each pixel and the ground truth feature center vector:

[0032]

[0033] Distance map consisting of distance values ​​from all pixel locations After mapping using the sigmoid function, a pixel-level forgery probability map is generated. :

[0034]

[0035] in, It is a pixel-level forgery probability map At pixel position The probability value at that location. For the sigmoid function, The scaling factor. This is the distance threshold parameter. Thresholding the probability map 𝑃 yields a binary forgery mask 𝑀, used to indicate the pixel-level location of the forgery region:

[0036]

[0037] in Represents pixels To forge the region, It is a probability threshold parameter used to control the strictness of the judgment of forged areas.

[0038] In one possible implementation, the decision network includes a cross-scale fusion module, a global convergence module, a hierarchical classification output module, and a gating constraint module.

[0039] The cross-scale fusion module consists of three cross-scale fusion units, each positioned between features of adjacent resolution scales. These units perform spatial size alignment and channel dimension alignment on adjacent scale features and obtain fused features through element-wise addition. The three cross-scale fusion units sequentially output type-level fusion features, method-level fusion features, and platform-level fusion features.

[0040] The global aggregation module is used to perform global average pooling on the fused features to obtain the global representation vector of the corresponding level.

[0041] The hierarchical classification output module includes a property-level classification output head, a type-level classification output head, a method-level classification output head, and a platform-level classification output head. Each classification output head adopts a fully connected mapping structure to map the global representation vector of the corresponding level to the label space of that level and output the classification score.

[0042] The gating constraint module constructs a gating vector based on the pre-defined label hierarchy mapping relationship of the multi-level label system, thereby obtaining the conditional probability vector of the corresponding level. Specifically, after obtaining the conditional probability vector of the property-level label, the type-level sourcing, method-level sourcing, and platform-level sourcing all construct gating vectors based on the conditional probability vector output from the previous level and the corresponding label hierarchy mapping relationship. These gating vectors then perform zero-weight suppression and normalization on the classification score of the current level's classification output head, thereby outputting the conditional probability vector of the current level's label.

[0043] Secondly, embodiments of this application provide a layered fine-grained image forgery detection system, comprising the following modules:

[0044] Dataset building module: Constructs a multi-level labeling system and obtains the training dataset.

[0045] Authenticity / fakeness detection module: performs a preliminary determination of the authenticity of the input image.

[0046] Feature extraction module: It obtains multi-scale feature representations from low resolution to high resolution through the multi-resolution feature extraction mechanism of the feature extraction backbone network.

[0047] Pixel-level localization module: Based on the obtained multi-scale features, a high-resolution forgery mask is predicted through a pixel-level localization network, which is used to perform pixel-level localization of the forgery area.

[0048] Decision Module: This module constructs a coarse-to-fine reasoning path using a multi-level labeling system through a decision network. Based on multi-scale feature representation, it sequentially completes forgery determination, forgery type determination, method-level tracing, and platform-level tracing, outputting the conditional probabilities of the corresponding level labels as the tracing results. Each level of tracing result after forgery determination uses the previous level's tracing result as a conditional constraint, thus forming a consistent, path-constrained, hierarchical reasoning process.

[0049] Training module: Trains a multi-branch pixel-level spoofing localization network based on the training dataset obtained from the dataset construction module.

[0050] Advantages and positive effects of the present invention:

[0051] (1) This invention has a unified multi-level label system. For the first time, it integrates image forgery detection, local tampering location and generative model tracing within the same system. It uses hierarchical labels to achieve step-by-step identification from "authenticity judgment" to "forgery type" and then to "forgery source", which significantly improves the systematicness and accuracy of detection. It intuitively presents the source and location of forgery through mask positioning and hierarchical classification, which is highly interpretable.

[0052] (2) This invention constructs a multi-branch pixel-level forgery localization network in the color domain and frequency domain through a multi-branch fine-grained learning mechanism. It can simultaneously capture subtle texture anomalies and high-level generation mode differences in forged images, and has strong cross-model and cross-dataset generalization capabilities. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the layered label construction of the present invention.

[0054] Figure 2 A schematic diagram of the multi-branch feature extraction network constructed in this invention.

[0055] Figure 3 A schematic diagram of the hierarchical fine-grained image forgery detection system constructed according to the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the specific embodiments of this invention will be further described in detail below with reference to the accompanying drawings. Those skilled in the art should understand that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0057] Please see Figure 1 This invention addresses the problems of diverse forgery forms, large cross-modal differences, and weak generalization ability in current image forgery detection tasks by proposing a multi-level forgery labeling system. This system progressively refines the labeling of forgery samples from five dimensions: authenticity detection, forgery nature, forgery type, method-level tracing, and platform-level tracing. This enables the detection model to learn hierarchical forgery semantic features during the training phase, thereby improving its generalization ability across methods, platforms, and datasets. The multi-level labeling system is mainly used for the construction and labeling of training data, and its construction process can be divided into five steps:

[0058] Step 1.1: In the multi-level forgery labeling system of the present invention, the top level first performs preliminary labeling on the authenticity of the training samples, which is used to divide the training samples into real images and images containing forged content, thereby providing a basic entry point for the subsequent construction of labels for forgery nature, forgery type and source level.

[0059] Based on the authentic annotation information of the training samples, real or fake labels are assigned to the training samples, which serve as the top-level labels in a multi-level labeling system. The training samples are derived from manual annotation, verified labels from public datasets, and content generated through a trusted generation process.

[0060] Step 1.2: In the second-level label, after completing the authenticity labeling, for the forged sample, construct the forgery nature label according to the forgery nature, and divide the forgery nature into two major categories: tampering forgery and generation forgery.

[0061] Tampering-based forgeries are based on real images and are achieved by editing, enhancing, repairing, or replacing local regions, such as common object removal, facial editing, and style transfer. Generative forgeries, on the other hand, rely entirely on models to generate new images, including images generated by GANs, diffusion models, and Transformer models.

[0062] Step 1.3: In the third-level label, after clarifying the nature of forgery, the forgery samples are further subdivided along the technical characteristics of the forgery method to construct forgery type labels.

[0063] For tampering-type forgeries, they are further divided into two types: image enhancement and image editing. Image enhancement includes processing methods that affect the overall visual style, such as color adjustment and style transfer. Image editing includes operations such as replacing, splicing, or repairing local areas. For generation-type forgeries, they are divided into GAN-based generation samples, diffusion model-based generation samples, Transformer-based generation samples, and hybrid generation samples that integrate multiple generation mechanisms, based on the technical route of their underlying generation framework.

[0064] Step 1.4: After completing the forgery type labeling, further construct method-level labels for the forged samples to identify their corresponding specific generation or tampering methods.

[0065] For image enhancement samples, method-level labels include color adjustment and style transfer; for image editing samples, method-level labels include image duplication and local repair. For GAN-based generated samples, method-level labels are categorized according to classic GAN model families, including but not limited to styleGAN, DCGAN, and cycleGAN; for diffusion model-based generated samples, method-level labels are categorized according to different model systems, including but not limited to Sora, Playground, Imagen, Midjourney, Gen-2, Labs, Tongyi Wanxiang, and Mita; for Transformer-based generated samples, method-level labels are categorized according to mainstream transformer generation architectures, including but not limited to MUSE and MaskGIT; for hybrid generated samples based on multi-model fusion, method-level labels are categorized according to the mainstream "transformer + diffusion model" hybrid generation architecture, including but not limited to DALL·E3, Firefly2, Wenxin Yige, and Keling.

[0066] Step 1.5: Based on the method-level tags, construct the finest-grained platform-level tags to achieve the final source attribution labeling for the forged content.

[0067] The fifth-level tags provide the finest-grained forgery type labels, used to map method-level tags to their actual deployment platforms to pinpoint the source of the forgery. Platform-level tags map method-level tags to their actual deployment platforms, including but not limited to OpenAI, Sora, Google, Runway, Pika, Adobe, Alibaba, Kuaishou, Baidu, and Mita Technology. Additionally, to address situations where the forgery method of a detected sample does not have a corresponding platform tag, an unknown platform tag is provided.

[0068] Steps 1.1-1.5 above construct a multi-level forgery label system from genuine / fake detection to final source tracing. This system provides a structured, scalable, and well-interpretable label benchmark for model training, laying the foundation for subsequent forgery detection, localization, and source tracing algorithms. This multi-level label system is not an independent manual classification rule, but rather corresponds one-to-one with the multi-branch pixel-level forgery localization network in step 2. Each level of label is used to supervise the learning objectives of different resolution flow branches and constrain the decision path during the inference stage, guiding the model to complete forgery detection, localization, and source tracing.

[0069] Through steps 1.1-1.5 above, the training samples are assigned real and fake labels, fake type labels, method-level labels and platform-level labels, respectively, forming a training dataset containing multi-level fake semantic information, providing a unified and structured training foundation for subsequent fake detection, pixel-level localization and source tracing tasks.

[0070] Step 2: Input the image to be detected into the authenticity discrimination module. This module performs a preliminary authenticity determination on the input image, analyzing its authenticity based on multi-domain discrimination clues in the color domain, frequency domain, and noise residual domain of the input RGB format image. This provides input enhancement and spatial cues for the subsequent multi-branch pixel-level forgery localization network.

[0071] The authenticity detection module includes:

[0072] 1) Multi-domain cue extraction unit:

[0073] Color gamut cues: The input image is normalized and scaled to obtain the color gamut baseline features as color gamut cues, which are used to preserve the image structure, texture and color distribution information.

[0074] Frequency domain cues: The multi-scale Laplacian operator is used to enhance the characterization of edge texture breaks and frequency domain anomalies in the input image at different scales, and the resulting multi-scale high-frequency response feature maps are used as frequency domain cues.

[0075] Noise residual domain cue: Perform SRM (Spatial Rich Model) residual filtering on the input image to obtain a noise residual map as a noise residual domain cue. This is used to enhance the separability between hidden tampering traces and the distribution differences of generated noise, and to provide residual signal basis for authenticity judgment and subsequent source tracing.

[0076] 2) Authenticity screening unit: Construct multi-channel input features of the same spatial scale by combining color domain cues, frequency domain cues and noise residual domain cues, and input them into a lightweight feature extraction network for feature learning.

[0077] Specifically, color domain cues, frequency domain cues, and noise residual domain cues are size-aligned and then concatenated along the channel dimension to obtain multi-channel input features at the same spatial scale. These multi-channel input features are then fed into a lightweight feature extraction network for feature extraction. After global pooling and fully connected layers, the features are passed through a binary classifier constructed using the sigmoid function to output a true / false confidence score. This score is used to determine whether the input image contains forged content and to perform early filtering. This initial screening unit emphasizes high recall to ensure that potentially forged images proceed to subsequent forgery nature and source analysis stages. When the true / false confidence score is higher than a set threshold, the true judgment result is directly output and subsequent calculations are terminated. When the true / false confidence score is lower than the threshold, the original image to be detected is input into a multi-branch pixel-level forgery localization network.

[0078] Step 3: Construct a multi-branch pixel-level fake location network.

[0079] The multi-branch pixel-level forgery localization network includes a feature extraction backbone network, a pixel-level localization network, and a decision network. It is used to perform coarse-to-fine spatial localization of forgery regions in the input image, and on this basis, jointly complete hierarchical source tracing reasoning of forgery nature, type, method, and platform.

[0080] Step 4: Input the image to be detected into the feature extraction backbone network to extract multi-resolution features.

[0081] The feature extraction backbone network comprises four resolution feature branches, each acquiring four types of resolution feature maps. When acquiring resolution feature maps, lower-resolution feature maps are obtained by downsampling the previous-level resolution feature map. Specifically, during resolution feature branch processing, the resolution feature map output from the previous-level resolution feature branch is used as input, and a convolution with a stride of 2 is used as a downsampling operator to reduce the spatial size of the feature map to half its original size, thereby obtaining a lower-resolution feature map. The resolution feature maps obtained from the four resolution feature branches are named and distinguished as follows:

[0082] L1: Lowest resolution feature map, used to characterize forgery patterns with global semantics and cross-regional consistency, suitable for coarse-grained discrimination of forgery properties.

[0083] L2: Medium to low resolution feature maps, used to capture regional consistency and global texture statistics under a larger receptive field.

[0084] L3: Medium-to-high resolution feature map, used to fuse local structural and mesoscale semantic information.

[0085] L4: Highest resolution feature map, used to obtain fine-grained spatial information.

[0086] Step 5: Based on the obtained multi-scale features, a high-resolution forgery mask is predicted through a pixel-level localization network, which is used to perform pixel-level localization of the forgery area.

[0087] The pixel-level localization network is set on the highest resolution feature branch of the multi-branch pixel-level spoofing localization network. By performing convolutional mapping and spatial modeling on the highest resolution feature map, a spoofing probability map aligned with the input image space is generated. The spoofing probability map is further thresholded to obtain a high-resolution binary spoofing mask M.

[0088] Specifically, the highest resolution feature map is input into the pixel-level localization network, and through convolutional mapping and spatial alignment, the output pixel-level embedding feature map is aligned with the input space. ,in For the target mask resolution, For the embedded dimension.

[0089] Subsequently, the true feature center vector is obtained by averaging the pixel-level embedded feature map with the pixel embeddings of all real samples. And calculate the position of each pixel in the pixel-level embedded feature map. Embedded vector With the center vector distance :

[0090]

[0091] By setting the distance scale parameter R, Calculate the normalized distance between the embedding vector of each pixel and the ground truth feature center vector:

[0092]

[0093] Distance map consisting of distance values ​​from all pixel locations After mapping using the sigmoid function, a pixel-level forgery probability map is generated. :

[0094]

[0095] in, It is a pixel-level forgery probability map At pixel position The probability value at that location. For the sigmoid function, The scaling factor. This is the distance threshold parameter. Thresholding the probability map 𝑃 yields a binary forgery mask 𝑀, used to indicate the pixel-level location of the forgery region:

[0096]

[0097] in Represents pixels To forge the region, This is a probability threshold parameter used to control the strictness of the determination of forged regions. The probability threshold parameter is set to a fixed value, with a default value of 0.5. In subsequent inference processes, the forgery mask M is used to spatially constrain the feature responses in the multi-branch pixel-level forgery localization network, so that when the network performs forgery nature determination, type identification, method-level tracing, and platform-level tracing, it prioritizes focusing on the regions identified as forgeries, thereby improving the accuracy and consistency of fine-grained tracing inference.

[0098] Step 6: The decision network constructs a coarse-to-fine reasoning path based on the multi-level label system. Based on multi-scale feature representation, it sequentially completes property-level tracing, type-level tracing, method-level tracing, and platform-level tracing, and outputs the conditional probability of the corresponding level label as the tracing result.

[0099] The decision network includes a cross-scale fusion module, a global convergence module, a hierarchical classification output module, and a gating constraint module.

[0100] The cross-scale fusion module consists of three cross-scale fusion units, each positioned between features of adjacent resolution scales. These units perform spatial size alignment and channel dimension alignment on adjacent scale features and obtain fused features through element-wise addition. The three cross-scale fusion units sequentially output type-level fusion features, method-level fusion features, and platform-level fusion features.

[0101] The global aggregation module is used to perform global average pooling on the fused features to obtain the global representation vector of the corresponding level.

[0102] The hierarchical classification output module includes a property-level classification output head, a type-level classification output head, a method-level classification output head, and a platform-level classification output head. Each classification output head adopts a fully connected mapping structure to map the global representation vector of the corresponding level to the label space of that level and output the classification score.

[0103] The gating constraint module constructs gating vectors based on the pre-defined label hierarchy mapping relationships of the multi-level label system, thereby obtaining the conditional probability vectors of the corresponding levels. Specifically, after obtaining the conditional probability vectors of the property-level labels, the type-level tracing, method-level tracing, and platform-level tracing all construct gating vectors based on the conditional probability vectors output from the previous level and the corresponding label hierarchy mapping relationships. These gating vectors are then used to perform zero-weight suppression and normalization on the classification scores of the current level's classification output head, thereby outputting the conditional probability vectors of the current level's labels, achieving hierarchical consistency and path-constrained step-by-step reasoning.

[0104] The decision network adopts a path reasoning structure from coarse-grained to fine-grained, consistent with the multi-level labeling system.

[0105] Property-level source tracing (corresponding to second-level labeling): The decision network takes the lowest-resolution feature map L1 as input and obtains a property-level global representation vector through a cross-scale fusion module and a global convergence module. This global representation vector is mapped by a fully connected layer of the property-level classification output head to output classification scores for tamper-type forgery and generation-type forgery, and then normalized to obtain a probability vector for the forgery property. .

[0106]

[0107] in This indicates the category of forgery. This probability vector serves as the root node probability and is used as a conditional constraint for subsequent hierarchical reasoning.

[0108] Type-level tracing (corresponding to third-level tags):

[0109] First, the L1 features are upsampled and aligned to the spatial size of the low-to-medium resolution feature map L2 through a cross-scale fusion unit, and channel alignment is completed through 1×1 convolution. Then, the L1 features are added element-wise to obtain the type-level fusion feature F2, which is then global average pooled by the global aggregation module to obtain the type-level global representation vector.

[0110] The type-level classification output head is fully connected to the type-level global representation vector to output the classification score of each candidate forgery type.

[0111] Subsequently, based on the hierarchical mapping relationship between "forgery property - forgery type" in the multi-level labeling system, the gating constraint module constructs a type-level gating vector based on the property-level output probability vector P(N). It then performs zero-weight suppression on type classification scores that do not conform to the current forgery property and normalizes the retained branches, thereby obtaining the type-level conditional probability vector. .

[0112] in Indicates the type of forgery category

[0113] when At that time, the type-level classification head outputs the conditional probabilities of image enhancement and image editing.

[0114] when At that time, the type-level classification head outputs the conditional probabilities of generation based on GAN, generation based on diffusion model, generation based on Transformer, and hybrid generation.

[0115] Method-level source tracing (corresponding to the fourth-level tag):

[0116] First, the L2 features are upsampled and aligned to the spatial size of the medium-to-high resolution feature map L3 through a cross-scale fusion unit, and channel alignment is completed through 1×1 convolution; then, the L3 features are added element-wise to obtain the method-level fusion feature F3.

[0117] The global pooling module performs global average pooling on the fused feature F3, and under the constraint of the fake mask M, it only performs feature pooling on the pixel features predicted as fake regions to obtain the method-level global representation vector.

[0118] The method-level classification output header outputs the classification score of each candidate forgery method based on this global representation vector.

[0119] The gating constraint module is based on the type-level conditional probability vector. Construct a method-level gating vector based on the predefined "forgery type - forgery method" hierarchy. Apply zero-weight suppression to the classification scores of methods inconsistent with the current forgery type to obtain a method-level conditional probability vector. ,in Indicates the category of forgery method.

[0120] Platform-level traceability (corresponding to the fifth-level tag):

[0121] First, the L3 features are upsampled and aligned to the spatial size of the highest resolution feature map L4 through a cross-scale fusion unit, and channel alignment is completed through 1×1 convolution; then, the fused features F4 are obtained by element-wise addition with the L4 features.

[0122] The global pooling module performs global average pooling on the fused feature F4, and under the constraint of the fake mask M, it only performs feature pooling on the pixel features predicted as fake regions to obtain the platform-level global representation vector.

[0123] The platform-level classification output head performs a fully connected mapping on the global representation vector to output the classification score for each candidate platform.

[0124] The gating constraint module is based on the method-level conditional probability vector. The system also establishes a hierarchical mapping between "forgery methods" and "platforms," ​​applies zero-weight suppression to platform classification scores that do not conform to the current forgery method, and normalizes the remaining branches to obtain the platform-level conditional probability vector. ,in Indicates the platform category.

[0125] When the forgery method category of the sample to be detected does not have a corresponding platform label set in the preset "forgery method - platform" hierarchical mapping relationship, the gating constraint module sets the platform-level gating vector to activate only the "unknown platform" label, and performs zero-weight suppression and normalization on the classification scores of the other platform labels, thereby outputting the platform-level conditional probability vector. At this point, the platform-level tracing result outputs "unknown platform," and the method-level tracing result serves as the final basis for determining the source.

[0126] In the above tracing process, the decision network outputs conditional probability vectors at the property level, type level, method level, and platform level, respectively.

[0127] The property-level output shows the probability of tampering / generation, while the type-level, method-level, and platform-level outputs more granular conditional probabilities under the constraints of the previous-level branch. As shown in the following formula, the confidence level of the final tracing result... It is obtained by combining the conditional probabilities of each level along its path through a product, thus achieving coarse-to-fine reasoning with consistent levels, and using the highest confidence level as the final classification result.

[0128]

[0129] The final output of this invention includes:

[0130] (1) A pixel-level forgery mask M is used to indicate the spatial location of the forgery area in the input image;

[0131] (2) The result of the forgery determination and its probability P(N);

[0132] (3) The tracing results of the forgery type, method and platform and their corresponding conditional probabilities P(T | N), P(Mtd | T, N), P(Plt | Mtd, T, N).

[0133] (4) Confidence of the final source tracing results .

[0134] Ultimately, the confidence level of a certain platform The confidence level is obtained by multiplying the conditional probabilities of each level along the inference path, and the highest confidence level is used as the final classification result.

[0135] Step 7: Train a multi-branch pixel-level fake localization network based on the acquired training dataset.

[0136] During the training phase, the multi-branch pixel-level spoofing localization network described in step 3 is trained end-to-end using the acquired training dataset, wherein:

[0137] (1) The pixel-level localization network learns to predict fake masks under the supervision of real and fake samples;

[0138] (2) Under the supervision of the nature, type, method and platform label of the forgery, the decision network optimizes the classification loss of the corresponding level respectively;

[0139] (3) By using the gating constraint mechanism, the classification loss at each level is jointly optimized on a consistent reasoning path.

[0140] Through the above training method, the network can simultaneously possess pixel-level forgery location capability and forgery source tracing capability.

[0141] During training, training images are input into the network. The backbone network extracts multi-resolution features, the pixel-level localization network predicts the pixel-level forgery probability map on the highest resolution branch and further obtains a binary forgery mask, and the decision network outputs hierarchical prediction results based on multi-scale features, sequentially including forgery nature, forgery type, method level, and platform level. To achieve coarse-to-fine hierarchical consistent inference, a conditional constraint mechanism is adopted for the output of each level during the training phase: the prediction probability of the previous level forms a gating vector, which weights and masks the output of the next level classification head, so that the next level only learns and updates on the branch consistent with the prediction of the previous level, thereby avoiding cross-level contradictory outputs.

[0142] The loss function consists of pixel-level localization loss and hierarchical source tracing loss. The pixel-level localization loss is used to constrain the consistency between the pixel-level forgery probability map and its supervision signal. The hierarchical source tracing loss is used to constrain the consistency between the classification outputs at the property level, type level, method level and platform level and the corresponding labels.

[0143] To improve training stability and generalization ability, data augmentation strategies are used during the training phase to process the input image by random scaling, cropping, flipping, color perturbation, and compression perturbation. Thresholding parameters can be set as learnable parameters or adaptively determined through the validation set, thereby enabling pixel-level localization and source inference to be optimized together during training.

[0144] This application also provides a layered fine-grained image forgery detection system, including the following modules:

[0145] Dataset building module: Constructs a multi-level labeling system and obtains the training dataset.

[0146] Authenticity / fakeness detection module: performs a preliminary determination of the authenticity of the input image.

[0147] Feature extraction module: It obtains multi-scale feature representations from low resolution to high resolution through the multi-resolution feature extraction mechanism of the feature extraction backbone network.

[0148] Pixel-level localization module: Based on the obtained multi-scale features, a high-resolution forgery mask is predicted through a pixel-level localization network, which is used to perform pixel-level localization of the forgery area.

[0149] Decision Module: This module constructs a coarse-to-fine reasoning path using a multi-level labeling system through a decision network. Based on multi-scale feature representation, it sequentially completes forgery determination, forgery type determination, method-level tracing, and platform-level tracing, outputting the conditional probabilities of the corresponding level labels as the tracing results. Each level of tracing result after forgery determination uses the previous level's tracing result as a conditional constraint, thus forming a consistent, path-constrained, hierarchical reasoning process.

[0150] Training module: Trains a multi-branch pixel-level spoofing localization network based on the training dataset obtained from the dataset construction module.

[0151] In one possible implementation, the dataset construction module operates as follows:

[0152] Step 1.1: Multi-level fake labeling system. The top level first performs preliminary labeling of the authenticity of the training samples, which is used to divide the training samples into real images and images containing fake content.

[0153] Step 1.2: In the second-level label, after completing the authenticity labeling, for the forged sample, construct the forgery nature label according to the forgery nature, and divide the forgery nature into two major categories: tampering forgery and generation forgery.

[0154] Tampering-type forgery is based on real images and is achieved by editing, enhancing, repairing, or replacing local areas; generative forgery, on the other hand, relies entirely on the model to generate new images.

[0155] Step 1.3: In the third-level label, after clarifying the nature of forgery, the forgery samples are further subdivided along the technical characteristics of the forgery method to construct forgery type labels.

[0156] For tampering-type forgeries, they are further divided into two types: image enhancement and image editing. For generation-type forgeries, they are divided into GAN-based generation samples, diffusion model-based generation samples, Transformer-based generation samples, and hybrid generation samples that integrate multiple generation mechanisms, based on the technical route of their underlying generation framework.

[0157] Step 1.4: After completing the forgery type labeling, further construct method-level labels for the forged samples to identify their corresponding specific generation or tampering methods.

[0158] Step 1.5: Based on the method-level tags, construct the finest-grained platform-level tags to achieve the final source attribution labeling for the forged content.

[0159] By employing the above steps, training samples are assigned labels for real and fake, fake type, method-level, and platform-level, respectively, forming a training dataset containing multi-level fake semantic information.

[0160] In one possible implementation, the authenticity determination module includes:

[0161] 1) Multi-domain cue extraction unit: Normalizes and scales the input image to obtain the color domain baseline features as color domain cues. A multi-scale Laplacian operator is used to enhance the characterization of edge texture breaks and frequency domain anomalies in the input image at different scales, obtaining multi-scale high-frequency response feature maps as frequency domain cues. SRM (Spatial Rich Model) residual filtering is performed on the input image to obtain a noise residual map as noise residual domain cues.

[0162] 2) Authenticity screening unit: Construct multi-channel input features of color domain cues, frequency domain cues and noise residual domain cues into the same spatial scale, and input them into a lightweight feature extraction network for feature learning, and output authenticity confidence.

[0163] In one possible implementation, the feature extraction backbone network includes four resolution feature branches, each acquiring four types of resolution feature maps. When acquiring resolution feature maps, lower-resolution feature maps are obtained by downsampling the previous-level resolution feature map. Specifically, during resolution feature branch processing, the resolution feature map output from the previous-level resolution feature branch is used as input, and a convolution with a stride of 2 is used as a downsampling operator to reduce the spatial size of the feature map to half of its original size, thereby obtaining a lower-resolution feature map.

[0164] In one possible implementation, the pixel-level localization network is set on the highest resolution feature branch of the multi-branch pixel-level spoofing localization network. By performing convolutional mapping and spatial modeling on the highest resolution feature map, a spoofing probability map aligned with the input image space is generated. The spoofing probability map is further thresholded to obtain a high-resolution binary spoofing mask M.

[0165] In one possible implementation, the pixel-level localization network operates as follows:

[0166] The highest resolution feature map is input into the pixel-level localization network, and spatial alignment is achieved through convolutional mapping, outputting a pixel-level embedded feature map aligned with the input space.

[0167] Subsequently, the true feature center vector is obtained by averaging the pixel-level embedded feature map with the pixel embeddings of all real samples. And calculate the position of each pixel in the pixel-level embedded feature map. Embedded vector With the center vector distance :

[0168]

[0169] By setting the distance scale parameter R, Calculate the normalized distance between the embedding vector of each pixel and the ground truth feature center vector:

[0170]

[0171] Distance map consisting of distance values ​​from all pixel locations After mapping using the sigmoid function, a pixel-level forgery probability map is generated. :

[0172]

[0173] in, It is a pixel-level forgery probability map At pixel position The probability value at that location. For the sigmoid function, The scaling factor. This is the distance threshold parameter. Thresholding the probability map 𝑃 yields a binary forgery mask 𝑀, used to indicate the pixel-level location of the forgery region:

[0174]

[0175] in Represents pixels To forge the region, It is a probability threshold parameter used to control the strictness of the judgment of forged areas.

[0176] In one possible implementation, the decision network includes a cross-scale fusion module, a global convergence module, a hierarchical classification output module, and a gating constraint module.

[0177] The cross-scale fusion module consists of three cross-scale fusion units, each positioned between features of adjacent resolution scales. These units perform spatial size alignment and channel dimension alignment on adjacent scale features and obtain fused features through element-wise addition. The three cross-scale fusion units sequentially output type-level fusion features, method-level fusion features, and platform-level fusion features.

[0178] The global aggregation module is used to perform global average pooling on the fused features to obtain the global representation vector of the corresponding level.

[0179] The hierarchical classification output module includes a property-level classification output head, a type-level classification output head, a method-level classification output head, and a platform-level classification output head. Each classification output head adopts a fully connected mapping structure to map the global representation vector of the corresponding level to the label space of that level and output the classification score.

[0180] The gating constraint module constructs a gating vector based on the pre-defined label hierarchy mapping relationship of the multi-level label system, thereby obtaining the conditional probability vector of the corresponding level. Specifically, after obtaining the conditional probability vector of the property-level label, the type-level sourcing, method-level sourcing, and platform-level sourcing all construct gating vectors based on the conditional probability vector output from the previous level and the corresponding label hierarchy mapping relationship. These gating vectors then perform zero-weight suppression and normalization on the classification score of the current level's classification output head, thereby outputting the conditional probability vector of the current level's label.

[0181] In summary, this invention, through the design of a multi-level labeling system, a multi-branch feature extraction structure, pixel-level mask localization, and a platform-level tracing mechanism, realizes a complete chain from authenticity judgment to forgery nature identification, forgery method classification, generation model determination, and specific platform attribution, achieving an interpretable, traceable, and highly generalizable forgery image detection method.

[0182] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0183] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0184] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A layered, fine-grained image forgery detection method, characterized in that, Includes the following steps: Step 1: Construct a multi-level labeling system and obtain the training dataset; Step 2: Perform a preliminary determination of the authenticity of the input image using the authenticity detection module; Step 3: Construct a multi-branch pixel-level spoofing localization network, including a feature extraction backbone network, a pixel-level localization network, and a decision network; Step 4: Obtain multi-scale feature representations from low resolution to high resolution through the multi-resolution feature extraction mechanism of the feature extraction backbone network; Step 5: Based on the obtained multi-scale features, a high-resolution forgery mask is predicted through a pixel-level localization network, which is used to perform pixel-level localization of the forgery area. Step 6: The decision network constructs a coarse-to-fine reasoning path based on a multi-level label system. Based on multi-scale feature representation, it sequentially completes the forgery nature determination, forgery type determination, method-level tracing, and platform-level tracing, and outputs the conditional probability of the corresponding level label as the tracing result. The tracing results at each level after the forgery nature determination are all constrained by the tracing result of the previous level. Step 7: Train a multi-branch pixel-level fake localization network based on the acquired training dataset.

2. The layered fine-grained image forgery detection method according to claim 1, characterized in that, Step 1 is performed as follows: Step 1.1: Multi-level fake labeling system. The top level first performs preliminary labeling of the authenticity of the training samples, which is used to divide the training samples into real images and images containing fake content. Step 1.2: In the second-level label, after completing the authenticity labeling, for the forged sample, construct the forgery nature label according to the forgery nature, and divide the forgery nature into two major categories: tampering forgery and generation forgery; Tampering-type forgery is based on real images and is achieved by editing, enhancing, repairing, or replacing local areas; generative forgery, on the other hand, relies entirely on models to generate new images. Step 1.3: In the third-level label, after clarifying the nature of forgery, the forgery samples are further subdivided along the technical characteristics of the forgery method to construct forgery type labels; For tampering-type forgeries, they are further divided into two types: image enhancement and image editing. For generation-type forgeries, they are divided into GAN-based generation samples, diffusion model-based generation samples, Transformer-based generation samples, and hybrid generation samples that integrate multiple generation mechanisms, based on the technical route of their underlying generation framework. Step 1.4: After completing the forgery type labeling, further construct method-level labels for the forged samples to identify their corresponding specific generation or tampering methods; Step 1.5: Based on the method-level tags, construct the finest-grained platform-level tags to achieve the final source attribution labeling for the forged content; By employing the above steps, training samples are assigned labels for real and fake, fake type, method-level, and platform-level, respectively, forming a training dataset containing multi-level fake semantic information.

3. The layered fine-grained image forgery detection method according to claim 1, characterized in that, The authenticity detection module includes: 1) Multi-domain cue extraction unit: Normalizes and scales the input image to obtain the color domain baseline feature input as color domain cue; uses a multi-scale Laplacian operator to enhance the characterization of edge texture breaks and frequency domain anomalies in the input image at different scales to obtain a multi-scale high-frequency response feature map as frequency domain cue; performs SRM residual filtering on the input image to obtain a noise residual map as noise residual domain cue. 2) Authenticity screening unit: Construct multi-channel input features of color domain cues, frequency domain cues and noise residual domain cues into the same spatial scale, and input them into a lightweight feature extraction network for feature learning, and output authenticity confidence.

4. The layered fine-grained image forgery detection method according to claim 3, characterized in that, The feature extraction backbone network includes four resolution feature branches, which respectively acquire four types of resolution feature maps. When acquiring resolution feature maps, the lower resolution feature map is obtained by downsampling the previous level resolution feature map. Specifically, when processing the resolution feature branches, the resolution feature map output by the previous level resolution feature branch is used as input, and a convolution with a stride of 2 is used as a downsampling operator to reduce the spatial size of the feature map to 1 / 2 of its original size, thereby obtaining a lower resolution feature map.

5. The layered fine-grained image forgery detection method according to claim 4, characterized in that, The pixel-level localization network is set on the highest resolution feature branch of the multi-branch pixel-level spoofing localization network. By performing convolutional mapping and spatial modeling on the highest resolution feature map, a spoofing probability map aligned with the input image space is generated. The spoofing probability map is further thresholded to obtain a high-resolution binary spoofing mask.

6. The layered fine-grained image forgery detection method according to claim 5, characterized in that, The pixel-level positioning network operates as follows: The highest resolution feature map is input into the pixel-level localization network, and spatial alignment is achieved through convolutional mapping, outputting a pixel-level embedding feature map that is aligned with the input space. Subsequently, the true feature center vector is obtained by averaging the pixel-level embedded feature map with the pixel embeddings of all real samples. ; And calculate the position of each pixel in the pixel-level embedding feature map. Embedded vector With the center vector distance : By setting the distance scale parameter R, Calculate the normalized distance between the embedding vector of each pixel and the ground truth feature center vector: Distance map consisting of distance values ​​from all pixel locations After mapping using the sigmoid function, a pixel-level forgery probability map is generated. : in, It is a pixel-level forgery probability map At pixel position The probability value at that location. For the sigmoid function, The scaling factor. The distance threshold parameter is used; thresholding the probability map 𝑃 yields a binary forgery mask 𝑀, which is used to indicate the pixel-level location of the forgery region: in Represents pixels To forge the region, It is a probability threshold parameter used to control the strictness of the judgment of forged areas.

7. The layered fine-grained image forgery detection method according to claim 6, characterized in that, The decision network includes a cross-scale fusion module, a global convergence module, a hierarchical classification output module, and a gating constraint module. The cross-scale fusion module consists of three cross-scale fusion units, which are respectively set between adjacent resolution scale features. They are used to perform spatial size alignment and channel dimension alignment on adjacent scale features, and obtain fused features by element-wise addition. The three cross-scale fusion units output type-level fusion features, method-level fusion features and platform-level fusion features in sequence. The global aggregation module is used to perform global average pooling on the fused features to obtain the global representation vector of the corresponding level; The hierarchical classification output module includes a property-level classification output head, a type-level classification output head, a method-level classification output head, and a platform-level classification output head. Each classification output head adopts a fully connected mapping structure to map the global representation vector of the corresponding level to the label space of that level and output the classification score. The gating constraint module constructs a gating vector based on the pre-defined label hierarchy mapping relationship of the multi-level label system, thereby obtaining the conditional probability vector of the corresponding level. Specifically, after obtaining the conditional probability vector of the property-level label, the type-level tracing, method-level tracing, and platform-level tracing all construct gating vectors based on the conditional probability vector output by the previous level and the corresponding label hierarchy mapping relationship, and perform zero-weight suppression and normalization processing on the classification score of the current level classification output head, thereby outputting the conditional probability vector of the current level label.

8. The layered fine-grained image forgery detection method according to claim 7, characterized in that, The decision network adopts a path reasoning structure from coarse-grained to fine-grained, consistent with the multi-level labeling system. Property-level tracing: The decision network takes the lowest resolution feature map L1 as input and obtains the property-level global representation vector through the cross-scale fusion module and the global convergence module; this global representation vector is mapped by the fully connected layer of the property-level classification output head to output the classification scores of tamper-type forgery and generation-type forgery, and then normalized to obtain the probability vector of the forgery property. Type-level tracing: First, the L1 features are upsampled and aligned to the spatial size of the low-to-medium resolution feature map L2 by a cross-scale fusion unit, and channel alignment is completed by 1×1 convolution. Subsequently, the L2 features are added element-wise to obtain the type-level fusion feature F2, which is then subjected to global average pooling by the global aggregation module to obtain the type-level global representation vector. The type-level global representation vector is fully connected and mapped by the type-level classification output head to output the classification score of each candidate forgery type. Subsequently, the gating constraint module constructs a type-level gating vector based on the attribute-level output probability vector according to the subordinate mapping relationship of "forgery property - forgery type" in the multi-level label system. It performs zero-weight suppression on the type classification score that does not conform to the current forgery property and normalizes the retained branches to obtain the type-level conditional probability vector. Method-level source tracing: First, the L2 features are upsampled and aligned to the spatial size of the medium-to-high resolution feature map L3 through a cross-scale fusion unit, and channel alignment is completed through 1×1 convolution; then, the L3 features are added element-wise to obtain the method-level fusion feature F3. The global pooling module performs global average pooling on the fused feature F3, and under the constraint of the fake mask M, it only performs feature pooling on the pixel features predicted as fake regions to obtain the method-level global representation vector. The method-level classification output header outputs the classification score of each candidate forgery method based on this global representation vector; The gating constraint module constructs a method-level gating vector based on the type-level conditional probability vector and the preset "forgery type - forgery method" dependency relationship; it performs zero-weight suppression on the classification scores of methods inconsistent with the current forgery type to obtain the method-level conditional probability vector, where... Indicates the category of forgery method; Platform-level traceability: First, the L3 features are upsampled and aligned to the spatial size of the highest resolution feature map L4 through a cross-scale fusion unit, and channel alignment is completed through 1×1 convolution; then, the fused features F4 are obtained by element-wise addition with the L4 features. The global pooling module performs global average pooling on the fused feature F4, and under the constraint of the fake mask M, it only performs feature pooling on the pixel features predicted as fake regions to obtain a platform-level global representation vector. The platform-level classification output head performs a fully connected mapping on the global representation vector and outputs the classification score for each candidate platform; The gating constraint module performs zero-weight suppression on the platform classification scores that do not conform to the current forgery method based on the method-level conditional probability vector and the subordinate mapping relationship of "forgery method - platform", and normalizes the remaining branches to obtain the platform-level conditional probability vector. When the forgery method category of the sample to be detected does not have a corresponding platform label set in the preset "forgery method - platform" subordinate mapping relationship, the gating constraint module sets the platform-level gating vector to activate only the "unknown platform" label, and performs zero-weight suppression and normalization on the classification scores of the other platform labels, thereby outputting the platform-level conditional probability vector; at this time, the platform-level tracing result is output with "unknown platform", and the method-level tracing result is used as the final source determination basis; The confidence level of the final tracing result is obtained by multiplying the conditional probabilities of each level along the path, and the highest confidence level is used as the final classification result.

9. A layered fine-grained image forgery detection system, characterized in that, Includes the following modules: Dataset construction module: Constructs a multi-level labeling system and obtains the training dataset; Authenticity detection module: performs a preliminary determination of the authenticity of the input image; Feature extraction module: Obtains multi-scale feature representations from low resolution to high resolution through the multi-resolution feature extraction mechanism of the feature extraction backbone network; Pixel-level localization module: Based on the obtained multi-scale features, a high-resolution forgery mask is predicted through a pixel-level localization network, which is used to perform pixel-level localization of the forgery area; Decision module: Constructs a coarse-to-fine reasoning path through a decision network based on a multi-level label system. Based on multi-scale feature representation, it sequentially completes forgery nature determination, forgery type determination, method-level tracing, and platform-level tracing, and outputs the conditional probability of the corresponding level label as the tracing result. The tracing results at each level after forgery nature determination are all constrained by the tracing result of the previous level. Training module: Trains a multi-branch pixel-level spoofing localization network based on the training dataset obtained from the dataset construction module.

Images