Multi-view fusion and regional decoupling detection system for diffusion model image restoration counterfeiting

Abstract

The invention relates to the technical field of image forgery detection, and discloses a diffusion model image restoration forgery-oriented multi-view fusion and regional decoupling detection system, which comprises a forgery trace extraction module and a tampered region positioning fusion module, the forgery trace extraction module comprises a noise residual extractor, a high-frequency texture extractor, a cross-view fusion device and a multi-stage contrast learning device. According to the system, through a multi-view feature fusion mechanism and a region decoupling strategy, the detection problem caused by visual consistency, edge smoothness and noise distribution homogeneity of a diffusion model repair image is effectively solved; the innovative multi-level contrast learning framework can forcibly separate the feature representation of the counterfeit and real areas, and significantly improve the ability to capture weak counterfeit traces; the dynamic interactive fusion module realizes collaborative optimization of main body and edge features through a multi-scale channel attention mechanism; the DMIL-Net can rapidly and accurately mark the diffusion repair area in the image, and the manual checking burden is relieved.

Description

Technical Field

[0001] This invention relates to the field of image forgery detection technology, specifically to a multi-view fusion and region decoupling detection system for image restoration forgery based on diffusion models. Background Technology

[0002] In recent years, diffusion models have demonstrated remarkable visual fidelity in image restoration, completion, and re-creation due to their ability to model the physical diffusion process. However, it is precisely their high consistency in maintaining the distribution of structure, texture, lighting, and noise that makes the restored or tampered areas almost seamlessly integrated with the original content. Traditional forensic methods based on block matching, resampling traces, JPEG quantization tables, or sensor pattern noise quickly fail in the face of such forgeries. Even when introducing convolutional neural networks to directly learn RGB features, it is difficult to converge to a discriminative representation because the forgery traces are too subtle. Subsequent noise view and frequency domain enhancement methods attempt to amplify tampering clues through SRM filtering, BayarConv, and DCT / FFT spectral anomalies. However, diffusion models have completely reshaped the original noise distribution during the noise addition-denoising iterative process, making it difficult to distinguish the forged area from the real background in the frequency domain. Edge enhancement methods rely on artificially introduced discontinuous edges as priors. However, the smooth and natural edge transitions in diffusion restoration lead to high-confidence edge priors misleading the network. Therefore, how to simultaneously capture complementary information between RGB semantics, noise residuals and high-frequency details, and how to decouple and model the main body region from the detail edge under the premise of extremely blurred forgery boundaries have become two major bottlenecks that need to be overcome in image forensics research in the diffusion era. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a multi-view fusion and region decoupling detection system for image restoration using diffusion models. This system has advantages such as quickly and accurately marking diffusion restoration regions in images, thus solving the aforementioned technical problems.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a multi-view fusion and region decoupling detection system for image restoration of diffusion model forgery, including a forgery trace extraction module and a localization and tampering region fusion module; The forgery trace extraction module includes a noise residual extractor, a high-frequency texture extractor, a cross-view fusion unit, and a multi-level contrast learner. The noise residual extractor is based on the original RGB tampered image. Get noise view ; The high-frequency texture extractor is based on the original RGB tampered image. Obtaining high-frequency information ; The cross-view fusion processor is based on the original RGB altered image. Noise view and high-frequency information Extract tampering features, concatenate the tampering features of the three modalities at the same level, and finally process... The CBR operation yields fusion features; The multi-level contrast learner is used to upsample the tampered features extracted by the cross-view fusion machine and calculate pixel-level contrast loss. The location tampering area fusion module includes three decoder branches: main area decoupling branch, detail area decoupling branch, and interactive integration branch. The main region decoupling branch includes a coarse prediction decoder, a region enhancement block, and four serially connected main sensing blocks. The detailed region decoupling branch includes an edge prediction module, an edge enhancement module, and four serial detail-aware blocks; The interactive integration branch includes four serial interactive fusion blocks. Each interactive fusion block is used to read the subject perception block and detail perception block at the same level, fuse them, and then pass them to the next level interactive fusion block, subject perception block, and detail perception block. The last level interactive fusion block is used to output the prediction result.

[0005] As a preferred embodiment of the present invention, the noise residual extractor is based on the original tampered image. Get noise view The specific expression is as follows:

[0006] in, This represents a nonlocal mean filter. This indicates a modified original RGB image. Indicates the product operation; The high-frequency texture extractor is based on the original RGB tampered image. Obtaining high-frequency information The specific expression is as follows:

[0007] in, This indicates a 2x upsampling operation. This indicates a 2x downsampling operation. Represents the convolution operator for a Gaussian filter; The cross-view fusion processor is based on the original RGB altered image. Noise view and high-frequency information The specific expression for extracting tampering features is as follows:

[0008]

[0009]

[0010] in, This represents the output features of levels 1-4 of the spatial branch Backbone. This represents the output characteristics of levels 1-4 of the noise branch backbone. This represents the output features of levels 1-4 of the high-frequency branch Backbone. This indicates PVTv2 pre-trained on ImageNet. and These represent PVTv2 instances that have not been pre-trained; The cross-view fusion processor performs a concatenation operation on the tampering features of three modalities at the same level, and finally... The expression for the fused features obtained by the CBR operation is as follows:

[0011] in, These represent the high-frequency, spatial, and noise branches of the Backbone, respectively. Hierarchical output features express CBR operation, Indicates splicing, the The CBR operations include convolution, batch normalization, and ReLU activation processing.

[0012] As a preferred embodiment of the present invention, the multi-level contrast learner is used to upsample the tampered features extracted by the cross-view fusion processor and calculate the pixel-level contrast loss. The specific steps are as follows: Step A1: Use different numbers of 4×4 transposed convolutions with a stride of 2 to... , , Upsampling by 2, 4, and 8 times respectively, the specific expressions are as follows:

[0013]

[0014]

[0015] in, Indicates the upsampling , Indicates the upsampling , Indicates the upsampling , express CBR operation, express The transpose convolution operation; Step A2: Downsample the real mask by a factor of 4 using sampling and reshaping operations; Step A3: Use a semi-hard sample sampling strategy to calculate pixel-level contrast loss. The specific expression is as follows:

[0016]

[0017] in, Represents pixels Positive samples, Represents pixels negative samples, , These correspond to the positive sample feature representation and the negative sample feature representation of the pixel, respectively. This represents the absolute value of the similarity probability between the features of the anchor pixel and the features of the target sample or negative sample. Indicates an exponential mapping. Represents a logarithmic mapping. This indicates temperature hyperparameters. Indicates pixel-level contrast loss. It indicates that it is aimed at f 2 Pixel-level contrast loss for hierarchical feature computation Indicating targeting f 3 Pixel-level contrast loss for hierarchical feature computation Indicates based on f 4 Pixel-level contrast loss for hierarchical features.

[0018] As a preferred embodiment of the present invention, the coarse prediction decoder is used to generate coarse localization results. The specific expression is as follows:

[0019]

[0020] in, This indicates a 2x upsampling operation. This indicates a 2x downsampling operation. This represents the product operation. , , These represent the fusion features at different levels. express CBR operation, Indicates splicing; The edge prediction module is used to generate an initial predicted edge map. The specific expression is as follows:

[0021] in, Represents the Sobel operator. This represents the Sigmoid function. Indicates splicing, This indicates an element-wise addition operation. This indicates an upsampling operation.

[0022] As a preferred embodiment of the present invention, the region enhancement block and the edge enhancement module have the same structure, and the edge enhancement module is used to enhance the edge map. Inject into the decoupling branch of the detail region and integrate global context features, as shown in the following expression:

[0023]

[0024] in, Indicates that the multi-view fusion features are processed through The main features corresponding to the current branch obtained after CBR processing This represents the product operation. Indicates splicing, This indicates an element-wise addition operation. Represents global context features, This represents a hierarchical multi-view fusion feature that has undergone edge enhancement preprocessing.

[0025] As a preferred embodiment of the present invention, the detail-aware block first integrates global context features. Interactive fusion features output by the upper-layer interactive fusion block and characteristics The specific expression is as follows:

[0026] in, Indicates integrated features representing multi-source features. express CBR operation, This indicates splicing; secondly, it means... Enter them into one respectively The specific expressions for CBR and the three parallel dilated convolutions with dilation rates of 2, 3 and 4 are as follows:

[0027]

[0028] in, This represents dilated convolution. Indicates the index identifier of adjacent scale pairs. This represents the characteristic residual of the m-th group of adjacent scales. The dilation rate of the dilated convolution is used to obtain the output features of the detail-aware block. and detailed diagrams The specific expression is as follows:

[0029]

[0030] in, This indicates an element-wise addition operation. express CBR operation, express CBR operation, , , Residuals representing characteristics at different adjacent scales.

[0031] As a preferred embodiment of the present invention, the subject-sensing block and the detail-sensing block have the same structure, and their specific input is: global context features. Interactive fusion features output by the upper-layer interactive fusion block The main area-guided multi-view fusion feature after area enhancement preprocessing Its output is the main map. Decoupling features of the main body For specific calculation formulas, please refer to the detailed perception block.

[0032] As a preferred embodiment of the present invention, the interactive fusion block includes the following expression:

[0033]

[0034]

[0035] in, express CBR operation, , , These represent the detailed optimization features, the main optimization features, and the original detailed decoupling features, respectively. This indicates the decoupling characteristics of the main body. This indicates a fusion of main features and detailed features. This represents the concatenation process, followed by the introduction of a multi-scale channel attention mechanism, ultimately yielding interactive fusion features, as shown in the following expression:

[0036] in, Indicates interactive fusion features, This represents a multi-scale channel attention mechanism.

[0037] Compared with existing technologies, this invention provides a multi-view fusion and region decoupling detection system for image restoration of diffusion models, which has the following advantages: This invention effectively solves the detection challenges of visual consistency, edge smoothness, and homogeneity of noise distribution in images restored by diffusion models through a multi-view feature fusion mechanism and region decoupling strategy. The innovative multi-level contrastive learning framework can forcibly separate the feature representations of fake and real regions, significantly improving the ability to capture weak forgery traces. The dynamic interactive fusion module achieves collaborative optimization of subject and edge features through a multi-scale channel attention mechanism. DMIL-Net can quickly and accurately mark diffusion restoration areas in images, reducing the burden of manual review. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall framework of the present invention; Figure 2 This is a schematic diagram of the noise view extraction process of the present invention; Figure 3 This is a schematic diagram of the multi-level comparison learning module structure of the present invention; Figure 4 This is a schematic diagram of the interactive fusion block of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Please see Figure 1 - Figure 4 A multi-view fusion and region decoupling detection system for image restoration of diffusion model forgery includes a forgery trace extraction module and a tampered region fusion module; The image forgery detection and forgery trace extraction module includes a noise residual extractor, a high-frequency texture extractor, a cross-view fusion unit, and a multi-level contrast learner; Image forgery detection noise residual extractor based on original RGB tampered image Get noise view ; The tampered region generated by the diffusion model may have noise patterns different from the real region, and these patterns can serve as clues to tampering; based on this, this invention designs a noise residual extractor; specifically, as follows... Figure 1 As shown, firstly, the NRE uses a nonlocal mean filter as a denoising method to generate a noise-free clean view corresponding to the original tampered image; then, the clean view is subtracted from the original tampered image to obtain the noise residual; finally, to further enhance the difference region, the value of the noise residual image is multiplied by 100. This process can be described as follows:

[0041] in, This represents a nonlocal mean filter. This indicates a modified original RGB image. Indicates the product operation; Image forgery detection high-frequency texture extractor based on original RGB tampered image Obtaining high-frequency information ; High-frequency features contain detailed information such as edges, textures, and patterns in an image, which is crucial for identifying subtle alterations. In image tampering forensics, the loss or anomaly of these details is often direct evidence of tampering. The Laplacian of Gaussian operator, approximated using the difference of Gaussians operator, effectively preserves the most salient high-frequency features. Based on this, a high-frequency extractor was designed to apply the Laplacian of Gaussian function to the original RGB tampered image to obtain high-frequency information. This process can be described as follows:

[0042] in, This indicates a 2x upsampling operation. This indicates a 2x downsampling operation. Represents the convolution operator for a Gaussian filter; Image forgery detection cross-view fusion based on original RGB tampered images Noise view and high-frequency information Extract tampering features, concatenate the tampering features of the three modalities at the same level, and finally process... The CBR operation yields fusion features; Cross-view fusion DMIL-Net uses three PVTv2 networks as its backbone to transform the original RGB-modified image. High-frequency information and noise view The data are then fed into the corresponding backbone networks to extract tampering features.

[0043]

[0044]

[0045] in, Indicates spatial characteristics, Indicates noise characteristics, Indicates high-frequency characteristics, This indicates PVTv2 pre-trained on ImageNet. and These represent untrained PVTv2; subsequently, the present invention performs a concatenation operation on the features of the three modalities at the same level, and utilizes... The CBR (Convolution, Batch Normalization, and ReLU) function adjusts the number of channels for the fused features to 64.

[0046] in, These represent spatial characteristics, noise characteristics, and high-frequency characteristics, respectively. express CBR operation, Indicates splicing, The CBR operations include convolution, batch normalization, and ReLU activation processing; The image forgery detection multi-level contrast learner is used to upsample the tampering features extracted by the cross-view fusion and calculate pixel-level contrast loss; Contrastive learning learns effective representations of data by comparing positive and negative sample pairs; in feature fusion tasks involving three views, it can better capture long-term dependencies between the three modalities and generate more comprehensive fused features; this invention proposes a multi-level contrastive learning strategy; firstly, applying a pixel-level contrastive learning strategy to features at multiple levels helps the model form a deep understanding of the scene at each level, thereby more accurately locating tampered areas; such as Figure 3 As shown, select As the feature input of the MCL module, different numbers of 4×4 transposed convolutions with a stride of 2 are used to... , , Upsampled by 2, 4, and 8 times respectively;

[0047]

[0048]

[0049] in, Indicates the upsampling , Indicates the upsampling , Indicates the upsampling , express CBR operation, express The transposed convolution operation is significant in restoring detailed information, enhancing feature representation capabilities, and ensuring feature space dimensionality matching, thereby improving the accuracy and efficiency of contrastive learning. Secondly, to reduce computational costs and improve efficiency, sampling and reshaping operations are used to downsample the real mask by a factor of 4, ensuring its accuracy and efficiency. The dimensions are the same; the meaning of the sampling operation is: if the proportion of tampered pixels in a batch exceeds 25%, then the batch is marked as "tampered"; if it is less than 25%, then it is marked as "true"; The core meaning of sampling operation is: during model training, the pixel proportion of the processed mask batch is statistically analyzed to quickly classify "tampered" or "real" labels, and to assist in comparative learning to calculate loss.

[0050] Key disassembly: 1. Statistical object: For a batch of data during model training, focus on the pixels of the groundtruthmask (real tampering label mask) within that batch.

[0051] 2. Judgment criteria: Calculate the proportion of "tampered pixels" in a batch to the total number of pixels in that batch.

[0052] 3. Labeling results: Percentage > 25% → This batch is marked as "tampered"; Percentage < 25% → This batch is marked as "authentic".

[0053] Core objective: This operation is designed to work in conjunction with the Multi-Level Contrast Learning (MCL) module, simplifying the calculation of pixel-level contrast loss. By quickly dividing batch labels, it helps the model efficiently learn the feature differences between the tampered and real regions, while reducing computational complexity. The features will subsequently participate in semi-hard sample sampling and pixel-level contrast loss calculation, helping the model capture the long-term dependencies of multi-view features and providing fusion feature optimization support for the subsequent accurate localization of tampered regions in DMIL-Net. These are key intermediate features for the MCL module to achieve hierarchical scene understanding. Finally, a semi-hard sample sampling strategy is used to calculate the pixel-level contrast loss. For a specific hierarchical feature of each training image, a memory bank is constructed by sampling 10 pixels for each class. Then, for each pixel anchor, 512 closest negative samples and 512 randomly selected negative samples are extracted from the memory bank to calculate the contrast loss. Therefore:

[0054]

[0055] in, Represents pixels Positive samples, Represents pixels negative samples, , These correspond to the positive sample feature representation and the negative sample feature representation of the pixel, respectively. This represents the absolute value of the similarity probability between the features of the anchor pixel and the features of the target sample or negative sample. Indicates an exponential mapping. Represents a logarithmic mapping. This indicates temperature hyperparameters. Indicates pixel-level contrast loss. It indicates that it is aimed at f Pixel-level contrast loss calculated using 2-level feature computation. Indicating targeting f Pixel-level contrast loss calculated using 3-level feature computation Indicates based on f Pixel-level contrast loss for 4-level features.

[0056] The image forgery detection and tampering region fusion module includes three decoder branches: main region decoupling branch, detail region decoupling branch, and interactive integration branch. The image forgery detection subject region decoupling branch includes a coarse prediction decoder, a region enhancement block, and four serial subject perception blocks connected in sequence. The image forgery detection detail region decoupling branch includes an edge prediction module, an edge enhancement module, and four serial detail-aware blocks; The image forgery detection interactive integration branch includes four serial interactive fusion blocks. Each interactive fusion block is used to read the subject perception block and detail perception block at the same level, fuse them, and then pass them to the next level interactive fusion block, subject perception block, and detail perception block. The last level interactive fusion block is used to output the prediction result. Specifically, in the stage of locating the tampered region, this invention designs three decoder branches: a main region decoupling branch, a detail region decoupling branch, and an interactive integration branch. The main region decoupling branch aims to decouple the main region of the tampered region and consists of a coarse prediction decoder (CPD), a region enhancement block (REB), and four main perceptual blocks (BAB). The detail region branch aims to decouple the detail region of the tampered region and consists of an edge prediction module (EPM), an edge enhancement module (EEB), and four detail perceptual blocks (DAB). The interactive integration branch aims to achieve the interactive integration of the output features of the two branches and consists of four interactive fusion blocks (IFB). The coarse prediction decoder is used to generate coarse localization results. The specific expression is as follows:

[0057]

[0058] in, This indicates a 2x upsampling operation. This indicates a 2x downsampling operation. This represents the product operation. , , These represent the fusion features at different levels. express CBR operation, Indicates splicing; The edge prediction module is mainly used to generate the initial predicted edge map. First, the edge prediction module uses low-level features... and high-level features As input, the low-level features contain rich edge details, while the high-level features possess rich semantic information. Secondly, to enhance the edge-related patterns, a Sobel layer is introduced. The Sobel layer distinguishes edge-related pixels from other pixels in a given feature map by using edge-related weights. Finally, an edge map is obtained through a series of convolutional operations. The specific expression is as follows:

[0059] in, Represents the Sobel operator. This represents the Sigmoid function. Indicates splicing, This indicates an element-wise addition operation. Indicates an upsampling operation; The region enhancement block has the same structure as the edge enhancement module. The region enhancement block aims to enhance the main region decoupling branch's perception of the tampered region using coarse localization results. Since the detail region is mainly composed of the edges of the tampered region and surrounding pixels, the region enhancement module has a completely identical structure to the edge enhancement module. Its core includes feature preprocessing, global context fusion, and size matching modules. The input is multi-view fusion features. (i=1~4, corresponding to different levels), first the number of channels is adjusted by the CBR module; then global contextual features are incorporated to grasp the overall content of the image, while all features are upsampled to the shallow features ( Matching dimensions; final output main body enhancement features The system provides optimized feature input for subsequent Subject Aware Block (BAB) analysis of the tampered main body of the region. The edge enhancement module utilizes the edge map to enhance the perception of edge regions by the detail region decoupling branch. To ensure the balance between the two decoupling branches, the region enhancement block and the edge prediction module are designed to have the same structure. Here, the invention will be described in detail using the edge enhancement module as an example. The edge enhancement module uses prior edge information... Injecting into the decoupling branch of the detail region effectively enhances the branch's ability to perceive the edges of camouflaged objects. Furthermore, the edge enhancement module integrates global contextual features, enabling the network to more comprehensively understand the content of the entire image, thereby improving the model's overall performance in complex scenes. The edge enhancement module is used to integrate edge maps... Inject into the decoupling branch of the detail region and integrate global context features, as shown in the following expression:

[0060]

[0061] in, Indicates that the multi-view fusion features are processed through The main features corresponding to the current branch obtained after CBR processing This represents the product operation. Indicates splicing, This indicates an element-wise addition operation. Represents global context features, This indicates a detail-oriented multi-view fusion feature that has undergone edge enhancement preprocessing. The dimensions were all upsampled to the same size as same, Right now , It is the mid-level detail feature association identifier of the detail region decoupling branch, and its core meaning is the integration or association form of the detail decoupling features of the second and third layers; Detail-aware blocks are primarily used to analyze the detailed regions of the tampered area and generate a detail region map; to obtain the most comprehensive information, detail-aware blocks first integrate global context features. Interactive fusion features output by the upper-layer interactive fusion block (Current level features are not) (time), and characteristics ; The detail-aware block first integrates global contextual features. Interactive fusion features output by the upper-layer interactive fusion block and characteristics The specific expression is as follows:

[0062] in, This represents the underlying integrated features used for multi-scale feature capture in the detail-aware block. express CBR operation, This indicates splicing; secondly, it means... Enter them into one respectively The specific expressions for CBR and the three parallel dilated convolutions with dilation rates of 2, 3 and 4 are as follows:

[0063]

[0064] in, This represents dilated convolution. Indicates the index identifier of adjacent scale pairs. This represents the characteristic residual of the m-th group of adjacent scales. The dilation rate of the dilated convolution is used to obtain the output features of the detail-aware block. and detailed diagrams The specific expression is as follows:

[0065]

[0066] in, This indicates an element-wise addition operation. express CBR operation, express CBR operation, , , These represent the third-level feature residuals obtained by calculating the difference between feature maps at adjacent scales within the detail-aware block; The subject-aware block and the detail-aware block have the same structure, and their input is global contextual features. Interactive fusion features output by the upper-layer interactive fusion block (Current level features are not) (time), and characteristics Its inputs are global context features, interactive fusion features output from the previous layer's interactive fusion block, and other features; its output is the main body map. Decoupling features of the main body The Subject-Aware Block (BAB) is the core module of the subject region decoupling branch in the "Locate Tampered Regions" stage of the DMIL-Net model. Its structure is completely consistent with that of the Detail-Aware Block (DAB). This module uses the subject-enhanced features processed by the Region Enhancement Block (REB). The system takes as input global contextual features (to ensure consistency between subject analysis and global image semantics) and features output from the previous layer's Interactive Fusion Block (IFB) (to ensure feature coherence between layers). It first integrates multi-source input information, then captures multi-scale features through a CBR module and parallel dilated convolution, and calculates the differences between adjacent scale features to fill in uncovered subject details. Finally, it outputs a subject map that marks the tampered area. and the main decoupling features used for subsequent feature fusion. This lays the foundation for the collaborative fusion of main and detailed features in the Interactive Fusion Block (IFB); The main purpose of the interactive fusion block is to fuse the output features of two decoupled branches at a specific level and generate a complete localization result; its structure is as follows: Figure 4 As shown; specifically, feature interaction and feature fusion are first achieved through interconnected forwarding; this process can be described as follows:

[0067]

[0068]

[0069] in, express CBR operation, The detail perception module integrates and fuses features. Key features of the detail region obtained by applying 3×3 CBR after the residuals of the third-level features. It is in the interactive integration module After 3×3 CBR processing, the key features of the main body region output by topic perception are compared with those of the main body region. Intermediate features of the splicing In IFB After being processed by 3×3CBR and Intermediate features of the splicing The process involves concatenation, followed by the introduction of a multi-scale channel attention mechanism (MSCA) to further optimize information fusion by enhancing feature selectivity. MSCA comprises two branches: one branch obtains global contextual information through global average pooling, and the other branch obtains local contextual information. This process can be described as follows:

[0070] in, Indicates interactive fusion features, This represents a multi-scale channel attention mechanism. It has two uses: one is to predict fine localization results, and the other is to decouple the next level of detail regions and edge regions. Furthermore, for this system, the loss function constructed during training adopts a combination of the weighted binary cross-entropy (WBCE) loss function and the weighted intersection-over-union (WIoU) loss function, which are used for fine-grained localization results. ) and coarse-grained localization results ( Optimization of the main map; ( ) and detailed map ( For supervised training, this invention uses the BCE loss function as its loss function; for edge-supervised tasks, the Dice loss function is used to handle the imbalance of positive and negative samples. in, , and Let the labels be detail region labels, main region labels, and edge labels, respectively; then the total loss function is: here, , , , , As hyperparameters, these are set to 1, 1, 1, 1, 0.5 respectively in this invention; Figure 2The diagram illustrates the noise view extraction process, showcasing the core workflow of the noise residual extractor: First, the original tampered image is used as input, and a corresponding noise-free clean view is generated using non-local mean filtering. Then, the noise residual is obtained by calculating the difference between the original tampered image and the clean view. Finally, the value of the noise residual map is multiplied by 100 to enhance the difference region, and the final output is a noise view that can be used to capture tampering traces, providing tampering clues in the noise dimension for subsequent multi-view feature learning of the network.

[0071] Figure 3 The diagram illustrates the structure of the multi-level contrastive learning module, showcasing the core process of MCL in capturing long-term dependencies of multi-view features: the module fuses features from multiple views. f2 , f3 , f4 As input, the above features are upsampled by 2x, 4x, and 8x respectively through different numbers of double-strut transposed convolutions; at the same time, the GroundTruth mask is downsampled by 4x (the sampling rule is that if the proportion of tampered pixels in the batch exceeds 25%, it is marked as tampered, otherwise it is marked as real) to match the size of the upsampled features; then, a semi-hard sample sampling strategy is adopted to sample pixels from each category to build a memory bank, and negative samples are selected for each pixel anchor point to calculate pixel-level contrast loss, which helps the model to deeply understand the image scene at each level and improve the accuracy of tampered region localization.

[0072] Figure 4 The diagram shows the structure of the Interactive Fusion Block (IFB), illustrating the core process of IFB fusion of main and detailed features: the module outputs main features at the same level as the Body Awareness Block (BAB). Detail-aware block (DAB) output detail features As input, preliminary fusion features are first obtained by concatenating (Cat) with the CBR module. Subsequently, Multi-Scale Channel Attention (MSCA) is introduced, which processes information in two branches: one branch obtains global contextual information through Global Average Pooling (GAP), and the other branch captures local contextual information. Feature selectivity is optimized through pointwise convolution (PWC), activation functions, and other operations. Finally, interactive fused features are output. This feature is used to predict the complete tamper location result at the current level, and also to pass coherent features to the decoupling task at the next level, achieving a synergy between the integrity of the subject and the richness of details. The specific ablation experiment of this invention is as follows: Six progressively enhanced model configurations (from Scheme 1 to Scheme 6) were designed on the test set of the self-built DMI50K dataset to systematically evaluate the specific contributions of each key module in DMIL-Net to the overall performance. The experiments used F1-score as the evaluation metric. The test set contained five subsets (BN, PE, IA, PP, RP), each containing 2000 images, corresponding to tampered samples generated by different diffusion inpainting methods. Scheme 1 was the basic model using only RGB views, with an average F1 score of approximately 0.81. Scheme 2 introduced a High Frequency Feature Extractor (HFE), improving the F1 score from 0.852 to 0.867 on the PP subset, indicating that high-frequency information helps capture subtle anomalies. Scheme 3 further added a Noise Residual Extractor (NRE), increasing the F1 score from 0.593 to [missing value] on the RP subset. The F1 score was 0.604, validating the sensitivity of the noisy view to traces left by the diffusion model. Scheme 4 introduced multi-level contrastive learning (MCL), which improved the F1 score from 0.857 to 0.873 on the IA subset, indicating that cross-modal dependency modeling significantly enhances the discriminative ability of fused features. Scheme 5 added region decoupling modules (BAB / DAB / IFB), which generally improved the F1 score across all subsets, demonstrating the contribution of structural decoupling to edge integrity. Scheme 6 is the complete model, which further adds a coarse prediction decoder (CPD) and an edge prediction module (EPM). Finally, it achieved F1 scores of 0.918, 0.928, 0.893, 0.936, and 0.627 on the five subsets BN, PE, IA, PP, and RP, respectively, with an average improvement of more than 10% compared to Scheme 1. This fully demonstrates that the synergistic effect of each module is crucial for improving the performance of tampered region localization. The SOTA comparison experiment of this invention is as follows: A comprehensive comparative experiment was conducted on the DMI50K dataset, comparing DMIL-Net with five current mainstream image tampering localization methods (MVSS-Net, MFI-Net, TA-Net, CFL-Net, and EMF-Net), using the F1 score as the evaluation metric. The test set included five subsets: BN, PE, IA, PP, and RP, each containing 2000 images to ensure fairness and representativeness in the comparison. Experimental results show that DMIL-Net achieved the highest F1 score on all subsets, with F1 scores of 0.918, 0.928, and 0.936 on the BN, PE, and PP subsets, respectively, significantly outperforming the second-best performing EMF-Net (0.899). The performance improvement reached 1.9%–6.1% (0.881, 0.875). On the IA subset, DMIL-Net's F1 score of 0.893 far surpassed EMF-Net's 0.855, leading by 3.8 percentage points. Even on the most challenging RP subset, DMIL-Net still ranked first with an F1 score of 0.627, demonstrating its robust recognition ability for complex restoration areas. Further visualization analysis showed that the localization map generated by DMIL-Net outperformed the comparison methods in terms of edge integrity, region continuity, and detail preservation. In particular, it can still effectively identify weak tampering traces in scenarios where the content generated by the diffusion model is highly integrated with the natural background, verifying its state-of-the-art performance in the diffusion model image restoration tampering localization task. The robustness experiments of this invention are as follows: DMIL-Net was robustly evaluated on the BN subset of the DMI50K dataset to test the model's stability against common image processing attacks. The experiments simulated image quality degradation in real-world scenarios by applying Gaussian noise and Gaussian blur filtering to the test images at different intensities, with noise variance and blur kernel size increasing progressively. The results show that although the F1-score of all methods decreased to varying degrees with increasing interference intensity, DMIL-Net consistently maintained the highest localization accuracy under all levels of attacks, exhibiting minimal performance fluctuations. For example, under high-intensity noise (σ=25) and strong blur (kernel size=7×7) conditions, DMIL-Net's F1 score decreased by only about 3.2%, significantly better than EMF-Net's 7.8% and MVSS-Net's 12.4%. Visual comparisons further demonstrate that DMIL-Net is least affected by interference in terms of edge preservation and region integrity, verifying its excellent robustness and resistance to attacks. This invention has the following significant advantages and positive effects: By using a multi-view feature fusion mechanism (RGB semantics, noise residuals, and high-frequency textures) and a region decoupling strategy (subject / detail hierarchical modeling), it effectively solves the detection challenges caused by visual consistency, edge smoothness, and homogeneity of noise distribution in images restored by diffusion models; the innovative multi-level contrastive learning framework can forcibly separate the feature representations of forged and real regions, significantly improving the ability to capture weak forgery traces; the dynamic interactive fusion module achieves collaborative optimization of subject and edge features through a multi-scale channel attention mechanism; DMIL-Net can quickly and accurately mark diffusion restoration areas in images, reducing the burden of manual review and providing a usable tool for evidence collection, media review, and copyright tracking.

[0073] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-view fusion and region decoupling detection system for image restoration forgery based on diffusion models, characterized in that: This includes a forgery trace extraction module and a location tampering area fusion module; The forgery trace extraction module includes a noise residual extractor, a high-frequency texture extractor, a cross-view fusion unit, and a multi-level contrast learner. The noise residual extractor is based on the original RGB tampered image. Get noise view ; The high-frequency texture extractor is based on the original RGB tampered image. Obtaining high-frequency information ; The cross-view fusion processor is based on the original RGB altered image. Noise view and high-frequency information Extract tampering features, concatenate the tampering features of the three modalities at the same level, and finally process... The CBR operation yields fusion features; The multi-level contrast learner is used to upsample the tampered features extracted by the cross-view fusion machine and calculate pixel-level contrast loss. The location tampering area fusion module includes three decoder branches: main area decoupling branch, detail area decoupling branch, and interactive integration branch. The main region decoupling branch includes a coarse prediction decoder, a region enhancement block, and four serially connected main sensing blocks. The detailed region decoupling branch includes an edge prediction module, an edge enhancement module, and four serial detail-aware blocks; The interactive integration branch includes four serial interactive fusion blocks. Each interactive fusion block is used to read the subject perception block and detail perception block at the same level, fuse them, and then pass them to the next level interactive fusion block, subject perception block, and detail perception block. The last level interactive fusion block is used to output the prediction result.

2. The multi-view fusion and region decoupling detection system for image restoration based on diffusion model forgery as described in claim 1, characterized in that: The noise residual extractor is based on the original tampered image. Get noise view The specific expression is as follows: in, This represents a nonlocal mean filter. This indicates a modified original RGB image. Indicates the product operation; The high-frequency texture extractor is based on the original RGB tampered image. Obtaining high-frequency information The specific expression is as follows: in, This indicates a 2x upsampling operation. This indicates a 2x downsampling operation. Represents the convolution operator for a Gaussian filter; The cross-view fusion processor is based on the original RGB altered image. Noise view and high-frequency information The specific expression for extracting tampering features is as follows: in, This represents the output features of levels 1-4 of the spatial branch Backbone. This represents the output characteristics of levels 1-4 of the noise branch backbone. This represents the output features of levels 1-4 of the high-frequency branch Backbone. This indicates PVTv2 pre-trained on ImageNet. and These represent PVTv2 instances that have not been pre-trained; The cross-view fusion processor performs a concatenation operation on the tampering features of three modalities at the same level, and finally... The expression for the fused features obtained by the CBR operation is as follows: in, These represent the high-frequency, spatial, and noise branches of the Backbone, respectively. Hierarchical output features express CBR operation, Indicates splicing, the The CBR operations include convolution, batch normalization, and ReLU activation processing.

3. The multi-view fusion and region decoupling detection system for image restoration and forgery based on a diffusion model as described in claim 2, characterized in that: The multi-level contrast learner is used to upsample the tampered features extracted by the cross-view fusion machine, and the specific steps for calculating the pixel-level contrast loss are as follows: Step A1: Use different numbers of 4×4 transposed convolutions with a stride of 2 to... , , Upsampling by 2, 4, and 8 times respectively, the specific expressions are as follows: in, Indicates the upsampling , Indicates the upsampling , Indicates the upsampling , express CBR operation, express The transpose convolution operation; Step A2: Use sampling and reshaping operations to obtain the true mask. Downsampling by 4 times; Step A3: Use a semi-hard sample sampling strategy to calculate pixel-level contrast loss. The specific expression is as follows: in, Represents pixels Positive samples Represents pixels negative samples, , These correspond to the positive sample feature representation and the negative sample feature representation of the pixel, respectively. This represents the absolute value of the similarity probability between the features of the anchor pixel and the features of the target sample or negative sample. Represents an exponential mapping. Represents a logarithmic mapping. This indicates temperature hyperparameters. Indicates pixel-level contrast loss. It indicates that it is aimed at f 2 Pixel-level contrast loss for hierarchical feature computation Indicating targeting f 3 Pixel-level contrast loss for hierarchical feature computation Indicates based on f 4 Pixel-level contrast loss for hierarchical features.

4. The multi-view fusion and region decoupling detection system for image restoration and forgery based on a diffusion model as described in claim 1, characterized in that: The coarse prediction decoder is used to generate coarse localization results. The specific expression is as follows: in, This indicates a 2x upsampling operation. This indicates a 2x downsampling operation. This represents the product operation. , , This represents the fusion features at different levels. express CBR operation, Indicates splicing; The edge prediction module is used to generate an initial predicted edge map. The specific expression is as follows: in, Represents the Sobel operator. This represents the Sigmoid function. Indicates splicing, This indicates an element-wise addition operation. This indicates an 8x upsampling operation.

5. The multi-view fusion and region decoupling detection system for image restoration of diffusion models according to claim 4, characterized in that: The region enhancement block has the same structure as the edge enhancement module, and the edge enhancement module is used to enhance the edge map. Inject into the decoupling branch of the detail region and integrate global context features, as shown in the following expression: in, Indicates that the multi-view fusion features are processed through The main features corresponding to the current branch obtained after CBR processing This represents the product operation. Indicates splicing, This indicates an element-wise addition operation. Represents global context features, This represents a hierarchical multi-view fusion feature that has undergone edge enhancement preprocessing.

6. The multi-view fusion and region decoupling detection system for image restoration of diffusion models according to claim 1, characterized in that: The detail-aware block first integrates global context features. Interactive fusion features output by the upper-layer interactive fusion block and characteristics The specific expression is as follows: in, Indicates integrated features representing multi-source features. express CBR operation, This indicates splicing; secondly, it means... Enter them into one respectively The specific expressions for CBR and the three parallel dilated convolutions with dilation rates of 2, 3 and 4 are as follows: in, This represents dilated convolution. Indicates the index identifier of adjacent scale pairs. This represents the characteristic residual of the m-th group of adjacent scales. The dilation rate of the dilated convolution is used to obtain the output features of the detail-aware block. and detailed diagrams The specific expression is as follows: in, This indicates an element-wise addition operation. express CBR operation, express CBR operation, , , Residuals representing characteristics at different adjacent scales.

7. The multi-view fusion and region decoupling detection system for image restoration of diffusion models according to claim 6, characterized in that: The subject-aware block and the detail-aware block have the same structure, and their specific input is: global context features. Interactive fusion features output by the upper-layer interactive fusion block The main area-guided multi-view fusion feature after area enhancement preprocessing Its output is the main map. Decoupling features of the main body For specific calculation formulas, please refer to the detailed perception block.

8. The multi-view fusion and region decoupling detection system for image restoration of diffusion models according to claim 7, characterized in that: The interactive fusion block includes the following expression: in, express CBR operation, , , These represent the detailed optimization features, the main optimization features, and the original detailed decoupling features, respectively. This indicates the decoupling characteristics of the main body. This indicates a fusion of main features and detailed features. This represents the concatenation process, followed by the introduction of a multi-scale channel attention mechanism, ultimately yielding interactive fusion features, as shown in the following expression: in, Indicates interactive fusion features, This represents a multi-scale channel attention mechanism.

Images