Image restoration detection method and system based on cross-domain representation modeling, and electronic equipment

Abstract

The invention provides an image restoration detection method and system based on cross-domain representation modeling, and electronic equipment. The method comprises the following steps of: 1, preprocessing an image by using a minimum filling strategy; step 2, extracting three kinds of noise information of Noiseprint + +, SRM filtering and Bayar convolution from the image, and inputting the noise information to a noise information gating unit for splicing and fusion to obtain fused noise information; 3, inputting the image preprocessed in the step 1 and the noise information fused in the step 2 into a mask double-flow encoder to generate a feature map; and 4, inputting the feature map generated in the step 3 into a simple feature pyramid to generate a multi-scale feature map, fusing all features, outputting final prediction, and calculating a loss function. Compared with the prior art, the method has higher detection capability and robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital forensics and computer vision, and specifically relates to an image restoration detection method, system, and electronic equipment based on cross-domain representation modeling. Background Art

[0002] With the continuous advancement of image restoration technology, the latest AI tools can generate visually realistic image content with almost no visual artifacts. This poses a great challenge to image restoration detection research. Currently, many methods use noise domain and frequency domain features as auxiliary clues to further explore potential restoration traces, but there are two problems:

[0003] First, noise-domain and frequency-domain features are highly sensitive to post-processing operations. Images are often compressed, smoothed, and format-converted as they spread across the internet and social media, a significant challenge to existing research. The ability to construct robust restoration representations is crucial to addressing this issue.

[0004] Second, due to limitations in model structure and hardware resources, most current algorithms use a fixed input resolution strategy during training, scaling images. This operation severely destroys the restoration traces left in the original image, weakening the model's learning and discrimination capabilities. Summary of the Invention

[0005] In response to the above-mentioned problems in the prior art, the present invention provides an image restoration detection method, system, and electronic device based on cross-domain representation modeling. The present invention uses a noise information gating unit to integrate various auxiliary information, construct a feature representation of the restoration area in the noise domain, and reduce the impact of post-processing operations on model performance. In addition, a masked dual-stream encoder based on the ViT architecture is designed, which uses window attention to replace part of the global attention, and at the same time draws on multi-modal feature processing methods to deeply fuse RGB domain and noise domain features, providing more comprehensive positioning clues for the model to locate the restoration area. Finally, a simple feature pyramid is introduced to maintain the model's generalization to multi-scale targets and inputs of different resolutions.

[0006] The technical solutions adopted by the present invention to solve the above problems are as follows:

[0007] An image restoration detection method based on cross-domain representation modeling includes the following steps:

[0008] Step 1: Use the minimum padding strategy (MPS) to preprocess the image to retain the original information of the input image to the greatest extent;

[0009] Step 2: Extract three types of noise information from the image: Noiseprint++, SRM filtering, and Bayar convolution, and input them into the noise information gating unit (NGU) for splicing and fusion;

[0010] Step 3: Input the preprocessed image in step 1 and the noise information fused in step 2 into the masked dual-stream encoder (MDE) to generate a feature map;

[0011] Step 4: Input the feature map generated in step 3 into the simple feature pyramid (SFPN) to generate a multi-scale feature map. After fusing all features, the final prediction is output and the loss function is calculated to obtain the detection result.

[0012] Furthermore, in step 1, in order to preserve the original information of the input image to the greatest extent, the minimum padding strategy (MPS) performs zero padding below and to the right of the image and its true label, so that its resolution is aligned to the minimum integer multiple of the image block size in the masked dual-stream encoder, thereby adapting to input images of any size.

[0013] Furthermore, in step 2, the noise feature f np++ 、f bayar and f srm The outputs are concatenated in the channel dimension and then integrated through a continuous convolution block to obtain the robust noise feature f noise At the same time, in order to avoid the model from being overly dependent on a specific noise information, f noise The noise gating unit (NGU) integrates multiple effective features, including Noiseprint++, SRM filtering, and Bayar convolution, while suppressing redundant components in the noise to enhance the noise domain's ability to resist interference from image post-processing operations.

[0014] Noiseprint++ is a camera "fingerprint" extractor trained through self-supervision and contrastive learning loss. Compared to the original Noiseprint, the latter's modeling method further improves the network's ability to recognize image editing operations. The SRM filter is a fixed local non-semantic feature extractor that uses a combination of designed filters to detect potential local anomalies. Bayar convolution is a learnable module that is constrained to actively focus on repair traces during training.

[0015] Furthermore, in step three, the masked dual-stream encoder (MDE) is based on the ViT architecture, and takes the outputs of step one and step two as dual-stream inputs, respectively, to simultaneously process noise domain and RGB domain information, improve the modeling ability of image restoration traces, and adopt a masked window attention mechanism to replace part of the global attention. While reducing the computational cost, it takes into account the modeling of micro details and macro information, which helps to improve the representation ability of the model.

[0016] Since noise domain features carry rich low-level detail information, while RGB domain features contain more high-level abstract semantic expressions, the masked dual-stream encoder (MDE) also introduces a feature rectification module (FRM) and a feature fusion module (FFM) to achieve information alignment and deep fusion between noise domain and RGB domain features.

[0017] Furthermore, in step 4, in order to introduce multi-scale features, the simple feature pyramid (SFPN) is used to mask the feature map f output by the dual-stream encoder. encoder Based on the multi-scale feature map, downsampling and upsampling are used to generate the multi-scale feature map. At the same time, in order to keep the model as lightweight as possible, the present invention adopts an MLP-based decoder to fuse the multi-scale features and restore them to a high-resolution prediction mask.

[0018] In summary, the method of the present invention uses modules such as a noise information gating unit, a mask dual-stream encoder, and a simple feature pyramid. First, the image is processed with a minimum filling strategy to retain the original repair traces. Secondly, the noise information gating unit preprocesses each auxiliary information to construct a noise domain representation of the repaired area. At the same time, the present invention uses mask window attention to replace part of the global attention in ViT, and introduces a feature correction and fusion module to further integrate the repair features in the noise domain and RGB domain information. Finally, the fused features are sent to a simple pyramid structure, and the model maintains generalization for inputs of different resolutions through multi-scale learning. By studying the multi-scale information interaction mode and the cross-domain feature processing process, the detection algorithm's positioning accuracy for the repaired area is improved, while the robustness and generalization of the model are improved. This provides a more general technical solution for future exploration of the application of multimodal large language models and unsupervised learning in the field of image repair detection.

[0019] The present invention also discloses an image restoration detection system based on cross-domain representation modeling, which is used to execute the above method and includes the following modules:

[0020] Preprocessing module: preprocess the image using the minimum filling strategy;

[0021] Noise information fusion module: extracts three types of noise information from the image: Noiseprint++, SRM filtering, and Bayar convolution, and inputs them into the noise information gating unit for splicing and fusion to obtain fused noise information;

[0022] Feature map generation module: The preprocessed image and fused noise information are input into the mask dual-stream encoder to generate a feature map;

[0023] Repair module: The generated feature map is input into a simple feature pyramid to generate a multi-scale feature map, and then all features are fused to output the final prediction and calculate the loss function.

[0024] The present invention also discloses an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and running on the processor; the processor is used to read the program in the memory to implement the above method or system.

[0025] In summary, compared with the prior art, the present invention has the following advantages:

[0026] 1. Inpainting artifacts around suspicious areas are crucial clues for identifying inpainting operations. The integrity of these artifacts determines the accuracy of determining the inpainted area. However, existing techniques typically set a fixed input resolution and perform interpolation and scaling on the original image during preprocessing. This process blurs or distorts key artifacts, increasing the difficulty of forensic analysis. To address this, the present invention utilizes a minimum padding strategy to maximize the preservation of the original information in the input image.

[0027] 2. With the continuous advancement of deep inpainting technology, it is difficult to detect traces left by inpainting operations based solely on RGB domain information. Noise features can help reveal inpainting traces, but they are more sensitive to post-processing operations. If the image is blurred, noisy, or interpolated, the performance of the model will be degraded. To this end, the present invention selects three widely used and complementary methods as noise inputs, integrates multiple effective features through a noise information gating unit, and suppresses redundant components in the noise to enhance the noise domain feature's ability to resist interference from image post-processing operations.

[0028] 3. The ViT architecture performs block embedding on the input image, transforming it into sequential data. It then uses a self-attention mechanism to capture global relationships, adapting to images of varying resolutions. This characteristic is highly compatible with the minimum padding strategy. Based on ViT, this paper designs a dual-stream encoder that simultaneously processes information in both the noise and RGB domains, improving the ability to model image restoration artifacts.

[0029] 4. When the input image resolution is high, the computational overhead of global self-attention increases quadratically. This paper introduces a window attention mechanism based on the traditional ViT structure to replace part of the global attention. The traditional window attention mechanism requires zero padding when the number of image blocks is not an integer multiple of the window size. However, when the padding ratio is high, irrelevant information will interfere with the local feature modeling effect. This paper uses a masked window attention mechanism to accurately block the padding information within each window, completely eliminating the impact of the padding content on feature extraction.

[0030] 5. Existing inpainting techniques are mostly used to remove objects or watermarks from photo backgrounds. However, editors can now use generative AI tools to inpaint complex objects in images. Furthermore, the ratio of inpainted to real areas varies significantly across datasets. Therefore, this invention incorporates multi-scale information through a feature pyramid, improving the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of an image restoration detection method based on cross-domain representation modeling provided by a preferred embodiment of the present invention;

[0032] Figure 2 A schematic diagram of the structure of a masked dual-stream encoder provided by a preferred embodiment of the present invention;

[0033] Figure 3 A schematic diagram of a window attention mechanism mask strategy provided by a preferred embodiment of the present invention;

[0034] Figure 4 A schematic diagram of a decoder based on a multi-layer perceptron (MLP) according to a preferred embodiment of the present invention;

[0035] Figure 5 A flowchart of an image restoration detection method based on cross-domain representation modeling provided by a preferred embodiment of the present invention;

[0036] Figure 6 A visualization diagram provided for a preferred embodiment of the present invention;

[0037] Figure 7 A block diagram of an image restoration detection system based on cross-domain representation modeling provided by a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to better understand the technical solutions of the present invention, the technical solutions in the preferred embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0039] like Figure 1 、 5As shown, this embodiment provides an image restoration detection method based on cross-domain representation modeling, and the specific steps include:

[0040] Step 1: Preprocess the image using the minimum padding strategy. In this step, zero padding is performed below and to the right of the image and its ground-truth label to align its resolution to the minimum integer multiple of the image block size in the masked dual-stream encoder, thus adapting to input images of any size.

[0041] Step 2: Extract three types of noise information from the image: Noiseprint++, SRM filtering, and Bayar convolution. These are then fed into the noise information gating unit for splicing and fusion. This step extracts and integrates multiple effective features from the image: Noiseprint++, SRM filtering, and Bayar convolution. This also suppresses redundant noise components, enhancing the noise domain's ability to resist interference from image post-processing operations.

[0042] Step 3: Input the image preprocessed in step 1 and the noise information fused in step 2 into the masked dual-stream encoder to generate a feature map. In this step, the masked dual-stream encoder processes both the noise domain and the RGB domain information.

[0043] Step 4: Input the feature map generated in Step 3 into a simple feature pyramid to generate a multi-scale feature map. After fusing all features, the final prediction is output and the loss function is calculated. In this step, to introduce multi-scale features, the simple feature pyramid uses the feature map output by the masked dual-stream encoder as a basis, generating a multi-scale feature map through downsampling and upsampling. Inpainting artifacts often appear at the interface between the inpainted and true regions. A corresponding objective function is set to target this characteristic, encouraging the model to focus on relevant cues.

[0044] Figure 1 This is a flow chart of a preferred embodiment of an image restoration detection method based on cross-domain representation modeling. Figure 1 As shown in the lower left corner, in step one, the original content of the padded image is concentrated in the upper left area. This can reduce the model's reliance on prior knowledge in the pre-training data, learn more key low-level visual cues that meet the needs, and retain the original information of the input image to the greatest extent.

[0045] Since images in the same batch must maintain the same resolution during model training, the present invention also introduces a gradient accumulation strategy in deep learning, setting the batch size to 1 and waiting for the batch to accumulate to a fixed value before performing backpropagation, thereby improving the training stability and generalization of the model.

[0046] like Figure 1 As shown in the upper left, in step 2, the noise feature f np++ , f bayarand f srm The outputs are concatenated in the channel dimension and then integrated through a continuous convolution block to obtain the robust noise feature f noise .

[0047] In order to avoid the model from being overly dependent on a specific noise information, f noise Dropout processing is performed and the dropout rate is set to 0.33. That is, by randomly discarding the activation values of 1 / 3 of the neurons, the model is forced to rely on a different feature subset in each forward propagation, thereby enhancing the generalization of the model.

[0048] Figure 2 This is a schematic diagram of the masked dual-stream encoder structure. Figure 2 As shown in Figure 3, in step three, the masked two-stream encoder uses windowed attention to capture local detail features (such as texture and edges), while global attention is used to address overall information exchange. Because noise-domain features carry rich low-level details, while RGB-domain features contain more high-level abstract semantic expressions, the masked two-stream encoder also introduces a feature rectification module (FRM) and a feature fusion module (FFM) (both of which can adopt existing technologies) to achieve information alignment and deep fusion between noise-domain and RGB-domain features. The FRM generates weighted channel feature maps and spatial feature maps using a simple convolutional attention mechanism, and employs residual connections to achieve rectification in the channel and spatial dimensions to enhance representation diversity. The FFM consists of two stages: information exchange and feature fusion. First, a cross-attention mechanism is used to exchange information between noise-domain and RGB-domain features, generating two mixed features. These mixed features are then integrated into a single output through a series of 1×1 and 3×3 convolutional layers and fed into a multi-scale pyramid architecture.

[0049] Figure 3 It is a masking strategy for the window attention mechanism. Furthermore, the masked dual-stream encoder uses masked window attention instead of traditional window attention, which can accurately block the padding information within each window and completely eliminate the impact of padding on feature extraction.

[0050] The feature map output in step 3 is input into the feature pyramid in step 4, such as Figure 1 As shown in the upper right corner. Specifically, first use maximum pooling with a stride of 2 and a kernel size of 2×2 for downsampling to generate a 1 / 32 scale feature map; then, retain the original feature map as a 1 / 16 scale feature map; use a deconvolution operation with a stride of 2 and a kernel size of 2×2 for upsampling to generate a 1 / 8 scale feature map; finally, use two consecutive deconvolution operations with a stride of 2 and a kernel size of 2×2 for further upsampling to generate a 1 / 4 scale feature map. This process is shown in the following formula:

[0051]

[0052] Figure 4 This is a schematic diagram of an MLP-based decoder. In order to keep the model as lightweight as possible, the present invention uses an MLP-based decoder to fuse multi-scale features and restore them to high-resolution prediction masks. Figure 4 As shown in the figure, all features are first upsampled to the current maximum feature scale (1 / 4) through bilinear interpolation. All features are then concatenated along the channel dimension. Finally, a linear layer is used to fuse all features and output the final prediction. The prediction result is restored to the original image size through bilinear interpolation before being fed into the loss function. The model input shape is arbitrary, but it is reshaped to a fixed 256*256 shape during processing. It must be restored to its original shape before calculating the loss function, otherwise the effect will be poor. The loss function is used to calculate the error between the machine learning model output and the true value, and the detection result is obtained through the loss function calculation.

[0053] The total loss function consists of two parts: edge loss and regional loss, as shown in the following formula:

[0054] L=λ r L region (Pred,M r )+λ e L edge (Pred*M e ,M e )

[0055] Among them, L region and L edge Both are binary cross entropy losses, Pred is the predicted mask, M r and M e denote the true regional label and edge true label respectively, λ r and λ e It is a hyperparameter that controls the amplification of each loss term.

[0056] The image inpainting detection method based on cross-domain representation modeling provided by the present invention has been fully tested on the IID-inpainting dataset. Compared with existing methods, it has advantages as shown in the following table:

[0057]

[0058] The visualization results of the experiment are as follows Figure 6 shown.

[0059] like Figure 7 As shown, this embodiment discloses an image restoration detection system based on cross-domain representation modeling, which is used to execute the above method and includes the following modules:

[0060] Preprocessing module: preprocess the image using the minimum filling strategy;

[0061] Noise information fusion module: extracts three types of noise information from the image: Noiseprint++, SRM filtering, and Bayar convolution, and inputs them into the noise information gating unit for splicing and fusion to obtain fused noise information;

[0062] Feature map generation module: The preprocessed image and fused noise information are input into the mask dual-stream encoder to generate a feature map;

[0063] Repair module: The generated feature map is input into a simple feature pyramid to generate a multi-scale feature map, and then all features are fused to output the final prediction and calculate the loss function.

[0064] For other contents of this embodiment, please refer to the above method embodiment.

[0065] This embodiment discloses an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and running on the processor; characterized in that:

[0066] The processor is used to read the program in the memory to implement the above method or system.

[0067] In summary, the present invention addresses the problems of improper use of original image information and poor anti-interference ability of noise domain features in existing research, and proposes an image restoration detection method and system based on cross-domain representation modeling. The minimum filling strategy proposed in the present invention retains the most original restoration traces in the image, providing complete clues for feature modeling; the three original noise features are preprocessed by the noise information gating unit to suppress redundant noise sources, and the dropout operation is introduced to reduce the model's dependence on fixed noise features; drawing on the idea of multimodal feature fusion, the present invention introduces feature correction and feature fusion modules to promote information exchange and deep fusion between noise domain features and RGB domain features; finally, a simple feature pyramid is used to maintain the generalization of the model for input data of different resolutions. Compared with the existing technology, the present invention has stronger detection capabilities and robustness.

[0068] It should be understood that the embodiments described above are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

Claims

1. An image restoration detection method based on cross-domain representation modeling, characterized in that: The steps include: Step 1: Preprocess the image using the minimum filling strategy; Step 2: Extract three types of noise information from the image: Noiseprint++, SRM filtering, and Bayar convolution, input them into the noise information gating unit for splicing and fusion to obtain fused noise information; Step 3: Input the image preprocessed in step 1 and the noise information fused in step 2 into the mask dual-stream encoder to generate a feature map; Step 4: Input the feature map generated in step 3 into the simple feature pyramid to generate a multi-scale feature map, then fuse all features to output the final prediction and calculate the loss function.

2. The image restoration detection method based on cross-domain representation modeling according to claim 1 is characterized in that: In step 1, the minimum padding strategy performs zero padding below and to the right of the image and its true label to align the resolution to the minimum integer multiple of the image block size in the mask dual-stream encoder.

3. The image restoration detection method based on cross-domain representation modeling according to claim 1 is characterized in that: In step 2, the noise features f corresponding to Noiseprint++, SRM filtering and Bayar convolution are np++ 、f bayar and f srm The outputs are concatenated in the channel dimension and then integrated through a continuous convolution block to obtain the robust noise feature f noise , for f noise Perform dropout processing.

4. The image restoration detection method based on cross-domain representation modeling according to claim 1 is characterized in that: In step 3, the masked dual-stream encoder processes the noise domain and RGB domain information simultaneously.

5. The image restoration detection method based on cross-domain representation modeling according to claim 4 is characterized in that: In step 3, the masked two-stream encoder adopts masked window attention.

6. The image restoration detection method based on cross-domain representation modeling according to claim 4 or 5, characterized in that: In step 4, the feature map f output by the masked dual-stream encoder is encoder Based on the proposed method, multi-scale feature maps are generated by downsampling and upsampling, and an MLP-based decoder is used to fuse the multi-scale features and restore them to the prediction mask.

7. An image restoration detection system based on cross-domain representation modeling, used to execute the method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Preprocessing module: preprocess the image using the minimum filling strategy; Noise information fusion module: extracts three types of noise information from the image: Noiseprint++, SRM filtering, and Bayar convolution, and inputs them into the noise information gating unit for splicing and fusion to obtain fused noise information; Feature map generation module: The preprocessed image and fused noise information are input into the mask dual-stream encoder to generate a feature map; Repair module: The generated feature map is input into a simple feature pyramid to generate a multi-scale feature map, and then all features are fused to output the final prediction and calculate the loss function.

8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and running on the processor; characterized in that: The processor is configured to read a program in a memory to implement the method according to any one of claims 1 to 6 or to implement the system according to claim 7.

Images