A method and system for image inpainting detection based on high-resolution networks

Abstract

This invention discloses an image inpainting detection method and system based on a high-resolution network. The method includes the following steps: acquiring a training dataset generated by an image inpainting method, inputting it into a preset network, and outputting multiple feature maps; extracting and fusing features from the multiple feature maps to obtain a coarse-localized global prediction mask; iterating the coarse-localized global prediction mask according to a preset iteration path based on the multiple feature maps to obtain a final prediction mask; wherein each iteration generates a prediction mask, which is used as the input for the next iteration; performing pixel-level supervision on all prediction masks using binary cross-entropy loss, and performing edge supervision on the final prediction mask using dice loss, thereby training an image inpainting detection model; and performing image inpainting detection based on the image inpainting detection model to locate the inpainting region. This invention has advantages such as high detection accuracy.

Description

Technical Field

[0001] This invention mainly relates to the fields of digital image processing and information security technology, specifically to an image restoration and detection method and system based on high-resolution networks. Background Technology

[0002] Image inpainting is an image processing technique often used in artifact restoration, image editing, and other applications. Its core principle is to repair missing areas based on the similarity and continuity between pixels, generating a visually acceptable image. Since the advent of context encoders, deep learning-based image inpainting techniques have developed rapidly. However, the misuse of image inpainting tools can cause serious social harm. Therefore, developing forensic tools for image inpainting operations is essential in the field of information security.

[0003] Traditional image restoration and forensic methods primarily rely on manually created datasets, which are cumbersome and yield poor results. With the development of neural networks, several deep learning-based image restoration and forensic models have been proposed, such as HP-FCN, IID-Net, and PSCC-Net. In implementing this invention, the inventors of this patent application discovered that existing models suffer from at least the following technical problems:

[0004] Most existing image inpainting and forensic networks primarily use cascaded convolutional layers, undergoing multiple downsampling steps to obtain deep and highly abstract features. In this process, shallow features are often ignored. However, shallow features retain more image details and edge information, which is crucial for location-sensitive tasks. Furthermore, while these methods achieve high scores on AUC and F1-Score metrics, their visualization results still have flaws. More specifically, existing methods often struggle to accurately identify details, leading to blurred, coarse, and incomplete edges in the prediction mask.

[0005] Meanwhile, some forensic methods analyze restored images and employ data-driven approaches to build network architectures. This leads to overfitting of their models to specific datasets, making them unsuitable for effective training on other datasets. Therefore, this overfitting poses a significant obstacle to reproducible research. Summary of the Invention

[0006] To address the technical problems existing in the prior art, this invention provides an image inpainting detection method and system based on high-resolution networks that provides accurate detection.

[0007] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0008] An image inpainting detection method based on high-resolution networks includes the following steps:

[0009] Obtain the training dataset generated by the image inpainting method, input it into the preset network, and output multiple feature maps;

[0010] Feature extraction and fusion are performed on multiple feature maps to obtain a coarse-localized global prediction mask;

[0011] Based on a preset iteration path of multiple feature maps, the coarsely localized global prediction mask is iterated according to the preset iteration path to obtain the final prediction mask; each iteration generates a prediction mask, which is used as the input for the next iteration.

[0012] A pixel-level supervision model is trained by using binary cross-entropy loss on all predicted masks and edge supervision on the final predicted mask.

[0013] The image restoration detection model is used to detect and restore images and locate restoration areas.

[0014] Preferably, a multi-scale convolution extraction module is used to extract and fuse features from multiple feature maps; wherein the multi-scale convolution extraction module includes a feature fusion stage and a multi-scale feature extraction stage;

[0015] In the feature fusion stage, multiple feature maps are upsampled to the same resolution and channel dimension, and then the multiple transformed feature maps are merged.

[0016] In the multi-scale feature extraction stage, further feature extraction is performed on the merged features; this includes multiple branches, and each branch has two network layers; the first layer is a convolutional layer, and the second layer is a dilated convolutional layer; finally, the results of multiple branches are concatenated together and fed into a single convolutional layer to generate a single-channel initial prediction mask, i.e., a coarse-localized global prediction mask.

[0017] Preferably, the first convolutional layer has a kernel size of (2b-1)×(2b-1); the second dilated convolutional layer has a kernel size of 3×3 and a dilation rate of 2×(b-1).

[0018] Preferably, a reverse attention module is introduced on a preset iteration path to iterate the global prediction mask for coarse localization.

[0019] Preferably, the reverse attention weight A of the reverse attention module i The calculation is as follows:

[0020]

[0021] Where σ is the Sigmoid function, E is a matrix with all elements equal to 1, and M... i+1It is the prediction mask from the previous iteration.

[0022] Preferably, the prediction mask M generated in each iteration i for:

[0023]

[0024] Where ⊕ represents bitwise summation; R i The reverse attention feature is as follows:

[0025] R i =F i *A i

[0026] Where F i This is a feature map.

[0027] Preferably, the binary cross-entropy loss L bce The calculation formula is:

[0028]

[0029] The loss for pixel-level supervision is as follows:

[0030]

[0031] Where G is the real mask, and M is the real mask. g It is a global prediction mask.

[0032] Preferably, the die loss L dice The calculation formula is:

[0033]

[0034] G edge M edge represent the edge masks obtained by morphological gradient calculation of the real mask G and the predicted mask M, respectively;

[0035] Edge supervision is performed only on the final predicted mask, and its loss is denoted as: L e =L dice (G·G edge ,M1·M edge ).

[0036] Preferably, the edge mask M is obtained through morphological gradient. e :

[0037]

[0038] Where B(k) represents a cross matrix of size (2k+1)×(2k+1), where only the elements in the k-th row and k-th column are 1, and the rest are 0; Indicates expansion. This indicates corrosion, and k is set to 1.

[0039] The present invention also discloses an image inpainting detection system based on a high-resolution network, comprising an interconnected memory and a processor, wherein the memory stores a computer program, and the computer program executes the steps of the method described above when run by the processor.

[0040] Compared with the prior art, the advantages of the present invention are as follows:

[0041] This invention employs an encoder-decoder architecture, using a high-resolution network as the encoder backbone to exchange information between feature maps at different levels, enabling sufficient information exchange between features of different scales and fully utilizing shallow feature information. Simultaneously, a multi-scale feature extraction module based on dilated convolution is designed to expand the model's receptive field without downsampling or increasing computational complexity, outputting a coarse-localized global prediction mask. In the decoder, this invention combines a reverse attention module to progressively iterate and refine the global prediction mask, finally outputting the final prediction mask. An edge supervision strategy is used to make the model pay more attention to the boundaries between repaired and unrepaired regions in the image, avoiding blurring of the prediction mask boundaries.

[0042] This invention utilizes a neural network model for forensic analysis of image restoration operations, achieving pixel-level detection and localization. Compared to similar methods in existing technologies, it achieves higher evaluations in AUC and F1-scores, and its visualization results are clearer. Furthermore, this invention demonstrates significant performance on both self-synthesized datasets and publicly available datasets. Attached Figure Description

[0043] Figure 1 This is a block diagram of the image inpainting detection based on a high-resolution network according to the present invention.

[0044] Figure 2 This is a framework diagram of the multi-scale extraction module based on dilated convolution in this invention.

[0045] Figure 3 The flowchart of the image inpainting detection method based on high-resolution networks of the present invention is shown in an embodiment. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0047] like Figure 1 and Figure 3 As shown, the image inpainting detection method based on high-resolution networks provided in this embodiment of the invention includes the following steps:

[0048] Step 1: Prepare the training dataset, such as obtaining publicly available image restoration datasets from the Internet; to ensure dataset diversity, a dataset containing five image restoration methods, namely NS, DeepFillV2, LaMa, RFR, and MISF, was synthesized.

[0049] Step 2: Determine the training details, including the machine used for training, batch size, learning rate, etc.

[0050] Step 3: The input batch of repaired images first enters the backbone of the network architecture, which is a lightweight variant of the high-resolution network (HR-Net) (HRNetV2p-W18), and outputs four feature maps F1, F2, F3, and F4 with different resolutions and different channel dimensions.

[0051] Step 4: Receive the four feature maps obtained in Step 3 as input to the multi-scale convolution extraction module. Through feature extraction and fusion, output a coarse-localized global prediction mask.

[0052] Step 5: Based on the preset iteration path of multiple feature maps, the coarsely localized global prediction mask is iterated and refined using a progressive inverse attention module according to the preset iteration path to obtain the final prediction mask; wherein each iteration generates a prediction mask, which is used as the input for the next iteration.

[0053] Step 6: Apply binary cross-entropy loss to all the predicted masks generated during the iteration process for pixel-level supervision, and apply dice loss to the final predicted mask for edge supervision, thereby training the image inpainting detection model.

[0054] Step 7: Detect the repaired image based on the image repair detection model and output the final prediction mask; the final prediction mask is a grayscale image, where white represents the area in the original image that has been processed by the repair operation.

[0055] like Figure 2 As shown, in step 4, the multi-scale convolution extraction module includes a feature fusion stage and a multi-scale feature extraction stage. It performs feature fusion and multi-scale feature extraction on the four feature maps generated by the high-resolution network and outputs a coarse-localized global prediction mask.

[0056] Specifically, in the feature fusion stage, the four feature maps F are... i Convert to the same resolution and channel dimension through upsampling:

[0057] F′ i =up(F i )

[0058] Then the four transformed features F′ i Merge:

[0059] F m =concat(F′1,F′2,F′3,F′4)

[0060] In the multi-scale feature extraction stage, the merged features F m Further feature extraction is performed: it contains four branches {b=1,…,4}, and each branch has two network layers; the first layer is a convolutional layer with a kernel size of (2b-1)×(2b-1); the second layer is a dilated convolutional layer with a kernel size of 3×3 and a dilation rate of 2×(b-1).

[0061] Finally, the results from the four branches are concatenated and fed into a 3×3 convolutional layer to generate a single-channel initial prediction mask, i.e., the global prediction mask M. g .

[0062] like Figure 1 As shown, in step 5, a bottom-up progressive iterative path (F4, F3, F2, F1) is designed based on the feature map. An inverse attention module (RA) is introduced along this path to iterate and refine the global prediction mask multiple times.

[0063] The iterative path incorporates the feature map F output by the high-resolution network. i and the reverse attention module RA;

[0064] The inverse attention module is typically used in object detection tasks, with inverse attention weights A. i The calculation is as follows:

[0065]

[0066] Where σ is the Sigmoid function, E is a matrix with all elements equal to 1, and M... i+1 It is the prediction result of the previous iteration; when i=4, the prediction mask of the previous iteration is M. g .

[0067] Reverse attention feature R i It is simply the reverse attention weight multiplied by the feature map, i.e.:

[0068] R i =F i *A i

[0069] Ultimately, each iteration outputs a prediction mask:

[0070]

[0071] in This indicates bitwise summation;

[0072] like Figure 1 M1 in the formula is the final prediction mask.

[0073] In step 6, to improve the model's prediction performance, all prediction masks M generated during the iteration process are processed. i {i=1,…,4} uses binary cross-entropy loss L bce Pixel-level supervision is performed, and a dice loss L is applied to the final predicted mask. dice Perform edge monitoring.

[0074] Specifically, pixel-level supervision is applied to all predicted masks M. i {i=1,…,4} uses binary cross-entropy loss L bce Its expression is as follows:

[0075]

[0076] Where H represents the image height, W represents the image width, G is the ground truth mask, and M is the predicted mask; the loss for pixel-level image supervision and L... p as follows:

[0077]

[0078] Where G is the real mask, and M is the real mask. g It is a global prediction mask.

[0079] Specifically, edge supervision is a special pixel-level supervision method used to encourage the model to pay more attention to the edges of the repaired area in the image, thereby obtaining clearer prediction results.

[0080] Because the positive and negative classes in the edge mask are severely imbalanced, this invention employs dice loss L. dice With edge monitoring, the dice loss is as follows:

[0081]

[0082] G edge M edge Let G and M represent the edge masks obtained by morphological gradient calculations, respectively, from the ground mask G and the predicted mask M. Specifically, the edge mask M is obtained by morphological gradient calculation from the predicted mask M. edge The formula is:

[0083]

[0084] Where M is the prediction mask, and B(k) represents a cross matrix of size (2k+1)×(2k+1), where only the elements in the k-th row and k-th column are 1, and the rest are 0; Indicates expansion. This indicates corrosion, and k is set to 1.

[0085] G edge The calculation process and M edge The calculation process is the same, and will not be repeated here.

[0086] This invention only performs edge supervision on the final predicted mask, and its loss L e Notation:

[0087] L e =L dice (G·G edge ,M1·M edge )

[0088] Ultimately, all the losses and Notation:

[0089]

[0090] Where λ is a hyperparameter, set to 10.

[0091] This invention uses an encoder-decoder architecture, employing a high-resolution network as the encoder backbone network to exchange information between feature maps at different levels, enabling full information exchange between features of different scales and making full use of shallow feature information.

[0092] Meanwhile, a multi-scale feature extraction module based on dilated convolution is designed to expand the receptive field of the model and output a coarse-localized global prediction mask without downsampling or increasing computational complexity.

[0093] In the decoder, this invention combines a reverse attention module to progressively iterate and refine the global prediction mask, and finally outputs the final prediction mask.

[0094] By employing an edge supervision strategy, the model focuses more on the boundary between repaired and unrepaired regions in the image, thus avoiding blurry prediction mask boundaries.

[0095] This invention utilizes a neural network model for forensic analysis of image restoration operations, achieving pixel-level detection and localization. Compared to similar methods in existing technologies, it achieves higher evaluations in AUC and F1-score, and its visualization results are clearer. AUC refers to the area under the Receiver Operating Characteristic (ROC) curve, a two-dimensional graph where the horizontal axis represents the False Positive Rate (FPR) and the vertical axis represents the True Positive Rate (TPR). The F1 score is the harmonic mean of precision and recall, where precision measures the proportion of samples predicted as positive by the model that are actually positive, and recall measures the proportion of all actual positive samples correctly predicted as positive by the model. This invention has achieved significant results on both self-synthesized datasets and publicly available datasets.

[0096] This invention also discloses an image inpainting detection system based on a high-resolution network, comprising an interconnected memory and a processor. The memory stores a computer program, which, when run by the processor, executes the steps of the method described above. The detection system of this invention corresponds to the detection method described above and also possesses the advantages described therein.

[0097] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium includes: any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0098] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An image inpainting detection method based on high-resolution networks, characterized in that, Including the following steps: The training dataset generated by the image inpainting method is obtained, input into a high-resolution network, and multiple feature maps are output. ,in ; The multi-scale convolution extraction module extracts and fuses features from multiple feature maps; the multi-scale convolution extraction module includes a feature fusion stage and a multi-scale feature extraction stage. In the feature fusion stage, multiple feature maps are upsampled to the same resolution and channel dimension, and then the multiple transformed feature maps are merged. In the multi-scale feature extraction stage, further feature extraction is performed on the merged features; It contains multiple branches, and each branch has two network layers; The first layer is a convolutional layer, and the second layer is a dilated convolutional layer. Finally, the results of multiple branches are concatenated and fed into a single convolutional layer to generate a single-channel initial prediction mask, resulting in a coarse-localized global prediction mask. ; Based on multiple feature maps, a preset iteration path is used to iterate the coarsely localized global prediction mask according to the preset iteration path to obtain the final prediction mask; Each iteration generates a prediction mask, which serves as the input for the next iteration. The preset iteration path is a bottom-up, progressive iteration path. A reverse attention module is introduced along this path to iterate over the coarse-localized global prediction mask; when... At that time, the prediction mask generated in each iteration for: in This indicates bitwise summation; It is the prediction mask from the previous iteration; The reverse attention feature is as follows: in For feature maps; These are the reverse attention weights for the reverse attention module; when At that time, the prediction mask of the previous iteration was ; A pixel-level supervision model is trained by using binary cross-entropy loss on all predicted masks and edge supervision on the final predicted mask. The image restoration detection model is used to perform restoration detection on the image under test and locate the restoration area.

2. The image inpainting detection method based on high-resolution networks according to claim 1, characterized in that, The first convolutional layer has a kernel size of... ; The second layer is a dilated convolutional layer with a kernel size of . The expansion rate is .

3. The image inpainting detection method based on high-resolution networks according to claim 1, characterized in that, Reverse attention weights of the reverse attention module The calculation is as follows: in It is the Sigmoid function. It is a matrix whose elements are all 1s.

4. The image inpainting detection method based on high-resolution networks according to claim 1, 2, or 3, characterized in that, Binary cross-entropy loss The calculation formula is: in Represents the height of the image. Represents the width of the image. It is a real mask. It is a predictive mask; The loss for pixel-level supervision is as follows: in It is a real mask. It is a global prediction mask.

5. The image inpainting detection method based on high-resolution networks according to claim 4, characterized in that, Dice loss The calculation formula is: , They represent the actual mask respectively. Predictive mask The edge mask is obtained by morphological gradient calculation; Edge supervision is performed only on the final predicted mask, and the loss is denoted as: .

6. The image inpainting detection method based on high-resolution networks according to claim 5, characterized in that, Edge mask obtained through morphological gradient : in Represents a size of The cross matrix, where only the first... row and number The column contains only 1s and all other elements are 0s. Indicates expansion. Indicates corrosion. It is set to 1.

7. An image inpainting and detection system based on a high-resolution network, comprising an interconnected memory and a processor, wherein the memory stores a computer program, characterized in that, The computer program, when run by a processor, performs the steps of the method as described in any one of claims 1-6.

Images