Image restoration detection method and system based on high-resolution network

Abstract

The invention discloses an image restoration detection method and system based on a high-resolution network, and the method comprises the steps: obtaining a training data set generated through an image restoration method, inputting the training data set into a preset network, and outputting a plurality of feature maps; performing feature extraction and fusion on the plurality of feature maps to obtain a roughly positioned global prediction mask; presetting an iteration path according to the plurality of feature maps, and carrying out iteration on the roughly positioned global prediction mask according to the preset iteration path to obtain a final prediction mask; wherein a prediction mask is generated in each iteration and is used as the input of the next iteration; performing pixel-level supervision on all the prediction masks by adopting binary cross entropy loss, performing edge supervision on the final prediction mask by adopting dice loss, and performing training to obtain an image restoration detection model; and performing repair detection on the image based on the image repair detection model, and positioning a repair area. The method has the advantages of accurate detection and the like.

Description

Technical Field

[0001] The present invention mainly relates to the field of digital image processing and information security technology, and in particular to an image restoration detection method and system based on a high-resolution network. Background Art

[0002] Image inpainting is an image processing operation that is often used in cultural relic restoration, image editing, and so on. Its core principle is to repair the missing area based on the similarity and continuity between pixels to generate a visually acceptable image. Since the emergence of context encoders, image inpainting technology based on deep learning has developed rapidly. However, the abuse of image inpainting tools may cause serious social harm. Therefore, it is very necessary to develop forensic tools for image inpainting operations in the field of information security.

[0003] Traditional restoration forensics methods are mainly based on hand-made data sets, which are cumbersome and have poor forensic results. With the development of neural networks, some deep learning-based image restoration forensics models have been proposed, such as HP-FCN, IID-Net, PSCC-Net, etc. In the process of implementing the present invention, the inventor of this patent application found that the existing models have at least the following technical problems:

[0004] Most of the existing image restoration forensics networks use concatenated convolutional layers as the main body. After multiple downsampling, the deep and highly abstract features of the network are obtained. In this process, the shallow features of the network are often ignored. However, shallow features retain more image details and edge information, which is very important for position-sensitive tasks. In addition, although these methods have obtained high scores in the two evaluation indicators of AUC and F1-Score, their visualization results are still flawed. More specifically, existing methods are usually difficult to accurately identify details, resulting in blurred, rough and incomplete edges of the predicted masks.

[0005] Meanwhile, some forensic methods analyze repaired images and adopt a data-driven approach to build network architectures. This causes their models to overfit to specific datasets, making them ineffective for training on other datasets. As a result, this overfitting poses a significant obstacle to reproducible research. Summary of the invention

[0006] In view of the technical problems existing in the prior art, the present invention provides an accurate detection method and system for image restoration detection based on a high-resolution network.

[0007] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0008] A high-resolution network-based image restoration detection method comprises the following steps:

[0009] Obtain a training data set generated by an image restoration method, input it into a preset network, and output multiple feature maps;

[0010] Extract and fuse multiple feature maps to obtain a roughly positioned global prediction mask;

[0011] According to the preset iteration paths of multiple feature maps, the roughly positioned global prediction mask is iterated according to the preset iteration paths to obtain the final prediction mask; each iteration generates a prediction mask and serves as the input of the next iteration;

[0012] Binary cross entropy loss is used for pixel-level supervision of all predicted masks, and dice loss is used for edge supervision of the final predicted mask to train the image inpainting detection model.

[0013] Perform image repair detection based on the image repair detection model to locate the repair area.

[0014] Preferably, feature extraction and fusion are performed on multiple feature maps through a multi-scale convolution extraction module; wherein the multi-scale convolution extraction module includes a feature fusion stage and a multi-scale feature extraction stage;

[0015] In the feature fusion stage, multiple feature maps are converted to the same resolution and channel dimension through upsampling operations, and then the multiple converted feature maps are merged;

[0016] In the multi-scale feature extraction stage, the merged features are further extracted; it contains multiple branches, and each branch has two network layers; the first layer is a convolutional layer, and the second layer is an expanded convolutional layer; finally, the results of multiple branches are connected together and input into a convolutional layer to generate a single-channel initial prediction mask, that is, a coarsely positioned global prediction mask.

[0017] Preferably, the convolution layer of the first layer has a kernel size of (2b-1)×(2b-1); the expansion convolution layer of the second layer has a kernel size of 3×3 and a dilation rate of 2×(b-1).

[0018] Preferably, a reverse attention module is introduced on a preset iteration path to iterate the coarsely positioned global prediction mask.

[0019] Preferably, the reverse attention weight A of the reverse attention module i The calculation of is as follows:

[0020]

[0021] Where σ is the Sigmoid function, E is a matrix with all elements set to 1, and M i+1is the prediction mask of the previous iteration.

[0022] Preferably, the prediction mask M generated in each iteration is i for:

[0023]

[0024] Where ⊕ represents bitwise sum; R i It is the reverse attention feature, specifically:

[0025] R i =F i *A i

[0026] Among them, F i is the feature graph.

[0027] Preferably, the binary cross entropy loss L bce The calculation formula is:

[0028]

[0029] The loss sum for pixel-level supervision is as follows:

[0030]

[0031] Where G is the real mask, M g is the global prediction mask.

[0032] Preferably, the dice loss L dice The calculation formula is:

[0033]

[0034] G edge 、M edge They represent the edge masks obtained by calculating the real mask G and the predicted mask M through morphological gradient respectively;

[0035] Only edge supervision is performed on the final predicted mask, and the loss is recorded as: L e =L dice (G.G edge ,M1·M edge ).

[0036] Preferably, the edge mask M is obtained by morphological gradient e :

[0037]

[0038] Where B(k) represents a cross matrix of size (2k+1)×(2k+1), in which only the elements in the k-th row and k-th column are 1, and the rest are 0; Indicates expansion, represents corrosion, k is set to 1.

[0039] The present invention also discloses an image restoration detection system based on a high-resolution network, comprising a memory and a processor connected to each other, wherein a computer program is stored on the memory, and when the computer program is run by the processor, the steps of the method described above are executed.

[0040] Compared with the prior art, the advantages of the present invention are:

[0041] The present invention uses an encoder-decoder architecture and a high-resolution network as the encoder backbone network to exchange information between feature maps at different levels, so that features of different scales can fully exchange information and make full use of shallow feature information. At the same time, a multi-scale feature extraction module based on dilated convolution is designed to expand the receptive field of the model without downsampling and increasing the computational complexity, and output a roughly positioned global prediction mask. In the decoder, the present invention combines a reverse attention module to progressively iterate and refine the global prediction mask, and finally outputs the final prediction mask. Through the edge supervision strategy, the model pays more attention to the boundary between the repaired area and the unrepaired area in the image, avoiding blurred boundaries of the prediction mask.

[0042] The present invention uses a neural network model to collect evidence of image restoration operations, and can achieve pixel-level detection and positioning. Compared with similar methods in the prior art, it has obtained higher evaluations in AUC and F1-scores, and its visualization results are clearer; and the present invention has achieved significant results on both self-synthesized datasets and public datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a block diagram of the image restoration detection based on high-resolution network of the present invention.

[0044] Figure 2 This is a framework diagram of the multi-scale extraction module based on dilated convolution in the present invention.

[0045] Figure 3 A flowchart of an embodiment of the high-resolution network-based image restoration detection method of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0047] like Figure 1 and Figure 3 As shown, the image restoration detection method based on high-resolution network provided by the embodiment of the present invention includes the following steps:

[0048] Step 1: prepare a training dataset, such as obtaining a public image restoration dataset on the Internet. In order to increase the diversity of the dataset, a dataset containing five image restoration methods, namely NS, DeepFillV2, LaMa, RFR, and MISF, is synthesized.

[0049] Step 2: Determine the details of the training, including the machine, batch, learning rate, etc.

[0050] Step 3: The input restoration image batch first enters the lightweight variant (HRNetV2p-W18) of the high-resolution network (HR-Net) as the backbone of the network architecture, and outputs four feature maps F1, F2, F3, and F4 with different resolutions and channel dimensions;

[0051] Step 4, receiving the four feature maps obtained in step 3 as the input of the multi-scale convolution extraction module, and outputting a roughly positioned global prediction mask through feature extraction and fusion;

[0052] Step 5: according to the preset iteration path of multiple feature maps, the roughly located global prediction mask is iterated and refined by a progressive reverse attention module according to the preset iteration path to obtain the final prediction mask; each iteration generates a prediction mask, which is used as the input of the next iteration;

[0053] Step 6: Binary cross entropy loss is used to perform pixel-level supervision on all predicted masks generated during the iteration process, and dice loss is used to perform edge supervision on the final predicted mask, thereby training an image restoration detection model.

[0054] Step 7: Detect the repaired image based on the image repair detection model and output the final prediction mask; the final prediction mask is a grayscale image, in which white represents the area in the original image that has been processed by the repair operation.

[0055] like Figure 2 As shown in FIG. 4 , in step 4, the multi-scale convolution extraction module includes a feature fusion stage and a multi-scale feature extraction stage, which performs feature fusion and multi-scale feature extraction on the four feature maps generated by the high-resolution network, and outputs a coarsely positioned global prediction mask.

[0056] Specifically, in the feature fusion stage, the four feature maps F i Convert to the same resolution and channel dimensions by upsampling:

[0057] F′ i = up(F i )

[0058] Then the four transformed features F′ i To perform the merge:

[0059] F m =concat(F′1,F′2,F′3,F′4)

[0060] In the multi-scale feature extraction stage, the merged feature F m Further feature extraction: It contains four branches {b=1,…,4}, and each branch has two network layers; the first layer is a convolutional layer with a kernel size of (2b-1)×(2b-1); the second layer is an expanded convolutional layer with a kernel size of 3×3 and an expansion rate of 2×(b-1);

[0061] Finally, the results of the four branches are concatenated and input into a 3×3 convolutional layer to generate a single-channel initial prediction mask, namely the global prediction mask M g .

[0062] like Figure 1 As shown, in step 5, a bottom-up progressive iterative path (F4, F3, F2, F1) is designed according to the feature map, and a reverse attention module (RA) is introduced on the path to perform multiple iterations and refinements on the global prediction mask.

[0063] The iterative path combines the feature map F output by the high-resolution network i and reverse attention module RA;

[0064] The reverse attention module is usually used in target detection tasks, and the reverse attention weight A i The calculation of is as follows:

[0065]

[0066] Where σ is the Sigmoid function, E is a matrix with all elements set to 1, and M i+1 is the prediction result of the previous iteration; when i=4, the prediction mask of the previous iteration is M g .

[0067] Reverse attention feature R i It is the reverse attention weight multiplied by the feature map, that is:

[0068] R i =F i *A i

[0069] Ultimately, each iteration outputs a predicted mask:

[0070]

[0071] in Indicates bitwise sum;

[0072] like Figure 1 The M1 in is the final prediction mask.

[0073] In step 6, in order to make the model obtain better prediction results, all prediction masks M generated in the iterative process are i {i=1,…,4} uses binary cross entropy loss L bce Perform pixel-level supervision and use dice loss L for the final predicted mask dice Conduct marginal supervision.

[0074] Specifically, pixel-level supervision is performed on all predicted masks M i {i=1,…,4} uses binary cross entropy loss L bce , which is expressed as follows:

[0075]

[0076] Where H represents the height of the image, W represents the width of the image, G is the real mask, and M is the predicted mask; the loss of image pixel-level supervision and L p as follows:

[0077]

[0078] Where G is the real mask, M g is the global prediction mask.

[0079] Specifically, edge supervision is also a special pixel-level supervision method that is used to encourage the model to pay more attention to the edges of the repaired areas in the image, thereby obtaining clearer prediction results.

[0080] Since the positive and negative classes in the edge mask are seriously unbalanced, the present invention adopts the dice loss L dice With marginal supervision, the dice loss is as follows:

[0081]

[0082] G edge 、M edge They represent the edge masks obtained by calculating the morphological gradient of the real mask G and the predicted mask M. Specifically, the edge mask M is obtained by calculating the morphological gradient of the predicted mask M. edge The formula is:

[0083]

[0084] Where M is the prediction mask, B(k) represents a cross matrix of size (2k+1)×(2k+1), where only the elements in the k-th row and k-th column are 1, and the rest are 0; Indicates expansion, represents corrosion, k is set to 1.

[0085] G edge The calculation process of M edge The calculation process is the same and will not be repeated here.

[0086] The present invention only performs edge supervision on the final prediction mask, and its loss L e Denoted as:

[0087] L e =L dice (G.G edge ,M1·M edge )

[0088] Ultimately, all losses and Denoted as:

[0089]

[0090] where λ is a hyperparameter and is set to 10.

[0091] The present invention uses an encoder-decoder architecture and a high-resolution network as the encoder backbone network to exchange information between feature maps at different levels, so that features of different scales can be fully exchanged and shallow feature information can be fully utilized.

[0092] At the same time, a multi-scale feature extraction module based on dilated convolution is designed to expand the receptive field of the model and output a coarsely positioned global prediction mask without downsampling or increasing the computational complexity.

[0093] In the decoder, the present invention combines the reverse attention module to progressively iterate and refine the global prediction mask, and finally outputs the final prediction mask;

[0094] Through the edge supervision strategy, the model pays more attention to the boundary between the repaired area and the unrepaired area in the image, avoiding the fuzzy prediction of the mask boundary.

[0095] The present invention collects evidence of image restoration operations through a neural network model, and can achieve pixel-level detection and positioning. Compared with similar methods in the prior art, it has obtained higher evaluations in AUC and F1-scores, and its visualization results are clearer. Wherein AUC refers to the area under the receiver operating characteristic curve (Receiver Operating Characteristic curve, referred to as ROC curve), where the ROC curve is a two-dimensional graph, the horizontal axis is the false positive rate (False Positive Rate, FPR), and the vertical axis is the true positive rate (True Positive Rate, TPR); the F1 score is the harmonic mean of the precision rate and the recall rate, where the precision rate measures the proportion of samples predicted by the model to be positive that are actually positive, and the recall rate measures the proportion of all actual positive samples that are correctly predicted by the model to be positive. The present invention has achieved remarkable results on both self-synthesized data sets and public data sets.

[0096] The present invention also discloses a high-resolution network-based image restoration detection system, comprising a memory and a processor connected to each other, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the above method are executed. The detection system of the present invention corresponds to the above detection method and also has the advantages described in the above method.

[0097] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. The memory is used to store computer programs and / or modules, and the processor implements various functions by running or executing computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, an internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0098] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A high-resolution network-based image restoration detection method, characterized in that: Includes steps: Obtain a training data set generated by an image restoration method, input it into a preset network, and output multiple feature maps; Extract and fuse multiple feature maps to obtain a roughly positioned global prediction mask; According to the preset iteration paths of the multiple feature maps, the roughly positioned global prediction mask is iterated according to the preset iteration paths to obtain the final prediction mask; Each iteration generates a prediction mask, which serves as the input for the next iteration. Binary cross entropy loss is used for pixel-level supervision of all predicted masks, and dice loss is used for edge supervision of the final predicted mask to train the image inpainting detection model. Based on the image restoration detection model, the image to be tested is restored and the restoration area is located.

2. The image restoration detection method based on high-resolution network according to claim 1 is characterized in that: The multi-scale convolution extraction module is used to extract and fuse multiple feature maps; wherein the multi-scale convolution extraction module includes a feature fusion stage and a multi-scale feature extraction stage; In the feature fusion stage, multiple feature maps are converted to the same resolution and channel dimension through upsampling operations, and then the multiple converted feature maps are merged; In the multi-scale feature extraction stage, further feature extraction is performed on the merged features; It contains multiple branches, and each branch has two network layers; The first layer is a convolutional layer, and the second layer is an expanded convolutional layer; finally, the results of multiple branches are connected together and input into a convolutional layer to generate a single-channel initial prediction mask, that is, a coarsely positioned global prediction mask.

3. The high-resolution network-based image restoration detection method according to claim 2, characterized in that: The first convolution layer has a kernel size of (2b-1)×(2b-1); the second dilated convolution layer has a kernel size of 3×3 and a dilation rate of 2×(b-1).

4. The high-resolution network-based image restoration detection method according to claim 1, 2 or 3, characterized in that: A reverse attention module is introduced on the preset iteration path to iterate the coarsely positioned global prediction mask.

5. The high-resolution network-based image restoration detection method according to claim 4, characterized in that: The reverse attention weight A of the reverse attention module i The calculation of is as follows: Where σ is the Sigmoid function, E is a matrix with all elements set to 1, and M i+1 is the prediction mask of the previous iteration.

6. The high-resolution network-based image restoration detection method according to claim 5, characterized in that: The prediction mask M generated at each iteration i for: in Indicates bitwise sum; R i It is the reverse attention feature, specifically: R i =F i *A i Among them, F i is the feature graph.

7. The high-resolution network-based image restoration detection method according to claim 1, 2 or 3, characterized in that: Binary cross entropy loss L bce The calculation formula is: Where H represents the height of the image, W represents the width of the image, G is the real mask, and M is the predicted mask; The loss sum for pixel-level supervision is as follows: Where G is the real mask, M g is the global prediction mask.

8. The high-resolution network-based image restoration detection method according to claim 7, characterized in that: Dice loss L dice The calculation formula is: G edge 、M edge They represent the edge masks obtained by calculating the real mask G and the predicted mask M through morphological gradient respectively; Only edge supervision is performed on the final predicted mask, and the loss is recorded as: L e =L dice (G.G edge ,M1·M edge ).

9. The high-resolution network-based image restoration detection method according to claim 8, characterized in that: Obtain edge mask M through morphological gradient edge : Where B(k) represents a cross matrix of size (2k+1)×(2k+1), in which only the elements in the k-th row and k-th column are 1, and the rest are 0; Indicates expansion, represents corrosion, k is set to 1.

10. A high-resolution network-based image restoration detection system, comprising a memory and a processor connected to each other, wherein a computer program is stored in the memory, characterized in that: When the computer program is executed by a processor, the computer program performs the steps of the method according to any one of claims 1 to 9.

Images