A deep image restoration forensics method based on multi-model fusion

Abstract

This invention discloses an image deep inpainting forensics method based on multi-model fusion. The method includes: S1, inputting an image into a trained forensics network; S2, obtaining the forensics result from the forensics network; the forensics result represents the tampered area of ​​the image; the forensics network includes several encoder-decoder structure network branches, a detail enhancement module, and an attention fusion module; the detail enhancement module is connected to the end of each network branch; the attention fusion module is connected to the end of the encoder in each network branch. This method can perform forensics on images with different JPEG compression levels, and the obtained forensics results have good robustness.

Description

Technical Field

[0001] This invention relates to the field of image processing and pattern recognition technology, specifically to an image deep restoration and forensics method based on multi-model fusion. Background Technology

[0002] With the rapid development of internet technology and the widespread adoption of smart devices, digital images, as a primary carrier of visual information, have become a crucial medium for information transmission and are widely used in various industries and fields of social life. However, in the digital media era, the ease of use of image processing and editing software has made editing and tampering with images convenient and simple; simultaneously, the forged images produced are difficult to identify, leading to a surge in image tampering and forgery incidents. Image inpainting is a technique used in image editing and tampering to repair damaged areas. Mature deep image inpainting algorithms are widely used in various commercial software, making them vulnerable to malicious exploitation. Furthermore, images altered using deep inpainting algorithms leave very few visual clues, making them even more difficult to identify. This poses a significant threat to network information security and has serious negative impacts on society. Therefore, research into this technology has significant practical implications.

[0003] Because traditional image restoration algorithms and depth image restoration differ significantly in their operational processes, previous forensic studies based on manually designed features are not applicable to more advanced depth image restoration algorithms. Furthermore, current forensic research focused on advanced depth image restoration is limited and has many shortcomings, particularly its poor resistance to interference after JPEG compression. Summary of the Invention

[0004] The purpose of this invention is to provide an image depth restoration forensics method based on multi-model fusion, which can achieve ideal forensics performance for images with different JPEG compression conditions.

[0005] To address the aforementioned technical problems, this invention provides a technical solution: an image depth restoration forensics method based on multi-model fusion, the method comprising:

[0006] S1. Input the image into the trained forensics network;

[0007] S2. Obtain the forensic results from the forensic network; the forensic results represent the tampered area of ​​the image;

[0008] The forensic network includes several encoder-decoder structured network branches, detail enhancement modules, and attention fusion modules; the detail enhancement modules are connected to the tail of each network branch; and the attention fusion modules are connected to the tail of the encoder in each network branch.

[0009] Different network branches, combined with the detail enhancement module, are used to obtain preliminary evidence collection results for images under different JPEG compression conditions. The attention fusion module first obtains aggregated features based on the output of the decoder in the network branch, then obtains several attention weight maps based on the aggregated features, and finally uses the attention weight maps to weight and sum the preliminary evidence collection results to obtain the evidence collection results.

[0010] According to the above scheme, the encoder in the network branch includes a first ordinary convolutional block, a second ordinary convolutional block, a first depthwise separable convolutional block, a first max pooling layer, a second depthwise separable convolutional block, a second max pooling layer, a third depthwise separable convolutional block, and a fourth depthwise separable convolutional block connected in sequence. It includes a first convolutional layer with residuals connected to the beginning of the first depthwise separable convolutional block and the end of the first max pooling layer, a second convolutional layer with residuals connected to the beginning of the second depthwise separable convolutional block and the end of the second max pooling layer, a third convolutional layer with residuals connected to the beginning and end of the third depthwise separable convolutional block, and a fourth convolutional layer with residuals connected to the beginning and end of the fourth depthwise separable convolutional block.

[0011] The first ordinary convolutional block includes a first ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the first ordinary convolutional layer is a 3×3 convolutional layer with 32 channels in its output feature map and a stride of 2.

[0012] The second ordinary convolutional block includes a second ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the second ordinary convolutional layer is a 3×3 convolutional layer with 64 channels in its output feature map and a stride of 1.

[0013] The first depthwise separable convolutional block includes a first depthwise separable convolutional layer, a BN layer, a ReLU activation function, a second depthwise separable convolutional layer, and a BN layer connected in sequence; both the first and second depthwise separable convolutional layers are 3×3 convolutional layers, and their output feature map has 128 channels.

[0014] The first, third, and fourth depthwise separable convolutional blocks each include a ReLU activation function, a third depthwise separable convolutional layer, a BN layer, a ReLU activation function, a fourth depthwise separable convolutional layer, and a BN layer connected in sequence; the third and fourth depthwise separable convolutional layers are both 3×3 convolutional layers, and their output feature map has 256 channels;

[0015] Both the first and second max-pooling layers are 3×3 pooling layers with a step size of 2.

[0016] The first, second, third, and fourth convolutional layers are all 1×1 convolutional layers with a stride of 2.

[0017] The tail end of the second ordinary convolutional block and the tail end of the first depth-separable convolutional block are hopped to the decoder in the network branch.

[0018] According to the above scheme, the decoder in the network branch includes a first upsampling layer, a first basic convolutional block, a second upsampling layer, a second basic convolutional block, a third upsampling layer, and a third basic convolutional block connected in sequence.

[0019] The first upsampling layer, the second upsampling layer, and the third upsampling layer all use a bilinear interpolation algorithm with a step size of 2;

[0020] The first basic convolutional block includes a first basic convolutional layer, a BN layer, a ReLU activation function, a second basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the first and second basic convolutional layers are both 3×3 convolutional layers, and their output feature map has 128 channels and a stride of 1.

[0021] The second basic convolutional block includes a third basic convolutional layer, a BN layer, a ReLU activation function, a fourth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the third and fourth basic convolutional layers are both 3×3 convolutional layers, with 64 channels in their output feature maps and a stride of 1;

[0022] The third basic convolutional block includes a fifth basic convolutional layer, a BN layer, a ReLU activation function, a sixth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the fifth and sixth basic convolutional layers are both 3×3 convolutional layers, with 32 channels in their output feature maps and a stride of 1;

[0023] The first end of the first basic convolutional block is skipped to the end of the first depth-separable convolutional block, and the first end of the second basic convolutional block is skipped to the end of the second ordinary convolutional block.

[0024] According to the above scheme, the detail enhancement module includes a third ordinary convolutional block, a fourth ordinary convolutional block, an average pooling layer, a fifth convolutional layer, a fourth upsampling layer, a first stacked layer, a sixth convolutional layer, and a first softmax layer connected in sequence.

[0025] The third ordinary convolutional block includes a third ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the third ordinary convolutional layer is a 3×3 convolutional layer with 20 channels in its output feature map;

[0026] The fourth ordinary convolutional block includes a fourth ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the third ordinary convolutional layer is a 3×3 convolutional layer with 20 channels in its output feature map;

[0027] The average pooling layer uses average pooling operations with sampling factors of 1 / 32, 1 / 16, 1 / 8 and 1 / 4 to perform downsampling and obtain multi-scale features;

[0028] The fifth convolutional layer is a 1×1 convolutional layer, and its output feature map has 1 channel, which maps multi-scale features to a single channel.

[0029] The fourth upsampling layer uses a bilinear interpolation algorithm to upsample the output of the fifth convolutional layer to the size of the image and then stacks the channel dimensions to obtain a 4-channel feature map.

[0030] The stacking layer stacks the 4-channel feature map output by the first upsampling layer with the 20-channel feature map output by the fourth ordinary convolutional block to obtain a 24-channel feature map.

[0031] The sixth convolutional layer is a 3×3 convolutional layer, and its output feature map has 2 channels; the sixth convolutional layer maps the 24-channel feature map output by the stacked layer to a 2-channel feature map;

[0032] The first Softmax layer normalizes the 2-channel feature map output by the sixth convolutional layer.

[0033] According to the above scheme, the number of network branches is 3; the attention fusion module includes a second stacking layer, an attention fusion module decoder, a second Softmax layer, a slicing layer, and a weighting layer;

[0034] The second stacking layer stacks the feature maps output by the decoders in each network branch to obtain the aggregated features;

[0035] The attention fusion module decoder includes a fourth basic convolutional block, a fifth upsampling layer, a fifth basic convolutional block, a sixth upsampling layer, a sixth basic convolutional block, a seventh upsampling layer, a seventh basic convolutional block, and a seventh convolutional layer connected in sequence.

[0036] The fourth basic convolutional block includes a seventh basic convolutional layer, a BN layer, a ReLU activation function, an eighth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the seventh and eighth basic convolutional layers are both 3×3 convolutional layers with 256 output feature channels and a stride of 1.

[0037] The fifth basic convolutional block includes a ninth basic convolutional layer, a BN layer, a ReLU activation function, a tenth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the ninth and tenth basic convolutional layers are both 3×3 convolutional layers with 128 output feature channels and a stride of 1.

[0038] The sixth basic convolutional block includes an eleventh basic convolutional layer, a BN layer, a ReLU activation function, a twelfth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the eleventh and twelfth basic convolutional layers are both 3×3 convolutional layers with 64 output feature channels and a stride of 1.

[0039] The seventh basic convolutional block includes a thirteenth basic convolutional layer, a BN layer, a ReLU activation function, a fourteenth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the thirteenth and fourteenth basic convolutional layers are both 3×3 convolutional layers with 32 output feature channels and a stride of 1.

[0040] The fifth, sixth, and seventh upsampling layers all employ a bilinear interpolation algorithm with a step size of 2.

[0041] The seventh convolutional layer is a 1×1 convolutional layer with 3 output feature channels and a stride of 1.

[0042] The second Softmax layer normalizes the 3-channel feature map output by the attention fusion module decoder;

[0043] The segmentation layer divides the normalized result of the second Softmax layer output into three attention weight maps;

[0044] The weighted layer performs a weighted summation of the three preliminary forensic results corresponding to the three network branches based on the three attention weight maps to obtain the forensic result.

[0045] According to the above scheme, the training method of the forensic network includes:

[0046] The deep inpainting algorithm was used to modify the images to different degrees, and the modified images were used to construct an image dataset.

[0047] The image dataset is divided into training set, validation set, and test set;

[0048] The training set is compressed to different degrees using JPEG to obtain a hybrid training set;

[0049] The forensic network is trained using a progressive training strategy and a boundary cross-entropy loss function on the hybrid training set.

[0050] According to the above scheme, the image is modified to different degrees using a deep repair algorithm. The modification location is randomly selected, and the modification shape includes circles, rectangles, and irregular shapes. The area ratio of the modification region includes 5%, 10%, and 15%.

[0051] According to the above scheme, the method for compressing the training set to different degrees of JPEG includes:

[0052] The training set is divided into several original equal parts, the same number of branches as the network.

[0053] The images in different original portions are subjected to JPEG compression operations with different quality factors to obtain several compressed portions, and each compressed portion constitutes the hybrid training set.

[0054] According to the above scheme, the method of training the forensic network using the progressive training strategy and the boundary cross-entropy loss function through the mixed training set includes a network branch pre-training stage, an attention fusion module pre-training stage, and a fine-tuning stage.

[0055] In the pre-training stage of the network branches, each network branch is trained using the compressed equal parts, and the output of each network branch is supervised pixel by pixel using standard cross-entropy to obtain the preset parameters of each network branch.

[0056] In the pre-training phase of the attention fusion module, a certain learning rate is set for the preset parameters of each network branch, and then the attention fusion module is trained using the mixed training set, and the evidence collection network is supervised using the boundary cross-entropy loss function.

[0057] In the fine-tuning stage, based on the forensic network obtained after the pre-training stage of the attention fusion module, a hybrid training set is used to jointly fine-tune each network branch and the attention fusion module.

[0058] According to the above scheme, in the boundary cross-entropy loss function, the weight factors for the tampered region boundary part, the tampered region non-boundary part, and the non-tampered region are 0.7, 0.2, and 0.1, respectively.

[0059] The beneficial effects of the present invention are as follows: 1. The forensic network used in this method contains multiple network branches. Each network branch is used to obtain preliminary forensic results of images under different JPEG compression conditions. The multiple preliminary forensic results are weighted and summed by the attention fusion module, which combines the advantages of different network branches, making the forensic results more robust. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the network structure of an evidence-gathering network according to an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of the network structure of an encoder with network branches according to an embodiment of the present invention;

[0062] Figure 3 This is a schematic diagram of the network structure of a detail enhancement module according to an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the network structure of a decoder for a network branch according to an embodiment of the present invention;

[0064] Figure 5 This is a schematic diagram of the network structure of the decoder of the attention fusion module according to an embodiment of the present invention;

[0065] Figure 6 This is a schematic diagram of different weighted regions of the cross-entropy loss function according to an embodiment of the present invention;

[0066] Figure 7 This is a flowchart of the training strategy for an evidence-gathering network according to an embodiment of the present invention.

[0067] Figure 8 This is a qualitative comparison chart of different schemes under different JPEG compression conditions;

[0068] Figure 9 This is a flowchart of an image restoration and forensics method based on multi-model fusion according to an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0070] See Figure 9 This embodiment discloses an image restoration and forensics method based on multi-model fusion, including:

[0071] S1. Input the image into the trained forensics network;

[0072] S2. Obtain the evidence collection results from the evidence collection network.

[0073] To obtain the aforementioned forensic network, the following three main steps are required:

[0074] (1) Select several typical depth restoration algorithms to construct an image dataset based on depth restoration;

[0075] (2) The network is constructed using three identical encoder-decoder structure network branches and an attention fusion module. See [link to relevant documentation] Figure 1The proposed progressive training strategy enables each branch of the multi-branch network to independently perform forensic analysis of tampered images under different JPEG compression conditions. The outputs of the multiple branches are then weighted using an attention fusion module to generate the final result. This ensemble strategy fully combines the advantages of different forensic branches, thereby maximizing robustness.

[0076] (3) Using the designed training strategy and the loss function that strengthens the focus on the boundary, the constructed evidence collection network is trained on the constructed image dataset to obtain the trained network model, and then the results are tested on the test set and the performance index is calculated.

[0077] Specifically:

[0078] (a) The image dataset is constructed using advanced deep inpainting algorithms, making the tampered images visually difficult to identify. The tampered regions are randomly selected, with shapes including circles, rectangles, and irregular shapes, and the area proportions of the tampered regions are set to 5%, 10%, and 15%. The dataset used in this chapter is constructed using one particular inpainting scheme as an example. First, 19350 inpainted images were obtained through the above operations, and randomly divided into image sets of 18000, 450, and 900 for use as training, validation, and test sets, respectively. Then, the 19350 inpainted images were compressed using JPEG with quality factors of 95 and 75, respectively, resulting in 19350×3 images (including the original images without JPEG compression). For the pre-training stage of the branch networks, the training set containing 18000×3 images was used as the input to the three network branches according to the different JPEG compression methods. Finally, the original training set containing 18,000 images was randomly divided into three equal parts. One part was left unprocessed, while the other two parts were subjected to JPEG compression with quality factors of 95 and 75, respectively. The three parts were then merged to obtain a mixed training set of 18,000 images. This dataset was used for training the network's attention fusion module during the pre-training and fine-tuning phases, as well as for training with other comparative schemes.

[0079] (b) The forensic network can consist of multiple branches. The network of this invention uses three identical encoder-decoder structure branches. The encoder uses a simplified Xception network. See [link to documentation]. Figure 2The system comprises two regular convolutional blocks, four depthwise separable convolutional blocks, two max-pooling layers, and four residual connection blocks. Each regular convolutional block consists of one 3×3 convolutional layer, a batch normalization (BN) layer, and a ReLU activation function. The strides of the convolutional layers in the two regular convolutional blocks are 2 and 1, respectively, resulting in 32 and 64 output feature map channels. Similarly, each depthwise separable convolutional block consists of two repeated depthwise separable convolutional layers and a BN layer, with a ReLU activation function inserted in between. The two depthwise separable convolutional layers within the same depthwise separable convolutional block have the same number of output feature map channels; the output feature map channels for the four depthwise separable convolutional blocks are 128, 256, 256, and 256, respectively. The two max-pooling layers are located after the first two depthwise separable convolutional blocks. Adjacent depthwise separable convolutional blocks are connected by residual connections using convolutional layers with a kernel size of 1×1 and a stride of 2. In addition, the encoder's skip connections feed the 64-channel feature map after the second ordinary convolutional block and the 128-channel feature map after the first max pooling layer to the corresponding scale feature locations in the decoder.

[0080] (c) Both ordinary convolutional blocks consist of a 3×3 convolutional layer, a BN layer, and a ReLU activation function, with strides of 2 and 1, respectively. The depthwise separable convolutional block consists of two repeated depthwise separable convolutional layers and a BN layer, with a ReLU activation function inserted in between.

[0081] (d) See also Figure 4 The decoder of the branch network consists of three upsampling layers and three basic convolutional blocks. Through three upsampling operations and three basic convolutional blocks, the feature map is restored to the input image size. The number of output feature map channels at 1 / 4, 1 / 2, and 1 input image scales are 128, 64, and 32, respectively. The basic convolutional block consists of a convolutional layer with a 3×3 kernel and a stride of 1, a BN layer, and a ReLU activation function, repeated twice. The basic convolutional block's role is to obtain the feature representation of the tampered region. The upsampling layers use a bilinear interpolation algorithm with a stride of 2. Three upsampling operations progressively restore the feature map obtained by the encoder to the input image size. The feature map obtained after the first two upsampling operations is stacked with the encoder feature map obtained through skip connections to compensate for the loss of detail information caused by the convolution and downsampling operations. Three basic convolutional blocks were inserted after two skip connections and a third upsampling, respectively, and the number of output feature map channels at the 1 / 4, 1 / 2 and 1 input image scales were set to 128, 64 and 32, respectively.

[0082] Add detail enhancement modules at the end of network branches to enhance the network's feature representation capabilities. See [link to documentation]. Figure 3The detail enhancement module consists of four parts: 1) Preliminary feature extraction: Two ordinary convolutional blocks are used to perform preliminary feature extraction on the 32-channel feature map output by the decoder to obtain a 20-channel feature map; 2) Pyramid pooling: The preliminarily extracted features are downsampled using average pooling operations with sampling factors of 1 / 32, 1 / 16, 1 / 8 and 1 / 4 respectively to obtain feature mappings at multiple scales, i.e., constructing a feature pyramid with four scales; 3) Feature aggregation: First, a convolution with a kernel size of 1×1 is used to map the multi-scale features to a single channel, and then bilinear interpolation is used to upsample them to the input image size before stacking them by channel dimensions. Finally, the stacked 4-channel feature map is stacked with the preliminarily extracted feature map to complete feature aggregation; 4) Branch result output: First, a convolution with a kernel size of 1×1 is used to map the aggregated features of the 24 channels of the three network branches to a 2-channel feature map, and then a Softmax classifier is used to normalize them to obtain the output result.

[0083] (e) An attention fusion module was designed for fusion. First, the feature maps output from the three encoders were stacked to obtain a 768-channel aggregated feature. Then, the aggregated feature was input into the decoder of the attention fusion module, see [link to relevant documentation]. Figure 5 The decoder in the attention fusion module uses four basic convolutional blocks (output feature map channels of 256, 128, 64, and 32 respectively) interspersed with upsampling layers (bilinear interpolation with a stride of 2) to progressively restore the feature maps to their original size. At the end of the decoder, a 1×1 convolution reduces the number of feature channels to 3. Next, the 3-channel feature map output by the decoder is normalized using a Softmax layer and then divided into three attention weight maps. Finally, the three preliminary results are weighted using their corresponding weight maps to obtain the final result.

[0084] (f) A boundary cross-entropy loss function is proposed based on the cross-entropy loss function to improve the supervision of the forensic network. By increasing the weight factor of the tampered region boundary, the impact of class imbalance is mitigated, and the supervision of the tampered region boundary is enhanced to optimize the boundary details of the forensic results, highlighting the importance of the tampered region boundary information. In the boundary cross-entropy loss function, the weight factors for the tampered region boundary part (obtained by subtracting from the 9×9 erosion operation of the true labels), the non-boundary part of the tampered region, and the non-tampered region are 0.7, 0.2, and 0.1, respectively.

[0085] Specifically, the boundary cross-entropy loss function can be expressed as:

[0086]

[0087] In the above formula, θ1, θ2, and θ3 represent the boundary parts of the tampered region, respectively. Figure 6 (a) White area), Non-boundary part of the altered area ( Figure 6(a) gray area) and unaltered area ( Figure 6 (a) Weighting factors for the black region, where W and H are the width and height of the feature map, respectively. In this embodiment, to emphasize boundary information and reduce class imbalance, θ1, θ2, and θ3 are set to 0.7, 0.2, and 0.1, respectively.

[0088] Y kij To output the pixel value of the k-th channel at coordinates (i,j) to the network, B ij and J ij These represent the pixel values ​​at the true labels (i,j) of the altered region boundary and the complete altered region, respectively, both of which have only two possible values: 1 and 0. For example... Figure 6 As shown, the boundary true label B ( Figure 6 (d) is obtained through the original true label J( Figure 6 (b) and the 9×9 etching operation on J ( Figure 6 (c) is obtained by subtraction.

[0089] (g) Propose a progressive training strategy comprising three stages, see [link to relevant documentation]. Figure 7 The three stages are the branch network pre-training stage, the attention fusion module pre-training stage, and the fine-tuning stage.

[0090] Branch network pre-training stage: The training data for the three branches are datasets without JPEG compression and datasets after JPEG compression with quality factors of 95 and 75, respectively. Standard cross-entropy is used to supervise each pixel of the output of each branch to obtain the parameters of the three trained network branches, that is, to obtain three network models that can prove tampered images under different JPEG compression conditions.

[0091] In the pre-training phase of the attention fusion module: the parameters learned in the pre-training phase of the branch network are first used as the preset parameters of the branch network, and a very small learning rate is set for these network parameters, making them almost frozen. Then, the attention fusion module is trained using a mixed dataset of tampered images under three different JPEG compression conditions, and the proposed boundary cross-entropy loss function is used to supervise the overall network, resulting in a model that has preliminary evidence-gathering capabilities for tampered images under different JPEG compression conditions.

[0092] Fine-tuning phase: Since step-by-step training can lead to the accumulation of model errors, a hybrid dataset was used to jointly fine-tune the three branch networks and the attention fusion module based on the model learned in the second phase, aiming for better overall forensic results. The parameters learned in the second phase were used as preset parameters, and all settings were the same as in the second phase except for the number of iterations; the difference was that the number of iterations in this phase was only 3.

[0093] (h) The network uses a corresponding loss function to calculate the loss between the predicted value and the label at each stage of training. Then, the network parameters are updated through backpropagation until the training loss reaches its minimum and the metrics of the training set and the validation set continue to rise and become similar.

[0094] The method proposed in this invention is implemented based on the PyTorch framework, and the model training and testing are completed using an NVIDIA 1080Ti GPU in an Ubuntu environment. The forensic metrics TPR, FPR, F1, and IOU obtained from testing on the test set are shown in Tables 1 and 2.

[0095] Table 1 Comparison of F1 (%) and IoU (%) for different schemes under different JPEG compression conditions

[0096]

[0097] Table 2 Comparison of TPR (%) and FPR (%) for different schemes under different JPEG compression conditions

[0098]

[0099] Note: Best results are marked in bold.

[0100] The above table and Figure 8 The comparison schemes 1, 2, and 3 mentioned above refer to the scheme proposed by Zhu et al. [A deep learning approach to patch-based image inpainting forensics[J]], DeeplabV3+, and the scheme proposed by Li et al. [Localization of deep inpainting using high-pass fully convolutional network[C]].

[0101] TPR stands for True Positive Rate: the proportion of correctly identified tampered regions out of the actual tampered regions; FPR stands for False Positive Rate: the proportion of incorrectly identified tampered regions out of the actual untampered regions; F1 score balances precision and recall; IOU stands for Intersection over Union: the ratio of the intersection and union of identified tampered regions and the actual tampered regions.

[0102] Figure 8 This demonstrates the forensic results of different methods on three tampered images under different JPEG compression conditions. First, observe the sample without JPEG compression ( Figure 8 (Lines 1-3) and samples compressed with a quality factor of 95 ( Figure 8 Lines 4-6), the solution of this invention ( Figure 8(f) Compared to other comparison schemes ( Figure 8 (c)-(e)) clearly not only have fewer false alarm pixels, but also more accurately fit the actual tampered area in terms of position and shape. Other comparison schemes all showed significant false positives and false negatives. Next, we observed JPEG compressed samples with a quality factor of 75 ( Figure 8 Lines 7-9 compare the evidence collection results of different schemes. Although the present invention's scheme is relatively different from some comparative schemes (comparisons) Figure 8 The localization result in line 7(d) does not show a significant advantage, but it still achieves a superior localization result compared to most of the comparison solutions (e.g., Figure 8 (Line 8) Overall, the forensic performance achieved is relatively satisfactory. Finally, comparing the forensic results of samples under different JPEG compression conditions shows that the forensic results of different schemes deteriorate as the quality factor decreases. This is because the decrease in the JPEG quality factor causes more traces of restoration operations to be removed. In summary, the scheme of this invention has superior forensic performance compared to other comparative schemes, especially for uncompressed samples or samples with high quality factors.

[0103] Tables 1 and 2 show the quantitative comparison results of different forensic methods. From the test results of the overall dataset (containing 900 restored tampered images each with JPEG compression (quality factor 95 and 75), totaling 2700 images), it can be seen that the forensic results of the present invention are significantly better than other comparative methods in all indicators. Furthermore, as the quality factor decreases, the forensic performance of all methods declines due to the removal of more restoration traces. For the uncompressed JPEG and JPEG compressed samples with a quality factor of 95, the present invention's method achieves significantly better forensic performance than other comparative methods in all indicators. For example, for JPEG compressed samples with a quality factor of 95, the present invention's method outperforms the second-best forensic method by 7.88%, 12.42%, 7.90%, and 0.56% in F1, IoU, TPR, and FPR, respectively, demonstrating a very significant performance advantage. For JPEG compressed samples with a quality factor of 75, although the present invention's method does not achieve the best performance in TPR and FPR, the difference compared to other superior comparative methods is very small. The reason for this may be that, apart from the solution of this invention, other forensic methods tend to learn the repair trace features of JPEG compressed samples with a quality factor of 75, while struggling to simultaneously learn the trace features of samples without JPEG compression and those compressed with a quality factor of 95. However, the solution of this invention, through multi-model fusion, strives to take into account samples with different JPEG compression conditions, fully leveraging the advantages of multi-branch networks. While maintaining good performance for JPEG compressed samples with a quality factor of 75, it significantly improves the forensic performance for samples with other JPEG compression conditions. In summary, the solution of this invention achieves superior forensic performance compared to other comparative solutions.

[0104] It should be understood that the multimedia file types applicable to the method described in this invention are not limited to images, the deep inpainting algorithm used in this method is not limited to that used in this embodiment, and the selection of configuration parameters for the de-integer network is not limited to the configurations in the embodiment (e.g., the number of network branches or loss function parameters).

[0105] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for image depth restoration and forensic investigation based on multi-model fusion, characterized in that, The method includes: S1. Input the image into the trained forensics network; S2. Obtain the forensic results from the forensic network; the forensic results represent the tampered area of ​​the image; The forensic network includes several encoder-decoder structured network branches, detail enhancement modules, and attention fusion modules; the detail enhancement modules are connected to the tail of each network branch; and the attention fusion modules are connected to the tail of the encoder in each network branch. Different network branches, combined with the detail enhancement module, are used to obtain preliminary forensic results of images under different JPEG compression conditions. The attention fusion module first obtains aggregated features based on the output of the decoder in the network branch, then obtains several attention weight maps based on the aggregated features, and finally uses the attention weight maps to weight and sum the preliminary forensic results to obtain the forensic results. The encoder in the network branch includes a first ordinary convolutional block, a second ordinary convolutional block, a first depthwise separable convolutional block, a first max pooling layer, a second depthwise separable convolutional block, a second max pooling layer, a third depthwise separable convolutional block, and a fourth depthwise separable convolutional block connected in sequence. It includes a first convolutional layer with residuals connected to the beginning of the first depthwise separable convolutional block and the end of the first max pooling layer, a second convolutional layer with residuals connected to the beginning of the second depthwise separable convolutional block and the end of the second max pooling layer, a third convolutional layer with residuals connected to the beginning and end of the third depthwise separable convolutional block, and a fourth convolutional layer with residuals connected to the beginning and end of the fourth depthwise separable convolutional block. The first ordinary convolutional block includes a first ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the first ordinary convolutional layer is a 3×3 convolutional layer with 32 channels in its output feature map and a stride of 2. The second ordinary convolutional block includes a second ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the second ordinary convolutional layer is a 3×3 convolutional layer with 64 channels in its output feature map and a stride of 1. The first depthwise separable convolutional block includes a first depthwise separable convolutional layer, a BN layer, a ReLU activation function, a second depthwise separable convolutional layer, and a BN layer connected in sequence; both the first and second depthwise separable convolutional layers are 3×3 convolutional layers, and their output feature map has 128 channels. The first, third, and fourth depthwise separable convolutional blocks each include a ReLU activation function, a third depthwise separable convolutional layer, a BN layer, a ReLU activation function, a fourth depthwise separable convolutional layer, and a BN layer connected in sequence; the third and fourth depthwise separable convolutional layers are both 3×3 convolutional layers, and their output feature map has 256 channels; Both the first and second max-pooling layers are 3×3 pooling layers with a step size of 2. The first, second, third, and fourth convolutional layers are all 1×1 convolutional layers with a stride of 2. The tail end of the second ordinary convolutional block and the tail end of the first depth-separable convolutional block are hopped to the decoder in the network branch. The decoder in the network branch includes a first upsampling layer, a first basic convolutional block, a second upsampling layer, a second basic convolutional block, a third upsampling layer, and a third basic convolutional block connected in sequence. The first upsampling layer, the second upsampling layer, and the third upsampling layer all use a bilinear interpolation algorithm with a step size of 2; The first basic convolutional block includes a first basic convolutional layer, a BN layer, a ReLU activation function, a second basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the first and second basic convolutional layers are both 3×3 convolutional layers, and their output feature map has 128 channels and a stride of 1. The second basic convolutional block includes a third basic convolutional layer, a BN layer, a ReLU activation function, a fourth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the third and fourth basic convolutional layers are both 3×3 convolutional layers, with 64 channels in their output feature maps and a stride of 1; The third basic convolutional block includes a fifth basic convolutional layer, a BN layer, a ReLU activation function, a sixth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the fifth and sixth basic convolutional layers are both 3×3 convolutional layers, with 32 channels in their output feature maps and a stride of 1; The first end of the first basic convolutional block is skipped to the end of the first depth-separable convolutional block, and the first end of the second basic convolutional block is skipped to the end of the second ordinary convolutional block.

2. The image depth restoration forensics method based on multi-model fusion according to claim 1, characterized in that, The detail enhancement module includes a third ordinary convolutional block, a fourth ordinary convolutional block, an average pooling layer, a fifth convolutional layer, a fourth upsampling layer, a first stacked layer, a sixth convolutional layer, and a first softmax layer connected in sequence. The third ordinary convolutional block includes a third ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the third ordinary convolutional layer is a 3×3 convolutional layer with 20 channels in its output feature map; The fourth ordinary convolutional block includes a fourth ordinary convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the third ordinary convolutional layer is a 3×3 convolutional layer with 20 channels in its output feature map; The average pooling layer uses average pooling operations with sampling factors of 1 / 32, 1 / 16, 1 / 8 and 1 / 4 to perform downsampling and obtain multi-scale features; The fifth convolutional layer is a 1×1 convolutional layer, and its output feature map has 1 channel, which maps multi-scale features to a single channel. The fourth upsampling layer uses a bilinear interpolation algorithm to upsample the output of the fifth convolutional layer to the size of the image and then stacks the channel dimensions to obtain a 4-channel feature map. The stacking layer stacks the 4-channel feature map output by the first upsampling layer with the 20-channel feature map output by the fourth ordinary convolutional block to obtain a 24-channel feature map. The sixth convolutional layer is a 3×3 convolutional layer, and its output feature map has 2 channels; the sixth convolutional layer maps the 24-channel feature map output by the stacked layer to a 2-channel feature map; The first Softmax layer normalizes the 2-channel feature map output by the sixth convolutional layer.

3. The image depth restoration forensics method based on multi-model fusion according to claim 1, characterized in that, The number of network branches is 3; the attention fusion module includes a second stacking layer, an attention fusion module decoder, a second Softmax layer, a slicing layer, and a weighting layer; The second stacking layer stacks the feature maps output by the decoders in each network branch to obtain the aggregated features; The attention fusion module decoder includes a fourth basic convolutional block, a fifth upsampling layer, a fifth basic convolutional block, a sixth upsampling layer, a sixth basic convolutional block, a seventh upsampling layer, a seventh basic convolutional block, and a seventh convolutional layer connected in sequence. The fourth basic convolutional block includes a seventh basic convolutional layer, a BN layer, a ReLU activation function, an eighth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the seventh and eighth basic convolutional layers are both 3×3 convolutional layers with 256 output feature channels and a stride of 1. The fifth basic convolutional block includes a ninth basic convolutional layer, a BN layer, a ReLU activation function, a tenth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the ninth and tenth basic convolutional layers are both 3×3 convolutional layers with 128 output feature channels and a stride of 1. The sixth basic convolutional block includes an eleventh basic convolutional layer, a BN layer, a ReLU activation function, a twelfth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the eleventh and twelfth basic convolutional layers are both 3×3 convolutional layers with 64 output feature channels and a stride of 1. The seventh basic convolutional block includes a thirteenth basic convolutional layer, a BN layer, a ReLU activation function, a fourteenth basic convolutional layer, a BN layer, and a ReLU activation function connected in sequence; the thirteenth and fourteenth basic convolutional layers are both 3×3 convolutional layers with 32 output feature channels and a stride of 1. The fifth, sixth, and seventh upsampling layers all employ a bilinear interpolation algorithm with a step size of 2. The seventh convolutional layer is a 1×1 convolutional layer with 3 output feature channels and a stride of 1. The second Softmax layer normalizes the 3-channel feature map output by the attention fusion module decoder; The segmentation layer divides the normalized result of the second Softmax layer output into three attention weight maps; The weighted layer performs a weighted summation of the three preliminary forensic results corresponding to the three network branches based on the three attention weight maps to obtain the forensic result.

4. The image depth restoration forensics method based on multi-model fusion according to claim 1, characterized in that, The training method for the forensic network includes: The deep inpainting algorithm was used to modify the images to different degrees, and the modified images were used to construct an image dataset. The image dataset is divided into training set, validation set, and test set; The training set is compressed to different degrees using JPEG to obtain a hybrid training set; The forensic network is trained using a progressive training strategy and a boundary cross-entropy loss function on the hybrid training set.

5. The image depth restoration forensics method based on multi-model fusion according to claim 4, characterized in that, The image is modified to varying degrees using a deep inpainting algorithm. The modification location is randomly selected, and the modification shape includes circles, rectangles, and irregular shapes. The area ratio of the modified region includes 5%, 10%, and 15%.

6. The image depth restoration forensics method based on multi-model fusion according to claim 4, characterized in that, The method for compressing the training set to different degrees using JPEG includes: The training set is divided into several original equal parts, the same number of branches as the network. The images in different original portions are subjected to JPEG compression operations with different quality factors to obtain several compressed portions, and each compressed portion constitutes the hybrid training set.

7. The image depth restoration forensics method based on multi-model fusion according to claim 6, characterized in that, The method of training the forensic network using the hybrid training set with a progressive training strategy and boundary cross-entropy loss function includes a network branch pre-training stage, an attention fusion module pre-training stage, and a fine-tuning stage. In the pre-training stage of the network branches, each network branch is trained using the compressed equal parts, and the output of each network branch is supervised pixel by pixel using standard cross-entropy to obtain the preset parameters of each network branch. In the pre-training phase of the attention fusion module, a certain learning rate is set for the preset parameters of each network branch, and then the attention fusion module is trained using the mixed training set, and the evidence collection network is supervised using the boundary cross-entropy loss function. In the fine-tuning stage, based on the forensic network obtained after the pre-training stage of the attention fusion module, a hybrid training set is used to jointly fine-tune each network branch and the attention fusion module.

8. The image depth restoration forensics method based on multi-model fusion according to claim 4 or 7, characterized in that, In the boundary cross-entropy loss function, the weight factors for the tampered region boundary part, the tampered region non-boundary part, and the non-tampered region are 0.7, 0.2, and 0.1, respectively.

Images