Image restoration detection method based on reverse tampering trace enhancement and adaptive edge perception

Abstract

The invention provides an image restoration detection method based on reverse tampering trace enhancement and adaptive edge perception, and the method comprises the steps: removing the homogeneous features of a tampering region and a non-tampering region through differential convolution in a feature enhancement stage, reversely enhancing the tampering trace, enlarging a feature receptive field through an expansion convolution layer, and carrying out the recognition of the tampering trace. And fusion of multi-scale information is realized. In the feature extraction stage, an adaptive edge extraction operator is set, the network is guided to pay attention to the inconsistency of the edge of the tampered area, and the positioning effect of the tampered area is refined. In the decision-making stage, an attention mechanism based on cosine similarity optimization features is introduced, and only features with large intra-class cosine similarity are screened for processing so as to reduce intra-class feature differences. The method can accurately detect and position various invisible repairing and tampering operations, and has good generalization ability for a deep repairing method and a traditional repairing method.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing and multimedia forensics, specifically relating to an image restoration and detection method based on reverse tampering trace enhancement and adaptive edge perception. Background Technology

[0002] With the rapid development of image inpainting technology, the latest generative networks can generate visually very realistic image content, making image inpainting detection increasingly challenging. However, existing inpainting forensics methods still have many problems.

[0003] To enhance tampering detection, preprocessing modules typically employ a combination of high-pass filters, SRM, and Bayar. However, this approach also amplifies homogeneous features between tampered and untampered regions, features that are weakly correlated with tampering detection. Furthermore, extracted high-frequency or residual features amplify noise, failing to capture more complex semantic information and hindering tamper detection remediation. In addition, current remediation methods mostly employ traditional edge detection operators to extract edge features, using fixed templates and relying on local gradient calculations, making them highly sensitive to noise and resulting in weakened edge responses in tampered regions. The lack of effective edge artifact capture methods leads to low edge localization accuracy. Moreover, some models introduce attention mechanisms, weighting all features based on global similarity relationships, which may introduce irrelevant information. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention provides an image inpainting and detection method based on reverse tampering trace enhancement and adaptive edge perception. The technical solution is as follows:

[0005] Feature enhancement includes: preprocessing the input image I using fused differential convolution to obtain feature F. fdc Then feature F fdc The feature F is obtained by inputting a two-scale dilated convolutional layer. m Simultaneously, ordinary convolution is performed on the input image I to extract the neglected detail features F. c , feature F c and feature F m Feature F is obtained by connecting on the channel. cm .

[0006] Feature extraction, including: extracting the features F obtained during the image enhancement stage. cm The feature F is obtained by inputting it into the backbone network ResNet50 for feature extraction.

[0007] The repair region decision-making includes: optimizing the features F extracted by the backbone network ResNet50 through a feature attention mechanism based on cosine similarity, including global attention and local attention. The features obtained through global attention are... The features obtained through local attention are Feature F, feature and characteristics Feature F is obtained by connecting on the channel. o The system employs a three-stage upsampling decoding process, consisting of upsampling, ordinary convolution, batch normalization, and activation functions, to obtain an output image of the same size as the input image. A region loss function is then calculated. A threshold of 0.5 is set: pixels with a probability greater than 0 and less than 0.5 are set to 0, and pixels with a probability greater than 0.5 and less than 1 are set to 1, ultimately yielding a binary detection result.

[0008] Adaptive edge awareness includes: passing the outputs of the four stages of ResNet50 sequentially through an adaptive edge extraction operator and an edge residual module to obtain feature F. s1 F s2 F s3 and F s4 For feature F s2 Perform upsampling, and compare with feature F s1 The features F are obtained by adding them pixel by pixel and then passing them through the edge residual module again. s12 For feature F s3 Perform upsampling and feature F s12 The features F are obtained by adding each pixel individually and then passing them through the edge residual module. s123 For feature F s4 Perform upsampling and feature F s123 The features F are obtained by adding each pixel individually and then passing them through the edge residual module. s1234 , feature F s1234 The predicted edge feature map is obtained by inputting the Sigmoid activation function, and the edge loss function is calculated. The edge loss function and the region loss function are jointly used to supervise the parameter updates in the network training.

[0009] The total loss function used during training in this invention consists of two parts: region loss and edge loss, as shown in the following formula:

[0010] L total =L region +βL edge

[0011] Among them, the edge loss L edge For binary cross-entropy loss, the region loss L region We also use binary cross-entropy loss, and further employ focus loss as a supplement. β is a hyperparameter, which we set to 0.5 in this invention.

[0012] In summary, compared with the prior art, the present invention has the following advantages:

[0013] This invention effectively enhances the repair of tampering traces and suppresses the homogeneity between tampered and untampered regions of an image by fusing differential convolution; it models edge features in the image more flexibly through an adaptive edge extraction operator, and the supervised network focuses on the inconsistency of the edges of the tampered region, refining the localization effect of the tampered region; at the same time, the attention mechanism reduces the interference of weakly correlated features, helping the network to better understand contextual information, thereby reducing pixel misjudgment.

[0014] Compared with similar methods in the prior art, the present invention has achieved higher evaluations in AUC, F1 and IoU metrics, can accurately detect and locate a variety of invisible repair and tampering operations, and has good generalization ability to deep repair methods and traditional repair methods. Attached Figure Description

[0015] Figure 1 This is a structural diagram of the model of the present invention;

[0016] Figure 2 Structure diagram of the adaptive edge extraction operator;

[0017] Figure 3 This is a structural diagram of the edge residual module;

[0018] Figure 4 This is a visualization of the invention. Detailed Implementation

[0019] This invention is based on multimedia forensics technology and comprises four parts: feature enhancement, feature extraction, restoration region decision-making, and adaptive edge perception. To better understand the technical solution of this invention, the appendix is ​​provided below. Figure 1 The present invention will be described in detail below.

[0020] (1) The specific steps of feature enhancement are as follows:

[0021] Step a1: Preprocess the input image I using fusion differential convolution to obtain feature F. fdc .

[0022] The fusion differential convolution used in this step is shown in the following formula:

[0023]

[0024] Where (x,y) represents the coordinates of the current pixel, and (x+u,y+v) represents the coordinates of its neighboring pixels. I(x,y) represents the pixel value of the input image at coordinates (x,y), and I(x+u,y+v) represents the neighboring pixel values ​​at coordinates (x,y). ω fdcThis represents a 3×3 convolution kernel, whose weights are obtained from data obtained during network training.

[0025] Step a2, the feature F obtained in step a1 fdc The input consists of two dilated convolutional layers with different dilation rates. The first layer produces an intermediate feature F1. Feature F1 is then compared with feature F... fdc Connect them together, input them into the second layer of dilated convolution to obtain feature F2, and then combine feature F... fdc F1 and F2 are concatenated along the channel dimension to obtain the final feature F. m .

[0026] The formula for calculating the receptive field L of dilated convolution is as follows:

[0027] L = k + (k-1)·(d-1)

[0028] Where k is the kernel size and d is the dilation rate.

[0029] In this step, let feature F fdc First, intermediate features F1 are obtained by using dilated convolutional layers with dilation rates d1 and d2.

[0030]

[0031] Then feature F1 and feature F fdc Connect them together and input them into a dilated convolutional layer with dilation rates d3 and d4 to obtain feature F2.

[0032]

[0033] Ultimately, feature F fdc F1 and F2 are connected along the channel dimension to obtain the final feature F. m To obtain multi-scale features.

[0034] F m =Concat(F fdc (F1, F2)

[0035] Step a3: Perform multiple ordinary convolutions on the input image I to obtain feature F. c .

[0036] In this step, the first layer is a 5×5 convolution, the second layer is a combination of 3×3 convolution, batch normalization, and ReLU activation function, and the third layer is also a combination of 3×3 convolution, batch normalization, and ReLU activation function. The final feature is denoted as F. c The weights used in the convolutions are obtained from the data through network training, and the formula for this process is shown below:

[0037] F c =Relu(BN(Conv)3×3 (Relu(BN(Conv 3×3 (Conv 5×5 (I)))))))

[0038] Step a4, the feature F obtained in step a2 m and the feature F obtained in step a3 c Feature F is obtained by connecting on the channel. cm .

[0039] (2) The specific steps of feature extraction are as follows:

[0040] Feature F obtained by feature enhancement cm The features are extracted from the backbone network ResNet50 and finally obtained as features F.

[0041] (3) The specific steps for repairing regional decisions are as follows:

[0042] Step b1: The features F extracted by the backbone network ResNet50 are processed through an attention mechanism based on cosine similarity optimization, including global attention and local attention. The features obtained through global attention are... The features obtained through local attention are

[0043] In this step, the multi-channel feature vector F(i,j,·) of feature F at position (i,j) can be denoted as F ij Calculate the cosine similarity S between it and the multi-channel feature vectors at other locations. ij,nm The calculation formula is as follows:

[0044]

[0045] Here, cosine[] represents calculating the cosine similarity of vectors, and || represents taking the modulus.

[0046] Set the similarity threshold η, and select S. ij,nm >η features regenerate new multi-channel features at position (i,j) The process is shown in the following formula:

[0047]

[0048] Local attention uses a 5×5 weight matrix ω l Acquired through a data-driven approach, and then compared with F. ij Convolution obtains updated features

[0049] Step b2, the features obtained in step b1 and characteristics Feature F is obtained by connecting it to feature F on the channel. o .

[0050] Step b3, for the feature F obtained in step b2 o A third-order upsampling decoding process is performed. Between the first and second upsampling, and between the second and third upsampling, 3×3 ordinary convolutions, batch normalization, and ReLU activation functions are embedded. After the third upsampling, a 3×3 ordinary convolution and a Sigmoid activation function are applied again, resulting in an output of the same size as the input image. The region loss function is then calculated. A threshold of 0.5 is set: pixels with a probability greater than 0 and less than 0.5 are set to 0, and pixels with a probability greater than 0.5 and less than 1 are set to 1, ultimately yielding a binary detection result.

[0051] (4) The specific steps of adaptive edge sensing are as follows:

[0052] The outputs of the four stages of ResNet50 are sequentially passed through the adaptive edge extraction operator and the edge residual module to obtain feature F. s1 F s2 F s3 and F s4 For feature F s2 Perform upsampling, and compare with feature F s1 The features F are obtained by adding them pixel by pixel and then passing them through the edge residual module again. s12 For feature F s3 Perform upsampling and feature F s12 The features F are obtained by adding each pixel individually and then passing them through the edge residual module. s123 For feature F s4 Perform upsampling and feature F s123 The features F are obtained by adding each pixel individually and then passing them through the edge residual module. s1234 , feature F s1234 The predicted edge feature map is obtained by inputting the Sigmoid activation function, the edge loss function is calculated, and the parameter updates in the supervised network training are monitored.

[0053] In this step, the adaptive edge extraction operator has two core principles in its structural design: the center weight value is always 0, and the sum of the weight values ​​of the neighborhood is also 0, as shown in the following formula.

[0054]

[0055] Where, ω edge For adaptive edge extraction operator, (u,v) is the position index relative to the center point, and the values ​​of u and v are both in the range of {-1,0,1}.

[0056] like Figure 2As shown, after the adaptive edge detection operator, L2 regularization is used to reduce the dimension N to 1. Then, the feature map is reweighted using edge attention weights and element-wise multiplication through a batch normalization layer and a Sigmoid activation function.

[0057] like Figure 3 As shown, the edge residual module first adjusts the feature map dimension to a uniform dimension through 1×1 convolution. The batch normalization layer in the middle normalizes the continuously added result and then uses 3×3 convolution, ReLU activation function and residual connection.

[0058] The total loss function used during training in this invention consists of two parts: region loss and edge loss, as shown in the following formula:

[0059] L total =L region +βL edge

[0060] Where β is a hyperparameter, and the edge loss L edge For binary cross-entropy loss, the region loss L region We also use binary cross-entropy loss, and further employ focus loss as a supplement.

[0061] Area loss L region The formula is shown below:

[0062] L region (M g M p ) = L B (M g M p )+L F (M g M p )

[0063] Among them, M g To repair the true value of the region, M p This represents the repaired area detected by the model. L B (M g M p ) represents the binary cross-entropy loss, L F (M g M p () is the focus of the loss.

[0064] The formula for calculating the binary cross-entropy loss function is shown below:

[0065]

[0066] The formula for calculating the focus loss function is shown below:

[0067]

[0068] Where α f It is a weighting parameter that balances the importance of positive and negative samples, γ f This is called the focusing parameter. In this invention, γ is set as... f =2, α f =0.75.

[0069] Figure 4 The image shown is a visualization of the results of this invention. By comparing the localization results of different image restoration and forensics schemes, it can be seen that our method is superior to other schemes for restoration and tampering samples of regular (second row), slender (fourth row), and small (fifth row) areas.

[0070] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. An image inpainting and detection method based on reverse tampering trace enhancement and adaptive edge perception, characterized in that: Feature enhancement includes: preprocessing the input image I using fused differential convolution to obtain feature F. fdc Then feature F fdc The feature F is obtained by inputting a two-scale dilated convolutional layer. m Simultaneously, ordinary convolution is performed on the input image I to extract the neglected detail features F. c , feature F c and feature F m Feature F is obtained by connecting on the channel. cm . Feature extraction, including: extracting the features F obtained during the image enhancement stage. cm The feature F is obtained by inputting it into the backbone network ResNet50 for feature extraction. The repair region decision-making includes: optimizing the features F extracted by the backbone network ResNet50 through a feature attention mechanism based on cosine similarity, including global attention and local attention. The features obtained through global attention are... The features obtained through local attention are Feature F, feature and characteristics Feature F is obtained by connecting on the channel. o The system employs a three-stage upsampling decoding process, consisting of upsampling, ordinary convolution, batch normalization, and activation functions, to obtain an output image of the same size as the input image. A region loss function is then calculated. A threshold of 0.5 is set: pixels with a probability greater than 0 and less than 0.5 are set to 0, and pixels with a probability greater than 0.5 and less than 1 are set to 1, ultimately yielding a binary detection result. Adaptive edge awareness includes: passing the outputs of the four stages of ResNet50 sequentially through an adaptive edge extraction operator and an edge residual module to obtain feature F. s1 F s2 F s3 and F s4 For feature F s2 Perform upsampling, and compare with feature F s1 The features F are obtained by adding them pixel by pixel and then passing them through the edge residual module again. s12 For feature F s3 Perform upsampling and feature F s12 The features F are obtained by adding each pixel individually and then passing them through the edge residual module. s123 For feature F s4 Perform upsampling and feature F s123 The features F are obtained by adding each pixel individually and then passing them through the edge residual module. s1234 , feature F s1234 The predicted edge feature map is obtained by inputting the Sigmoid activation function, and the edge loss function is calculated. The edge loss function and the region loss function are jointly used to supervise the parameter updates in the network training.

2. The feature enhancement method for image inpainting and detection based on reverse tampering trace enhancement and adaptive edge perception according to claim 1, characterized in that: The feature enhancement is implemented through the following steps: Step a1: Preprocess the input image I using fusion differential convolution to obtain feature F. fdc . Step a2, let the feature F obtained in step a1 be... fdc By using two layers of dilated convolution with different dilation rates, feature F1 is obtained through the first layer. Feature F1 is then compared with feature F... fdc Connect them together, input them into the second layer of dilated convolution to obtain feature F2, and then combine feature F... fdc F1 and F2 are concatenated along the channel dimension to obtain the final feature F. m . Step a3: Perform ordinary convolution on the input image I to obtain feature F. c The first layer is a 5×5 convolution, the second layer is a combination of 3×3 convolution, batch normalization and ReLU activation function, and the third layer is also a combination of 3×3 convolution, batch normalization and ReLU activation function. Step a4, the feature F obtained in step a2 m and the feature F obtained in step a3 c Feature F is obtained by connecting on the channel. cm .

3. The image restoration and detection method based on reverse tampering trace enhancement and adaptive edge perception according to claim 2, characterized in that, The fused differential convolution used in step a1 is a weighted combination of ordinary convolution and differential convolution. For the input image I, the output feature F is obtained through fused differential convolution. fdc The calculation formula is as follows: Where (x,y) represents the coordinates of the current pixel, and (x+u,y+v) represents the coordinates of its neighboring pixels. I(x,y) represents the pixel value of the input image at coordinates (x,y), and I(x+u,y+v) represents the neighboring pixel values ​​at coordinates (x,y). ω fdc This represents a 3×3 convolution kernel, whose weights are obtained from data obtained during network training. The hyperparameter θ∈[0,1] controls the contribution of gradient-based details.

4. The restoration region decision-making method of the image restoration detection method based on reverse tampering trace enhancement and adaptive edge perception according to claim 1, characterized in that: The specific steps for making the repair area decision are as follows: Step b1: The features F extracted by the backbone network ResNet50 are processed through an attention mechanism based on cosine similarity optimization, including global attention and local attention. The features obtained through global attention are... The features obtained through local attention are Step b2, the features obtained in step b1 and characteristics Feature F is obtained by connecting it to feature F on the channel. o . Step b3, for the feature F obtained in step b2 o A third-order upsampling decoding is performed. Between the first and second upsampling, as well as between the second and third upsampling, 3×3 ordinary convolution, batch normalization, and ReLU activation functions are embedded. After the third upsampling, a 3×3 ordinary convolution and Sigmoid activation function are applied to obtain an output of the same size as the input image. The region loss function is then calculated.

5. The image restoration and detection method based on reverse tampering trace enhancement and adaptive edge perception according to claim 4, characterized in that, The global attention used in step b1 selects features with high similarity to F using cosine similarity, and generates new features by weighting the similarity values. The multi-channel feature vector F(i,j,·) of feature F at position (i,j) can be denoted as F ij Calculate the cosine similarity S between it and the multi-channel feature vectors at other locations. ij,nm The calculation formula is as follows: Here, cosine[] represents calculating the cosine similarity of vectors, and || represents taking the modulus. Set the similarity threshold η, and select S. ij,nm >η features regenerate new multi-channel features at position (i,j) The process is shown in the following formula:

6. The image inpainting and detection method based on reverse tampering trace enhancement and adaptive edge perception according to claim 1, characterized in that, The adaptive edge extraction operator used in the adaptive edge sensing has two core principles in its structural design: the center weight value is always 0, and the sum of the weight values ​​of the neighborhood is also 0. The calculation formula is as follows: Where, ω edge For adaptive edge extraction operator, (u,v) is the position index relative to the center point, and the values ​​of u and v are both in the range of {-1,0,1}.

7. The loss function of the image inpainting and detection method based on reverse tampering trace enhancement and adaptive edge perception according to claim 1, characterized in that, The loss function consists of two parts: region loss and edge loss, and the calculation formula is as follows: L total =L region +βL edge Among them, the edge loss L edge For binary cross-entropy loss, the region loss L region We also use binary cross-entropy loss, and further employ focus loss as a supplement. β is a hyperparameter, which we set to 0.5 in this invention.

Images