Image tampering positioning method based on conditional diffusion model

Abstract

The invention discloses an image tampering positioning method based on a conditional diffusion model. The conditional diffusion model comprises a conditional network and a reverse denoising network. Wherein the condition network can utilize different noise states in the denoising process to provide fine control over denoising operation of different time steps, destructive noise in the image is effectively resisted, the robustness of post-processing is improved, and the positioning capability of the model is improved. The reverse denoising network comprises an encoder, a decoder and a conditional feature injection module, and the conditional feature injection module is used for more effectively injecting evidence obtaining conditional features into the denoising network and finely controlling the denoising process of the mask. According to the method, tampering localization is described as a conditional denoising process, evidence obtaining traces in different denoising stages are fully mined by utilizing the generation capability and the detail preservation capability of the conditional diffusion model, the mask is gradually recovered under condition guidance, and an effective method is provided for image tampering localization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information security, and particularly relates to an image tampering positioning method based on a conditional diffusion model. BACKGROUND

[0002] With image editing and generation tools becoming increasingly popular and easy to use, image content can be easily tampered with without leaving obvious traces. However, malicious use of image tampering for fraud or false propaganda to seek benefits will have a great negative impact on society.

[0003] Traditional image tampering detection methods mostly rely on hand-designed features or statistical rules, and are difficult to cope with diversified tampering methods in complex scenes. In recent years, deep learning-based tampering detection methods have attracted widespread attention, which can automatically learn tampering features and improve detection performance. Specifically, in the literature “MVSS-Net: Multi-view multi-scale supervised networks for image manipulation detection”, Dong Chengbo et al. constructed a multi-scale supervision strategy to mine tampering traces from tampering edges and regional noise, and improved the performance of the detection model. In the literature “TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization”, Guillaro Fabrizio et al. used a contrast learning technology to construct a noise-sensitive fingerprint extraction network, which can simultaneously capture camera noise fingerprints and image editing information, and combine them with pixel information to fuse high-level and low-level tampering traces. In the invention patent “An image tampering detection method based on a cross-window self-attention correlation network” (publication number CN118711008A), Lu Tongwei et al. designed a network structure based on cross-window self-attention correlation, including local and global feature extraction modules and a prediction module for tampering detection. In the patent application “An image tampering positioning method based on VisionTransformer” (publication number CN118155021A), Liu Zhisong et al. detected the inconsistency of local noise of real images and tampering regions of tampered images, and used Vision Transformer to increase the receptive field, thereby improving the positioning ability.

[0004] Diffusion model has become an important technology for image generation due to its powerful ability to model image pixel distribution, and has shown great potential in image segmentation tasks. In the literature "Segdiff: Image segmentation with diffusion probabilistic models", Amit Tomer et al. use diffusion model as conditional generation to solve the image segmentation problem by extending the U-Net encoder. The positioning of the tampered image is similar to image semantic segmentation, which assigns a class label to each pixel. Diffusion model can learn the potential distribution and detail change process in the image by gradually recovering the original data from noise data, and is expected to capture subtle tampering traces.

[0005] However, these methods still face many challenges. When encountering more complex tampering methods and information loss caused by post-processing operations, existing methods often show limited detection performance, and still lack satisfactory generalization ability. SUMMARY

[0006] The purpose of the present application is to provide an image tampering positioning method based on a conditional diffusion model.

[0007] The technical solution to achieve the purpose of the present application is as follows:

[0008] The image tampering positioning method based on the conditional diffusion model comprises the steps of training the conditional diffusion model, and using the trained conditional diffusion model to perform tampering positioning on the image to be tested.

[0009] The conditional diffusion model comprises K conditional feature extraction modules; the first conditional feature extraction module comprises a first convolutional layer, a second convolutional layer, a time step information fusion module and a Transformer encoder, and the other conditional feature extraction modules comprise convolutional layers, time step information fusion modules and Transformer encoders connected in sequence; an image x and a noise mask image m t After being subjected to the first convolutional layer and the second convolutional layer respectively, the input data of the time step information fusion module is obtained by element-wise addition, and the output data of the time step information fusion module is subjected to the Transformer encoder to obtain the conditional feature F1; F1 is subjected to the second conditional feature extraction module to obtain the conditional feature F2, …, F K-1 is subjected to the Kth conditional feature extraction module to obtain the conditional feature F K ;

[0010] The time step information fusion module adjusts the dimension of the input data, performs normalization, and then splices the time embedding p time to obtain the output data; the time embedding p time is obtained according to the noise mask image m tThe time step t is obtained through sine and cosine position encoding;

[0011] The conditional diffusion model further includes K feature size alignment modules and K-1 convolutional layers; each conditional feature extraction module is connected to a feature size alignment module; c K With c K-1 After concatenation, c′ is obtained by passing the concatenation through a convolutional layer. K-1 , c′ K-1 With c K-2 After concatenation, c′ is obtained through another convolutional layer. K-2 By analogy, we obtain c1′ as the evidence-gathering condition feature c. f The c K , by conditional feature F K The result is obtained after alignment by the Kth feature size alignment module;

[0012] The conditional diffusion model also includes an encoder, a channel attention module, and a decoder; noise mask image m t The encoder obtains feature z, and z is compared with the evidence-gathering condition feature c. f After concatenation, z is obtained through the channel attention module. c , z c The noise mask image m is obtained by the decoder. t-1 ;

[0013] The training conditional diffusion model includes:

[0014] Construct a training set consisting of multiple images x and their mask images m0. For each mask image m0, randomly select time steps t from 1 to T and add noise to obtain a noise mask image m. t ;

[0015] Image x and its noise mask m in the training set t The noise mask image m is obtained through the conditional diffusion model. t-1 As a denoising mask image

[0016] The conditional diffusion model is trained using the training set, and the model parameters are updated using an optimization algorithm in conjunction with the loss function until the preset convergence condition is met, thus completing the training.

[0017] The loss function is: in,

[0018]

[0019] In the formula, N is the total number of pixels in the mask image. m0(i,j) are respectively The value of pixel (i,j) in m0; λ1 and λ2 are hyperparameters;

[0020] The step of using a trained conditional diffusion model to locate tampering in the test image includes:

[0021] Let the image to be tested be image x, and let the noise image of the same size as the image to be tested and conforming to a standard Gaussian distribution be the noise mask image m. t Set the time step t = T′; image x and noise mask m t The denoised mask image is obtained from the trained conditional diffusion model through T′ iterations. right Binarization is performed to obtain the tampering location result of the image under test.

[0022] Preferably, the second convolutional layer of the first conditional feature extraction module is a zero convolutional layer.

[0023] In the above technical solution, the encoder is a U-Net encoder and the decoder is a U-Net decoder. Alternatively, the encoder is a U-ViT encoder and the decoder is a U-ViT decoder.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The tampering location is described as a conditional denoising process. By utilizing the generative and detail-preserving capabilities of the diffusion model, the evidence traces in different denoising stages are fully explored, and the mask is gradually restored under conditional guidance, providing an effective method for image tampering location.

[0026] 2. By applying multi-scale feature fusion to obtain evidence-gathering condition features, the model can be guided to focus more accurately on the tampered area. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the forward diffusion of training samples.

[0028] Figure 2 This is a schematic diagram of the conditional diffusion model.

[0029] Figure 3 This is a schematic diagram of the evidence collection condition feature extraction module.

[0030] Figure 4 This is a schematic diagram of the multi-scale feature fusion module.

[0031] Figure 5 This is a structural diagram of the conditional feature injection module. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0033] The application provides an image tampering positioning method based on a conditional diffusion model, and a specific implementation is as follows:

[0034] S1. Image sample set construction and preprocessing.

[0035] S1.1. The open source dataset MS COCO is taken as a training set, and the CASIA, RSIID and SciSp-C are taken as test sets, and the tampering types include splicing, copying and moving and deleting;

[0036] S1.2. Data set preprocessing, data augmentation is performed on the data set, including random scaling, horizontal flipping, vertical flipping and rotation, and then normalization processing is performed, and the size of the image is uniformly adjusted to WxH image, wherein W=384 and H=384.

[0037] S2. Forward diffusion process: as shown in Figure 1 , the core idea is to add Gaussian noise to the mask m0 corresponding to the training sample constructed in step S1 to gradually destroy the structure and content of the original image.

[0038] A time step t is randomly selected from 1 to T, and the mask m0 is added with noise to obtain a noise mask image m t after noise addition, t The calculation formula is as follows:

[0039]

[0040] In the formula, β t represents a predefined parameter for controlling the amount of Gaussian noise added at each step in the forward diffusion process, is the added Gaussian noise, represents a Gaussian distribution, and I is an identity matrix.

[0041] S3. The structure of the conditional diffusion model is as shown in Figure 2 :

[0042] Conditional feature extraction: input the image x and the corresponding noise mask m t into the conditional network to extract the forensic condition features.

[0043] S3.1. The image x and the noise mask m t are input into K forensic condition feature extraction modules to obtain multi-scale features F=[F1, F2,…, F K ] of the image x; in the formula, K is the number of forensic condition feature extraction modules, and in this embodiment, K is set to 4.

[0044] As shown in Figure 3As shown, the first forensic condition feature extraction module includes a multi-view aggregation layer, a time step information fusion module, and a Transformer encoder, and the other K-1 forensic condition feature extraction modules include a convolutional layer, a time step information fusion module, and a Transformer encoder. In this embodiment, the PVTv2-B4 pre-trained weights are used to initialize the parameters of the forensic condition feature extraction modules.

[0045] The image x and the noise mask m t The first forensic condition feature extraction module is input, and the multi-view aggregation layer is used to introduce the noise mask, which can be:

[0046] x MVA =Conv(x)+Conv(m t )

[0047] The multi-view aggregation layer used in this embodiment is as follows:

[0048] x MVA =Conv(x)+Zero(m t )

[0049] In the formula, Conv(·) represents a convolutional layer, and Zero(·) represents a zero convolutional layer, which is initialized to zero to prevent the negative effects of randomly initialized parameters in the early training stage of multi-view aggregation.

[0050] Then the dimensions are adjusted to obtain the feature embedding p j , the time embedding p time is calculated using the sine-cosine position encoding of the time step t, the feature embedding p j and the time embedding p time are input to the time step information fusion module:

[0051] p j =Reshape(x MVA )

[0052] p concat =Cat(Norm(p j ),p time )

[0053] In the formula, Reshape(·) is a dimension adjustment operation, Cat(·) is a concatenation operation, and Norm(·) represents a normalization operation.

[0054] p concat is input to the Transformer encoder to obtain F1, and F k-1 is sequentially input to the kth forensic condition feature extraction module to obtain F k (k=2,3,…,K).

[0055] The conditional network can utilize different noise states in the denoising process, provide fine control of denoising operations at different time steps, effectively resist destructive noise in the image, improve the robustness of post-processing, and improve the positioning ability of the model.

[0056] The multi-scale features F = [F1, F2, …, F K ] are input to the multi-scale feature fusion module for layer-by-layer fusion, as shown in Figure 4 First, F K and F K-1 are fused, and then the fused features are fused with F K-2 , and so on, finally the previously fused features are fused with F1 to obtain the forensic condition feature c f , and each step of the fusion process is as follows:

[0057] c i = c(F i )

[0058] c′ K = c K

[0059] c i ′= Conv(Cat(c i ′ +1 ,c i )), i ∈ {K-1, K-2, …, 1}

[0060] In the formula, Conv(·) represents convolution operation, Cat(·) is concatenation operation, FDA(·) is feature size alignment module, including convolution, ReLU activation and up-sampling operation, c f = c1′.

[0061] S4. Reverse denoising network: input the noise mask map m t into the reverse denoising network, and gradually denoise under the guidance of the forensic condition feature c f .

[0062] The reverse denoising network includes a U-Net encoder, a U-Net decoder (or a U-ViT encoder and a U-ViT decoder), and a conditional feature injection module, wherein the conditional feature injection module is used to more effectively inject the forensic condition feature c f into the denoising network and finely control the denoising process of the mask.

[0063] The noise mask map m t is input into the reverse denoising network, and at each time step of the reverse denoising process, the reverse denoising network will be controlled by the forensic condition feature c f to gradually reconstruct the mask from Gaussian noise.

[0064] S4.1 input the noise mask m t to the U-Net encoder to obtain the output feature z;

[0065] S4.2 input the output feature z to the conditional feature injection module, as shown in Figure 5 , the output feature z and the forensic conditional feature c f are spliced to obtain z f . Input z f to a channel attention module, and the calculation formula is as follows:

[0066] z f =Cat(z,c f )

[0067]

[0068] In the formula, Avg(·) represents the average pooling layer; the above operations are performed to obtain z c ;

[0069] S4.3 input z c to the U-Net decoder to output the denoising result m t-1 .

[0070] The loss function of the conditional diffusion model is:

[0071]

[0072]

[0073] In the formula, is the cross-entropy loss function, is the set similarity measure loss function, and N is the total number of pixels, and m0(i,j) are the predicted value and the true value of the pixel at position (i,j) in the image. That is, if m0(i,j) is 1, the pixel belongs to the tampered region. Otherwise, it is 0, and the pixel belongs to the non-tampered region. λ1 and λ2 are hyperparameters for adjusting the two loss functions, which are set to 1 and 0.3 respectively in this embodiment.

[0074] When training the conditional diffusion model using the training set:

[0075] S1. Sample a batch from the training samples, including images and their masks, which is set to 32 in this embodiment;

[0076] S2. Randomly select a time step t from 1 to T, and sample a random noise for the mask m0 corresponding to each image x in the batch, and add it to the mask to obtain the noise map mt In the present embodiment, the training period T is set to 1000;

[0077] S3. inputting the image x and the noise m t to the conditional network, inputting the noise m t to the reverse denoising network, and the reverse denoising network outputting a predicted mask

[0078] S4. calculating the loss of all samples in a batch and averaging, and then back-propagating to update the weights.

[0079] S5. repeating the steps S1-S4 until a preset convergence condition (the loss function no longer decreases or a preset training round is reached) is met, to obtain a trained positioning model. In the present embodiment, the training round is set to 150.

[0080] When the trained conditional diffusion model is used:

[0081] S1. pre-processing a to-be-detected image;

[0082] S2. sampling a noise image with a size consistent with the to-be-detected image from a standard Gaussian distribution as a noise mask corresponding to the to-be-detected image

[0083] S3. starting from t=T to t=1, iteratively inputting the to-be-detected image and the noise image to the trained reverse denoising network for denoising to obtain In the present embodiment, T is set to 10 when used;

[0084] S4. obtaining the mask image after the loop ends

[0085] S5. binarizing the mask image to obtain the tampering positioning result of the image.

[0086] In order to illustrate the performance of the above scheme of the present application, relevant experiments are performed.

[0087] Experimental parameters refer to: the data set uses three data sets CASIA, RSIID and SciSp-C in the field of image tampering positioning to test the positioning performance of the model. The training batch size is set to 32, the optimizer uses Adam optimizer, the learning rate is set to 0.001, and the weights of the loss functions and are respectively set to 1 and 0.3. In the experiment, F1(%) is used as the evaluation index of the model, and the existing image tampering positioning method is compared to verify the effectiveness of the present application.

[0088] ​Table 1 Performance test of the present application and other image tampering positioning algorithms

[0089] CASIA (%) RSIID (%) SciSp-C (%) RRU-Net 40.8 80.9 31.9 ManTra 18.0 63.4 20.7 IF-OSN 55.3 77.8 14.3 PSCC-Net 52.0 67.8 6.5 CAT-Net2 75.2 80.0 22.9 MVSS 52.8 78.8 15.8 TruFor 73.7 71.1 17.5 CoDE 72.3 77.5 22.6 The present invention 76.9 90.4 60.1

[0090] Table 1 shows the experimental results of the present application in the in-house experimental setting on three data sets, with the best results in bold. The compared method RRU-Net comes from the literature RRU-Net: The ringed residual UNet for image splicing forgery detection, ManTra comes from the literature ManTra-Net: Manipulation tracing network for detection and localization of image forgeries with anomalous features, IF-OSN comes from the literature Robust image forgery detection over online social network shared images, PSCC-Net comes from the literature PSCC-Net: Progressive spatio channel correlation network for image manipulation detection and localization, CAT-Net2 comes from the literature Learning JPEG compression artifacts for image manipulation detection and localization, MVSS comes from the literature MVSS-Net: Multiview multi-scale supervised networks for image manipulation detection, TruFor comes from the literature TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization, and CoDE comes from the literature Employing Reinforcement Learning to Construct a Decision-Making Environment for Image Forgery Localization. The experimental results show that the present application has more accurate positioning performance and generalization ability compared with other positioning schemes. Among them, the RSIID and SciSp-C data sets contain images mainly from the biomedical field, and the image content is mainly manifested as microscopic images, strip images and other image types, which have great differences in visual features compared with the natural images contained in the CASIA data set.Even in such image structure complex and lack of obvious semantic information scene, the application can still maintain stable positioning performance, has good generalization ability, can adapt to different types of image tamper detection needs.

Claims

1. An image tampering localization method based on a conditional diffusion model, characterized in that, The steps include: training a conditional diffusion model, and using the trained conditional diffusion model to locate tampering in the image under test; The conditional diffusion model includes K conditional feature extraction modules; the first conditional feature extraction module includes a first convolutional layer, a second convolutional layer, a time step information fusion module, and a Transformer encoder, and the other conditional feature extraction modules include convolutional layers, time step information fusion modules, and Transformer encoders connected in series. Image x, noise mask map m t The input data of the time step information fusion module is obtained by element-wise addition after the first convolutional layer and the second convolutional layer, respectively, and the output data of the time step information fusion module is obtained by a Transformer encoder to obtain the conditional feature F1; F1 is obtained by the second conditional feature extraction module to obtain the conditional feature F2, …, F K-1 The conditional feature F is obtained by the Kth conditional feature extraction module. K ​ The time step information fusion module adjusts the dimensions of its input data, normalizes it, and then merges it with the time-embedded p. time The output data is obtained after concatenation; the time embedding p time According to the noise mask image m t The time step t is obtained through sine and cosine position encoding; The conditional diffusion model further includes K feature size alignment modules and K-1 convolutional layers; each conditional feature extraction module is connected to a feature size alignment module; c K With c K-1 After concatenation, c′ is obtained by passing the concatenation through a convolutional layer. K-1 , c′ K-1 With c K-2 After concatenation, c′ is obtained through another convolutional layer. K-2 By analogy, we obtain c′1 as the evidence-gathering feature c. f The c K , by conditional feature F K The result is obtained after alignment by the Kth feature size alignment module; The conditional diffusion model also includes an encoder, a channel attention module, and a decoder; noise mask image m t The encoder obtains feature z, and z is compared with the evidence-gathering condition feature c. f After concatenation, z is obtained through the channel attention module. c , z c The noise mask image m is obtained by the decoder. t-1 ; The training conditional diffusion model includes: Construct a training set consisting of multiple images x and their mask images m0. For each mask image m0, randomly select time steps t from 1 to T and add noise to obtain a noise mask image m. t ; Image x and its noise mask m in the training set t The noise mask image m is obtained through the conditional diffusion model. t-1 As a denoising mask image The conditional diffusion model is trained using the training set, and the model parameters are updated using an optimization algorithm in conjunction with the loss function until the preset convergence condition is met, thus completing the training. The loss function is: in, In the formula, N is the total number of pixels in the mask image. m0(i,j) are respectively The value of pixel (i,j) in m0; λ1 and λ2 are hyperparameters, respectively; The step of using a trained conditional diffusion model to locate tampering in the test image includes: Let the image to be tested be image x, and let the noise image of the same size as the image to be tested and conforming to a standard Gaussian distribution be the noise mask image m. t Set the time step t = T′; image x and noise mask m t The denoised mask image is obtained from the trained conditional diffusion model through T′ iterations. right Binarization is performed to obtain the tampering location result of the image under test.

2. The conditional diffusion model as described in claim 1, characterized in that, The second convolutional layer of the first conditional feature extraction module is a zero convolutional layer.

3. The conditional diffusion model as described in claim 1, characterized in that, The encoder is a U-Net encoder, and the decoder is a U-Net decoder.

4. The conditional diffusion model as described in claim 1, characterized in that, The encoder is a U-ViT encoder, and the decoder is a U-ViT decoder.

Images