Deepfake image detection method fusing content-independent features and content-dependent features

Abstract

The application discloses a deep fake image detection method fusing content-independent features and content-dependent features. Firstly, smooth images, texture images and detail images are obtained through Gaussian filters, Gabor filters and wavelet filters respectively, and after fusion, the true and false probability based on frequency domain features is obtained through feature pyramid network processing. Secondly, difference images and noise images are obtained through unsharp mask and DnCNN denoising model, and after fusion, the true and false probability based on spatial domain features is also obtained through feature pyramid network processing. The frequency domain and spatial domain features jointly constitute the content-independent features of the image. In addition, the content-dependent features of the image are extracted through a ViT model to obtain the true and false probability based on the content-dependent features. Finally, the three kinds of true and false probabilities are weighted and fused to output the true and false judgment result of the image. The application performs excellently in cross-generation model and different data source fake image detection, and has strong generalization and universality.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and forgery detection, and particularly relates to a deep fake image detection method fusing content-independent features and content-dependent features. BACKGROUND

[0002] Artificial Intelligence Generated Content (AIGC) is a technology that learns from large-scale data sets to create new original content through complex algorithms, models, and rules. This technology has a wide range of applications in text and image generation, game development, and film and television production, bringing convenience to various industries. However, the improper use of this technology has also raised some problems and posed security risks.

[0003] Early image generation technologies, such as Generative Adversarial Networks (GAN), generate high-quality fake images through the adversarial training of generators and discriminators. However, these technologies have problems such as high training difficulty, weak scalability, and insufficient diversity of generated images. Current detection methods for GAN-generated images have made some progress, including using spatial or frequency domain feature differences for detection, such as color space differences, gradient texture features, high-frequency patterns, and noise residuals. In recent years, diffusion models (DM) have emerged, which generate images from noise samples by gradually removing prediction noise, demonstrating better stability, controllable training, and image quality. Since this technology is still in its infancy, there is less related research, and existing detection methods are mostly directly migrated from GAN detectors, but the effect is not good, with low detection accuracy, poor generalization and robustness.

[0004] In addition, existing models often struggle to deal with fake images from unknown data sources and unknown generation models, and the universality and generalization of the models are generally poor. Some research has tried to improve the generalization of the model through data augmentation and dataset expansion, but the improvement effect is limited, and it is difficult to fully utilize various effective features of the generated images. Some research directly migrates existing detectors for cross-model image detection, but the classification accuracy of the related models cannot meet the practical requirements.

[0005] With the rapid progress of generative artificial intelligence technology, models such as generative adversarial networks and diffusion models have been able to create realistic fake images, i.e. deepfake images. Although image generation technology has made significant achievements, the corresponding deepfake image detection technology has lagged behind, especially for images generated by the currently widely used diffusion model, the existing detection methods have obvious deficiencies in accuracy. In addition, most existing detection methods are only for specific image generation models, and lack the ability to detect different sources of fake images. SUMMARY

[0006] The purpose of the present application is to provide a deepfake image detection method that fuses content-independent features and content-dependent features, which can detect images generated by generative adversarial networks and diffusion models with high precision.

[0007] Technical scheme: The deepfake image detection method of the present application fuses content-independent features and content-dependent features, comprising the following steps:

[0008] S1: input the original image I into a Gaussian filter, a Gabor filter and a wavelet filter to obtain a smooth image I smooth , a texture image I Gabor and a detail image I detail . I smooth , I Gabor and I detail are called frequency domain features of the image, and the three features are spliced in the channel dimension to obtain the fused frequency domain feature I freq . I freq is input into a feature pyramid network (FPN), and then processed by a classification head to obtain a true or false probability p freq based on the frequency domain feature;

[0009] S2: input the original image I into an unsharp masking model and a DnCNN (Denoising Convolutional Neural Network) denoising model to obtain a difference image I sharp and a noise image I noise . I sharp and I noise are called spatial domain features of the image, and the two features are spliced in the channel dimension to obtain the fused spatial domain feature I spatial . I spatial is input into a feature pyramid network, and then processed by a classification head to obtain a true or false probability pspatial The three frequency domain features obtained by S1 and the two spatial domain features obtained by S2 are collectively referred to as content-independent features of the image;

[0010] S3: inputting the original image I into a pre-trained visual transformer (ViT) model to extract content-dependent features of the image, replacing the original classification head of the visual transformer (ViT) model with a fully connected layer to obtain a content-dependent feature-based authenticity probability p content ;

[0011] S4: weighting and adding the three authenticity probabilities p freq , p spatial , and p content obtained by S1, S2, and S3 respectively to obtain a final authenticity probability p final , and obtaining an authenticity classification result I final of the image according to p result .

[0012] Further, the step S1 specifically comprises the following steps,

[0013] S1.1: inputting the original image I into a Gaussian filter, performing weighted average processing on the image using the Gaussian filter to reduce high-frequency noise, and obtaining a smoothed image I smooth ;

[0014] S1.2: inputting the original image I into a Gabor filter to capture texture information in different directions of the image, and obtaining a texture image I Gabor ;

[0015] S1.3: inputting the original image I into a wavelet filter to perform image decomposition and reconstruction, and obtaining a detail image I detail ;

[0016] S1.4: concatenating I smooth , I Gabor , and I detail obtained by S1.1, S1.2, and S1.3 respectively in the channel dimension to obtain a fused frequency domain feature I freq , inputting I freq into a feature pyramid network, and then processing it through a classification head comprising a pooling layer, a fully connected layer, and a softmax activation function to obtain a frequency domain feature-based authenticity probability p freq ;

[0017] Further, the step S2 specifically comprises the following steps,

[0018] S2.1: processing the original image I using a de-sharpening mask technique, first performing Gaussian blur processing on the original image I to obtain a smoothed image I smoothSubtracting the smoothed image I from the original image I smooth to obtain a difference image I sharp ;

[0019] S2.2: input the original image I into the DnCNN denoising model to obtain a noise image I by using multi-layer convolution and activation function noise ;

[0020] S2.3: respectively input I sharp and I noise obtained by S2.1 and S2.2 into the fusion network to obtain a fused spatial domain feature I spatial ; spatial input I spatial into the feature pyramid network, and then perform the same classification head processing as S1.4 to obtain a true-false probability p based on the spatial domain feature content .

[0021] Further, in step S3, a content-related feature of the image is extracted by using a pre-trained ViT model. The original classification head of the ViT model is replaced by a fully connected layer with an output dimension of 2 to directly generate a prediction result of binary classification, i.e., a true-false probability p based on the content-related feature of the image content .

[0022] Further, in step S4, the true-false probability p freq based on the frequency domain feature obtained in step S1, the true-false probability p spatial based on the spatial domain feature obtained in step S2, and the true-false probability p content based on the content-related feature obtained in step S3 are weighted and added to obtain a final true-false probability p final . According to p final , a final image true-false classification result I result is obtained.

[0023] The application also discloses a computer device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to realize the steps of the method of the application.

[0024] Beneficial effects: Compared with the prior art, the present application has the following remarkable advantages: the present application extracts the content-independent features of the image by combining multiple filters (Gaussian, Gabor, wavelet filter), unsharp masking technology, and DnCNN denoising model; and uses a fine-tuned pre-trained Vision Transformer (ViT) model to extract the content-dependent features of the image. Among them, the content-independent features contain frequency domain features and spatial domain features, thereby constructing a complete image feature representation. Experimental results show that the model shows higher detection accuracy than existing methods for both GAN-generated images and diffusion model-generated images. Especially for cross-generation model, different data sources of fake images, the model has good detection performance, which proves its excellent generalization and universality. The design of the model enables it to effectively deal with the problem of detecting unknown source deepfake images, which is an important requirement in the field of fake image detection. The model represents a technical innovation in the field of deepfake image detection and provides a new solution for the field. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 For the two-dimensional wavelet decomposition process in the specific embodiment of the present application.

[0026] Figure 2 For the feature pyramid network structure diagram in the specific embodiment of the present application.

[0027] Figure 3 For the DnCNN structure diagram in the specific embodiment of the present application.

[0028] Figure 4 For the ViT model schematic diagram in the specific embodiment of the present application.

[0029] Figure 5 For the detection network structure diagram of fusing content-independent features and content-dependent features in the specific embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions of the present application will be further described below with reference to the accompanying drawings.

[0031] The present application is realized by the following measures: a deepfake image detection model fusing content-independent features and content-dependent features, comprising the following steps:

[0032] S1: input the original image I into a Gaussian filter (Gaussian filter), a Gabor filter (Gabor filter) and a wavelet filter (Wavelet filter) to obtain a smooth image I smooth , a texture image I Gabor and a detail image I detail , respectively. Ismooth , I Gabor and I detail The three features are spliced in the channel dimension to obtain the fused frequency domain feature I freq . freq The input feature pyramid network (FPN) is inputted, and then the classification head is processed to obtain the authenticity probability p freq based on the frequency domain feature;

[0033] S2: The original image I is inputted into the unsharp masking model and the DnCNN (Denoising Convolutional Neural Network) denoising model to obtain the difference image I sharp and the noise image I noise , respectively. sharp and I noise are called spatial domain features of the image, and the two features are spliced in the channel dimension to obtain the fused spatial domain feature I spatial . spatial The input feature pyramid network is inputted, and then the classification head is processed to obtain the authenticity probability p spatial based on the spatial domain feature. The three frequency domain features obtained by S1 and the two spatial domain features obtained by S2 are collectively referred to as content-independent features of the image;

[0034] S3: The original image I is inputted into the pre-trained Vision Transformer (ViT) model to extract the content-related features of the image, and the original classification head of the ViT model is replaced by a fully connected layer to obtain the authenticity probability p content based on the content-related features;

[0035] S4: The three authenticity probabilities p freq , p spatial , and p content obtained by S1, S2, and S3 are weighted and added to obtain the final authenticity probability p final , and the authenticity classification result I final of the image is obtained according to p result ;

[0036] Further, the step S1 specifically includes the following steps,

[0037] S1.1: The original image I is inputted into a Gaussian filter to obtain a smoothed image I smooth .

[0038] Specifically, the Gaussian filter is used to smooth the image and reduce high-frequency noise, and the two-dimensional Gaussian distribution density function is as follows:

[0039]

[0040] In the formula, G(x, y) represents the value of the Gaussian function at the position (x, y), that is, the weight of the position. Sigma is the standard deviation of the Gaussian distribution, and mu represents the mean of the Gaussian distribution. The application adopts a 3*3 size convolution kernel, and the standard deviation and the mean are both set to 1, and the Gaussian convolution kernel shape is as follows:

[0041]

[0042] The Gaussian convolution kernel is normalized, and the sum of the weights is 1, so that the total brightness of the image remains unchanged. The image is processed by applying the above Gaussian convolution kernel to weighted average. The pixels in the central region have a larger weight and make a more significant contribution to the final result, while the pixels in the edge region have a smaller weight, thereby achieving image smoothing and low-frequency feature extraction.

[0043] S1.2: input the original image I into the Gabor filter to obtain a texture image I Gabor ;

[0044] The Gabor filter detects the texture features in a specific direction in the image by combining a Gaussian function and a sine wave function. The mathematical expression of the Gabor filter is as follows:

[0045]

[0046] In the formula, x' and y' are the coordinates after rotation transformation, lambda represents the wavelength of the filter, theta represents the direction of the parallel stripes of the Gabor function, psi represents the phase shift of the sine wave, sigma represents the standard deviation of the Gaussian function, and gamma represents the space aspect ratio.

[0047] The application sets the Gabor filter kernel size to 31*31, the wavelength lambda to 10, the phase shift psi to 0, the standard deviation sigma to 4, and the space aspect ratio gamma to 0.5. In order to comprehensively consider the features in different directions, the application adopts a maximum value merging strategy of multi-directional response, uses 16 Gabor filter kernels in different directions, and sets their stripe directions theta to be uniformly distributed between 0° and 360°. Finally, the Gabor kernel is normalized as follows, thereby ensuring that the filter has consistent response strength in different directions. Through the above settings, the Gabor filter can capture multi-directional texture information in the image.

[0048]

[0049] S1.3: input the original image I into the wavelet filter to obtain a detail image I detail ;

[0050] Haar is used as the base function of wavelet filtering in the present application, and the image is decomposed and reconstructed by two-dimensional wavelet transform, and the two-dimensional wavelet decomposition process is as shown in Figure 1 The process of wavelet decomposition and reconstruction is as shown in the following formula:

[0051] DWT(I)→(LL,LH,HL,HH)

[0052] I detail =∑(|LH|+|HL|+|HH|)

[0053] In the formula, I represents an input image, LL represents a low-frequency part of the image, that is, the overall structure and large-scale features of the image; LH, HL and HH respectively represent high-frequency components of the image in the horizontal direction, the vertical direction and the diagonal direction. The present application first decomposes the original input image by two-dimensional wavelet transform to obtain a low-frequency vector LL and high-frequency vectors LH, HL and HH in three directions. Then, the high-frequency components in the three directions are combined to obtain a complete high-frequency image, that is, a detail image I detail Before applying the wavelet transform, the image pixel value is normalized to the range of [0, 255] to ensure the stability and consistency of the image data in the transformation process. The result after the wavelet transform is also de-normalized to ensure that the image data is suitable for further processing and analysis.

[0054] S1.4: I smooth , I Gabor , I detail obtained by S1.1, S1.2 and S1.3 respectively are spliced in the channel dimension to obtain the fused frequency domain feature I freq , I freq is input into the feature pyramid network, and then processed by the classification head containing the pooling layer, the fully connected layer and the softmax activation function to obtain the authenticity probability p freq based on the frequency domain feature, which is in the range of [0, 1]. The structure of the feature pyramid network is as shown in Figure 2 .

[0055] Further, the step S2 specifically comprises the following steps,

[0056] S2.1: obtaining a difference image I sharp by the anti-sharpening mask model from the original image I.

[0057] The present application first divides the original image I into three color channels I R , I G , I B , and performs Gaussian blur processing on I R , I G , I B respectively to generate smooth images Ismooth_R , I smooth_G , I smooth_B For each color channel, the smoothed image is subtracted from the original image, and the results are recorded as I diff_R , I diff_G , I diff_B , that is I diff_x =I x -I smooth_x Then the three color channels i diff_R , I diff_G , I diff_B Add together to get the complete I diff Here we set a threshold T = 50, and the final difference image I sharp Only I is retained diff The pixel values ​​in the image are greater than or equal to T, and the other pixel values ​​are set to 0. The above process can be expressed as:

[0058]

[0059] S2.2: Input the original image I into the DnCNN denoising model, and obtain the noisy image I through multiple layers of convolution and activation functions. noise ;

[0060] The DnCNN structure of the present invention is as follows Figure 3 As shown in the figure, the blind denoising model (DnCNN with Blind NoiseLevel) is selected to adapt to noises of different types and intensities, where the number of layers of the model and the noise level of the test set are set to 20 and 25 respectively. For each original image I, it is separated into I R , I G and I B Three color channels. Apply DnCNN to each channel to get the predicted noise image I noise_R , I noise_G and I noise_B The noise images of each color channel are spliced ​​in the channel dimension to obtain the complete noise image I noise .

[0061] S2.3: The I obtained by S2.1 and S2.2 sharp with I noise The fused spatial domain features I are obtained by splicing in the channel dimension spatial , will I spatial Input the feature pyramid network, and then go through the same classification head processing as S1.4 to obtain the true and false probability p based on spatial domain features. spatial , its value range is [0,1].

[0062] Furthermore, in step S3, the Vision Transformer (ViT) model is used to extract content-related features of the image. The structure of the ViT model used is as follows: Figure 4 As shown. Select the ViT "vit_b_16" pre-trained model and replace the original classification head of the model with a fully connected layer with an output dimension of 2, outputting the true or false probability p based on content-related features. content , its value range is [0,1].

[0063] Furthermore, in step S4, the three true and false probabilities p obtained by S1, S2, and S3 are freq 、p spatial 、p content The final probability of authenticity p is obtained by adding them together through the weighted voting mechanism. final . freq 、p spatial 、p content The weights are recorded as weight1, weight2, and weight3, and the initial values ​​of the three weights are set to 0.4, 0.2, and 0.4 respectively. final =weight1*p freq +weight2*p spatial +weight3*p content The overall structure of the model is as follows Figure 5 As shown in Figure 2. During the training process, the model can automatically adjust the weights according to the contribution of different features to better balance the influence of each branch. The weighted voting mechanism can comprehensively consider the feature information from different branches and reduce possible conflicts, thereby maintaining the high performance of the model when processing complex and diverse forged images. Finally, the true and false probability p is used to determine the true and false probability of the model. final The output of the model is obtained, and the image is classified into the category with a larger probability of authenticity, thereby determining its authenticity classification result.

Claims

1. A deepfake image detection method that fuses content-agnostic features with content-dependent features, characterized in that, The method comprises the following steps: Step S1: input the original image I into a Gaussian filter, a Gabor filter and a wavelet filter respectively to obtain a smooth image I smooth , a texture image I Gabor and a detail image I detail respectively; I smooth , I Gabor and I detail are called frequency domain features of the image, and the three features are spliced in the channel dimension to obtain a fused frequency domain feature I freq ; input I freq into a feature pyramid network FPN, and then perform processing through a classification head to obtain a true-false probability p freq based on the frequency domain feature; Step S2: input the original image I into the deblurring mask model and the DnCNN denoising model to obtain difference image I sharp and noise image I nois e; I sharp and I noise The two features are spliced in the channel dimension to obtain the fused spatial domain feature I spatial ; I spatial is input into the feature pyramid network FPN, and then processed by the classification head to obtain the authenticity probability p based on the spatial domain feature spatial ; The three frequency domain features obtained in step S1 and the two spatial domain features obtained in step S2 are collectively referred to as content-independent features of the image; Step S3: input the original image I into the pre-trained visual transformer ViT model to extract content-related features of the image, replace the original classification head of the visual transformer ViT model with a fully connected layer, and obtain a true-false probability p based on the content-related features content ; Step S4: the three probabilities p freq , p spatial , p content obtained in steps S1, S2, S3 are weighted and added to obtain the final probability p final of authenticity, and according to p final the authenticity classification result I result of the image is obtained.

2. The method of claim 1, wherein the deepfake image detection method fuses content- independent features and content-dependent features. The step S1 specifically comprises the following steps: S1.1: input the original image I into a Gaussian filter, perform weighted average processing on the image using the Gaussian filter, reduce high-frequency noise, and obtain a smoothed image I smooth ; S1.2: input the original image I into the Gabor filter, capture the texture information in different directions in the image, and obtain a texture image I Gabor ; S1.3: input the original image I into the wavelet filter, perform image decomposition and reconstruction, and obtain the detail image I detail ; S1.4: I smooth Gabor detail In the channel dimension, the fused frequency domain features I freq freq are input into the feature pyramid network, and then processed by the classification head including the pooling layer, the fully connected layer and the softmax activation function to obtain the authenticity probability p freq based on the frequency domain features.​​​ 3. A deepfake image detection method that fuses content-agnostic features with content-dependent features according to claim 2, characterized in that, The step S1.1 specifically is: the Gaussian filter is used for smoothing the image and reducing high-frequency noise, and a two-dimensional Gaussian distribution density function is as follows: In the formula, G(x, y) represents a value of the Gaussian function at a position (x, y), that is, a weight of the position, sigma is a standard deviation of the Gaussian distribution, and mu represents a mean value of the Gaussian distribution; The Gaussian convolution kernel is normalized, the sum of the weights is 1, the total brightness of the image is kept unchanged, and the image is subjected to weighted average processing by applying the Gaussian convolution kernel.

4. The method of claim 2, wherein the content-independent features are fused with the content-dependent features. The step S1.2 specifically is: the Gabor filter is used for detecting texture features in a specific direction of the image by combining a Gaussian function and a sine wave function, and a mathematical expression of the Gabor filtering is as follows: In the formula, x' and y' are coordinates after rotation transformation, lambda represents a wavelength of the filter, theta represents a direction of parallel stripes of the Gabor function, psi represents a phase offset of the sine wave, sigma represents a standard deviation of the Gaussian function, and gamma represents a space aspect ratio; A maximum value merging strategy of multi-direction response is adopted, 16 Gabor filter kernels in different directions are used, and stripe directions theta of the 16 Gabor filter kernels are uniformly distributed between 0° and 360°; and finally, the Gabor kernels are normalized, as follows: In the formula, kernel represents an original weight matrix of the filter, and sigma kernel represents a sum of all elements in the matrix; a weight matrix normalized kernel obtained through the normalization operation maintains consistent response intensity in different directions.

5. The method of claim 2, wherein the content-independent features are fused with the content-dependent features. The step S1.3 specifically is: Haar is used as a base function of wavelet filtering, two-dimensional wavelet transform is used for wavelet decomposition and reconstruction of the image, and a process of the wavelet decomposition and reconstruction is as follows: DWT(I)→(LL, LH, HL, HH) I detail =∑(LH|+|HL|+|HH|) In the formula, I represents an input image, LL represents a low-frequency part of the image, that is, an overall structure and large-scale features of the image, LH, HL, and HH respectively represent high-frequency components of the image in a horizontal direction, a vertical direction and a diagonal direction; before the wavelet transform is applied, the image pixel value is standardized to a range of [0, 255]; and the result after the wavelet transform is subjected to inverse standardization.

6. The method of claim 1, wherein the method is a deepfake image detection method that fuses content-agnostic features and content-dependent features, and wherein the method comprises: The step S2 specifically comprises the following steps: S2.1: process the original image I using the de-sharpening mask technique, first perform Gaussian blur processing on the original image I to obtain a smoothed image I smooth , and then subtract the smoothed image I from the original image I smooth to obtain a difference image I sharp ; S2.2: input the original image I into the DnCNN denoising model, and obtain the noise image I by using multi-layer convolution and activation function noise ; S2.3: I sharp with I noise In the channel dimension, the fused spatial domain feature I spatial , I spatial The input feature pyramid network is input into the same classification head as S1.4 to obtain the authenticity probability p based on the spatial domain feature spatial .

7. A deepfake image detection method that fuses content-agnostic features with content-dependent features according to claim 6, characterized in that, Step S2.1 is specifically: dividing the original image I into three color channels I R , G , B , and respectively performing Gaussian blur processing on I R , G , B to generate smooth images I smooth_R , smooth_G , smooth_B . For each color channel, the original image is subtracted from the smooth image, and the results are respectively denoted as I diff_R , diff_G , diff_B , that is, I diff_x = I x -I smooth_x . Then, I diff_R , I diff_G , and I diff_B of the three color channels are added to obtain the complete I diff ; a threshold T = 50 is set, and only the pixel values greater than or equal to T in I sharp are retained in the final difference image I diff , and other pixel values are set to 0. The above process is represented as:

8. The method of claim 1, wherein the method is a deepfake image detection method that fuses content-agnostic features and content-dependent features, and wherein the method further comprises: In step S3, the content-related features of the image are extracted using the pre-trained ViT model, and the original classification head of the ViT model is replaced with a fully connected layer with an output dimension of 2 to directly generate a prediction result of binary classification, i.e., a probability p of authenticity based on the content-related features of the image content .

9. The method of claim 1, wherein the method is a deepfake image detection method that fuses content-agnostic features and content-dependent features, and wherein the method further comprises: Step S4 is specifically: setting weights of p freq , p spatial , and p content as weight1, weight2, and weight3 respectively, and setting initial values of the three weights; and the final probability p final = weight1*p freq + weight2*p spatial + weight3*p content .

10. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, causes the processor to perform the method of any one of claims 1 to 9. The processor executes the computer program to implement the steps of the method in claim 1.

Images