Depth forgery detection method fusing spatial domain texture difference and frequency domain information

Through the deep forgery detection method that fuses the difference in airspace texture and frequency domain information, the average difference convolution and cross attention mechanism are used to solve the problems of poor detection accuracy and insufficient generalization ability in the existing technology, and the forged image is realized Accurate detection and good generalization ability.

CN118968269APending Publication Date: 2024-11-15HENAN UNIV OF SCI & TECH
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411041806.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing deep forgery detection methods have poor detection accuracy and insufficient generalization capabilities, making it difficult to adapt to changes in different forgery technologies and image quality, resulting in a decrease in detection effect.

Method used

The depth forgery detection method that combines the texture difference and frequency domain information of the forged image is extracted through average differential convolution, and combined with pixel intensity and local relative gradient information, combined with potential artifact information in the frequency domain, and used cross attention. The mechanism performs feature fusion to achieve accurate detection of fake images.

Benefits of technology

It realizes more accurate detection of forged images, has good generalization capabilities, can maintain excellent detection performance in unknown forgery technology, and improves detection accuracy and adaptability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968269A_ABST
    Figure CN118968269A_ABST
Patent Text Reader

Abstract

The invention discloses a depth forgery detection method fusing spatial domain texture difference and frequency domain information. The method comprises the following steps: firstly, preprocessing an input image to obtain a face image; secondly, respectively inputting the face image into the upper branch network and the lower branch network to extract texture difference information and frequency domain potential artifact information in the space domain of the forged image, and outputting a feature map; projecting the feature map into a feature block, splicing the feature block with the CLS token, and outputting a feature vector as the input of a feature fusion module to obtain the CLSS token and the CLSF token after feature fusion; and finally, taking the CLSS token and the CLSF token as input of a multi-layer sensor to obtain a classification result and outputting the classification result. According to the method, spatial domain texture differences and frequency domain information are combined as classification bases, the authenticity images are classified after feature fusion is performed through a cross attention mechanism, and the method has better detection performance and generalization ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of forged image detection, and in particular relates to a deep forgery detection method that integrates spatial texture differences and frequency domain information. Background Art

[0002] In recent years, the generation model has been continuously developed, and the deep fake related technology has gradually matured. There are some bad behaviors that use deep fake technology to manipulate the facial area of ​​the face, such as replacing the facial area, manipulating the facial features, etc. Malicious users can easily generate fake face images or videos, and these fake contents can be quickly spread through social networks. With the frequent occurrence of illegal and criminal acts such as extortion, pornography, and false information in the global "face-changing" incidents, it has brought many risks to the national security field such as social security and national security.

[0003] For the detection of deep fake content, the existing advanced methods mainly use the forged features in the image to achieve binary classification of true and false images by training complex neural network models. However, these detection methods have certain limitations. First, the detection accuracy of the detection method that relies on a single forged feature as the basis for classification will drop significantly when faced with different forgery technologies and different image quality. Secondly, the continuous updating and iteration of deep fake technology has made the realism of forged content continue to improve, making detection more difficult. New forgery technology may introduce new forgery features, and existing detection methods may not be able to adapt to these changes in time, resulting in the failure of the detection method.

[0004] Therefore, a more generalized detection method that can accurately detect forged images is needed. Summary of the invention

[0005] In view of this, in order to solve the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a deep fake detection method that integrates spatial texture differences and frequency domain information, designs average differential convolution and combines pixel intensity information and local relative gradient information to extract texture difference information in the spatial domain of forged images, thereby solving the problems of poor detection accuracy and insufficient generalization ability of existing deep fake detection methods.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is: a deep fake detection method integrating spatial texture difference and frequency domain information, comprising the following steps: S1, obtaining an input image containing facial information, and performing preprocessing to obtain a facial image; S2, inputting the face image into the upper branch network and the lower branch network respectively to extract the texture difference information in the spatial domain of the forged image and the potential artifact information in the frequency domain, and outputting the feature map; S3, project the feature maps output by the upper branch network and the lower branch network into feature blocks, respectively, and concatenate them with the CLS token to output the final feature vector; S4, the final feature vector output by the upper branch network and the lower branch network is used as the input of the feature fusion module, and the CLS_S token and CLS_F token after feature fusion are output; S5. Use the CLS_S token and the CLS_F token as the input of the multi-layer perceptron to obtain the final detection result and output it.

[0007] Furthermore, the upper branch feature extraction network uses the modified Efficient-0 as the backbone network; the face image is combined with the intensity information of the pixel and the local relative gradient information through the average difference convolution to extract the texture difference information in the shallow features of the spatial domain of the forged image, and then the upper branch feature vector is output through the backbone network.

[0008] Furthermore, the lower branch network includes an average grayscale image frequency-aware decomposition module and an Efficient-0 backbone network; the face image is transformed into a spatial domain display by the average grayscale image frequency-aware decomposition module, and then the lower branch feature vector is outputted by the backbone network.

[0009] Furthermore, for the feature map extracted by the backbone network, it is segmented and flattened into a vector, and then the feature map is linearly projected into a feature block to change the dimension of the feature vector, and it is spliced ​​with the newly added CLS token to perform position embedding and output the final feature vector.

[0010] Furthermore, the S5 comprises the following steps: The CLS_S token and the CLS_F token are input into a multilayer perceptron to obtain corresponding scores, the CLS_S score and the CLS_F score are added as the final score of the input image, and then the final score is mapped to the interval of 0 to 1 through the sigmoid function, a threshold is set and the final score is compared with the threshold to obtain the classification result of the face image as the detection result output.

[0011] The beneficial effects of the present invention are: The present invention combines spatial texture differences and frequency domain information as the basis for classification, and classifies true and false images after feature fusion through a cross-attention mechanism, thereby achieving more accurate detection of forged images and having better detection performance. At the same time, it also has good generalization ability for unknown forgery technologies, solving the problems of poor detection accuracy and insufficient generalization ability of existing deep forgery detection methods. The average difference convolution is proposed. It combines the intensity information of pixels and the local relative gradient information to extract the texture difference information in the spatial domain of the forged image. This is used as one of the bases for distinguishing true and false images. Compared with other convolution methods, it can better reflect the texture difference in the forged image, and after replacing the ordinary convolution, it will not change the parameters of the network. In the mining of potential artifact information in the frequency domain, frequency-aware decomposition based on the average grayscale image is used to convert the frequency domain information of the input image into a spatial domain display in order to better mine the artifact information; further, the feature vector after feature extraction is mapped to a patch and concatenated with the CLS token as the input of the feature fusion module, and the feature fusion module is used to further process the feature information to achieve feature fusion of the upper and lower branch networks, making the classification results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 It is the overall structure diagram of the detection method of the present invention; Figure 2 It is a structural diagram of the average difference convolution of the detection method of the present invention; Figure 3 This is a structural diagram of the detection method of the present invention based on the frequency perception decomposition of the average grayscale image. DETAILED DESCRIPTION

[0014] The following specific examples are given to further clearly, completely and in detail illustrate the technical solution of the present invention. This example is the best example based on the technical solution of the present invention, but the protection scope of the present invention is not limited to the following examples.

[0015] Example 1 The present invention provides a deep forgery detection method integrating spatial texture difference and frequency domain information, comprising the following steps: Step 1: Obtain an input image containing facial information and perform preprocessing to obtain a facial image.

[0016] In the pre-processing process, the facial area of ​​the human face is cropped and resized. Optionally, in this embodiment, it is unified to 224*224 pixels.

[0017] Furthermore, the cropped face image is randomly data enhanced. In this embodiment, the face image can be processed by means including adding Gaussian noise, horizontally flipping the image, adjusting the image brightness and contrast, and translating or rotating the image. The image after the data enhancement is then converted into a tensor for subsequent feature extraction.

[0018] Step 2: Input the face image into the upper branch network and the lower branch network respectively to extract the texture difference information in the spatial domain of the forged image and the potential artifact information in the frequency domain, and output the feature map.

[0019] Furthermore, the upper branch feature extraction network uses the modified Efficient-0 as a backbone network; the lower branch network includes an average grayscale image frequency perception decomposition module and an Efficient-0 backbone network; The face image is extracted from the forged image through average differential convolution, which combines the pixel intensity information and local relative gradient information to extract the texture difference information in the shallow spatial features of the image, and then the backbone network outputs the upper branch feature vector (the feature map of the upper branch); the face image is transformed into a spatial domain display through the average grayscale image frequency perception decomposition module, and then the backbone network outputs the lower branch feature vector (the feature map of the lower branch).

[0020] The spatial texture difference information is extracted through average difference convolution, which can better reflect the texture difference in the forged image than other convolution methods, and after replacing the ordinary convolution, the network parameters will not change. Based on the frequency-aware decomposition of the average grayscale image, the frequency domain information of the input image is converted into a spatial domain display to better mine artifact information.

[0021] Step 3: Project the feature maps output by the upper branch network and the lower branch network into feature blocks, respectively, and concatenate them with the CLS token to output the final feature vector; specifically, the following contents are included: For the feature map extracted by the backbone network, it is segmented and flattened into a vector, and then the feature map is linearly projected into a feature block to change the dimension of the feature vector, and it is spliced ​​with the newly added CLS token to perform position embedding and output the final feature vector.

[0022] Step 4: The final feature vectors output by the upper branch network and the lower branch network are used as the input of the feature fusion module, and the CLS_S token and CLS_F token after feature fusion are output.

[0023] Furthermore, the feature fusion module is composed of a stack of Transformer encoders and a cross attention mechanism.

[0024] Step 5: Use the CLS_S token and CLS_F token as the input of the multi-layer perceptron to obtain the final detection and output, which specifically includes the following steps: The CLS_S token and the CLS_F token are input into a multilayer perceptron to obtain corresponding scores, the CLS_S score and the CLS_F score are added as the final score of the input image, and then the final score is mapped to the interval of 0 to 1 through the sigmoid function, a threshold is set and the final score is compared with the threshold to obtain the classification result of the face image as the detection result output.

[0025] Furthermore, the multilayer perceptron consists of a normalization layer and a fully connected layer.

[0026] In summary, the deep fake detection method combining spatial texture difference and frequency domain information proposed in the present invention combines two forgery features, namely spatial texture difference and frequency domain potential artifacts, and uses the designed average differential convolution (ADC) to extract the forged texture difference information. The frequency domain potential artifact information is extracted using frequency-aware decomposition based on the average grayscale image, and after feature fusion through the cross-attention mechanism, more accurate detection of forged images is achieved, and at the same time, it has excellent generalization performance.

[0027] Example 2 The deep fake detection method of the present invention combines spatial texture differences and frequency domain potential artifact information, and includes the following steps: Step 1: Get the input image containing face information, use the face detector MTCNN to crop the face area, and resize it to 224*224. After cropping, randomly perform data enhancement.

[0028] Step 2: Input the face image into the upper branch network to extract the texture difference information in the spatial domain of the forged image; input the face image after the average grayscale image frequency perception decomposition module into the lower branch network to extract the potential artifact information in the frequency domain.

[0029] In this embodiment, the upper branch network replaces the ordinary convolution of Stage 1 in Efficient-0 with average differential convolution (ADC), and the network structure after replacement is shown in Table 1. ADC does not increase the number of network parameters, and its structure is referred to as Figure 1 shown. Table 1 Network structure of texture difference information extraction module

[0030] When the upper branch network performs image processing to extract texture difference information, the following steps are included: A1. For ordinary two-dimensional convolution, from a certain position P in the input feature map x CThe process of extracting the next-level feature map y is mainly to sample the local receptive field and aggregate the sampled values ​​through weighted summation. The process can be formulated as: Where R is the local receptive field, P r Traverse the positions in R, and w represents its weight. On the basis of extracting pixel intensity information through ordinary convolution, during the ADC traversal inside the local receptive field, the difference between each pixel point and the mean value of the receptive field pixel is used as the final pixel value, thereby representing the local relative pixel gradient information of the image; A2. Use ADC to input feature map x from a certain position P C The process of extracting the next level feature map y can be formulated as:

[0031] When the lower branch network performs image processing to extract potential artifact information in the frequency domain, the following steps are included: B1. The image first passes through the average grayscale image frequency perception decomposition module to realize the display of the original image frequency domain information in the form of spatial domain. The average grayscale image frequency perception decomposition module is as follows Figure 2 As shown, the image is first converted into an average grayscale image and DCT transformed. After the dot product operation with the combined mask filter, the inverse DCT transform is performed to a three-channel image displayed in the spatial domain. The process can be formulated as: F=D -1 (X*D*Filters)(1) Where D represents DCT transform, D-1 represents inverse DCT transform, and Filters represents combined mask filter; B2. The combined mask filter is composed of three binary masks added to the learnable filter and then passed through Sigmoid. The binary mask divides the frequency domain image into a low-frequency band (the first 1 / 16 of the spectrum), a mid-frequency band (1 / 16 to 1 / 8 of the spectrum), and a high-frequency band (the remaining spectrum). The learnable filter is used to select the attention area outside the binary mask. The process can be formulated as: Filters=σ(f b +f att )(2) Among them, σ represents the Sigmoid function, f b is the base binary mask, f att represents a learnable filter, which is randomly drawn from a standard normal distribution (mean 0, standard deviation 1). Combining formulas (1) and (2), the frequency-aware decomposition module of the average grayscale image can be formulated as: F=D -1 (X*D*Sigmoid(f b +f att))(3)

[0032] Step 3: Linearly project the feature map extracted by the upper branch of step 2 and the CLS token into a feature block of size 7*7; linearly project the feature map extracted by the lower branch of step 2 and the CLS token into a feature block of size 56*56. Specifically, the following steps are included: C1. In order to utilize the multi-scale information of the image in the subsequent feature fusion, the linear projection of the feature map extracted by the upper branch is first defined as a feature block of size 7*7, and the linear projection of the feature map extracted by the lower branch is a feature block of size 56*56. For the feature map extracted by the backbone network, it is segmented and flattened into a vector, and then these vectors are mapped to a new dimension through the linear layer; C2. Add the classification identifier CLS token to the beginning of the feature vector after the mapping is completed, then perform position embedding and output a final feature vector containing the classification identifier and position information.

[0033] Step 4: Use the feature block in step 3 as the input of the feature fusion module, and output the CLS_S token and CLS_F token after feature fusion.

[0034] In this embodiment, each layer of the feature fusion module is composed of a small-scale Transformer encoder, a large-scale Transformer encoder, and a CrossTransformer. The Transformer is composed of multi-head attention, feedforward network and normalization. The Transformer can customize parameters such as input and output dimensions and the number of multi-head attention.

[0035] Furthermore, step 4 includes the following sub-steps: D1: The input vector passes through the stacked multi-head attention and feed-forward networks, and is finally normalized as the input of CrossTransformer; D2: CrossTransformer first separates the classification tokens of the two input tensors, fuses and updates the two classification tokens through cross attention, and finally concatenates the classification tokens with the original input tensor to achieve feature fusion of the upper and lower branch networks; D3: The feature fusion module consists of the above-mentioned Transformer encoders of different scales and CrossTransformer stacks. The input feature vector is cross-fused through this module, and finally the classification tokens of the upper and lower branches are output for subsequent processing.

[0036] Step 5: Use the upper branch token CLS_S and the lower branch token CLS_F output in step 4 as the input of the multi-layer perceptron to produce the final classification result; Among them, the multilayer perceptron consists of a normalization layer and a fully connected layer. The CLS_S token and the CLS_F token generate corresponding scores through the multilayer perceptron. The CLS_S score and the CLS_F score are added as the final score of the input image. After that, the score is mapped to the interval of 0 to 1 through the sigmoid function, and the threshold is set to 0.5 to generate the final classification result of the input face image.

[0037] Comparative experiment: Experiments were conducted on three public datasets: FF++, DFDC, and Celeb-DF. The experimental results on the three datasets are shown in Tables 2 and 3. Table 2 Experimental results of different methods on FF++ dataset (ACC%) Table 3 Experimental results of different methods on DFDC and Celeb-DF-v2 datasets (AUC%) The results show that in the detection of the four forgery methods of FF++(HQ), the method of the present invention achieved the best accuracy; in the detection of the four forgery methods of FF++(LQ), the method of the present invention achieved the best accuracy for the detection of FS, and the other three methods were also at an advanced level, close to the best detection accuracy results. The detection results on the DFDC and Celeb-DF-v2 datasets show that the AUC (an evaluation index to measure the quality of a binary classification model) of the method of the present invention achieved the best results, indicating that the classification effect of the method of the present invention is better than that of other comparison methods. In summary, the test results conducted in each dataset show that the method of the present invention has achieved good results on the three benchmark datasets and has excellent performance.

[0038] In terms of generalization ability testing, the experimental results of training on FF++(HQ) and testing on DFDC and Celeb-DF-v2 are shown in Tables 4 and 5. Table 4 Experimental results of different methods on DFDC training on FF++(HQ) (%) Table 5 Experimental results (AUC%) of different methods on Celeb-DF-v2 trained on FF++(HQ)

[0039] On the DFDC dataset, the ACC and AUC of the proposed method achieved the best results; on the Celeb-DF-v2 dataset, the AUC of the proposed method achieved the best result. In summary, the generalization test results show that the proposed method achieved the best results on both DFDC and Celeb-DF-v2, which is better than other comparison methods and has excellent generalization ability. The experimental results show that the proposed method has excellent performance in detecting images of unknown forgery types, which confirms that the model has good generalization ability. The above comparative experimental results with existing advanced methods show that the method has good detection performance on various datasets, and also has good generalization ability on unknown forgery techniques.

[0040] The above shows and describes the main features, basic principles and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements according to actual conditions, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A deep fake detection method integrating spatial texture difference and frequency domain information, characterized in that: The following steps are involved: S1, obtaining an input image containing facial information, and performing preprocessing to obtain a facial image; S2, inputting the face image into the upper branch network and the lower branch network respectively to extract the texture difference information in the spatial domain of the forged image and the potential artifact information in the frequency domain, and outputting the feature map; S3, project the feature maps output by the upper branch network and the lower branch network into feature blocks, respectively, and concatenate them with the CLS token to output the final feature vector; S4, the final feature vector output by the upper branch network and the lower branch network is used as the input of the feature fusion module, and the CLS_S token and CLS_F token after feature fusion are output; S5. Use the CLS_S token and the CLS_F token as the input of the multi-layer perceptron to obtain the final detection result and output it.

2. According to claim 1, a deep fake detection method integrating spatial texture difference and frequency domain information is characterized in that: In the S1 preprocessing process, the face area of ​​the input image is cropped and the size is adjusted.

3. According to claim 1, a deep fake detection method integrating spatial texture difference and frequency domain information is characterized in that: The upper branch feature extraction network in S2 uses the modified Efficient-0 as the backbone network; wherein, the face image is combined with the intensity information of the pixels and the local relative gradient information through the average difference convolution to extract the texture difference information in the shallow features of the spatial domain of the forged image, and then the upper branch feature map is output through the backbone network.

4. The deep forgery detection method of combining spatial texture difference and frequency domain information according to claim 1, characterized in that: The lower branch network in S2 includes an average grayscale image frequency perception decomposition module and an Efficient-0 backbone network; wherein, the face image is transformed into a spatial domain display by the average grayscale image frequency perception decomposition module, and then the lower branch feature map is output through the backbone network.

5. A deep fake detection method integrating spatial texture difference and frequency domain information according to any one of claims 3-4, characterized in that: In S3, the feature map extracted by the backbone network is segmented and flattened into a vector, and then the dimension of the feature vector is changed by linearly projecting the feature map into a feature block, which is then concatenated with the newly added CLS token, positionally embedded, and the final feature vector is output.

6. The deep forgery detection method of combining spatial texture difference and frequency domain information according to claim 1, characterized in that: The S3 specifically includes the following sub-steps: S31, linearly project the feature map extracted by the upper branch network of S2 into a feature block of size 7*7, and linearly project the feature map extracted by the lower branch network into a feature block of size 56*56; S32, concatenating the classification identifier CLS token at the beginning of the feature vector after the mapping is completed, and then performing position embedding and outputting a final feature vector containing the classification identifier and position information.

7. The deep forgery detection method of claim 4 that combines spatial texture differences and frequency domain information, characterized in that: The feature fusion module described in S4 is composed of a stack of Transformer encoders and cross-attention mechanisms.

8. The deep forgery detection method of claim 1 that combines spatial texture differences and frequency domain information, characterized in that: The multilayer perceptron in S4 is composed of a normalization layer and a fully connected layer.

9. The deep forgery detection method of claim 8 that integrates spatial texture differences and frequency domain information, characterized in that: The S5 specifically includes the following contents: The CLS_S token and the CLS_F token are input into a multilayer perceptron to obtain corresponding scores, the CLS_S score and the CLS_F score are added as the final score of the input image, and then the final score is mapped to the interval of 0 to 1 through the sigmoid function, a threshold is set and the final score is compared with the threshold to obtain the classification result of the face image as the detection result output.

Citation Information

Cited By

  • Pluggable forged face detection method based on test time domain adaptation

    CN120048008A

  • A method for detecting a pluggable fake face based on test time domain adaptation

    CN120048008B

  • Identity data analysis method based on spatial frequency sensing fusion network

    CN120561699A

  • Multi-step sequence depth forgery detection method and system fused with frequency domain

    CN122223518A