A deepfake detection method based on frequency domain filtering residual

Abstract

A deep forgery detection method based on frequency domain filtering residual belongs to the technical field of image classification, including a preprocessing module and a classification module, including the following steps: by performing Haar wavelet transform on the image, a low-frequency information map of the image is obtained; residual operation is performed on the grayscale map and the low-frequency information map of the original image to obtain the residual map of the medium and high frequency information of the original image; in the classification module, the original image and the residual map of the medium and high frequency information are spliced ​​and input into the convolutional neural network for classification processing. The features of the medium and high frequency domains of the image are obtained by Haar wavelet transform and residual, so as to mine the forgery traces weakened or contaminated in the RGB domain due to image compression in the frequency domain, and improve the accuracy of the detection method when detecting compressed images. The RGB image and the high-frequency residual image are spliced ​​together as the input of the convolutional neural network, making full use of the rich semantic information in the RGB domain and the detailed texture information in the medium and high frequency domains, so that the detection method has good robustness to image compression.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and specifically relates to a deep forgery detection method based on frequency domain filtering residual. Background Art

[0002] In recent years, with the development of science and technology, social media and short video platforms have risen rapidly, occupying most of people's entertainment time. More and more people like to share images and videos of their daily lives on various social media and short video platforms. These images and videos will spread rapidly on the Internet and become the main carrier for people to communicate and share information with each other. However, image tampering technology, especially face image tampering technology, has brought certain negative effects to the spread of images and videos. Tampered face images and videos may be used for improper purposes. Therefore, the detection of tampered face images and videos has very important practical significance and value.

[0003] With the rapid development of deep learning and generative adversarial networks, the current face tampering technology has become very mature. The main tampering methods include Deepfakes, Face2Face, Faceswap and NeuralTextures.

[0004] Deepfakes is based on two autoencoders with a shared encoder that reconstruct training images of source and target faces, respectively. The encoder and decoder trained on the source face are then applied to the target face to generate a fake image. The output of the autoencoder is then blended with the rest of the image using Poisson image editing. Face2Face is a facial reenactment system that transfers the expression of a source video to a target video while maintaining the identity of the target person. FaceSwap is a computer graphics-based method that uses detected facial landmarks to fit a 3D face model that generates a fake image by back-projecting it onto the target image using the texture of the input image to minimize the difference between the projected shape and local landmarks. NeuralTextures is a technique that uses expression transfer to modify the facial texture map of the target video to match the expression of the source target to generate a fake video. Although these techniques all rely on advanced image processing algorithms, there are now some integrated software or program frameworks that can be used by people without professional knowledge to quickly process facial images, making face forgery easy and common.

[0005] In order to combat the harm and negative impact of face image tampering, people have proposed a variety of methods to detect tampered face images and videos. There are currently two main detection methods, one is to use traditional manual features, and the other is to use neural network to extract features. Compared with the features extracted by neural networks, manual features are clearer and more robust, more interpretable, and can better utilize the fine-grained information of the image. Therefore, this method has achieved good accuracy on certain specific data sets, but on larger data sets, the neural network method can achieve better results. Neural networks can learn more and richer image features, unlike the method using manual features that relies on specific features. Therefore, this type of method has received more and more attention and has been widely used. The detection of forged faces can be regarded as a classification task, and the accuracy of detection depends on the selection of features and the selection of classifiers. The process of all detection methods can be roughly divided into two stages: feature extraction and classification. Some methods divide these two stages into two independent sub-problems, while others integrate these two stages in an end-to-end manner based on deep neural networks. In the feature extraction stage, there are two main feature selections, one is based on the features of a single image, and the other is based on the features between video frames. Since methods based on single image features can also be used to determine the authenticity of videos, there are many studies on this type of method. In the classification stage, the main choices of classifiers are SVM, CNN, RNN, MLP, etc., among which SVM and CNN are more widely used.

[0006] Existing methods have achieved good detection accuracy on some public datasets, but there is still a common problem, that is, the detection accuracy for highly compressed videos and images will drop significantly. In other words, the robustness of most methods is very poor. For uncompressed or lightly compressed images and videos, existing methods can achieve satisfactory detection accuracy, but when forged face images and videos are highly compressed, the detection accuracy of existing methods is greatly reduced. This is because the compression operation can eliminate forgery traces at the feature level and reduce the difference in feature distribution between real images and fake images, and this feature difference is exactly the detection principle that most existing methods rely on. However, in real applications, the quality of images and videos may be relatively low after a variety of post-processing operations similar to compression, which greatly affects the performance of existing methods. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes: a deep fake detection method based on frequency domain filtering residual, including a preprocessing module and a classification module, including the following steps:

[0008] In S1, the preprocessing module, the image is subjected to Haar wavelet transform to obtain the low-frequency information map of the image;

[0009] S2, in the preprocessing module, the grayscale image and the low-frequency information image of the original image are subjected to residual operation to obtain the residual image of the medium and high frequency information of the original image;

[0010] In S3, the classification module, the original image and the residual image of medium and high frequency information are spliced ​​and then input into the convolutional neural network for classification processing.

[0011] Wherein, in the step S1, the RGB image I O Grayscale to get the grayscale image I of the original image G , to I G Perform Haar wavelet transform to obtain low-frequency information graph I L ;

[0012] For the one-dimensional signal of the image, when performing Haar wavelet decomposition, the signal is first decomposed by a scaling function, the frequency band of the scaling function is the same as the frequency band of the original signal, and then the approximation function of the original signal is decomposed in the scale space and the wavelet space respectively to obtain the low-frequency coarse part and the high-frequency detail part of the signal. Through the linear correlation between the scaling function and the wave function and the selection of the scaling factor and the translation parameter, a one-dimensional signal is decomposed by Haar wavelet;

[0013] For the two-dimensional signal of the image, when performing wavelet transform, each row of the image pixel matrix is ​​first transformed, and then each column of the matrix after the row transform is transformed to obtain the final decomposition transform result; or when performing wavelet transform, each column of the image pixel matrix is ​​first transformed, and then each row of the matrix after the column transform is transformed to obtain the final decomposition transform result; a geometric image will be decomposed into four frequency domain components after Haar wavelet transform, that is, four sub-images are generated, namely, a low-frequency information image that concentrates most of the information of the geometric image; a horizontal high-frequency information image that reflects the edge information of the geometric image along the horizontal direction; a vertical high-frequency information image that reflects the edge information of the geometric image along the vertical direction; a diagonal high-frequency information image that reflects the detail features of the geometric image along the diagonal direction, and the side length of the above sub-images is 1 / 2 of the original image.

[0014] Wherein, in the step S2, after obtaining I L Afterwards, I L Enlarge the size to the original image size, and then convert the grayscale image of the original image into G Subtract low frequency information graph I L , and obtain the mid-high frequency information residual map I H ; Get I H After that, the original RGB image I O and I H Splice along the image channel direction to get I C .

[0015] Among them, in the step S3, the Xception network is selected as the backbone network of the classification stage, and I C As the input of the network, Xception is trained through end-to-end learning to classify true and false face images.

[0016] The beneficial effects of the present invention are:

[0017] 1. The features of the mid- and high-frequency domains of the image are obtained through Haar wavelet transform and residual, so as to mine the forgery traces that are weakened or contaminated in the RGB domain due to image compression in the frequency domain, and improve the accuracy of the detection method when detecting compressed images.

[0018] 2. The RGB image and the high-frequency residual image are spliced ​​together as the input of the convolutional neural network, which makes full use of the rich semantic information in the RGB domain and the detailed texture information in the medium and high frequency domains, so that the subsequent convolutional neural network can fully mine the forgery traces of the forged image in multiple feature domains, making the detection method robust to image compression and obtaining more satisfactory performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is an overall block diagram of the method of the present invention;

[0020] Figure 2 It is the Xception network structure of the present invention. DETAILED DESCRIPTION

[0021] A deep fake detection method based on frequency domain filtering residual, the technical scheme is as follows: including a preprocessing module and a classification module, such as Figure 1 As shown, the following steps are included:

[0022] In S1, the preprocessing module, the image is subjected to Haar wavelet transform to obtain the low-frequency information map of the image;

[0023] S2, in the preprocessing module, the grayscale image and the low-frequency information image of the original image are subjected to residual operation to obtain the residual image of the medium and high frequency information of the original image;

[0024] In S3, the classification module, the original image and the residual image of medium and high frequency information are spliced ​​and then input into the convolutional neural network for classification processing.

[0025] Wherein, in the step S1, the RGB image I O Grayscale to get the grayscale image I of the original image G , to I G Perform Haar wavelet transform to obtain low-frequency information graph I L ;

[0026] Since the Haar wavelet transform requires a grayscale image as input, first convert the RGB image I O Grayscale to get the grayscale image I of the original image G ; to I G Perform Haar wavelet transform.

[0027] Wavelet is an oscillation whose amplitude starts from zero amplitude and rises to a certain value and then returns to zero at a very high frequency. Wavelet transform uses wavelet function to decompose the original image into different sub-waves, and these waves have frequency function and space function. In wavelet transform, if the scaled and shifted wavelets are decomposed into multi-resolution form, the numerical complexity can be effectively avoided. Decomposing a high-resolution signal into multiple lower-resolution signals is called tree decomposition of wavelet. In many cases, low-frequency signals are often the most important part of the information contained in the target image in the frequency domain, and the original information contained in the decomposed signal is given in the low-frequency part of the signal. High-frequency signals often contain some subtle differences. After wavelet decomposition, the image provides approximate values ​​of the main details and other important information, which can be used for further analysis of this information in other ways later. Wavelet transform divides the image into four parts. It provides approximate information of the image and image information in the vertical, horizontal and diagonal directions.

[0028] Haar wavelet transform was proposed by Alfréd Haar in 1909. It is the simplest type of wavelet transform and the earliest wavelet transform. Compared with other types of wavelet transforms, Haar wavelet transform has the following advantages:

[0029] 1) Haar wavelet is compactly supported in the time domain, that is, its non-zero interval is (0,1);

[0030] 2) Haar wavelet is an orthogonal wavelet. The orthogonality reduces the correlation of sub-band data and reduces redundancy.

[0031] 3) Haar wavelet is symmetrical. If the unit impulse response of a system is symmetrical, the system has a linear phase, which is very beneficial for removing phase aberrations.

[0032] 4) Haar wavelet only takes +1 and -1, and the calculation is very simple.

[0033] 5) The Haar wavelet is currently the only orthogonal wavelet that is both symmetric and has finite support.

[0034] Based on these advantages of the Haar wavelet, the present invention selects it for wavelet transform.

[0035] For the one-dimensional signal of the image, when performing Haar wavelet decomposition, the signal is first decomposed using a scaling function, the frequency band of the scaling function is the same as the frequency band of the original signal, and then the approximation function of the original signal is decomposed in the scale space and wavelet space respectively. The low-frequency coarse part and the high-frequency detailed part of the signal are obtained.

[0036] The scaling function of the Haar wavelet transform is as follows:

[0037]

[0038]

[0039] Among them, φ(x) represents the scale function, x represents time,

[0040] The wave function of the Haar wavelet transform is shown below:

[0041]

[0042]

[0043] Among them, ψ(x) represents the wave function, j is the scale factor, changing j can make the function graph shrink or enlarge; i is the translation parameter, changing i makes the function graph translate along the x-axis. The scaling function and wave function satisfy the following relationship:

[0044]

[0045]

[0046] The essence of Haar wavelet transform is to decompose a one-dimensional signal into a linear representation of a scaling function and multiple wave functions. Through the linear correlation between the scaling function and the wave function and the selection of the scaling factor and the translation parameter, a one-dimensional signal is decomposed by Haar wavelet.

[0047] For the two-dimensional signal of the image, when performing wavelet transform, each row of the image pixel matrix is ​​first transformed, and then each column of the matrix after row transform is transformed to obtain the final decomposition transform result; or when performing wavelet transform, each column of the image pixel matrix is ​​first transformed, and then each row of the matrix after column transform is transformed to obtain the final decomposition transform result; a geometric image will be decomposed into four frequency domain components after Haar wavelet transform, that is, four sub-images are generated, namely, a low-frequency information image that concentrates most of the information of the geometric image; a horizontal high-frequency information image that reflects the edge information of the geometric image along the horizontal direction; a vertical high-frequency information image that reflects the edge information of the geometric image along the vertical direction; a diagonal high-frequency information image that reflects the detail features of the geometric image along the diagonal direction, and the side length of the sub-image is 1 / 2 of the original image.

[0048] The present invention only retains the low-frequency information graph, that is, Figure 1 I L .

[0049] Wherein, in the step S2, after obtaining I L Afterwards, I L Enlarge the size to the original image size, and then convert the grayscale image of the original image into G Subtract low frequency information graph I L , and obtain the mid-high frequency information residual map I H The reason for this is that low-frequency information concentrates most of the semantic information of the image, and this rich semantic information can also be obtained in the RGB domain. The method of the present invention is to find the rich texture detail information contained in the mid-high frequency area, which is more likely to find the existence of forgery traces. Therefore, by making a residual of the original grayscale image and the low-frequency information image, the present invention can obtain the required mid-high frequency information residual image I H .

[0050] Get I H After that, the original RGB image I O and I H Splice along the image channel direction to get I C The reason is that such an operation not only retains rich semantic information, which can ensure the accuracy in detecting uncompressed images, but also combines a large amount of medium and high frequency texture detail information, which can provide more forgery traces and feature selection when detecting compressed images, thereby ensuring the robustness of the method of the present invention to image compression.

[0051] Among them, in the step S3, the Xception network is selected as the backbone network of the classification stage, and I C As the input of the network, Xception is trained through end-to-end learning to classify true and false face images. The Xception network structure is as follows Figure 2 shown.

[0052] 3. Experimental results

[0053] The present invention selects four methods as comparison methods and conducts experiments on three data sets respectively. The detection accuracy results are shown in Table 1. Among them, Meso4 represents the method of directly using MesoNet for classification, MesoInception represents the method of directly using MesoInceptionNet for classification, and Xception represents the method of directly using XceptionNet for classification. These three methods are the methods previously proposed in this field. H Indicates that the classification module in this method only uses I HThat is, the high-frequency residual map is used as the input of the classification module. C Indicates that the classification module in this method uses I C That is, the concatenation of the RGB image and the high-frequency residual image is used as the input of the classification module. The three datasets are FaceForensics++, Celeb-DF-V2 and UADFV, all of which are publicly available and commonly used datasets in this field. C0 represents the original quality version of the FaceForensics++ dataset, i.e., the uncompressed version; C23 represents the lightly compressed version of the FaceForensics++ dataset, i.e., the compression factor is 23; C40 represents the heavily compressed version of the FaceForensics++ dataset, i.e., the compression factor is 40. Each version of the FaceForensics++ dataset contains four deep fake methods, so there are four rows of results for each version.

[0054] Table 1 Experimental results

[0055]

[0056]

[0057] It can be seen from the test results in Table 1 that the method proposed in the present invention (i.e., C The method of the present invention achieves leading results in 12 out of 14 sub-datasets (including C23 and C40) and 6 out of 8 sub-datasets (including C23 and C40) of the compressed versions. The experimental results demonstrate the robustness of the method of the present invention to image compression, while maintaining leading performance in real image detection of uncompressed images.

[0058] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A deep fake detection method based on frequency domain filtering residual, characterized in that: It includes a preprocessing module and a classification module, including the following steps: In S1, in the preprocessing module, the low-frequency information map of the original RGB map is obtained by performing Haar wavelet transform on the original RGB map; S2, in the preprocessing module, the grayscale image and the low-frequency information image of the original RGB image are subjected to residual operation to obtain a residual image of the medium and high frequency information of the original RGB image; In S3, the classification module, the original RGB image and the medium and high frequency information residual image are spliced ​​and then input into the convolutional neural network for classification processing.

2. The deep fake detection method based on frequency domain filtering residual according to claim 1, characterized in that: In the step S1, the original image RGB image I O Grayscale to get the grayscale image I of the original RGB image G， to I G Perform Haar wavelet transform to obtain low-frequency information graph I L ; For the one-dimensional signal of the original RGB image, when performing Haar wavelet decomposition, the signal is first decomposed using a scaling function, the frequency band of the scaling function is the same as the frequency band of the original signal, and then the approximation function of the original signal is decomposed in the scale space and the wavelet space respectively to obtain the low-frequency coarse part and the high-frequency detail part of the signal. Through the linear correlation between the scaling function and the wave function and the selection of the scaling factor and the translation parameter, a one-dimensional signal is decomposed by Haar wavelet; For the two-dimensional signal of the original RGB image, when performing wavelet transform, each row of the image pixel matrix is ​​first transformed, and then each column of the matrix after row transform is transformed to obtain the final decomposition transform result; or when performing wavelet transform, each column of the image pixel matrix is ​​first transformed, and then each row of the matrix after column transform is transformed to obtain the final decomposition transform result; After the Haar wavelet transform, a geometric image will be decomposed into four frequency domain components, that is, four sub-images will be generated, namely, a low-frequency information image that concentrates most of the information of the geometric image; a horizontal high-frequency information image that reflects the edge information of the geometric image along the horizontal direction; a vertical high-frequency information image that reflects the edge information of the geometric image along the vertical direction; and a diagonal high-frequency information image that reflects the detail features of the geometric image along the diagonal direction. The side length of the above sub-images is 1 / 2 of the original RGB image.

3. The deep fake detection method based on frequency domain filtering residual according to claim 2, characterized in that: In the step S2, after obtaining I L Afterwards, I L Enlarge the size to the original RGB image size, and then convert the grayscale image of the original RGB image into G Subtract low frequency information graph I L , and obtain the mid-high frequency information residual map I H ; Get I H After that, the original RGB image I O and I H Splice along the image channel direction to get I C .

4. The deep fake detection method based on frequency domain filtering residual according to claim 3, characterized in that: In step S3, the Xception network is selected as the backbone network of the classification stage. C As the input of the network, Xception is trained through end-to-end learning to classify true and false face images.

Images