A deep fake face detection method based on frequency learning

Abstract

The present invention discloses a deep fake face detection method based on frequency learning, and proposes a frequency-aware attention module to detect fake face images, which can refine the representation of the frequency domain, promote the learning of frequency artifacts, effectively extract the artifact information in the frequency domain, extract the features of the image through a dual-stream framework, and capture the artifact information in the spatial domain and the artifact information that is difficult to capture in the spatial domain but more obvious in the frequency domain. Through the fusion learning of the spatial domain and the frequency domain, the existing data is fully utilized to obtain the best characterization results. At the same time, by introducing a deep fake contrast loss function, the relative distribution of fake faces obtained by different tampering techniques can be retained on the basis of distinguishing between real and fake faces, the intra-class variance of real faces can be reduced, and the inter-class diversity in the feature space can be amplified, which greatly improves the generalization of fake face detection and the robustness against various unknown tampering methods in real scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of deep fake detection, and in particular to a deep fake face detection method based on frequency learning. Background Art

[0002] With the remarkable progress of deep fake technology, fake face detection has also attracted widespread attention from the society. At first, fake faces were mainly obtained by replacing faces from different identities. Although operations such as boundary blurring, color adjustment, and expression adjustment were used to make the fake faces more realistic, people can also find traces of image tampering in the spatial domain through means such as steganalysis. Later, thanks to the rapid development of Generative Adversarial Networks (GANs) technology, a variety of face forgery methods based on GANs came into people's view. Since the forged images obtained by the generative method do not require direct face replacement operations, their forgery traces are very limited in the spatiotemporal domain. However, through the average frequency spectrum analysis of 1,000 real faces and 1,000 generative forged faces, such as Figure 1 As shown in the figure, the left figure shows the average spectrum of the real face, and the right figure shows the average spectrum of the generative forged face. It can be clearly found that there are obvious rasterization artifacts in the average spectrum of the generative forged face. This is because the GANs network framework contains upsampling operations, and the images generated by this operation will inevitably produce such artifacts in the frequency domain, so further mining the frequency domain information will be an effective method. Since such false images may be used maliciously or even cause adverse social impacts, effective and generalized forgery detection technology is urgently needed to compete with them.

[0003] Fake face detection technology is usually regarded as a binary classification problem that only needs to distinguish true from fake. There are many ways to tamper with fake faces, but each of them will leave more or less fake traces on the final composite image, which provides detection clues for fake face detection. The deep learning network can obtain a model that can separate the representations of real and fake images by learning fake images generated by different forgery methods and real face images. The purpose of deep forgery detection technology is to find the tampered artifacts in fake face images. Most existing methods extract forgery clues from the spatial domain and the ordinary frequency domain.

[0004] The framework diagram of face forgery detection based on space is as follows Figure 2As shown. This type of network focuses on extracting the features of the image in the spatial domain, and is committed to mining artifact information in the spatial domain through multi-scale information. The specific operation is to input the data into the backbone network, which is usually a deep neural network, obtain the low-level, middle-level and deep-level features of the image from the shallow, middle and deep layers of the network and connect them to obtain multi-scale features. Then, the feature map is divided into blocks, the regional similarity between blocks is calculated, and finally the result is predicted to be true or false.

[0005] The frequency-based face forgery detection block diagram is as follows Figure 3 As shown. First, the input image is subjected to discrete cosine transform (DCT) and regional discrete cosine transform respectively, forming two branches, upper and lower. For the first branch, the spectrum obtained by DCT enters filters of different frequency bands to obtain high-frequency, medium-frequency, and low-frequency spectra, and then converted back to the spatial domain, connected in the channel dimension, and input into the convolutional neural network to further extract features. For the second branch, the spectrum of each region is numerically counted for different frequency band responses to generate a regional spectrum statistics map, which is input into the convolutional neural network to further obtain the frequency connection between regions. The two branches add fusion modules at different levels of the network to share and fuse the obtained information. Finally, the features of the two branches are connected and input into the pooling layer to obtain the prediction result of the authenticity of the image.

[0006] In real scenarios, the model should have a good ability to identify forged images with unknown tampering techniques, but the existing forged face detection technology always overfits the training set data and the forgery method. This will lead to a serious decline in effectiveness across databases and forgery methods, poor generalization, weak migration ability, and low practicality.

[0007] At the same time, since fake face generation technology often includes upsampling operations, and this operation will leave unconcealable pseudo clues in the frequency domain information of the generated fake face, the effective use of frequency domain information is one of the important methods to improve fake face detection. However, general frequency domain information does not have unique invariance and regional consistency, and it is impossible to directly use convolutional neural networks to extract fake clues. Existing technologies use spectral statistics and low-frequency, medium-frequency and high-frequency superposition and then convert them back to the spatial domain to enable them to be input into deep learning networks, but they often cannot fully obtain the corresponding information, resulting in poor model effects. Summary of the invention

[0008] In view of the shortcomings of the existing technology, such as poor generalization and poor model effect, the present invention provides a deep fake face detection method based on frequency learning. The frequency domain upsampling artifacts are captured by using the frequency attention mechanism, and the artifact information in the frequency domain of the forged face image is further mined to obtain an image representation that is more distinguishable between true and false, and a more robust model. At the same time, the generalization of the forged cheek technology is improved, so that it can have better resolution even when facing unknown tampering technologies.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A deep fake face detection method based on frequency learning, comprising the following steps:

[0011] S1. Input the image into two different branches respectively. The backbone networks of the two branches use lightweight convolutional neural networks. The first branch obtains the information of the image space domain through the backbone network, that is, the basic visual features of the image; the second branch strengthens the frequency domain artifact information by introducing the frequency perception attention module, and then inputs it into the backbone network to obtain the frequency domain information to enhance the visual features of the image;

[0012] S2, merge the output characteristics of the two branches in step S1;

[0013] S3. Input the fused features output by step S2 into three different branches. The first branch obtains the forged trace mask regeneration image through the upsampling operation of the decoder, and uses the real forged trace mask as supervision to design the mask loss to prompt the model to pay attention to the artifact information; the second branch obtains a prediction result through the pooling layer and the fully connected layer to judge the authenticity, and optimizes it with the cross entropy loss function; the third branch obtains the feature vector of the high-dimensional feature space of the input image through a fully connected layer, an activation layer and another fully connected layer, and uses the deep forged contrast learning loss function to optimize the feature vector of the high-dimensional feature space to reduce the intra-class variance of the real face and obtain better image representation information.

[0014] Furthermore, in step S1, the process of the frequency-aware attention module is as follows: for the input image, a discrete cosine transform is first performed to obtain a spectrum of the input image; then, the spectrum is input into the average pooling layer and the maximum pooling layer for average pooling and maximum pooling respectively to obtain two different views of the spectrum, and the two different views are connected in the channel dimension, and then the final frequency attention map is obtained through a 1×1 convolution layer and a nonlinear activation function; then the spectrum attention map is converted back to the spatial domain using an inverse discrete cosine transform, and finally, the frequency attention map is superimposed on the input image through a residual short connection and output.

[0015] Furthermore, the nonlinear activation function uses the Sigmoid function

[0016] Furthermore, in step S3, the decoder is used to predict the position of the pixel points of the tampered image. The specific operation is: pair the tampered image with the real image before tampering, subtract the pixel values ​​to generate a mask image, and set a threshold between (0,1) to generate a mask image. Figure 2 The value of the real area is set to 0, and the value of the tampered area is set to 1. This is used as supervision, and the pixel-level cross entropy loss is used to optimize the decoder for forged trace regeneration.

[0017] Furthermore, the second branch in step S3 obtains a prediction value in the interval [0,1] through the pooling layer and the fully connected layer. The result is judged to be true or false with a threshold of 0.5, that is, the prediction value in the interval [0,0.5) is judged to be a real image, and in the interval [0.5, 1] ​​it is judged to be a forged image, and optimized with the cross entropy loss function.

[0018] Furthermore, the activation function used by the activation layer of the third branch in step S3 is the ReLU function.

[0019] Furthermore, in the process of optimizing the high-dimensional representation using the deep fake contrastive learning loss function in the third branch of step S3, the real sample pairs are brought closer, and the real sample and the fake sample pairs are pulled apart.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. The deep fake face detection method based on frequency learning provided by the present invention proposes a new frequency-aware attention module (FAM) to detect fake face images. The module can refine the representation of the frequency domain and promote the learning of frequency artifacts, which is difficult to find when using the spatial attention mechanism and the channel attention mechanism in the past, and effectively extracts the artifact information in the frequency domain. In addition, the FAM module is plug-and-play and can be directly applied to the existing CNN structure without making any other changes to the CNN network framework. It has low computational cost and is simple and efficient.

[0022] 2. The present invention designs a dual-stream framework combining the spatial domain and the frequency domain based on FAM to extract the features of the image, while capturing the artifact information in the spatial domain and the artifact information that is difficult to capture in the spatial domain but is more obvious in the frequency domain. By integrating the learning of the spatial domain and the frequency domain, the present invention makes full use of the existing data to obtain the best characterization results.

[0023] 3. The present invention introduces the deep fake contrast loss (DFCL) function. The model is optimized through the loss function to retain the relative distribution of forged faces obtained by different tampering techniques on the basis of distinguishing real and fake faces, further reduce the intra-class variance of real faces, distinguish real faces from forged faces and the representations obtained after the deep network, and amplify the inter-class diversity in the feature space, which greatly improves the generalization of forged face detection and the robustness against various unknown tampering methods in real scenes, and improves the resolution ability of the model, so that it can maintain a good effect in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0025] Figure 1 The average spectrogram of real faces and generated fake faces in the frequency domain.

[0026] Figure 2 A framework diagram for spatial-based face forgery detection.

[0027] Figure 3 Block diagram for frequency-based face forgery detection.

[0028] Figure 4 A principle block diagram of a deep fake face detection method based on frequency learning provided in an embodiment of the present invention.

[0029] Figure 5 A flow chart of a frequency-aware attention module (FAM) provided for an embodiment of the present invention.

[0030] Figure 6 This is a diagram of the deep fake comparative learning effect provided by an embodiment of the present invention.

[0031] Figure 7 A comparison diagram of a forged trace mask and a forged trace mask regeneration provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to better understand the technical solution, the method of the present invention is described in detail below with reference to the accompanying drawings.

[0033] The deep fake face detection method based on frequency learning provided by the embodiment of the present invention uses a dual-stream framework, and considers the information in the spatial domain and the frequency domain as clues for fake face detection. The principle block diagram is as follows: Figure 4 As shown. Specifically including the following steps:

[0034] S1. Input the image into two different branches respectively. The backbone networks of both branches use the lightweight convolutional neural network Xception. The first branch (RGB stream) obtains the image spatial domain information, that is, the basic visual features of the image, through the backbone network; the second branch (frequency stream) introduces the frequency-aware attention module (FAM) to enhance the frequency domain artifact information that is not easily perceived, including Figure 1 The generative forged image has obvious rasterization information in the frequency domain, which is then input into the backbone network to obtain the frequency domain information to enhance the visual features of the image.

[0035] About Frequency-Aware Attention Module (FAM):

[0036] Existing methods that use the attention mechanism mostly use the attention mechanism from a spatial or channel perspective, and improper settings of the attention mechanism module usually result in the model paying similar attention to different types of images, which greatly reduces the effectiveness of the attention mechanism and even has a misleading negative effect on model training. Furthermore, when using the generative adversarial network method to generate fake images, due to the unavoidable upsampling operation in the generation process, artifact information will be left in the frequency domain, and such artifact information is usually not available in the spatial domain. This is one of the reasons why many fake face detection methods are greatly reduced in effectiveness when facing generative images.

[0037] For an image Where H, W, and C represent the length, width, and number of channels of the image, respectively. The frequency domain expression can be obtained by discrete cosine transform (DCT), as shown in Formula 1, where K represents a constant coefficient, (i, j) and (h, w) represent x and x, respectively. f The coordinates of .

[0038]

[0039] Thus, the spectrum of the image is obtained, which is expressed as x f =D(x).

[0040] Based on the theory of digital image processing, we can derive the inverse discrete cosine transform (IDCT) expression, as shown in Equation 2:

[0041]

[0042] By further derivation of the IDCT transformation formula, it is noted that each pixel of the image in the spatial domain is linearly correlated with the coefficient in the frequency domain, and ζ(i,j,h,w) is the correlation coefficient between the spatial domain and the frequency domain. Therefore, operations in the frequency domain can effectively act on the RGB domain. Based on this, the present invention proposes a frequency-aware attention module (FAM), which introduces an attention mechanism in the frequency domain. FAM is conducive to capturing and refining obvious frequency artifacts hidden in the spatial domain.

[0043] The detailed flow chart of FAM is as follows: Figure 5 As shown. For the input image The image label is y∈{0,1}. We first perform discrete cosine transform (DCT) as shown in equation 1 to obtain the spectrum of the input image x f =D(x). Next, in order to explore the importance of each frequency component, the present invention designs a learning-based attention map generator G(·,θ), in which the spectrum map is average pooled and maximum pooled through the average pooling layer and the maximum pooling layer respectively to obtain two different views of the spectrum map: and The two different views are connected in the channel dimension, and then through a 1×1 convolution layer and a nonlinear activation function, here, the present invention uses the Sigmoid function, represented by σ(·), to obtain the final frequency attention map Mask F , and its specific implementation process is shown in Formula 3:

[0044] The dot product of is highlighted. Since the spectrum map does not satisfy rotation invariance and local consistency, the frequency attention map cannot be directly input into the backbone network based on the convolutional neural network. Here, the spectrum map is first converted back to the spatial domain using the inverse discrete cosine transform (IDCT) shown in Formula 2. At the same time, a residual short connection is designed at the end of the FAM to superimpose the input image with the frequency attention image to obtain richer information, thereby completing the subsequent tasks. The transformation from the frequency domain attention map to the spatial domain of the attention frequency domain output by the FAM module is shown in Formula 4:

[0045] x out =D -1 (x f Mask F )+x=D -1 (D(x)·G(D(x),θ))+x# (4)

[0046] S2, according to the complementary characteristics of the spatial domain and the frequency domain, the output characteristics of the two branches of step S1 are merged;

[0047] There are generally two types of face forgery methods: one is replacement, that is, part or all of the facial area of ​​the source face is cut, deformed, adjusted and replaced into the same area of ​​the target face. This method will leave traces of tampering in the spatial domain. The model can discover this artifact information through supervised learning and make judgments on the authenticity of the image based on this. The other is generative, that is, forged faces generated based on generative adversarial networks. For rougher generative forged images, there will also be traces of forgery in the spatial domain, but for more refined generative forged face images, the artifacts in the spatial domain are difficult to be captured by the model, which will greatly reduce the accuracy of the model. However, due to the inherent upsampling operation of the generative adversarial network, such a fine image will inevitably leave artifacts in the frequency domain. The present invention enhances the area with artifacts in the frequency domain through FAM, and further mines the frequency domain information. The two branches of the model extract the features of the image in the spatial domain and the frequency domain respectively, and directly connect the outputs of the two branches, thereby completing multi-dimensional information fusion.

[0048] S3: Input the fused features output from step S2 into three different branches to perform multi-scale supervision on model learning.

[0049] Among them, the first branch is the decoder module. The fused features output by step S2 are input into the decoder. The forged trace mask regeneration map is obtained through the upsampling operation in the decoder, and the real forged trace mask is used as supervision. Its generation process is shown in Formula 5. The designed mask loss is shown in Formula 6. The model learning is supervised to encourage the model to pay attention to the artifact information in the image.

[0050]

[0051] L exp =∑ i,j -[M(i,j)log M(i,j)+(1-M(i,j))log(1-M(i,j))]# (6)

[0052] The second branch obtains a predicted value y' in the interval [0,1] through the pooling layer and the fully connected layer. The result is judged as true or false with a threshold of 0.5, that is, the predicted value in the interval [0,0.5) is judged as a real image, and in the interval [0.5, 1] ​​is judged as a forged image, and optimized with the cross entropy loss function, as shown in Formula 7:

[0053] L CE =-[ylogy'+(1-y)log(1-y')]# (7)

[0054] The third branch obtains the feature vector of the input image in the high-dimensional feature space through a fully connected layer, an activation layer and another fully connected layer, where the activation function used in the activation layer is the ReLU function. Next, the high-dimensional representation is optimized using the DeepFake Contrastive Learning (DFCL) loss function, as shown in Equation 8, where P and N represent real faces and fake faces, respectively, and τ is a scalar representing the temperature coefficient. In Equation 8, δ(·,·) calculates the similarity of the two feature vectors, and the specific calculation process is shown in Equation 9 to reduce the intra-class variance of the real face and maintain the inter-class diversity in the feature space. During the optimization process, the real sample pairs are brought closer, and the real and fake sample pairs are pulled apart, so that the model obtains more discriminative image representation information.

[0055]

[0056]

[0057] The final loss function is shown in Equation 10, where λ1, λ2, and λ3 are hyperparameters used to balance these loss functions, and the entire model is trained and optimized through the back-propagation algorithm based on this supervision.

[0058] L total =λ1L CE +λ2L DFDC +λ3L exp # (10)

[0059] About Deep Fake Contrastive Learning:

[0060] Existing face forgery detection methods usually use cross entropy loss to supervise the training process. However, the image representation obtained by cross entropy loss supervision is not distinctive enough in nature. The main reason is that cross entropy loss assumes that all instances in the same category should have a tight distribution, while ignoring the differences between them and focusing only on finding a decision boundary to separate different classes. Therefore, the separability between classes and the compactness within classes are not explicitly considered. Although a few methods based on metric learning have been proposed, they do not take into account the particularity of face forgery detection, that is, manipulated images are generated through various different forgery methods. The representations of forged images generated by different manipulation methods are quite different.

[0061] The key to the classification task is whether the model can obtain a high-dimensional representation with the characteristics required by the downstream task through learning. This also largely determines the quality of the results of the downstream task. For the task of forged face detection, the representation of the real face should be relatively concentrated in the feature space, and far away from the representation of the forged face, that is, the similarity is low. However, many existing technologies usually use cross-entropy loss for supervised learning, and this loss function that emphasizes category differences more will limit the generality of the model in the unseen domain. Specifically, the use of the cross-entropy loss method assumes that all samples in each category are similar in feature distribution. Its purpose is to find a hyperplane to separate samples of different categories, without considering the distribution of samples within the category and the deeper differences between samples. Therefore, these methods cannot further explore the essential differences between real and fake faces, nor do they take into account the differences between different forgery methods. Therefore, in the absence of other constraints, these methods using cross-entropy loss are prone to overfitting the forgery methods that appear in the training set, thereby limiting the generalization of the model when facing data with unknown tampering methods.

[0062] Therefore, this paper designs a unified deep fake contrastive learning (DFCL) loss function, which pulls the real and fake image pairs apart and brings the real image pairs closer. The DFCL loss is based on the currently popular InfoNCE loss. For positive sample pairs, positive-negative sample pairs, and negative sample pairs, its effect is as follows: Figure 6 As shown in the figure. For pictures in real scenes, real pictures should be similar and have no signs of forgery, while fake pictures are obtained by various invisible tampering techniques. Therefore, the representation of real images should be similar, while the representation of fake images should not be similar. Therefore, when designing the contrastive learning loss, only the positive sample pairs are brought closer and the positive and negative sample pairs are pulled apart, without bringing the negative samples closer, so that the negative samples maintain their original distribution, so that the learned representation is more conducive to the judgment of real and fake faces.

[0063] About forged trace mask learning:

[0064] Since the tampered image may be caused by partial tampering of the real image, or it may be caused by the complete replacement of the real image, the present invention designs a decoder to predict the specific location of the tampered image pixels. In this way, an auxiliary output can be obtained to explain which spatial regions the network is based on to make decisions, which can also prove the interpretability of the present invention.

[0065] The specific operation of the decoder is to pair the tampered image with its source image (the real image before tampering), subtract the pixel values ​​to generate a mask image, and set a threshold between (0,1) to Figure 2The tampered area is quantized, and the real area is set to 0, and the tampered area is set to 1. This is used as supervision, and the decoder for forged trace regeneration is optimized using pixel-level cross entropy loss. The comparison of the designed forged trace mask real value and the forged trace mask regenerated by the decoder is shown in the figure. Figure 7 As shown in the first and second rows, we can see that the network effectively focuses on the tampered areas of the image, proving the effectiveness of this method.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep fake face detection method based on frequency learning, characterized in that: The following steps are involved: S1. Input the image into two different branches respectively. The backbone networks of the two branches use lightweight convolutional neural networks. The first branch obtains the information of the image space domain through the backbone network, that is, the basic visual features of the image; the second branch strengthens the frequency domain artifact information by introducing the frequency perception attention module, and then inputs it into the backbone network to obtain the frequency domain information to enhance the visual features of the image; The process of the frequency-aware attention module is as follows: for the input image, first use discrete cosine transform to obtain the spectrum of the input image; then, input the spectrum into the average pooling layer and the maximum pooling layer for average pooling and maximum pooling respectively, obtain two different views of the spectrum, and connect these two different views in the channel dimension, and then use a 1×1 convolution layer and a nonlinear activation function to obtain the final frequency attention map; then use the inverse discrete cosine transform to convert the spectrum attention map back to the spatial domain, and finally use a residual short connection to superimpose the frequency attention map with the input image and output it; S2, merge the output characteristics of the two branches in step S1; S3, input the fusion features output from step S2 into three different branches. The first branch obtains the forged trace mask regeneration image through the upsampling operation of the decoder, and uses the real forged trace mask as supervision to design the mask loss to prompt the model to pay attention to the artifact information; the second branch obtains a prediction result through the pooling layer and the fully connected layer to judge the authenticity, and optimizes it with the cross entropy loss function; the third branch obtains the feature vector of the high-dimensional feature space of the input image through a fully connected layer, an activation layer and another fully connected layer, and uses the deep forged contrast learning loss function to optimize the feature vector of the high-dimensional feature space to reduce the intra-class variance of the real face and obtain better image representation information; The first branch predicts the position of the pixel points of the tampered image through the decoder. The specific operation is: pair the tampered image with the real image before tampering, subtract the pixel values ​​to generate a mask image, set a threshold between (0,1), binarize the mask image, set the value of the real area to 0, and the value of the tampered area to 1. This is used as supervision, and the pixel-level cross entropy loss is used to optimize the decoder for forged trace regeneration; The second branch obtains a prediction value in the interval [0,1] through the pooling layer and the fully connected layer. The result is judged as true or false with a threshold of 0.

5. That is, the predicted value in the interval [0,0.5) is judged as a real image, and in the interval [0.5, 1] ​​it is judged as a forged image, and optimized with the cross entropy loss function.

2. The frequency learning-based deep fake face detection method according to claim 1, characterized in that: The nonlinear activation function uses the Sigmoid function.

3. The frequency learning-based deep fake face detection method according to claim 1, characterized in that: The activation function used by the activation layer of the third branch in step S3 is the ReLU function.

4. The frequency learning-based deep fake face detection method according to claim 1, characterized in that: In the process of optimizing the high-dimensional representation using the deep fake contrastive learning loss function in the third branch of step S3, the real sample pairs are brought closer, and the real sample and fake sample pairs are pulled apart.

Images