Face forgery detection method based on visual converter

Abstract

The present invention particularly relates to a face forgery detection method based on visual transformer, comprising the following steps: S100, using Transformer as a basic component and combining a variant residual connection module, an attention guidance module and a multi-level forensics module to jointly construct a network model for modeling global and local information; S200, using a data set to train the network model to obtain a face forgery detection model; S300, importing a picture or video to be detected into the network model for recognition, and obtaining a final prediction result. Using Transformer as the basic structure instead of a convolutional network, global and local information can be fully modeled, the variant residual connection module ensures that the network can capture enough texture information, the attention guidance module can help select the most valuable and recognizable areas, while ignoring areas with more redundant information such as background, and in addition, the multi-level forensics module combines features from different network depths to further improve detection performance.

Description

Technical Field

[0001] The invention relates to the technical field of face forgery detection, and in particular to a face forgery detection method based on a visual converter. Background Art

[0002] Deepfake refers to a type of synthetic media that uses computer technology to replace the image of a person in an existing image or video with someone else. While the act of falsifying media content is not new, Deepfake uses the powerful techniques of machine learning and artificial intelligence to forge or generate visual and audio content with high potential for deception, which can easily be used for malicious purposes. As of now, it is almost impossible for humans to distinguish whether certain media is credible. If Deepfake gets out of control on social media, it is likely to seriously threaten the reputation of celebrities and even trigger a political crisis, so effective detection methods are very important at the moment.

[0003] For security reasons, researchers have proposed a series of face forgery detection methods in recent years. Among them, the detection methods designed based on convolutional neural networks (CNN) have shown outstanding performance. However, these CNN-based methods may have the problem of being seriously over-fitted to specific datasets. Most of these detection models can achieve near-perfect performance after training on the FaceForensics++ dataset, but their accuracy is seriously reduced when directly tested on the Celeb-DF dataset. This is mainly because CNN-based methods identify forged faces by learning local texture information, which is usually related to the means of forgery. There are differences in the means of face forgery between different datasets, which directly leads to the problem of reduced performance in their migration tests. Therefore, in order to solve this problem, we try to take into account both global information and local information to achieve face forgery detection, rather than just local texture information.

[0004] In recent years, the visual transformer Transformer has achieved great success in classic classification tasks. It uses the innate attention mechanism to apply the transformer to a series of image patches, effectively broadening the receptive field, which helps to capture global information. A series of related extension works, such as object detection segmentation and semantic segmentation, further confirm the ability of the transformer to capture global and local information, and also show its great potential in the task of face forgery detection. Summary of the invention

[0005] The object of the present invention is to provide a face forgery detection method based on visual converter, which has high accuracy and optimal transfer performance.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is: a face forgery detection method based on visual transformer, comprising the following steps: S100, using Transformer as the basic component and combining a variant residual connection module, an attention guidance module and a multi-level forensics module to jointly construct a network model for modeling global and local information; S200, using a data set to train the network model to obtain a face forgery detection model; S300, importing the image or video to be detected into the network model for recognition to obtain the final prediction result.

[0007] Compared with the prior art, the present invention has the following technical effects: Transformer is used as the basic structure instead of convolutional network, so as to fully model global and local information; the variant residual connection module ensures that the network can capture sufficient texture information; the attention guidance module can help select the most valuable and recognizable areas (i.e., the areas most likely to be tampered with) while ignoring areas with much redundant information such as background (i.e., the areas least likely to be tampered with); in addition, the multi-level forensics module combines features from different network depths to further improve detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is the overall structure diagram of the network in the present invention;

[0009] Figure 2 This is a visualization example of the model attention distribution heat map when detecting forged face images in the present invention;

[0010] Figure 3 This is the accuracy comparison between this method and other methods on the FF++ dataset;

[0011] Figure 4 This is a comparison of the migration capabilities of this method and other methods on image-level detection on the Celeb-DFv2 dataset;

[0012] Figure 5 This is a comparison of the migration capabilities of this method and other methods in video-level detection on more public datasets. DETAILED DESCRIPTION

[0013] Combine the following Figures 1 to 5 , the present invention is further described in detail.

[0014] See also Figure 1, a face forgery detection method based on visual transformer, comprising the following steps: S100, using Transformer as the basic component and combining with variant residual connection module, attention guidance module and multi-level forensics module to jointly construct a network model for modeling global and local information; S200, using the data set to train the network model to obtain a face forgery detection model; S300, importing the image or video to be detected into the network model for recognition, and obtaining the final prediction result. Aiming at the limitations of the current Deepfake detection method at the architectural level and the bottleneck faced by the migration performance, the present invention proposes a face forgery detection method based on Transformer. In this case, three modules are specially designed, including attention guidance module, variant residual connection module and multi-level forensics module. In addition, contrast loss is also designed during the training process. The present invention uses Transformer as the basic structure rather than convolutional network, so as to fully model global and local information. The variant residual connection module ensures that the network can capture sufficient texture information. The attention guidance module can help select the most valuable and recognizable areas (i.e., the areas most likely to be tampered with), while ignoring areas with more redundant information such as background (i.e., the areas least likely to be tampered with). In addition, the multi-level forensics module combines features from different network depths to further improve the detection performance. Figure 2 In the grayscale image shown, the brighter parts of the facial features are the redder parts in the heat map.

[0015] Furthermore, in step S100, the network model includes N Transformer blocks, N-1 variant residual connection modules, an attention guidance module, N second Transformer layers, and a multi-level forensics module; the Transformer block is connected to the next Transformer block through the variant residual connection module, and the output tokens of each Transformer block are sent to the attention guidance module, the attention guidance module selects the most valuable token from all tokens and outputs it to the second Transformer layer, and the multi-level forensics module merges the sub-prediction results output by the N second Transformer layers to obtain the final prediction result. The network model using this structure can fully model global and local information, and has high detection performance and accuracy.

[0016] When training or testing, the input face image is first segmented into a 16×16 pixel network and projected into the embedding space, then the learnable position embedding is added and input into the Transformer block connected by the variant residual. The embedding space and position embedding are the conventional pre-processing of the visual Transformer. These two steps follow the original design of VIT. You can refer to the article "AN IMAGE IS WORTH16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE" published by Alexey Dosovitskiy. This is not an improvement point for this case, so it will not be elaborated in detail.

[0017] Furthermore, the Transformer block is composed of multiple first Transformer layers, and the output form of each first Transformer layer is as follows: ; ,in Indicates The hidden features encoded by the first Transformer layer. The Transformer blocks here all contain multiple first Transformer layers. When actually designing the network model, the number of Transformer blocks and the number of Transformer layers contained in each Transformer block can be set.

[0018] The method based on convolutional neural network (CNN) can easily extract rich texture information, but it is difficult for the Transformer structure to obtain sufficient texture information. In order to solve this problem, in this case, it is preferred to use a variant residual connection between adjacent Transformer blocks: ;in, represents the input of the i-th Transformer block, Indicates The output of the Transformer block as input. Through such a setting, it is ensured that the network model in the present invention can capture sufficient texture information.

[0019] By fusing the attention weights of all previous Transformer layers, the one-to-one correspondence between input tokens and attention weights is ensured as much as possible. Specifically, the attention guidance module is processed as follows: recursively normalize the original attention weights output by all network layers and then accumulate them; find the token with the largest weight from the H self-attention heads; combine the token with the largest weight and the classification token as the input sequence of the second Transformer layer, which can be expressed as follows: ; In the formula, represents the token corresponding to the i-th self-attention head encoded by the selected L-1-th layer Transformer, Represents the classification token at layer L-1.

[0020] Considering the complexity and diversity of forged face images, the face forgery detection model should not only focus on the high-level features of the network output, but also on the low-level features at the same time, and allow features at different levels to participate in the final decision. Therefore, this method believes that for the sample deep fake detection method, the corresponding features of the real face should show a "true" nature at all levels, and vice versa. Based on this, this method proposes a multi-level forensics module to obtain more convincing and detailed predictions. Specifically, the multi-level forensics module merges the sub-prediction results according to the following formula to obtain the final prediction result: ; In the formula, Indicates that the sub-classification confidence obtained by analyzing the output features of the second Transformer layer of the i-th layer is the sub-prediction result. The sub-classification confidence is a floating point number between 0 and 1. Indicates the average confidence of each sub-classification.

[0021] Furthermore, training such a deep stacked Transformer network requires strong and detailed supervision. Therefore, in the step S200, when training the network model, the classification loss and the contrast loss are combined as the training loss, which is characterized as follows: ; Among them, the forms of cross entropy loss and contrast loss are as follows:

[0022] ;

[0023] ;

[0024] In the formula, y and Represent the label value and predicted value of the input image respectively, Represents the normalized result of the classification token features output by the Transformer layer.

[0025] When setting the network model, the number of Transformer blocks and the number of Transformer layers in each Transformer block can be set according to actual conditions. The present invention provides a more preferred implementation: the network model includes 4 Transformer blocks, each Transformer block includes 3 first Transformer layers. When these two parameters are taken in this way, experimental results show that this number has the best effect.

[0026] With the rapid development of various face forgery technologies, the differences between real and forged faces are becoming more and more subtle. In order to reveal these differences, the present invention attempts to build a deep network with Transformer as the basic module, modeling the global information of the entire input face for feature extraction. In order to further improve the detection ability of the model, a variant residual connection module is used to assist in extracting the texture information of the image. At the same time, considering the particularity of face forgery detection, the tampered areas are all concentrated on the face, which means that the background information is useless. Therefore, this method proposes an attention guidance module to guide the network to focus on the areas most likely to be tampered with. In view of the problem that the previous training loss was too simple, a feature-level contrast loss was designed to provide stronger supervision for the face forgery detection model.

[0027] There are four key components in this invention: (1) Building a network purely with Transformer basic components to model global and local information. (2) Using the Attention Guided Module (ALM) to guide the network to focus on the most valuable areas (areas most likely to be tampered with). (3) Using the Variant Residual Connection (VRC) and the Multi-Level Forensic Module (MFM) to combine and analyze features at different network depths. (4) Using the contrast loss to provide stronger supervision for model training.

[0028] With these four key parts, the network model we built works very well. Figure 3-Figure 5 It is the result of comparing the detection method in the present invention with the common detection method in the prior art. Figure 3 We can see that the accuracy of the detection method in the present invention on the image level detection of the FF++ dataset is not the best, but it is also good. Figure 4 We can see the advantages of the detection method of the present invention. When we compared the transfer ability of image-level detection on the Celeb-DFv2 dataset, we found that the accuracy of each detection method in the prior art had a significant decline. Although the detection method of the present invention also declined, the decline was very small, indicating that the network model of the present invention has a strong transfer ability. Not only is the transfer ability of image-level detection strong, Figure 5 The migration capability of video-level detection on more public datasets is demonstrated in comparison with other methods. It can also be seen that the detection method in the present invention also has strong migration capability in video detection.

Claims

1. A face forgery detection method based on visual converter, characterized by: The steps include: S100, using Transformer as a basic component and combining it with a variant residual connection module, an attention guidance module and a multi-level forensics module to jointly construct a network model for modeling global and local information, wherein the network model includes N Transformer blocks, N-1 variant residual connection modules, an attention guidance module, N second Transformer layers and a multi-level forensics module; the Transformer block is connected to the next Transformer block through the variant residual connection module, the output tokens of each Transformer block are sent to the attention guidance module, the attention guidance module selects the most valuable token from all tokens and outputs it to the second Transformer layer, and the multi-level forensics module merges the sub-prediction results output by the N second Transformer layers to obtain the final prediction result; The Transformer block is composed of multiple first Transformer layers, and the output of each first Transformer layer is as follows: l ’ =MSA(LN(Z l-1 ))+Z l-1 , Z l =MLP(LN(Z l ’ )+Z l-1 )+Z l ’ ; Where Z l Represents the hidden features encoded by the first Transformer layer; Adjacent Transformer blocks use a variant residual connection according to the following formula: Ti+1 =F(X Ti )-X Ti ; Among them, X Ti represents the input of the i-th Transformer block, F(X Ti ) indicates that X Ti The output of the Transformer block as input; The attention guidance module is processed as follows: recursively normalize the original attention weights output by all network layers and then accumulate them; find the token with the largest weight from the H self-attention heads; combine the token with the largest weight and the classification token as the input sequence of the second Transformer layer, which can be expressed as follows: In the formula, represents the token corresponding to the i-th self-attention head encoded by the selected L-1-th layer Transformer, represents the classification token of layer L-1; The multi-level forensics module combines the sub-prediction results according to the following formula to obtain the final prediction result: Pred = Mean (Sub1, Sub2, ..., Sub N ); Where, Sub i It indicates that the sub-classification confidence obtained by analyzing the output features of the second Transformer layer of the i-th layer is the sub-prediction result, the sub-classification confidence is a floating point number between 0 and 1, and Mean() indicates taking the average of the confidences of each sub-classification; S200, using the data set to train the network model to obtain a face forgery detection model; S300: Import the image or video to be detected into the network model for recognition to obtain the final prediction result.

2. The method for detecting face forgery based on visual converter according to any one of claims 1, characterized in that: In step S200, when training the network model, the classification loss and the contrast loss are combined as the training loss, which is represented as follows: L = L cls +L con ; Among them, the forms of classification loss and contrast loss are as follows: In the formula, y and Represents the label value and predicted value of the input image, Z i Represents the normalized result of the classification token features output by the Transformer layer, Z j Represents the normalized result of the classification token features output by the Transformer layer.

3. The face forgery detection method based on visual converter according to any one of claims 1, characterized in that: The network model includes 4 Transformer blocks, and each Transformer block includes 3 first Transformer layers.

Images