Depth forgery detection method based on mask supervision

Abstract

The invention relates to the technical field of computer vision and deep learning, and relates to a deep forgery detection method based on mask supervision. According to the technical scheme, an image is input to a preprocessing module for preprocessing; the processed image is respectively input into a mask supervision module for extracting spatial domain features and a frequency domain feature extraction module for extracting frequency domain features; and the spatial domain features and the frequency domain features are connected and input to a classification module to obtain a final prediction result. The method has the beneficial effects that the attention of the model to the irrelevant region is supervised and corrected through the mask, and a pixel-level label is used for guiding the model to face the synthesis region of the face, so that more accurate extraction of spatial features is ensured; the depth forgery detection method based on mask supervision has remarkable advantages in the aspects of detection accuracy, robustness, generalization ability, calculation efficiency, interpretability, overall performance optimization and the like, and a new solution is provided for the field of depth forgery detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and relates to a deep counterfeit detection method based on mask supervision. Background Art

[0002] The field of deepfake detection has made remarkable progress in recent years, with innovative research emerging continuously. Detection methods in spatial, frequency, and cross-modal domains have been developed, and interdisciplinary methods such as semantic segmentation, model compression, and fake voice detection have been introduced, bringing new vitality to the field of deepfake detection. Despite significant performance improvements compared to the machine learning era, deepfake detection in compressed scenarios remains a major challenge. Specifically, most images and videos circulated on social platforms have undergone post-processing operations such as compression and filtering. These post-processing operations weaken or even eliminate detection cues, causing existing deepfake detection methods to significantly degrade or even fail.

[0005] Deep fake detection methods are classified into manual feature-based detection methods and deep feature-based detection methods according to the type of extracted features. At present, deep fake detection methods based on deep features are gradually becoming mainstream. In addition, the present invention subdivides the features extracted by the deep learning model into three types on this basis: spatial domain, frequency domain, and a combination of spatial domain and frequency domain.

[0006] A. Spatial Domain

[0007] The spatial domain, also known as the image space, processes the pixels of the image. Typical spatial domain features include four categories: biometric features, texture and edge features, identity features, and latent features.

[0008] Biometrics: refers to features based on biometric information such as face, fingerprint, iris, etc.

[0009] Texture and edge features: These features describe the texture structure of the area in the image and the edge information of the object, and are often used for image segmentation and recognition.

[0010] Identity features: related to the identification of specific objects in an image, such as faces, license plates, etc.

[0011] Latent features: These are features that are not easily observed directly and may need to be extracted by more complex algorithms or models, such as the features that deep learning models are able to learn from large amounts of data.

[0012] Biometrics. Since eyebrows are located at the edge of the face, the eyebrow region is more susceptible to fusion boundaries during face swapping. Nguyen et al. established the utility of eyebrows in deep fake detection through biometric comparison. In addition, researchers noticed that many fake videos only involve tampering in a single field, namely the exchange of faces or sound sources, so lip shape features are used as the basis for detection. Deep fake detection using lip shape is usually accompanied by voice, and abnormal videos are identified through joint recognition of the two modalities. Agarwal et al. extracted key phonemes and their corresponding frames from the video, and identified fake videos by the mismatch between the correspondence between lip shape and typical phonemes such as M, B, and P. However, these methods mainly focus on facial information that is sensitive to the human eye, may not be applicable to all datasets, and there is a risk of failure if the data source changes.

[0013] Texture and edge features. Fusion is an indispensable step in the face tampering process: face replacement involves the fusion process between faces of different identities, and face reproduction involves the fitting process of different facial feature points. Artifacts introduced by fusion (such as forged boundaries, unnatural fusion areas, GAN fingerprints, etc.) can serve as clues for deep fake detection. Sun et al. extracted facial feature points and input them into a recurrent neural network to distinguish forged videos by unnatural expressions and discontinuous movements. Zhao et al. designed a texture enhancement module and an attention module to enhance the distinguishability of features. Using only real images for deep fake detection is one of the main methods of texture and boundary feature based methods. The forged data required by this type of method is actively generated by the model based on real data, which requires a small amount of data and performs well, so it has received a lot of attention. Li et al. proposed a method called "face X-ray" to guide the model to detect the fusion boundary of the image by actively generating forged faces. In order to generate higher quality training data, Chen et al. adopted the GAN structure, designed different face swapping areas and fusion types, and trained the generator to produce realistic forged images in an adversarial manner, while enhancing the detection ability of the discriminator. In order to exclude irrelevant background, identity and other information from detection, Shiohara et al. innovatively proposed a "self-face swapping" data generation method. The generated fake images maintain the same identity, but add artifacts of different face swapping operations, allowing the model to focus more on the artifacts themselves.

[0014] Latent features. The specific meaning of latent features is not clearly defined, and they are mainly extracted through networks such as MesoNet, Xception, ResNet, EfficientNet, MkfaNet, etc. These networks have good portability and the features they extract are universal, making them commonly used backbone networks in subsequent research and making important contributions. Although these methods can achieve good performance on raw images, the spatial domain information is easily affected by post-processing operations of the image (such as compression). Once the image quality decreases, their performance may be greatly reduced.

[0015] B. Frequency Domain

[0016] Existing deepfake detection models usually face the problem of being not robust enough. To solve this problem, researchers began to explore the frequency domain information of images. Common frequency domain transformation methods include wavelet transform and discrete cosine transform (DCT). Jia et al. trained a two-branch network based on static wavelet decomposition to extract inconsistencies within and between images to distinguish between real and fake images. Wolter et al. believed that wavelet transform retains spatial information to a certain extent, so they compared the wavelet coefficients of real and fake images to make authenticity judgments. Since DCT is a key step in JPEG compression, its application is more extensive. Qian et al. proposed the Frequency Forged Face Network (F3-Net), which uses DCT to transform from the spatial domain to the frequency domain and adopts complementary frequency-aware cues to achieve good performance in detecting low-quality forged faces. Frank et al. used two-dimensional DCT to extract frequency domain information to efficiently detect forged images. In addition, frequency domain transformation also plays an important role in identifying GAN fingerprints. Jeong et al. designed FrePGAN, which can generate frequency-level perturbation maps to enhance the generalization ability of deepfake detectors. Huang et al. argue that GANs primarily use isotropic convolutions to generate their outputs, thus leaving fingerprints on subbands extracted by anisotropic transforms. Their model uses fully separable wavelet transforms and multiwavelets to obtain anisotropic features as a basis for classification. These methods show that frequency domain information can provide more robust and effective clues for deep fake detection.

[0017] C. Combination of spatial domain and frequency domain

[0018] In order to obtain more comprehensive information, researchers have tried to integrate spatial and frequency domain features many times. Liu et al. proposed a shallow network that combines spatial images and phase spectra to capture upsampling artifacts. Li et al. designed a spatial-frequency domain fusion module and confirmed the effectiveness of the fusion operation through ablation experiments. This method also introduced a new loss function, called single center loss, to enhance intra-class compression and inter-class separability. Luo et al. integrated the high-frequency features extracted by SRM with spatial features, and their detection model achieved excellent performance in cross-dataset evaluation. Gu et al. proposed a dual-branch deep fake detection network with enhanced learning capabilities, each branch dedicated to extracting fine-grained features in the spatial and frequency domains. In the method proposed by Wang et al., wavelet transformed features are combined with spatial features for classification.

[0019] These studies show that deepfake images can be detected more effectively by combining features from the spatial and frequency domains. These methods not only improve the accuracy of detection, but also enhance the generalization ability of the model to different types of forgeries by using different features and loss functions. Summary of the invention

[0020] In order to solve the above-mentioned technical problems existing in the prior art, the present invention provides a deep fake detection method based on mask supervision. The method is based on mask supervision and frequency domain conversion. By introducing pixel-level labels, mask supervision is introduced into the model to correct the model's attention to the forged area and guide the extraction of spatial features. The frequency domain clues are more robust and not easily affected by post-processing effects such as compression.

[0021] To achieve the above object, the present invention provides the following technical solutions:

[0022] A deep fake detection method based on mask supervision, the steps are as follows:

[0023] S1, input image to preprocessing module for preprocessing;

[0024] S2, the processed image is input into the mask supervision module for extracting spatial domain features and the frequency domain feature extraction module for extracting frequency domain features respectively;

[0025] S3. The spatial domain features and the frequency domain features are connected and input into a classification module to obtain a final prediction result.

[0026] Furthermore, image cropping and / or data enhancement are performed in the preprocessing module.

[0027] Furthermore, after the image enters the mask supervision module, the backbone network HRNet extracts the intermediate features of the image and then divides them into two directions, one for upsampling to generate a predicted mask, and the real mask is used for supervision; the other is used to generate features connected with the frequency domain.

[0028] Furthermore, in the frequency domain feature extraction module, the image is first subjected to DCT transformation, and then data of different frequency bands are obtained through three different frequency band filters, which are reconstructed and spliced; the connected data is input into the backbone network Xception to obtain the features of the frequency domain stream.

[0029] Furthermore, the data enhancement includes image flipping, grayscale conversion, and color jittering.

[0030] Furthermore, the frequency domain feature extraction module performs the following steps:

[0031] S31. Perform discrete cosine transform on the preprocessed image Ip. Let DCT(·) represent discrete cosine transform and IDCT(·) represent inverse discrete cosine transform. Therefore, the space-frequency conversion process can be expressed as follows:

[0032] I DCT =DCT(Ip) (1)

[0033] The obtained frequency domain image I DCT It is input to the low-pass filter f0(·), the band-pass filter f1(·), and the high-pass filter f2(·) to obtain a frequency domain image containing different frequency bands. The filtering process is represented by the following equation, where f i (·) represents different frequency band filters; this process allows the model to analyze the image at different frequency levels, thereby capturing features at different scales; the mathematical expression of the filtering process can be written as:

[0034]

[0035] S32, performing inverse discrete cosine transform on the frequency domain images of different frequency bands to reconstruct the image; the reconstructed image is converted into recon__i Expressed by the following formula:

[0036]

[0037] S33, the reconstructed images of different frequency bands are connected along the channel dimension, represented by concat(·), to form the input Ic to the backbone network, which is expressed as:

[0038] I c =concat(i recon__i ),i=0,1,2 (4)

[0039] S34. Will I c Input to the backbone network f re (·) to extract the frequency domain features I fre , expressed as follows

[0040] I fre =f fre (I c ) (5)

[0041] The frequency domain feature extraction module selects the input stream and intermediate stream of the Xception network as the backbone network; when the input stream and intermediate stream are used to extract frequency domain features, the number of input channels of the backbone network increases from 3 in the original Xception structure to 9.

[0042] Furthermore, the mask supervision module performs the following steps: using HRNet as the backbone network for extracting spatial features to perform human posture estimation and semantic segmentation; inputting the preprocessed image Ip into the backbone network HRNet and obtaining the intermediate feature Im, as shown below:

[0043] I m =f spatial (I p ) (6)

[0044] Among them, f spatial (·) represents the backbone network, the intermediate feature I m It consists of four feature maps of different sizes, denoted as The value of i is 1 to 4, and the sizes of these feature maps are arranged in order from large to small, namely 64×64, 32×32, 16×16, and 8×8; then, these features are sent to two branches for processing; one branch is the mask prediction branch responsible for generating predictions, and the other branch is the spatial feature extraction branch for spatial feature extraction.

[0045] Further, after entering the mask prediction branch, feature maps Im2, Im3 and Im4 are upsampled to match the size of Im1 and then connected along the channel dimension; this process is expressed as follows:

[0046] I c1 =concat(f up (I m__i )),i=2,3,4 (7)

[0047] Then, the mean operation is applied to obtain the prediction mask I mask ; This process is expressed as follows:

[0048] I mask =f mean (Ic1 ) (8)

[0049] Finally, the calculated prediction mask I mask The loss is calculated by comparing with the ground truth pixel-level label Igt; the loss function used is the mean square error, and the calculation process of the loss function is summarized by the following formula:

[0050]

[0051] Where M and N represent the size of the mask, M = N = 256; I mask,ij and I gt,ij represent the pixel values ​​at the corresponding positions in the mask and the ground truth pixel-level labels, respectively.

[0052] Furthermore, in the spatial feature extraction branch, upsampling and downsampling operations are included; the size of the intermediate feature is adjusted to match I m3 , that is, 16×16; the process of obtaining the connection feature Ic2 is expressed by the following formula

[0053] I c2 =concat(f down (I m__1 )),f down ( I__2 ),I m__3 ,f up (I m__4 ) (10)

[0054] Finally, the connected feature Ic2 is expanded to obtain the spatial feature I spatial

[0055] I spatial =f e (I c2 ) (11)

[0056] Furthermore, the classification module performs the following steps:

[0057] S41, connecting the features of the spatial domain and the frequency domain;

[0058] S42, input the connected features into the classification network f exit (·) to obtain the prediction results It is expressed as follows.

[0059] =f exit (concat(f fre ,f spatial )) (12)

[0060] The classification network uses the output stream of the Xception network. In this module, the classification network only adjusts the size of the input features according to the original output stream;

[0061] The classification module uses cross entropy as the loss function, which is defined as follows:

[0062]

[0063] Where y represents the ground truth label, y = 1 represents a real image, y = 0 represents a fake image, and The label representing the model's prediction;

[0064] The overall loss function consists of two parts: the mean square error function of the mask supervision module and the cross entropy function of the classification module; the total loss function is the sum of these two:

[0065] Loss = L MSE +L ce (14).

[0066] Beneficial effects of the present invention:

[0067] Compared with the prior art, the deep fake detection method based on mask supervision described in the present invention corrects the model's attention to irrelevant areas through mask supervision, and uses pixel-level labels to guide the model towards the synthetic area of ​​the face, ensuring more accurate extraction of spatial features. In addition, the present invention also combines a frequency domain feature extraction module, which uses the robustness of frequency domain cues to compression artifacts; first preprocess the input image, and then input it into the mask supervision module and the frequency domain feature extraction module; the mask supervision module uses the HRNet network to extract intermediate features, and refines the spatial features by guiding the predicted mask with the real mask; the frequency domain module applies DCT transform and filtering of different frequency bands to extract features; finally, the spatial and frequency domain features are connected and input into the classification network, and the final prediction result is output; experimental results show that the method of the present invention maintains good robustness in compression scenarios.

[0068] The deep fake detection method based on mask supervision in the present invention realizes effective detection of deep fake content by combining feature extraction in spatial domain and frequency domain and utilizing mask supervision mechanism. Its beneficial effects are mainly reflected in the following aspects:

[0069] Improve detection accuracy: Combining the features of the spatial domain and the frequency domain enables the detection method to capture detailed information of the image in different dimensions. The spatial domain features focus on the pixel-level information and local structure of the image, while the frequency domain features reveal the global characteristics and frequency distribution of the image. The two complement each other and improve the comprehensiveness and accuracy of detection.

[0070] Enhanced robustness: Through the use of DCT transform and frequency band filters, the detection method has a certain robustness to post-processing operations such as image compression and filtering. These operations are widely used in social media, and the method of the present invention can still maintain high detection performance in these cases.

[0071] Mask supervision mechanism improves detection effect: The mask supervision mechanism guides the model to focus on key areas in the image by generating a predicted mask and comparing it with the real mask. This mechanism helps the model focus more on the forged areas, thereby improving the pertinence and accuracy of detection.

[0072] Flexible data enhancement strategies: In the preprocessing stage, various data enhancement methods such as image cropping, flipping, grayscale, and color jittering are used to increase the diversity and complexity of training data, which helps to improve the generalization ability of the model.

[0073] Efficient feature extraction network: HRNet and Xception are used as the backbone network to extract features in the spatial domain and frequency domain respectively. These networks have powerful feature extraction capabilities and high computational efficiency, which can reduce computational costs while ensuring detection performance.

[0074] Strong interpretability: By generating prediction masks, this method not only provides the final detection results, but also can visually display the areas that the model focuses on, enhancing the interpretability of the detection results and helping users understand and trust the detection results.

[0075] Joint optimization of multiple loss functions: The mean square error and cross entropy loss functions are jointly optimized, so that the model can simultaneously consider the performance of mask prediction and classification tasks during training, thereby achieving the optimization of overall performance.

[0076] In summary, the deep fake detection method based on mask supervision of the present invention shows significant advantages in detection accuracy, robustness, generalization ability, computational efficiency, interpretability and overall performance optimization, providing a new solution for the field of deep fake detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below in combination with the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0078] Figure 1 is a flow chart of a deep fake detection method based on mask supervision of the present invention;

[0079] Figure 2 It visualizes real faces and four common ways they are forged in FaceForensics++;

[0080] Figure 3 It is the framework of the mask supervision module of the present invention;

[0081] Figure 4 is a structural diagram of the mask prediction branch of the present invention;

[0082] Figure 5 It is a structural diagram of the feature extraction branch of the present invention;

[0083] Figure 6 is an ablation experiment result diagram of the deep forgery detection method based on mask supervision of the present invention;

[0084] Figure 7 is the mask image predicted by the model of the present invention;

[0085] Figure 8 It is a Grad-CAM visualization diagram based on the mask supervision model of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Figure 1-8 The deep fake detection method based on mask supervision is further explained, and the technical solutions in the embodiments of the present application are clearly and completely described; obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and all other embodiments obtained by ordinary technicians in this field without creative work based on the embodiments in the present application are within the scope of protection of the present application.

[0087] Example 1

[0088] In recent years, artificial intelligence technology has been widely used, which has brought people security and convenience, but also brought certain risks. Among them, deep fake technology creates convincing but fake performances by manipulating facial images, which has caused major security issues. In order to improve detection performance in this case, the present invention introduces a novel deep fake detection method based on mask supervision. The deep fake detection method based on mask supervision proposed in the present invention is based on mask supervision and frequency domain conversion. After compression, the artifacts in the deep fake portrait are weakened, resulting in the model's inability to distinguish between real and fake areas in the image, thereby diverting attention from the facial area. In addition, Xception exhibits an overly small attention area problem when detecting fake images generated by Deepfake and NeuralTexture, and therefore fails to cover the facial fake area. Therefore, the present invention introduces mask supervision into the model by introducing pixel-level labels to correct the model's attention to the fake area and guide the extraction of spatial features. In addition, studies have shown that frequency domain cues are more robust and less susceptible to post-processing effects such as compression. Therefore, the present invention integrates a frequency domain information extraction module into the proposed method.

[0089] A. Overall process of the method

[0090] The overall framework of the deep fake detection method based on mask supervision is as follows Figure 1 As shown. The whole model consists of four modules, namely the preprocessing module, the mask supervision module, the frequency domain feature extraction module and the classification module. The training process of the model can be summarized as follows: First, the input image is preprocessed, such as image cropping and data enhancement, and the processed image is input into the mask supervision module and the frequency domain feature extraction module respectively. The mask supervision module mainly extracts spatial domain features. After the image enters the module, the backbone network HRNet (high resolution network) extracts the intermediate features of the image, and then divides it into two directions, one for upsampling to generate a prediction mask, and the real mask is used for supervision. In this way, the forged area concerned by the intermediate feature can be more accurate. The other is used to generate features connected to the frequency domain. In the frequency domain feature extraction module, the image is first transformed by DCT, and then the data of different frequency bands are obtained through three different frequency band filters for reconstruction and splicing. The connected data is input into the backbone network Xception to obtain the features of the frequency domain stream. In the classification module, the spatial domain and frequency domain features are connected and input into the classification network to obtain the final prediction result.

[0091] B. Introduction to main modules

[0092] This section introduces the preprocessing module, frequency domain feature extraction module, mask supervision module and classification module in the deep fake detection method based on mask supervision.

[0093] 1) Preprocessing module: The preprocessing module is the first step of the method of the present invention, which mainly includes two operations: image cropping and data enhancement. The purpose of image cropping is to recognize faces and adjust the image size to fit the model input. Data enhancement refers to applying a series of random transformations and perturbations to the original image to generate new training samples. The data enhancement used in the present invention includes image flipping, grayscale, color jittering, etc. The images generated by the above transformations are used as new training data to expand the scale of the training set. In addition, by introducing new samples, the model's dependence on specific data distribution can be reduced, enabling it to better adapt to different types of images and transformations, learn more generalized feature representations, and effectively alleviate overfitting problems. The data-enhanced image Ip is used as input to the mask supervision module and the frequency domain feature extraction module.

[0094] 2) Frequency domain feature extraction module: The frequency domain feature extraction module consists of five steps: space-frequency conversion, filtering, reconstruction, connection and feature extraction.

[0095] First, the preprocessed image Ip is subjected to discrete cosine transform (DCT). Let DCT(·) represent discrete cosine transform and IDCT(·) represent inverse discrete cosine transform. Therefore, the space-frequency conversion process can be expressed as follows:

[0096] I DCT =DCT(I p ) (1)

[0097] The obtained frequency domain image IDCT is input to a low-pass filter f0(·), a band-pass filter f1(·), and a high-pass filter f2(·) to obtain a frequency domain image IDCTi containing different frequency bands. The filtering process is represented by the following equation, where fi(·) represents different frequency band filters. This process allows the model to analyze the image at different frequency levels, thereby capturing features of different scales. The mathematical expression of the filtering process can be written as:

[0098]

[0099] Subsequently, the frequency domain images of different frequency bands are subjected to inverse discrete cosine transform (IDCT) to reconstruct the image. The reconstructed image is represented by Irecon__i. This process can be represented by the following formula:

[0100]

[0101] The reconstructed images of different frequency bands are connected along the channel dimension, represented by concat(·), to form the input Ic to the backbone network, which can be expressed as:

[0102] I c=concat(i recon__i ),i=0,1,2 (4)

[0103] Finally, Ic is input into the backbone network fre(·) to extract the frequency domain feature Ifre, which can be expressed as follows

[0104] I fre =f fre (I c ) (5)

[0105] The frequency domain feature extraction module selects the input stream and the intermediate stream of the Xception network as the backbone network. It is worth noting that since the present invention extracts information from the input image in the low frequency, medium frequency and high frequency bands and reconstructs it in the spatial domain, when the input stream and the intermediate stream are used to extract frequency domain features, the number of input channels of the backbone network increases from 3 of the original Xception structure to 9.

[0106] Despite the spatial reconstruction and connection, the generated feature map Ifre already contains multi-scale frequency domain information, not just pure spatial features. This information can provide rich feature representation for subsequent models.

[0107] 3) Mask Supervision Module: The purpose of the mask supervision module is to use pixel-level labels to guide the feature extraction process in the spatial domain. Figure 2 Four common tampering methods in FaceForensics++ and their corresponding pixel-level masks are shown. In these masks, black areas represent untampered areas, while white areas represent tampered areas. The mask of the real image is composed entirely of black pixels. The mask generated by the Deepfake technology is relatively simple, and the tampered area is presented as a rectangle. The masks of the other three tampering techniques are also similar, except that the mouth area in the FaceSwap image is real.

[0108] The mask supervision module uses HRNet as the backbone network for extracting spatial features. HRNet is a deep neural network architecture used for human pose estimation and semantic segmentation. Its main feature is that it retains information of the same resolution at the same time and better captures details and global context by forward passing information in parallel. The specific structure of the backbone network in the mask supervision module can be found in Figure 3 The process of inputting the preprocessed image Ip into the backbone network HRNet and obtaining the intermediate feature Im can be expressed as follows:

[0109] I m =f spatial (I p ) (6)

[0110] In this module, fspatial(·) represents the backbone network, that is, HRNet. From the output of the last layer of the backbone network, we can see that the intermediate feature I m It consists of four feature maps of different sizes, denoted as I mi ,i has a value of 1 to 4. The sizes of these feature maps are arranged from large to small, namely 64×64, 32×32, 16×16, and 8×8. Then, these features are sent to two branches for processing. Figure 8 The branch marked as “upsampling” in the figure is the mask prediction branch, which is responsible for generating the predicted mask; the other branch is used for spatial feature extraction. Figure 4 and Figure 5 The specific structures of these two branches are shown.

[0111] Mask prediction branch. Figure 4 As shown, after entering the mask prediction branch, the feature map I m2 ,I m3 and I m4 is upsampled to match I m1 The dimensions are then connected along the channel dimension. This process can be expressed as follows:

[0112] I c1 =concat(f up (I m__i )),i=2,3,4 (7)

[0113] Then, the mean operation is applied to obtain the prediction mask I mask . This process can be expressed as follows:

[0114] I mask =f mean (I c1 ) (8)

[0115] Finally, the calculated prediction mask I mask Compared with the ground truth pixel-level labels I gt Compare to calculate the loss. The loss function used here is the mean square error (MSE), and the calculation process of the loss function can be summarized by the following formula:

[0116]

[0117] In the present invention, M and N represent the size of the mask, M = N = 256. mask,ij and Igt,ij represent the pixel values ​​at the corresponding positions in the mask and ground truth pixel-level labels, respectively. The introduction of mean squared error (MSE) loss helps to make the distribution of intermediate features closer to the distribution of forged areas, thereby correcting the focal area of ​​the model.

[0118] Spatial feature extraction branch. Figure 5 As shown in Figure 2, in the spatial feature extraction branch, there are upsampling and downsampling operations. In the HRNet model, the feature map I m1 ,I m2 and I m3 is downsampled to match I m4 In the present invention, since the spatial features extracted by the mask supervision module need to be combined with the frequency domain features, the size of the intermediate features is adjusted to match I m3 , that is, 16×16. Get the connection feature I c2 The process can be expressed by the following formula

[0119] I c2 =concat(f down (I m__1 )),f down (I __ 2),I m__3 ,f up (I m__4 ) (10)

[0120] Finally, the extended connection feature I c2 To obtain the spatial feature I spatial

[0121] I spatial =f e (I c2 ) (11)

[0122] 4) Classification module: First, the features of the spatial domain and the frequency domain are concatenated. Then, the concatenated features are input into the classification network f exit (·) to obtain the prediction results This can be expressed as follows.

[0123] =f exit (concat(f fre ,f spatial ))(12)

[0124] The classification network uses the output stream of the Xception network. In this module, the classification network only adjusts the size of the input features according to the original output stream.

[0125] The classification module uses cross entropy as the loss function, which is defined as follows:

[0126]

[0127] In this context, y represents the ground truth label, y=1 represents a real image, y=0 represents a fake image, and Represents the label predicted by the model.

[0128] The overall loss function of the deep fake detection method based on mask supervision consists of two parts: the mean square error function of the mask supervision module and the cross entropy function of the classification module. The total loss function is the sum of these two:

[0129] Loss = L MSE +L ce (14)

[0130] The reason why the model's performance drops significantly when tested on low-quality data is that after compression, some image artifacts that can serve as detection clues are weakened or even eliminated. Based on this, the present invention designs two strategies: one is to obtain more comprehensive features by integrating information in the spatial and frequency domains, and the second is to reduce the possibility of key features disappearing; and restore the network's attention to the forged area during the compression process. Combining these two strategies, in order to improve the model's detection performance on compressed facial images, the present invention proposes a deep forgery detection method based on mask supervision. In order to make full use of the advantage that the combination of spatial and frequency domain features is not easily affected by compression operations, the present invention designs a frequency domain feature extraction module to capture the frequency domain information of the image. At the same time, in order to enhance the model's ability to identify forged areas, a mask supervision mechanism is introduced in the spatial feature extraction process, and the model is guided by pixel-level labels so that it can focus on and identify the tampered parts of the image more accurately. Finally, the spatial and frequency domain features are concatenated and input into the classification network to obtain the final prediction results.

[0131] The contributions of the present invention are summarized as follows:

[0132] This paper proposes a deep fake detection method based on mask supervision, which combines spatial and frequency domain features and is suitable for detecting fake images with different compression levels. By cleverly integrating these two types of features, the impact of image compression on detection effectiveness is reduced.

[0133] The present invention designs a detection framework, including preprocessing, mask supervision, frequency domain feature extraction and classification modules. The collaborative operation of these modules not only enhances the expressiveness of features, but also improves the accuracy and robustness of detection.

[0134] The present invention has conducted extensive experiments on public datasets with different compression levels, fully verifying the effectiveness of the proposed method. The experimental results show that the method of the present invention has significant performance advantages in detecting compression forged images.

[0135] Example 2

[0136] This example provides experimental verification using the method described in Example 1.

[0137] A. Experimental Setup

[0138] Dataset: The FaceForensics++ (FF++) dataset is used to evaluate the proposed method. The FaceForensics++ dataset contains 1,000 original videos that are processed by four different facial manipulation techniques and converted into corresponding fake videos. The four manipulation techniques include Deepfake (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTexture (NT). The videos in the FF++ dataset are compressed into two versions: lightly compressed (c23) and heavily compressed (c40), using the H.264 codec and obtained with constant rate quantization parameters of 23 and 40, respectively.

[0139] The present invention uses the facial recognition tool MTCNN to detect, identify and align faces in video frames, and the cropped faces are resized to 256×256 pixels. The present invention uses 68 facial landmarks from Dlib to generate corresponding masks for the four forgery methods in the FF++ dataset. The mask of the real image is an image with all black pixels. 50 frames are selected as training data for each video in the training set, while 100 frames are selected for each video in the validation set and the test set to ensure a balance in the amount of training and test data.

[0140] Experimental settings. The backbone networks Xception and HRNet have been pre-trained on ImageNet. The Adam optimizer is used with the following parameters: β1 = 0.9, β2 = 0.999, ε = 10 -8 ,l r =2×10 -4 The batch size is 32 and the training cycle is 8 epochs.

[0141] Evaluation indicators. The present invention uses accuracy (ACC) and the area under the receiver operating characteristic curve (ROC) (AUC) as evaluation indicators.

[0142] B. Experimental Results

[0143] In order to comprehensively evaluate the performance of the proposed deep fake detection method, this paper sets up three experimental scenarios according to the image compression level, namely, uncompressed, lightly compressed and heavily compressed scenarios.

[0144] 1) Uncompressed scene: In the uncompressed scene, the evaluation results of the proposed method are shown in Table I. It can be seen from the table that due to the rich artifact information in the uncompressed scene, the detector can find the forgery clues more easily, so the comparison method and the method proposed in the present invention maintain high performance. Specifically, the accuracy of all methods reached above 80%. Except for SupCon, which only achieved 81.57% ACC on the FaceSwap dataset, all other experimental results were above 93%. The method proposed in the present invention achieved the best and second best performance on the Deepfake and FaceSwap datasets, respectively, with an average accuracy of 98.81% on the four sub-datasets.

[0145] TABLEI

[0146] ACC results in uncompressed scenarios (%)

[0147] Method DF FS F2F NT Avg Steg.Features

[30] 99.03 98.89 99.13 99.88 99.23 D-CNN

[39] 98.03 98.94 98.96 96.06 98.00 MesoNet

[17] 96.37 98.17 97.95 93.30 96.45 SupCon

[40] 98.31 97.10 97.75 96.45 97.40 Xception

[18] 99.11 81.57 96.57 99.11 94.09 ResNet50

[19] 99.04 99.16 98.93 97.82 98.74 Ours 99.21 99.13 98.98 97.90 98.81

[0148] In addition, the present invention evaluates the AUC index of the proposed mask supervision method, and the results are shown in Table II. As can be seen from the table, the mask supervision method performs better than the backbone network Xception on the FF++ dataset, especially on the NeuralTexture dataset with higher forgery quality, its AUC is 2.1% higher than Xception, which fully proves the effectiveness of mask supervision.

[0149] TABLE II

[0150] AUC results in the uncompressed scenario (%)

[0151] Method DF FS F2F NT Avg MesoNet 98.72 99.37 98.60 96.87 98.39 Xception 98.98 98.04 99.15 97.49 98.42 SupCon 99.27 86.50 99.34 99.59 96.18 Ours 99.84 99.81 99.76 99.59 99.75

[0152] Combining the observations in Table I and Table II, it can be seen that most methods show high detection performance when tested on the uncompressed version of the public dataset. Therefore, the present invention will not further explore the performance of the model in the uncompressed scenario.

[0153] 2) Mildly compressed scenario: The evaluation results of ACC in the mildly compressed scenario are shown in Table III. Due to the weakening of artifact information in the forged portrait, the overall performance of the detection method is reduced. In the mildly compressed scenario, the mask supervision method proposed in this paper achieves an average accuracy of 96.50%, and achieves the second and best accuracy on Face2Face and NeuralTexture, respectively.

[0154] It is worth mentioning that NeuralTexture is a powerful deep fake generation technology that can extract textures from an original image and apply them to another image using a neural network. This process ensures that the output fake face has high resolution and high detail texture. Since NeuralTexture retains the original color and details, effectively detecting fake faces generated by it remains a major challenge. The model proposed in this paper achieved the best performance on the NeuralTexture dataset, which is 5.85% higher than the second best method, proving the robustness of the method of this invention.

[0155] The evaluation results of the AUC index in the mild compression scenario are shown in Table IV. As can be seen from the table, the method of the present invention has achieved the best AUC on different datasets, achieving an average AUC of 99.38%. The AUC on the Deepfake dataset is the highest, reaching 99.52%. Mask supervision can more accurately locate and utilize subtle changes.

[0156] The subtle changes associated with deep fakes enhance the model's ability to identify this dataset. On other datasets, the proposed method also maintains a high level of AUC, demonstrating its robustness in detecting different types of fakes.

[0157] 3) Severely compressed scenarios: The evaluation results of ACC and AUC in heavily compressed scenarios are shown in Table V and Table VI, respectively. As can be seen from the table, in heavily compressed scenarios, the performance of traditional deep fake detection methods (Steg.Features and Cozzolino et al.) on the heavily compressed subset of FF++ is relatively low because these methods rely on handcrafted features. In contrast, deep learning methods such as F3-Net, MesoNet, Xception, and the method proposed in this invention can automatically learn complex feature representations of the data, so they outperform traditional methods. This shows that deep learning methods are effective in the task of forged portrait detection in heavily compressed scenarios.

[0158] Although the performance of the proposed method is not optimal, it still provides some improvement in overall performance compared to traditional image processing techniques and methods like MesoNet. On the NeuralTexture dataset, the test accuracy of the proposed method exceeds 97.5% and the AUC exceeds 99.5%. The NeuralTexture dataset is the most challenging detection object, but even on the most challenging heavily compressed (c40) subset of the NeuralTexture dataset, the mask supervision method proposed in this paper achieves a high accuracy of 97.76%, an increase of nearly 30% compared to ADD, and an AUC of 99.56%, an increase of 13.94% compared to MesoNet.

[0159] The method proposed in the present invention uses Xception as the backbone network. However, on the Face2Face dataset, the ACC of the proposed model is lower than that of Xception, and the same phenomenon is observed when evaluating the AUC indicator. The present invention analyzes the reasons for this phenomenon in detail: the present invention believes that the reason is feature conflict, that is, in a heavily compressed scenario, pixel-level supervision conflicts with the intrinsic characteristics of the model. Although the model is learning global features, it has difficulties considering pixel-level detailed information. Compared with Xception, the spatial-frequency fusion branch is more complex in structure, and the increased complexity leads to overfitting, which in turn leads to a decrease in model performance.

[0160] TABLE III

[0161] ACC results in mild compression scenarios (%)

[0162]

[0163]

[0164] TABLE IV

[0165] AUC results in the light compression scenario. (%)

[0166] Method DF FS F2F NT Avg MesoNet 95.79 98.97 97.57 86.56 94.72 Xception 97.77 99.27 98.92 92.52 97.12 SupCon 98.85 78.42 99.12 97.59 93.50 Ours 99.52 99.47 99.41 99.10 99.38

[0167] TABLE V

[0168] ACC results in a heavy compression scenario. (%)

[0169]

[0170]

[0171] TABLE VI

[0172] AUC results in heavy compression scenarios. (%)

[0173] Method DF FS F2F NT Avg MesoNet 85.32 87.46 92.15 85.72 87.66 Xception 90.93 94.13 95.01 95.21 93.82 SupCon 92.69 53.52 71.81 95.93 78.49 Ours 94.15 92.37 90.13 99.56 94.05

[0174] C. Ablation studies

[0175] 1) Effectiveness of spatial-frequency feature integration and mask supervision: In order to evaluate the contribution of spatial-frequency feature integration and mask supervision, the present invention compares the performance of the backbone networks HRNet and Xception with the mask supervision method, using ACC as the evaluation metric. The results are presented in Table VII. In the lightly compressed scenario, the accuracy of the model is comprehensively improved after the integration of mask supervision and spatial-frequency combination, especially on the NeuralTexture dataset, where a significant improvement is observed. In the heavily compressed scenario, the performance of the mask supervision method on the Face2Face and FaceSwap datasets is slightly reduced compared to Xception. We believe that there are two reasons for this phenomenon. First, overfitting occurs on these two subsets. Second, there is a conflict between the features extracted by the two backbone networks, and when these features are combined, they cancel each other out, resulting in a decrease in the discriminability of the obtained features and a decrease in model performance. Compared with HRNet, the performance of the mask supervision method on all datasets is improved. In summary, the overall performance of the model is better after mask supervision and spatial-frequency integration are incorporated.

[0176] 2) Effectiveness of mask supervision. In order to evaluate the contribution of real pixel-level label (mask) supervision to the overall method, the performance of the model before and after integrating mask supervision is tested on the heavily compressed version of the FaceForensics++ dataset, using ACC as the evaluation metric. Figure 5 The structure of the “no mask supervision” model in the figure is the same as the mask supervision method, except for the lack of pixel-level label supervision. As can be seen from the figure, the integration of mask supervision leads to an overall improvement in model performance, especially on the Deepfake dataset.

[0177] D. Visualization

[0178] First, we visualize the predicted masks in the deepfake detection method based on mask supervision, and the results are shown in Figure 7 The predicted mask is a grayscale image, where black pixels indicate that the predicted pixels are real and white pixels indicate that the predicted pixels are fake. For the predicted masks of different categories of images, in most cases, as shown in the first two rows in the figure, the pixel distribution is close to the distribution of the true pixel-level labels.

[0179] It is worth noting that in the Face2Face dataset, white pixels appear in the unmodified background area. The present invention believes that the reason for this phenomenon is the bidirectional influence between the predicted mask and the intermediate features, that is, the predicted mask not only affects the formation of the intermediate features, but the intermediate features also adjust the shape of the predicted mask in turn. This interaction leads to the complexity of the intermediate features, which are not only directly supervised by the true pixel-level labels, but also indirectly affected by the cross-entropy loss. This may cause the pixels in the predicted mask to not completely match the true pixel-level labels, such as Figure 7 The last row shows the extreme case.

[0180] TABLE VII

[0181] Comparison of mask supervision method and backbone network performance (%).

[0182]

[0183] Specifically, for Deepfake and FaceSwap images, the predicted masks are very close to the real masks, but there are some cases, such as the third row in the figure, where the predicted masks lack facial contours. For Face2Face images, the generated predicted masks have facial contours, but the background area is more chaotic compared to the first two forgery methods, which indirectly explains why the model performs poorly on this dataset. The predicted masks obtained on the NeuralTexture dataset are clearer than the first three datasets, and there are almost no cases of chaotic predicted masks. Even the worst mask can predict the contours of the face, which indirectly explains why the model performs better on the NeuralTexture dataset. In the worst case, the predicted mask of the real image may show the contours of the face, causing the model to make a misjudgment. Since the purpose of the branch where the prediction mask is located is to obtain better intermediate features rather than the final classification features, the prediction mask can only reflect the internal working mechanism and feature extraction capabilities of the model to a certain extent. Therefore, the present invention also uses Grad-CAM to visualize the final classification features of the model, and the results are shown in Figure 8 In general, the model's attention is focused on the face area, with almost no attention paid to the background or hair area. This attention distribution reflects the model's good ability to capture key features in the classification task. For Deepfake, FaceSwap, and NeuralTexture images, the model's attention is allocated to the face swap fusion boundary or the entire face swap area. However, for Face2Face images, the model's focus is on the corners of the mouth. This deviation in the attention area leads to a decrease in the model's performance on this dataset.

[0184] In summary, due to the supervision of pixel-level labels, the attention of the deep fake detection model based on mask supervision can be focused on the face, ensuring the good performance of the model. However, on a specific fake dataset, the focus area of ​​the model is relatively fixed, that is, the focus area does not change with the movement of the face, and there is a problem that the focus area is too small. This leads to a low performance ceiling for this method.

[0185] In summary, in order to solve the problem of decreased detection performance of compressed deep fake images, the present invention proposes two strategies: obtaining more comprehensive features and restoring the attention of the model. Based on this, the present invention introduces a deep fake detection method based on mask supervision. The present invention first analyzes the problems existing in the current deep fake detection method, and then proposes to solve these problems through mask supervision and frequency domain conversion. Subsequently, the present invention comprehensively explains the structure of the deep fake detection method based on mask supervision. The experimental results show that the method proposed in the present invention maintains robustness on deep fake images with different compression levels, which fully demonstrates the effectiveness of mask supervision and frequency domain conversion. In addition, the present invention also performs a visual analysis of the mask and Grad-CAM images predicted by the model, which further proves the effectiveness of the method of the present invention. However, there is still room for improvement in the performance of the method of the present invention in heavily compressed scenarios, and future work will also focus on this aspect.

[0186] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A deep fake detection method based on mask supervision, characterized in that: Here are the steps: S1, input image to preprocessing module for preprocessing; S2, the processed image is input into the mask supervision module for extracting spatial domain features and the frequency domain feature extraction module for extracting frequency domain features respectively; S3. The spatial domain features and the frequency domain features are connected and input into a classification module to obtain a final prediction result.

2. The deep fake detection method based on mask supervision as claimed in claim 1, characterized in that: Image cropping and / or data enhancement are performed in the preprocessing module.

3. The deep fake detection method based on mask supervision as claimed in claim 1, characterized in that: After the image enters the mask supervision module, the backbone network HRNet extracts the intermediate features of the image and then divides them into two directions, one for upsampling to generate a predicted mask, and the real mask is used for supervision; the other is used to generate features connected with the frequency domain.

4. The deep fake detection method based on mask supervision as claimed in claim 1, characterized in that: In the frequency domain feature extraction module, the image is first subjected to DCT transformation, and then data in different frequency bands are obtained through three different frequency band filters, which are then reconstructed and spliced; the connected data is input into the backbone network Xception to obtain the features of the frequency domain stream.

5. The deep fake detection method based on mask supervision as claimed in claim 2, characterized in that: The data enhancement includes image flipping, grayscale conversion, and color jittering.

6. The deep fake detection method based on mask supervision as claimed in claim 3, characterized in that: The frequency domain feature extraction module performs the following steps: S31, the pre-processed image I p Perform discrete cosine transform, let DCT(·) represent discrete cosine transform, IDCT(·) represent inverse discrete cosine transform; therefore, the space-frequency conversion process can be expressed as follows: I DCT =DCT(I p ) (1) The obtained frequency domain image I DCT It is input to the low-pass filter f0(·), the band-pass filter f1(·), and the high-pass filter f2(·) to obtain a frequency domain image containing different frequency bands. The filtering process is represented by the following equation, where f i (·) represents different frequency band filters; this process allows the model to analyze the image at different frequency levels, thereby capturing features at different scales; the mathematical expression of the filtering process can be written as: S32, performing inverse discrete cosine transform on the frequency domain images of different frequency bands to reconstruct the image; the reconstructed image is converted into recon__i Expressed by the following formula: S33, the reconstructed images of different frequency bands are connected along the channel dimension, represented by concat(·), to form an input I c To the backbone network, it is expressed as: I c =concat(i recon__i ),i=0,1,2 (4) S34. Will I c Input to the backbone network f re (·) to extract the frequency domain features I fre , expressed as follows I fre =f fre (I c ) (5) The frequency domain feature extraction module selects the input stream and intermediate stream of the Xception network as the backbone network; when the input stream and intermediate stream are used to extract frequency domain features, the number of input channels of the backbone network increases from 3 in the original Xception structure to 9.

7. The deep fake detection method based on mask supervision as claimed in claim 3, characterized in that: The mask supervision module performs the following steps: using HRNet as the backbone network for extracting spatial features to perform human posture estimation and semantic segmentation; p The process of inputting into the backbone network HRNet and obtaining the intermediate feature Im is expressed as follows: I m =f spatial (I p ) (6) Among them, f spatial (·) represents the backbone network, the intermediate feature I m It consists of four feature maps of different sizes, denoted as I mi , the value of i is 1 to 4, and the sizes of these feature maps are arranged in order from large to small, namely 64×64, 32×32, 16×16, and 8×8; then, these features are sent to two branches for processing; one branch is the mask prediction branch responsible for generating predictions, and the other branch is the spatial feature extraction branch used for spatial feature extraction.

8. The deep fake detection method based on mask supervision as claimed in claim 7, characterized in that: After entering the mask prediction branch, the feature map I m2 ,I m3 and I m4 is upsampled to match I m1 The dimensions are then connected along the channel dimension; this process is expressed as follows: I c1 =concat(f up (I m__i )),i=2,3,4 (7) Then, the mean operation is applied to obtain the prediction mask I mask ; This process is expressed as follows: I mask =f mean (I c1 ) (8) Finally, the calculated prediction mask I mask The loss is calculated by comparing with the ground truth pixel-level label Igt; the loss function used is the mean square error, and the calculation process of the loss function is summarized by the following formula: Where M and N represent the size of the mask, M = N = 256; I mask,ij and I gt,ij represent the pixel values ​​at the corresponding positions in the mask and the ground truth pixel-level labels, respectively.

9. The deep fake detection method based on mask supervision as claimed in claim 7, characterized in that: In the spatial feature extraction branch, upsampling and downsampling operations are included; the size of the intermediate features is adjusted to match I m3 , that is, 16×16; obtain the connection feature I c2 The process is expressed by the following formula I c2 =concat(f down (I m__1 )),f down ( I__2 ),I m__3 ,f up (I m__4 ) (10) Finally, the connected feature Ic2 is expanded to obtain the spatial feature I spatial I spatial =f e (I c2 ) (11) 10. The deep fake detection method based on mask supervision according to any one of claims 1 to 9, characterized in that: The classification module performs the following steps: S41, connecting the features of the spatial domain and the frequency domain; S42, input the connected features into the classification network f exit (·) to obtain the prediction results It is expressed as follows. =f exit (concat(f fre ,f spatial )) (12) The classification network uses the output stream of the Xception network. In this module, the classification network only adjusts the size of the input features according to the original output stream; The classification module uses cross entropy as the loss function, which is defined as follows: Where y represents the ground truth label, y = 1 represents a real image, y = 0 represents a fake image, and The label representing the model's prediction; The overall loss function consists of two parts: the mean square error function of the mask supervision module and the cross entropy function of the classification module; the total loss function is the sum of these two: Loss=L MSE +L ce (14)。

Images