A false face detection and tracing method based on feature-level decoupling

Abstract

The present invention provides a fake face detection and tracing method based on feature-level decoupling, which is divided into a progressive forgery feature extraction stage and a cross-decoupling stage. On the basis of the extracted overall forgery features, the forgery features are further divided into common forgery features and special forgery features. Common forgery features can effectively improve the generalization ability of deep forgery detectors in detection tasks, and avoid overfitting of forgery-irrelevant features and method-specific features. Special forgery features can further enhance the ability to trace the forgery method. In addition, the method also introduces an orthogonalization regularization loss to further help disentangle and enhance the generalization ability of decoupled features. Experimental results show that the method has good detection and tracing performance, providing a new perspective for face forgery detection and tracing tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of computer data processing, and in particular to a false face detection and tracing method based on feature level decoupling. Background Art

[0002] The deep fake face detection task aims to determine whether the face in a given image or video has been tampered with or synthesized by deep fake technology. From the perspective of the modeling method of forgery features, existing work can be divided into two categories: artificial construction-based methods and deep learning-based methods. Artificial construction-based methods rely on rich expert knowledge and explicitly model forgery features for detection, such as visual artifacts, head posture inconsistency, abnormal blinking frequency, irregular pupil shape, and inconsistent corneal highlights. However, this type of method can generally only construct a clear single forgery detection clue for a specific forgery method, and may not be effectively applied to data from other forgery methods, so its applicability is relatively limited. In comparison, deep learning-based methods can directly learn a variety of forgery features from large-scale forgery data, thereby achieving better detection results. This type of method captures various forgery traces in a data-driven manner, such as contour fitting traces, frequency domain artifacts, temporal forgery traces, identity information inconsistency, and texture defects. In addition, some methods try to combine task characteristics to design more suitable network structures, loss functions, data augmentation, and training strategies to improve detection performance.

[0003] Deep fake tracing is a more sophisticated task, which requires analyzing the generation details behind the fake image or video, including information on synthesis methods, network structure, and hyperparameters. Depending on whether the target data to be traced can be accessed in advance, deep fake tracing can be divided into active tracing and passive tracing. Active tracing generally requires pre-embedding the designed fingerprint information into the generation model during the training phase, and directly extracting the embedded fingerprint information for tracing during inference. Compared with passive tracing, active tracing is usually superior in terms of robustness and accuracy, but it requires pre-embedding information into deep fake data, which limits its application scenarios. Passive tracing does not require access to the target data to be traced, and can be further divided into model-level tracing, network structure-level tracing, and method-level tracing. The literature extracts the model fingerprint of Generative Adversarial Networks (GANs) to determine which specific model the image is synthesized by. Yang et al. empirically proved the existence of network structure fingerprints, and that they have globally consistent properties, and proposed a structural tracing network DNA-Det for network structure-level tracing. Girish et al. considered the rapid update and iteration of GANs methods and explored the GANs method tracing scenario in an open environment. Jia et al. trained a video-level forgery method tracing model on a dataset generated by five different face replacement methods.

[0004] Existing works often regard the face forgery detection task and the traceability task as independent research directions, and design independent detection models. For example, some traceability works regard it as a follow-up task after the detection task. However, these works often ignore the connection between the forgery detection task and the traceability task. The present invention notes that the detection task focuses on the common features of forgery, while the traceability task focuses on the special features of forgery. From the perspective of the process of fake face forgery, different forgery methods have both common operations and unique operations. Therefore, face forgery features can also be divided into common forgery traces and unique forgery traces. Due to the complementary connection between the feature components concerned by the two tasks, it is possible to unify the face forgery detection task and the face forgery tracing task.

[0005] Previous work has attempted to separate the common features and special features of multiple forgery methods by designing a forgery classification head and a domain classification head in the classifier to learn the texture and generalizable features of a specific method respectively. However, the present invention finds that even for a binary loss supervised encoder, if some parameters of the encoder are frozen and only the domain classification head is retrained, a certain detection effect can be achieved. Figure 1 As shown in the figure, the forged features obtained by Xception are always clustered according to the forged domain. This also suggests that simple binary loss supervision cannot obtain ideal common features. Therefore, how to separate common features from specific features deserves more consideration.

[0006] The forged features of forged face images include method-independent forged features (common features) and method-dependent forged features (specific features). Since these two forged feature components correspond to the detection task and the tracing task respectively, this implicit connection also makes a unified framework possible.

[0007] On this basis, the present invention further explores how to separate the common features and special features of forged images, and uses them as the basis for unifying the forged face detection task and the traceability task. Due to the complementary characteristics of the two, the present invention believes that integrating a unified framework can promote both tasks. However, the unified fake face detection and traceability framework has the following challenges:

[0008] (1) How to eliminate the impact of forged irrelevant features: Deep neural networks may inadvertently learn content or identity information, which prevents the network from learning purely forged features.

[0009] (2) How to encode forged features: In the feature encoding stage, both special forgeries and common forgeries are encoded, and low-level forgery features and high-level forgery features are encoded. This solves the problem that low-level forgery features such as visual artifacts and forged boundaries are difficult to learn.

[0010] (3) Joint optimization problem: The present invention notes that there is an implicit connection between common features and special features. How to use the network to separate common features from special features and finally perform detection and tracing tasks separately. Summary of the invention

[0011] An embodiment of the present invention provides a false face detection and tracing method based on feature level decoupling, comprising:

[0012] Fake face detection and tracing are performed based on the forgery feature-level decoupling framework.

[0013] Preferably, the forgery feature level decoupling framework comprises two stages: a progressive forgery feature extraction stage and a cross decoupling stage.

[0014] Preferably, the progressive forgery feature extraction stage includes:

[0015] N-stage overall forged feature encoder: The encoder is divided into N stages from shallow to deep layers, and the overall forged features are learned from low-level forged features (corresponding to shallow features) to high-level forged features (corresponding to deep features) in a way that limits the depth of the network, and then high-level forged features are gradually learned based on low-level forged features;

[0016] Triple similarity calculation: An image with the same label as the input sample and an image with a different label from the input sample will be randomly selected from the dataset to construct a triplet, and then the triplet will be input into the overall forged feature encoder to extract features and calculate the triplet loss. The triplet loss is used to reduce the distance between features of the same type and increase the distance between features of different types. It is defined as follows:

[0017]

[0018] Among them, x anc represents the anchor image, x pos With x anc The labels are the same, x neg Belong to different labels; E p represents the function of the progressive feature encoder, the symbol d represents the Euclidean distance, and α is the predefined margin;

[0019] Multi-target feature extraction module: binary classification loss calculated by cross entropy To supervise the model to learn forgery related features:

[0020]

[0021] in, represents the cross entropy loss, H cIt is the classification head of forgery related features, which is implemented by multiple MLP layers. y∈{fake,real} is a binary classification label, fake means that the image is a forged image, and real means that the image is a real image;

[0022] or,

[0023] Forgery method by guiding the model to identify forged images:

[0024]

[0025] Among them, H s is the classification head for a specific forged feature, represents the cross entropy loss, f n represents the overall forged feature map of the nth stage, y′∈{real,GAN1,GAN2,…} is an instance of the label that identifies the image;

[0026] Pixel-level forgery guidance module: Divide M into non-overlapping patches of h1×w1; where h1 represents the height of the feature map and w1 represents the width of the feature map; then obtain the corresponding label M k (k=1,2,…,h1w1), for each patch P k Take the average value of all M values get:

[0027]

[0028] Assumptions For the predicted location map, the accuracy of the prediction is measured using the cross entropy loss:

[0029]

[0030] Among them, M k Indicates the label of the forged area downsampled to the pixel level corresponding to the size of the feature map, represents the prediction of the forged feature map for the forged area, and k represents the pixel number;

[0031] Overall loss in the pre-training phase: Balance the three loss functions based on the three loss weight parameters, and involve to balance them. Note that more parameters will be involved in training. Automatic weighted loss (AWL) is used to automatically select parameters. The final loss compensation is as follows:

[0032]

[0033] in, is the pre-training loss, s1, s2, s3, and s4 are learnable parameters that are optimized during network training. Corresponding to the triplet loss, represents the true-false cross entropy loss, represents the multi-classification loss for the fake class, represents the segmentation loss of the forged region;

[0034] The cross-decoupling phase includes:

[0035] Common and special encoders: The encoder processes a pair of images (x0, x1), where x0 represents a forged image with manipulation method A and x1 represents an image with manipulation method B; the encoder E includes a pre-trained forged feature encoder E p and specificity fake encoder E s and commonality forged encoder E c , respectively extracting common forged features and special forged features based on forged features; in E s and E c A multi-scale feature fusion network based on bilinear interpolation upsampling is added to the head of ; the encoder is applied to each pair, and the corresponding forged features are obtained as follows:

[0036] {f1,…,f n} i =E p (x i )

[0037]

[0038] Where i∈{0,1} is the index of the image, {f1,…,f n} i 、f i s and f i c Corresponding to the input image x i The overall counterfeit characteristics, special counterfeit characteristics and common counterfeit characteristics of w j , α j represents the learnable weight of the jth layer when feature fusion is performed, x i represents the input image, E p represents the pre-trained overall forged feature encoder, E s represents the special feature encoder, E c represents the common feature encoder;

[0039] Feature encoding loss: Domain loss and forgery loss are used to extract special features and common features. The formula is:

[0040]

[0041] Feature reconstruction loss: self-reconstruction and cross-reconstruction; for self-reconstruction, common discriminators and special discriminators are used for identification respectively, and the formula is:

[0042]

[0043] Orthogonal Regularization Loss: Use orthogonal diversity loss To reduce the correlation of attention features, orthogonal diversity loss It is expressed as:

[0044]

[0045] Overall loss in the training phase: The training process uses automatic weighted loss for training, and the final AWL calculation result is:

[0046]

[0047] Among them, s5, s6, s7, and s8 are parameters that are optimized during the training process. represents the feature encoding loss, represents the self-reconstruction loss, represents the cross reconstruction loss, represents the orthogonal regularization loss.

[0048] Common and special encoders: The encoder processes a pair of images (x0, x1), where x0 represents a forged image with manipulation method A and x1 represents an image with manipulation method B; the encoder E includes a pre-trained forged feature encoder E p and specificity fake encoder E s and commonality forged encoder E c , respectively extracting common forged features and special forged features based on forged features; in E s and E c A multi-scale feature fusion network based on bilinear interpolation upsampling is added to the head of ; the encoder is applied to each pair, and the corresponding forged features are obtained as follows:

[0049] {f1,…,f n} i =E p (x i )

[0050]

[0051] Where i∈{0,1} is the index of the image, {f1,…,f n} i 、f i s and f i cCorresponding to the input image x i The overall counterfeit characteristics, special counterfeit characteristics and common counterfeit characteristics of w j , α j represents the learnable weight of the jth layer when feature fusion is performed, x i represents the input image, E p represents the pre-trained overall forged feature encoder, E s represents the special feature encoder, E c represents the common feature encoder;

[0052] Feature encoding loss: Domain loss and forgery loss are used to extract special features and common features. The formula is:

[0053]

[0054] Feature reconstruction loss: self-reconstruction and cross-reconstruction; for self-reconstruction, common discriminators and special discriminators are used for identification respectively, and the formula is:

[0055]

[0056] Orthogonal Regularization Loss: Use orthogonal diversity loss To reduce the correlation of attention features, orthogonal diversity loss It is expressed as:

[0057]

[0058] Overall loss during training: The training process uses automatic weighted loss for training, and the final AWL calculation result is:

[0059]

[0060] Among them, s5, s6, s7, and s8 are parameters that are optimized during the training process. represents the feature encoding loss, represents the self-reconstruction loss, represents the cross reconstruction loss, represents the orthogonal regularization loss.

[0061] Preferably, the false face detection and tracing method based on feature-level decoupling further includes:

[0062] Testing phase: Use the pre-trained encoder to extract forgery features, use the special encoder to extract special forgery features for forgery tracing tasks, and use the common encoder to extract common forgery features for forgery classification tasks.

[0063] An embodiment of the present invention provides an electronic device, the electronic device comprising:

[0064] at least one processor; and

[0065] A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any of the above-mentioned false face detection and tracing methods based on feature-level decoupling.

[0066] An embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement any of the above-mentioned false face detection and tracing methods based on feature-level decoupling when executed.

[0067] A computer program product provided by an embodiment of the present invention includes computer instructions. When executed by an electronic device, the electronic device executes the computer program code of any of the above-mentioned false face detection and tracing methods based on feature-level decoupling.

[0068] This application has achieved the following beneficial effects:

[0069] The contributions of this invention can be divided into four parts:

[0070] (1) The present invention innovatively integrates face forgery detection and tracing into one task, regards forgery features as complementary common features and specific features, and jointly optimizes the two subtasks based on feature decoupling.

[0071] (2) The present invention proposes a progressive forged feature encoder, which uses the pixel-level forged features contained in multi-scale segmentation to guide the network to learn more abstract forged features. The progressive approach can ensure that high-level forged features are built on low-level forgeries. And this learning method can effectively eliminate the problem of deep neural networks learning forged irrelevant features by limiting the depth of the neural network in multiple stages.

[0072] (3) The present invention proposes a forged feature cross-decoupling module, which realizes the “commonality-specificity” cross-reconstruction of the forged feature level by introducing carefully designed forged pairs during the training process, and introduces the orthogonal regularization loss to reduce the correlation between common features and specific features.

[0073] (4) A large number of experiments show that the framework of the present invention achieves the unification of two tasks, and the tasks promote each other. In the detection and tracing tasks, the framework of the present invention has more advanced performance.

[0074] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0075] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0077] Figure 1 A schematic diagram for visualizing the features of Xception in the detection task;

[0078] Figure 2 This is a schematic diagram of the training process framework;

[0079] Figure 3 This is the flow chart of the testing phase;

[0080] Figure 4 Schematic diagram of t-SNE visualization on FF++.

[0081] Figure 5 Schematic diagram of t-SNE visualization of FF++ generalized to Celeb-DF. DETAILED DESCRIPTION

[0082] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0083] The embodiment of the present invention provides a false face detection and tracing method based on feature level decoupling, such as Figure 1 As shown, including:

[0084] Fake face detection and tracing are performed based on the forgery feature-level decoupling framework.

[0085] The forgery feature level decoupling framework consists of two stages: the progressive forgery feature extraction stage and the cross decoupling stage.

[0086] The progressive forgery feature extraction phase includes:

[0087] N-stage overall forged feature encoder: The encoder is divided into N stages from shallow to deep layers, and the overall forged features are learned from low-level forged features (corresponding to shallow features) to high-level forged features (corresponding to deep features) in a way that limits the depth of the network, and then high-level forged features are gradually learned based on low-level forged features;

[0088] Triple similarity calculation: An image with the same label as the input sample and an image with a different label from the input sample will be randomly selected from the dataset to construct a triplet, and then the triplet will be input into the overall forged feature encoder to extract features and calculate the triplet loss. The triplet loss is used to reduce the distance between features of the same type and increase the distance between features of different types. It is defined as follows:

[0089]

[0090] Among them, x anc represents the anchor image, x pos With x anc The labels are the same, x neg Belong to different labels; E p represents the function of the progressive feature encoder, the symbol d represents the Euclidean distance, and α is the predefined margin;

[0091] Multi-target feature extraction module: binary classification loss calculated by cross entropy To supervise the model to learn forgery related features:

[0092]

[0093] in, represents the cross entropy loss, H c It is the classification head of forgery related features, which is implemented by multiple MLP layers. y∈{fake,real} is a binary classification label, fake means that the image is a forged image, and real means that the image is a real image;

[0094] or,

[0095] Forgery method by guiding the model to identify forged images:

[0096]

[0097] Among them, H s is the classification head for a specific forged feature, represents the cross entropy loss, f n represents the overall forged feature map of the nth stage, y′∈{real,GAN1,GAN2,…} is an instance of the label that identifies the image;

[0098] Pixel-level forgery guidance module: Divide M into non-overlapping patches of h1×w1; where h1 represents the height of the feature map and w1 represents the width of the feature map; then obtain the corresponding label M k (k=1,2,…,h1w1), for each patch P k Take the average value of all M values get:

[0099]

[0100] Assumptions For the predicted location map, the accuracy of the prediction is measured using the cross entropy loss:

[0101]

[0102] Among them, M k Indicates the label of the forged area downsampled to the pixel level corresponding to the size of the feature map, represents the prediction of the forged feature map for the forged area, and k represents the pixel number;

[0103] Overall loss in the pre-training phase: Balance the three loss functions based on the three loss weight parameters, and involve to balance them. Note that more parameters will be involved in training. Automatic weighted loss (AWL) is used to automatically select parameters. The final loss compensation is as follows:

[0104]

[0105] in, To compensate for the loss, s1, s2, s3, and s4 are preset parameters. Corresponding to the triplet loss, represents the true-false cross entropy loss, represents the multi-classification loss for the fake class, represents the segmentation loss of the forged region;

[0106] Optimization is performed during the network training process. The false face detection and tracing method based on feature-level decoupling also includes:

[0107] Testing phase: Use the pre-trained encoder to extract forgery features, use the special encoder to extract special forgery features for forgery tracing tasks, and use the common encoder to extract common forgery features for forgery classification tasks.

[0108] like Figure 2As shown, the present invention proposes a forgery feature-level decoupling framework, which includes two stages: a progressive forgery feature extraction stage (pre-training stage) and a cross-decoupling stage (training stage). Based on the carefully extracted forgery features, the present invention further divides the forgery features into common forgery features and specific forgery features. Common forgery features can effectively improve the generalization ability of deep forgery detectors in detection tasks, avoiding overfitting of forgery-irrelevant features and method-specific features. Specific forgery features can further enhance the traceability of forgery methods. In addition, the present invention also introduces an orthogonalization regularization loss to further help disentangle and enhance the generalization ability of decoupled features.

[0109] The idea of ​​the present invention is that there are common features and special features in the forgery representation of the image. The main purpose is to disentangle them and use the disentangled artifact features for subsequent tasks. The disentanglement framework is mainly composed of a pre-trained progressive encoder, which simultaneously extracts multi-level generalizable features of textures and different forgeries for learning specific methods. The special encoder and the common encoder perform feature secondary encoding based on the pre-trained features, and use special discriminators and common discriminators to judge the special and common representations in the reconstructed features, respectively. In the process of identifying multiple sets of reconstructed features, the parameters of the special discriminator and the common discriminator are shared.

[0110] Direct feature-level decoupling of forged features is difficult to optimize and will cause instability in network training. The idea of ​​the present invention is to first extract the forged features as a whole and perform cross-decoupling on their basis. And the features extracted by the progressive forged feature extraction method themselves have excellent generalization. It can effectively reduce the interference of forged irrelevant features. Here, the present invention represents the images in the training set as (x, y, y′, M), where x represents the image, y is the true or false label, M is the corresponding forged mask, and if x is forged, y' is the forged category.

[0111] First, generalized forged features (including domain-related forged features and domain-independent forged features) are extracted by a progressive forged feature extraction method. In this stage, multi-scale segmentation loss is used to emphasize the guidance of pixel-level forged features for abstract forged features. In the training process of each stage of progressive training, the network is designed to include an N-stage forged feature encoder, a ternary similarity calculation module, a multi-target feature extraction module, and a pixel-level forged guidance module. The present invention uses automatic weighted loss (AWL) to balance these losses.

[0112] N-stage forged feature encoder: Previous work has shown that even a binary supervised forged encoder will inadvertently learn the ID representation and content features of the face. If a deep neural network is directly used as a forged feature encoder, the learned forged features will not be "pure" enough. The present invention solves the problem of forged feature extraction from a completely new perspective. First of all, the present invention takes into account that the forged features of the face have both low-level forged features (local forged features such as artifacts) and high-level forged features (globally consistent forged features). Drawing on the idea of ​​ProGAN, the present invention hopes to progressively learn the "low-level-high-level" features by controlling the encoder depth in multiple stages. Specifically, the present invention divides the backbone network into N-stage forged feature encoders. The encoder of the nth stage contains the previous n-1 stages. In the primary stage, the expression ability of low-level forged features is improved by limiting the network depth, and then high-level forged features are gradually learned on the basis of low-level forged features. Such a progressive design can well alleviate the problems in the previous forged feature learning, and this more refined forged feature is also beneficial to the subsequent feature decoupling stage.

[0113] Ternary similarity calculation: Since inherent forgery features are difficult to define and extract, this paper designs ternary similarity calculation to calculate three sets of losses, focusing on improving the classification ability of features. Figure 3 As shown, for each input image (real or fake), the present invention will randomly select a homogeneous image and a heterogeneous image from the dataset to construct a triple. The triple is then input into the network to extract features and calculate the triple loss. The triple loss is used to reduce the distance between features of the same type and increase the distance between features of different types, and is defined as follows:

[0114]

[0115] Among them, x anc represents the anchor image, x pos With x anc Belong to the same category, x neg Belongs to the opposite category. E p represents the function of the progressive feature encoder, the symbol d represents the Euclidean distance, and α is a predefined margin.

[0116] Multi-objective feature extraction module: For classification loss, the present invention proposes two different optimization objectives. First, the present invention proposes a binary classification loss calculated by cross entropy To supervise the model to learn forgery related features:

[0117]

[0118] in represents the cross entropy loss, H cis the head of the fake related features, which is implemented by multiple MLP layers. y∈{fake,real} is the binary classification label. In addition, Learn method-specific patterns by guiding the model to identify which forgery method was used in the forgery image:

[0119]

[0120] Among them, H s For the head of a particular forged feature, y′∈{real,GAN1,GAN2,…} gives the label identifying which instance belongs to the image. Note that H c and H s They share the same architecture but not parameters. The multi-task classification loss enables the model to learn both method-specific textures and common features of different fakes, enhancing the generalization ability of the model.

[0121] Pixel-level forgery guidance module: The present invention divides M into h1×w1 non-overlapping patches and then obtains the corresponding label M k (k=1,2,…,h1w1), for each patch P k Take the average value of all M values get:

[0122]

[0123] Assumptions For the predicted location map, the accuracy of the prediction is measured using the cross entropy loss:

[0124]

[0125] Overall loss in the pre-training phase: Three loss functions are considered during the pre-training process, and three loss weight parameters are involved to balance them. Note that more parameters will be involved during training. A grid search strategy can be used to search for good parameters. However, due to the large number of parameters and the uncertain setting range, the grid search strategy is neither realistic nor accurate. To solve this problem, the present invention uses automatic weighted loss (AWL) to automatically select parameters. The final loss compensation is as follows:

[0126]

[0127] Among them, s1, s2, s3, and s4 are parameters that are optimized during the network training process.

[0128] Common and special encoders: The encoder of the present invention processes a pair of images (x0, x1), where x0 represents a forged image with manipulation method A and x1 represents an image with manipulation method B. The encoder E includes a pre-trained forged feature encoder Ep and specificity fake encoder E s and commonality forged encoder E c , respectively extracting common forged features and special forged features based on forged features. Inspired by the image pyramid, in E s and E c The head of is added with a multi-scale feature fusion network based on bilinear interpolation for upsampling. The present invention applies the encoder to each pair and obtains the corresponding forged features as follows:

[0129] {f1,…,f n} i =E p (x i )

[0130]

[0131] Where i∈{0,1} is the index of the image. n} i 、f i s and f i c Corresponding to the input image x i The overall counterfeit characteristics, special counterfeit characteristics and common counterfeit characteristics. j , α j Represents the learnable weight of the jth layer during feature fusion.

[0132] Feature coding loss: This invention uses domain loss and forgery loss to extract special features and common features. Please note that coding loss cannot completely decouple special features and common features. The formula is:

[0133]

[0134] Feature reconstruction loss: There are two types of reconstruction in the framework of the present invention: self-reconstruction and cross-reconstruction. For self-reconstruction, the present invention uses common discriminators and special discriminators for identification respectively, and the formula is:

[0135]

[0136] Orthogonal regularization loss: The present invention believes that common features and special features should have as small an intersection as possible, and hopes that common features and special features can focus on as many forged features as possible. Inspired by the orthogonal regularizer, the present invention uses orthogonal diversity loss To reduce the correlation of attention features. Since the correlation of attention is independent of time, the orthogonal diversity loss Expressed as:

[0137]

[0138] Overall loss during training: Similar to the pre-training process, the training process uses automatic weighted loss for training. The final AWL calculation result is:

[0139]

[0140] Among them, s5, s6, s7, and s8 are parameters that are optimized during the training process.

[0141] During the test, the present invention uses a pre-trained encoder to extract forgery features, uses a special encoder to extract special forgery features for forgery tracing tasks, and uses a common encoder to extract common forgery features for forgery classification tasks, such as Figure 3 shown.

[0142] The present invention reports the area under the curve (AUC) metric to compare the proposed method with previous work, which is consistent with the evaluation method adopted by many previous works. The default evaluation metrics used are ACC and AUC.

[0143] FaceForensics++ (FF++): FF++ is a standard benchmark for face forgery detection. It includes 1000 real videos and 4000 fake videos, including four face processing techniques: Deepfakes (DF), Face2Face (F2F), FaceSwap (FS), and NeuralTextures (NT). In addition, each video in FF++ has three quality levels: original, high-quality (C23), and low-quality (C40) data. According to previous work, the HQ version of FF++ is adopted by default.

[0144] Celeb-DF-v2 (CDF): CDF is a large-scale public dataset containing 590 real videos and 5639 fake videos. The fake videos in CDF have better visual quality than previous datasets, which makes detection more challenging.

[0145] Deepfake Detection Challenge (DFDC): DFDC is a challenging face swapping dataset for evaluating the performance of detectors in different scenarios. It includes 1133 real videos and 4080 fake videos generated by Deepfake, GAN and traditional face swapping methods.

[0146] DeepFakeDetection(DFD):DFD is a Deepfake dataset produced by Google. It contains more than 3,000 manipulated videos from 28 actors in various scenes with a high level of visual realism.

[0147] ForgeryNet is a large-scale fake face dataset, containing 99,630 real face videos from 4 real data sources and 121,617 fake face videos based on 15 forgery methods. Since the test set does not provide category labels, and 36 types of data augmentation are used for post-processing on the validation set, in order to reduce the impact of data augmentation, the original training set is divided in a ratio of 8:1:1 in this experiment to create new training sets, validation sets, and test sets.

[0148]

[0149] Table 1. In-library test results in detection tasks

[0150]

[0151] Table 2 Test results of traceability tasks

[0152] As shown in Table 1, in the in-library detection task, the results of all indicators of this method on the FaceForensic++ and ForgeryNet datasets are better than the baseline model Xception. These results fully demonstrate the superiority of the unified framework proposed in this paper in the task of true and false binary classification.

[0153] As shown in Table 2, the proposed method achieves better results than the baseline (Xception) on both the FaceForensics++ and ForgeryNet datasets. Considering that the unified framework proposed in the present invention improves the deep face forgery detection performance while maintaining the traceability performance to the most advanced level, it has high practical application value.

[0154]

[0155] Table 3 Cross-database test results in detection tasks

[0156] The present invention uses two different data configurations: "FF-ALL", which includes all four forged data, and "FF-wo-DF", "FF-wo-F2F", "FFwo-FS" and "FF-wo-NT", which use the FF++ dataset but delete DF, F2F, FS and NT respectively. The experimental results show that our framework achieves good results in cross-database testing of detection tasks compared with the baseline.

[0157] In the visualization of FF++, the overall forgery has the situation that the forgery domain is partially aggregated, while the special forgery can completely separate the forgery domain, and the common forgery features can discover the common features between the forgery methods, such as Figure 4 The visualization generalized to Celeb-DF is shown in Figure 5 shown.

[0158] An embodiment of the present invention provides an electronic device, the electronic device comprising:

[0159] at least one processor; and

[0160] A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute any of the above-mentioned false face detection and tracing methods based on feature-level decoupling.

[0161] An embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement any of the above-mentioned false face detection and tracing methods based on feature-level decoupling when executed.

[0162] A computer program product provided by an embodiment of the present invention includes computer instructions. When executed by an electronic device, the electronic device executes the computer program code of any of the above-mentioned false face detection and tracing methods based on feature-level decoupling.

[0163] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A false face detection and tracing method based on feature-level decoupling, characterized in that: include: Based on the forgery feature-level decoupling framework, fake face detection and tracing are performed; The forgery feature level decoupling framework consists of two stages: the progressive forgery feature extraction stage and the cross decoupling stage; The progressive forgery feature extraction phase includes: Stage overall forged feature encoder: The encoder is divided into In the first stage, the overall forged features are learned from low-level forged features (corresponding to shallow features) to high-level forged features (corresponding to deep features) in a way that limits the depth of the network, and then high-level forged features are gradually learned based on low-level forged features; Triple similarity calculation: An image with the same label as the input sample and an image with a different label from the input sample will be randomly selected from the dataset to construct a triplet, and then the triplet will be input into the overall forged feature encoder to extract features and calculate the triplet loss. The triplet loss is used to reduce the distance between features of the same type and increase the distance between features of different types. It is defined as follows: in, Represents an anchor image, and The labels are the same, Belong to different labels; Denotes the function of the progressive feature encoder, symbol represents the Euclidean distance, are predefined margins; Multi-target feature extraction module: binary classification loss calculated by cross entropy To supervise the model to learn forgery related features: in, represents the binary classification loss, represents the cross entropy loss, is a classification head that forges correlation features, which are implemented by multiple MLP layers. It is a binary classification label, fake means the image is a fake image, and real means the image is a real image; or, Forgery method by guiding the model to identify forged images: in, is the classification head for a specific forged feature, represents the cross entropy loss, represents the overall forged feature map of the nth stage, To give an instance of identifying a label belonging to an image; Pixel-level forgery guidance module: Divide into Non-overlapping patches; where Represents the height of the feature map, Represents the width of the feature map; then get the corresponding label , for each patch All The average value of ,get: For the predicted location map, the accuracy of the prediction is measured using the cross entropy loss: ； in, Indicates the label of the forged area downsampled to the pixel level corresponding to the size of the feature map, represents the prediction of the forged feature map for the forged area, Indicates the number of the pixel; Overall loss in the pre-training phase: Balance the three loss functions based on the three loss weight parameters, and involve to balance them. Please note that more parameters will be involved in training. Automatic weighted loss AWL is used to automatically select parameters. The final loss compensation is as follows: in, is the pre-training loss, , , , are learnable parameters that are optimized during network training. Corresponding to the triplet loss, represents the true-false cross entropy loss, represents the multi-classification loss for the fake class, represents the segmentation loss of the forged region; The cross-decoupling phase includes: Common and special encoders: The encoder processes a pair of images ,in represents a forged image with manipulation method A, Represents an image with manipulation method B; encoder Includes pre-trained fake feature encoder and specificity forged encoder and commonality forged encoder , respectively extracting common forged features and special forged features based on forged features; and A multi-scale feature fusion network based on bilinear interpolation upsampling is added to the head of ; the encoder is applied to each pair, and the corresponding forged features are obtained as follows: in, is the index of the image, , and Corresponding to the input image The overall counterfeit characteristics, special counterfeit characteristics and common counterfeit characteristics of , Indicates the feature fusion The learnable weights of the layers, represents the input image, represents the pre-trained overall forged feature encoder, represents a specific fake encoder, Indicates commonality forged encoder; Feature encoding loss: Domain loss and forgery loss are used to extract special features and common features. The formula is: Feature reconstruction loss: self-reconstruction and cross-reconstruction; for self-reconstruction, common discriminators and special discriminators are used for identification respectively, and the formula is: Orthogonal Regularization Loss: Use orthogonal diversity loss To reduce the correlation of attention features, orthogonal diversity loss It is expressed as: Overall loss in the training phase: The training process uses automatic weighted loss for training, and the final AWL calculation result is: in , , , is a parameter that is optimized during training. represents the feature encoding loss, represents the self-reconstruction loss, represents the cross reconstruction loss, represents the orthogonal regularization loss.

2. The method for detecting and tracing false faces based on feature-level decoupling according to claim 1, characterized in that: Also includes: Testing phase: Use the pre-trained encoder to extract forgery features, use the special encoder to extract special forgery features for forgery tracing tasks, and use the common encoder to extract common forgery features for forgery classification tasks.

3. An electronic device, characterized in that: The electronic device comprises: at least one processor; and A memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the false face detection and tracing method based on feature-level decoupling as described in any one of claims 1-2.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the false face detection and tracing method based on feature-level decoupling according to any one of claims 1 to 2 when executed.

5. A computer program product, characterized in that The computer program product includes computer instructions, and when executed by an electronic device, the electronic device executes the computer program code of the false face detection and tracing method based on feature-level decoupling as described in any one of claims 1 to 2.

Images