A method and system for identifying fake faces

Abstract

The present invention provides a method and system for identifying forged faces. The method comprises the following steps: obtaining a face image to be identified and a dataset of paired real and forged face images, and clustering and dividing the dataset; using the divided dataset to input multiple models for joint training of feature extractors; fixing the trained feature extractor and using a variational autoencoder to map the extracted features to a shared latent feature space, thereby achieving deep collaborative fusion of features from different models; fixing the trained feature extractor and variational autoencoder, obtaining features of the input image and feeding them into a classifier for training; and using the trained forged face identification network to perform forged face identification on the face image to be detected. The method effectively improves the generalization of the model by clustering the dataset, and fuses features from different models by using a variational autoencoder, making the feature expression more compact and robust, thereby promoting complementarity between different models.

Description

Technical Field

[0001] The present invention belongs to the technical field of forged face identification, and in particular relates to a forged face identification method and system. Background Art

[0002] With the rapid development of deep learning, AI-based image generation techniques have gained widespread application. In particular, driven by generative models such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Diffusion Models, the synthesis of highly realistic facial images has become a reality. This technology has played a significant role in the creation of virtual characters, film and television special effects, and augmented reality (AR). However, it also poses serious security risks, such as identity forgery, the spread of false information, and malicious fraud. Therefore, efficient, accurate, and robust methods for detecting forged facial images have become a key technology in the current field of AI security.

[0003] Currently, forged face detection primarily falls into two categories: spatial-domain analysis and frequency-domain analysis. Spatial-domain methods rely on deep neural networks to extract local features, such as texture, edge information, and unnatural facial details, and then use classification models for identification. Representative methods include the CNN-based XceptionNet, ResNet, and EfficientNet, as well as the Transformer-based ViT and Swin Transformer. While these methods perform well on standard datasets, their generalization capabilities are limited when dealing with new forgery methods, cross-dataset transfer, and low-resolution images. Frequency-domain methods exploit anomalies in the frequency domain distribution of GAN-generated images for detection. For example, Fourier transform analysis can reveal high-frequency anomalies in forged images, while wavelet transforms can extract features of forged images at different frequencies through multi-scale analysis. While these methods partially address the shortcomings of spatial-domain methods, their detection performance degrades when faced with image compression, degradation, or noise. Existing technologies generally suffer from insufficient generalization, susceptibility to adversarial attacks, sensitivity to image compression, and a lack of multi-dimensional feature fusion. Summary of the Invention

[0004] Based on this, an embodiment of the present invention provides a method and system for identifying fake faces, which aims to improve the accuracy, robustness and generalization ability of detection by constructing a multi-feature fusion framework.

[0005] A first aspect of an embodiment of the present invention provides a method for identifying a forged face, the method comprising:

[0006] Obtain a dataset of facial images to be identified, as well as paired real and forged facial images. Use a lightweight CNN to extract features, perform K-nearest neighbor clustering in the feature space, and divide the data into multiple target datasets.

[0007] Inputting the target data set into a deep learning model for joint training of feature extractors, wherein the deep learning model includes at least Inception, ResNet50, ViT, and EfficientNet;

[0008] Determine the trained feature extractor and use a variational autoencoder to map the extracted features to a shared latent feature space. Fuse the features extracted by different models and constrain the distribution through KL divergence. Then, use the decoder to reconstruct the original features from the latent space to ensure that key information is not lost during the encoding process.

[0009] Use the trained feature extractor and variational autoencoder to obtain features, and input the features into the classifier for fine-tuning to complete the training of the fake face identification network;

[0010] The face image to be identified is input into the trained forged face identification network, and an identification result is output.

[0011] Furthermore, in the step of inputting the target dataset into the deep learning model for joint training of the feature extractor, the target dataset is randomly input into the deep learning model; after feature extraction, comparative learning is performed, and then each deep learning model inputs the features into the classification head to output the results, calculates the loss according to the loss function, and performs iterative training, so that each deep learning model can serve as a unique feature extractor, and uses the corresponding test set for performance evaluation.

[0012] Furthermore, in the step of calculating the loss according to the loss function and performing iterative training, the calculation formula of the total loss function is:

[0013] ;

[0014] Among them, L is the total loss function, L NCE is the contrastive learning loss function, L CE is the cross entropy loss function;

[0015] The calculation formula of contrastive learning loss function is:

[0016] ;

[0017] in, Represents the features extracted from the current positive sample, Represents the features obtained by different deep learning models to extract positive samples, Represents the features extracted by different deep learning models, T represents the temperature coefficient, which is used to control the steepness of the distribution. express and similarity;

[0018] The calculation formula of the cross entropy loss function is:

[0019] ;

[0020] Among them, N represents the number of samples, M represents the number of models, Indicates the j The weight of the deep learning model, Indicates the i The correct labels of samples, Indicates the i Sample No. j The prediction results made by a deep learning model.

[0021] Furthermore, in the step of using the corresponding test set to perform performance evaluation, three indicators, Precision, Recall and F1-score, are used for evaluation. Specifically,

[0022] ;

[0023] Among them, TP represents the number of correctly predicted samples, FP represents the number of incorrectly predicted samples, Precision represents the precision rate, Recall represents the recall rate, and F1-score represents the harmonic mean of Precision and Recall.

[0024] Furthermore, the trained feature extractor is determined, and the extracted features are mapped to a shared latent feature space using a variational autoencoder, the features extracted by different models are fused, and the distribution is constrained by KL divergence, and then the original features are reconstructed from the latent space by the decoder. In the step, the trained feature extractor is fixed, the features of the data set are extracted, and the features are obtained after simple splicing. , is the number of input channels, H and W are the height and width of the input respectively, the feature is input to the variational autoencoder, and is mapped to the shared potential space through convolution to obtain the feature and features , specifically,

[0025] The features Input into the first convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is:

[0026] ;

[0027] in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1. represents the batch normalization operation, represents a nonlinear activation function;

[0028] Will Input into the second convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is:

[0029] ;

[0030] in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1;

[0031] Will Input into two convolutional layers respectively to obtain features and features , the calculation formula is:

[0032] ;

[0033] ;

[0034] in, and Both represent 3×3 convolution operations with a stride of 1 and a padding of 1;

[0035] Then the KL divergence loss is used to constrain the features and features The characteristic distribution of

[0036] Then, the reparameterization technique is used to obtain the potential representation z, that is, the fused feature z, and the decoder is used to reconstruct the features of z. The calculation formula of the potential representation z is:

[0037] ;

[0038] in, As random variables, the samples are sampled through the reparameterization trick to ensure that the gradient of the latent representation z can be transferred so that the feature and features It can still be optimized through backpropagation.

[0039] Furthermore, the feature constraint is achieved by KL divergence loss. and features In the step of feature distribution, KL divergence is used for regularization, and the calculation formula is:

[0040] ;

[0041] Among them, D KL represents the KL divergence loss, d represents the dimension of the latent space, σ i and μ i They represent the characteristics of the i-th sample respectively.

[0042] Furthermore, in the step of reconstructing the features of z using the decoder, the potential representation z is input into the first transposed convolution block, and R1 is obtained through the nonlinear activation function and batch normalization. The calculation formula is:

[0043] ;

[0044] in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1;

[0045] Then R1 is input into the second transposed convolution block, and R2 is obtained through nonlinear activation function and batch normalization. The calculation formula is:

[0046] ;

[0047] in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1;

[0048] Finally, R2 is input into the last transposed convolution layer to obtain the reconstructed features. , the calculation formula is:

[0049] ;

[0050] in, Represents a 4×4 transposed convolution operation with a stride of 2 and padding of 1.

[0051] Furthermore, the goal of the decoder is to reconstruct features generated from the latent representation z , as close as possible to the original input features, and the mean square error is used to measure the quality of the decoder reconstructed data. The calculation formula is:

[0052] ;

[0053] in, represents the reconstruction loss, N represents the total number of features, X Represents the original feature, Represents the reconstruction feature.

[0054] Furthermore, in the step of inputting the features into the classifier for fine-tuning, the features fused by the variational autoencoder, that is, the potential representation z, are gradually input into the convolution block to obtain z1, which is calculated as follows:

[0055] ;

[0056] in, Represents a 3×3 transposed convolution operation with a stride of 1 and a padding of 1;

[0057] Then z1 is globally averaged pooled and input into the fully connected layer for dimensionality reduction to obtain z1. The calculation formula is:

[0058] ;

[0059] in, represents the global average pooling operation, Represents the dimensionality reduction operation of the fully connected layer;

[0060] Finally, after normalization, the predicted probability is obtained after the Sigmoid activation function , the calculation formula is:

[0061] ;

[0062] Among them, Sigmoid means that the activation function maps the output to [0,1];

[0063] The loss function used by the classifier is cross entropy loss, and the calculation formula is:

[0064] ;

[0065] in, represents the cross entropy loss, N represents the number of all samples, represents the true label, Represents the classification result of the classifier.

[0066] A second aspect of the embodiments of the present invention provides a forged face identification system for implementing the forged face identification method of the first aspect, the system comprising:

[0067] The acquisition module is used to obtain the face image to be identified, as well as a dataset of paired real and forged face images, and use a lightweight CNN to extract features. It performs K-nearest neighbor clustering in the feature space and divides the data into multiple target datasets.

[0068] A training module, configured to input the target dataset into a deep learning model for joint training of a feature extractor, wherein the deep learning model includes at least Inception, ResNet50, ViT, and EfficientNet;

[0069] The fusion module is used to determine the trained feature extractor and use the variational autoencoder to map the extracted features to a shared latent feature space. The features extracted by different models are fused and distributed through the KL divergence constraint. The original features are then reconstructed from the latent space through the decoder to ensure that key information is not lost during the encoding process.

[0070] The fine-tuning module is used to obtain features using the trained feature extractor and variational autoencoder, and input the features into the classifier for fine-tuning to complete the training of the fake face identification network;

[0071] The input module is used to input the face image to be identified into the trained forged face identification network and output the identification result.

[0072] The present invention provides a method and system for identifying fake faces. The method obtains a face image to be identified and a dataset of paired real and fake face images and clusters the datasets. The clustered datasets are used to input multiple models for joint feature extraction. The trained feature extractor is fixed, and a variational autoencoder is used to map the extracted features to a shared latent feature space, thereby achieving deep collaborative fusion of features from different models. The trained feature extractor and variational autoencoder are fixed, and the features of the input image are obtained and fed into a classifier for training. The trained fake face identification network is used to identify fake faces from the face image to be detected. Clustering the dataset effectively improves the generalization of the model, and the variational autoencoder is used to fuse features from different models, making the feature expression more compact and robust, thereby promoting complementarity between different models. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A flowchart of a method for identifying a forged face provided in the first embodiment of the present invention;

[0074] Figure 2 Schematic diagram of the process of data set partitioning and multi-model feature extraction;

[0075] Figure 3 Schematic diagram of the classifier training process based on multi-model feature extraction and variational autoencoder feature fusion;

[0076] Figure 4 Schematic diagram of the structure of feature fusion based on variational autoencoder;

[0077] Figure 5 This is a structural block diagram of a fake face identification system provided in Example 2 of the present invention;

[0078] Figure 6This is a structural block diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0079] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0080] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0082] Example 1

[0083] According to an embodiment of the present invention, an embodiment of a method for identifying a forged face is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0084] In this embodiment, a method for identifying a fake face is provided, which can be used in electronic devices, such as computers. Figure 1 , Figure 1 A flowchart of a method for identifying a forged face provided in the first embodiment of the present invention is shown, which specifically includes steps S01 to S05.

[0085] Step S01: Obtain a facial image to be identified, as well as a dataset of paired real and forged facial images, extract features using a lightweight CNN, perform K-nearest neighbor clustering in the feature space, and divide the data into multiple target datasets.

[0086] Specifically, using the publicly available datasets DeepFake and FaceForensics++, the lightweight MobileNet was used to perform basic feature extraction on the datasets. The extracted facial features were then clustered using the K-nearest neighbor method, resulting in N categories. One category of data was selected as the validation set, while the remaining N-1 categories served as the training set to ensure the model's generalization capabilities across diverse data distributions. Subsequently, following the same strategy, one category of data was used as the validation set and the remaining data as the training set for each partition, ultimately forming N independent training-validation dataset combinations. Both the face images to be identified and the dataset were uniformly cropped to 224×224 to ensure input consistency.

[0087] Step S02: Input the target data set into a deep learning model for joint training of feature extractors, wherein the deep learning model includes at least Inception, ResNet50, ViT, and EfficientNet.

[0088] See also Figure 2 and Figure 3 , Figure 2 Schematic diagram of the process of data set partitioning and multi-model feature extraction, Figure 3 This is a diagram of the classifier training process based on multi-model feature extraction and variational autoencoder feature fusion. In this embodiment of the present invention, the divided data set is randomly input into deep learning models such as Inception, ResNet50, ViT, and EfficientNet. After feature extraction, comparative learning is performed. In order to prevent different models from learning similar features and encourage them to extract more diverse and complementary information, this embodiment uses InfoNCE (Information

[0089] Noise-Contrastive Estimation) loss function optimization model, its formula is:

[0090] ;

[0091] in, Represents the features extracted from the current positive sample, Represents the features obtained by different deep learning models to extract positive samples, Represents the features extracted by different deep learning models, T represents the temperature coefficient, which is used to control the steepness of the distribution. express and This formula emphasizes that the model should learn important features relevant to the target task, so that samples of the same category (such as real / fake) remain relatively consistent in the latent space. At the same time, by imposing regularization constraints, we prevent features extracted by different models from being too similar, thereby enhancing model specificity and ensuring that each model has its own characteristic expression.

[0092] Each model feature is then input into the classification head to output the result, and the cross entropy loss function is used, whose formula is:

[0093] ;

[0094] Among them, N represents the number of samples, M represents the number of models, Indicates the j The weight of the deep learning model, Indicates the i The correct labels of samples, Indicates the i Sample No. j The prediction results made by a deep learning model.

[0095] The calculation formula of the total loss function is:

[0096] ;

[0097] Among them, L is the total loss function, L NCE is the contrastive learning loss function, L CE The cross entropy loss function is used to optimize the loss and reduce the feature redundancy of different models, so that each model can learn information from different perspectives.

[0098] In addition, the corresponding test set is used to evaluate the performance of the feature classification results, among which the three indicators of Precision, Recall and F1-score are used for evaluation. Specifically,

[0099] ;

[0100] TP represents the number of correctly predicted samples, and FP represents the number of incorrectly predicted samples. Precision represents the accuracy rate, reflecting the proportion of samples predicted as positive (generated by the AI) that are actually positive. Recall represents the recall rate, reflecting the proportion of samples correctly predicted as positive among all truly positive samples. F1-score represents the harmonic mean of Precision and Recall.

[0101] In step S03, a trained feature extractor is determined, and the extracted features are mapped to a shared latent feature space using a variational autoencoder. The features extracted by different models are fused and distributed through KL divergence constraints. The original features are then reconstructed from the latent space through a decoder to ensure that key information is not lost during the encoding process.

[0102] See also Figure 4 , is a structural diagram of feature fusion based on variational autoencoder. After step S02, the feature extractor has been trained. Then we fix the feature extractor network and simply splice the extracted features to obtain feature , is the number of input channels, H and W are the height and width of the input respectively, the feature is input to the variational autoencoder, and is mapped to the shared potential space through convolution to obtain the feature and features Specifically, the characteristics Input into the first convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is:

[0103] ;

[0104] in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1. represents the batch normalization operation, represents a nonlinear activation function;

[0105] Will Input into the second convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is:

[0106] ;

[0107] in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1;

[0108] Will Input into two convolutional layers respectively to obtain features and features , the calculation formula is:

[0109] ;

[0110] ;

[0111] in, and Both represent 3×3 convolution operations with a stride of 1 and a padding of 1;

[0112] Then the KL divergence loss is used to constrain the features and features The characteristic distribution of

[0113] Then, the reparameterization technique is used to obtain the potential representation z, that is, the fused feature z, and the decoder is used to reconstruct the features of z. The calculation formula of the potential representation z is:

[0114] ;

[0115] in, As random variables, the samples are sampled through the reparameterization trick to ensure that the gradient of the latent representation z can be transferred so that the feature and features It can still be optimized through backpropagation.

[0116] Constraining features through KL divergence loss and features In the step of feature distribution, KL divergence is used for regularization, and the calculation formula is:

[0117] ;

[0118] Among them, D KL represents the KL divergence loss, d represents the dimension of the latent space, σ i and μ i Represent the characteristics of the i-th sample. It can be understood that the characteristics and features They are obtained by variational autoencoders respectively, and the feature distribution in the latent space is constrained by KL divergence to make it close to the standard normal distribution, thereby improving the smoothness of the latent space and thus improving the robustness of the encoder to noise and disturbances.

[0119] In the step of reconstructing the features of z using the decoder, the potential representation z is input into the first transposed convolution block, and R1 is obtained through the nonlinear activation function and batch normalization. The calculation formula is:

[0120] ;

[0121] in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1;

[0122] Then R1 is input into the second transposed convolution block, and R2 is obtained through nonlinear activation function and batch normalization. The calculation formula is:

[0123] ;

[0124] in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1;

[0125] Finally, R2 is input into the last transposed convolution layer to obtain the reconstructed features. , the calculation formula is:

[0126] ;

[0127] in, Represents a 4×4 transposed convolution operation with a stride of 2 and padding of 1.

[0128] At the same time, in order to ensure that key information is not lost during the encoding process, the decoder is used to generate the reconstructed features through the potential representation z , as close as possible to the original input features, and the mean square error is used to measure the quality of the decoder reconstructed data. The calculation formula is:

[0129] ;

[0130] in, represents the reconstruction loss, N represents the total number of features, X Represents the original feature, Represents the reconstructed features, and backpropagation is performed by calculating the mean square error while iterating the encoder and decoder to ensure that key information is not lost during the encoding process.

[0131] Step S04: Use the trained feature extractor and variational autoencoder to obtain features, and input the features into the classifier for fine-tuning to complete the training of the fake face identification network.

[0132] Fix the parameters of the feature extraction network and the variational autoencoder, and input the extracted and fused features into the final classifier for final training. It should be noted that the features fused by the variational autoencoder, that is, the potential representation z, are gradually input into the convolution block to obtain z1. The calculation formula is:

[0133] ;

[0134] in, Represents a 3×3 transposed convolution operation with a stride of 1 and a padding of 1;

[0135] In order to reduce the amount of calculation while retaining key information, z1 is globally averaged pooled and input into the fully connected layer for dimensionality reduction to obtain z1. The calculation formula is:

[0136] ;

[0137] in, represents the global average pooling operation, Represents the dimensionality reduction operation of the fully connected layer;

[0138] Finally, after normalization, the predicted probability is obtained after the Sigmoid activation function , the calculation formula is:

[0139] ;

[0140] Among them, Sigmoid means that the activation function maps the output to [0,1];

[0141] The loss function used by the classifier is cross entropy loss, and the calculation formula is:

[0142] ;

[0143] in, represents the cross entropy loss, N represents the number of all samples, represents the true label, Represents the classification result of the classifier.

[0144] In order to evaluate the performance of the model, all data sets are randomly input into the trained model to obtain the corresponding predicted labels. AUC-ROC is used as the evaluation indicator. The AUC calculation formula is as follows:

[0145] ;

[0146] Where TP represents the number of correctly predicted samples, FP represents the number of incorrectly predicted samples, and the AUC value represents the area under the ROC curve, ranging from 0 to 1. The closer the AUC value is to 1, the better the classifier performance.

[0147] Step S05: input the face image to be identified into the trained forged face identification network and output the identification result.

[0148] Specifically, the training of the fake face identification network is completed, and then the face image to be identified is input into the trained multi-model feature fusion network, and the probability of the image to be identified (between 0 and 1) is output. The category of the image to be identified is a real image (0) or a fake face image (1). The closer the probability is to 1, the more likely it is a fake face image.

[0149] In summary, the forged face identification method in the above embodiment of the present invention obtains a face image to be identified and a paired dataset of real and forged face images and clusters the dataset; uses the partitioned dataset to input multiple models for joint training of feature extractors; fixes the trained feature extractor and uses a variational autoencoder to map the extracted features to a shared latent feature space to achieve deep collaborative fusion of features from different models; fixes the trained feature extractor and variational autoencoder, obtains the features of the input image and feeds them into a classifier for training; uses the trained forged face identification network to perform forged identification on the face image to be detected. Among them, the generalization of the model is effectively improved by clustering the dataset, and the feature expression is made more compact and robust by fusing the features of different models through the variational autoencoder, thereby promoting complementarity between different models.

[0150] Example 2

[0151] See also Figure 5 , Figure 5 This is a block diagram of a counterfeit face identification system provided in Example 2 of the present invention. This counterfeit face identification system 200 is used to implement the above-mentioned embodiments and preferred embodiments, and details already described are omitted. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0152] Specifically, the fake face identification system 200 includes: an acquisition module 21, a training module 22, a fusion module 23, a fine-tuning module 24, and an input module 25, wherein:

[0153] An acquisition module 21 is configured to acquire a facial image to be identified, as well as a dataset of paired real and forged facial images, extract features using a lightweight CNN, perform K-nearest-neighbor clustering in the feature space, and divide the data into multiple target datasets.

[0154] The training module 22 is used to input the target data set into the deep learning model for joint training of the feature extractor. The deep learning model includes at least Inception, ResNet50, ViT, and EfficientNet. Specifically, the target data set is randomly input into the deep learning model; after feature extraction, comparative learning is performed, and then each deep learning model inputs the feature into the classification head to output the result, calculates the loss according to the loss function, and performs iterative training so that each deep learning model can serve as a unique feature extractor and uses the corresponding test set for performance evaluation. In the step of calculating the loss according to the loss function and performing iterative training, the calculation formula of the total loss function is:

[0155] ;

[0156] Among them, L is the total loss function, L NCE is the contrastive learning loss function, L CE is the cross entropy loss function;

[0157] The calculation formula of contrastive learning loss function is:

[0158] ;

[0159] in, Represents the features extracted from the current positive sample, Represents the features obtained by different deep learning models to extract positive samples, Represents the features extracted by different deep learning models, T represents the temperature coefficient, which is used to control the steepness of the distribution. express and similarity;

[0160] The calculation formula of the cross entropy loss function is:

[0161] ;

[0162] Among them, N represents the number of samples, M represents the number of models, Indicates the j The weight of the deep learning model, Indicates the i The correct labels of samples, Indicates the i Sample No. j The prediction results made by the deep learning model. In addition, in the step of using the corresponding test set to perform performance evaluation, three indicators, Precision, Recall and F1-score, are used for evaluation. Specifically,

[0163] ;

[0164] Among them, TP represents the number of correctly predicted samples, FP represents the number of incorrectly predicted samples, Precision represents the precision rate, Recall represents the recall rate, and F1-score represents the harmonic mean of Precision and Recall;

[0165] The fusion module 23 is used to determine the trained feature extractor and use the variational autoencoder to map the extracted features to the shared latent feature space, fuse the features extracted by different models, and constrain the distribution through KL divergence, and then reconstruct the original features from the latent space through the decoder to ensure that key information is not lost during the encoding process. Specifically, the trained feature extractor is fixed, the features of the data set are extracted, and the features are obtained after simple splicing. , is the number of input channels, H and W are the height and width of the input respectively, the feature is input to the variational autoencoder, and is mapped to the shared potential space through convolution to obtain the feature and features , specifically,

[0166] The features Input into the first convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is:

[0167] ;

[0168] in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1. represents the batch normalization operation, represents a nonlinear activation function;

[0169] Will Input into the second convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is:

[0170] ;

[0171] in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1;

[0172] Will Input into two convolutional layers respectively to obtain features and features , the calculation formula is:

[0173] ;

[0174] ;

[0175] in, and Both represent 3×3 convolution operations with a stride of 1 and a padding of 1;

[0176] Then the KL divergence loss is used to constrain the features and features The characteristic distribution of

[0177] Then, the reparameterization technique is used to obtain the potential representation z, that is, the fused feature z, and the decoder is used to reconstruct the features of z. The calculation formula of the potential representation z is:

[0178] ;

[0179] in, As random variables, the samples are sampled through the reparameterization trick to ensure that the gradient of the latent representation z can be transferred so that the feature and features It can still be optimized by back propagation, and the feature is constrained by KL divergence loss and features In the step of feature distribution, KL divergence is used for regularization, and the calculation formula is:

[0180] ;

[0181] Among them, D KL represents the KL divergence loss, d represents the dimension of the latent space, σ i and μ i Represent the features of the i-th sample respectively. In the step of reconstructing the features of z using the decoder, the potential representation z is input into the first transposed convolution block, and R1 is obtained through the nonlinear activation function and batch normalization. The calculation formula is:

[0182] ;

[0183] in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1;

[0184] Then R1 is input into the second transposed convolution block, and R2 is obtained through nonlinear activation function and batch normalization. The calculation formula is:

[0185] ;

[0186] in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1;

[0187] Finally, R2 is input into the last transposed convolution layer to obtain the reconstructed features. , the calculation formula is:

[0188] ;

[0189] in, represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1. The goal of the decoder is to reconstruct features generated from the latent representation z. , as close as possible to the original input features, and the mean square error is used to measure the quality of the decoder reconstructed data. The calculation formula is:

[0190] ;

[0191] in, represents the reconstruction loss, N represents the total number of features, X Represents the original feature, Represents the reconstruction features;

[0192] The fine-tuning module 24 is used to obtain features using the trained feature extractor and variational autoencoder, and input the features into the classifier for fine-tuning to complete the training of the fake face identification network. Specifically, the features fused by the variational autoencoder, that is, the potential representation z, are gradually input into the convolution block to obtain z1. The calculation formula is:

[0193] ;

[0194] in, Represents a 3×3 transposed convolution operation with a stride of 1 and a padding of 1;

[0195] Then z1 is globally averaged pooled and input into the fully connected layer for dimensionality reduction to obtain z1. The calculation formula is:

[0196] ;

[0197] in, represents the global average pooling operation, Represents the dimensionality reduction operation of the fully connected layer;

[0198] Finally, after normalization, the predicted probability is obtained after the Sigmoid activation function , the calculation formula is:

[0199] ;

[0200] Among them, Sigmoid means that the activation function maps the output to [0,1];

[0201] The loss function used by the classifier is cross entropy loss, and the calculation formula is:

[0202] ;

[0203] in, represents the cross entropy loss, N represents the number of all samples, represents the true label, Represents the classification result of the classifier;

[0204] The input module 25 is used to input the face image to be identified into the trained forged face identification network and output the identification result.

[0205] Example 3

[0206] Another aspect of the present invention provides an electronic device, see Figure 6 , shown is an electronic device in embodiment 3 of the present invention, including a memory 20, a processor 10, and a computer program 30 stored in the memory and executable on the processor. When the processor 10 executes the computer program 30, the forged face identification method as described above is implemented.

[0207] In some embodiments, the processor 10 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, used to run program codes or process data stored in the memory 20, such as executing access restriction programs.

[0208] The memory 20 includes at least one type of readable storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 20 may be an internal storage unit of the electronic device, such as the hard disk of the electronic device. In other embodiments, the memory 20 may also be an external storage device of the electronic device, such as a plug-in hard disk equipped on the electronic device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory 20 may include both an internal storage unit of the electronic device and an external storage device. The memory 20 can be used not only to store application software and various types of data of the electronic device, but also to temporarily store data that has been output or is about to be output.

[0209] It should be pointed out that Figure 6 The structure shown does not constitute a limitation to the electronic device. In other embodiments, the electronic device may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0210] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the forged face identification method described above is implemented.

[0211] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0212] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0213] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0214] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0215] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for identifying fake faces, characterized in that: The method comprises: Obtain a dataset of facial images to be identified, as well as paired real and forged facial images. Use a lightweight CNN to extract features, perform K-nearest neighbor clustering in the feature space, and divide the data into multiple target datasets. The target dataset is input into a deep learning model for joint training of feature extractors, wherein the deep learning model includes at least Inception, ResNet50, ViT, and EfficientNet, and the target dataset is randomly input into the deep learning model; after feature extraction, comparative learning is performed, and then each deep learning model inputs the features into a classification head to output the results, calculates the loss according to the loss function, and performs iterative training so that each deep learning model can serve as a unique feature extractor, and uses the corresponding test set for performance evaluation; Determine the trained feature extractor and use a variational autoencoder to map the extracted features to a shared latent feature space. Fuse the features extracted by different models and constrain the distribution through KL divergence. Then, use the decoder to reconstruct the original features from the latent space to ensure that key information is not lost during the encoding process. Use the trained feature extractor and variational autoencoder to obtain features, and input the features into the classifier for fine-tuning to complete the training of the fake face identification network; Inputting the face image to be identified into the trained forged face identification network and outputting the identification result; The trained feature extractor is determined, and the extracted features are mapped to a shared latent feature space using a variational autoencoder, the features extracted by different models are fused, and the distribution is constrained by KL divergence, and then the original features are reconstructed from the latent space by the decoder. In the step, the trained feature extractor is fixed, the features of the data set are extracted, and the features are obtained after simple splicing. , is the number of input channels, H and W are the height and width of the input respectively, the feature is input to the variational autoencoder, and is mapped to the shared potential space through convolution to obtain the feature and features , specifically, The features Input into the first convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is: ； in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1. represents the batch normalization operation, represents a nonlinear activation function; Will Input into the second convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is: ； in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1; Will Input into two convolutional layers respectively to obtain features and features , the calculation formula is: ； ； in, and Both represent 3×3 convolution operations with a stride of 1 and a padding of 1; Then the KL divergence loss is used to constrain the features and features characteristic distribution of Then, the reparameterization technique is used to obtain the potential representation z, that is, the fused feature z, and the decoder is used to reconstruct the features of z. The calculation formula of the potential representation z is: ； in, As random variables, the samples are sampled through the reparameterization trick to ensure that the gradient of the latent representation z can be transferred so that the feature and features It can still be optimized through backpropagation.

2. The forged face identification method according to claim 1, characterized in that: In the step of calculating the loss according to the loss function and performing iterative training, the calculation formula of the total loss function is: ； Among them, L is the total loss function, L NCE is the contrastive learning loss function, L CE is the cross entropy loss function; The calculation formula of contrastive learning loss function is: ； in, Represents the features extracted from the current positive sample, Represents the features obtained by different deep learning models to extract positive samples, Represents the features extracted by different deep learning models, T represents the temperature coefficient, which is used to control the steepness of the distribution. express and similarity; The calculation formula of the cross entropy loss function is: ； Among them, N represents the number of samples, M represents the number of models, Indicates the j The weight of the deep learning model, Indicates the i The correct labels of samples, Indicates the i Sample No. j The prediction results made by a deep learning model.

3. The forged face identification method according to claim 2, characterized in that: In the step of using the corresponding test set to perform performance evaluation, three indicators, Precision, Recall and F1-score, are used for evaluation. Specifically, ； Among them, TP represents the number of correctly predicted samples, FP represents the number of incorrectly predicted samples, Precision represents the precision rate, Recall represents the recall rate, and F1-score represents the harmonic mean of Precision and Recall.

4. The forged face identification method according to claim 3, characterized in that: The feature is constrained by KL divergence loss and features In the step of feature distribution, KL divergence is used for regularization, and the calculation formula is: ； Among them, D KL represents the KL divergence loss, d represents the dimension of the latent space, σ i and μ i They represent the characteristics of the i-th sample respectively.

5. The forged face identification method according to claim 4, characterized in that: In the step of reconstructing the features of z using the decoder, the potential representation z is input into the first transposed convolution block, and R1 is obtained through the nonlinear activation function and batch normalization. The calculation formula is: ； in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1; Then R1 is input into the second transposed convolution block, and R2 is obtained through nonlinear activation function and batch normalization. The calculation formula is: ； in, Represents a 4×4 transposed convolution operation with a stride of 2 and a padding of 1; Finally, R2 is input into the last transposed convolution layer to obtain the reconstructed features. , the calculation formula is: ； in, Represents a 4×4 transposed convolution operation with a stride of 2 and padding of 1.

6. The forged face identification method according to claim 5, characterized in that: The goal of the decoder is to reconstruct features generated from the latent representation z , as close as possible to the original input features, and the mean square error is used to measure the quality of the decoder reconstructed data. The calculation formula is: ； in, represents the reconstruction loss, N represents the total number of features, X Represents the original feature, Represents the reconstruction feature.

7. The forged face identification method according to claim 6, characterized in that: In the step of inputting the features into the classifier for fine-tuning, the features fused by the variational autoencoder, i.e., the potential representation z, are gradually input into the convolution block to obtain z1, which is calculated as follows: ； in, Represents a 3×3 transposed convolution operation with a stride of 1 and a padding of 1; Then z1 is globally averaged pooled and input into the fully connected layer for dimensionality reduction to obtain z1. The calculation formula is: ； in, represents the global average pooling operation, Represents the dimensionality reduction operation of the fully connected layer; Finally, after normalization, the predicted probability is obtained after the Sigmoid activation function , the calculation formula is: ； Among them, Sigmoid means that the activation function maps the output to [0,1]; The loss function used by the classifier is cross entropy loss, and the calculation formula is: ； in, represents the cross entropy loss, N represents the number of all samples, represents the true label, Represents the classification result of the classifier.

8. A fake face identification system, characterized in that: For implementing the forged face identification method according to any one of claims 1 to 7, the system comprises: The acquisition module is used to obtain the face image to be identified, as well as a dataset of paired real and forged face images, and use a lightweight CNN to extract features. It performs K-nearest neighbor clustering in the feature space and divides the data into multiple target datasets. A training module is configured to input the target dataset into a deep learning model for joint training of feature extractors. The deep learning model includes at least Inception, ResNet50, ViT, and EfficientNet. The target dataset is randomly input into the deep learning model; after feature extraction, comparative learning is performed. Each deep learning model then inputs the features into a classification head to output the results. The loss is calculated based on a loss function, and iterative training is performed so that each deep learning model can serve as a unique feature extractor. The corresponding test set is used for performance evaluation. The fusion module is used to determine the trained feature extractor and use the variational autoencoder to map the extracted features to the shared latent feature space, fuse the features extracted by different models, and constrain the distribution through KL divergence, and then reconstruct the original features from the latent space through the decoder to ensure that key information is not lost during the encoding process. Specifically, the trained feature extractor is fixed, the features of the data set are extracted, and the features are obtained after simple splicing. , is the number of input channels, H and W are the height and width of the input respectively, the feature is input to the variational autoencoder, and is mapped to the shared potential space through convolution to obtain the feature and features , specifically, The features Input into the first convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is: ； in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1. represents the batch normalization operation, represents a nonlinear activation function; Will Input into the second convolution block, through nonlinear activation function and batch normalization, we get , the calculation formula is: ； in, Represents a 4×4 convolution operation with a stride of 2 and a padding of 1; Will Input into two convolutional layers respectively to obtain features and features , the calculation formula is: ； ； in, and Both represent 3×3 convolution operations with a stride of 1 and a padding of 1; Then the KL divergence loss is used to constrain the features and features characteristic distribution of Then, the reparameterization technique is used to obtain the potential representation z, that is, the fused feature z, and the decoder is used to reconstruct the features of z. The calculation formula of the potential representation z is: ； in, As random variables, the samples are sampled through the reparameterization trick to ensure that the gradient of the latent representation z can be transferred so that the feature and features Can still be optimized through backpropagation; The fine-tuning module is used to obtain features using the trained feature extractor and variational autoencoder, and input the features into the classifier for fine-tuning to complete the training of the fake face identification network; The input module is used to input the face image to be identified into the trained forged face identification network and output the identification result.

Images