Face deepfake detection method and system based on hybrid expert network

Abstract

The application provides a face deep fake detection method and system based on a hybrid expert network, relates to the technical field of face deep fake detection, and can realize rapid identification of simple fake images, improve the processing efficiency of the system in a high-throughput scene, and greatly reduce the calculation burden and the corresponding time through the construction of a two-level identification cascade architecture; for difficult images that are difficult to identify, the physical expert branch and the semantic expert branch are deeply coupled, and a classification-oriented dynamic gate adjustment mechanism is introduced, so that the adaptive trade-off of image texture defects and logical contradictions is realized, the general identification ability of the model for unknown fake types is greatly enhanced, and the detection accuracy for complex samples is improved.

Description

Technical Field

[0001] This invention relates to the field of deepfake face technology, specifically to a deepfake face detection method and system based on a hybrid expert network. Background Technology

[0002] With the evolution of generative artificial intelligence, deepfake face technology has diverged into two categories: "generative synthesis" (such as full-face synthesis generated by diffusion models) and "manipulation attacks" (such as face swapping and expression replay). Existing face forgery detection technologies typically employ a single convolutional neural network or visual transformer architecture to process all input images uniformly. When faced with massive amounts of internet data and diverse forgery methods, they face a bottleneck where computational efficiency and detection accuracy cannot be balanced. Due to the lack of an effective mechanism for differentiating the difficulty of image processing, detection systems often waste expensive computing resources on low-quality, easily identifiable forged images. This results in a significant increase in the average response latency of the system under high-throughput scenarios, making it difficult to meet real-time requirements.

[0003] Furthermore, traditional detection methods struggle to simultaneously examine both microscopic physical discontinuities and macroscopic logical consistency when faced with complex forged images, resulting in a high false negative rate. Additionally, existing models often exhibit insufficient generalization performance when confronted with unknown or novel forgery algorithms due to their over-reliance on specific training datasets, thus limiting their overall applicability.

[0004] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for detecting deepfake faces based on hybrid expert networks, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A deepfake face detection method based on hybrid expert networks includes the following steps: S1: Use a pre-trained lightweight convolutional neural network as a statistical filter, and then use the collected face images as input to the statistical filter to extract the spatial statistical features of the face images and calculate the first anomaly score. S2: Compare the first anomaly score of the face image with a preset first score threshold to obtain the first detection result. Based on the first detection result, classify the face image into a fake image or a difficult image. If it is classified as a fake image, output the first detection result directly. If it is classified as a difficult image, execute step S3. S3: Perform double normalization preprocessing on the difficult image, and then input it into the parallel physical expert branch and semantic expert branch in the hybrid expert network to extract the physical and logical features of the difficult image. S4: Use a classification-oriented gating network to identify the potential classification tendency of difficult images, and then generate fusion weights for the physical expert branch and the semantic expert branch according to the potential classification tendency. Then, perform weighted fusion of the physical features and logical features of the difficult images according to the fusion weights. S5: Input the fused features obtained after weighted fusion into a preset classifier to calculate the second anomaly score of the fused features. Then compare the second anomaly score with a preset second score threshold to obtain the second detection result of the difficult image and output it.

[0007] Preferably, the lightweight convolutional neural network adopts the Xception network structure; The calculation steps for the first anomaly score include: resizing the input face image, extracting the feature map of the face image through the Xception network structure, mapping it through a global average pooling layer and a fully connected layer to obtain a two-dimensional logical value, and finally using the Softmax function to calculate the normalized first forgery probability as the first anomaly score.

[0008] Preferably, both the first detection result and the second detection result are binary labels, and satisfy the following definition: When the first anomaly score is greater than the first score threshold, the first detection result is output as false, the binarization label is set to 0, and the corresponding face image is classified as a fake image. When the first anomaly score is not greater than the first score threshold, the first detection result is output as true, the binarization label is set to 1, and the corresponding face image is classified as a difficult image. When the second anomaly score is greater than the second score threshold, the second detection result is output as false, and the binarized label is set to 0. When the score of the second anomaly is not greater than the second score threshold, the output of the second detection result is true, and the binarized label is set to 1.

[0009] Preferably, in step S3, when performing double normalization preprocessing on the difficult image, the original pixel value range of the difficult image is first normalized, and then a split input strategy is adopted for the physical expert branch and the semantic expert branch: For difficult images input from the physics expert branch, standardization is performed using the first standard mean and first standard deviation of the ImageNet dataset. For the image data input to the semantic expert branch, the second standard mean and second standard variance of the CLIP pre-trained model are used for standardization.

[0010] Preferably, the physical features include physical texture features and boundary artifact features; The physics expert branch uses EfficientNet-B4 as the backbone network and uses the feature vector of the penultimate convolutional layer as the physical features. During the training phase, the physics expert branch uses the SBI data augmentation method to generate training samples online, including: A convex hull mask is generated using a facial landmark detection algorithm. The source face and the target face are then alpha-blended within the mask area, and elastic deformation and Gaussian blur are applied to simulate physical stitching artifacts.

[0011] Preferably, the logical feature is a high-level semantic logic inconsistency feature; The semantic expert branch uses a pre-trained CLIP-ViT model image encoder as its backbone network; During the training phase, the semantic expert branch is trained using a partial fine-tuning strategy, including: The parameters of the shallow Transformer modules of the image encoder in the CLIP-ViT model are frozen, and only the last few Transformer modules and the visual projection layer are unfrozen to preserve the pre-trained semantic priors.

[0012] Preferably, in step S4, the fusion weights are calculated based on a multi-task joint loss function, the total loss of which includes the main classification loss, classification-guided gating loss, and auxiliary classification cross-entropy loss of the physical expert branch and the semantic expert branch.

[0013] Preferably, the classification-guided gating loss is calculated based on the binary classification cross-entropy loss function, and the specific steps include: The potential classification tendencies of difficult images are divided into physical tampering and semantic tampering, and a binary classification label is assigned to each difficult image. When the potential classification tendency is physical tampering, the label value is 0, the fusion weight of the physical expert branch is increased, and the fusion weight of the semantic expert branch is decreased. When the potential classification tendency is semantic tampering, the label value is 1, the fusion weight of the physical expert branch is reduced, and the fusion weight of the semantic expert branch is increased.

[0014] Preferably, the face deepfake detection system is used to execute the above-described face deepfake detection method, specifically including: The statistical filtering module is used to use a pre-trained lightweight convolutional neural network as a statistical filter, and the collected face image as the input of the statistical filter to extract the spatial statistical features of the face image and calculate the first anomaly score. The first-level recognition module compares the first anomaly score of the face image with a preset first score threshold to obtain a first detection result. Based on the first detection result, the face image is classified as a fake image or a difficult image. If it is classified as a fake image, the first detection result is directly output. If it is classified as a difficult image, step S3 is executed. The feature extraction module is used to perform double normalization preprocessing on the difficult image, and then input it into the parallel physical expert branch and semantic expert branch in the hybrid expert network to extract the physical and logical features of the difficult image. The feature fusion module is used to identify the potential classification tendency of difficult images using a classification-oriented gating network, and then generate fusion weights for the physical expert branch and the semantic expert branch according to the potential classification tendency, and perform weighted fusion of the physical features and logical features of the difficult images according to the fusion weights. The secondary recognition module is used to input the fused features obtained after weighted fusion into a preset classifier to calculate the second anomaly score of the fused features, and then compare the second anomaly score with a preset second score threshold to obtain the second detection result of the difficult image and output it.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention, by constructing a cascaded architecture for two-level recognition, enables rapid identification of simple forged images, improving the system's processing efficiency in high-throughput scenarios and significantly reducing computational burden and response time. For more difficult images to identify, by deeply coupling physical and semantic expert branches and introducing a classification-oriented dynamic gating adjustment mechanism, an adaptive trade-off between image texture defects and logical contradictions is achieved, greatly enhancing the model's general recognition capability for unknown forgery types, thereby improving the detection accuracy of complex samples. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall method flow of the present invention; Figure 2 This is a schematic diagram of the test curve of the present invention on the FaceForensics++ dataset; Figure 3 This is a schematic diagram of the module structure of the deep face spoofing detection system in this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0018] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0019] Example: Please see Figures 1-2 The present invention provides a technical solution: A deepfake face detection method based on hybrid expert networks includes the following steps: S1: Use a pre-trained lightweight convolutional neural network as a statistical filter, and then use the collected face images as input to the statistical filter to extract the spatial statistical features of the face images and calculate the first anomaly score.

[0020] The lightweight convolutional neural network uses the Xception network architecture; The steps for calculating the first anomaly score include: resizing the input face image, typically by adjusting it to... The system first extracts the feature map of the face image through the Xception network structure, and then maps it through the global average pooling layer and the fully connected layer to obtain the two-dimensional logical value. Finally, the system uses the Softmax function to calculate the normalized first forgery probability as the first anomaly score.

[0021] A lightweight convolutional neural network is used for initial screening. In order to maximize inference efficiency while ensuring detection accuracy, the legacy_xception from the timm library is used as the backbone network. The last fully connected layer of the network is modified to output a classification head with a dimension of 2.

[0022] In this initial screening step, the Xception network structure is used because it employs depthwise separable convolutions and can sensitively capture subtle noise inconsistencies, compression artifacts, and frequency distribution anomalies in images. Compared to ordinary convolutional neural networks, it has a significantly reduced number of parameters and computational cost while maintaining a greater number of layers, making it more suitable as an initial filter in the field of deepfake faces. This approach can filter out easily identifiable, simple samples, which not only greatly improves inference throughput but also avoids the risk of misjudgment due to overfitting of simple images.

[0023] S2: Compare the first anomaly score of the face image with a preset first score threshold to obtain the first detection result. Based on the first detection result, classify the face image as a fake image or a difficult image. If it is classified as a fake image, output the first detection result directly. If it is classified as a difficult image, execute step S3.

[0024] The initial score threshold can be set based on the best F1-Score on the validation set, typically 0.96. A higher threshold ensures that only images with extremely obvious false features are directly blocked. This high threshold reduces the false positive rate and ensures the authority of the initial screening process. Furthermore, setting the threshold using the best F1-Score allows for a balance between precision and recall, ensuring the objectivity of the initial screening process.

[0025] Both the first and second detection results are binary labels and satisfy the following definition: When the first anomaly score is greater than the first score threshold, the first detection result is output as false, the binarization label is set to 0, and the corresponding face image is classified as a fake image. When the first anomaly score is not greater than the first score threshold, the first detection result is output as true, the binarization label is set to 1, and the corresponding face image is classified as a difficult image.

[0026] Understandably, computational latency accumulates throughout the image data processing process. By filtering out low-quality forged images and skipping subsequent steps in this step, not only can the GPU's computing power be focused on recognizing highly complex images, but the average system response time can also be significantly reduced.

[0027] S3: Perform double normalization preprocessing on the difficult image, and then input it into the parallel physical expert branch and semantic expert branch in the hybrid expert network to extract the physical and logical features of the difficult image.

[0028] In step S3, when performing double normalization preprocessing on the difficult image, the original pixel value range of the difficult image is first normalized, and then a split input strategy is adopted for the physical expert branch and the semantic expert branch: For difficult images input to the physics expert branch, standardization is performed using the first standard mean and first standard deviation from the ImageNet dataset. The first standard mean and first standard deviation are typically taken as follows: , To better adapt to EfficientNet; For the image data input to the semantic expert branch, standardization is performed using the second standard mean and second standard deviation of the CLIP pre-trained model. The second standard mean and second standard deviation are typically taken as follows: , To better adapt to CLIP-ViT.

[0029] For the parameters used in the standardization process of both models, the standardized values ​​are obtained from the statistical distribution of the datasets used during their respective pre-training. For the physics expert branch, these standardized values ​​are the statistical results of the mean and standard deviation of all pixels in the R, G, and B channels of the ImageNet training set. For the semantic expert branch, they are the statistical results of the large-scale image and text dataset used during CLIP pre-model training. Since the CLIP training data differs from the ImageNet dataset, containing more diverse network images, different resolutions, and color distributions, their statistical characteristics are also slightly different.

[0030] Physical features include physical texture features and boundary artifact features; The physics expert branch uses EfficientNet-B4 as the backbone network. In this embodiment, the feature dimension is set to 1792, and the feature vector of the penultimate convolutional layer is used as the physical feature. During the training phase, the physics expert branch uses SBI (Self-Fusion Image) data augmentation to generate training samples online, including: A convex hull mask is generated using a facial landmark detection algorithm. The source face and the target face are then alpha-blended within the mask area, and elastic deformation and Gaussian blur are applied to simulate physical stitching artifacts.

[0031] This study employs the SBI method for training, which utilizes techniques such as convex hull masking, elastic deformation, Gaussian blur, and alpha mixing to generate "fake evidence." This effectively simulates the physical discontinuities in the digital image stitching process, giving the trained physics expert branch a stronger general recognition capability for novel forged images that have never been seen before. Furthermore, using the penultimate layer features of EfficientNet-B4 avoids over-compression of the final classification layer, preserving the rich spatial texture features of the image. This allows the physics expert branch to accurately capture frequency loss caused by Gaussian blur or pixel steps at stitching points.

[0032] The logical characteristic is an inconsistency between high-level semantic logic; The semantic expert branch uses the image encoder of the pre-trained CLIP-ViT model as the backbone network, and in this embodiment, the feature dimension is set to 1024. During the training phase, the semantic expert branch is trained using a partial fine-tuning strategy, including: The parameters of the shallow Transformer modules of the image encoder in the CLIP-ViT model are frozen, and only the last few Transformer modules and the visual projection layer are unfrozen to preserve the pre-trained semantic priors.

[0033] In this embodiment, the parameters of the first 22 Transformer modules of the CLIP-ViT model are frozen, while only the last two layers are unfrozen. Freezing the shallow layers preserves the general visual cognition of the CLIP-ViT model, improving the success rate of the training process. Unfreezing the last two layers directs the model's cognition towards the specific task of "real vs. fake recognition." This setup not only fully utilizes the model's performance but also prevents overfitting, thereby reducing training costs.

[0034] Understandably, different deep learning models see different image distributions during the pre-training phase. For example, EfficientNet, used in the physics expert branch, is a highly optimized convolutional neural network that excels at capturing local textures and spatial details in images. Since it was pre-trained on the ImageNet dataset, it serves as the physics expert in this embodiment, specifically designed to identify physical defects in images, such as seams and blurred edges on faces. CLIP-ViT, used in the semantic expert branch, is a model based on the VisionTransformer architecture, trained using the CLIP (contrastive language-image pre-training) method. It excels at understanding long-distance relationships between different parts of an image. Furthermore, because it was trained using text descriptions and image pairing, it can identify overall logical flaws in images, such as inconsistencies between the gaze direction and the head tilt direction.

[0035] S4: Use a classification-oriented gating network to identify the potential classification tendency of difficult images, and then generate fusion weights for the physical expert branch and the semantic expert branch according to the potential classification tendency. Finally, perform weighted fusion of the physical and logical features of the difficult images according to the fusion weights.

[0036] Here, we need to extract the physical feature vector from the physics expert branch. Logical feature vectors extracted from semantic expert branches The mapping is performed to a unified feature space for dimensional alignment, followed by concatenation along the channel dimension. For gated networks, they can typically be built upon a pre-trained multilayer perceptron, specifically including: The first fully connected layer is used to reduce the dimension of the concatenated vector from 1024 to 256. The activation function layer uses the ReLU activation function to increase nonlinearity; The second fully connected layer is used to further reduce the dimensionality of the concatenated vector from 256 to 2, corresponding to the fusion weights of the physical expert branch and the semantic expert branch, respectively. The normalization layer is used to normalize the fusion weights of the physical expert branch and the semantic expert branch so that their sum is 1.

[0037] In step S4, the fusion weights are calculated based on the multi-task joint loss function. The total loss of the multi-task joint loss function includes the main classification loss, the classification-guided gating loss, and the auxiliary classification cross-entropy loss of the physical expert branch and the semantic expert branch.

[0038] The functional expression of the multi-task joint loss function is: In the formula Indicates the total loss. Indicates the primary classification loss. Indicates classification-oriented gating loss. , Let the cross-entropy losses for auxiliary classification be represented by the physics expert branch and the semantic expert branch, respectively. , The weights corresponding to the two types of losses are set to 1 and 0.5 respectively in this embodiment.

[0039] Because the DeepFake dataset used for training often has an imbalance in the ratio of positive to negative samples (i.e., more fake images than real images), Focal Loss is used instead of the traditional cross-entropy loss for the main classification loss. Its calculation formula is as follows: In the formula This represents the balance factor, used to adjust the weight ratio of positive and negative samples. This represents the classification probability, that is, the probability that a sample belongs to the correct category. This represents the focus parameter, which controls the degree of attention given to difficult-to-classify samples. The larger the value, the higher the degree of attention given to difficult-to-classify images. In this embodiment, it is set to 2.

[0040] The classification-guided gating loss is calculated based on the binary cross-entropy loss function, and the specific steps include: The potential classification tendencies of difficult images are divided into physical tampering and semantic tampering, and a binary classification label is assigned to each difficult image. When the potential classification tendency is physical tampering, the label value is 0, the fusion weight of the physical expert branch is increased, and the fusion weight of the semantic expert branch is decreased. When the potential classification tendency is semantic tampering, the label value is 1, the fusion weight of the physical expert branch is reduced, and the fusion weight of the semantic expert branch is increased.

[0041] A gating network is essentially a miniature classifier. During training, it is fed labeled data. When the training samples are of the physical tampering class, the classification-oriented gating loss is forced to guide the gating network to assign high weights to the physical expert branch. Conversely, when the training samples are of the semantic tampering class, the semantic expert branch is assigned high weights. Therefore, after training, it can identify the potential classification tendencies of difficult images.

[0042] The formula for calculating the classification-guided gating loss is: In the formula Indicates category labels, , ... Approaching 1 (in engineering applications, it is directly considered equal to 1), when the difficult image is of the semantic tampering type, it makes Approaching 1.

[0043] To prevent the physical expert branch and the semantic expert branch from degenerating before fusion (i.e., one branch becoming completely ineffective), auxiliary classification heads need to be added to the outputs of both the physical expert branch and the semantic expert branch, and the corresponding auxiliary classification cross-entropy loss needs to be calculated. Specifically, the following two-stage training strategy can be adopted: Phase 1: Train the Xception network structure separately for 15 epochs using the stage1 training code, with a learning rate set to [missing value]. Save the model weights with the highest AUC (area under the receiver operating characteristic curve) on the validation set.

[0044] Phase 2: Freeze the stage 1 parameters and train the coupled hybrid expert network for 20 epochs using the stage 2 training code. Employ the Adam optimizer with an initial learning rate of [value missing]. It also dynamically adjusts the learning rate in conjunction with a cosine annealing strategy.

[0045] By attaching auxiliary classification heads to the ends of the physical and semantic branches respectively, it is required that these two branches can independently identify true and false even without fusion, thereby ensuring that the extracted physical feature vectors and logical feature vectors contain high-quality discriminative information and enhancing the reliability of the final detection results.

[0046] The final fused feature vector This can be expressed as: S5: Input the fused features obtained after weighted fusion into a preset classifier to calculate the second anomaly score of the fused features. Then compare the second anomaly score with a preset second score threshold to obtain the second detection result of the difficult image and output it.

[0047] The classifier here is usually composed of a fully connected layer, which is used to reduce the dimensionality of the fused vector to a 2-dimensional space, corresponding to the two-dimensional logic of true and false respectively. Then, the Softmax function is used to convert the logic value into a second forgery probability and use it as the second anomaly score.

[0048] When the second anomaly score is greater than the second score threshold, the second detection result is output as false, and the binarized label is set to 0. When the second anomaly score is not greater than the second score threshold, the second detection result is output as true, the binarized label is set to 1, and the final recognition result is output.

[0049] In this embodiment, the overall operating environment is implemented under the PyTorch 1.12 deep learning framework, and OpenCV and Dlib are used for image preprocessing. The hardware environment uses an NVIDIA RTX 3060 GPU, and the area under the receiver operating characteristic curve (AUC) and inference speed (FPS) are used as the main evaluation metrics.

[0050] To verify the model's robustness under unknown attacks, this embodiment trains the model on FaceForensics++ and performs cross-domain testing on the Celeb-DF-v2 dataset. The AUC on Celeb-DF reaches 87.30%, while the benchmark method SBI (Self-Blended Images) has an AUC of 81.16%. Compared to SBI, this invention achieves a 6.14% improvement, indicating that the introduction of the semantic expert branch effectively compensates for the shortcomings of relying solely on physical texture detection. The latest state-of-the-art method, More Clues, has an AUC of 86.40%, which is also lower than this invention. Furthermore, removing the gating network in this scheme reduces the AUC to 84.5%, and removing the semantic expert branch significantly reduces the AUC to 81.2%, further demonstrating the effectiveness and superiority of the "two-stream decoupling" and "classification adaptive gating" in this scheme.

[0051] like Figure 2 As shown, this scheme maintains a high AUC across subsets of the same dataset FaceForensics++ (c23), where the horizontal and vertical axes represent the probabilities of being identified as fake / real, respectively. Each curve represents a different subset, and each subset represents a different type of forgery. The figure shows that the AUC for the FaceSwap subset is 99.98%, the Face2Face subset is 99.91%, the Deepfakes subset is 99.98%, and the NeuralTextures subset is 99.51%, with an overall average AUC of 99.98%. Furthermore, the average inference speed during recognition reaches 65.8 FPS, indicating saturated detection performance.

[0052] Please see Figure 3 The present invention also provides a face deepfake detection system based on a hybrid expert network for performing the above-described face deepfake detection method, specifically including: a statistical screening module, a first-level recognition module, a feature extraction module, a feature fusion module, and a second-level recognition module.

[0053] The statistical filtering module uses a pre-trained lightweight convolutional neural network as a statistical filter, and the acquired face image as the input of the statistical filter to extract the spatial statistical features of the face image and calculate the first anomaly score. The first-level recognition module compares the first anomaly score of the face image with a preset first score threshold to obtain the first detection result. Based on the first detection result, the face image is classified as a fake image or a difficult image. If it is classified as a fake image, the first detection result is directly output. If it is classified as a difficult image, step S3 is executed. The feature extraction module is used to perform double normalization preprocessing on difficult images, and then input them into the parallel physical expert branch and semantic expert branch in the hybrid expert network to extract the physical and logical features of the difficult images. The feature fusion module is used to identify the potential classification tendency of difficult images using a classification-oriented gating network. Then, based on the potential classification tendency, it generates fusion weights for the physical expert branch and the semantic expert branch respectively, and performs weighted fusion of the physical and logical features of the difficult image based on the fusion weights. The secondary recognition module is used to input the fused features obtained after weighted fusion into a preset classifier to calculate the second anomaly score of the fused features. The second anomaly score is then compared with a preset second score threshold to obtain the second detection result of the difficult image and output it.

[0054] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0055] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0056] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0057] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for detecting deepfake faces based on hybrid expert networks, characterized in that, The specific steps include: S1: Use a pre-trained lightweight convolutional neural network as a statistical filter, and then use the collected face images as input to the statistical filter to extract the spatial statistical features of the face images and calculate the first anomaly score. S2: Compare the first anomaly score of the face image with a preset first score threshold to obtain the first detection result. Based on the first detection result, classify the face image into a fake image or a difficult image. If it is classified as a fake image, output the first detection result directly. If it is classified as a difficult image, execute step S3. S3: Perform double normalization preprocessing on the difficult image, and then input it into the parallel physical expert branch and semantic expert branch in the hybrid expert network to extract the physical and logical features of the difficult image. S4: A classification-oriented gating network is used to identify the potential classification tendency of difficult images. The potential classification tendency of difficult images includes physical tampering and semantic tampering. Then, based on the potential classification tendency, the fusion weights of the physical expert branch and the semantic expert branch are generated respectively. The physical features and logical features of the difficult images are weighted and fused based on the fusion weights. S5: Input the fused features obtained after weighted fusion into a preset classifier to calculate the second anomaly score of the fused features. Then compare the second anomaly score with a preset second score threshold to obtain the second detection result of the difficult image and output it.

2. The face deepfake detection method based on a hybrid expert network according to claim 1, characterized in that: The lightweight convolutional neural network adopts the Xception network structure; The calculation steps for the first anomaly score include: resizing the input face image, extracting the feature map of the face image through the Xception network structure, mapping it through a global average pooling layer and a fully connected layer to obtain a two-dimensional logical value, and finally using the Softmax function to calculate the normalized first forgery probability as the first anomaly score.

3. The face deepfake detection method based on a hybrid expert network according to claim 1, characterized in that: Both the first and second detection results are binary labels and satisfy the following definition: When the first anomaly score is greater than the first score threshold, the first detection result is output as false, the binarization label is set to 0, and the corresponding face image is classified as a fake image. When the first anomaly score is not greater than the first score threshold, the first detection result is output as true, the binarization label is set to 1, and the corresponding face image is classified as a difficult image. When the second anomaly score is greater than the second score threshold, the second detection result is output as false, and the binarized label is set to 0. When the score of the second anomaly is not greater than the second score threshold, the output of the second detection result is true, and the binarized label is set to 1.

4. The face deepfake detection method based on a hybrid expert network according to claim 1, characterized in that: In step S3, when performing double normalization preprocessing on the difficult image, the original pixel value range of the difficult image is first normalized, and then a split input strategy is adopted for the physical expert branch and the semantic expert branch: For difficult images input from the physics expert branch, standardization is performed using the first standard mean and first standard deviation of the ImageNet dataset. For the image data input to the semantic expert branch, the second standard mean and second standard variance of the CLIP pre-trained model are used for standardization.

5. The face deepfake detection method based on a hybrid expert network according to claim 1, characterized in that: The physical features include physical texture features and boundary artifact features; The physics expert branch uses EfficientNet-B4 as the backbone network and uses the feature vector of the penultimate convolutional layer as the physical features. During the training phase, the physics expert branch uses the SBI data augmentation method to generate training samples online, including: A convex hull mask is generated using a facial landmark detection algorithm. The source face and the target face are then alpha-blended within the mask area, and elastic deformation and Gaussian blur are applied to simulate physical stitching artifacts.

6. The face deepfake detection method based on a hybrid expert network according to claim 1, characterized in that: The logical feature is a high-level semantic logic inconsistency feature; The semantic expert branch uses a pre-trained CLIP-ViT model image encoder as its backbone network; During the training phase, the semantic expert branch is trained using a partial fine-tuning strategy, including: The parameters of the shallow Transformer modules of the image encoder in the CLIP-ViT model are frozen, and only the last few Transformer modules and the visual projection layer are unfrozen to preserve the pre-trained semantic priors.

7. The face deepfake detection method based on a hybrid expert network according to claim 1, characterized in that: In step S4, the fusion weights are calculated based on the multi-task joint loss function. The total loss of the multi-task joint loss function includes the main classification loss, the classification-guided gating loss, and the auxiliary classification cross-entropy loss of the physical expert branch and the semantic expert branch.

8. The face deepfake detection method based on a hybrid expert network according to claim 7, characterized in that: The classification-guided gating loss is calculated based on the binary classification cross-entropy loss function, and the specific steps include: The potential classification tendencies of difficult images are divided into physical tampering and semantic tampering, and a binary classification label is assigned to each difficult image. When the potential classification tendency is physical tampering, the label value is 0, the fusion weight of the physical expert branch is increased, and the fusion weight of the semantic expert branch is decreased. When the potential classification tendency is semantic tampering, the label value is 1, the fusion weight of the physical expert branch is reduced, and the fusion weight of the semantic expert branch is increased.

9. A face deepfake detection system based on a hybrid expert network, characterized in that: The face deepfake detection system is used to execute the face deepfake detection method as described in any one of claims 1-8, specifically including: The statistical filtering module is used to use a pre-trained lightweight convolutional neural network as a statistical filter, and the collected face image as the input of the statistical filter to extract the spatial statistical features of the face image and calculate the first anomaly score. The first-level recognition module compares the first anomaly score of the face image with a preset first score threshold to obtain a first detection result. Based on the first detection result, the face image is classified as a fake image or a difficult image. If it is classified as a fake image, the first detection result is directly output. If it is classified as a difficult image, step S3 is executed. The feature extraction module is used to perform double normalization preprocessing on the difficult image, and then input it into the parallel physical expert branch and semantic expert branch in the hybrid expert network to extract the physical and logical features of the difficult image. The feature fusion module is used to identify the potential classification tendency of difficult images using a classification-oriented gating network, and then generate fusion weights for the physical expert branch and the semantic expert branch according to the potential classification tendency, and perform weighted fusion of the physical features and logical features of the difficult images according to the fusion weights. The secondary recognition module is used to input the fused features obtained after weighted fusion into a preset classifier to calculate the second anomaly score of the fused features, and then compare the second anomaly score with a preset second score threshold to obtain the second detection result of the difficult image and output it.

Images