Method and system for detecting deeply-forged face image

Abstract

The invention provides a detection method and system for a deeply-forged face image, and the method comprises the steps: constructing a face image data set and a deeply-forged image detection model which comprises a router, an expert model and a Transform module; the face image data set is adopted to carry out joint training on the router, the expert model and the Transform module; and inputting a to-be-detected face image into the trained deep counterfeit image detection model to obtain the probability of whether the to-be-detected face image is a deep counterfeit image or not. According to the method, the robustness and generalization ability of deep counterfeiting detection are enhanced, and the weakness of the existing method in facing diversified counterfeiting technologies and antagonism disturbance is solved, especially for complex propagation scenes and environments with many confrontation factors; by combining the multi-expert model and the Transform module, the accuracy and robustness of detection are improved, and the generalization ability and adaptability of the model to different types of forgery are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for detecting deep fake facial images. Background Art

[0002] Artificial intelligence (AI) techniques for creating images and videos have made significant progress in the past few decades, with the rapid development of deep learning. Artificial digital images produced by these techniques are difficult to distinguish from images taken by digital cameras.

[0003] Deepfake methods can be divided into four types: synthesis, attribute editing, identity swapping, and face reenactment. Synthesis (e.g., PGGAN, StyleGAN, and StyleGAN2) refers to the process of creating fake facial images of people who do not exist in the real world. Attribute editing (e.g., STGAN, StarGAN, and StarGANv2) refers to the process of changing a person's facial features, such as hairstyle or facial hair, or adding or removing signs of aging. Meanwhile, identity swapping (e.g., FaceSwap) is a method that replaces one person's face with another's. Finally, in the reenactment process (e.g., Face2Face), although the person in the image remains the same, their facial expressions change. If the fake image is created using an existing deepfake generator and the source of the image has been identified, deepfake detection is relatively easy and achieves relatively high accuracy.

[0004] At present, deep fake detection technology is not mature, and there are research difficulties such as poor robustness against adversarial perturbations (such as resizing, blurring, etc.), weak generalization ability, and increased difficulty in detecting biological signals. It can be divided into:

[0005] 1. Passive detection methods for deep fakes (lack of high-quality data sets, inconsistent evaluation mechanisms, and serious mismatches between experimental environments and real-world scenario data) are unable to eliminate the harm that has already been caused. There is little research on active defense technologies, and data sources are uncontrollable, which limits their practicality.

[0006] 2. Due to the complexity of propagation scenarios and many adversarial factors, the generalization ability of existing technologies is weak and their universality is low.

[0007] 3. The distribution of data sets is single, the evaluation standards are inconsistent, the initiative is insufficient, and the practicality needs to be strengthened. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention provides a method and system for detecting deep fake facial images. The present invention can solve the weaknesses of the prior art when facing diversified forgery techniques and adversarial disturbances, especially for complex propagation scenarios and environments with many adversarial factors.

[0009] The technical solution of the present invention is: a method for detecting deep fake face images, comprising the following steps:

[0010] S1), constructing a face image dataset;

[0011] S2), constructing a deep fake image detection model, wherein the deep fake image detection model includes a router, an expert model and a Transformer module;

[0012] S3), using the face image dataset to jointly train the router, expert model and Transformer module;

[0013] S4) inputting the face image to be detected into the trained deep fake image detection model to obtain the probability of whether the face image to be detected is a deep fake image.

[0014] Preferably, in step S2), the MobileNet V2 model is used as the router; and n CNN models are used as expert models.

[0015] Preferably, in step S2), the last layer of the MobileNet V2 model is replaced by a fully connected layer, and the output dimension is set to the number of expert models.

[0016] Preferably, in step S3), the joint training of the router, the expert model and the Transformer module specifically includes the following steps:

[0017] S31), initialize the pre-trained weights of the router and the expert model, freeze the underlying weights of the expert model, and only train the router and Transformer modules independently;

[0018] S32), entering the joint training mode, updating the weights of the Router and Transformer modules at the same time, and fine-tuning or synchronously updating the high-level weights of the expert model as needed;

[0019] S33) According to the performance of the validation set, adjust the learning rate or weight update strategy to optimize the feature extraction and binary classification capabilities of the Router, expert model, and Transformer modules.

[0020] Preferably, in step S4), the trained deep fake image detection model is used to detect the face image to be detected, which specifically includes the following steps:

[0021] S41), using the trained router Router to extract high-level features of the face image to be detected; outputting an n-dimensional weight vector as the weights of different experts through the MobileNet V2 model;

[0022] S42), using the trained n CNN models as expert models, and generating a corresponding feature map according to the face image to be detected through each expert model;

[0023] S43), obtaining a mixed feature map by weighted summing the n-dimensional weight vector output by the router and the feature maps output by the n expert models;

[0024] S44), by flattening the mixed feature map into a feature vector, and capturing the dependency between the feature vectors through the trained Transformer module; finally, the probability of whether the face image to be detected is a deep fake image is obtained.

[0025] Preferably, in step S43), the calculation expression of the n-dimensional weight vector output by the weighted sum router Router and the n expert models is:

[0026]

[0027] In the formula, w i represents the i-th weight vector of the router, f i represents the feature map of the i-th expert model; F represents the weighted mixed feature map.

[0028] Preferably, in step S44), the Transformer module performs binary classification on the feature vector through a self-attention mechanism to obtain the probability of whether the face image to be detected is a deep fake image.

[0029] The present invention also provides a deep fake face image detection system, comprising:

[0030] A data acquisition module, used for acquiring facial images;

[0031] Router module, used to extract high-level features of the face image to be detected;

[0032] A multi-expert module is used to generate a corresponding number of feature maps according to the face image to be detected;

[0033] The weighted summation module is used to perform weighted summation on the weight vector output by the Router module and the feature map output by the multi-expert module to obtain a mixed feature map;

[0034] The binary classification module is used to perform binary classification based on the mixed feature map to detect whether the face image is forged.

[0035] Preferably, the Router module extracts high-level features of the face image to be detected by using the MobileNet V2 model, and outputs an n-dimensional weight vector as the weights of different experts through the MobileNet V2 model.

[0036] Preferably, the multi-expert module adopts n CNN models, and each CNN model generates a corresponding feature map according to the face image to be detected.

[0037] Preferably, the binary classification module uses a self-attention mechanism through a Transformer module to perform binary classification on the feature vector to obtain the probability of whether the face image to be detected is a deep fake image.

[0038] The beneficial effects of the present invention are:

[0039] 1. The present invention improves the accuracy and robustness of detection by combining the multi-expert model and the Transformer module;

[0040] 2. This invention enhances the robustness and generalization of deep fake detection, addressing the weaknesses of existing methods in the face of diverse fake techniques and adversarial perturbations, especially for complex propagation scenarios and environments with many adversarial factors;

[0041] 3. The present invention improves the generalization ability and adaptability of the model to different types of forgeries by combining multiple CNN models as expert models;

[0042] 4. The present invention solves the problems of lack of existing data sets and uneven distribution by constructing a more comprehensive and high-quality training data set, thereby improving the generalization ability and applicability of the detection model;

[0043] 5. The present invention uses the self-attention mechanism of Transformer to capture the dependencies between features and optimizes the model structure and parameters to reduce the computational cost and optimization difficulty. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the process of the present invention;

[0045] Figure 2 This is a framework result diagram of the system of the present invention. DETAILED DESCRIPTION

[0046] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0047] Example 1

[0048] like Figure 1As shown, this embodiment provides a method for detecting deep fake face images, comprising the following steps:

[0049] S1), constructing a face image dataset;

[0050] In this embodiment, existing public face image datasets are used, including the FaceForensics++ dataset, the DeepFake Detection Challenge Dataset (DFDC) dataset, and the Celeb-DF (Celeb

[0051] Deepfake) dataset, DeeperForensics-1.0 dataset, UADFV (YouTube Aligned

[0052] The FaceForensics++ dataset contains various types of forged images and uses FaceSwap, Face2Face, Deepfakes and other technologies to generate forged videos. It includes 1,000 original videos and their forged versions (high-quality and low-quality versions), as well as related annotations.

[0053] The DeepFake Detection Challenge Dataset (DFDC) dataset was released by Facebook. The DeepFake Detection Challenge Dataset (DFDC) dataset covers various types of forgery methods, including more complex forgery samples. It includes about 100,000 videos, some of which are real videos and the rest are forged videos generated by different algorithms.

[0054] The Celeb-DF (Celeb Deepfake) dataset consists of 590 high-quality fake videos and synthesized real videos.

[0055] The DeeperForensics-1.0 dataset includes 10,000 video clips featuring a variety of forgery scenarios in real-world environments.

[0056] The UADFV (YouTube Aligned DeepFake Videos) dataset contains YouTube videos synthesized using Deepfake technology, which has about 50 fake videos and 50 real videos.

[0057] In addition, after obtaining the face image data set, this embodiment needs to preprocess it, including alignment, cropping, standardization, denoising, and histogram equalization. Among them, standardization refers to standardizing or normalizing the pixel values. This embodiment adopts 0-1 normalization or zero mean normalization; denoising uses a filtering method (such as median filtering or Gaussian filtering) to remove noise in the image and improve the clarity of the image; and histogram equalization refers to equalizing the brightness distribution of the image to enhance the contrast of the image, which is suitable for images with uneven lighting.

[0058] The preprocessed face image dataset is then enhanced, including random cropping and scaling, rotation and flipping, brightness, contrast and saturation adjustment, Gaussian noise addition, color jitter, Cutout or random occlusion, etc.

[0059] Then, this embodiment also generates more fake face images through a generative adversarial network (GAN) to make up for the lack of data, and synthesizes new samples through an image deformation network to generate enhanced data that meets the target features; and combines two face images into a new image through linear interpolation to generate more diverse samples.

[0060] In addition, this embodiment also simulates real-life occlusion situations for facial images, such as adding random occlusion objects (masks, sunglasses), which helps the model's detection capabilities in occlusion situations, and performs different degrees of blurring on the image (such as motion blur or Gaussian blur) to make the model more robust.

[0061] Finally, the face image dataset obtained after the above processing is divided into a training set and a validation set.

[0062] S2), constructing a deep fake image detection model, wherein the deep fake image detection model includes a router, an expert model and a Transformer module;

[0063] In this embodiment, the MobileNet V2 model is used as the router; the last layer of the MobileNet V2 model is replaced by a fully connected layer, and the output dimension is set to the number of expert models; n CNN models are used as expert models. In this embodiment, the CNN model adopts EfficientNet, XceptionNet, or ResNet network.

[0064] S3) Use the training set to jointly train the router, expert model and Transformer module; specifically, the following steps are included:

[0065] S31), initialize the pre-trained weights of the router and the expert model, freeze the underlying weights of the expert model, and only train the router and Transformer modules independently;

[0066] S32), entering the joint training mode, updating the weights of the Router and Transformer modules at the same time, and fine-tuning or synchronously updating the high-level weights of the expert model as needed;

[0067] S33) According to the performance of the validation set, adjust the learning rate or weight update strategy to optimize the feature extraction and binary classification capabilities of the Router, expert model, and Transformer modules.

[0068] The MobileNet V2 model is normalized into a weight vector [w1,w2,...,w n ], the MobileNet V2 model uses the cross entropy loss function as the loss function and uses Adam or SGD as the optimizer. The MobileNet V2 model freezes some of the underlying weights and only updates the last few weights and the fully connected layers.

[0069] Moreover, the weight update of the expert model can be selected between "freeze" and "synchronous update". When resources are limited, the pre-trained expert model weights are frozen to reduce training overhead; when resources allow, some parameters of the expert model are allowed to be updated synchronously during the training process. If synchronous weight update is selected, the high-level weights of the expert model are updated and the low-level weights are frozen; and the preset learning rate is used for fine-tuning and updating to prevent the expert model parameters from deviating significantly from the pre-trained distribution.

[0070] Similarly, the expert model and Transformer module both use the cross entropy loss function as the loss function and use Adam or SGD as the optimizer.

[0071] S4), inputting the face image to be detected into the trained deep fake image detection model to obtain the probability of whether the face image to be detected is a deep fake image, specifically comprising the following steps:

[0072] S41), using the trained router Router to extract high-level features of the face image to be detected; outputting an n-dimensional weight vector as the weights of different experts through the MobileNet V2 model;

[0073] S42), using the trained n CNN models as expert models, and generating a corresponding feature map according to the face image to be detected through each expert model;

[0074] S43), by weighted summing the n-dimensional weight vector output by the router and the feature graphs output by the n expert models, a mixed feature graph is obtained; its expression is:

[0075]

[0076] In the formula, w i represents the i-th weight vector of the router, f i represents the feature map of the i-th expert model; F represents the weighted mixed feature map;

[0077] S44), by flattening the mixed feature map into a feature vector, and using the trained Transformer module to perform binary classification on the feature vector through a self-attention mechanism; finally, the probability of whether the face image to be detected is a deep fake image is obtained.

[0078] Example 2

[0079] like Figure 2 As shown, this embodiment provides a deep fake face image detection system, including:

[0080] A data acquisition module, used for acquiring facial images;

[0081] Router module, used to extract high-level features of the face image to be detected;

[0082] A multi-expert module is used to generate a corresponding number of feature maps according to the face image to be detected;

[0083] The weighted summation module is used to perform weighted summation on the weight vector output by the Router module and the feature map output by the multi-expert module to obtain a mixed feature map;

[0084] The binary classification module is used to perform binary classification based on the mixed feature map to detect whether the face image is forged.

[0085] As a preferred embodiment of this embodiment, the Router module extracts high-level features of the face image to be detected by using the MobileNet V2 model, and outputs an n-dimensional weight vector as the weights of different experts through the MobileNet V2 model.

[0086] As a preferred embodiment of this embodiment, the last layer of the MobileNet V2 model is replaced by a fully connected layer, and the output dimension is set to the number of expert models.

[0087] As a preferred embodiment of the present invention, the multi-expert module adopts n CNN models, and each CNN model generates a corresponding feature map according to the face image to be detected.

[0088] As a preferred embodiment of the present invention, the calculation expression for calculating the mixed feature map by the weighted summation module is:

[0089]

[0090] In the formula, w i represents the i-th weight vector of the router, f i represents the feature map of the i-th expert model; F represents the weighted mixed feature map.

[0091] As a preferred embodiment of the present invention, the binary classification module flattens the mixed feature map into a feature vector, and then uses a self-attention mechanism through a Transformer module to perform binary classification on the feature vector to obtain the probability of whether the face image to be detected is a deep fake image.

[0092] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A method for detecting deep fake face images, characterized in that: The following steps are involved: S1), constructing a face image dataset; S2), constructing a deep fake image detection model, wherein the deep fake image detection model includes a router, an expert model and a Transformer module; S3), using the face image dataset to jointly train the router, expert model and Transformer module; S4) inputting the face image to be detected into the trained deep fake image detection model to obtain the probability of whether the face image to be detected is a deep fake image.

2. A method for detecting deep fake face images according to claim 1, characterized in that: In step S2), the MobileNet V2 model is used as the router; and n CNN models are used as expert models.

3. A method for detecting deep fake face images according to claim 2, characterized in that: In step S2), the last layer of the MobileNet V2 model is replaced by a fully connected layer, and the output dimension is set to the number of expert models.

4. The method for detecting deep fake face images according to claim 1, characterized in that: In step S3), the joint training of the router, the expert model and the Transformer module specifically includes the following steps: S31), initialize the pre-trained weights of the router and the expert model, freeze the underlying weights of the expert model, and only train the router and Transformer modules independently; S32), enter the joint training mode, update the weights of the router and transformer modules at the same time, and fine-tune or synchronously update the high-level weights of the expert model as needed; S33) According to the performance of the validation set, adjust the learning rate or weight update strategy to optimize the feature extraction and binary classification capabilities of the Router, expert model, and Transformer modules.

5. The method for detecting deep fake face images according to claim 1, characterized in that: In step S4), the trained deep fake image detection model is used to detect the face image to be detected, which specifically includes the following steps: S41), using the trained router Router to extract high-level features of the face image to be detected; outputting an n-dimensional weight vector as the weights of different experts through the MobileNetV2 model; S42), using the trained n CNN models as expert models, and generating a corresponding feature map according to the face image to be detected through each expert model; S43), obtaining a mixed feature map by weighted summing the n-dimensional weight vector output by the router and the feature maps output by the n expert models; S44), by flattening the mixed feature map into a feature vector, and capturing the dependency between the feature vectors through the trained Transformer module; the feature vectors are classified into two categories through the self-attention mechanism, and finally the probability of whether the face image to be detected is a deep fake image is obtained.

6. A method for detecting deep fake face images according to claim 5, characterized in that: In step S43), the calculation expression of the n-dimensional weight vector output by the weighted sum router Router and the n expert models is: In the formula, w i represents the i-th weight vector of the router, f i represents the feature map of the i-th expert model; F represents the weighted mixed feature map.

7. A deep fake face image detection system, characterized in that: The detection uses the method described in any one of claims 1 to 6 to detect deep fake images, and the system includes: A data acquisition module, used for acquiring facial images; Router module, used to extract high-level features of the face image to be detected; A multi-expert module is used to generate a corresponding number of feature maps according to the face image to be detected; The weighted summation module is used to perform weighted summation on the weight vector output by the Router module and the feature map output by the multi-expert module to obtain a mixed feature map; The binary classification module is used to perform binary classification based on the mixed feature map to detect whether the face image is forged.

8. A deep fake face image detection system according to claim 7, characterized in that: The Router module extracts high-level features of the face image to be detected by using the MobileNet V2 model, and outputs an n-dimensional weight vector as the weights of different experts through the MobileNetV2 model.

9. A deep fake face image detection system according to claim 7, characterized in that: The multi-expert module adopts n CNN models, and each CNN model generates a corresponding feature map according to the face image to be detected.

10. A deep fake face image detection system according to claim 7, characterized in that: The weighted summation module calculates the mixed feature map according to the following formula: In the formula, w i represents the i-th weight vector of the router, f i represents the feature map of the i-th expert model; F represents the weighted mixed feature map; The binary classification module flattens the mixed feature map into a feature vector, and uses the self-attention mechanism through the Transformer module to perform binary classification on the feature vector to obtain the probability of whether the face image to be detected is a deep fake image.

Images