A face deep forgery detection method and system based on multi-modal artifacts

Abstract

The present application relates to a kind of face depth forgery detection method and system based on multimodal artifact.The method includes extracting video frame from face depth forgery video frame by frame, and obtaining face region image from the extracted video frame;Multi-scale texture difference features, multi-level noise features and deep semantic features of the face region image are extracted using a dual-flow feature extraction network;Multi-scale texture difference features, deep semantic features and multi-level noise features are concatenated to perform multimodal feature fusion and determine the fusion features;Supervised contrast learning and classification of fusion features are performed based on a collaborative supervision strategy;The collaborative supervision strategy determines face depth forgery detection features by supervised contrast learning of fusion features;Classification is performed according to face depth forgery detection features and fusion feature vectors.The present application can effectively extract depth forgery detection features with generalization performance, improve intra-class detection accuracy and improve cross-database generalization detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia information forensics, and particularly relates to a face deepfake detection method and system based on multi-modal artifacts. BACKGROUND

[0002] With the rapid development of computer graphics, especially deep generative models, open source face deepfake technologies such as DeepFake, FaceSwap, Face2Face and NeuralTexture have emerged. On the one hand, the development of face deepfake technology promotes the development of the field of artistic creation and the film manufacturing industry. On the other hand, the spread of maliciously tampered face images or videos in Internet social media poses a huge threat to multimedia information security, causing serious harm such as invasion of personal privacy and forgery of legal evidence.

[0003] Existing face deepfake detection methods can be roughly divided into two categories: one class uses hand-crafted features to extract the differences between real faces and tampered faces. For example, X. Yang et al. published a paper entitled "Exposing deep fakes using inconsistent head poses" in IEEE International Conference on Acoustics, Speech and Signal Processing in 2019, which used hand-crafted features to extract head pose inconsistency artifacts as detection features. The features extracted by this method have clear purposes and strong interpretability, but the use of hand-crafted features introduces expert prior knowledge, and as the quality of tampered faces continues to improve, detection methods based on hand-crafted features cannot be generalized to improved face deepfake methods. The other class uses the structure of a neural network as a discriminator for face deepfake. For example, H. Zhao et al. published a paper entitled "Multi-attentional deepfake detection" in IEEE Conference on Computer Vision and Pattern Recognition in 2021, which introduced a multi-head attention mechanism into the EfficientNetB4 network to extract fine-grained difference features and achieved good performance in intra-class scene detection. This method relies on the powerful data fitting ability of neural networks to learn rich fake detection features, but it is prone to overfitting to the data distribution of the training data set and is difficult to generalize to unknown face deepfake methods.

[0004] In order to solve the problem of poor cross-database detection generalization performance of the current face deep fake detection method, a new face deep fake detection method or system is needed to effectively extract deep fake detection features with generalization performance, improve the in-class detection accuracy of the data set, and improve the generalization detection accuracy of the cross-database. SUMMARY

[0005] The purpose of the present application is to provide a face deep fake detection method and system based on multi-modal artifacts, which can effectively extract deep fake detection features with generalization performance, improve the in-class detection accuracy of the data set, and improve the generalization detection accuracy of the cross-database.

[0006] To achieve the above purpose, the present application provides the following scheme:

[0007] A face deep fake detection method based on multi-modal artifacts, comprising:

[0008] Extracting video frames from a face deep fake video frame by frame, and obtaining face region images from the extracted video frames;

[0009] Using a double-flow feature extraction network to extract multi-scale texture difference features, multi-level noise features and deep semantic features of the face region images; the double-flow feature extraction network uses an RGB branch and an SRM noise branch to extract features from the input face region images respectively; the double-flow feature extraction network includes a texture difference feature extraction module and a multi-level adaptive noise feature extraction module; the texture difference feature extraction module extracts multi-scale texture difference features based on different shallow feature maps of the RGB branch; the multi-level adaptive noise feature extraction module uses the deep semantic feature map of the RGB branch to interact with and enhance the noise feature extraction deep semantic features and multi-level noise features in the shallow noise feature map of the SRM noise branch;

[0010] Concatenating the multi-scale texture difference features, deep semantic features and multi-level noise features to perform multi-modal feature fusion and determine the fusion features;

[0011] Supervised contrast learning and classification of the fusion features based on a cooperative supervision strategy; the cooperative supervision strategy determines the face deep fake detection features by supervised contrast learning of the fusion features; and classifying according to the determined face deep fake detection features.

[0012] Optionally, an EfficientNetB4 pre-trained on ImageNet is used as the backbone network of the RGB branch and the SRM noise branch;

[0013] EfficientNetB4 comprises: a head convolution Conv1, seven layers, and a tail convolution connected in sequence; the RGB branch and the SRM noise branch share parameters within the same flow, while the parameters between flows are independent.

[0014] Optionally, the texture difference feature extraction module includes: different central difference convolutional layers.

[0015] Optionally, the inlet of the SRM noise branch includes an adaptive SRM filter layer.

[0016] Optionally, the multi-level noise feature extraction module includes: an RGB downsampling path, an SRM downsampling path, a cross-modal upsampling interactive enhancement path, and a lateral connection path.

[0017] Optionally, the loss function of the collaborative supervision strategy is:

[0018] L total =α*L ce +β*L sc ;

[0019] Among them, L total L is the loss function for the collaborative supervision strategy. sc Loss function for supervised contrastive learning L ce For the loss function of classification, α and β are both balanced weights, z is the standardized fusion feature, N is the batch size during training, and |N p | is the size of the positive sample set in the batch data, N. a For the negative sample set in the batch data, z i For anchor samples, z p For positive examples, z a For negative samples, τ∈R + It is the temperature coefficient, p j and y j , respectively, represent the probability that the model predicts the true value and the label of the input sample j.

[0020] A face deepfake detection system based on multimodal artifacts includes:

[0021] The face region image acquisition module is used to extract video frames frame by frame based on deepfake face videos and obtain face region images from the extracted video frames;

[0022] The feature extraction module is configured to extract multi-scale texture difference features, multi-level noise features and deep semantic features of the face region image by using a dual-stream feature extraction network; the dual-stream feature extraction network is configured to extract features of the input face region image by using an RGB branch and an SRM noise branch respectively; the dual-stream feature extraction network comprises a texture difference feature extraction module and a multi-level adaptive noise feature extraction module; the texture difference feature extraction module is configured to extract multi-scale texture difference features based on different shallow feature maps of the RGB branch; the multi-level adaptive noise feature extraction module is configured to interact and enhance noise features in the shallow noise feature map of the SRM noise branch by using deep semantic feature maps of the RGB branch and shallow noise feature maps in the SRM noise branch.

[0023] The multi-modal feature fusion module is configured to concatenate the multi-scale texture difference features, the deep semantic features and the multi-level noise features, perform multi-modal feature fusion, and determine fusion features.

[0024] The supervised contrast learning and classification module is configured to perform supervised contrast learning and classification of the fusion features based on a collaborative supervision strategy; the collaborative supervision strategy is configured to determine face deep fake detection features by performing supervised contrast learning on the fusion features; and classification is performed according to the determined face deep fake detection features.

[0025] Optionally, an EfficientNetB4 pre-trained on an ImageNet is used as a backbone network of the RGB branch and the SRM noise branch.

[0026] The EfficientNetB4 comprises a head convolution Conv1, seven levels and a tail convolution connected in sequence; the RGB branch and the SRM noise branch share parameters within the network, and the network parameters are independent between the branches.

[0027] According to the embodiments of the present application, the following technical effects are achieved:

[0028] This invention provides a method and system for detecting deepfake faces based on multimodal artifacts. The method extracts the video to be detected frame by frame from the dataset, extracts face region images from the video frames, and then inputs these face region images into a dual-stream feature extraction network to extract multi-scale texture difference features, deep semantic features, and multi-level noise features. Finally, a cooperative supervision strategy is used to perform supervised contrastive learning and classification on the multimodal fusion features. The dual-stream feature extraction network includes a texture difference feature extraction module and a multi-level noise extraction module, and can effectively extract deepfake detection features with generalization performance. The introduction of a cooperative supervision strategy enhances the inter-class separability and intra-class compressibility in the multimodal fusion feature space. This improves the intra-class detection accuracy of existing deepfake face detection methods while also improving the generalization detection accuracy across different datasets. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A schematic diagram of the process for a face depth forgery detection method based on multimodal artifacts provided by the present invention;

[0031] Figure 2 This is a typical diagram of a deepfake face process;

[0032] Figure 3 This invention provides a framework diagram for a face depth forgery detection method based on multimodal artifacts.

[0033] Figure 4 This is a schematic diagram of the texture difference feature extraction module;

[0034] Figure 5 This is a schematic diagram of a multi-level noise feature extraction module;

[0035] Figure 6 This is a schematic diagram of a face depth forgery detection system based on multimodal artifacts provided by the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] The application aims to provide a face deepfake detection method and system based on multi-modal artifacts, which can effectively extract deepfake detection features with generalization performance, improve the in-class detection accuracy of the data set, and improve the generalization detection accuracy across libraries.

[0038] In order to make the above-mentioned purposes, characteristics and advantages of the application more obvious and easy to understand, the application will be further described in detail below in combination with the drawings and specific embodiments.

[0039] As shown in the figure, Figure 2 The typical face deepfake process includes three processing stages. In the first stage, a face detection model such as MTCNN or Dlib is used to detect and recognize the face area in the face deepfake video frame, and the face area is extracted according to the face feature points. In the second stage, the deepfake face area image is rendered by a 3D parameter model based on computer graphics or generated by a deep generative model. In this stage, different face tampering algorithms will produce different texture patterns from the original face image. In the third stage, the deepfake face area image generated in the second stage is fused with the background area in the original image after a series of post-processing steps such as color correction and affine transformation to obtain the final face deepfake video frame. The fusion processing step in this stage will expose the noise features that are inconsistent between the face area and the background area. In summary, the texture difference and area noise inconsistency are complementary artifacts inherent in the typical face deepfake process.

[0040] Figure 1 A face deepfake detection method based on multi-modal artifacts provided by the application, Figure 3 A face deepfake detection method based on multi-modal artifacts provided by the application, Figure 1 and Figure 3 The face deepfake detection method based on multi-modal artifacts provided by the application includes:

[0041] S101, based on face deepfake video, extract video frames frame by frame, and obtain face area images from the extracted video frames.

[0042] Currently, most open-source face deepfake data sets are video data sets. First, the video data set needs to be extracted frame by frame to obtain face deepfake images. Then, for all real and fake images, a face detection model Dlib is used to detect and extract the face area. To further expose the inconsistency of the noise area, the face area is expanded by 1.3 times in width and height, and finally the size of the cropped face is uniformly adjusted to 256*256.

[0043] S102, a multi-scale texture difference feature, a multi-level noise feature and a deep semantic feature of the face region image are extracted by using a double-flow feature extraction network; the double-flow feature extraction network extracts features of the input face region image by using an RGB branch and an SRM noise branch respectively; the double-flow feature extraction network comprises a texture difference feature extraction module and a multi-level adaptive noise feature extraction module; the texture difference feature extraction module extracts the multi-scale texture difference feature based on different shallow feature maps of the RGB branch; the multi-level adaptive noise feature extraction module extracts deep semantic features and multi-level noise features by interacting the deep semantic feature map of the RGB branch with the shallow noise feature map in the SRM noise branch and enhancing the noise feature in the shallow noise feature map of the SRM noise branch.

[0044] EfficientNetB4 pre-trained on ImageNet is used as the backbone network of the RGB branch and the SRM noise branch.

[0045] EfficientNetB4 comprises a head convolution Conv1, seven levels Layer1-Layer7 and a tail convolution Conv2 connected in sequence; the RGB branch and the SRM noise branch share parameters within the network and are independent of each other in terms of network parameters.

[0046] The texture difference feature extraction module comprises different center difference convolution layers. The texture difference feature extraction module extracts the multi-scale texture difference feature based on different shallow feature maps of the RGB branch. Specifically, as shown in Figure 4 , the feature map of the input RGB branch is sequentially subjected to two different center difference convolution layers to obtain a texture feature map, wherein the center difference convolution layer combines pixel intensity information and gradient information to learn the texture information in the feature map, and is expressed as:

[0047]

[0048] In the above formula, p c represents the current position in the receptive field input and output feature map F o and F i , p n enumerates the pixels in the current local receptive field range N, and θ is a balance parameter of the pixel intensity and the pixel gradient information, which is set to θ=0.7 in the embodiment. In the center difference convolution, K represents the number of convolution kernels, S represents the step length, and G represents the grouping number of input channels and output channels, which is set to M=16 in the embodiment. Subsequently, the texture feature map is subjected to Gram matrix coding and vector flattening operation to obtain the texture feature TDF. The Gram matrix calculation expression is:

[0049]

[0050] In the above formula is the vectorized texture feature map i = 1, 2…M. The texture difference feature extraction module is applied to the shallow feature maps output by Conv1, Layer1, Layer2 in the RGB branch to obtain multi-scale texture difference features TDF1, TDF2, TDF3.

[0051] The input of the SRM noise branch contains an adaptive SRM filtering layer. The adaptive SRM filtering layer overcomes the limited noise representation ability of traditional fixed kernel parameter filters and adaptively learns the SRM kernel parameters through network learning. The calculation expression of the adaptive SRM filtering layer is:

[0052]

[0053] where W(0,0) represents the parameters of the center of the adaptive SRM filtering kernel, W(m,n) represents the kernel parameters outside the center of the adaptive SRM filtering kernel, and R represents the residual map obtained by performing convolution operation on the input image X and the adaptive SRM filter. The learnable SRM filtering layer contains 3 learnable SRM convolution kernels of 5*5, and takes the 3 filters in the classic SRM algorithm as the initialization parameters. Then it is expanded to a 3*3*5*5 SRM convolution kernel in the second dimension to synchronize the input and output resolution of 3x256x256.

[0054] As Figure 4 shown, the multi-level noise feature extraction module includes an RGB downsampling path, an SRM downsampling path, a cross-modal upsampling interaction enhancement path, and a lateral connection path. The level in the backbone network that generates a feature map of the same resolution is defined as a stage, and the output feature map of the last layer of each stage is defined as a hierarchy. The outputs of Layer1, Layer2, Layer3, Layer5, and Layer7 in the backbone network EfficientNetB4 are five main hierarchical feature maps. The 1x1 convolution layer in the lateral connection path is used to adjust and unify the channel number of each scale feature. The 3x3 convolution kernel is used to eliminate the aliasing effect in the upsampling process. Since the deep feature maps in the RGB branch contain rich semantic features, the subtle noise features in the SRM noise branch are easy to disappear in the downsampling process, therefore, the deep semantic features T5, T4 and the multi-level noise features T3, T2, T1 are further extracted by using the deep semantic feature maps output by Layer5 and Layer7 in the RGB branch to interact with the shallow noise features output by Layer1, Layer2, and Layer3 in the SRM branch;

[0055] S103, concatenate the multi-scale texture difference features, the deep semantic features, and the multi-level noise features to perform multi-modal feature fusion and determine the fused features FG = concat(TDF1, TDF2, TDF3, T1, ··· T5).

[0056] In S104, supervised contrast learning and classification of the fusion feature are performed based on a collaborative supervision strategy. The collaborative supervision strategy is used to determine a face deepfake detection feature with generalization performance for supervised contrast learning of the fusion feature. The supervised contrast learning encourages the network to further aggregate the features of the same class and further separate the features of different classes, thereby learning a feature expression with better generalization performance. Classification is performed based on the determined face deepfake detection feature, and a cross-entropy loss function is used as the supervision.

[0057] The loss function of the collaborative supervision strategy is:

[0058] L total = α * L ce + β * L sc ;

[0059] wherein L total is the loss function of the collaborative supervision strategy, L sc is the loss function of the supervised contrast learning L ce is the loss function of the classification, α and β are both balance weights, z is the standardized fusion feature, N is the size of a batch of data in the training process, |N p | is the size of the positive sample set in the batch data, N a is the negative sample set in the batch data, z i is the anchor sample, z p is the positive sample, z a is the negative sample, τ ∈ R + is a temperature coefficient, p j and y j are the probability of the model predicting the real and the label of the input sample j, respectively. Specifically, α = 1, β = 0.5, and τ = 0.1.

[0060] In this embodiment, the training data set is a large face deepfake detection data set FaceForensic++, and the cross-library general detection data set uses Celeb-DF and DFDC. Among them, FaceForensic++ contains 1000 real videos and 4000 videos of 4 kinds of DeepFake, FaceSwap, Face2Face, and NeuralTexture in the non-compressed C0, low-compressed C23, and high-compressed C40 scenes. Celeb-DF contains 408 real videos and 795 fake videos. DFDC includes 100000 real videos and 19154 fake videos.

[0061] In the intra-class detection of the FaceForensic++ face deepfake dataset, the test results are shown in Table 1, and for the detection of face deepfake in a real scene, the low compression C23 and high compression C40 settings are selected for the test dataset, the selected evaluation indexes ACC represent the accuracy, and AUC represents the area under the curve, the above two evaluation indexes can fully evaluate the classification ability of the neural network classifier, and it can be seen from Table 1 that compared with the advanced classification network Xception, EfficientNetB4, and the advanced face fake detection methods SPSL and DFFD, the method proposed in the embodiment of the application can improve the detection ability of deepfake faces with different compression rates.

[0062] Table 1

[0063]

[0064] In the cross-library generalization performance detection of face deepfake detection, the embodiment of the application is trained on FaceForensic++, and the generalization performance is tested in the Celeb-DF and DFDC datasets. The test results are shown in Table 2, and advanced generalization detection performance is obtained on the Celeb-DF and DFDC datasets.

[0065] Table 2

[0066]

[0067]

[0068] Figure 6 The structure diagram of the face deepfake detection system based on multi-modal artifacts provided by the application is shown in Figure 6 The face deepfake detection system based on multi-modal artifacts provided by the application comprises:

[0069] The face region image acquisition module 601 is configured to extract video frames frame by frame based on a face deepfake video, and obtain a face region image from the extracted video frames.

[0070] The feature extraction module 602 is configured to extract multi-scale texture difference features, multi-level noise features and deep semantic features of the face region image by using a double-stream feature extraction network; the double-stream feature extraction network extracts features of the input face region image by using an RGB branch and an SRM noise branch respectively; the double-stream feature extraction network comprises a texture difference feature extraction module and a multi-level adaptive noise feature extraction module; the texture difference feature extraction module extracts multi-scale texture difference features based on different shallow feature maps of the RGB branch; the multi-level adaptive noise feature extraction module extracts deep semantic features and multi-level noise features by interacting the deep semantic feature map of the RGB branch with the shallow noise feature map in the SRM noise branch and enhancing the noise features in the shallow noise feature map of the SRM noise branch.

[0071] The multi-modal feature fusion module 603 is configured to concatenate the multi-scale texture difference features, the deep semantic features and the multi-level noise features, perform multi-modal feature fusion, and determine fusion features.

[0072] The supervised contrast learning and classification module 604 is configured to perform supervised contrast learning and classification of the fusion features based on a cooperative supervision strategy; the cooperative supervision strategy is used to determine face deep fake detection features by performing supervised contrast learning on the fusion features; and classification is performed according to the determined face deep fake detection features.

[0073] The EfficientNetB4 pre-trained on the ImageNet is used as the backbone network of the RGB branch and the SRM noise branch.

[0074] The EfficientNetB4 comprises a head convolution Conv1, seven levels and a tail convolution connected in sequence; the RGB branch and the SRM noise branch share parameters within the network, and the network parameters are independent between the branches.

[0075] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0076] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A face deepfake detection method based on multi-modal artifacts, characterized in that, The method comprises the steps of: Based on face deep fake video frame by frame extraction video frame, from the extracted video frame to obtain face region image; A dual-stream feature extraction network is used to extract multi-scale texture difference features, multi-level noise features, and deep semantic features from the face region image. The dual-stream feature extraction network uses an RGB branch and an SRM noise branch to extract features from the input face region image. The dual-stream feature extraction network includes a texture difference feature extraction module and a multi-level adaptive noise feature extraction module. The texture difference feature extraction module extracts multi-scale texture difference features based on different shallow feature maps from the RGB branch. The multi-level adaptive noise feature extraction module utilizes the deep semantic feature map from the RGB branch and the shallow noise feature map from the SRM noise branch. The noise feature extraction module extracts deep semantic features and multi-level noise features from the shallow noise feature map of the SRM noise branch through interaction and enhancement. The multi-level adaptive noise feature extraction module includes: an RGB downsampling path, an SRM downsampling path, a cross-modal upsampling interactive enhancement path, and a lateral connection path. In the backbone network, the layers that generate feature maps of the same resolution are defined as a stage, and the output feature map of the last layer of each stage is defined as a hierarchical structure. Layers 1, 2, 3, 5, and 7 in the backbone network output five main hierarchical feature maps. In the lateral connection path... The convolutional layers are used to adjust and unify the number of channels for each scale feature; The convolutional kernels are used to eliminate aliasing during the upsampling process; the deep semantic features are further extracted by interacting and enhancing the deep semantic features with the deep semantic feature maps output from Layer 5 and Layer 7 of the RGB branch and the shallow noise features output from Layer 1, Layer 2, and Layer 3 of the SRM branch. , and multi-level noise characteristics , and ; The multi-modal feature fusion module is used for concatenating the multi-scale texture difference feature, the deep semantic feature and the multi-level noise feature, performing multi-modal feature fusion, and determining the fusion feature. Based on the collaborative supervision strategy, the supervised contrast learning and classification of the fusion feature are performed; the collaborative supervision strategy is to determine the face deep fake detection feature by supervised contrast learning on the fusion feature; and classification is performed according to the determined face deep fake detection feature.

2. The method of claim 1, wherein the method is based on multimodal artifacts. EfficientNetB4 pre-trained on ImageNet is used as the backbone network of the RGB branch and the SRM noise branch; EfficientNetB4 comprises a head convolution Conv1, 7 levels and a tail convolution connected in sequence; the RGB branch and the SRM noise branch share parameters within the network, and the network parameters between the branches are independent.

3. The method of claim 2, wherein the method is based on multimodal artifacts. The texture difference feature extraction module comprises different center difference convolution layers.

4. The method of claim 1, wherein the method is based on multimodal artifacts. The entrance of the SRM noise branch comprises an adaptive SRM filtering layer.

5. The method of claim 1, wherein the method is based on multimodal artifacts for face deepfake detection. The loss function of the collaborative supervision strategy is: ； where, is the loss function for the supervised policy, is the loss function for the supervised contrastive learning , is the loss function for the classification, , α and β are both balance weights, z is the normalized fused feature, N is the batch size in the training process, is the size of the positive sample set in the batch data, N a is the negative sample set in the batch data, z i is the anchor sample, z p is the positive sample, z a is the negative sample, is the temperature coefficient, and are the probability of the model predicting as real and the label of the input sample j , respectively.

6. A face deepfake detection system based on multi-modal artifacts, applied to the face deepfake detection method based on multi-modal artifacts of any one of claims 1-5, characterized in that, The method comprises the steps of: A face region image acquisition module is used for extracting video frames from face deep fake videos frame by frame, and obtaining face region images from the extracted video frames; A feature extraction module is used for extracting multi-scale texture difference features, multi-level noise features and deep semantic features of the face region images by using a double-flow feature extraction network; the double-flow feature extraction network uses an RGB branch and an SRM noise branch to extract features from input face region images; the double-flow feature extraction network comprises a texture difference feature extraction module and a multi-level adaptive noise feature extraction module; the texture difference feature extraction module extracts multi-scale texture difference features based on different shallow feature maps of the RGB branch; the multi-level adaptive noise feature extraction module extracts deep semantic features and multi-level noise features by interacting and enhancing the noise features in the shallow noise feature maps of the SRM noise branch using the deep semantic feature maps of the RGB branch and the shallow noise feature maps in the SRM noise branch; A multi-modal feature fusion module is used for concatenating the multi-scale texture difference feature, the deep semantic feature and the multi-level noise feature, performing multi-modal feature fusion, and determining the fusion feature. A supervised contrast learning and classification module is used for performing supervised contrast learning and classification of the fusion feature based on a collaborative supervision strategy; the collaborative supervision strategy is to determine the face deep fake detection feature by supervised contrast learning on the fusion feature; and classification is performed according to the determined face deep fake detection feature.

7. The face deepfake detection system based on multi-modal artifacts according to claim 6, wherein, EfficientNetB4 pre-trained on ImageNet is used as the backbone network of the RGB branch and the SRM noise branch; EfficientNetB4 comprises a head convolution Conv1, 7 levels and a tail convolution connected in sequence; the RGB branch and the SRM noise branch share parameters within the network, and the network parameters between the branches are independent.

Images