A deepfake detection method based on spatiotemporal attention-guided fusion

Abstract

The present invention relates to the field of face recognition technology and discloses a deep fake detection method based on spatiotemporal attention guidance fusion. By using spatial domain and temporal feature extraction branches based on the Xception network, an attention mutual guidance module is designed to promote the two branches to adaptively focus on more subtle forgery traces and eliminate the interference of irrelevant information. At the same time, a cross-modal attention fusion module is designed to filter important temporal information and integrate it into spatial domain features, capturing the inconsistency problems existing in most forged videos and improving the model's detection ability for unknown forgery methods.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a deep fake detection method based on spatiotemporal attention guided fusion. Background Art

[0002] With the development of deepfake technology, it has become very easy to fake facial videos. The spread of these forged products has a huge negative impact on the country, society and individuals. More and more researchers are beginning to pay attention to deepfake detection technology.

[0003] Yang et al. noted that forgeries are created by splicing synthetic facial regions onto the original image, resulting in 3D head pose errors. They first experimentally verified this phenomenon and then constructed a classifier based on head pose inconsistencies to evaluate the head pose estimated by facial landmarks extracted by Dlib, thereby detecting forged videos. Finally, they tested their model on the UADFV dataset, achieving an AUC of 0.974 at the video level. Liu et al. observed that most existing forgery methods involve upsampling operations, and that the accumulation of upsampling operations leads to significant changes in the frequency domain, particularly the phase spectrum. Therefore, they proposed a method based on spatial phase shallow learning (SPSL). This method combines the spatial domain and phase spectrum to capture the upsampling artifacts generated during the forgery process, thereby improving the portability of face forgery detection and theoretically demonstrating the effectiveness of using the phase spectrum. Furthermore, noting that local texture information is more important than high-level semantic information in face forgery detection, they reduced the number of network layers to reduce the receptive field, allowing the network to suppress high-level features and focus on local regions. Zheng et al. observed that most video forgery methods ignore the connections between frames, resulting in temporal inconsistencies in the forged videos. By exploring how to fully utilize temporal consistency for forged video detection, an end-to-end framework is proposed. The framework mainly consists of two stages: the first stage is the Fully Temporal Convolution Network (FTCN), which modifies the spatial convolution kernel size to 1 while keeping the temporal convolution kernel size unchanged. This design helps to extract temporal features and improve generalization ability; the second stage is the temporal attention network, which is used to explore the consistency of long sequences in the video.

[0004] Most existing forgery detection methods target intra-database detection, using the same dataset for training and testing. This approach achieves good results, but generalization remains a major challenge in face forgery detection. Different forgery techniques generate varying data distributions. While some methods can achieve high detection rates within a database, they often experience significant performance degradation in cross-database scenarios, limiting the detector's general applicability.

[0005] The prior art discloses a forged face detection method based on multi-channel dual-stream attention. The method includes the following steps: preprocessing forged face video data to generate multi-channel information to obtain a multi-channel face information dataset; constructing a detection model based on a dual-stream attention feature extractor and a classifier; inputting the obtained multi-channel face information dataset into the detection model for training to obtain a detection model that can identify genuine faces; and inputting the generated multi-channel information into the detection model for testing to obtain a result indicating whether the face video is genuine or fake. This prior art suffers from performance degradation in cross-database scenarios, affecting detection effectiveness. Summary of the Invention

[0006] The purpose of this invention is to provide a deep fake detection method based on spatiotemporal attention guided fusion to solve the problem in the prior art that performance will be degraded in cross-database scenarios, affecting the detection effect.

[0007] To achieve the above objectives, the present invention provides a deep fake detection method based on spatiotemporal attention guided fusion, comprising:

[0008] S1. Obtain a dataset, obtain the residuals between adjacent frames of the dataset, construct a residual graph, and use video processing tools to obtain the I frame of the dataset;

[0009] S2. Use the DLIB face extraction model to extract the residual image and the face area of the I frame, form a new dataset based on the face area, and divide the new dataset into a training set and a test set;

[0010] S3. Set the resolution of the training set and the test set, and perform data augmentation on the training set and the test set to obtain the augmented training set and the test set;

[0011] S4. The face area and the residual map of the I frame are convolved by the convolution block to obtain the first feature map and the second feature map respectively. The branch for convolving the face area of the I frame is the spatial domain branch, and the branch for convolving the residual map is the residual branch. The first feature map and the second feature map are input into the cross-modal attention fusion module to obtain the temporal features of attention fusion. The temporal features are merged into the spatial domain branch to obtain the feature map input into the next layer of convolution. The construction process of the cross-modal attention fusion module is as follows: the first feature map is convolved by the convolution block to obtain the first query, the second feature map is convolved by two different convolution blocks to obtain the second query and the time domain feature value, the similarity of the two modalities of the first query and the second query is calculated to obtain the similarity value, and the attention map is obtained by combining the similarity value and the weight matrix;

[0012] S5. Obtain feature representations of the residual image and the face area of the I frame, which are the first feature representation and the second feature representation respectively. Perform average pooling and maximum pooling on the first feature representation to obtain two first 2D maps. Perform average pooling and maximum pooling on the second feature representation to obtain two second 2D maps. Concatenate the two first 2D maps and the two second 2D maps and convolve them through a standard convolutional layer to obtain two 2D spatial attention maps. Normalize the two 2D spatial attention maps to obtain two final attention maps. Use the two final attention maps to represent the RGB features and the inter-frame residual features, respectively, to obtain the attention mutual guidance module.

[0013] S6. Input the training set and test set into the two-stream network model for learning, set the learning batch and learning rate, and obtain the pre-trained weights. The two-stream network contains two branches, each of which uses the Xception network as the backbone network for time domain and spatial domain feature extraction. Input the pre-trained weights into the attention mutual guidance module for training, and obtain the trained attention mutual guidance module.

[0014] S7. Adjust the learning rate and repeat step S6 to obtain multiple sets of model weights. Select an evaluation criterion, test the obtained multiple sets of model weights, and select the model weight with the best cross-database effect as the final result of the cross-database detection.

[0015] Preferably, in step S1, the datasets are FaceForensics++ and Celeb-DF.

[0016] Preferably, in step S1, the video processing tool is FFmpeg.

[0017] Preferably, in step S3, data enhancement includes random cropping, random horizontal inversion and Gaussian blur.

[0018] Preferably, in step S4, the formula for obtaining the feature map input to the next convolution layer is as follows:

[0019]

[0020] F′=F T +F

[0021] Where ft represents the flattening operation, K represents the query obtained in the spatial domain features, and K R represents the query obtained in the time domain feature, C represents the similarity value, W represents the weight matrix, V R represents the value obtained in the time domain feature, A represents the attention map, F T represents the fused temporal features, represents matrix multiplication, and softmax represents the normalized exponential function.

[0022] Preferably, in step S5, the size of the first feature representation and the second feature representation is C×H×W, where C is the number of channels, H is the height, and W is the width. The formulas for the two final attention maps are as follows:

[0023] F′=F1×Sig(Conv(Concat(Avg(F r ),Max(F r )))+F1

[0024] F′ r =F r ×Sig(Conv(Concat(Avg(F1),Max(F1)))+F r

[0025] Where F1 is the first feature representation, Fr is the second feature representation, Sig is the sigmod activation function, Conv is a 1×1 convolution operation, concat represents the concatenation of two vectors, Avg is the average pooling operation, and Max is the maximum pooling operation.

[0026] Preferably, in step S5, the two 2D spatial attention maps are normalized by a sigmoid function to obtain two final attention maps.

[0027] Preferably, in step S6, the fusion mode of inputting the training set and the test set into the two-stream network model for learning is a cross-modal attention fusion mode, and the loss function is a cross entropy loss function, and the formula is as follows:

[0028]

[0029] The number of samples y is the true label value, Represents the predicted probability value.

[0030] Preferably, in step S7, the evaluation criteria are selected as ACC and AUC.

[0031] Preferably, in step S7, the model weight with the best cross-database effect is selected as the final result of the cross-database detection. Specifically, the model weight with the best cross-method effect is selected as the final result of the cross-method detection after training on FaceForensics++ and testing on Celeb-DF.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] By using the spatial domain and temporal feature extraction branches based on the Xception network, an attention mutual guidance module is designed to promote the two branches to adaptively focus on more subtle traces of forgery and eliminate the interference of irrelevant information. At the same time, a cross-modal attention fusion module is designed to filter important temporal information and integrate it into the spatial domain features, capturing the inconsistencies existing in most forged videos and improving the model's detection ability for unknown forgery methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of a deep fake detection method based on spatiotemporal attention guided fusion according to an embodiment of the present invention;

[0035] Figure 2 is a face region map of an I frame and a residual map according to an embodiment of the present invention;

[0036] Figure 3 This is a diagram showing the effect of the data enhancement strategy according to an embodiment of the present invention;

[0037] Figure 4 is a structural diagram of a cross-modal attention fusion module according to an embodiment of the present invention;

[0038] Figure 5 This is a structural diagram of the attention mutual guidance module according to an embodiment of the present invention;

[0039] Figure 6 This is a structural diagram of a dual-stream network with a cross-modal attention fusion module added in an embodiment of the present invention;

[0040] Figure 7 This is a structural diagram of a dual-stream network adding attention mutual guidance module according to an embodiment of the present invention;

[0041] Figure 8 Schematic diagram of the training process of an embodiment of the present invention;

[0042] Figure 9 Schematic diagram of the test process of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0044] In the description of the present invention, it should be noted that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "back," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate the description of the present invention and simplify the description. They are not intended to indicate or imply that the devices or components referred to must have a specific orientation, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0045] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0046] Furthermore, in the description of the present invention, unless otherwise specified, “plurality” means two or more.

[0047] Example 1

[0048] like Figure 1-5 As shown, a deep fake detection method based on spatiotemporal attention guided fusion in a preferred embodiment of the present invention includes:

[0049] S1. Obtain a dataset, obtain the residuals between adjacent frames of the dataset, construct a residual graph, and use video processing tools to obtain the I frame of the dataset;

[0050] S2. Use the DLIB face extraction model to extract the residual image and the face area of the I frame, form a new dataset based on the face area, and divide the new dataset into a training set and a test set;

[0051] S3. Set the resolution of the training set and the test set, and perform data augmentation on the training set and the test set to obtain the augmented training set and the test set;

[0052] S4. The face area and the residual map of the I frame are convolved by the convolution block to obtain the first feature map and the second feature map respectively. The branch for convolving the face area of the I frame is the spatial domain branch, and the branch for convolving the residual map is the residual branch. The first feature map and the second feature map are input into the cross-modal attention fusion module to obtain the temporal features of attention fusion. The temporal features are merged into the spatial domain branch to obtain the feature map input into the next layer of convolution. The construction process of the cross-modal attention fusion module is as follows: the first feature map is convolved by the convolution block to obtain the first query, the second feature map is convolved by two different convolution blocks to obtain the second query and the time domain feature value, the similarity of the two modalities of the first query and the second query is calculated to obtain the similarity value, and the attention map is obtained by combining the similarity value and the weight matrix;

[0053] like Figure 4 As shown in the figure, the attention mechanism is used to generate time domain and spatial domain attention maps, and then the effective information in the time series is filtered out. Finally, multiple sets of convolution operations are used to further adaptively extract the significant information of the time series branches to improve the model detection accuracy.

[0054] S5. Obtain feature representations of the residual image and the face area of the I frame, which are the first feature representation and the second feature representation respectively. Perform average pooling and maximum pooling on the first feature representation to obtain two first 2D maps. Perform average pooling and maximum pooling on the second feature representation to obtain two second 2D maps. Concatenate the two first 2D maps and the two second 2D maps and convolve them through a standard convolutional layer to obtain two 2D spatial attention maps. Normalize the two 2D spatial attention maps to obtain two final attention maps. Use the two final attention maps to represent the RGB features and the inter-frame residual features, respectively, to obtain the attention mutual guidance module.

[0055] like Figure 5 As shown in the figure, a face image contains rich pixel information, while forged content only accounts for a small part. How to filter out effective information is very critical. The attention mutual guidance module uses the mutual guidance of time and space domains to gradually guide the other branch to focus on significant forgery traces, thereby improving the model's detection ability.

[0056] S6. Input the training set and test set into the two-stream network model for learning, set the learning batch and learning rate, and obtain the pre-trained weights. The two-stream network contains two branches, each of which uses the Xception network as the backbone network for time domain and spatial domain feature extraction. Input the pre-trained weights into the attention mutual guidance module for training, and obtain the trained attention mutual guidance module.

[0057] S7. Adjust the learning rate and repeat step S6 to obtain multiple sets of model weights. Select an evaluation criterion, test the obtained multiple sets of model weights, and select the model weight with the best cross-database effect as the final result of the cross-database detection.

[0058] The entire network model structure is as follows Figure 7 As shown, Table 1 and Table 2 show the detection results of the network in cross-method and cross-library situations, and Xception, EfficientNet and Mulit-Attention models are selected for comparison, where HQ represents the compressed FaceForensics++ dataset. It can be seen that the model exceeds the existing mainstream technology in most cases, confirming the effectiveness of the present invention. Table 3 shows that the present invention still has good results compared with mainstream technology under in-library (i.e., training and testing on FaceForensics++) detection, where c23 represents the high-quality FaceForensics++ dataset. Table 4 shows the detection results under different combinations, where AMG1 represents adding one attention mutual guidance module, AMG2 represents adding two, CMA1 represents adding one cross-modal attention fusion module, and CMA2 represents adding two, indicating that the attention mutual guidance module and cross-modal attention fusion module proposed in the present invention are both effective.

[0059] Table 1. Cross-method detection results

[0060]

[0061]

[0062] Table 2. Cross-library detection results

[0063]

[0064] Table 3. In-store testing results

[0065]

[0066] Table 4. Detection results of different combinations

[0067]

[0068] By using the spatial domain and temporal feature extraction branches based on the Xception network, an attention mutual guidance module is designed to promote the two branches to adaptively focus on more subtle traces of forgery and eliminate the interference of irrelevant information. At the same time, a cross-modal attention fusion module is designed to filter important temporal information and integrate it into the spatial domain features, capturing the inconsistencies existing in most forged videos and improving the model's detection ability for unknown forgery methods.

[0069] Example 2

[0070] In step S1, the datasets are FaceForensics++ and Celeb-DF.

[0071] In this embodiment, the resolution of the training set and the test set is set to 280×280, and then data enhancement is performed.

[0072] In step S1, the video processing tool is FFmpeg.

[0073] In step S2, the ratio of the training set to the test set is 80% and 20%.

[0074] In step S3, data augmentation includes random cropping, random horizontal inversion and Gaussian blur.

[0075] In this embodiment, the feature maps F and F of the spatial domain branch and the residual branch are first obtained. R , and then input the feature map into the cross-modal attention fusion module to obtain the attention fusion temporal feature F T , and finally merged into the spatial domain branch. Specifically, the feature map F, after passing through the convolution block, obtains the query K. Feature map F R After two different convolution blocks, the query K is obtained R Sum V R Then by querying K and K R Calculate the similarity of the two modalities to get C, and combine C with the weight matrix W to get the attention map A. The next step is the attention fusion process, through A and V R Further refine the time series features and obtain F T Finally, the extracted features are fused into the spatial domain (branch) to obtain the feature map F′ that is input to the next layer of convolution.

[0076] In step S4, the formula for obtaining the feature map input to the next layer of convolution is as follows:

[0077]

[0078] F′=F T +F

[0079] Where ft represents the flattening operation, K represents the query obtained in the spatial domain features, and K R represents the query obtained in the time domain feature, C represents the similarity value, W represents the weight matrix, V R represents the value obtained in the time domain feature, A represents the attention map, F T represents the fused temporal features, represents matrix multiplication, and softmax represents the normalized exponential function.

[0080] In step S5, the size of the first feature representation and the second feature representation is C×H×W, where C is the number of channels, H is the height, and W is the width. The formulas for the two final attention maps are as follows:

[0081] F′=F1×Sig(Conv(Concat(Avg(F r ),Max(F r )))+F1

[0082] F′ r =F r ×Sig(Conv(Concat(Avg(F1),Max(F1)))+F r

[0083] Where F1 is the first feature representation, Fr is the second feature representation, Sig is the sigmod activation function, Conv is a 1×1 convolution operation, concat represents the concatenation of two vectors, Avg is the average pooling operation, and Max is the maximum pooling operation.

[0084] In step S5, the two 2D spatial attention maps are normalized by the sigmoid function to obtain two final attention maps.

[0085] Example 3

[0086] like Figure 8-9 As shown, in step S6, the training set and the test set are input into the two-stream network model for learning. The fusion method is the cross-modal attention fusion method, and the loss function is the cross entropy loss function, which is as follows:

[0087]

[0088] The number of samples y is the true label value, Represents the predicted probability value.

[0089] In this embodiment, the learning batch is set to 16 and the learning rate is set to 0.0002 to obtain the pre-training weights.

[0090] In step S7, the evaluation criteria are selected as ACC and AUC, where ACC is the accuracy index and AUC is the area under the ROC curve index.

[0091] In step S7, the model weight with the best cross-database effect is selected as the final result of the cross-database detection. Specifically, the model weight with the best cross-method effect is selected as the final result of the cross-method detection after training on FaceForensics++ and testing on Celeb-DF.

[0092] In this example, the obtained model weights were tested using AUC and ACC as evaluation criteria, and the model weight with the best cross-database performance (i.e., trained on FaceForensics++ and tested on Celeb-DF) was selected as the final result of the cross-database test. The model weight with the best cross-method performance (one of the four forgery methods was selected as the training set, and the other three methods were selected as the test set) was selected as the final result of the cross-method test.

[0093] In this embodiment, after training the attention mutual guidance module, the entire network model needs to be fine-tuned, and the learning rate is adjusted to one tenth of the original every 10 rounds, and the training is completed after 30 rounds.

[0094] The working process of the present invention is:

[0095] S1. Obtain a dataset, obtain the residuals between adjacent frames of the dataset, construct a residual graph, and use video processing tools to obtain the I frame of the dataset;

[0096] S2. Use the DLIB face extraction model to extract the residual image and the face area of the I frame, form a new dataset based on the face area, and divide the new dataset into a training set and a test set;

[0097] S3. Set the resolution of the training set and the test set, and perform data augmentation on the training set and the test set to obtain the augmented training set and the test set;

[0098] S4. The face area and the residual map of the I frame are convolved by the convolution block to obtain the first feature map and the second feature map respectively. The branch for convolving the face area of the I frame is the spatial domain branch, and the branch for convolving the residual map is the residual branch. The first feature map and the second feature map are input into the cross-modal attention fusion module to obtain the temporal features of attention fusion. The temporal features are merged into the spatial domain branch to obtain the feature map input into the next layer of convolution. The construction process of the cross-modal attention fusion module is as follows: the first feature map is convolved by the convolution block to obtain the first query, the second feature map is convolved by two different convolution blocks to obtain the second query and the time domain feature value, the similarity of the two modalities of the first query and the second query is calculated to obtain the similarity value, and the attention map is obtained by combining the similarity value and the weight matrix;

[0099] S5. Obtain feature representations of the residual image and the face area of the I frame, which are the first feature representation and the second feature representation respectively. Perform average pooling and maximum pooling on the first feature representation to obtain two first 2D maps. Perform average pooling and maximum pooling on the second feature representation to obtain two second 2D maps. Concatenate the two first 2D maps and the two second 2D maps and convolve them through a standard convolutional layer to obtain two 2D spatial attention maps. Normalize the two 2D spatial attention maps to obtain two final attention maps. Use the two final attention maps to represent the RGB features and the inter-frame residual features, respectively, to obtain the attention mutual guidance module.

[0100] S6. Input the training set and test set into the two-stream network model for learning, set the learning batch and learning rate, and obtain the pre-trained weights. The two-stream network contains two branches, each of which uses the Xception network as the backbone network for time domain and spatial domain feature extraction. Input the pre-trained weights into the attention mutual guidance module for training, and obtain the trained attention mutual guidance module.

[0101] S7. Adjust the learning rate and repeat step S6 to obtain multiple sets of model weights. Select an evaluation criterion, test the obtained multiple sets of model weights, and select the model weight with the best cross-database effect as the final result of the cross-database detection.

[0102] In summary, the embodiments of the present invention provide a deep fake detection method based on spatiotemporal attention guidance fusion. By using the spatial domain and temporal feature extraction branches based on the Xception network, an attention mutual guidance module is designed to promote the two branches to adaptively focus on more subtle traces of fakes and eliminate the interference of irrelevant information. At the same time, a cross-modal attention fusion module is designed to screen important temporal information and integrate it into the spatial domain features, capturing the inconsistency problems existing in most fake videos and improving the model's detection ability for unknown fake methods.

[0103] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A deepfake detection method based on spatiotemporal attention-guided fusion, comprising: S1. Obtain a dataset, obtain the residuals between adjacent frames of the dataset, construct a residual graph, and use video processing tools to obtain the I frame of the dataset; S2. Use the DLIB face extraction model to extract the residual image and the face area of the I frame, form a new dataset based on the face area, and divide the new dataset into a training set and a test set; S3. Set the resolution of the training set and the test set, and perform data augmentation on the training set and the test set to obtain the augmented training set and the test set; S4. The face area and the residual map of the I frame are convolved by the convolution block to obtain the first feature map and the second feature map respectively. The branch for convolving the face area of the I frame is the spatial domain branch, and the branch for convolving the residual map is the residual branch. The first feature map and the second feature map are input into the cross-modal attention fusion module to obtain the temporal features of attention fusion. The temporal features are merged into the spatial domain branch to obtain the feature map input into the next layer of convolution. The construction process of the cross-modal attention fusion module is as follows: the first feature map is convolved by the convolution block to obtain the first query, the second feature map is convolved by two different convolution blocks to obtain the second query and the time domain feature value, the similarity of the two modalities of the first query and the second query is calculated to obtain the similarity value, and the attention map is obtained by combining the similarity value and the weight matrix; In step S4, the formula for obtaining the feature map input to the next layer of convolution is as follows: F′=F T +F Where ft represents the flattening operation, K represents the query obtained in the spatial domain features, and K R represents the query obtained in the time domain feature, C represents the similarity value, W represents the weight matrix, V R represents the value obtained in the time domain feature, A represents the attention map, F T represents the fused temporal features, represents matrix multiplication, and softmax represents the normalized exponential function; S5. Obtain feature representations of the residual image and the face area of the I frame, which are the first feature representation and the second feature representation respectively. Perform average pooling and maximum pooling on the first feature representation to obtain two first 2D maps. Perform average pooling and maximum pooling on the second feature representation to obtain two second 2D maps. Concatenate the two first 2D maps and the two second 2D maps and convolve them through a standard convolutional layer to obtain two 2D spatial attention maps. Normalize the two 2D spatial attention maps to obtain two final attention maps. Use the two final attention maps to represent the RGB features and the inter-frame residual features, respectively, to obtain the attention mutual guidance module. S6. Input the training set and test set into the two-stream network model for learning, set the learning batch and learning rate, and obtain the pre-trained weights. The two-stream network contains two branches, each of which uses the Xception network as the backbone network for time domain and spatial domain feature extraction. Input the pre-trained weights into the attention mutual guidance module for training, and obtain the trained attention mutual guidance module. S7. Adjust the learning rate and repeat step S6 to obtain multiple sets of model weights. Select an evaluation criterion, test the obtained multiple sets of model weights, and select the model weight with the best cross-database effect as the final result of the cross-database detection.

2. A deep fake detection method based on spatiotemporal attention guided fusion according to claim 1, characterized in that: In step S1, the datasets are FaceForensics++ and Celeb-DF.

3. A deep fake detection method based on spatiotemporal attention guided fusion according to claim 1, characterized in that: In step S1, the video processing tool is FFmpeg.

4. The deep fake detection method based on spatiotemporal attention guided fusion according to claim 1 is characterized in that In step S3, data augmentation includes random cropping, random horizontal inversion and Gaussian blur.

5. The deep fake detection method based on spatiotemporal attention guided fusion according to claim 1 is characterized in that In step S5, the size of the first feature representation and the second feature representation is C×H×W, where C is the number of channels, H is the height, and W is the width. The formulas for the two final attention maps are as follows: F′=F1×Sig(Conv(Concat(Avg(F r ),Max(F r )))+F1 F′ r =F r ×Sig(Conv(Concat(Avg(F1),Max(F1)))+F r Where F1 is the first feature representation, Fr is the second feature representation, Sig is the sigmod activation function, Conv is a 1×1 convolution operation, concat represents the concatenation of two vectors, Avg is the average pooling operation, and Max is the maximum pooling operation.

6. The deep fake detection method based on spatiotemporal attention guided fusion according to claim 1 is characterized in that In step S5, the two 2D spatial attention maps are normalized by the sigmoid function to obtain two final attention maps.

7. The deep fake detection method based on spatiotemporal attention guided fusion according to claim 1 is characterized in that In step S6, the training set and the test set are input into the two-stream network model for learning. The fusion mode is cross-modal attention fusion mode, and the loss function is the cross entropy loss function. The formula is as follows: The number of samples y is the true label value, Represents the predicted probability value.

8. The deep fake detection method based on spatiotemporal attention guided fusion according to claim 1 is characterized in that In step S7, the evaluation criteria are selected as ACC and AUC.

9. The deep fake detection method based on spatiotemporal attention guided fusion according to claim 1 is characterized in that In step S7, the model weight with the best cross-database effect is selected as the final result of the cross-database detection. Specifically, the model weight with the best cross-method effect is selected as the final result of the cross-method detection after training on FaceForensics++ and testing on Celeb-DF.

Images