Deepfake synthetic video detection method, system, and device

Abstract

The application relates to a Deepfake synthetic video detection method, system, device and medium, which comprises the following steps: dividing a data set to frame and extracting an RGB image I of a detection area of each frame; using a dual-tree complex wavelet transform to decompose the RGB image I into a low-frequency subband S L and a high-frequency subband set S H in different directions; inputting the low-frequency subband S L and the high-frequency subband set S H into a frequency domain feature extraction network respectively to obtain frequency domain features F DT‑CWT ; inputting the low-frequency subband S L and the high-frequency subband set S H into an image enhancement branch based on energy adjustment for processing to obtain space domain features F RGB ; fusing the frequency domain features F DT‑CWT and the space domain features F RGB to obtain single-frame classification features F C ; inputting the single-frame classification features F C into a preselected classifier network module to realize Deepfake video identification and output a judgment category.

Description

Technical Field

[0001] This invention relates to a method, system, device, and medium for detecting Deepfake synthesized videos based on dual-tree complex wavelet transform (DT-CWT), and relates to the field of Deepfake synthesized video tampering detection technology. Background Technology

[0002] In recent years, thanks to the rapid development of deep generative models, deepfake video spoofing techniques have become increasingly popular on the internet. With the emergence of numerous open-source technologies in deep learning, the technical barrier to video spoofing has become lower and lower. This allows some criminals to easily synthesize fake videos of specific individuals without permission and use them maliciously. Therefore, the detection of deepfake videos is crucial.

[0003] Existing Deepfake synthetic video detection algorithms are mainly implemented through neural networks. Such detection methods can usually achieve an accuracy of over 99% within a database. However, their performance drops significantly when tested across different databases, indicating insufficient generalization performance.

[0004] To improve the generalization performance of algorithms, existing detection techniques utilize DCT to extract frequency domain information and analyze local frequency statistical characteristics for Deepfake video forgery detection. While achieving good detection results on compressed videos, these techniques still show a significant decline in cross-database detection. Existing detection techniques also disclose dual-stream networks based on RGB images and high-frequency noise for Deepfake video detection. These networks promote feature learning through information interaction between the two modalities; however, noise features are easily affected by the source of face data and the face forgery method, leading to differences in noise characteristics across different databases. Therefore, the generalization ability of these algorithms still needs improvement. Summary of the Invention

[0005] To address the aforementioned problems, the purpose of this invention is to provide a method, system, device, and medium for detecting Deepfake synthetic videos that can help improve detection performance and enhance algorithm robustness.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for detecting Deepfake synthesized videos, comprising:

[0008] The dataset is divided into segments, the videos in each dataset are divided into frames, and the RGB images of the detection regions in each frame are extracted.

[0009] The RGB image I is decomposed into low-frequency subband S using dual-tree complex wavelet transform. L and the set of high-frequency subbands in different directions SH ;

[0010] Low-frequency subband S L and the high-frequency subband set S H The inputs are fed into a frequency domain feature extraction network to obtain frequency domain features F. DT-CWT ;

[0011] Low-frequency subband S L and the high-frequency subband set S H The input is processed by the energy-adjusted image enhancement branch to obtain the spatial domain features F. RGB ;

[0012] Frequency domain feature F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C ;

[0013] The single-frame classification features F C Inputting the data into a pre-trained classifier network module enables Deepfake video identification and outputs a decision category.

[0014] The aforementioned Deepfake synthesized video detection method further utilizes dual-tree complex wavelet transform to decompose the RGB image I into low-frequency subbands S. L and the set of high-frequency subbands in different directions S H ,include:

[0015] The dual-tree complex wavelet transform decomposes the RGB image I in multiple directions, generating six high-frequency sub-bands pointing in {±15°, ±45°, ±75°} and one low-frequency sub-band.

[0016] The aforementioned Deepfake synthetic video detection method further includes a frequency domain feature extraction network comprising a frequency domain feature extraction branch network and a frequency domain feature fusion network based on a cross-attention mechanism; wherein,

[0017] The frequency domain feature extraction branch network is used to extract the low-frequency subband S from the input. L and the high-frequency subband set S H Extracting low-frequency detection feature map F L and high-frequency detection feature map F H ;

[0018] A frequency domain feature fusion network based on a cross-attention mechanism is used to integrate low-frequency detection feature maps F. L and high-frequency detection feature map F H By fusing the components, the frequency domain feature F is obtained. DT-CWT .

[0019] The Deepfake synthetic video detection method further comprises a frequency domain feature extraction branch network consisting of a high-frequency SF-Net network and a low-frequency SF-Net network. Both networks employ a structure of a 3x3 kernel ordinary convolution and five inverse residual modules connected in sequence. Specifically, for the high-frequency SF-Net network, the output feature map of the fourth inverse residual module is selected as the high-frequency feature map, and for the low-frequency SF-Net network, the output feature map of the fifth inverse residual module is selected as the low-frequency feature map.

[0020] The aforementioned Deepfake synthetic video detection method further utilizes a frequency domain feature fusion network based on a cross-attention mechanism. This network uses a cross-attention module to process the high-frequency and low-frequency feature maps, passing them through a self-attention module to obtain an attention-guided feature map in the (0,1) open interval. The newly generated self-attention weight map is then recombined into a multi-channel attention-guided feature map T, as shown in the equation:

[0021]

[0022] In the formula, This represents element-wise multiplication. This indicates element-wise addition, and M(HF) and M(LF) represent the self-attention weight maps generated based on the high-frequency feature map HF and the low-frequency feature map LF, respectively.

[0023] The aforementioned Deepfake synthesized video detection method further includes low-frequency subband S... L and the high-frequency subband set S H The input is processed by an energy-adjusted image enhancement branch, including: low-frequency attenuation, high-frequency enhancement, dual-tree complex wavelet inverse transform, Gamma correction, and / or Xception network feature extraction.

[0024] The Deepfake synthetic video detection method further includes the following training process for the classifier module:

[0025] The RGB face images in the training set are subjected to dual-tree complex wavelet transform, and then input into the frequency domain feature extraction network, the energy-adjusted image enhancement branch, and the classifier network module to obtain the predicted probability p.

[0026] Calculate the cross-entropy loss using the predicted probability p and the given binary classification label;

[0027] The model is trained with the goal of minimizing the cross-entropy loss.

[0028] The adaptive Adam algorithm is used as the optimizer for model training.

[0029] Secondly, the present invention provides a Deepfake synthetic video detection system, comprising:

[0030] The frame extraction unit is configured to divide the dataset, divide the video of each dataset into frames, and extract the RGB image I of the detection region of each frame.

[0031] The frequency band decomposition unit is configured to decompose the RGB image I into a low-frequency subband S using a dual-tree complex wavelet transform. L and the set of high-frequency subbands in different directions S H ;

[0032] The frequency domain feature extraction unit is configured to extract low-frequency subband S L and the high-frequency subband set S H The inputs are fed into a frequency domain feature extraction network to obtain frequency domain features F. DT-CWT ;

[0033] The spatial feature extraction unit is configured to extract low-frequency subband S L and the high-frequency subband set S H The input is processed by the energy-adjusted image enhancement branch to obtain the spatial domain features F. RGB ;

[0034] The fusion unit is configured to combine the frequency domain features F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C ;

[0035] The video prediction unit is configured to classify single-frame features F C Inputting the data into a pre-trained classifier network module enables Deepfake video identification and outputs a decision category.

[0036] Thirdly, the present invention also provides an electronic device including computer program instructions, wherein the program instructions, when executed by a processor, are used to implement the Deepfake synthetic video detection method.

[0037] Fourthly, the present invention also provides a computer-readable storage medium storing computer program instructions, wherein the program instructions, when executed by a processor, are used to implement the Deepfake synthetic video detection method.

[0038] Because the present invention adopts the above technical solution, it has the following characteristics:

[0039] 1. The Deepfake Synthetic Video Detection Algorithm Based on Dual-Tree Complex Wavelet Frequency Domain Features provided by this invention can obtain tampering features with stronger representation capabilities, thereby improving the generalization ability of the detection algorithm.

[0040] 2. The frequency domain feature extraction network designed in this invention can effectively extract the frequency domain features of dual-tree complex wavelet. At the same time, the network model is shallow and has few parameters, avoiding the problem of overfitting or even degradation caused by excessively deep networks. The algorithm runs fast and is conducive to application in real-world scenarios.

[0041] 3. This invention employs a frequency domain feature fusion module based on a cross-attention mechanism. Compared to feature splicing methods, this feature fusion approach can extend the attention function from one stream to another, generating rich feature representations. This helps to further strengthen the relationship between features, reduce ambiguity, and thus improve the performance of the model.

[0042] 4. This invention improves and designs the frequency domain feature extraction network and frequency domain feature fusion method. At the same time, it adopts energy-adjustment-based image enhancement technology to preprocess the spatial domain image, improves the contrast of the location of interest in the image, which helps the model improve detection performance and enhances the robustness of the algorithm.

[0043] In summary, this invention can be widely applied to the detection of Deepfake synthesized video tampering. Attached Figure Description

[0044] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings:

[0045] Figure 1 This is a schematic diagram of the network architecture of the Deepfake synthesized video detection method according to an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the training phase of the Deepfake synthesized video detection method according to an embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the SF-Net network architecture according to an embodiment of the present invention;

[0048] Figure 4 This is a diagram of the frequency domain feature extraction network architecture according to an embodiment of the present invention;

[0049] Figure 5 This is a diagram of the energy-adjustment-based image enhancement branch architecture according to an embodiment of the present invention;

[0050] Figure 6 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0051] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0052] Although terms such as first, second, third, etc., may be used in this document to describe multiple elements, components, regions, layers, and / or segments, these elements, components, regions, layers, and / or segments should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or segment from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and other numerical terms used herein do not imply order or sequence. Therefore, the first element, component, region, layer, or segment discussed below may be referred to as the second element, component, region, layer, or segment without departing from the teachings of the exemplary embodiments.

[0053] For ease of description, spatial relative terms may be used in the text to describe the relationship of one element or feature relative to another element or feature as shown in the figure. These relative terms include, for example, "inside," "outside," "middle," "outer," "below," "above," etc. Such spatial relative terms are intended to include different orientations of the device in use or operation, other than those depicted in the figure.

[0054] The Deepfake Synthetic Video Detection Method, System, Device, and Medium provided by this invention include: dividing a dataset, segmenting the video in each dataset into frames, and extracting the RGB image I of the detection region in each frame; and using dual-tree complex wavelet transform to decompose the RGB image I into a low-frequency subband S. L and the set of high-frequency subbands in different directions S H ; to reduce the low-frequency subband S L and the high-frequency subband set S H The inputs are fed into a frequency domain feature extraction network to obtain frequency domain features F. DT-CWT ; to reduce the low-frequency subband S L and the high-frequency subband set S H The input is processed by the energy-adjusted image enhancement branch to obtain the spatial domain features F. RGB; frequency domain features F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C ; classify the single-frame features F C The input is fed into a pre-trained classifier network module to identify Deepfake videos and output a decision category. Therefore, this invention can obtain tamper detection features with stronger representational capabilities through dual-tree complex wavelets, achieving better detection results and effectively improving generalization performance.

[0055] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the invention and to fully convey the scope of the invention to those skilled in the art.

[0056] Example 1: The Deepfake Synthesis Video Detection Method Based on Dual-Tree Complex Wavelet provided in this example includes:

[0057] S1: Divide the dataset, segment the video of each dataset into frames, and extract the RGB image I of the detection region of each frame.

[0058] S2: The RGB image I is decomposed into a low-frequency subband S using dual-tree complex wavelet transform. L and the set S of high-frequency subbands in 6 different directions H .

[0059] S3: Construct a frequency domain feature extraction network, including a frequency domain feature extraction branch network and a frequency domain feature fusion network based on a cross-attention mechanism, including:

[0060] S31: Construct a frequency domain feature extraction branch network to extract low-frequency subband S L and the high-frequency subband set S H The inputs are fed into the frequency domain feature extraction branch network to obtain the low-frequency detection feature map F. L and high-frequency detection feature map F H .

[0061] Specifically, the frequency domain feature extraction branch network consists of a high-frequency SF-Net network and a low-frequency SF-Net network, both employing a structure of a 3x3 kernel convolution followed by five inverse residual modules connected sequentially. Each inverse residual module consists of an expansion layer, a batch normalization layer, and a ReLU activation function layer, followed by a 1x1 depthwise separable convolution layer, a batch normalization layer, and a ReLU activation function layer, finally connected to a 1x1 convolutional layer and a batch normalization layer. The number of repetitions of each inverse residual module is determined by the hyperparameter n. For the high-frequency SF-Net network, the output feature map of the fourth inverse residual module is selected as the high-frequency feature map; for the low-frequency SF-Net network, the output feature map of the fifth inverse residual module is selected as the low-frequency feature map.

[0062] S32: Construct a frequency domain feature fusion network based on a cross-attention mechanism to integrate low-frequency detection feature maps F L and high-frequency detection feature map F H The input is fused into a frequency domain feature fusion network based on a cross-attention mechanism to obtain frequency domain features F. DT-CWT .

[0063] A frequency domain feature fusion network based on cross-attention mechanism is used to fuse the output low-frequency feature map and high-frequency feature map.

[0064] Specifically, a frequency domain feature fusion network based on a cross-attention mechanism is constructed. The cross-attention weight maps are attention-guided feature maps of the (0,1) open interval obtained from either the high-frequency or low-frequency feature map through a self-attention module. The high-frequency feature map is processed in the same way as the low-frequency feature map; adding the high-frequency or low-frequency feature map to its corresponding cross-attention weight map enhances important latent features in either the high-frequency or low-frequency feature map. Finally, the two newly generated feature maps are recombine into a multi-channel attention-guided feature map T, as shown in the equation:

[0065]

[0066] In the formula, This represents element-wise multiplication. This indicates element-wise addition, and M(HF) and M(LF) represent the self-attention weight maps generated based on the high-frequency feature map HF and the low-frequency feature map LF, respectively.

[0067] S4: Construct an image enhancement branch based on energy adjustment, and enhance the low-frequency subband S L and the high-frequency subband set S HThe input is fed into the energy-adjusted image enhancement branch, and undergoes processes such as low-frequency attenuation, high-frequency enhancement, dual-tree complex wavelet inverse transform, Gamma correction, and Xception network feature extraction to obtain the spatial domain features FRGB.

[0068] Specifically, the energy-adjusted image enhancement branch uses DT-CWT to decompose the face region image into components with different sizes, positions, and orientations, weakening the low-frequency sub-band and enhancing the high-frequency sub-band. The specific calculation formula is as follows:

[0069]

[0070] Where F(x,y) represents the low-frequency sub-image obtained after the second-level decomposition of DT-CWT, (m,n) represents the position of the point with the maximum amplitude in the sub-image, and F′(x,y) represents the low-frequency sub-band image after low-frequency attenuation. The constant k represents the degree of attenuation.

[0071] The high-frequency enhancement in the energy-adjusted image enhancement branch is specifically calculated using the following formula:

[0072] D′ a =D α / s O<s<1

[0073] In the formula, the six high-frequency subbands obtained by the two-level dual-tree complex wavelet transform are denoted as D. α α = {±15°, ±45°, ±75°}, the enhanced subband is D′ α The constant s represents the degree of enhancement. High-frequency enhancement means enhancing the detail coefficients.

[0074] Furthermore, to ensure that the total energy of the images remains unchanged before and after enhancement, Gamma correction is used for adjustment, with the specific formula as follows:

[0075]

[0076]

[0077] Where u(i,j) is the image generated after the inverse transform, g(i,j) is the image after Gamma correction, γ is the correction factor, and m u Let m be the mean gray value of u(i,j). f The grayscale mean of the original image is τ, which is a constant used to adjust the brightness of the image.

[0078] Furthermore, during the feature extraction process of the Xception network, the energy-adjusted enhanced image is fed into the Xception network, where the 2048-dimensional output tensor of the last average pooling layer of the Xception network is the learned color feature F. RGB .

[0079] S5: Frequency domain features F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C .

[0080] S6: Classification features F C The video is fed into a pre-trained classifier network module consisting of fully connected layers and softmax activation layers to obtain the predicted probability p. If p is greater than 0.5, it is judged as a real video; otherwise, it is judged as a synthetic video.

[0081] Specifically, the training process of the classifier network module includes:

[0082] Calculate the loss function and backpropagate to update the network weight coefficients, saving the network model and optimal weights.

[0083] The RGB face images in the training set are subjected to dual-tree complex wavelet transform, and then input into the frequency domain feature extraction network, the energy-adjusted image enhancement branch, and the classifier network module to obtain the predicted probability p.

[0084] Calculate the cross-entropy loss using the predicted probability p and the given binary labels.

[0085] The model is trained with the goal of minimizing the cross-entropy loss.

[0086] During training, the adaptive Adam algorithm was used as the optimizer, with an initial learning rate of 0.0001. The optimization scheme for the learning rate was to reduce it by a factor of 0.5 if the loss on the validation set did not decrease after two training epochs. The training epochs were set to 20, and the batch size was set to 32. Finally, the model and weights with the minimum loss on the validation set were saved to complete the model training and obtain the trained classifier network module.

[0087] The implementation process of the Deepfake Synthesis Video Detection Method based on Dual-Tree Complex Wavelet in this embodiment is described in detail below through specific examples.

[0088] This embodiment selects the TIMIT-DF, FaceForensics++ (hereinafter referred to as FF++), DeepfakeDetection (hereinafter referred to as DFD), and FFW databases for intra-database testing and cross-database testing. In this embodiment, the C23 data from the DFD database and the C0 and C23 data from the FF++ database are divided into training, validation, and test sets according to a video ratio of 7:2:1. The TIMIT-DF and FFW databases will be used for cross-database testing. Since FFW lacks real videos, 50 real videos are selected from the FF++ test set to supplement it. When partitioning the databases, considering that duplicate data in the training, validation, and test sets would cause severe overfitting to the source domain during training, leading to a decrease in the algorithm's generalization performance, this embodiment chooses to partition the dataset by person, ensuring that the real face and Deepfake face of the same person can only appear in one of the training, validation, or test datasets. The databases partitioned and used in this embodiment are shown in Table 1. This experiment was conducted on a Linux system, primarily based on the deep learning framework Keras, with a GTX 1080 Ti graphics card and CUDA version 9.0.176.

[0089] Table 1 Database Partitioning (Unit: Frames)

[0090]

[0091] like Figure 1 , Figure 2 As shown, the Deepfake Synthetic Video Detection Method based on Dual-Tree Complex Wavelet Transform in this embodiment includes the following steps:

[0092] S1: Decode the video to be detected into a frame sequence and extract the RGB image I of the detection region of each frame;

[0093] Specifically, the videos in each database are first decoded into frame sequences, and sampling is performed at intervals, which can be set to 5. The frame sequences are then saved as a unit of video. Next, the face detector get_frontal_face_detector of the open-source framework Dlib library is used to perform face recognition on the video frame sequences, extracting 68 feature points of the face. The 30th feature point is the tip of the nose. Finally, a region of size 256×256 is cropped with the tip of the nose as the center point as the feature extraction region.

[0094] S2: Perform dual-tree complex wavelet transform on the feature extraction region, decomposing it into high-frequency subbands S H and low-frequency subband S L .

[0095] The feature extraction region is adjusted to a uniform-sized RGB image I and normalized to serve as the color feature data for the feature detection region. The dual-tree complex wavelet transform decomposes the RGB image I in multiple directions, generating six high-frequency sub-bands and one low-frequency sub-band pointing in {±15°, ±45°, ±75°}.

[0096] S3: Construct a frequency domain feature extraction network, including a high- and low-frequency feature extraction branch network and a frequency domain feature fusion network based on a cross-attention mechanism.

[0097] Low-frequency subband S L and the high-frequency subband set S H The data are fed into the corresponding frequency domain feature extraction branch networks to obtain low-frequency detection feature maps F. L and high-frequency detection feature map F H The low-frequency and high-frequency detection feature maps are input into a frequency domain feature fusion module based on a cross-attention mechanism for fusion to obtain the frequency domain feature F. DT-CWT ;

[0098] S31: Constructing a frequency domain feature extraction branch network

[0099] In this embodiment, the frequency domain feature extraction branch consists of a high-frequency SF-Net network and a low-frequency SF-Net network. The SF-Net network uses the inverse residual module of MoileNet V2 as the basic module of the network model, and its network structure is as follows: Figure 3 As shown in Table 2, the SF-Net network employs a structure consisting of a 3x3 convolutional kernel and five inverse residual modules connected sequentially. Detailed parameters of the SF-Net structure are shown in Table 2.

[0100] Table 2 Detailed parameters of SF-Net structure

[0101]

[0102] In this embodiment, a low-frequency subband with a size of 256×256×1 is input into the low-frequency SF-Net network, and the output feature map of the 5th inverse residual module is selected as the low-frequency feature map. A high-frequency subband with a size of 128×128×6 is input into the high-frequency SF-Net network, and the output feature map of the 4th inverse residual module is selected as the low-frequency feature map. The data dimensions of the output low-frequency feature map and high-frequency feature map are both 16×16×320.

[0103] S32: Constructing a frequency domain feature fusion network based on cross-attention mechanism

[0104] like Figure 4As shown, in this embodiment, a frequency domain feature fusion network based on a cross-attention mechanism is used. Taking low-frequency features as an example, in the cross-attention module, the cross-attention weight map is an attention-guided feature map with an open interval of (0,1) obtained from the low-frequency feature map through the self-attention module. The high-frequency feature map is processed in the same way as the low-frequency feature map. Finally, the two newly generated feature maps are recombined into a multi-channel attention-guided feature map.

[0105] S4: Construct an energy-adjusted image enhancement branch, which will enhance the low-frequency subband S L and the high-frequency subband set S H The image is fed into an energy-adjusted image enhancement branch, where it undergoes low-frequency attenuation, high-frequency enhancement, dual-tree complex wavelet inverse transform, Gamma correction, and Xception network feature extraction to obtain the spatial domain feature F. RGB ;

[0106] In this embodiment, as Figure 5 As shown, the face region image is decomposed into components of different sizes, positions, and orientations using DT-CWT, and the low-frequency subband is weakened while the high-frequency subband is enhanced, thereby amplifying the high-frequency components that represent details. Next, a dual-tree complex wavelet inverse transform is applied. To ensure that the total energy of the image remains unchanged before and after enhancement, a Gamma nonlinear transform is used to adjust the grayscale values ​​of the original image, thereby regulating the image brightness and enhancing the visual effect.

[0107] S5: Frequency domain features F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C .

[0108] S6: Prediction is based on the classifier network module.

[0109] Classification features F C The video is fed into a classifier network module consisting of fully connected layers and softmax activation layers to obtain the predicted probability p. If p is greater than 0.5, the video is judged as real; otherwise, it is judged as synthetic video.

[0110] The training process of the classifier network module is as follows: calculate the loss function and backpropagate to update the network weight coefficients, saving the network model and optimal weights. Perform dual-tree complex wavelet transform on the RGB face images in the training set, and input them into the frequency domain feature extraction network, the energy-adjusted image enhancement branch, and the classifier network module to obtain the predicted probability p. Calculate the cross-entropy loss using the predicted probability p and the given classification label. Train the model with the goal of minimizing the cross-entropy loss. During training, the adaptive Adam algorithm is used as the optimizer, with an initial learning rate of 0.0001. The optimization scheme for the learning rate is to reduce it by a factor of 0.5 if the loss on the validation set does not decrease after two training epochs. The training epoch is set to 20, and the batch size is 32. Finally, save the model and weights that minimize the loss on the validation set.

[0111] The cross-entropy loss function is defined as follows:

[0112]

[0113] Where x represents a sample, y represents the actual label, a represents the predicted output, and n represents the total number of samples.

[0114] S8: Model Application: Load the classification network and parameters saved from the model training steps as the backend module of the detection system, input each video frame of the test set into the detection system, and predict the classification result.

[0115] In this embodiment, the model and weights trained using the training sets of the DFD database and the FF++ database are loaded and tested respectively. The performance metric of the Deepfake fake face detection algorithm in this embodiment is the Half Total Error Rate (HTER), and the formula for calculating the Half Total Error Rate (HTER) is as follows:

[0116]

[0117] Here, FAR (False Acceptance Rate) is the false acceptance rate, which is the percentage of times the algorithm identifies a Deepfake face as a real face in a video; FRR (False Rejection Rate) is the false rejection rate, which is the percentage of times the algorithm identifies a real face in a video as a Deepfake face.

[0118] To comprehensively explore the generalization ability of the algorithm, this embodiment uses the DFD(C23) database and the FF++(C0&C23) database as training databases, respectively, and performs cross-database performance verification on the DFD(C23) database, FF++(C0), FF++(C23) database, TIMIT-DF database, and FFW database. Tables 3 and 4 show the HTER test results of the models trained on the DFD(C23) and FF++(C0&C23) databases, respectively.

[0119] Table 3 Comparison of HTER Algorithm Experiment Results (%)

[0120]

[0121] Table 4 Comparison of HTER Algorithm Experiment Results II (%)

[0122]

[0123] As can be seen from Table 3, the HTER of the algorithm of this invention trained on the DFD(C23) database is 1.93% in the in-database test, and 11.95, 16.73, 13.74 and 18.42 in the cross-database tests of FF++(C0), FF++(C23), TIMIT-DF and FFW, respectively. The algorithm of this invention performs well in both in-database and cross-database tests.

[0124] As observed in Table 4, in the model trained on FF++ (C0 & C23), the HTER of the algorithm of this invention is 0.54% in the in-library test. In the cross-library test of FF++ (C23), DFD (C23), TIMIT-DF, and FFW, the HTERs are 1.84, 22.81, 24.23, and 13.15, respectively, further demonstrating that the algorithm of this invention performs excellently in both in-library and cross-library tests.

[0125] In summary, this embodiment utilizes the dual-tree complex wavelet transform method to design a frequency domain feature extraction branch and an energy-adjustment-based image enhancement branch network. Input data is fed into the frequency domain feature extraction branch and the energy-adjustment-based image enhancement branch, respectively, to obtain frequency domain features F. DT-CWT and spatial characteristics F RGB These two types of features are fed into the classifier network module to identify Deepfake synthetic videos. Experimental results show that dual-tree complex wavelet transform, feature fusion method based on cross-attention mechanism, and image enhancement can all improve the performance of Deepfake synthetic video detection system.

[0126] Example 2: Following the Deepfake Synthetic Video Detection Method provided in Example 1, this example provides a Deepfake Synthetic Video Detection System. The system provided in this example can implement the Deepfake Synthetic Video Detection Method of Example 1. This system can be implemented through software, hardware, or a combination of both. For ease of description, this example is described by dividing the system into various functional units. Of course, in implementation, the functions of each unit can be implemented in one or more software and / or hardware components. For example, the system may include integrated or separate functional modules or units to execute the corresponding steps in the methods of Example 1. Since the system in this example is basically similar to the method example, the description process of this example is relatively simple. Relevant details can be found in the description of Example 1. The embodiment of the Deepfake Synthetic Video Detection System provided by this invention is merely illustrative.

[0127] Specifically, the Deepfake Synthesis Video Detection System based on Dual-Tree Complex Wavelet provided in this embodiment includes:

[0128] The frame extraction unit is configured to divide the dataset, divide the video of each dataset into frames, and extract the RGB image I of the detection region of each frame.

[0129] The frequency band decomposition unit is configured to decompose the RGB image I into a low-frequency subband S using a dual-tree complex wavelet transform. L and the set of high-frequency subbands in different directions S H ;

[0130] The frequency domain feature extraction unit is configured to extract low-frequency subband S L and the high-frequency subband set S H The inputs are fed into a frequency domain feature extraction network to obtain frequency domain features F. DT-CWT ;

[0131] The spatial feature extraction unit is configured to extract low-frequency subband S L and the high-frequency subband set S H The input is processed by the energy-adjusted image enhancement branch to obtain the spatial domain features F. RGB ;

[0132] The fusion unit is configured to combine the frequency domain features F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C ;

[0133] The video prediction unit is configured to classify single-frame features F C Inputting the data into a pre-trained classifier network module enables Deepfake video identification and outputs a decision category.

[0134] Example 3: This example provides an electronic device corresponding to the Deepfake synthesized video detection method provided in Example 1. The electronic device can be an electronic device for client use, such as a mobile phone, laptop, tablet computer, desktop computer, etc., to execute the method of Example 1.

[0135] like Figure 6 As shown, the electronic device includes a processor, memory, communication interface, and bus. The processor, memory, and communication interface are connected via the bus to enable communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes the Deepfake synthetic video detection method provided in Embodiment 1. Those skilled in the art will understand that... Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computing device on which the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0136] In some implementations, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), and optical discs.

[0137] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0138] Example 4: The Deepfake synthesized video detection method of Example 1 can be specifically implemented as a computer program product. The computer program product may include a computer-readable storage medium on which computer-readable program instructions for executing the Deepfake synthesized video detection method described in Example 1 are loaded.

[0139] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0140] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the reference to terms such as "an embodiment," "some implementations," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the embodiments in this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0141] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting Deepfake synthesized videos, characterized in that, include: The dataset is divided into segments, the videos in each dataset are divided into frames, and the RGB images of the detection regions in each frame are extracted. The RGB image I is decomposed into low-frequency subband S using dual-tree complex wavelet transform. L and the set of high-frequency subbands in different directions S H ; Low-frequency subband S L and the high-frequency subband set S H The inputs are fed into a frequency domain feature extraction network to obtain frequency domain features F. DT-CWT ; Low-frequency subband S L and the high-frequency subband set S H The input is processed by the energy-adjusted image enhancement branch to obtain the spatial domain features F. RGB ; Frequency domain feature F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C ; The single-frame classification features F C The input is fed into a pre-trained classifier network module to identify Deepfake videos and outputs a decision category, where: The frequency domain feature extraction network includes a frequency domain feature extraction branch network and a frequency domain feature fusion network based on a cross-attention mechanism; among them, The frequency domain feature extraction branch network is used to extract the low-frequency subband S from the input. L and the high-frequency subband set S H Extracting low-frequency detection feature map F L and high-frequency detection feature map F H ; A frequency domain feature fusion network based on a cross-attention mechanism is used to integrate low-frequency detection feature maps F. L and high-frequency detection feature map F H By fusing the components, the frequency domain feature F is obtained. DT-CWT ; The frequency domain feature fusion network based on the cross-attention mechanism uses a cross-attention module to obtain an attention-guided feature map in the (0, 1) open interval from the high-frequency feature map and the low-frequency feature map through a self-attention module. The newly generated self-attention weight map is then recombined into a multi-channel attention-guided feature map T, as shown in the equation: In the formula, This represents element-wise multiplication. This indicates element-wise addition, and M(HF) and M(LF) represent the self-attention weight maps generated based on the high-frequency feature map HF and the low-frequency feature map LF, respectively.

2. The Deepfake synthesized video detection method according to claim 1, characterized in that, The RGB image I is decomposed into low-frequency subband S using dual-tree complex wavelet transform. L and the set of high-frequency subbands in different directions S H ,include: The dual-tree complex wavelet transform decomposes the RGB image I in multiple directions, generating six high-frequency sub-bands and one low-frequency sub-band pointing in {±15°, ±45°, ±75°}.

3. The method for detecting Deepfake synthesized videos according to claim 2, characterized in that, The frequency domain feature extraction branch network consists of a high-frequency SF-Net network and a low-frequency SF-Net network. Both networks use a structure of a regular convolution with a kernel size of 3x3 and five inverse residual modules connected in sequence. Specifically, for the high-frequency SF-Net network, the output feature map of the fourth inverse residual module is selected as the high-frequency feature map, and for the low-frequency SF-Net network, the output feature map of the fifth inverse residual module is selected as the low-frequency feature map.

4. The Deepfake synthesized video detection method according to claim 1, characterized in that, Low-frequency subband S L and the high-frequency subband set S H The input is processed by an energy-adjusted image enhancement branch, including: low-frequency attenuation, high-frequency enhancement, dual-tree complex wavelet inverse transform, Gamma correction, and / or Xception network feature extraction.

5. The Deepfake synthesized video detection method according to claim 1, characterized in that, The training process of the classifier module includes: The RGB face images in the training set are subjected to dual-tree complex wavelet transform, and then input into the frequency domain feature extraction network, the energy-adjusted image enhancement branch, and the classifier network module to obtain the predicted probability p. Calculate the cross-entropy loss using the predicted probability p and the given binary classification label; The model is trained with the goal of minimizing the cross-entropy loss. The adaptive Adam algorithm is used as the optimizer for model training.

6. A Deepfake synthetic video detection system, characterized in that, include: The frame extraction unit is configured to divide the dataset, divide the video of each dataset into frames, and extract the RGB image I of the detection region of each frame. The frequency band decomposition unit is configured to decompose the RGB image I into a low-frequency subband S using a dual-tree complex wavelet transform. L and the set of high-frequency subbands in different directions S H ; The frequency domain feature extraction unit is configured to extract low-frequency subband S L and the high-frequency subband set S H The inputs are fed into a frequency domain feature extraction network to obtain frequency domain features F. DT-CWT ; The spatial feature extraction unit is configured to extract low-frequency subband S L and the high-frequency subband set S H The input is processed by the energy-adjusted image enhancement branch to obtain the spatial domain features F. RGB ; The fusion unit is configured to combine the frequency domain features F DT-CWT and spatial characteristics F RGB Feature concatenation and fusion are performed to obtain single-frame classification features F. C ; The video prediction unit is configured to classify single-frame features F C The input is fed into a pre-trained classifier network module to identify Deepfake videos and outputs a decision category, where: The frequency domain feature extraction network includes a frequency domain feature extraction branch network and a frequency domain feature fusion network based on a cross-attention mechanism; among them, The frequency domain feature extraction branch network is used to extract the low-frequency subband S from the input. L and the high-frequency subband set S H Extracting low-frequency detection feature map F L and high-frequency detection feature map F H ; A frequency domain feature fusion network based on a cross-attention mechanism is used to integrate low-frequency detection feature maps F. L and high-frequency detection feature map F H By fusing the components, the frequency domain feature F is obtained. DT-CWT ; The frequency domain feature fusion network based on the cross-attention mechanism uses a cross-attention module to obtain an attention-guided feature map in the (0, 1) open interval from the high-frequency feature map and the low-frequency feature map through a self-attention module. The newly generated self-attention weight map is then recombined into a multi-channel attention-guided feature map T, as shown in the equation: In the formula, This represents element-wise multiplication. This indicates element-wise addition, and M(HF) and M(LF) represent the self-attention weight maps generated based on the high-frequency feature map HF and the low-frequency feature map LF, respectively.

7. An electronic device, characterized in that, It includes computer program instructions, wherein when the program instructions are executed by a processor, they are used to implement the Deepfake synthetic video detection method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, wherein the program instructions, when executed by a processor, are used to implement the Deepfake synthetic video detection method as described in any one of claims 1 to 5.

Images