A deep fake video discrimination method based on frequency domain information and multi-task learning

Abstract

The present application relates to a kind of deep fake video identification method based on frequency domain information and multi-task learning, using discrete cosine transform in frequency domain analysis, part of the spatial information of RGB three-channel image is reserved by the way of block processing, frequency domain features are obtained as input data;Using the feature of input data is extracted by multi-task learning deep neural network, Xception network is used as backbone network module, and the segmentation module based on deconvolution operation and the classification module based on feature fusion are designed, the features extracted by backbone network module and segmentation module are fused;While designing optimization training guide target algorithm, the relationship between the features after fusion is converted into geometric distance under three-dimensional condition, the training of multi-task learning deep neural network model is completed by optimization training guide target algorithm, and a deep fake video identification model is obtained, to complete the identification of deep fake video.

Description

Technical Field

[0001] This invention relates to a method for identifying deepfake videos based on frequency domain signal processing and analysis and combined with a multi-task deep learning model, belonging to the field of deep learning and computer vision, specifically the field of deepfake video detection. Background Technology

[0002] With the development of deep learning technology, especially generative networks such as Generative Adversarial Networks (GANs), deep learning models can now produce increasingly more high-definition, high-quality images and videos. Models for generating human faces have made particularly significant progress, now capable of generating synthetic faces that are difficult for the naked eye to distinguish, and are virtually indistinguishable from real faces. This type of technology is collectively known as deepfake generation technology. On the other hand, as deepfake generation technology is increasingly being used to infringe upon the rights of others, the field of deepfake detection, specifically designed to identify such deepfake videos, has emerged. The fields of deepfake generation and detection present a complementary relationship, mutually promoting each other's development through competition, and together constitute the field of deepfake technology.

[0003] In recent years, deepfake detection technology has developed into three main approaches: detection based on human-defined traces, detection based on single images, and detection based on image sequences. However, the specific methods used in practice are diverse, resulting in a flourishing landscape. Early detection methods were mostly designed based on human-defined trace detection. Based on early visual observation of forged videos, people first discovered visible visual flaws in details such as the blinking patterns of people in the video, head posture angles, and highlights in the teeth and pupils. Therefore, various technologies were applied to develop targeted detection methods for these visual flaws. Single-image detection methods mostly employ data-driven deep learning algorithms. They enhance certain features in the frames of forged video images through preprocessing, and then use a data-driven approach to drive the deep learning model to spontaneously explore forgery traces. The basis for these methods is often subtle texture information invisible to the naked eye. Image sequence-based detection methods introduce temporal features between frame sequences in a video on top of a single image, providing more criteria for deepfake detection. These methods often employ deep learning networks with larger parameters and higher computational costs, such as 3D convolution and temporal models, to more thoroughly mine traces.

[0004] Existing deepfake detection methods only demonstrate good accuracy on specific datasets. However, their generalization ability is severely hampered when dealing with cross-domain scenarios across different datasets. Furthermore, the scenes in these datasets are relatively simple and controllable, with consistent and straightforward image quality, character cooperation, and the types and approaches of forgery techniques. In contrast, deepfake videos circulating online differ fundamentally in origin, technology, and effect from those datasets, often resulting in unsatisfactory performance from existing methods. This problem can be attributed to three main factors: First, the technological origins of deepfake videos circulating online are unknown. Many forgery sources, such as applications developed using deepfake technology and released on collection platforms, possess technologies that are not publicly available or accessible. Second, during online circulation, videos undergo significant compression through reposting and uploading, erasing crucial details essential for deepfake detection and increasing the difficulty of identification. Finally, forged videos on the internet are often meticulously crafted by forgers using carefully designed processes to achieve specific purposes, fundamentally different from mass-generated datasets.

[0005] In conclusion, existing identification methods in the field of deepfake detection have certain limitations. Summary of the Invention

[0006] The problem this invention aims to solve is to overcome the poor stability and generalization of existing methods, which are unable to effectively deal with deepfake videos on the Internet. This invention provides a deepfake video identification method based on frequency domain information and multi-task learning, which improves the accuracy of deepfake video identification, enhances the ability of the identification method to cope with the network video environment, and improves the interpretability and generalization of the model.

[0007] The deepfake video identification method based on frequency domain information and multi-task learning of the present invention includes the following steps:

[0008] Step 1: Decompose the deepfake video into image frames to obtain RGB three-channel images. Then, use the discrete cosine transform in frequency domain analysis to operate on the RGB three-channel images. Combined with block processing, retain some spatial information of the RGB three-channel images to obtain the preprocessed frequency domain features as input data.

[0009] Step 2: Use a multi-task learning deep neural network to extract features from the input data obtained in Step 1. The multi-task learning deep neural network model uses the Xception network as the backbone network module, and designs a segmentation module based on deconvolution operation and a classification module based on feature fusion. The features extracted by the backbone network module and the segmentation module are fused to obtain the fused features. These features contain hundreds of dimensions of attributes and are called fused high-dimensional features or features in high-dimensional space. These fused high-dimensional features are further fed into the classifier part of the classification module to calculate the classification result.

[0010] Step 3: Transform the relationship between the fused high-dimensional features into the geometric distance of a sphere under three-dimensional conditions. The fused high-dimensional features obtained in Step 2 are divided into high-dimensional features of the real video and high-dimensional features of the fake video. The high-dimensional features of the real video are constrained to the vicinity of the center of the three-dimensional sphere, and the high-dimensional features of the fake video are constrained to the surface of the three-dimensional sphere and kept at a buffer distance from the features of the real video. By designing an optimized training-guided target algorithm, the training of a deep neural network model for multi-task learning is realized, and this model is used to complete the identification of deep fake videos.

[0011] In step 1: the frequency domain features after block processing As shown below:

[0012]

[0013] in Refers to a local RGB image with sides of 8 pixels. This refers to the two-dimensional discrete cosine transform described above.

[0014] In step 2, the structure of the deep neural network for multi-task learning is as follows:

[0015] The structure of a deep neural network for multi-task learning includes a backbone network module, a segmentation module, and a classification module. The backbone network module adopts the structure of the Xception model, retaining only the features output by the 12th convolutional module in the original Xception model without using the Xception model's classifier. The output features are then input into the segmentation and classification modules. The segmentation module contains four sets of deconvolutional modules, each containing one deconvolutional layer, one convolutional layer, and one activation layer. The segmentation module is responsible for further refining the features and then passing the refined features to the classification module. The classification module obtains the outputs of the backbone network module and the segmentation module respectively, performs feature fusion based on convolution operations, and then inputs the fused features into the classifier part of the classification module to determine the authenticity of the video.

[0016] In step 3, the optimized training guidance target algorithm is implemented as follows:

[0017] This is achieved using a loss function, which is expressed by the following formula:

[0018]

[0019] in To ultimately optimize training and guide the learning process. and To balance the parameters, in actual use, they are taken as 1.0 and 5.0 respectively. The cross-entropy loss function is the relationship between the predicted result and the actual category. These represent the model output and the pre-labeled actual classification results, respectively. , and These are predefined control parameters, which are actually used in practice at values ​​of 2.0, 0.5, and 0.9. The angle between `pred` and the parameters corresponding to `label` in the classifier of the classification module. Since this problem is a binary classification problem to distinguish between true and false cases, the class opposite to `label` is called `other`. The angle between the parameter corresponding to "other" in the classifier and "pred".

[0020] The advantages and benefits of this invention compared to existing technologies are as follows:

[0021] (1) This invention proposes an image preprocessing method based on frequency domain analysis, which can more effectively expose the essential forgery traces. In the process of processing, considering the lack of spatial location information and low spatial convolution efficiency of frequency domain features during feature fusion with features extracted from traditional spatial domain RGB images, the discrete cosine transform method of local block calculation is adopted to improve the quality of frequency domain analysis preprocessing and improve the accuracy of deep forgery identification.

[0022] (2) This invention proposes a deep neural network based on multi-task learning. Compared with traditional neural networks, this invention can effectively avoid the problem that the model focuses on background information and edge features, which may reduce the generalization and robustness of the identification method. It integrates the features extracted by the backbone network module and the segmentation module, thereby achieving higher accuracy in identifying deepfake videos of specific people.

[0023] (3) This invention proposes an optimized training guidance target algorithm that is more in line with the nature of fake videos. This algorithm enables the model to constrain the features of real and fake videos to a more reasonable geometric position in high-dimensional space during the learning process of how to identify deepfake videos, thereby improving robustness and having better generalization ability when faced with fake videos that have never been seen before. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention;

[0025] Figure 2 This describes the implementation process of step 1 of the present invention;

[0026] Figure 3 This is a diagram of the deep neural network structure for multi-task learning in this invention.

[0027] Figure 4 This is a specific implementation process of step 3 in the present invention. Detailed Implementation

[0028] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] like Figure 1 As shown, the present invention provides a deep forgery video identification method based on frequency domain information and multi-task learning, which consists of three parts: frequency domain preprocessing, training a feature extraction model based on multi-task learning, and optimizing the training guidance target. The specific implementation steps are as follows:

[0030] Step 1: Frequency Domain Analysis and Preprocessing of Image Data

[0031] This invention uses the two-dimensional discrete cosine transform (DCT) of frequency domain analysis. The two-dimensional DCT is equivalent to performing a one-dimensional DCT in both the length and width dimensions of the image. The basic formula for the two-dimensional DCT is:

[0032]

[0033]

[0034] ,in These are the coordinates in the spectrum after discrete cosine transform. The coordinates in the RGB image and For standardization constants, In the original RGB image Pixel values ​​at the location, is the side length of the image.

[0035] This invention, based on traditional frequency domain processing, considers the need to effectively combine RGB and frequency domain features. It employs a block-based processing method to effectively preserve some spatial information in the feature path of the frequency domain features. The resulting frequency domain features are obtained through block processing. As shown below:

[0036]

[0037] in Refers to a local RGB image with sides of 8 pixels, such as Figure 2 As shown, the three-channel RGB image is divided into multiple 8x8 local blocks, and the parts whose side lengths are not divisible are filled with 8x8 blocks. For the above two-dimensional discrete cosine transform, the transformed spectrum is used... Figure 2 The above method concatenates the preprocessed frequency domain features. This workflow allows the frequency domain features to be combined with other local features, improving the utilization rate of both features by the convolutional layer.

[0038] Step 2: Training a deep neural network based on multi-task learning

[0039] This invention employs the Xception network as its backbone network module and designs a segmentation module based on deconvolution operations and a classification module based on feature fusion. The features extracted by the backbone network module and the segmentation module are fused to obtain fused features. These features contain attributes with hundreds of dimensions and are referred to as fused high-dimensional features or features in high-dimensional space. These fused high-dimensional features are further fed into the classifier part of the classification module to calculate the classification result.

[0040] This invention, when verifying the authenticity of videos, not only considers the features extracted by the Xception backbone module, but also fuses multiple features to improve the overall verification effect, such as... Figure 3 As shown, the backbone network module adopts the structure of the Xception model, retaining only the features output by the 12th convolutional module in the original Xception model without using the Xception model's classifier. It extracts features from the input data and then inputs the output features into the segmentation and classification modules. The segmentation module contains four sets of deconvolutional modules, each containing one deconvolutional layer, one convolutional layer, and one activation layer. The segmentation module is responsible for further refining features, receiving the output of the backbone network module as input, and further processing the extracted features based on deconvolution operations to become the features of the segmentation module. These refined features are then passed to the classification module. The classification module obtains the outputs of the backbone network module and the segmentation module respectively, and performs feature fusion based on convolution operations. Therefore, when performing classification, the model can make more effective judgments regarding deepfake videos and videos featuring distinctive individuals circulating on the internet.

[0041] Step 3: Optimize the training guidance algorithm

[0042] The proposed optimization training-guided target algorithm transforms the relationships between fused high-dimensional features into spherical geometric distances under three-dimensional conditions. The fused high-dimensional features obtained in the previous step are divided into high-dimensional features of the real video and high-dimensional features of the fake video, such as... Figure 4As shown, the high-dimensional features of real videos are constrained to the vicinity of the center of a three-dimensional sphere. However, the high-dimensional features of fake videos are fundamentally different. For example, for two different forgery techniques A and B, the high-dimensional features extracted from the deepfake videos they produce are represented as the features of forgery type A and the features of forgery type B, respectively. The high-dimensional features of the fake videos are constrained to the surface of a three-dimensional sphere and kept at a buffer distance from the features of real videos. This distance is predefined by humans. By designing an optimized training-guided target algorithm, the training of a deep neural network model for multi-task learning is realized, resulting in a deepfake video identification model. This model is then used to identify deepfake videos.

[0043] Therefore, the optimized training guidance target proposed in this invention can provide a certain fault tolerance range for the identification system to deal with deepfake videos circulating on the Internet and deepfake videos of specific individuals.

[0044] The deepfake video identification method proposed in this invention is based on the essential common features of deepfake generation methods and the observational basis of the differences in features between real videos and deepfake videos in reality. It can not only identify deepfake videos with higher accuracy, but also improve the ability to deal with deepfake videos in more complex real-world scenarios in the network. Finally, the method is also optimized from the feature location space in high-dimensional space, which has better generalization ability.

[0045] While specific implementation methods of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples. Various changes or modifications can be made to these implementation methods without departing from the principles and implementation of the present invention. Therefore, the scope of protection of the present invention is defined by the appended claims.

Claims

1. A deep forgery video identification method based on frequency domain information and multi-task learning, characterized in that, Includes the following steps: Step 1: Decompose the deepfake video into image frames to obtain RGB three-channel images. Then, use the discrete cosine transform in frequency domain analysis to operate on the RGB three-channel images. Combined with block processing, retain some spatial information of the RGB three-channel images to obtain the preprocessed frequency domain features as input data. Step 2: Use a multi-task learning deep neural network to extract features from the input data obtained in Step 1. The multi-task learning deep neural network model uses the Xception network as the backbone network and designs an image segmentation module based on deconvolution operation to predict regions containing forgery traces in RGB images. A feature fusion module is also designed to fuse the features extracted by the segmentation module with the features extracted by the backbone network to obtain fused features. The fused features contain hundreds of dimensions of attributes and are called fused high-dimensional features. The fused high-dimensional features are entered into the classifier in the classification module to calculate the classification result. The structure of the deep neural network for multi-task learning is as follows: The structure of a deep neural network for multi-task learning includes a backbone network module, a segmentation module, and a classification module. The backbone network module adopts the structure of the Xception model, retaining only the features output by the 12th convolutional module in the original Xception model without using the Xception model's classifier. The output features are then input into the segmentation and classification modules. The segmentation module contains four sets of deconvolutional modules, each containing one deconvolutional layer, one convolutional layer, and one activation layer. The segmentation module is responsible for further refining the features and then passing the refined features to the classification module. The classification module obtains the outputs of the backbone network module and the segmentation module respectively, performs feature fusion based on convolution operation, and then inputs the fused features into the classifier part of the classification module to judge the authenticity of the video. Step 3: Transform the relationship between the fused high-dimensional features into the geometric distance of a sphere under three-dimensional conditions. The fused high-dimensional features obtained in Step 2 are divided into high-dimensional features of the real video and high-dimensional features of the fake video. The high-dimensional features of the real video are constrained to the vicinity of the center of the three-dimensional sphere, and the high-dimensional features of the fake video are constrained to the surface of the three-dimensional sphere and kept at a buffer distance from the features of the real video. By designing an optimized training-guided target algorithm, the training of a deep neural network model for multi-task learning is realized, and a deep fake video identification model is obtained. This model is used to complete the identification of deep fake videos.

2. The deepfake video identification method based on frequency domain information and multi-task learning according to claim 1, characterized in that: In step 1: the frequency domain features after block processing As shown below: in Refers to a local RGB image with sides of 8 pixels. It is a two-dimensional discrete cosine transform.

3. The deepfake video identification method based on frequency domain information and multi-task learning according to claim 1, characterized in that: In step 3: the optimized training guidance algorithm is implemented using a loss function, which is expressed by the following formula: in To ultimately optimize training and guide the learning process. and To balance the parameters, in actual use, they are taken as 1.0 and 5.0 respectively. The cross-entropy loss function is the relationship between the predicted result and the actual category. These represent the model output and the pre-labeled actual classification results, respectively. , and These are predefined control parameters, which are actually used in practice at values ​​of 2.0, 0.5, and 0.

9. The angle between `pred` and the parameters corresponding to `label` in the classifier of the classification module. Since this problem is a binary classification problem to distinguish between true and false cases, the class opposite to `label` is called `other`. The angle between the parameter corresponding to "other" in the classifier and "pred".

Images