A Deepfake Detection Method Based on Artifact Noise

Abstract

The present invention is a deepfake detection method based on artifact noise. The method comprises the following steps: (1) Face extraction: dividing a video into frames, extracting the face part from each frame image, and saving it as the original face image P of only-face; (2) DIP generation: inputting the original face image P into a DIP generation model, performing iterative fitting, and obtaining an image P* with specific noise; (3) Noise extraction: subtracting the original face image P from the image P* with specific noise to obtain artifact noise d; (4) Classification detection: inputting the artifact noise d into a classification convolutional neural network for training to obtain a classification model M; (5) Model test and inference: extracting the artifact noise d from a test video according to steps (1)-(3), inputting the artifact noise into the classification model M, and obtaining a classification result of deepfake detection. This method does not need to find specific differences of a specific model, and judges according to the extracted difference features, and has strong generalization ability.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and particularly relates to a deepfake detection method based on artifact noise. Background Art

[0002] In recent years, with the gradual development of artificial intelligence, the deepfake technology has developed rapidly and been widely spread and applied, becoming an inevitable risk in the era of artificial intelligence. Deepfakes utilize artificial intelligence and generative deep learning algorithms to realistically simulate and forge multimedia content such as news texts, images, and audio and video. Due to its high degree of authenticity, wide applicability, and low usage threshold, deepfakes can easily be exploited by malicious users and used for activities such as creating and spreading highly fraudulent false information. Among them, AI face-swap is the most representative and harmful. AI face-swap confuses the public by replacing the portraits of people, greatly violating the rights and interests of others, and even being able to manipulate public opinion, posing a severe challenge to social stability and network security. Therefore, the detection technology for related deepfakes led by AI face-swap has been widely studied.

[0003] Under the current situation where the authenticity of AI face-swap deepfakes has exceeded the cognitive discrimination ability of normal humans and is still developing at a high speed, and in the realistic background of the widespread malicious abuse of AI face-swap deepfakes, the forgery detection technology has broad development prospects and necessary practical needs. The related research in the field of deepfake detection has grown explosively with the wide spread of the open-source AI face-swap project on the Internet in 2017. The detection of forged videos can, on the one hand, promote further research on the forgery traces of generative adversarial networks and synthetic video forgery traces, and on the other hand, it also meets the current public's realistic need for forgery identification. Using this technology, targeted detection of videos on the Internet can be carried out to identify the false information therein.

[0004] Deepfake detection technology mainly goes through steps such as feature extraction, model establishment, and detection classification. The key to determining the detection performance lies in how to select relevant features that can effectively distinguish real and fake faces. Currently, the mainstream deepfake detection technology mainly realizes detection by looking for pixel distributions that are difficult to observe with the naked eye between forged products and real images or videos, and relying on machine learning to distinguish pixel-level differences. These differences are generally called artifacts or noise. These works effectively locate the noise for the detection model by using the inherent differences in texture, color, brightness, etc. between forged images and real images. Further work in this direction has found that the generative models for realizing deepfakes will leave specific noise on the generated content. This kind of noise is similar to a fingerprint. Different types of generative models and the slight differences existing when training the generative models can all lead to different fingerprint features in the image. Therefore, it is possible to judge whether it is generated by deepfake by identifying the fingerprint of the image or video generative model. Given this characteristic, many works will look for specific noise in the image to judge whether the image is generated by the corresponding specific model. However, the generalization of this type of method is poor. Since the specific generation method of the forged product cannot be determined, it is often limited to detecting deepfakes generated by specific models. At the same time, although the existing methods try to find these artifacts, noise, and fingerprints, there is no way to explain these key features that distinguish the original content from the forged products. This leads to problems such as poor generality, poor interpretability, and inability to effectively detect various different types of deepfake images in the existing detection methods. Summary of the Invention

[0005] The purpose of the present invention is to provide a detection method for deepfake recognition by extracting noise alternative features based on a generative network. The present invention uses a general method to extract noise and finally concludes that these noises are essentially a kind of difference. At the same time, through deep learning technology, the extracted differences are recognized, and the features and identification of the differences are directly analyzed to achieve more accurate prediction of deepfakes and stronger generalization. It solves the problems of poor generalization and low detection accuracy caused by the insufficient use of artifact noise, which is the key to detection, in visual deepfake detection methods such as AI face swapping.

[0006] The technical solution for realizing the purpose of the present invention is: a deepfake detection method based on artifact noise, including the following steps:

[0007] Step (1): Face extraction: Divide the input video into frames, extract the face part from each frame image F, and save it as the original face image P of only-face.

[0008] Step (2): DIP Generation: Input the original face image P extracted in step (1) into the DIP generation model for iterative fitting to obtain the newly generated image P with specific noise. * ;

[0009] Step (3): Noise Extraction: Subtract the original face image P from the specific noise image P generated by DIP in step (2) to obtain the artifact noise d. * ;

[0010] Step (4): Classification Detection: Input the artifact noise d extracted in step (3) into the classification convolutional neural network for training to obtain the trained classification model M.

[0011] Step (5): Model Testing and Inference: Extract the artifact noise d from the test video in the same way as in steps (1)-(3), and input the obtained artifact noise into the trained classification model M in step (4) to obtain the classification result of deepfake detection.

[0012] Further, step (2) is specifically as follows:

[0013] Scale the original face image P of only-face extracted to a fixed size (x, y), and each scaled image P ∈ R x×y×3 is directly input into the DIP generation model without adding noise manually.

[0014] The backbone part of the DIP network uses the hourglass structure. The generation model extracts the prior features of the image itself through multiple layers of downsampling and skip structures for encoding, and reconstructs and generates the image through the decoder of upsampling and convolution. The specific formula is as follows:

[0015] θ * = min E(f θ (z); P)

[0016]

[0017] where θ * is the network parameter initialized by the machine and the optimal solution of the parameter obtained through training. P is the input original image P, z is a set of fixed random encodings initially input into the network, with the same size as the image P but different channels. Adopting the idea of the GAN model, P * is the final output, that is, the image P with specific noise. * ;

[0018] In each iteration process, evaluate the similarity between P and P * and update the parameters to generate the image P with specific noise * to replace the unobtainable real image P. 0;

[0019] Using the generated image P with specific noise according to the learning rate lr = 0.01 * and the mse of the original face image P of only-face as the loss function, perform iterative fitting under the adam optimizer to obtain a newly generated image with specific noise, and package the data into an npz file package;

[0020] Further, step (3) is specifically as follows:

[0021] Read the npz file package packed in step (2). When reading, assign labels to each package according to the specific deepfake category during packaging, and organize it into a dictionary of category - {original image, DIP-generated image, image number};

[0022] According to the above dictionary, subtract the original face image P from the DIP-generated image P with specific noise * to obtain the noise d(P, P * ) as the feature x, label the feature of each image with the category as y, make training sets, validation sets and test sets, create a dataset using the dataset class, and encapsulate it with dataloader.

[0023] Further, step (4) is specifically as follows:

[0024] Input the training set and validation set containing the extracted artifact noise into a classification convolutional neural network for training. Use the cross-entropy loss function, Leaky ReLU as the activation function, obtain an 8-dimensional feature vector after 4 layers of convolution, put it into the classifier to get the result, with lr = 10 -3 Train for 1000 epochs to obtain a trained classification model.

[0025] Further, step (5) is specifically as follows:

[0026] Process the test set sample features in the same way as the training set in step (4), input them into the trained classification model, obtain the classification result of deepfake detection, and identify whether it is a forged product.

[0027] Compared with the prior art, the remarkable advantages of the present invention are:

[0028] (1) The present invention specifically analyzes the difference between counterfeit products and authentic content, and proposes the conclusion that the difference between counterfeit products and authentic content is a kind of artifact noise, which provides a qualitative explanation for the object that should be mainly studied and analyzed in the deep counterfeit detection task (i.e., artifact noise); at the same time, the present method also draws the conclusion that the artifact noise as the difference is the key feature for classification and identification; the explanation and conclusion proposed by the present method can promote further focus on the research on artifact noise in the deep detection task.

[0029] (2) The present invention takes artifact noise as the key feature for deep fake detection as its starting point, proposes a method for realizing accurate deep fake detection by using an image generated by a DIP model as a reference image, replacing the image to be tested to calculate the difference, and extracting a substitute for artifact noise, and proves the rationality of this method; since the artifact noise feature d input into the classification model of the present invention is a key identification feature that can distinguish between authenticity, it can further improve the learning ability of the classification model and enhance the accuracy of deep fake detection; at the same time, the detection model of this method does not need to look for specific differences of a specific model, but judges based on the extracted difference features, and has strong generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the DIP flow chart of the present invention.

[0031] Figure 2 It is a structural diagram of DIP downsampling, skip connection and upsampling of the present invention.

[0032] Figure 3 The invention discloses an original face image, a mask, a generated image and a difference display image.

[0033] Figure 4 It is the classification detection model of the present invention.

[0034] Figure 5 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0036] The main idea of ​​the present invention is:

[0037] Assuming that the original image in nature is P, when it is converted to a computer, it has already been processed many times, including discretization of RAW images, color interpolation, and JPEG compression. This series of processing is collectively referred to as Δ. After an original image is processed, the image presented to the user is already a processed image P′. The processed image can be regarded as the original image with processing added, and P′=P+Δ. People generally cannot find the difference between the original image and the processed image, so it can be naturally deduced that the two are very similar, and P′≈P. Their difference d(P′,P) is defined as noise or fingerprint. One of the simplest ways to define noise is the difference between the processed image and the original image, that is: d(P′,P)=P′-P.

[0038] In the field of deepfakes, any generative model G (defined here in a broad sense, including a series of operations to generate results) used to generate fake products is intended to generate a result P that is closer to the original image P. G However, due to the limitations of the model itself (such as the limited receptive field size of the convolution kernel, the inability to establish overall pixel-to-pixel associations, or the limitation of convolution parameters), some unnecessary information will inevitably be discarded, and the P obtained by the generation model will be G Only infinitely close to P, with P G ≈P, but it cannot be exactly the same as P, so P G =G(ω) also has noise, which can be expressed as d(P G ,P).

[0039] For different generative models G 1 With G 2 , the fake images they generate are P G1 and P G2 , due to their different implementation methods, the noise they generate is also biased, and the noise of the same generative model is often similar, so this noise deviation can be regarded as the fingerprint of the recognition model, so there is

[0040] The present invention intends to perform deep forgery detection based on the noise difference described in 1 to 3, but it is necessary to extract the image P to be tested. T The difference d(P T ,P) is unrealistic because it is impossible to obtain the original image P without any processing. Therefore, the present invention chooses to use an alternative representation of the difference instead of the unobtainable difference for deep fake detection.

[0041] The present invention converts the image P to be tested into T Input the DIP (Deep Image Prior) generation model to obtain the reference image P D DIP is essentially a generative model similar to the deep fake model.T There are also specific differences with P D , and the difference d(P T , P D ).

[0042] Under the definition of subtracted noise, for the image P to be measured T and the reference image P D , the difference can be expressed as: the difference d(P G , P) between the image to be measured and the original image and the difference d(P D , P) between the reference image and the original image, that is, d(P T , P D ) = d(P T , P) - d(P D , P). And since all Ps D are generated by the known DIP model, the present invention believes that the difference between the image to be measured and the reference image is positively correlated with the difference between the image to be measured and the original image, that is, there is d(P T , P D ) ~ d(P T , P). Therefore, it is concluded that d(P T , P D ) can be used to replace d(P T , P), that is, the alternative representation of noise can be extracted through DIP.

[0043] After extracting the difference, this method uses d(P T , P D ) as the discriminative feature required for classification, and establishes a classification model to distinguish deep fakes from real images.

[0044] Embodiment

[0045] The specific implementation of the present invention mainly includes 1) extraction of artifact noise; 2) distinguishing authenticity according to artifact noise, and the specific steps are as follows:

[0046] Step (1): For a video V to be measured, the read video V is divided into frames (in order to improve the detection efficiency, a certain number of frame images can be selected), and the face contour in the frame image F is extracted using the Python face recognition library dlib. Then, according to the face contour, the original frame image F is cropped, and the content of the face part is retained, and all the non-face parts remaining in the cropped image are covered to eliminate the interference formed by other non-related regions. Finally, it is saved as an image P of only-face.

[0047] Step (2): Scale the extracted original image P of only-face to a fixed size (x, y), such as (256, 256). Each scaled image P ∈ R x×y×3Directly input into the DIP generation model without adding artificial noise. In the DIP network, the backbone part uses the hourglass structure shown in Appendix Figure 1 and Appendix Figure 2 . This generation model is a self-encoder decoder similar to a GAN. It encodes the prior features of the image itself by performing multi-layer downsampling and skip structures on the image, and reconstructs and generates the image through an upsampling and convolutional decoder. The specific formula is as follows:

[0048] θ * = min E(f θ (z); P)

[0049]

[0050] where θ * is the network parameter based on random initialization, the optimal solution of the parameter obtained through training, P is the original image P we input, z is a fixed set of random encodings initially input into the network (the same size as the image x 0 , but with inconsistent channels, adopting the idea of the GAN model), P * is the final output, that is, the generated image P * .

[0051] In each iteration process, evaluate the similarity between P and P * and update the parameters to further generate P * similar to P. After a large number of iterations, DIP can generate an image consistent with the image P to be measured. However, the purpose of the present invention is to generate an image P * with specific noise to replace the unobtainable real image P 0 . Obtain the noise through the relationship of d(P, P * ) ~ d(P, P 0 ), so only a certain number of iterations are required.

[0052] According to the learning rate lr = 0.01, use the mse of the generated image P * and the image P to be measured as the loss function, and perform 500 iterations of fitting under the adam optimizer to obtain a newly generated image with specific noise. The generated image P * , together with the original image P, face mask Mask, etc., are packaged into an npz file according to the specific deepfake type and saved for deepfake detection. The above operations need to be performed for each picture;

[0053] Step (3): Read the npz file package generated by DIP. When reading, assign a label to each package (image) according to the specific deepfake category when packaging. The labels include: deepfakes, neuraltextures, face2face, faceswap and real, organized into a dictionary of categories - {original image, DIP generated image, image number}. According to the above dictionary, compare the image to be tested P with the image generated by DIP P * The noise d(P,P * ) as feature x, use the category to label the feature of each image y, create training set, validation set and test set, use dataset class to create dataset, and use dataloader to encapsulate with batch_size=200. The above steps (1) to (3) complete the first main content of this experiment and complete the work of artifact noise extraction in this invention.

[0054] Step (4): The training set and validation set containing the extracted artifact noise are input into the classification convolutional neural network for training. The core idea of ​​the present invention is to use the direct analysis of the artifact noise as the core difference between the true and false images to achieve high-precision deep fake detection. The cross entropy loss function and Leaky ReLU are used as the activation function. After 4 layers of convolution, an 8-dimensional feature vector is obtained, which is put into the classifier to obtain the result. The lr=10 -3 Train for 1000 epochs. Since the input noise feature is the key identification feature that can distinguish true from false, it can further improve the learning ability of the classification model.

[0055] Step (5): Process the sample features of the test set in the same way as the training set and input them into the trained classification model to obtain the classification results of deep fake detection and identify whether the product is a counterfeit.

[0056] The above steps (4) to (5) complete the second main content and main goal of this experiment. By directly performing feature engineering and classification on the extracted artifact noise, a higher precision discrimination is achieved.

Claims

1. A method for deepfake detection based on artifact noise, characterized in that, it includes the following steps: Step (1): Face extraction: Divide the input video into frames, extract the face part from each frame image F, and save it as the original face image P of only-face; Step (2): DIP generation: Input the original face image P extracted in step (1) into the DIP generation model for iterative fitting to obtain the newly generated image P with specific noise * ; Step (3): Noise extraction: Subtract the specific noise image P generated by DIP in step (2) from the original face image P * to obtain the artifact noise d; Step (4): Classification detection: Input the artifact noise d extracted in step (3) into a classification convolutional neural network for training to obtain a trained classification model M; Step (5): Model test inference: Extract the artifact noise d from the test video in the same way as in steps (1)-(3), and input the obtained artifact noise into the trained classification model M in step (4) to obtain the classification result of deepfake detection; Further, step (2) is specifically: The original face image P of the extracted only-face is scaled to a fixed size (x, y), and each scaled image P ∈ R x×y×3 is directly input into the DIP generation model without adding noise manually; The backbone part of the DIP network uses an hourglass structure. The generation model encodes the prior features of the image itself by performing multi-layer downsampling and skip structures on the image, and reconstructs and generates the image through an upsampling and convolutional decoder. The specific formula is as follows: θ * = min E(f θ (z); P) where θ * is the optimal solution of the parameters obtained through training based on randomly initialized network parameters. P is the original input image P, and z is a set of fixed random codes initially input into the network, which has the same size as the image P but different channels. Adopting the idea of the GAN model, P * is the final output, that is, the image P with specific noise * ; In each iteration, evaluate the similarity between P and P * and update the parameters to generate an image P * with specific noise to replace the unobtainable real image P 0 ; Using the generated image P with specific noise according to the learning rate lr = 0.01 * and the mean squared error (MSE) between the original face image P of only-face and the generated image as the loss function, perform iterative fitting under the Adam optimizer to obtain a newly generated image with specific noise, and package the data into an npz file package.

2. The method according to claim 1, characterized in that, step (3) is specifically: Read the npz file package packed in step (2). When reading, assign labels to each package according to the specific deepfake category during packing, and organize it into a dictionary of category-{original image, DIP-generated image, image number}; According to the above dictionary, the original face image P and DIP are used to generate an image P with specific noise. * The difference is obtained to get the noise f(P, P * ) as the feature x, and the features of each image are labeled with the category y. Then, a training set, a validation set, and a test set are made. The dataset is created using the dataset class and encapsulated with the dataloader.

3. The method according to claim 2, characterized in that, step (4) is specifically: The training set and validation set containing the extracted artifact noise are input into a classification convolutional neural network for training. The cross-entropy loss function is used, and Leaky ReLU is used as the activation function. After 4 layers of convolution, an 8-dimensional feature vector is obtained and put into the classifier to get the result, with lr = 10 -3 Train for 1000 epochs to obtain a trained classification model.

4. The method according to claim 3, characterized in that, step (5) is specifically: Process the test set sample features in the same way as the training set in step (4), input them into the trained classification model, obtain the classification result of deepfake detection, and identify whether it is a forged product.

Images