A method for detecting tampered face videos and images based on improved Siamese network

Abstract

A method for detecting tampered face videos and images based on an improved twin network belongs to the field of computer image processing technology. Technical solution: S1, pre-processing module Pre-processing, cropping the face area and background area of ​​the image to obtain face image blocks and background image blocks respectively; S2, feature extraction module Feature extraction, using the improved twin network to extract image features, and obtain the grayscale spatial features of the face image block and the background image block; S3, feature alignment module Feature alignment, using the feature splicing method to splice its grayscale spatial features, and classify to obtain the predicted label Y; S4, using the voting principle to correct the classification and obtain the result label Yt. This method uses an improved twin network to extract the features of the face area and background area of ​​the image, and then measures their similarity, and finally classifies the true and false images according to the similarity. At the same time, a voting principle is proposed to correct the final classification result to make the result more accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer image processing, and in particular to a method for detecting tampered face videos and images based on an improved twin network. Background Art

[0002] In recent years, with the development of science and technology, social media and short video platforms have risen rapidly, occupying most of people's entertainment time. More and more people like to share images and videos of their daily lives on various social media and short video platforms. These images and videos will spread rapidly on the Internet and become the main carrier for people to communicate and share information with each other. However, image tampering technology, especially face image tampering technology, has brought certain negative effects to the dissemination of images and videos. Tampered face images and videos may be used for improper purposes, such as blackmail, illegally obtaining citizens' personal privacy and property through facial information, spreading negative news, and even making vulgar videos. After the videos of some important political celebrities are manipulated, they may even seriously damage national and social security. The widespread dissemination of tampered images and videos will also reduce citizens' trust in the news media industry and make the public more susceptible to conspiracy theories. Therefore, the detection of tampered face images and videos has very important practical significance and value.

[0003] With the rapid development of deep learning and generative adversarial networks, the current face tampering technology has become very mature. The main tampering methods include Deepfake, Face2Face, Faceswap and Neural Textures.

[0004] Deepfake is based on two autoencoders with a shared encoder that reconstruct training images of the source and target faces, respectively. The encoder and decoder trained on the source face are then applied to the target face to generate a fake image. The output of the autoencoder is then blended with the rest of the image using Poisson image editing. Face2Face is a facial reenactment system that transfers the expression of a source video to a target video while maintaining the identity of the target person. FaceSwap is a computer graphics-based method that uses detected facial landmarks to fit a 3D face model that generates a fake image by back-projecting it onto the target image using the texture of the input image to minimize the difference between the projected shape and local landmarks. Neural Textures is a technique that uses expression transfer to modify the facial texture map of the target video to match the expression of the source target to generate a fake video. Although these techniques all rely on advanced image processing algorithms, there are now some integrated software or program frameworks that can be used by people without professional knowledge to quickly process facial images, making face forgery easy and common.

[0005] In order to combat the harm and negative impact of face image tampering, people have proposed a variety of methods to detect tampered face images and videos. There are currently two main detection methods, one is to use traditional manual features, and the other is to use neural network to extract features. Compared with the features extracted by neural networks, manual features are clearer and more robust, more interpretable, and can better utilize the fine-grained information of the image. Therefore, this method has achieved good accuracy on certain specific data sets, but on larger data sets, the neural network method can achieve better results. Neural networks can learn more and richer image features, unlike the method using manual features that relies on specific features. Therefore, this type of method has received more and more attention and has been widely used. The detection of forged faces can be regarded as a classification task, and the accuracy of detection depends on the selection of features and the selection of classifiers. The process of all detection methods can be roughly divided into two stages: feature extraction and classification. Some methods divide these two stages into two independent sub-problems, while others integrate these two stages in an end-to-end manner based on deep neural networks. In the feature extraction stage, there are two main feature selections, one is based on the features of a single image, and the other is based on the features between video frames. Since methods based on single image features can also be used to determine the authenticity of videos, there are many studies on this type of method. In the classification stage, the main choices of classifiers are SVM, CNN, RNN, MLP, etc., among which SVM and CNN are more widely used.

[0006] Existing methods have achieved good detection accuracy on some public datasets, but there are still some problems to be solved. The first problem is that most methods have poor robustness. For uncompressed or lightly compressed images and videos, existing methods can achieve satisfactory detection accuracy, but when forged face images and videos are highly compressed, the detection accuracy of existing methods is greatly reduced. Because the compression operation can eliminate the traces of forgery at the feature level and reduce the difference in feature distribution between real images and fake images, and this feature difference is exactly the detection principle that most existing methods rely on. However, in real applications, the quality of images and videos may be relatively low after a variety of post-processing operations similar to compression, which greatly affects the performance of existing methods. The second problem is that almost all methods only use the features of the face area or the boundary area where the face merges with the background, while ignoring the features of the background. Although the tampering algorithm only tampered with the face area of ​​the image and video, it is worth noting that in the real image without tampering, the face area and the background area are consistent at a certain feature level, because they come from the same camera and the subsequent processing operations are also completely consistent, while the forged image does not have such a feature. Therefore, background features are actually very useful, and it is crucial to find a feature space in which the face area and the background area are consistent in the real image. Summary of the invention

[0007] In order to solve the problem of accuracy in detecting true and false videos, the present invention proposes: a method for detecting tampered face videos and images based on an improved twin network, and the technical solution is as follows:

[0008] S1, pre-processing module Pre-processing, cropping the face area and background area of ​​the image to obtain face image blocks and background image blocks respectively;

[0009] S2, feature extraction module Feature extraction, uses the modified twin network to extract image features and obtain the grayscale spatial features of the face image block and the background image block;

[0010] S3, feature alignment module, uses the feature splicing method to splice its grayscale spatial features, and classifies them to obtain the predicted label Y;

[0011] S4. Use the voting principle to correct the classification and obtain the result label Yt.

[0012] Furthermore, in step S1, the feature similarity of the face area and the background area of ​​the image is measured using the twin network, and in the data set of the pre-processing module Pre-processing, all video frame images are cropped into face image blocks and background image blocks of the same size, and the face image blocks and background image blocks are stored in different folders respectively, and the image blocks of the same video frame are stored in the same folder; the process of step S1 is as follows:

[0013] S11. For each video in the dataset, use the software package dlib to detect the face area of ​​each frame in the video, and crop a fixed-size face image block based on the center of the face;

[0014] S12, cutting out n background image blocks of the same size as the face image block from the background area of ​​the image, where n is an odd number;

[0015] S13, creating a multi-level folder to store the processed face image blocks and background image blocks.

[0016] Furthermore, in step S13, for each sub-data set, a folder F is created to store corresponding data, and two folders Fr and Ft are created in the folder F to store the data of the real video and the tampered video respectively; in Fr and Ft, N folders Fk are created (k=1, 2, ..., N), where N is the number of real videos or tampered videos; two folders Ff and Fb are created in Fk to store the cropped face image blocks and background image blocks of each video respectively; in the training stage, an image block is randomly selected from the Fb and Ff folders of the Fk folder to form a set of image block pairs, IB represents the background image block, and IF represents the face image block.

[0017] Furthermore, in step S2: the twin network learns similarity measurement and applies it to face verification, and the network learns the similarity between the face image block and the background image block through the improved feature extraction module Feature extraction in the twin network, and the process is as follows:

[0018] S21, converting IB and IF in the image block pair into a grayscale image IBG of the background image block and a grayscale image IFG of the face image block, respectively, wherein the grayscale image reduces the influence of the image semantic content;

[0019] S22, input IBG and IFG into two Xception networks with shared weights, and obtain two sets of 512-dimensional grayscale spatial features GW(B) of the background image block and grayscale spatial features GW(F) of the face image block respectively;

[0020] An improved twin network is used to extract the features of face image blocks and background image blocks. Two identical Xception networks are used as the backbone of the twin network. Through end-to-end training and supervised learning, the convolutional neural network itself can extract appropriate and effective features in the grayscale space of the image, respectively extracting the features of face image blocks and background image blocks, and sharing parameters between the two networks.

[0021] Furthermore, in the step S3, in the feature alignment module Feature alignment, the similarity is measured by feature alignment, the features of the extracted face image blocks and background image blocks are spliced ​​together, and the final classification is performed through three fully connected layers.

[0022] Furthermore, the feature alignment method is used to measure the similarity to distinguish whether they are from real images or tampered images. The method is as follows: after obtaining the features of the grayscale space of the image blocks, namely GW(B) and GW(F), their similarity is measured in the feature alignment module, and GW(B) and GW(F) are connected to obtain a 1024-dimensional alignment feature, namely CW, which is defined as: It represents the concatenation operation. GW(B) and GW(F) are concatenated to obtain CW, which is input into the fully connected layer. The three fully connected layers have 256 nodes, 10 nodes, and 2 nodes respectively, and the binary prediction label Y is obtained.

[0023] Furthermore, in step S4, in the test phase, a voting principle is defined to correct the classification result, by cutting out n pairs of image blocks from the same frame image, where n is an odd number, and counting the classification results of each pair of image blocks, and the classification result is based on the majority result; the steps of data selection are as follows:

[0024] S41, when a face image block is randomly selected from the folder Ff storing face image blocks, n background image blocks are selected from the folder Fb storing background image blocks of the same video frame, where n is an odd number, and the face image blocks are copied n times to form n groups of image pairs;

[0025] S42, inputting the n sets of image block pairs into the trained feature extraction module and feature alignment module to obtain n binary prediction labels Yn, where n=1, 3, 5, ...;

[0026] S43. According to the voting principle of minority obeys majority, the result label Yt is obtained, that is, the classification result of the face image block and the background image block belonging to the image; Y=1 means that Fk belongs to the folder Fr, which means that a group of image block pairs come from a real video, and Y=0 means that Fk belongs to the folder Ft, which means that a group of image block pairs come from a fake video.

[0027] The beneficial effects of the present invention are:

[0028] 1. The voting principle defined in the test phase corrects the classification results, making them more accurate and achieving better detection accuracy;

[0029] 2. The aligned features retain all the feature information of the image patch pairs, so that the subsequent fully connected layer can fully explore the similarities between them, making the learning process more stable and robust, and obtaining more satisfactory performance;

[0030] 3. In a real image that has not been tampered with, the facial area and the background area are consistent at a certain feature level, and the subsequent processing operations are also completely consistent, while a forged image does not have such a feature. By utilizing this feature, the present invention not only uses the features of the facial area, but also makes full use of the features of the image background area that were ignored by previous methods. By comparing the similarity between the facial area and the background area of ​​the image, the forged image is detected. It has very good robust performance, making it crucial to find a feature space in which the facial area and the background area are consistent in the real image. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flowchart of a method for detecting tampered face videos and images based on an improved Siamese network;

[0032] Figure 2 This is the overall structural diagram of the method for detecting tampered face videos and images based on the improved twin network;

[0033] Figure 3 This is the Xception network framework diagram;

[0034] Figure 4 For the principle of voting;

[0035] Figure 5 This is a graph of test data. DETAILED DESCRIPTION

[0036] A method for detecting tampered face videos and images based on an improved twin network, comprising: Figure 1-2 The overall structural diagram of the method for detecting tampered face videos and images based on the improved twin network.

[0037] S1, pre-processing module Pre-processing, cropping the face area and background area of ​​the image to obtain face image blocks and background image blocks respectively;

[0038] S2, feature extraction module Feature extraction, uses the improved twin network to extract image features and obtain the grayscale spatial features of the face image block and the background image block;

[0039] S3, feature alignment module, uses the feature splicing method to splice its grayscale spatial features, and classifies them to obtain the predicted label Y;

[0040] S4. Use the voting principle to correct the classification and obtain the result label Yt.

[0041] In S1, the twin network is used to measure the feature similarity of the face area and the background area of ​​the image. In the data set of the pre-processing module Pre-processing, all video frame images are cropped into face image blocks and background image blocks of the same size, and the face image blocks and background image blocks are stored in different folders respectively. The image blocks of the same video frame are stored in the same folder. The process of step S1 is as follows:

[0042] S11. For each video in the dataset, use the software package dlib to detect the face area of ​​each frame in the video, and crop a fixed-size face image block based on the center of the face;

[0043] S12, cutting out n background image blocks of the same size as the face image block from the background area of ​​the image, where n is an odd number;

[0044] S13, creating a multi-level folder to store the processed face image blocks and background image blocks.

[0045] Among them, in S2, for each sub-dataset, a folder F is created to store the corresponding data. We create two folders Fr and Ft in folder F to store the data of real videos and tampered videos respectively; in Fr and Ft, we create N folders Fk, (k = 1, 2, ..., N), where N is the number of real videos or tampered videos; two folders Ff and Fb are created in Fk to store the cropped face image blocks and background image blocks of each video respectively; in the training stage, we randomly select an image block from the Fb and Ff folders of the Fk folder to form a set of image block pairs, IB represents the background image block, and IF represents the face image block.

[0046] Among them, in the step S2: the twin network learns the similarity metric and applies it to face verification, and the network learns the similarity between the face image block and the background image block through the improved feature extraction module Feature extraction in the twin network, and the process is as follows:

[0047] S21, converting IB and IF in the image block pair into a grayscale image IBG of the background image block and a grayscale image IFG of the face image block, respectively, wherein the grayscale image reduces the influence of the image semantic content;

[0048] S22, input IBG and IFG into two Xception networks with shared weights, and obtain two sets of 512-dimensional grayscale spatial features GW(B) of the background image block and grayscale spatial features GW(F) of the face image block respectively;

[0049] The improved twin network is used to extract the features of face image blocks and background image blocks. Two identical Xception networks are used as the backbone of the twin network. The Xception network is as follows: Figure 3 As shown in the figure, through end-to-end training and supervised learning, the convolutional neural network can extract appropriate and effective features in the grayscale space of the image, respectively extract the features of the face image blocks and the background image blocks, and share parameters between the two networks; the shared weights ensure that the two networks extract features from the same space, which is equivalent to enriching the feature data of each network, making the network more efficient.

[0050] Among them, in S3, in the feature alignment module Feature alignment, the measurement method of the original twin network is abandoned, and the feature alignment method is used to measure the similarity. The features of the extracted face image blocks and background image blocks are spliced ​​together, and the final classification is performed through three fully connected layers.

[0051] Due to differences in hardware and software and imperfections in manufacturing, the process of generating videos will leave a unique imprint on the video. In a video, the imprint should usually be consistent and continuous, so in real videos, the face area and the background area have a high similarity. In videos forged by Deepfake and FaceSwap, the face area is completely replaced, so the similarity between the face area and the background area is low, because the face area and the background area come from different videos; although the tampered videos generated using Face2Face and Neural Textures only modify facial expressions and individual attributes, rather than the face area directly coming from other videos, the tampering of the face will still destroy the consistency of the imprint. To this end, we use an improved twin network to measure the similarity between the face area and the background area of ​​the video frame. We use the Xception network as the backbone of the twin network. Through end-to-end training and supervised learning, we let the convolutional neural network extract suitable and effective features in the grayscale space of the image.

[0052] Furthermore, the feature alignment method is used to measure the similarity to distinguish whether they are from real images or tampered images. The method is as follows: after obtaining the features of the grayscale space of the image blocks, namely GW(B) and GW(F), their similarity is measured in the feature alignment module, and GW(B) and GW(F) are connected to obtain a 1024-dimensional alignment feature, namely CW, which is defined as: It represents the concatenation operation, concatenating GW(B) and GW(F) to obtain CW, which is input into the fully connected layer. The three fully connected layers have 256 nodes, 10 nodes, and 2 nodes respectively, and the binary prediction label Y is obtained. The most direct way to align features is to use the original twin network to perform residual operations on the two feature maps, but this is not suitable for image blocks with large differences in semantic content.

[0053] Among them, in S4, in the test phase, a voting principle is defined to correct the classification results. By cutting out n pairs of image blocks from the same frame image, n is an odd number, and counting the classification results of each pair of image blocks, the classification result is based on the majority result; the steps of data selection are as follows:

[0054] S41, when a face image block is randomly selected from the folder Ff storing face image blocks, n background image blocks are selected from the folder Fb storing background image blocks of the same video frame, where n is an odd number, and the face image blocks are copied n times to form n groups of image pairs;

[0055] S42, inputting the n sets of image block pairs into the trained feature extraction module and feature alignment module to obtain n binary prediction labels Yn, where n=1, 3, 5, ...;

[0056] S43. According to the voting principle of minority obeys majority, the result label Yt is obtained, that is, the classification result of the face image block and the background image block belonging to the image; Y=1 means that Fk belongs to the folder Fr, which means that a group of image block pairs come from a real video, and Y=0 means that Fk belongs to the folder Ft, which means that a group of image block pairs come from a fake video.

[0057] like Figure 5 As shown in the figure, IB1, IB2, and IB3 are three background image blocks of the same video frame, and IF is a face image block. Y1, Y2, and Y3 represent the test result labels of the three image pairs, and Yt represents the final test result label after voting. Yt = 1 indicates that the image is a real image, and Yt = 0 indicates that the image is a tampered image. Table 1 illustrates the voting principle between Yt and the labels of the three sets of image block pairs.

[0058] Table 1 Voting results

[0059]

[0060] After repeated experiments, the experimental data obtained in the test phase are as follows Figure 4As shown, DP in Chinese is Deepfake, F2 in Chinese is Face2Face, FS in Chinese is FaceSwap, NT in Chinese is Neural Textures, these four are the names of image tampering methods, LQ in Chinese is Low Quality, HQ in Chinese is High Quality, Raw in Chinese is original quality, these three represent the image quality of the video;

[0061] Meso4, MesoInception4, Xception, Our method, Our method|(voting) respectively represent the network models used, Meso4, MesoInception4, Xception are network models in the prior art, Our method is the twin network model of the present invention, and Ourmethod|(voting) is a twin network model using the voting principle;

[0062] It can be seen from the experimental data that the present invention improves the robustness to a great extent.

[0063] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for detecting tampered face videos and images based on an improved twin network, characterized in that: S1, pre-processing module Pre-processing, cropping the face area and background area of ​​the image to obtain face image blocks and background image blocks respectively; S2, feature extraction module Feature extraction, uses the modified twin network to extract image features and obtain the grayscale spatial features of the face image block and the background image block; S3, feature alignment module, uses the feature splicing method to splice its grayscale spatial features, and classifies them to obtain the predicted label Y; S4, use the voting principle to correct the classification and obtain the result label Yt; In step S2, the twin network learns similarity measurement and applies it to face verification. The improved feature extraction module Feature extraction in the twin network enables the network to learn the similarity between the face image block and the background image block. The process is as follows: S21, converting the background image block IB and the face image block IF in the image block pair into a grayscale image IBG of the background image block and a grayscale image IFG of the face image block, respectively, wherein the grayscale image reduces the influence of the image semantic content; S22, input IBG and IFG into two Xception networks with shared weights, and obtain two sets of 512-dimensional grayscale spatial features GW(B) of the background image block and grayscale spatial features GW(F) of the face image block respectively; The improved twin network is used to extract the features of face image blocks and background image blocks. Two identical Xception networks are used as the backbone of the twin network. Through end-to-end training and supervised learning, the features of face image blocks and background image blocks are extracted respectively. The parameters are shared between the two networks. In the step S3, in the feature alignment module, the similarity is measured by feature alignment, the features of the extracted face image block and the background image block are spliced ​​together, and the final classification is performed through three fully connected layers; The feature alignment method is used to measure the similarity to distinguish whether they are from real images or tampered images. The method is as follows: after obtaining the features of the grayscale space of the image blocks, namely GW(B) and GW(F), their similarity is measured in the feature alignment module, and GW(B) and GW(F) are connected to obtain a 1024-dimensional alignment feature, namely CW, which is defined as: It represents the concatenation operation. GW(B) and GW(F) are concatenated to obtain CW, which is input into the fully connected layer. The three fully connected layers have 256 nodes, 10 nodes, and 2 nodes respectively, and the binary prediction label Y is obtained.

2. The method for detecting tampered face videos and images based on the improved twin network as claimed in claim 1, characterized in that: In the step S1, in the data set of the pre-processing module Pre-processing, all video frame images are cut into face image blocks and background image blocks of the same size, and the face image blocks and background image blocks are respectively stored in different folders, and the image blocks of the same video frame are stored in the same folder; the process of step S1 is as follows: S11. For each video in the dataset, use the software package dlib to detect the face area of ​​each frame in the video, and crop a fixed-size face image block based on the center of the face; S12, cutting out n background image blocks of the same size as the face image block from the background area of ​​the image, where n is an odd number; S13, creating a multi-level folder to store the processed face image blocks and background image blocks.

3. The method for detecting tampered face videos and images based on the improved twin network as claimed in claim 2, characterized in that: In step S13, For each sub-dataset, create a folder F to store the corresponding data, and create two folders Fr and Ft in folder F to store the data of real videos and tampered videos respectively; in Fr and Ft, create N folders Fk, k = 1, 2, ..., N, where N is the number of real videos or tampered videos; create two folders Ff and Fb in Fk to store the cropped face image blocks and background image blocks of each video respectively; in the training stage, randomly select an image block from the Fb and Ff folders of the Fk folder to form a set of image block pairs, IB represents the background image block, and IF represents the face image block.

4. The method for detecting tampered face videos and images based on an improved twin network as claimed in claim 1, characterized in that: In step S4, in the test phase, a voting principle is defined to correct the classification result, by cutting out n pairs of image blocks from the same frame image, where n is an odd number, and counting the classification results of each pair of image blocks, and the classification result is based on the majority result; the steps of data selection are as follows: S41, when a face image block is randomly selected from the folder Ff storing face image blocks, n background image blocks are selected from the folder Fb storing background image blocks of the same video frame, where n is an odd number, and the face image blocks are copied n times to form n groups of image pairs; S42, inputting the n sets of image block pairs into the trained feature extraction module and feature alignment module to obtain n binary prediction labels Yn, where n=1, 3, 5, ...; S43. According to the voting principle of minority obeys majority, the result label Yt is obtained, that is, the classification result of the face image block and the background image block; Y=1 means that Fk belongs to the folder Fr, which means that a group of image block pairs come from a real video, and Y=0 means that Fk belongs to the folder Ft, which means that a group of image block pairs come from a fake video.

Images