Identity-invariant feature-based deep fake face detection method, system and device

Abstract

The application discloses a kind of based on identity invariant feature's deep counterfeit face detection method, system and equipment.This application first pre-processes several original face images and corresponding counterfeit face images into the size of 112x112x3 and respectively input into ResNet50 network, obtains the identity feature of face image, then using ArcFace loss function supervises the identity feature extracted by original face and counterfeit face to keep consistent.Finally, the counterfeit face image to be detected and corresponding target face image are respectively input into the ResNet50 main feature extraction network that has been trained, respectively obtain the identity feature f and g of face image, by calculating the cosine similarity of f and g, obtain the identity similarity of counterfeit face and target face, the picture that identity similarity is less than preset value is counterfeit face picture.This application can effectively extract the identity invariant feature of true and false face, with more accurate deep counterfeit face detection effect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence security, and relates to a deep fake face detection method, system and device, in particular to a deep fake face detection method, system and device based on identity-invariant features. BACKGROUND

[0002] Deepfake is a technology that creates or synthesizes fake content (such as images, videos, and sounds) based on intelligent methods such as deep learning. In recent years, with the development of deep learning technology, deepfake is developing at an unprecedented speed. Currently, deepfake technology can not only generate face-swapped images and mimic real people's actions and expressions, but also create characters that do not exist in reality and are difficult to distinguish, overturning the traditional concept of "seeing is believing".

[0003] Once misused, deepfake technology will cause great harm to individuals, society, and the state. In view of the potential risks of deepfake technology, countries have introduced relevant laws and regulations to prevent and control deepfake technology.

[0004] The best way to combat "deepfake" is "deepfake detection". The purpose of deepfake detection technology is to detect whether an image or video has been forged by deepfake technology. The current mainstream detection methods include detection methods using traditional image features and detection methods based on deep learning. With the development of deep learning technology, more and more novel deepfake detection technologies are being applied. Researchers extract deep features from images by constructing different convolutional neural network structures and use deep features to determine whether a face has been deepfake. In order to improve the expression ability of features, researchers continue to propose new network architectures, including Xception network and residual network. Some researchers also introduce frequency domain information into the convolutional neural network, increasing the expression ability of the features.

[0005] However, although these convolutional neural network structures can well extract the main features of the image, these single-sample input networks are prone to focus on feature expressions that are irrelevant to the authenticity of the picture, such as background features, skin color features, etc., and it is difficult to capture intrinsic feature expressions related to the authenticity of the picture, especially when multiple pictures have similar appearances but different authenticity attributes. These networks are prone to extract similar image features, thereby affecting the accuracy of detection. SUMMARY

[0006] To solve the above technical problems, the present application proposes a deep fake face detection method, system and device based on identity-invariant features.

[0007] The technical scheme adopted by the method of the present application is: a deep fake face detection method based on identity-invariant features, comprising the following steps:

[0008] Step 1: pre-process the fake face image to be detected and the target face image into a preset size of 112x112x3, and then input them into a ResNet50 backbone feature extraction network respectively to obtain the identity feature f of the fake face image and the identity feature g of the target face image; wherein the target face image is a face image obtained by forging an original face image;

[0009] Step 2: calculate the cosine similarity between the identity feature f and the identity feature g as the identity similarity of the face to be detected and the target face; if the identity similarity is less than a preset value, the face image to be detected is a fake image;

[0010] The ResNet50 backbone feature extraction network comprises an inlet flow layer, an intermediate flow layer and an outlet flow layer; the inlet flow layer converts a 112x112x3 picture into a 112x112x64 feature map, including 1 7x7 convolution and one max pooling layer; the intermediate flow layer includes 4 stages, including 3, 4, 6 and 3 residual blocks in turn, each residual block is composed of 1 1x1 convolution, 1 3x3 convolution and 1 1x1 convolution; the outlet flow layer converts a 7x7x512 feature map into a 512-dimensional feature vector, including an average pooling layer.

[0011] The technical scheme adopted by the system of the present application is: a deep fake face detection system based on identity-invariant features, comprising the following modules:

[0012] An identity feature extraction module is configured to pre-process a fake face image to be detected and a target face image into a preset size of 112x112x3, and then input them into a ResNet50 backbone feature extraction network respectively to obtain the identity feature f of the fake face image and the identity feature g of the target face image; wherein the target face image is a face image obtained by forging an original face image;

[0013] A fake image detection module is configured to calculate the cosine similarity between the identity feature f and the identity feature g as the identity similarity of the face to be detected and the target face; if the identity similarity is less than a preset value, the face image to be detected is a fake image;

[0014] The ResNet50 backbone feature extraction network comprises an entrance flow layer, a middle flow layer and an exit flow layer; the entrance flow layer is used for converting a picture of 112*112*3 into a feature map of 112*112*64, and comprises 1 7*7 convolution and a maximum pooling layer; the middle flow layer comprises 4 stages, and each stage comprises 3, 4, 6 and 3 residual blocks in sequence, each residual block is composed of 1 1*1 convolution, 1 3*3 convolution and 1 1*1 convolution; the exit flow layer is used for converting a feature map of 7*7*512 into a feature vector of 512 dimensions, and comprises an average pooling layer.

[0015] The technical scheme adopted by the device of the present application is: a deep fake face detection device based on identity-invariant features, comprising:

[0016] One or more processors;

[0017] A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the deep fake face detection method based on the identity-invariant features.

[0018] The advantages and positive effects of the present application are:

[0019] (1) The present application uses original images and fake images as training samples to learn the input network, which can make the network capture the real identity information of the fake image and make the identity features of the fake image and the original image consistent. In this way, the fake face looks similar to the real target face, but the identity features extracted by the above network have low similarity, thereby realizing the differentiation between the fake face and the real face.

[0020] (2) Compared with the original method of identifying authenticity by image appearance, the present application uses the real identity information of the fake image to judge the authenticity of the image, which is suitable for various fake methods and has higher robustness.

[0021] (3) The present application realizes the identification of deep fake face from a new angle, and solves the security problem caused by fake face in practical application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The method principle diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0024] The face deepfake process forges face A into face B, resulting in fake face C. In the present application, A, B and C are respectively referred to as the original face (i.e. the real face before forgery), the target face (i.e. the forged target object) and the fake face (i.e. the forgery result).

[0025] The existing convolutional neural network structure can only extract the deep features of the image, but these deep features do not pay attention to the feature expression related to the identity attribute of the face. Learning the input network by taking the original image and the fake image as training samples can make the network capture the real identity information of the fake image and make the identity features of the fake image and the original image consistent.

[0026] The deepfake algorithm tries to make the fake face look the same as the target face, achieving the effect of confusing the real with the fake. Based on the observation that the fake face is a modified version of the original face, although there is no necessary connection between the two in appearance, they share the same identity space; on the contrary, the fake face that looks very similar to the target face is not in the same identity space. Therefore, if multiple faces that look the same person have a very low similarity in identity features, there must be a fake face among them. Based on the above principle, the present application takes the original image and the fake image as training samples to learn the input network, constructs an identity-invariant feature extraction model, makes the network capture the real identity information of the fake image and makes the identity features of the fake image and the original image consistent; in the true-false discrimination test, the face image to be detected and the corresponding real face image are respectively input into the above trained network, the identity feature vectors of each are extracted, the similarity of the two vectors is measured, and if the similarity is not high enough, the face to be detected is considered to be a fake face.

[0027] See Figure 1 The present application provides a deepfake face detection method based on identity-invariant features, comprising the following steps:

[0028] Step 1: Preprocess the fake face image to be detected and the target face image into the size of 112x112x3, and then input them into the trained ResNet50 main feature extraction network respectively to obtain the identity feature f of the fake face image and the identity feature g of the target face image; wherein the target face image is a face image obtained by forging the original face image;

[0029] In this embodiment, the preprocessing is to detect the face area by the RetinaFace face detection algorithm, extract 5 facial feature points including left eye, right eye, nose, left corner of mouth and right corner of mouth; align the face by the facial feature points so that the aligned face is located at the center of the image; adjust the size of the picture to 112 pixels long, 112 pixels wide and 3 channels; recognize and crop the face area of each image; align the face by the facial feature points so that the aligned face is located at the center of the image.

[0030] Step 2: Calculate the cosine similarity between the identity features f and g as the identity similarity of the face to be detected and the target face; if the identity similarity is less than a preset value 0.35, the face image to be detected is a fake image.

[0031] The ResNet50 backbone feature extraction network used in this embodiment includes an inlet flow layer, a middle flow layer and an outlet flow layer; the inlet flow layer converts a 112x112x3 picture into a 112x112x64 feature map, including 1 7x7 convolution and 1 max pooling layer; the middle flow layer includes 4 stages, which include 3, 4, 6 and 3 residual blocks in turn, and each residual block is composed of 1 1x1 convolution, 1 3x3 convolution and 1 1x1 convolution; the outlet flow layer converts a 7x7x512 feature map into a 512-dimensional feature vector, including an average pooling layer.

[0032] The ResNet50 backbone feature extraction network of this embodiment is a trained backbone feature extraction network; its training process includes the following sub-steps:

[0033] Step 1.1: Collect a number of original face images of different identities and fake face images obtained by faking them, and the identity of the fake face image is represented by the identity of the corresponding original face image;

[0034] Firstly, in this embodiment, the fake face video and the original face video are down-sampled, and frame images of a fixed frame interval starting from a specific frame in the video are selected, so that the down-sampled video frames of the fake face video and the original face video from which the fake face video is derived correspond one by one. In this embodiment, 25 frame images starting from 0 seconds, one frame per second and per video are selected as the original data set;

[0035] Next, in this embodiment, the RetinaFace face detection algorithm is used to detect the face area in the obtained frame images, and the face images are cropped. 5 facial feature points are extracted, including left eye, right eye, nose, left corner of mouth and right corner of mouth; the face is aligned by the facial feature points so that the aligned face is located at the center of the image; the picture is adjusted to a size of 112 pixels long, 112 pixels wide and 3 channels.

[0036] The embodiment takes the processed face image in step 1.1 as input and supervision sample, and continuously trains the ResNet50 backbone feature extraction network.

[0037] Step 1.2: For each face sample, the identity feature feat is extracted through the ResNet50 network, and then the ArcFace loss of feat and the corresponding identity label is calculated.

[0038] The ArcFace loss of the embodiment is:

[0039]

[0040] Wherein N represents the number of samples in a Batch, and θ j represents the included angle between the identity feature extracted by the i th sample and the j th category vector, and θ yi represents the included angle between the identity feature extracted by the i th sample and the label category vector corresponding to it, m represents the interval hyperparameter and is set to 0.5, s represents the scale expansion hyperparameter and is set to 64, and n represents the number of classification categories.

[0041] Step 1.3: After the loss is calculated, the ResNet50 backbone feature extraction network is optimized by using the Adam optimizer to perform back propagation and optimization, so that the identity of the fake face image and the corresponding original face image is consistent;

[0042] Step 1.4: Repeat steps 1.1-1.3, train the ResNet50 backbone feature extraction network to convergence, and obtain the trained ResNet50 backbone feature extraction network.

[0043] The application first pre-processes a plurality of original face images and corresponding fake face images to 112x112x3 size and respectively inputs into the ResNet50 backbone feature extraction network to obtain the identity features of the face images, and then uses the ArcFace loss function to supervise the identity features extracted by the original face and the fake face to be consistent. Finally, the fake face image to be detected and the corresponding target face image are respectively input into the trained ResNet50 backbone feature extraction network to obtain the identity features f and g of the face images, respectively. The identity similarity of the fake face and the target face is obtained by calculating the cosine similarity of f and g, and the picture with an identity similarity less than a preset value is a fake face picture. The application can effectively extract the identity invariant features of true and fake faces, and has more accurate deep fake face detection effect.

[0044] It should be understood that parts not elaborated in the specification are all prior art.

[0045] It should be understood that the above description is merely a detailed explanation of the preferred embodiments and is not intended to limit the patent protection scope of the present application. Any modification or alternation made by those skilled in the art without departing from the scope of the present application shall fall within the patent protection scope of the present application. The patent protection scope of the present application shall be subject to the appended claims.

Claims

1. An identity-invariant feature based deep fake face detection method, characterized in that, The method comprises the following steps: Step 1: the fake face image to be detected and the target face image are preprocessed into a preset size of 112x112x3, and then are input into a ResNet50 backbone feature extraction network respectively to obtain identity features f of the fake face image and identity features g of the target face image; wherein the target face image is a face image obtained by forgery using an original face image; Step 2: the cosine similarity between the identity features f and the identity features g is calculated as the identity similarity of the face to be detected and the target face; if the identity similarity is less than a preset value, the face image to be detected is a fake image; The ResNet50 backbone feature extraction network comprises an inlet flow layer, a middle flow layer and an outlet flow layer; the inlet flow layer is used for converting a 112x112x3 picture into a 112x112x64 feature map, and comprises one 7x7 convolution and one maximum pooling layer; the middle flow layer comprises four stages, and each stage comprises 3, 4, 6 and 3 residual blocks in sequence, each residual block is composed of one 1x1 convolution, one 3x3 convolution and one 1x1 convolution; the outlet flow layer is used for converting a 7x7x512 feature map into a 512-dimensional feature vector, and comprises one average pooling layer; The ResNet50 backbone feature extraction network is a trained ResNet50 backbone feature extraction network; the training process of the ResNet50 backbone feature extraction network comprises the following steps: (1) collect a plurality of original face images of different identities and fake face images obtained by forgery using the original face images, and the identity of the fake face image is represented by the identity of the corresponding original face image; wherein, the original face video and the corresponding fake face video are selected from the specified frames, and the frame images of the fixed frame interval are selected; the original face video frames and the fake face video frames are one-to-one corresponding; the original face-fake face image samples are generated; The images of the fake face video and the original face video are preprocessed, the face area of each image is recognized and cropped through face detection technology, and the face is aligned through face feature points so that the aligned face is located at the center of the image; (2) for each face sample, the identity feature feat is extracted through the ResNet50 backbone feature extraction network, and then the ArcFace loss of the feat and the corresponding identity label is calculated; wherein, the ArcFace loss is: where N denotes the number of samples in a Batch, denotes the angle between the identity feature extracted from the i th sample and the j th class vector, denotes the angle between the identity feature extracted from the i th sample and its corresponding label class vector, m denotes the margin hyper-parameter, s denotes the scale expansion hyper-parameter, n denotes the number of classification classes. (3) after the loss is calculated, the ResNet50 backbone feature extraction network is optimized through the Adam optimizer and back propagation, so that the identity of the fake face image is consistent with that of the corresponding original face image; (4) repeat steps (1)-(4) to train the ResNet50 backbone feature extraction network until convergence, and obtain the trained ResNet50 backbone feature extraction network.

2. The identity-invariant feature based deep fake face detection method of claim 1, wherein: The preprocessing in step 1 is detecting a face region through a RetinaFace face detection algorithm, extracting 5 facial feature points including a left eye, a right eye, a nose, a left corner of a mouth and a right corner of a mouth, aligning the face through the facial feature points so that the aligned face is located at the center of the image, adjusting the picture through opencv to have a size of 112 pixels in length, 112 pixels in width and 3 channel numbers, recognizing and cropping a face region of each image, and aligning the face through the facial feature points so that the aligned face is located at the center of the image.

3. A system for detecting deepfake face based on identity-invariant features, the system comprising: Comprise the following modules: An identity feature extraction module is configured to preprocess a fake face image to be detected and a target face image into a preset size of 112x112x3, and then input the fake face image and the target face image into a ResNet50 backbone feature extraction network respectively to obtain an identity feature f of the fake face image and an identity feature g of the target face image; wherein the target face image is a face image obtained by falsification using an original face image; A fake image detection module is configured to calculate a cosine similarity between the identity feature f and the identity feature g as an identity similarity of the face to be detected and the target face; and if the identity similarity is less than a preset value, the face image to be detected is a fake image; The ResNet50 backbone feature extraction network comprises an inlet flow layer, a middle flow layer and an outlet flow layer; the inlet flow layer is configured to convert a picture of 112x112x3 into a feature map of 112x112x64, and comprises one 7x7 convolution and one maximum pooling layer; the middle flow layer comprises four stages, and each stage comprises 3, 4, 6 and 3 residual blocks in sequence, each residual block is composed of one 1x1 convolution, one 3x3 convolution and one 1x1 convolution; and the outlet flow layer is configured to convert a feature map of 7x7x512 into a feature vector of 512 dimensions, and comprises one average pooling layer; The ResNet50 backbone feature extraction network is a trained ResNet50 backbone feature extraction network; and a training process of the ResNet50 backbone feature extraction network comprises the following steps: (1) collecting a plurality of original face images of different identities and fake face images obtained by falsification using the original face images, and representing the identity of each fake face image by the identity of the corresponding original face image; wherein, the original face video and the corresponding fake face video are selected to have frame images with a fixed frame number interval starting from a specified frame; the original face video frames and the fake face video frames are one-to-one corresponding; and original face-fake face image samples are generated; The images of the fake face video and the original face video are preprocessed, the face region of each image is recognized and cropped through face detection technology, and the face is aligned through facial feature points so that the aligned face is located at the center of the image. (2) for each face sample, an identity feature feat is extracted through the ResNet50 backbone feature extraction network, and then an ArcFace loss of the feat and a corresponding identity label is calculated; wherein, the ArcFace loss is: where N denotes the number of samples in a Batch, denotes the angle between the identity feature extracted from the i th sample and the j th class vector, denotes the angle between the identity feature extracted from the i th sample and the label class vector corresponding to it, m denotes the margin hyper-parameter, s denotes the scale expansion hyper-parameter, n denotes the number of classification categories; (3) After the loss is calculated, the ResNet50 backbone feature extraction network is back propagated and optimized using the Adam optimizer, so that the identity of the fake face image and the corresponding original face image remains consistent; (4) Repeat steps (1)-(4) to train the ResNet50 backbone feature extraction network to convergence, and obtain the trained ResNet50 backbone feature extraction network.

4. An identity-invariant feature based deep fake face detection device, characterized by, comprise: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the identity-invariant feature-based deep fake face detection method according to any one of claims 1-2.

Images