A method and related equipment for detecting deep fake face images

Abstract

The present invention provides a method for detecting deep fake facial images and related equipment, including: obtaining a set of real facial images and a set of fake facial images to construct a training set for feature learning, thereby obtaining a multidimensional feature vector set; performing similarity calculation on any two feature vectors in the multidimensional feature vector set, generating a set of class label vectors for the training set, and calculating any two of the class label vectors therein; inputting the multidimensional feature vector set into a classification network to obtain a two-dimensional probability vector set and performing classification loss calculation; then calculating the overall loss and using the back propagation method to train a deep learning network model based on the loss and updating the network parameters of the deep learning network until the overall loss approaches a preset threshold, thereby obtaining a fake facial image detection model; inputting a block facial image to be detected into the fake facial image detection model to perform deep fake face detection and obtain a detection result; thereby improving the accuracy of fake facial image detection.

Description

Technical Field

[0001] The present invention relates to the technical field of face forgery detection, and in particular to a method for detecting deep fake face images and related equipment. Background Art

[0002] With the rapid development of face tampering technology, forged faces are becoming increasingly difficult for the human eye to discern, which will have a great impact on human and social security. Therefore, how to detect face forgery is crucial.

[0003] Metric learning, also known as similarity learning, has been widely used in many fields of artificial intelligence. It involves autonomously learning a metric distance function tailored to a specific task, based on the task itself. AKash Kumar used metric learning (ternary networks) for face forgery detection, and experimental results demonstrated that effectively measuring the distance between data can indeed improve the detection performance of deep networks. However, the ternary loss in this method primarily considers inter-class distances, while intra-class distances may not be as compact. Jiaming Li believes that the distribution of forged faces is diverse, while the distribution of real faces is relatively deterministic. They proposed a single center loss for face forgery detection, which makes real data more compact within classes and achieves considerable performance. Yunsheng Ni et al. proposed a continuous representation learning loss. By constraining the similarity of the embedding spaces of samples under two different data augmentations, the network is able to learn consistent features extracted from the same sample under different forms. In fact, an analysis of the proposed constraint reveals that it is equivalent to a special case of constraining the similarity between similar data. This is because in forged video data, the sets of real and forged face images of the same person are not unique. Secondly, the erasure-type data augmentation method they use may cause important traces of tampering and forgery to be erased or weakened, thereby reducing the difference between forged data and real data, resulting in the model being unable to learn the difference between the real face image set and the forged face image set well.

[0004] In the field of detecting forged facial image sets, methods based on recurrent convolutional neural networks are currently the primary approach. These networks are primarily used to analyze forgery features such as image texture, facial edges, and head pose. Most of these features are effective against traditional forgery techniques, such as Photoshop images and moiré patterns from screen captures. However, these traditional detection methods are no longer effective against the latest forgery techniques: forged facial image sets generated using generative adversarial networks.

[0005] Although there are currently a few methods that detect the latest methods of generating fake face image sets, such detection methods are still not mature enough, with low detection accuracy and poor robustness. Summary of the Invention

[0006] The present invention provides a method and related equipment for detecting deep fake facial images, the purpose of which is to improve the accuracy of forged facial image detection.

[0007] To achieve the above objectives, the present invention provides a method for detecting deep fake facial images, comprising:

[0008] Step 1: Obtain a set of real face images and a set of forged face images, where the face images in the real face image set correspond one-to-one to the face images in the forged face image set;

[0009] Step 2: Randomly extract multiple images from the real face image set and the fake face image set to construct a training set, and input all the face images in the training set into the deep learning network for feature learning to obtain a multi-dimensional feature vector set;

[0010] Step 3: perform similarity calculation on any two dimensional feature vectors in the multidimensional feature vector set to obtain an estimated similarity value matrix. Generate a class label vector corresponding to each face image in the training set to obtain a class label vector set. Calculate the true similarity value matrix for any two class label vectors in the class label vector set. Calculate the estimated similarity value matrix and the true similarity value matrix to obtain the overall similarity loss value.

[0011] Step 4: Input the multidimensional feature vector set into the classification network to obtain a two-dimensional probability vector set. The classification loss is calculated on the two-dimensional probability vector set and the class label vector using the cross entropy loss function to obtain multiple classification loss values. The multiple classification loss values ​​are summed to obtain the overall classification loss value. The two-dimensional probability vectors in the two-dimensional probability vector set correspond one to one to the feature vectors.

[0012] Step 5: Multiply the overall similarity loss value by the balance factor, and the sum of the obtained product and the overall classification loss value is taken as the overall loss. Based on the overall loss, the deep learning network is trained and the network parameters of the deep learning network are updated through backpropagation until the overall loss is close to the preset threshold, thereby obtaining a fake face image detection model;

[0013] Step 6: Input the block face image to be detected into the forged face image detection model to perform deep forged face detection to obtain the detection result.

[0014] Furthermore, step 2 also includes:

[0015] Perform data augmentation on the training set;

[0016] The augmented training set is input into the deep learning network for feature learning.

[0017] Furthermore, the deep learning network includes a feature extraction network and a fully connected neural network. The output of the feature extraction network is connected to the input of the fully connected neural network. The input of the feature extraction network is the input of the deep learning network, and the output of the fully connected neural network is the output of the deep learning network.

[0018] The augmented training set is input into the feature extraction network for feature extraction to obtain a multi-dimensional feature vector set;

[0019] The multidimensional feature vector set is input into a fully connected neural network, and the multidimensional feature vector set is mapped by the fully connected neural network to obtain a 2-dimensional probability vector set.

[0020] Furthermore, a training set is constructed by randomly selecting multiple images from the real face image set and the fake face image set, and data augmentation processing is performed on the training set, including:

[0021] Given a training set (x1, x2, x3…xN) containing multiple sets of real face images and multiple sets of forged face images, arbitrarily select an augmentation method t1 suitable for forged face detection in the data augmentation method set T, and generate a training set constructed by multiple augmented real face images and multiple augmented forged face images.

[0022] More specifically, step 3 includes:

[0023] Performing L2 norm normalization calculation on all multidimensional feature vectors in the multidimensional feature vector set to obtain a normalized multidimensional feature vector set;

[0024] Use the cosine similarity loss function to calculate the similarity loss of any two vectors in the normalized multidimensional feature vector set to obtain the estimated similarity matrix as follows:

[0025]

[0026] Generate a class label vector corresponding to each face image in the training set, and get the class label vector set y=(y1,y2,y3…y N ), calculate any two class label vectors in the class label vector set to obtain the true similarity value matrix;

[0027] By calculating the estimated similarity value matrix and the true similarity value matrix, the overall similarity loss value is obtained and the overall similarity loss value is calculated. The function is:

[0028]

[0029] in, is the i-th dimension feature vector f' i The L2 norm normalized vector of is the j-th dimension feature vector f' j The L2 norm normalized vector of , the calculation method of L2 norm normalization is y i is the class label corresponding to the i-th sample, y j is the class label corresponding to the jth sample.

[0030] More specifically, step 4 includes:

[0031] Input the multidimensional feature vector set into the classification network to obtain a two-dimensional probability vector set;

[0032] According to the cross entropy loss function, the overall classification loss value is calculated for:

[0033]

[0034] Among them, p' i is the 2D probability vector corresponding to the i-th sample, y i is the category label corresponding to the i-th sample.

[0035] More specifically, the formula for calculating the overall loss is:

[0036]

[0037] Among them, λ is the balance factor between similarity loss and classification loss, and the value range of λ is [0,5].

[0038] The present invention also provides a device for detecting deep fake facial images, comprising:

[0039] An acquisition module is used to acquire a real face image set and a forged face image set, wherein the face images in the real face image set correspond one to one with the face images in the forged face image set;

[0040] The feature learning module is used to randomly extract multiple images from the real face image set and the fake face image set to construct a training set, and input all the face images in the training set into the deep learning network for feature learning to obtain a multi-dimensional feature vector set;

[0041] A first calculation module is configured to perform similarity calculation on any two dimensional feature vectors in the multidimensional feature vector set to obtain an estimated similarity value matrix, generate a class label vector corresponding to each face image in the training set to obtain a class label vector set, calculate a true similarity value matrix for any two class label vectors in the class label vector set, and obtain an overall similarity loss value by calculating the estimated similarity value matrix and the true similarity value matrix;

[0042] A second calculation module is used to input the multidimensional feature vector set into the classification network to obtain a two-dimensional probability vector set, perform classification loss calculation on the two-dimensional probability vector set and the class label vector using a cross entropy loss function to obtain multiple classification loss values, and sum the multiple classification loss values ​​to obtain an overall classification loss value; the two-dimensional probability vectors in the two-dimensional probability vector set correspond one-to-one to the feature vectors;

[0043] The training module is used to multiply the similarity loss by the balance factor, and the sum of the obtained product and the classification loss is used as the overall loss. Based on the overall loss, the deep learning network is trained and the network parameters of the deep learning network are updated through backpropagation until the overall loss approaches a preset threshold, thereby obtaining a fake face image detection model.

[0044] The detection module is used to input the block face image to be detected into the forged face image detection model to perform deep forged face detection and obtain the detection result.

[0045] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method for detecting deep fake facial images is implemented.

[0046] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, a method for detecting deep fake facial images is implemented.

[0047] The above solution of the present invention has the following beneficial effects:

[0048] The present invention obtains a set of real face images and a set of forged face images, randomly extracts multiple images from the real face image set and the forged face image set to construct a training set, and inputs them into a deep learning network for feature learning to obtain a multi-dimensional feature vector set; performs similarity calculation on any two-dimensional feature vectors in the multi-dimensional feature vector set to obtain an estimated similarity value matrix, generates a class label vector corresponding to each face image in the training set to obtain a class label vector set, calculates any two class label vectors in the class label vector set to obtain a true similarity value matrix, and obtains an overall similarity loss value by calculating the estimated similarity value matrix and the true similarity value matrix; effectively increases the similarity between samples of the same type and expands the difference between the real face image set and the forged face image set; inputs the multi-dimensional feature vector set into the training set to obtain a class label vector set; A classification network is provided to obtain a two-dimensional probability vector set, and the classification loss is calculated on the two-dimensional probability vector set and the class label vector through the cross entropy loss function to obtain multiple classification loss values, and the multiple classification loss values ​​are summed to obtain an overall classification loss value; the similarity loss is multiplied by the balance factor, and the sum of the obtained product and the classification loss is used as the overall loss, and then the overall loss and the back propagation method are used to train the deep learning network model to obtain a forged face image detection model; the generalization ability of the forged face image detection model is improved through model training, and finally the block face image to be detected is input into the forged face image detection model for deep forged face detection to obtain the detection result; this solves the problem that the face detection model in the existing technology cannot well learn the difference between the real face image set and the forged face image set, and improves the accuracy of forged face image detection.

[0049] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0051] Figure 2 This is a graph showing the experimental results of verifying the similarity loss on the FF++(c40) dataset according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] To make the technical problems, technical solutions, and advantages to be solved by the present invention more clear, the following is a detailed description with reference to the accompanying drawings and specific embodiments. It is obvious that the embodiments described are only some of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0053] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0054] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood broadly. For example, they may refer to a locking connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0055] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0056] In response to existing problems, the present invention provides a method and related equipment for detecting deep fake facial images.

[0057] like Figure 1 As shown, an embodiment of the present invention provides a method for detecting deep fake face images, comprising:

[0058] Step 1: Obtain a real face image set and a forged face image set, where the face images in the real face image set correspond one-to-one to the face images in the forged face image set.

[0059] In an embodiment of the present invention, in order to facilitate the subsequent training of the deep learning network, it is necessary to obtain multiple real face image sets and forged face image sets. Therefore, real face image sets and forged face image sets are extracted from three commonly used forged face datasets: FaceForensics++ (FF++), DFDC, and Celeb-DF to form a data-balanced face detection dataset. Among them, FaceForensics++ (FF++) contains 1,363 real face videos collected from the video website Youtube and 4,000 fake videos synthesized using four tampering methods. DFDC includes 119,197 videos, each video is 10 seconds long, the frame rate ranges from 15 to 30 frames, and the resolution ranges from 320×240 to 3840×2160. Celeb-DF contains 590 original videos collected from the video website YouTube and 5,639 DeepFake deep forged videos generated from real videos. This example divides the above three datasets into train / real and train / fake training datasets and val / real and val / real test datasets in a 5:1 ratio. real is the saved real face image set, and fake is the saved fake face image set. train / real contains 50,000 real face images, train / fake contains 50,000 fake face images, val / real contains 10,000 real face images, and val / fake contains 10,000 fake face images.

[0060] Step 2: Randomly extract multiple images from the real face image set and the fake face image set to construct a training set, and input all the face images in the training set into the deep learning network for feature learning to obtain a multi-dimensional feature vector set;

[0061] Step 3: perform similarity calculation on any two dimensional feature vectors in the multidimensional feature vector set to obtain an estimated similarity value matrix. Generate a class label vector corresponding to each face image in the training set to obtain a class label vector set. Calculate the true similarity value matrix for any two class label vectors in the class label vector set. Calculate the estimated similarity value matrix and the true similarity value matrix to obtain the overall similarity loss value.

[0062] Step 4: Input the multidimensional feature vector set into the classification network to obtain a two-dimensional probability vector set. The classification loss is calculated on the two-dimensional probability vector set and the class label vector using the cross entropy loss function to obtain multiple classification loss values. The multiple classification loss values ​​are summed to obtain the overall classification loss value. The two-dimensional probability vectors in the two-dimensional probability vector set correspond one to one to the feature vectors.

[0063] Step 5: Multiply the similarity loss by the balance factor, and the sum of the obtained product and the classification loss is taken as the overall loss. Based on the overall loss, the deep learning network is trained and the network parameters of the deep learning network are updated through backpropagation until the overall loss is close to the preset threshold, thereby obtaining a fake face image detection model.

[0064] Step 6: Input the block face image to be detected into the forged face image detection model to perform deep forged face detection to obtain the detection result.

[0065] Specifically, step 2 also includes:

[0066] Perform data augmentation on the training set;

[0067] The augmented training set is input into the deep learning network for feature learning.

[0068] Specifically, multiple images are randomly selected from the real face image set and the fake face image set to construct a training set, and data augmentation processing is performed on the training set, including:

[0069] Given a training set (x1, x2, x3…x1) containing multiple real face images and fake face images N ), arbitrarily select an augmentation method t1 suitable for forged face detection from the data augmentation method set T. The four default recommended data augmentation methods include random erasure, random horizontal flipping, random cropping, and combined data augmentation (including random affine transformation, random compression, random downsampling, etc.); generate a training set constructed by multiple augmented real face images and multiple augmented forged face images. Because some data augmentation methods may destroy forged face information, the embodiment of the present invention increases the range of data augmentation, such as expanding the size of the random erasure area to verify the similarity between constrained samples, so that the deep learning network can learn the ability of data invariance.

[0070] In some embodiments of the present invention, Random Erasing is selected by default for data augmentation. Random Erasing is a lightweight method that does not require any additional parameters or memory consumption. It can be integrated with various CNN models without changing the learning strategy. As a supplement to existing data augmentation and regularization methods, the combination of the two further improves recognition performance. In terms of image classification, object detection and pedestrian re-identification, the performance of deep models is improved and the robustness is increased.

[0071] Specifically, the deep learning network includes a feature extraction network and a fully connected neural network. The output end of the feature extraction network is connected to the input end of the fully connected neural network, the input end of the feature extraction network is the input end of the deep learning network, and the output end of the fully connected neural network is the output end of the deep learning network.

[0072] The augmented training set is input into the feature extraction network for feature extraction to obtain a multi-dimensional feature vector set. In the embodiment of the present invention, the classic CNN network ResNet18 with 17 convolutional layers and 1 fully connected layer and the lightweight network MobileNet-v2 with 3 convolutional layers, 7 inverted residual structures and 1 average pooling layer are selected as the feature extraction network f. The feature extraction network f is trained on the input training set (x'1, x'2x'3...x' N ) to extract features and obtain the multi-dimensional feature vector:

[0073] (f'1,f'2,f'3…f′ N )=f(x'1,x'2,x'3…x′ N )

[0074] Among them, f' N Represents the N-th dimension feature vector, the value of N can be 512 or 1024, x' N Represents the Nth face image.

[0075] It should be noted that the deep learning network can be a commonly used convolutional neural network, but is not limited to commonly used convolutional neural networks.

[0076] The multidimensional feature vector is input into a fully connected neural network, and the multidimensional feature vector is mapped by the fully connected neural network to obtain a K-dimensional vector. In the embodiment of the present invention, the value of K is 2, corresponding to binary classification.

[0077] Specifically, step 3 includes:

[0078] In order to simply and efficiently increase the similarity of samples of the same category and reduce the similarity between different categories, the embodiment of the present invention introduces the cosine similarity loss function. Specifically:

[0079] In order to facilitate the calculation of the cosine angle, all multidimensional feature vectors in the multidimensional feature vector set are respectively subjected to L2 norm normalization calculation to obtain a normalized multidimensional feature vector set.

[0080] Use the cosine similarity loss function to calculate the similarity loss of any two vectors in the normalized multidimensional feature vector set to obtain the estimated similarity matrix as follows:

[0081]

[0082] Generate a class label vector corresponding to each face image in the training set, and get the class label vector set y=(y1,y2,y3…y N ), calculate any two class label vectors in the class label vector set to obtain the true similarity value matrix.

[0083] The cosine loss function is often used to evaluate the similarity between two vectors. The higher the cosine value of the two vectors, the higher the similarity.

[0084] By calculating the estimated similarity value matrix and the true similarity value matrix, the overall similarity loss value is obtained, so that the eigenvectors of the same type are close to 1, and the eigenvectors of the real face and the forged face are close to -1. The function expression is:

[0085]

[0086] in, is the i-th dimension feature vector f' i The L2 norm normalized vector of is the j-th dimension feature vector f' j The L2 norm normalized vector of , the calculation method of L2 norm normalization is y i is the class label corresponding to the i-th sample, y j is the class label corresponding to the jth sample. As an optional example, the above norm normalization method can be P=2 norm.

[0087] It should be noted that, in the embodiment of the present invention, the calculation of the similarity between feature vectors does not require any other modules to process and does not increase the parameters of the model.

[0088] Specifically, step 4 includes:

[0089] Input the multidimensional feature vector set into the classification network to obtain a two-dimensional probability vector set;

[0090] According to the cross entropy loss function, the overall classification loss value is calculated for:

[0091]

[0092] Among them, p' i is the 2D probability vector corresponding to the i-th sample, y i is the category label corresponding to the i-th sample.

[0093] Specifically, the formula for calculating the overall loss is:

[0094]

[0095] Here, λ is a balancing factor between similarity loss and classification loss, and the value range of λ is [0, 5]. Before training the deep learning network, the embodiment of the present invention needs to select different values ​​of the balancing factor λ for verification.

[0096] Specifically, based on the overall loss, the deep learning network is trained and the network parameters of the deep learning network are updated through the back propagation method. The training is iterated 100 times until the overall loss is close to the preset threshold, and a fake face image detection model is obtained.

[0097] Specifically, in an embodiment of the present invention, during the training process, the classification label corresponding to the input face image can be used as the true label, and the classification label obtained based on the classification network can be used as the predicted label to calculate the distribution difference between the true label and the predicted label.

[0098] It should be noted that since the process of training a deep learning network using block images as a training data set is the same as the commonly used model training process, the specific steps of the training process will not be described in detail in the embodiments of the present invention.

[0099] The following describes the deep fake face image detection method in conjunction with the specific verification process.

[0100] To verify the two advantages of the deep fake face image detection method proposed in this invention:

[0101] (1) Similarity loss can simply and effectively increase the similarity between similar samples and expand the difference between real images and forged images, allowing the deep learning network to learn more essential forged information. The verification is verified through specific examples. The verification process is as follows:

[0102] The comparison primarily uses two basic convolutional networks, ResNet18 and MobileNet-v2, which have been widely used in image recognition. For ease of comparison, methods using similarity loss are labeled ResNet18+ and MobileNet-v2+, with the balance factor λ set to 1 by default. Internal ablation experiments were conducted on the FF++ and DFDC datasets, as shown in Table 1.

[0103] Table 1 (Please replace the English expressions in the table with Chinese)

[0104]

[0105] Table (1) above illustrates that similarity loss is achieved by increasing the similarity between similar samples, enlarging the difference between real data and forged data, and improving the detection performance of deep learning networks. In order to make a visual comparison, the embodiment of the present invention visualizes the Class Activation Mapping (CAM, class heat map) of ResNet18 and ResNet18+, which shows that ResNet18+ is more accurate in discovering forged information areas. In order to observe the effectiveness of our proposed method from the perspective of clustering, the embodiment of the present invention uses the t-SNE method to visualize the two-dimensional graphs of Resnet18 and Resnet18+ on the FF++(c23) test dataset, as shown in the attached figure. Figure 2 As shown in , our method makes the clusters more compact, thus relatively increasing the distance between clusters.

[0106] (2) Constraining the similarity between samples allows the forged face image detection model to have a certain data consistency capability, thereby effectively utilizing the advantages of data augmentation. The verification is carried out through specific examples. The verification process is as follows:

[0107] Based on the horizontal flip data augmentation method, this embodiment of the present invention uses three data augmentation methods: Random Erasing, Random Erasing, a set of multiple data augmentation methods (DFDC_selmi), and Random Resized Crop (RC). We use ResNet18 as the benchmark network and verify it on the DFDC dataset, where the balance factor λ is set to 2. This embodiment of the present invention uses AUC as the evaluation indicator, as shown in Table 2 below:

[0108] Table 2

[0109]

[0110] As can be seen from the above table, the method for deep fake face image detection provided by the present invention performs better than the basic convolutional network under various data augmentation methods.

[0111] The embodiment of the present invention obtains a real face image set and a fake face image set, randomly extracts multiple pictures from the real face image set and the fake face image set respectively to construct a training set, and inputs them into a deep learning network for feature learning to obtain a multi-dimensional feature vector set; performs similarity calculation on any two-dimensional feature vectors in the multi-dimensional feature vector set to obtain an estimated similarity value matrix, generates a class label vector corresponding to each face image in the training set to obtain a class label vector set, calculates any two class label vectors in the class label vector set to obtain a true similarity value matrix, and obtains an overall similarity loss value by calculating the estimated similarity value matrix and the true similarity value matrix; effectively increases the similarity between samples of the same type, and expands the difference between the real face image set and the fake face image set; the multi-dimensional feature vector set is used to obtain the real similarity value matrix. Input a classification network to obtain a two-dimensional probability vector set, and use a cross-entropy loss function to calculate the classification loss on the two-dimensional probability vector set and the class label vector to obtain multiple classification loss values, which are summed to obtain an overall classification loss value; multiply the similarity loss by the balance factor, and use the sum of the obtained product and the classification loss as the overall loss; then use the overall loss and back propagation method to train the deep learning network model to obtain a forged face image detection model; the generalization ability of the forged face image detection model is improved through model training, and finally the block face image to be detected is input into the forged face image detection model for deep forged face detection to obtain a detection result; this solves the problem that the face detection model in the existing technology cannot well learn the difference between the real face image set and the forged face image set, and improves the accuracy of forged face image detection.

[0112] An embodiment of the present invention further provides a device for detecting deep fake facial images, comprising:

[0113] An acquisition module is used to acquire a real face image set and a forged face image set, wherein the face images in the real face image set correspond one to one with the face images in the forged face image set;

[0114] The feature learning module is used to randomly extract multiple images from the real face image set and the fake face image set to construct a training set, and input all the face images in the training set into the deep learning network for feature learning to obtain a multi-dimensional feature vector set;

[0115] A first calculation module is configured to perform similarity calculation on any two dimensional feature vectors in the multidimensional feature vector set to obtain an estimated similarity value matrix, generate a class label vector corresponding to each face image in the training set to obtain a class label vector set, calculate a true similarity value matrix for any two class label vectors in the class label vector set, and obtain an overall similarity loss value by calculating the estimated similarity value matrix and the true similarity value matrix;

[0116] A second calculation module is used to input the multidimensional feature vector set into the classification network to obtain a two-dimensional probability vector set, perform classification loss calculation on the two-dimensional probability vector set and the class label vector using a cross entropy loss function to obtain multiple classification loss values, and sum the multiple classification loss values ​​to obtain an overall classification loss value; the two-dimensional probability vectors in the two-dimensional probability vector set correspond one-to-one to the feature vectors;

[0117] The training module is used to multiply the similarity loss by the balance factor, and the sum of the obtained product and the classification loss is used as the overall loss. Based on the overall loss, the deep learning network is trained and the network parameters of the deep learning network are updated through backpropagation until the overall loss approaches a preset threshold, thereby obtaining a fake face image detection model.

[0118] The detection module is used to input the block face image to be detected into the forged face image detection model to perform deep forged face detection and obtain the detection result.

[0119] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiments of the embodiments of the present invention. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0121] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method for detecting deep fake facial images is implemented.

[0122] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying computer program code to a construction device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0123] An embodiment of the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the computer program, a method for detecting deep fake facial images is implemented.

[0124] The terminal device may be a computing device such as a desktop computer, a notebook computer, a PDA, a server, a server cluster, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0125] The processor may be a central processing unit (CPU), other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0126] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk equipped on the terminal device, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, etc. Furthermore, the memory may include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0127] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiments of the embodiments of the present invention. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0128] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0129] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present invention.

[0131] In the embodiments provided in the embodiments of the present invention, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0133] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for detecting deep fake face images, characterized in that: include: Step 1: obtaining a set of real face images and a set of forged face images, wherein the face images in the real face image set correspond one-to-one to the face images in the forged face image set; Step 2: randomly extracting multiple images from the real face image set and the forged face image set to construct a training set, and inputting all the face images in the training set into a deep learning network for feature learning to obtain a multidimensional feature vector set; Step 3: performing similarity calculation on any two dimensional feature vectors in the multidimensional feature vector set to obtain an estimated similarity value matrix; generating a class label vector corresponding to each face image in the training set to obtain a class label vector set; calculating any two class label vectors in the class label vector set to obtain a true similarity value matrix; and obtaining an overall similarity loss value by calculating the estimated similarity value matrix and the true similarity value matrix; Step 4: Input the multidimensional feature vector set into the classification network to obtain a two-dimensional probability vector set, perform classification loss calculation on the two-dimensional probability vector set and the class label vector using a cross entropy loss function to obtain multiple classification loss values, and sum the multiple classification loss values ​​to obtain an overall classification loss value; the two-dimensional probability vectors in the two-dimensional probability vector set correspond one to one to the feature vectors; Step 5: Multiply the overall similarity loss value by the balance factor, and use the sum of the obtained product and the overall classification loss value as the overall loss. Based on the overall loss, the deep learning network is trained by backpropagation and the network parameters of the deep learning network are updated until the overall loss is close to a preset threshold, thereby obtaining a fake face image detection model. Step 6: Input the block face image to be detected into the forged face image detection model to perform deep forged face detection to obtain a detection result.

2. The method for detecting deep fake face images according to claim 1, characterized in that: The step 2 further comprises: Performing data augmentation processing on the training set; The augmented training set is input into the deep learning network for feature learning.

3. The method for detecting deep fake face images according to claim 2, characterized in that: The deep learning network includes a feature extraction network and a fully connected neural network, the output end of the feature extraction network is connected to the input end of the fully connected neural network, the input end of the feature extraction network is the input end of the deep learning network, and the output end of the fully connected neural network is the output end of the deep learning network; Inputting the augmented training set into the feature extraction network to perform feature extraction to obtain a multi-dimensional feature vector set; The multidimensional feature vector set is input into the fully connected neural network, and the multidimensional feature vector set is mapped by the fully connected neural network to obtain a two-dimensional probability vector set.

4. The method for detecting deep fake face images according to claim 2, wherein: A plurality of images are randomly selected from the real face image set and the forged face image set to construct a training set, and data augmentation processing is performed on the training set, including: Given a training set (x1, x2, x3…x1) containing multiple real face image sets and fake face image sets, N ), arbitrarily select an augmentation method t1 suitable for forged face detection from the data augmentation method set T, and generate a training set constructed by multiple augmented real face images and multiple augmented forged face images.

5. The method for detecting deep fake face images according to claim 3, wherein: The step 3 comprises: Performing L2 norm normalization calculation on all multidimensional feature vectors in the multidimensional feature vector set to obtain a normalized multidimensional feature vector set; Use the cosine similarity loss function to calculate the similarity loss of any two vectors in the normalized multidimensional feature vector set to obtain the estimated similarity matrix as follows: Generate a class label vector corresponding to each face image in the training set, and obtain a class label vector set y=(y1,y2,y3…y N ), calculating any two class label vectors in the class label vector set to obtain a true similarity value matrix; By calculating the estimated similarity value matrix and the true similarity value matrix, the overall similarity loss value is obtained, and the overall similarity loss value is calculated. The function is: in, is the i-th dimension feature vector f' i The L2 norm normalized vector of is the j-th dimension feature vector f' j The L2 norm normalized vector of , the calculation method of L2 norm normalization is y i is the class label corresponding to the i-th sample, y j is the class label corresponding to the jth sample.

6. The method for detecting deep fake face images according to claim 3, wherein: The step 4 comprises: Inputting the multidimensional feature vector set into a classification network to obtain a two-dimensional probability vector set; According to the cross entropy loss function, the overall classification loss value is calculated for: Among them, p' i is the 2D probability vector corresponding to the i-th sample, y i is the category label corresponding to the i-th sample.

7. The method for detecting deep fake face images according to claim 3, characterized in that: The formula for calculating the overall loss is: Among them, λ is the balance factor between the overall similarity loss value and the overall classification loss value, and the value range of λ is [0,5].

8. A device for detecting deep fake face images, characterized in that: include: An acquisition module, configured to acquire a set of real face images and a set of forged face images, wherein the face images in the real face image set correspond one-to-one to the face images in the forged face image set; a feature learning module, configured to randomly extract a plurality of images from the real face image set and the forged face image set to construct a training set, and input all the face images in the training set into a deep learning network for feature learning to obtain a multidimensional feature vector set; a first calculation module, configured to perform similarity calculation on any two-dimensional feature vectors in the multidimensional feature vector set to obtain an estimated similarity value matrix, generate a class label vector corresponding to each face image in the training set to obtain a class label vector set, calculate any two class label vectors in the class label vector set to obtain a true similarity value matrix, and obtain an overall similarity loss value by calculating the estimated similarity value matrix and the true similarity value matrix; a second computing module, configured to input the multidimensional feature vector set into a classification network to obtain a two-dimensional probability vector set, perform classification loss calculation on the two-dimensional probability vector set and the class label vector using a cross entropy loss function to obtain multiple classification loss values, and sum the multiple classification loss values ​​to obtain an overall classification loss value; wherein the two-dimensional probability vectors in the two-dimensional probability vector set correspond one-to-one to the feature vectors; a training module configured to multiply the overall similarity loss value by a balancing factor, and use the sum of the obtained product and the overall classification loss value as the overall loss; and to perform model training on the deep learning network and update the network parameters of the deep learning network through a backpropagation method based on the overall loss until the overall loss approaches a preset threshold, thereby obtaining a forged face image detection model; The detection module is used to input the block-shaped face image to be detected into the forged face image detection model to perform deep forged face detection and obtain a detection result.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting deep fake facial images according to any one of claims 1 to 7 is implemented.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for detecting deep fake facial images according to any one of claims 1 to 7 is implemented.

Images