Face key point-based fake image detection method, terminal and storage medium

Abstract

The application provides a face key point-based fake image detection method, a terminal and a storage medium, wherein the method comprises constructing a face fake detection classification model; the face fake detection classification model comprises a face key point template generation module Gm, an RGB feature extraction module Fr, a mask feature extraction module Fm and a mask local attention module Bm; a training sample image X is acquired; the face fake detection classification model is trained by using the training sample image X, and a trained face fake detection classification model is obtained; a face image to be detected is acquired; the face image to be detected is input into the trained face fake detection classification model, and a recognition result is obtained. The application has reasonable design, and the mask of each organ region of a face is obtained through face key point detection, and redundant background information is discarded.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image recognition, and particularly relates to a fake image detection method based on face key points, a terminal and a storage medium. BACKGROUND

[0002] In the past few decades, the popularity of smartphones and the development of social networks have made digital images and videos very common digital objects. According to several reports, nearly 2 billion pictures are uploaded to the Internet every day. With the massive use of digital images, techniques for changing image content have also emerged, such as using editing software such as Photoshop. Today, the danger of fake news has been widely recognized, and with more than 100 million hours of video content watched on social networks every day, the spread of fake videos has attracted increasing attention, and a large number of face spoofing detection algorithms have emerged. There are many face spoofing image detection algorithms, among which the more common ones are improving neural network models and using multi-modal feature fusion. Improving neural network models usually modify neural network models using attention mechanisms to achieve better detection accuracy, but the problem is that the improved model may only perform well on the data distribution of the training data and not perform well on unknown spoofing operation data sets. Multi-modal based spoofing detection algorithms fuse information from multiple modalities, and when one modality fails, other modalities can make up for it, achieving better generalization. Multiple modalities interact with each other, often resulting in higher detection accuracy, and the focus of multi-modal face spoofing image detection is the selection of modalities and the effective fusion of multiple modalities.

[0003] This field usually uses other features as a supplement to another type of information, and other features are also referred to as another modality. Currently, other features involve the frequency domain, image depth, or heartbeat, blood flow, etc. The key point information of the face is also a kind of supplementary information, and more rich information can be obtained by capturing the key parts of the face. Deepfake technology usually transplants a person's face onto another person's face to achieve the effect of changing the identity of the person. Because Deepfake only changes the foreground area (face area) of the person, and does not change the background area of the person. Therefore, some artifacts may be produced in the process of transferring the face, especially at the junction of the foreground and background of the face. Deepfake technology also blurs the artifacts of the mixed boundary in the post-processing process, making it more difficult to detect the artifacts of the mixed boundary, especially when the model does not have a priori knowledge to guide it, the detection effect of the face spoofing image is not good. SUMMARY

[0004] Therefore, the present application aims to provide a face key point-based fake image detection method, a terminal and a storage medium to solve the problem of poor detection effect of existing detection methods on face fake images.

[0005] To achieve the above object, the technical scheme of the present application is as follows:

[0006] In a first aspect, the present application provides a face key point-based fake image detection method, comprising:

[0007] A face fake detection classification model is constructed, wherein the face fake detection classification model comprises a face key point template generation module Gm, an RGB feature extraction module Fr, a mask feature extraction module Fm and a mask local attention module Bm.

[0008] A training sample image X is obtained.

[0009] The face fake detection classification model is trained using the training sample image X to obtain a trained face fake detection classification model.

[0010] A face image to be detected is obtained.

[0011] The face image to be detected is input into the trained face fake detection classification model to obtain a recognition result.

[0012] Further, the training of the face fake detection classification model using the training sample image X to obtain a trained face fake detection classification model comprises:

[0013] The face key point template generation module Gm is used to generate a face key point template M of the training sample image X.

[0014] The RGB feature extraction module Fr is used to extract features fr from the training sample image X.

[0015] The mask feature extraction module Fm is used to extract features fm from the face key point template M.

[0016] The face key point template M and the mask local attention module Bm are used to fuse the features fr and the features fm, wherein the mask local attention module comprises an adaptive learning weight submodule and a channel fusion submodule.

[0017] obtaining outputs of a classification block of the mask feature extraction module Fm and a classification block of the RGB feature extraction module Fr, and adding the output of the classification block of the mask feature extraction module Fm to the output of the classification block of the RGB feature extraction module Fr to obtain an output result;

[0018] performing binary classification on the output result to determine authenticity of the input image;

[0019] performing gradient update on network model weights of the face forgery detection classification model according to a loss function and an optimizer to obtain a trained face forgery detection classification model.

[0020] Further, the face key point template generation module Gm generates a face key point template M of the training sample image X, including:

[0021] obtaining 68 key points of the training sample image X by using a dlib library function, and saving the 68 key points of the training sample image X in a label document for reading;

[0022] normalizing coordinates (x, y) of the 68 key points of the training sample image X to obtain key points p1 to p68;

[0023] restoring the (x, y) coordinates of the key points p1 to p68 into pixel coordinates;

[0024] generating five templates of left eye, right eye, nose, mouth and face contour according to a polygon filling method and the (x, y) coordinates of the key points p1 to p68, and performing convolution operation on each template using a convolution kernel C1 with a size of 5*5 and a kernel value of 1, and then obtaining mask1 to mask5 by using a Relu1 activation function;

[0025] splicing the mask1 to mask5 by channel to obtain the face key point template M.

[0026] Further, the RGB feature extraction module Fr extracts features of the training sample image X to obtain features fr, including:

[0027] inputting the training sample image X into the RGB feature extraction module Fr; wherein the RGB feature extraction module Fr includes a ResNet50 and a Fr classification block, the ResNet50 includes five stages S1, S2, S3, S4 and S5, and the Fr classification block includes a global average pooling and a fully connected layer;

[0028] The five stages in the RGB feature extraction module Fr are used for feature extraction respectively, and features fri are obtained, wherein i={1, 2, 3, 4, 5}.

[0029] Further, the mask feature extraction module Fm is used to extract features from the face key point template M, and features fm are obtained.

[0030] The face key point template M is input into the mask feature extraction module Fm; wherein the mask feature extraction module Fm includes a stage network and an Fm classification block, the stage network includes five stages, namely Sm1, Sm2, Sm3, Sm4 and Sm5, and the Fm classification block includes a global average pooling and a full connection layer.

[0031] The five stages in the mask feature extraction module Fm are used for feature extraction respectively, and features fmi are obtained, wherein i={1, 2, 3, 4, 5}.

[0032] Further, the stage network includes:

[0033] Sm1 includes a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 7*7, a filter of 5 and a stride of 2 connected in sequence;

[0034] Sm2 includes two DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5 and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5 and a stride of 2 connected in sequence;

[0035] Sm3 includes three DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5 and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5 and a stride of 2 connected in sequence;

[0036] Sm4, comprising 5 "DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1 connected in sequence, and one "DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2 connected in sequence;

[0037] Sm5, comprising 2 "DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1 connected in sequence, and one "DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2 connected in sequence.

[0038] Further, the feature fr and the feature fm are fused by using the face key point template M and the mask local attention module Bm; wherein the mask local attention module comprises an adaptive learning weight submodule and a channel fusion submodule, comprising:

[0039] The feature fm is input into the adaptive learning weight submodule, and each channel learns the attention of each pixel position by using the face key point template M through the adaptive learning weight submodule;

[0040] The output of the adaptive learning weight submodule is obtained, and the output of the adaptive learning weight submodule is summed in the channel direction by using the channel fusion submodule to obtain an attention map;

[0041] The attention map is normalized by using a sigmoid activation function to obtain fatt;

[0042] The fatt is multiplied by the feature fr element by element, and then the element corresponding feature fr is added to obtain a feature fr'.

[0043] Further, the outputs of the classification block of the mask feature extraction module Fm and the classification block of the RGB feature extraction module Fr are obtained, and the output of the classification block of the mask feature extraction module Fm is added to the output of the classification block of the RGB feature extraction module Fr to obtain an output result, comprising:

[0044] The feature fm is input into the classification block of the mask feature extraction module Fm, and the output of the classification block of the mask feature extraction module Fm is obtained;

[0045] input the feature fr' into a classification block of the RGB feature extraction module Fr, and obtain an output of the classification block of the RGB feature extraction module Fr;

[0046] add the output of the classification block of the mask feature extraction module Fm and the output of the classification block of the RGB feature extraction module Fr to obtain an output result.

[0047] In a second aspect, the present application further provides a terminal, comprising:

[0048] one or more processors;

[0049] a storage device configured to store one or more programs;

[0050] a camera configured to capture images;

[0051] When the one or more programs are executed by the one or more processors, the one or more processors implement the face key point based fake image detection method as described above.

[0052] In a third aspect, the present application further provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to perform the face key point based fake image detection method as described above.

[0053] Compared with the prior art, the face key point based fake image detection method, the terminal and the storage medium have the following advantages:

[0054] (I) The present application is designed reasonably. The mask of each organ region of the face is obtained through face key point detection, and the mask adaptively learns the weight feature to guide the RGB feature to focus on the local core region of the face, and the redundant background information is discarded. In this way, the model can find the fine-grained artifacts after the forgery. At the same time, the complete RGB feature can detect whether the face is deformed from a global perspective.

[0055] (II) The face forgery detection classification model involved in the present application can also find traces of forgery operation from local and global scales, thereby effectively detecting fake face images. The present application detects the fake face from multiple scales, has high detection accuracy and good generalization ability, and can be widely used in the field of fake face image detection. BRIEF DESCRIPTION OF DRAWINGS

[0056] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. The accompanying drawings should not be construed as an improper limitation of the present application. In the drawings:

[0057] Figure 1A face key point based fake image detection method flow chart according to the first embodiment of the present application;

[0058] Figure 2 A face key point based fake image detection method flow chart according to the second embodiment of the present application;

[0059] Figure 3 A structure schematic diagram of the convolution C1 according to the second embodiment of the present application;

[0060] Figure 4 A schematic diagram of the mask according to the second embodiment of the present application;

[0061] Figure 5 A schematic diagram of the RGB feature extraction module Fr according to the second embodiment of the present application;

[0062] Figure 6 A schematic diagram of the mask feature extraction module Fm according to the second embodiment of the present application;

[0063] Figure 7 A fusion process schematic diagram of the feature fr and the feature fm according to the second embodiment of the present application;

[0064] Figure 8 A structure schematic diagram of the face fake detection classification model according to the second embodiment of the present application;

[0065] Figure 9 A flow chart of obtaining an output result according to the second embodiment of the present application;

[0066] Figure 10 A structure schematic diagram of the face key point based fake image detection device according to the third embodiment of the present application. DETAILED DESCRIPTION

[0067] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0068] Embodiment one

[0069] Figure 1 A face key point based fake image detection method flow chart according to the first embodiment of the present application, the present embodiment can be used for face fake image detection and recognition, referring to Figure 1 , specifically comprising the following steps:

[0070] Step 101, constructing a face forgery detection classification model; wherein the face forgery detection classification model comprises a face key point template generation module Gm, an RGB feature extraction module Fr, a mask feature extraction module Fm, and a mask local attention module Bm.

[0071] The face key point refers to locating the key region position of the face part of a given face image, including eyebrows, eyes, nose, mouth, face contour, etc. By determining the face key point, it is beneficial to focus on the subsequent detection. The original RGB feature of the RGB color image contains the most abundant discriminative information, which is beneficial to improve the accuracy of subsequent detection.

[0072] The mask is a mask. The concept of mask in digital image processing is borrowed from the process of PCB manufacturing. In semiconductor manufacturing, many chip process steps use photolithography technology. The "negative" used for these steps is called a mask (also called a "mask"). Its function is to cover a non-transparent pattern template in the selected area on the silicon wafer, and then the subsequent etching or diffusion will only affect the area outside the selected area. Image masks are similar to them. By selecting images, graphics or objects, the processed image (all or part) is shielded to control the image processing area or processing process.

[0073] Step 102, obtaining a training sample image X.

[0074] In order to realize the training of the face forgery detection classification model to obtain better detection effect, the training sample image X can use the existing training sample. Illustratively, the face image contained in the training sample is a front face occupying more than 50% of the area of the whole image, and the resolution of the image can be 224*224.

[0075] Step 103, training the face forgery detection classification model using the training sample image X to obtain a trained face forgery detection classification model.

[0076] By training the face forgery detection classification model using the training sample image X, a trained face forgery detection classification model can be obtained. In the actual training process, the training sample can be divided into a training set and a test set to achieve better training effect.

[0077] Step 104, obtaining a face image to be detected.

[0078] Since the forged face image is generally in the form of a picture or a video, in the actual detection process, the picture or the video screenshot can be directly obtained, or the image can be collected through an image collection device, and the image can be appropriately preprocessed by a person skilled in the art according to actual needs, including but not limited to cropping, repairing and the like, so that the obtained to-be-detected image can be better detected by the trained face forgery detection classification model.

[0079] Step 105, inputting the to-be-detected face image into the trained face forgery detection classification model to obtain a recognition result.

[0080] The trained face forgery detection classification model can detect the image of the to-be-detected face image to determine whether the to-be-detected face image is real or forged, so as to feed back the recognition result to the user, thereby achieving the recognition effect of the forged face image.

[0081] The face key point-based forged image detection method provided in the embodiment can construct a face forgery detection classification model, wherein the face forgery detection classification model comprises a face key point template generation module Gm, an RGB feature extraction module Fr, a mask feature extraction module Fm and a mask local attention module Bm; a training sample image X is obtained; the face forgery detection classification model is trained by using the training sample image X to obtain a trained face forgery detection classification model; a to-be-detected face image is obtained; the to-be-detected face image is input into the trained face forgery detection classification model to obtain a recognition result, so that the forged face image can be accurately detected, and the influence of the forged face image on real life can be reduced.

[0082] Embodiment Two

[0083] Figure 2The embodiment is based on the above-mentioned embodiment and is optimized. In the embodiment, the training of the face forgery detection classification model by using the training sample image X is performed to obtain a trained face forgery detection classification model. The optimization is specifically as follows: the face key point template M of the training sample image X is generated by using the face key point template generation module Gm; the feature fr is obtained by performing feature extraction on the training sample image X by using the RGB feature extraction module Fr; the feature fm is obtained by performing feature extraction on the face key point template M by using the mask feature extraction module Fm; the features fr and fm are fused by using the face key point template M and the mask local attention module Bm. The mask local attention module includes an adaptive learning weight submodule and a channel fusion submodule. The output of the classification block of the mask feature extraction module Fm and the output of the classification block of the RGB feature extraction module Fr are obtained, and the output of the classification block of the mask feature extraction module Fm is added to the output of the classification block of the RGB feature extraction module Fr to obtain an output result. The output result is binary classified to determine the authenticity of the input image. The gradient update of the network model weight of the face forgery detection classification model is performed according to a loss function and an optimizer to obtain the trained face forgery detection classification model.

[0084] Correspondingly, the face forgery detection method based on face key points provided in the embodiment specifically includes the following steps.

[0085] Step 201, constructing a face forgery detection classification model; wherein the face forgery detection classification model includes a face key point template generation module Gm, an RGB feature extraction module Fr, a mask feature extraction module Fm, and a mask local attention module Bm.

[0086] Step 202, obtaining a training sample image X.

[0087] Step 203, generating a face key point template M of the training sample image X by using the face key point template generation module Gm.

[0088] Specifically, first, the 68 key points of the training sample image X can be obtained by using a dlib library function, and the 68 key points of the training sample image X are saved in a labeling document for reading. For example, in the training process, the 68 key points are saved in a labeling document, and when needed, the labeling document is read instead of re-detecting the key point information, so that the training speed is accelerated.

[0089] Secondly, the 68 key point coordinates (x, y) of the training sample image X are normalized to obtain key points p1 to p68. For example, the normalization formula is as follows: x = Hi / H; y = Wi / W. In the formula, Hi (ranging from 1 to H) is the height of the image in which the i-th key point is located; and Wi (ranging from 1 to W) is the width of the image in which the i-th key point is located. H and W respectively represent the height and width of the entire training sample image X. In addition, H and W herein are usually 224 pixels, and a person skilled in the art can also make reasonable adjustments according to needs. Each element pi (i ∈ [1~68]) in p1 to p68 herein respectively represents the normalized coordinate position (x, y) of the point on the face image X, and the value range of x and y is 0 to 1.

[0090] Thirdly, the (x, y) coordinates of the key points p1 to p68 are all restored to pixel coordinates. For example, the (x, y) coordinates of the key point pi are restored to pixel coordinates by multiplying pi(x, y) by the height and width of the preprocessed RGB input image. The purpose of this is to change the width and height of the image by resize operation when the training sample image X is input into the model, so as to prevent the coordinates of the face key points from being misaligned on the image after resize.

[0091] Then, five templates of left eye, right eye, nose, mouth and face contour are generated according to the polygon filling method and the (x, y) coordinates of the key points p1 to p68, and a convolution kernel of 5*5 and a kernel value of 1 is used for convolution C1 to perform convolution operation on each template, and then Relu1 activation function is used to obtain mask1 to mask5. For example, the Relu1 activation function is as follows: in the formula, x is the input feature, and y is the output feature.

[0092] Relu1 activation: y = min((max(0, x)), 1)

[0093] In addition, a person skilled in the art can also adjust the number of templates according to actual detection needs, for example, selecting any one or several of the left eye, right eye, nose, mouth and face contour to generate a template and performing corresponding operation.

[0094] Figure 3 The structure diagram of the convolution C1 in the second embodiment of the application is shown in FIG. 2. Figure 3 The pixel value of the mask is 0 and 1, the value of the edge between the background and the foreground is 1, the foreground is also 1, the background is 0, which makes the mask include both the foreground area and the edge area of the face. This is because the artifacts of face forgery are more likely to appear in the foreground, the edge between the foreground and the background, and the region of each tissue and organ.

[0095] Finally, the mask1 to mask5 are spliced by channel to obtain a face key point template M. Figure 4 For the schematic diagram of the mask in the second embodiment of the present application, see Figure 4 For example, the key point information of the eyes: p37 to p42 are the left eyes, and p43 to p48 are the right eyes. The nose is p28 to p36, the mouth is p49 to p60, and the contour is p1 to p27.

[0096] In step 204, the RGB feature extraction module Fr is used to perform feature extraction on the training sample image X to obtain a feature fr.

[0097] Figure 5 For the schematic diagram of the RGB feature extraction module Fr in the second embodiment of the present application, see Figure 5 . Specifically, the training sample image X is first input into the RGB feature extraction module Fr; wherein the RGB feature extraction module Fr includes a ResNet50 and a Fr classification block, the ResNet50 includes five stages, respectively S1, S2, S3, S4, S5; the Fr classification block includes a global average pooling and a full connection layer. Then the five stages in the RGB feature extraction module Fr are used to perform feature extraction respectively to obtain a feature fri, wherein i={1, 2, 3, 4, 5}.

[0098] In actual application process, by inputting the RGB face image into the RGB feature extraction module Fr, the features fr1, fr2, fr3, fr4, fr5 are obtained respectively, and these features can be fused with the features extracted by the mask.

[0099] In step 205, the mask feature extraction module Fm is used to perform feature extraction on the face key point template M to obtain a feature fm.

[0100] Figure 6 For the schematic diagram of the mask feature extraction module Fm in the second embodiment of the present application, see Figure 6Specifically, the face key point template M can be input into the mask feature extraction module Fm first; wherein the mask feature extraction module Fm includes a stage network and an Fm classification block, the stage network includes five stages, namely Sm1, Sm2, Sm3, Sm4 and Sm5, and the Fm classification block includes a global average pooling and a full connection layer. Then, the five stages in the mask feature extraction module Fm are used for feature extraction respectively to obtain features fmi, wherein i={1, 2, 3, 4, 5}. For example, the face key point template M is input into the mask feature extraction module Fm, and each stage obtains features fm1, fm2, fm3, fm4 and fm5 respectively.

[0101] Since the texture or pattern of the mask feature is less, a too deep network is not needed, and the features extracted by a deeper network have little guiding significance, therefore, the stage network with five stages is used to correspond to the RGB feature extraction module.

[0102] For example, the stage network includes:

[0103] Sm1 includes a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 7*7, a filter of 5 and a stride of 2 connected in sequence.

[0104] Sm2 includes two DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5 and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5 and a stride of 2 connected in sequence.

[0105] Sm3 includes three DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5 and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5 and a stride of 2 connected in sequence.

[0106] Sm4, including 5 "DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1" connected in sequence, and one "DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2" connected in sequence.

[0107] Sm5, including 2 "DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1" connected in sequence, and one "DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2" connected in sequence.

[0108] Here, the number of convolution operations for mask feature extraction is guaranteed to be the same as the number of RGB feature extraction, which makes the RGB feature fri and the mask feature fmi have the same receptive field when they are fused. In this way, the same size receptive field is obtained. The Fm classification block is used to learn the mask data and determine which face region is more prone to artifacts, so that the mask can better guide the RGB feature to learn the more important region.

[0109] Step 206, fusing the feature fr and the feature fm by using the face key point template M and the mask local attention module Bm; wherein the mask local attention module includes an adaptive weight learning submodule and a channel fusion submodule.

[0110] For example, the adaptive weight learning submodule is connected in sequence with one DepthwiseConvolution convolution with a convolution kernel of 3*3, a filter of 5, and a stride of 1. Since each channel of the mask represents a different face region, the face key point template M allows each channel to learn the attention of each pixel position independently through the adaptive weight learning submodule, so as to ensure that each channel does not interfere with each other and learns independently.

[0111] Figure 7 For the fusion process of the feature fr and the feature fm of the second embodiment of the present application, see Figure 7In actual application, the feature fmi can be input into the adaptive learning weight submodule first, and each channel can learn the attention of each pixel position by the adaptive learning weight submodule using the face key point template M. Then the output of the adaptive learning weight submodule is obtained, and the outputs of the adaptive learning weight submodule are summed in the channel direction using the channel fusion submodule to obtain an attention map. The attention map is normalized by using a sigmoid activation function to obtain fatti. Finally, the fatti is multiplied by the feature fri element by element, and then the feature fri corresponding to the element is added to obtain the feature fri'.

[0112] Figure 8 The structure diagram of the face forgery detection classification model of the second embodiment of the present application is shown in Figure 8 In actual application, after the face image to be detected is input into the face forgery detection classification model, the following steps are experienced:

[0113] First, a face key point template M of the image X is generated by using a face key point template generation module Gm. The face key point template M includes five face key point elements of left eye, right eye, nose, mouth and face contour.

[0114] Secondly, the feature fr of the input RGB image is extracted by using an RGB feature extraction module Fr.

[0115] Thirdly, the feature fm of the input face key point template M is extracted by using a mask feature extraction module Fm.

[0116] Then, the fr and the fm are fused by using a mask local attention module Bm.

[0117] Finally, the outputs of the classification block of the mask feature extraction module Fm and the classification block of the RGB feature extraction module Fr are added for binary classification to determine whether the input image is fake or real, that is, the recognition result of the face image to be detected can be obtained.

[0118] Step 207, the outputs of the classification block of the mask feature extraction module Fm and the classification block of the RGB feature extraction module Fr are obtained, and the output of the classification block of the mask feature extraction module Fm is added to the output of the classification block of the RGB feature extraction module Fr to obtain an output result.

[0119] Figure 9 The flowchart for obtaining the output result of the second embodiment of the present application is shown in Figure 9Specifically, the obtained feature fm5 can be input into the classification block of the mask feature extraction module Fm, and the output of the classification block of the mask feature extraction module Fm is obtained. Then the feature fr5' is input into the classification block of the RGB feature extraction module Fr, and the output of the classification block of the RGB feature extraction module Fr is obtained. Finally, the output of the classification block of the mask feature extraction module Fm is added to the output of the classification block of the RGB feature extraction module Fr to obtain the output result.

[0120] For example, the specific method of using the mask local attention module Bm to fuse the feature fri and the feature fmi to obtain the feature fr5' is as follows:

[0121] First, the feature fm1 is input into the adaptive learning weight submodule, and the output of the adaptive learning weight submodule is summed in the channel direction by using the channel fusion submodule to obtain an attention map. Then the attention map is normalized by using a sigmoid activation function to obtain fatt1 corresponding to the feature fm1. Finally, fatt1 is multiplied by fr1 and then added to fr1 to obtain fr1'.

[0122] Secondly, the feature fm2 is input into the adaptive learning weight submodule, and the output of the adaptive learning weight submodule is summed in the channel direction by using the channel fusion submodule to obtain an attention map. Then the attention map is normalized by using a sigmoid activation function to obtain fatt2 corresponding to the feature fm2. Finally, fatt2 is multiplied by fr2 and then added to fr2 to obtain fr2'. Wherein, fr2 is fr1' obtained by fusing fm1 and fr1.

[0123] Thirdly, the feature fm3 is input into the adaptive learning weight submodule, and the output of the adaptive learning weight submodule is summed in the channel direction by using the channel fusion submodule to obtain an attention map. Then the attention map is normalized by using a sigmoid activation function to obtain fatt3 corresponding to the feature fm3. Finally, fatt3 is multiplied by fr3 and then added to fr3 to obtain fr3'. Wherein, fr3 is fr2' obtained by fusing fm2 and fr2.

[0124] Then, the feature fm4 is input into the adaptive learning weight submodule, and the output of the adaptive learning weight submodule is summed in the channel direction by using the channel fusion submodule to obtain an attention map. Then the attention map is normalized by using a sigmoid activation function to obtain fatt4 corresponding to the feature fm4. Finally, fatt4 is multiplied by fr4 and then added to fr4 to obtain fr4'. Wherein, fr4 is fr3' obtained by fusing fm3 and fr3.

[0125] Finally, the feature fm5 is input into the adaptive learning weight submodule, and the output of the adaptive learning weight submodule is summed in the channel direction by using the channel fusion submodule to obtain an attention map. Then, the attention map is normalized by using a sigmoid activation function to obtain fatt5 corresponding to the feature fm5. Finally, fatt5 is multiplied by fr5 and then added to fr5 to obtain fr5'. Wherein, fr5 is fr4' obtained by fusing fm4 and fr4.

[0126] Step 208, the output result is binary classified to determine the authenticity of the input image. Exemplarily, the binary classification can use existing SVM, decision tree, FM, etc., and a person skilled in the art can also select according to actual needs, which will not be repeated here.

[0127] Step 209, gradient update of network model weight of the face fake detection classification model according to the loss function and the optimizer, to obtain a trained face fake detection classification model. Exemplarily, the loss function is cross-entropy loss or binary cross-entropy loss, the optimizer can be SGD or Adam, and the learning rate can be 0.0002 and 0.0001. A person skilled in the art can also select other suitable loss functions and optimizers to perform gradient update of network model weight of the face fake detection classification model according to actual needs, which will not be repeated here.

[0128] Step 210, obtaining a face image to be detected;

[0129] Step 211, inputting the face image to be detected into the trained face fake detection classification model to obtain a recognition result.

[0130] The face key point-based fake image detection method provided in the embodiment is characterized in that the face key point-based fake image detection method comprises the following steps: training the face fake detection classification model by using the training sample image X to obtain a trained face fake detection classification model, and the specific optimization is as follows: generating a face key point template M of the training sample image X by using a face key point template generation module Gm; extracting features of the training sample image X by using an RGB feature extraction module Fr to obtain features fr; extracting features of the face key point template M by using a mask feature extraction module Fm to obtain features fm; fusing the features fr and the features fm by using the face key point template M and a mask local attention module Bm; wherein the mask local attention module comprises an adaptive learning weight submodule and a channel fusion submodule; obtaining outputs of a classification block of the mask feature extraction module Fm and a classification block of the RGB feature extraction module Fr, and adding the output of the classification block of the mask feature extraction module Fm to the output of the classification block of the RGB feature extraction module Fr to obtain an output result; performing binary classification on the output result to determine the authenticity of an input image; and performing gradient update on network model weights of the face fake detection classification model according to a loss function and an optimizer to obtain the trained face fake detection classification model.

[0131] The face fake detection classification model provided in the embodiment can obtain templates of various organ parts by performing key point detection on a face region, and use the templates as prior knowledge to guide the model to pay attention to regions where artifacts are more likely to occur, thereby obtaining more accurate recognition results.

[0132] In addition, the backbone network model used for RGB feature extraction is not modified in the embodiment, so that a person skilled in the art can fine-tune the backbone network pre-trained on a large-scale dataset by using transfer learning in the actual application process. Generally, such fine-tuning can make the model converge faster and improve the detection accuracy.

[0133] Meanwhile, since the main parameters and the amount of calculation are in the RGB feature extraction module Fr, the RGB feature extraction module Fr can be well selected to use various models. Specifically, a lightweight model or a complex model can be used according to the actual time and accuracy requirements. Compared with a multi-branch network model, the model used in the application is a single-branch network model Fr module and a lightweight mask feature extraction module Fm, which has advantages in detection time and model training difficulty, is conducive to improving the model training efficiency, and shortens the actual detection time.

[0134] Embodiment three

[0135] Figure 10A structure schematic diagram of the device for detecting a forged image based on a face key point according to Embodiment Three of the present application is described; Figure 10 A block diagram of an exemplary terminal system suitable for implementing embodiments of the present application is shown. Figure 10 The terminal system shown is merely one example and should not be taken as limiting the scope of the functionality or use of embodiments of the present application.

[0136] As shown, Figure 10 The terminal 12 is represented in the form of a general purpose computing device. The components of the terminal 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processing unit 16.

[0137] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics bus (e.g., AGP or Accelerated Graphics Port bus), and a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0138] The terminal 12 typically includes a variety of computer system readable media. Such media can be any available media that is located either in or out of the computing device 12, such as volatile and non-volatile media, removable and non-removable media.

[0139] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The terminal 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Figure 10 not shown, a magnetic hard disk drive for reading from and writing to non-removable, non-volatile magnetic media (e.g., a "hard drive"). Although not specifically shown, such can Figure 10 In alternative embodiments, a magnetic hard disk drive (not shown) can be provided for reading from and writing to a removable, non-volatile magnetic medium (e.g., a "floppy disk"), and an optical disk drive (not shown) can be provided for reading from or writing to a removable, non-volatile optical medium (e.g., a CD-ROM, DVD-ROM or other optical media). In these instances, each drive can be connected to the bus 18 by one or more data media interfaces. The memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the present application.

[0140] Program / utility 40 having a set of program modules 42 can be stored in memory 28 by way of example, such program modules 42 include an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which may

[0141] Terminal 12 can also be communicatively coupled to one or more external devices 14, such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with terminal 12; and / or any devices (e.g., network card, modem, etc.) that enable terminal 12 to communicate with one or more other computing devices. Such communication can be facilitated by input / output (I / O) interfaces 22. Still yet, terminal 12 can be communicatively coupled to one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and / or the Internet, through network adapter 20, as is depicted in FIG. 1. It should be appreciated that the network adapter 20 can also be used to enable connection to other types of networks and / or remote computing devices not shown in FIG. 1, such as mobile networks (e.g., 3G, 4G, etc.), cellular networks, etc. It will be appreciated that the network connections shown and described herein are illustrative and other means of establishing a communications link between the computing devices can be used. Still further, the term "cracking" as used herein is not meant to limit the scope of the present application to any particular type of cracking, but is meant to encompass any type of cracking, including, but not limited to, image cracking, video cracking, etc.

[0142] The processing unit 16 performs various functions and data processing by running programs stored in the system memory 28, such as implementing the face key point based fake image detection method provided by the embodiments of the present application.

[0143] Embodiment Four

[0144] The embodiment four of the present application also provides a storage medium containing computer executable instructions, which when executed by a computer processor, are used to perform the face key point based fake image detection method provided by any of the above embodiments.

[0145] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0146] The computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which computer-readable program code is embodied. Such propagated data signals can take a wide variety of forms, including but not limited to electro-magnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium that is not a storage medium, that is, that is not a tangible medium, and that can communicate, propagate or transport programming for use by or in connection with an instruction execution system, apparatus, or device.

[0147] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the above.

[0148] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments of the present application, electronic mail (email) can be utilized as the

[0149] Note that the above merely describes preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, modifications and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A face key point based fake image detection method, characterized in that, The method comprises the following steps: constructing a face forgery detection classification model; wherein the face forgery detection classification model comprises a face key point template generation module Gm, an RGB feature extraction module Fr, a mask feature extraction module Fm, and a mask local attention module Bm; obtaining a training sample image X; training the face forgery detection classification model using the training sample image X to obtain a trained face forgery detection classification model; obtaining a face image to be detected; inputting the face image to be detected into the trained face forgery detection classification model to obtain a recognition result; the training of the face forgery detection classification model using the training sample image X to obtain a trained face forgery detection classification model comprises: generating a face key point template M of the training sample image X using the face key point template generation module Gm; extracting features fr of the training sample image X using the RGB feature extraction module Fr; extracting features fm of the face key point template M using the mask feature extraction module Fm; fusing the features fr and the features fm using the face key point template M and the mask local attention module Bm; wherein the mask local attention module comprises an adaptive learning weight submodule and a channel fusion submodule; obtaining the output of the classification block of the mask feature extraction module Fm and the output of the classification block of the RGB feature extraction module Fr, and adding the output of the classification block of the mask feature extraction module Fm to the output of the classification block of the RGB feature extraction module Fr to obtain an output result; performing binary classification on the output result to determine the authenticity of the input image; updating the gradient of the network model weight of the face forgery detection classification model according to a loss function and an optimizer to obtain a trained face forgery detection classification model; the fusion of the features fr and the features fm using the face key point template M and the mask local attention module Bm; wherein the mask local attention module comprises an adaptive learning weight submodule and a channel fusion submodule, which comprises: inputting the features fm into the adaptive learning weight submodule, and using the face key point template M to let each channel learn the attention of each pixel position through the adaptive learning weight submodule; obtaining the output of the adaptive learning weight submodule, and using the channel fusion submodule to sum the output of the adaptive learning weight submodule in the channel direction to obtain an attention map; normalizing the attention map using a sigmoid activation function to obtain fatt; multiplying fatt and the features fr element by element, and then adding the features fr corresponding to the elements to obtain features fr'; The output of the classification block of the mask feature extraction module Fm and the output of the classification block of the RGB feature extraction module Fr are acquired, and the output of the classification block of the mask feature extraction module Fm is added to the output of the classification block of the RGB feature extraction module Fr to obtain an output result, including: The feature fm is input into the classification block of the mask feature extraction module Fm, and the output of the classification block of the mask feature extraction module Fm is acquired; The feature fr' is input into the classification block of the RGB feature extraction module Fr, and the output of the classification block of the RGB feature extraction module Fr is acquired; The output of the classification block of the mask feature extraction module Fm is added to the output of the classification block of the RGB feature extraction module Fr to obtain an output result.

2. The method of claim 1, wherein, The face key point template generation module Gm generates a face key point template M of the training sample image X, including: The 68 key points of the training sample image X are acquired by using a dlib library function, and the 68 key points of the training sample image X are saved in a label document for reading; The (x, y) coordinates of the 68 key points of the training sample image X are normalized to obtain key points p1 to p68; The (x, y) coordinates of the key points p1 to p68 are restored to pixel coordinates; According to the (x, y) coordinates of the key points p1 to p68, five templates of left eye, right eye, nose, mouth and face contour are generated by using a polygon filling method, and then a convolution C1 with a convolution kernel of 5*5 and a kernel value of 1 is used for convolution operation on each template, and then a Relu1 activation function is used to obtain mask1 to mask5; The mask1 to mask5 are spliced by channel to obtain a face key point template M.

3. The method of claim 2, wherein, The RGB feature extraction module Fr is used for feature extraction on the training sample image X to obtain a feature fr, including: The training sample image X is input into the RGB feature extraction module Fr; wherein the RGB feature extraction module Fr includes a ResNet50 and a Fr classification block, the ResNet50 includes five stages S1, S2, S3, S4 and S5; the Fr classification block includes a global average pooling and a fully connected layer; The five stages in the RGB feature extraction module Fr are used for feature extraction respectively to obtain a feature fri, wherein i={1, 2, 3, 4, 5}.

4. The method of claim 2, wherein, The mask feature extraction module Fm is used for feature extraction on the face key point template M to obtain a feature fm, including: The face key point template M is input into the mask feature extraction module Fm; wherein the mask feature extraction module Fm includes a stage network and a Fm classification block, the stage network includes five stages Sm1, Sm2, Sm3, Sm4 and Sm5, and the Fm classification block includes a global average pooling and a fully connected layer; The five stages in the mask feature extraction module Fm are used for feature extraction respectively, and the features fmi are obtained, where i={1, 2, 3, 4, 5}.

5. The method of claim 4, wherein, The stage network comprises: Sm1 comprises DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 7*7, a filter of 5, and a stride of 2 connected in sequence; Sm2 comprises two DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2 connected in sequence; Sm3 comprises three DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2 connected in sequence; Sm4 comprises five DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2 connected in sequence; Sm5 comprises two DepthwiseConvolution-BatchNorm-ReLU layers with a convolution kernel of 3*3, a filter of 5, and a stride of 1 connected in sequence, and a DepthwiseConvolution-BatchNorm-ReLU layer with a convolution kernel of 3*3, a filter of 5, and a stride of 2 connected in sequence.

6. A terminal, characterized by comprising: comprise: one or more processors; a storage device for storing one or more programs; a camera for capturing images; When the one or more programs are executed by the one or more processors, the one or more processors implement the face key point based fake image detection method as claimed in claims 1-5.

7. A storage medium containing computer executable instructions for performing the face key point based fake image detection method as claimed in claims 1-5 when executed by a computer processor.

Images