A deep learning method for face anti-fraud

Abstract

The present invention discloses a deep learning method for face anti-fraud, comprising the following steps: building an MMCNN network framework, allowing multi-modal data input and fully integrating its information; modal expansion, in order to improve the accuracy, the original tri-modal data is expanded by two additional modalities by converting its color space; calculating the loss function; quantitative analysis of different modalities of true and false pictures, by quantitatively analyzing the CIS IA-SURF data set, to determine the similarity of true and false face pictures of each modality. By increasing the modality to improve the model accuracy of the MMCNN network framework, not only can the accuracy of the model be enhanced, but it can also help increase the stability of the model.

Description

Technical Field

[0001] The present invention relates to the field of face recognition, and specifically to a deep learning method for face anti-fraud. Background Art

[0002] In addition to the field of pedestrian re-identification, face anti-fraud also has many applications in other fields. For example, face verification in biometric technology has been widely used in many scenarios in our real life, such as mobile payment, face verification door locks, identity verification, etc. Unfortunately, these systems are sometimes attacked by fake faces, which leads to major security risks. For example, if the mobile phone face verification payment system is deceived by the attacker, it will cause economic losses. In today's information age, it is not difficult to obtain the other party's "fake face". It can be easily obtained through the other party's circle of friends, Weibo, Douyin and other social software.

[0003] Face anti-spoofing is not only the basic work of face recognition / verification / retrieval systems, but also important for robust pedestrian re-identification systems, so we believe that research on this issue is very meaningful. Summary of the invention

[0004] In order to solve the deficiencies of the prior art, the present invention provides the following technical solutions:

[0005] A deep learning method for face anti-fraud includes the following steps:

[0006] Build the MMCNN network framework, which allows multi-modal data input and can fully integrate its information;

[0007] Modal expansion: In order to improve the accuracy, the original three-modal data is expanded by two additional modalities by converting its color space;

[0008] Calculate the loss function;

[0009] Quantitative analysis of different modalities of real and fake images,The similarity of real and fake face images in each modality is determined by quantitatively analyzing the CISIA-SURF dataset.

[0010] The above-mentioned deep learning method for face anti-spoofing expands the RGB image to two color space modes, HSV and YCbCr, and uses ResNet-34 to extract features of the data of several modes layer by layer, extracts the features after res2, connects these features, and then forms a new set of features, and performs convolution on this new set of features.

[0011] The above-mentioned deep learning method for face anti-spoofing makes a SElayer for the features after res3, and makes each modality channel-weighted; the entire SElayer is divided into two parts: compression and activation. The compression part performs an average pooling on the entire feature map in the channel dimension, and the activation part allows the channel to learn weights through two fully connected layers.

[0012] In the above-mentioned deep learning method for face anti-fraud, the formula of the compression part is:

[0013] In the above-mentioned deep learning method for face anti-fraud, the formula of the activation part is:

[0014] s=F e (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0015] in σ is the activation function and r is the scaling parameter.

[0016] The above-mentioned deep learning method for face anti-spoofing connects SEFeature and performs connected convolution to obtain a set of features called CatSEFeature, and makes a residual between the middle-level feature CatFrature and the high-level feature CatSEFeature, and then sends the feature to res4 and res5 in sequence; after res5, the feature is pooled and stretched to change the feature map to 1*1, and then a fully connected layer is used, and the result is softmax classified to obtain the scores of real faces and fake faces.

[0017] In the above-mentioned deep learning method for face anti-spoofing, the modal expansion converts the original visible light image into an HSV color space image and a YCbCr image, separates chromaticity and brightness, and provides more detailed texture features for the entire model.

[0018] In the above-mentioned deep learning method for face anti-spoofing, the loss function assists the cross entropy loss by expanding the inter-class distance between the clustering centers of the real face and fake face features.

[0019] In the above-mentioned deep learning method for face anti-spoofing, the quantitative analysis is performed by extracting LBP features from several modes of the face to be identified, calculating the histogram of the LBP features of each mode, and calculating the chi-square distance of the histogram.

[0020] In the above-mentioned deep learning method for face anti-fraud, the chi-square distance of the calculated histogram is: The larger the chi-square distance, the greater the difference between the real and fake faces in this mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0022] Figure 1 A schematic diagram of the overall architecture of the MMCNN provided in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram comparing the ResNet structure provided by an embodiment of the present invention with VGG-19;

[0024] Figure 3 A schematic diagram of the SELayer structure provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of ResNetDE provided by SELayer in an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of the expansion of RGB data to HSV and YCbCr modes provided in an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of the HSV color space provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of comparing various channels of an HSV image provided by an embodiment of the present invention;

[0029] Figure 8 A schematic diagram of the YCbCr chi-square distance between a true and a false face provided in an embodiment of the present invention;

[0030] Fig. 9 A schematic diagram of the Push loss principle provided by an embodiment of the present invention;

[0031] Fig.10 A schematic diagram of the comparison between the real and fake face LBP images provided by an embodiment of the present invention;

[0032] Fig.11 A schematic diagram of the LBP chi-square distance between two real faces provided in an embodiment of the present invention;

[0033] Fig.12 A schematic diagram of the chi-square distances between true and false faces of different modalities provided by an embodiment of the present invention;

[0034] Fig.13 A schematic diagram of the LBP chi-square distance of each channel of HSV and YcbCr provided in an embodiment of the present invention;

[0035] Fig.14 A schematic diagram of a feature map visualization provided by an embodiment of the present invention;

[0036] Fig.15 A schematic diagram of an attack method on the CISIA-SURF dataset provided by an embodiment of the present invention;

[0037] Fig.16 A schematic diagram of CISIA-SURF face segmentation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] An embodiment of the present invention provides a deep learning method for face anti-fraud, which is applied to face recognition and anti-fraud recognition.

[0040] In one embodiment provided by the present invention, the deep learning method for face anti-spoofing comprises the following steps:

[0041] Step 1: Build the MMCNN network framework, which allows multimodal data input and can fully integrate its information.

[0042] Specifically, the overall architecture of MMCNN is as follows Figure 1 As shown, ResNet-34 is used as the main training framework, and ResNet-34 is disassembled into five parts, res1-res5, as shown in Figure 2 As shown, the information of the intermediate layer is extracted to effectively fuse multimodal data.

[0043] First, the RGB image is expanded to two color space modes, HSV and YCbCr, so that the network can obtain texture information more fully. Then, ResNet-34 is used to extract features from the data of several modes layer by layer. The features after res2 (middle-level features) are extracted, and these features are connected to form a new set of features. This new set of features is concatenated and convolved, and the features after convolution are CatFrature.

[0044] At the same time, the features after res3 (high-level features) are made into a SElayer, such as Figure 4 As shown, each mode is channel-weighted, and this feature is SEFeature;

[0045] Furthermore, the SElayer structure diagram is shown in Figure 3:

[0046] For any transformation: U = F t (X)

[0047] where F t is the convolution operation, and V = [v1,v2,.....v c ] is the convolution kernel, and the output feature map is U=[U1,U2,.....U c ]

[0048] Then there is

[0049] Furthermore, the entire SElayer is divided into two parts: compression (Squeeze) and activation (Excitation).

[0050] Compression part: average the entire feature map in the channel dimension, that is, do an average pooling, the formula is as follows:

[0051]

[0052] In this way, weighted averaging along the channel is equivalent to having a global receptive field for each channel. This information is the most concentrated feature map.

[0053] Activation part: After obtaining the concentrated features, the purpose of SElayer is to obtain a weight value for each channel, so that the model can automatically learn the importance of the channel, and let the channel learn the weight through two layers of fully connected layers. The formula is as follows:

[0054] s=F e (z,W)=σ(g(z,W))=σ(W2δ(W1z))

[0055] in σ is the activation function, ReLU is generally used, and r is the scaling parameter, the purpose is to reduce the number of channels and reduce the amount of calculation.

[0056] Multiply the obtained channel weight value by the original feature map as shown below:

[0057] x′=F scale (u c ,s c )=u c *s c .

[0058] After that, SEFeature is connected and convolved to obtain a set of features called CatSEFeature. The middle-level feature CatFrature is used as a residual with the high-level feature CatSEFeature, and then the feature is sent to res4 and res5 in sequence. After res5, the feature is pooled and stretched to change the feature map to 1*1. Then, a fully connected layer is used and the result is softmax classified to obtain the scores of real faces and fake faces.

[0059] Further, such as Figure 1 As shown, the calculation results of the size of the entire feature map operation process are as follows:

[0060] Change the size of all input images to image(H*W*C)=112*112*3 and send them to the network. After res1, due to the effects of convolution and pooling, the output size becomes res1(H*W*C)=28*28*64; after res2, the output is res2(H*W*C)=28*28*64. After splicing the five modes together, the feature becomes cat(H*W*C)=28*28*320. Then, the average pooling method is used to select the connection features, so that its feature map becomes avgpool(H*W*C)=14*14*320. Use Catconv to compress the feature map channel to 128 dimensions, so that its size becomes CatFeature(H*W*C)=14*14*128. After res3, the feature map size becomes res3(H*W*C)=14*14*128. Since SElayer only adds one more weight at the channel, it does not change the feature map size. So SElayer(H*W*C)=14*14*128, we concatenate the five high-level features Highcat(14*14*128)=14*14*640, and after Catconv it becomes CatSEFeature(14*14*128)=14*14*128.

[0061] Step 2: Modal expansion. In order to improve the accuracy, the original tri-modal (or single-modal) data is expanded by two additional modalities by converting its color space.

[0062] The MMCNN network framework built in step 1 above is characterized by allowing multimodal data input and fully integrating its information. Multimodal data plays a vital role in estimating accuracy because different modalities contain different information and attributes. For example, visible light images generally contain more texture detail information, depth images contain information about facial details and camera distance, which is very useful in preventing image and video attacks, while near-infrared images provide thermal radiation information of the face.

[0063] The expansion of data modality helps improve accuracy by converting the original visible light image into HSV color space image and YCbCr image. These two modalities separate chromaticity and brightness, providing more detailed texture features for the entire model. Figure 5 As shown, the two color space channels of HSV and YCbCr can provide effective color texture description.

[0064] HSV is a color model, different from the traditional RGB, the three channels of this model are hue, saturation, and value. Figure 6 As shown, the entire HSV color space can be represented by a cone, where hue is the basic color attribute, but it is measured by angle in HSV space. Saturation refers to the brightness of the color. High saturation means that its spectral color component is large and the color is more vivid. Brightness refers to the brightness of the color. Use the opencv library to convert the original RGB space to HSV space, so that it can be used as a new mode to provide the model with clear chromaticity information and brightness information. Further, as Figure 7 As shown in the figure, the real faces and fake faces in the data set are extracted and the color space decomposition is performed.

[0065] YCbCr is a color space that includes a luminance component (Luma), a blue chrominance component (Cb), and a red chrominance component (Cr). The characteristic of this space is that the luminance signal Y is separated from the chrominance signals Cb and Cr. The human eye is more sensitive to the luminance component, so after stripping the luminance component, the other two components can be downsampled, and the human eye will not easily distinguish them, so this chrominance space can be used for image compression.

[0066] like Figure 8 As shown in Figure 2, the chi-square distance between the true and false faces of the chrominance component is much greater than that of the grayscale image and the brightness component.

[0067] Step 3: Calculate the loss function.

[0068] Specifically, in the MMCNN network framework, since the face data anti-fraud problem is a binary classification problem and the intra-class spacing is relatively small, the clustering centers of the real face and fake face features are "pushed apart" to assist the cross entropy loss, and the loss function is push loss.

[0069] Assume that the characteristics of the logical layer of the network are F = (f1, f2…f N ), where the softmax result of the final classification is P = (p1, p2…p N ), the true label is Y = (y1, y2…y N ) where y i∈(0,1), using the P vector as a mask, the features of real faces and fake faces can be screened out as shown in the following formula:

[0070] F fake =F*|Y-1|

[0071] F real =F*|Y|

[0072] Calculate the cluster center C of the two categories fake and C real

[0073]

[0074] Then use the mean square error of the two cluster centers to "push" the two samples apart.

[0075]

[0076] As with general tasks, we define the cross entropy loss function to expand the distance between classes

[0077]

[0078] Finally, we define our loss function as

[0079] L=a*L softmax +b*L push

[0080] Among them, a and b are hyperparameters, which can adjust the ratio between the two.

[0081] Design L push The purpose of is to increase the interval between the two classification surfaces to avoid confusion due to large differences within the class. In addition, this loss can assist the cross entropy in classifying the wrong classifications in each training. Fig. 9 shown.

[0082] Step 4: Quantitative analysis of different modes of real and fake images

[0083] In this step, the similarity of real and fake face images of each modality is determined by quantitatively analyzing the CISIA-SURF dataset.

[0084] Specifically, three pairs of faces are randomly selected, and LBP feature extraction is performed on their five modes (taking the 8-neighborhood of the pixel point and the radius range as 1). Fig.10 As shown; calculate the histogram of each modality LBP feature (the number of boxes is unified as N = 50), and then calculate the chi-square distance of the histogram:

[0085]

[0086] The larger the chi-square distance, the greater the difference between the real and fake faces in this mode;

[0087] Furthermore, the comparison results of two real faces are used as the baseline. Fig.11 As shown in the figure, the chi-square distance between true and false faces of different modalities is as follows Fig.12 As shown; it can be seen that the depth image has the largest difference in the histograms of the real and fake faces in the three groups of experiments. The basic chi-square distance can reach more than 1000, while the RGB image is relatively stable, at about 300.

[0088] In order to further explore the role of the two additional modes, we separated the two additional mode channels and conducted similar experiments. The results are as follows: Fig.13 As shown; from the analysis of this group of experiments, it can be seen that the differences between the brightness channel (V) and the lightness channel (Y) are relatively stable and not much different from RGB, and the S channel has better results.

[0089] In another embodiment provided by the present invention, a face recognition authenticity measurement standard is defined. Assuming that the current prediction result of the sample is p i , the true result of this sample is y i , then:

[0090] If p i =0 and y i =1 means false negative error (False Negative FN)

[0091] If p i =1 and y i =0 is a false positive error (False Positive FP)

[0092] If p i =0 and y i =0 means True Negative TN

[0093] If p i =1 and y i =1 means True Positive (True Negative TN)

[0094] Attack Presentation Classification Error Rate

[0095] APCER=FP / (TN+FP)

[0096] Normal Presentation Classification Error Rate

[0097] NPCER=FN / (TP+FN)

[0098] Average Classification Error Rate

[0099] ACER=(APCER+NPCER) / 2

[0100] False Positive Rate

[0101] FPR=FP / (FP+TN)

[0102] True Positive Rate

[0103] TPR=TP / (TP+FN)

[0104] Half Total Error Rate

[0105]

[0106] Furthermore, the ROC curve is an important evaluation indicator for classification problems. It measures the balance between the false positive rate and the true positive rate. Each point on the curve represents the value of false positive and true positive at different thresholds. Furthermore, TPR@FPR=10E-2, TPR@FPR=10E-3, and TPR@FPR=10E-4 are also used as one of the measurement criteria.

[0107] In another embodiment provided by the present invention, the performance of the MMCNN network structure is verified by experiment using the CISIA-SURF dataset.

[0108] Specifically, by inputting multimodal data, including visible light image (RGB), depth image (Depth) and near infrared image (IR) face data types; Fig.15 As shown, there are 6 printing attack methods. The attack methods of this dataset are:

[0109] Attack1: Flatten the photo and dig out the eyes;

[0110] Attack2: Bend the photo and dig out the eyes;

[0111] Attack3: Flatten the photo and remove the eyes and nose;

[0112] Attack4: Bend the photo to remove the eyes and nose;

[0113] Attack5: Flatten the photo and remove the eyes, nose and mouth;

[0114] Attack6: Bend the photo to remove the eyes, nose and mouth.

[0115] Furthermore, the dataset uses Dlib for face detection and PRNet for 3D reconstruction and face alignment, making face cropping more complete and unaffected by the background environment. The detection process is as follows: Fig.16 As shown. By using cross-experiments, we can see the effects of PushLoss and adding modalities on the entire model. By comparing the third and fifth rows, we can see that adding two color spaces as modal channels improves the accuracy of the model. By introducing PushLoss, we can not only enhance the accuracy of the model, but also help increase the stability of the model.

[0116] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A deep learning method for face anti-fraud, comprising the following steps: Build the MMCNN network framework to allow multi-modal data input and fully integrate its information; modal expansion, in order to improve the accuracy, the original three-modal data is expanded by two additional modes by converting its color space; Calculate the loss function; Quantitative analysis of different modalities of real and fake images, by quantitatively analyzing the CISIA-SURF dataset to determine the similarity of real and fake face images in each modality; Expand the RGB image to the two color space modes of HSV and YCbCr, use ResNet-34 to extract the features of the data of the two color space modes layer by layer, extract the features after res2, connect the features after res2, and then form a new set of features, and perform convolution on this new set of features; Make a SElayer for the features after res3, and make each mode channel weighted; the whole SElayer is divided into two parts: compression and activation. The compression part makes an average pooling of the entire feature map in the channel dimension, and the activation part allows the channel to learn weights through two layers of fully connected layers; The formula for the compression part is: ; The formula for the activation part is: ; in , , is the activation function, is the scaling parameter; Connect SEFeature and perform convolution to obtain a set of features called CatSEFeature, and make residuals between the middle-level feature CatFrature and the high-level feature CatSEFeature, and then send the features to res4 and res5 in sequence; after res5, perform a pooling stretch on the features to change the feature map to 1*1, then use a fully connected layer, and perform softmax classification on the results to obtain the scores of real faces and fake faces; The modal expansion converts the original visible light image into an HSV color space image and a YCbCr image, separates chromaticity and brightness, and provides more detailed texture features for the entire model; The loss function assists the cross entropy loss by enlarging the inter-class distance between cluster centers of real and fake face features; The quantitative analysis is performed by extracting LBP features from several modes of the face to be identified, calculating the histogram of the LBP features of each mode, and calculating the chi-square distance of the histogram; The chi-square distance of the calculated histogram is: The larger the chi-square distance is, the greater the difference between the real and fake faces in this mode.

Images