Face living body detection method and system based on multi-scale anti-color method

Abstract

The application discloses a face living body detection method and system based on a multi-scale anti-color method, mainly aiming at the problems of weak cross-dataset generality, large method parameter quantity and method effectiveness being easily affected by illumination conditions of existing face living body detection methods, and specifically comprising the following steps: frame extraction and preprocessing are performed on a video to obtain a target image, and a training set and a test set are divided; a double-flow convolutional neural network is built based on a multi-scale anti-color method, a parallel CBAM attention module and an improved MobileNetV3 network as a backbone network; a binary cross-entropy function is used as a loss function to train the network model on the training set; and the trained network model is used for face living body detection on the test set. The application has the beneficial effects that a small amount of parameters can greatly improve the model performance, effectively solve the problem of sensitivity to illumination in face living body detection, and has good generalization ability and wide applicability.

Description

Technical Field

[0001] This invention relates to the field of face liveness detection technology, specifically to a face liveness detection method and system based on a multi-scale inverse color method. Background Technology

[0002] Currently, electronic device systems typically use various biometrics such as fingerprints, iris scans, and facial recognition for personal identification, with facial recognition being the most common method. The widespread application of facial recognition technology in daily life has also provided opportunities for criminals to steal the personal information of legitimate users. Criminals can obtain personal photos posted by users through social networks and then display them to facial capture devices, thereby deceiving the facial recognition system through facial authentication. Therefore, facial liveness detection technology has become a hot topic of research in academia and society.

[0003] Traditional methods often use LBP, HOG, and DOG to extract handcrafted features for face liveness detection, but they generally suffer from poor cross-dataset generalization. Unlike the shallow representations of handcrafted features extracted by traditional methods, deep learning-based methods can extract feature tables with high-level semantics. Distinguishing between real and fake faces using a single difference cue may cause these methods to encounter bottlenecks in accuracy and generalization. Furthermore, the effectiveness of existing methods is affected by lighting conditions. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the prior art, the present invention provides a face liveness detection method based on a multi-scale inversion color method. This method can solve the problems of low face detection accuracy, poor versatility and great influence of lighting in the prior art. The present invention also provides a face liveness detection system based on a multi-scale inversion color method.

[0005] Technical solution: The face liveness detection method based on multi-scale inversion color method of the present invention includes:

[0006] (1) Read the video, preprocess the video to obtain a set of face images, and divide the set of face images into a training set and a test set;

[0007] (2) Construct a network model based on the multi-scale inverse color method and attention mechanism, with the improved MobileNetV3 network as the backbone network;

[0008] (3) Train the network model on the training set;

[0009] (4) Use the trained network model to perform face liveness detection on the test set.

[0010] Furthermore, including:

[0011] The video preprocessing in step (1) includes the following steps:

[0012] (11) Read each frame of the video, use the dlib library to detect facial landmarks, and use affine transformation to align the detected faces to obtain the face image;

[0013] (12) Data augmentation is performed on the obtained face image. The data augmentation methods include random rotation, random translation, random brightness adjustment, random horizontal flipping, and random scaling.

[0014] Furthermore, including:

[0015] In step (2), the overall structure of the network model based on the multi-scale inversion method and attention mechanism is a two-stream convolutional neural network. One branch converts the input face image to be tested into a grayscale image, and then obtains the MSCI image through the multi-scale inversion method. The other branch directly inputs the face image to be tested. The MSCI image and the face image to be tested are respectively input into the improved MobileNetV3 network to extract the reflection features and color features of the face. The features of the two streams are fused and passed to the fully connected network to determine whether the input image is a real face image.

[0016] Furthermore, including:

[0017] In step (3), the binary cross-entropy function is used as the loss function to train the network model.

[0018] Furthermore, including:

[0019] The multi-scale color inversion method includes:

[0020] (21) The input image S(x,y) is filtered and normalized by a Gaussian function, and then subtracted from the identity matrix I to obtain the illumination image L(x,y), i.e.

[0021] (22) Use logpS(x,y)[subtract log[L(x,y)] to obtain r(x,y). Each r(x,y) is the output of a single-scale inversion color method. The specific formula is as follows:

[0022]

[0023] Where (x,y) represents the pixel coordinates of the input image; G(x,y) is the Gaussian filter function, and different values ​​of the standard deviation in the Gaussian function represent different scales of Gaussian filtering. i (x,y) represents the Gaussian filter function in the i-th single-scale color inversion method; r i(x,y) represents the output of the i-th single-scale inversion color method. The symbol for convolution;

[0024] (23) The output of the multi-scale inversion method is obtained by weighted fusion of the outputs of multiple single-scale inversion methods. The specific formula is as follows:

[0025]

[0026] To ensure that the single-scale inversion method has the advantages of low, medium, and high scales simultaneously, the number of scales is set to K=3, and That is, a multi-scale inversion method is obtained by weighted fusion of the three single-scale inversion methods.

[0027] Furthermore, including:

[0028] The improved MobileNetV3 network specifically includes: changing the stride of the first layer of the MobileNetV3 network from 2 to 1, meaning that the feature scale remains unchanged during the convolution operation in this layer; adding a parallel CBAM attention module after the convolution of the first layer of the MobileNetV3 network; merging the bottleneck layers of the second and third layers of the MobileNetV3 network in conjunction with the modification of the stride of the first layer of the MobileNetV3 network; deleting the last two layers of the MobileNetV3 network; and changing the number of output channels of the last layer of the MobileNetV3 network to 2.

[0029] Furthermore, including:

[0030] The parallel CBAM attention module changes the connection method of the CBAM attention module from serial to parallel. That is, for the input intermediate feature map, attention weights are inferred through independent channel attention mechanism and spatial attention mechanism respectively. Then, they are multiplied with the original intermediate feature map to perform adaptive adjustment of the features. Finally, the two feature maps after feature adaptive adjustment are fused with the original feature map to make the network pay more attention to the region of interest.

[0031] On the other hand, the present invention also provides a face liveness detection system based on a multi-scale inversion color method, comprising:

[0032] The preprocessing module is used to read the video, preprocess the video to obtain a set of face images, and divide the set of face images into a training set and a test set.

[0033] The model building module is used to build a network model based on a multi-scale inverse color method and an attention mechanism. The network model uses an improved MobileNetV3 network as its backbone network.

[0034] The training module is used to train the network model on the training set;

[0035] The testing module is used to perform face liveness detection on the test set using the trained network model.

[0036] Furthermore, including:

[0037] In the model construction module, the overall structure of the network model based on the multi-scale inversion method and attention mechanism is a two-stream convolutional neural network. One branch first converts the input face image to be tested into a grayscale image, and then obtains the MSCI image through the multi-scale inversion method. The other branch directly inputs the face image to be tested. The MSCI image and the face image to be tested are respectively input into the improved MobileNetV3 network to extract the reflection features and color features of the face. The features of these two streams are fused and passed to the fully connected network to determine whether the input image is a real face image.

[0038] Based on the above, the present invention also provides a computer storage medium storing a computer program thereon, which, when executed by a computer processor, implements the method described above.

[0039] Beneficial effects: (1) In order to make the method more widely applicable, each frame of the video image is read and face aligned and data augmented; (2) In order to enable the network to extract reflective features with better discrimination to solve the problem of light sensitivity, this invention proposes a multi-scale inversion color method and embeds it into a branch of a two-stream convolutional neural network; (3) In view of the problem of large number of parameters in two-stream networks, the lightweight network MobileNetV3 is selected and improved to improve the detection performance of the algorithm while reducing the number of parameters; (4) In order to help the network pay more attention to the pixel regions in the image that play a key role in the face liveness detection task, a parallel CBAM attention module is introduced, and the connection method of spatial attention and channel attention is improved from serial to parallel to improve the detection performance of the algorithm. Attached Figure Description

[0040] Figure 1 This is a flowchart of the network model based on the multi-scale inverse color method and attention mechanism described in the embodiments of the present invention;

[0041] Figure 2 This is a flowchart of the multi-scale color inversion method described in an embodiment of the present invention;

[0042] Figure 3 This is a structural diagram of the parallel CBAM described in an embodiment of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] First, this invention provides a face liveness detection method based on a two-stream convolutional neural network, comprising the following steps:

[0045] Step 1: Read the video, preprocess the video to obtain the target image, and divide it into training set and test set;

[0046] Step 2: Build a network model based on the multi-scale inversion method and attention mechanism, with MobileNetV3 as the backbone network;

[0047] Step 3: Train the network model on the training set;

[0048] Step 4: Use the trained network model to perform face liveness detection on the test set.

[0049] The specific content and methods of data preprocessing are as follows:

[0050] Each frame of the video is read, facial landmark detection is performed using the dlib library, and affine transformation is used to align the detected faces to obtain the face image;

[0051] To broaden the application of the method, data augmentation is performed on the obtained face images. Data augmentation methods include: random rotation, random translation, random brightness adjustment, random horizontal flipping, and random scaling.

[0052] like Figure 1 As shown, the structure of the network model described in step two is as follows:

[0053] The overall network structure is a two-stream convolutional neural network. One branch of the two-stream network first converts the input RGB image to grayscale and then uses a multi-scale color reversion (MSCI) method to obtain an MSCI image. The other branch directly inputs the RGB image. The MSCI image and RGB image are then fed into an improved MobileNetV3 network to extract the reflection and color features of the face, respectively. The features from these two streams are fused and passed to a fully connected network to determine whether the input image is a real face image. The network model is trained using a binary cross-entropy function as the loss function.

[0054] like Figure 2 As shown, the multi-scale inversion method specifically includes:

[0055] (1) The input source image S(x,y) is filtered and normalized by a Gaussian function, and then subtracted from the identity matrix I to obtain the illumination image L(x,y), i.e. Next, we subtract log[L(x,y)] from log[S(x,y)] to obtain r(x,y). Each r(x,y) is the output of a single-scale inversion color method, and the specific formula is as follows:

[0056]

[0057] Where G(x,y) is the Gaussian filter function, and r(x,y) is the output. This represents the convolution symbol. Different values ​​of the standard deviation in the Gaussian function represent Gaussian filtering at different scales.

[0058] (2) The output of the multi-scale inversion method is obtained by weighted fusion of the outputs of multiple single-scale inversion methods. The specific formula is as follows:

[0059]

[0060] To ensure that the single-scale inversion method has the advantages of low, medium and high scales, the number of scales is set to K=3, and w1=w2=w3=1 / 3.

[0061] The specific improvements to the MobilenetV3 network used in the network model are as follows:

[0062] (1) In the first layer, the stride is changed from 2 to 1, which means that the feature scale remains unchanged during the convolution operation in this layer;

[0063] (2) After the first convolutional layer, a parallel CBAM attention module (PCBAM) is added to enable the network to pay more attention to the region of interest;

[0064] (3) Combining the modification of the step size in the first layer, the bottleneck layers (bottleneck, bneck) in the original MobileNetV3 network structure are redundant. The second and third layers are merged to reduce resource consumption.

[0065] (4) To improve computational efficiency, the last two layers of the original network were removed;

[0066] (5) Considering that face liveness detection is a binary classification problem, the number of output channels in the last layer is changed to 2.

[0067] The specific parameters of the improved MobileNetV3 network are shown in Table 1 below:

[0068] Table 1

[0069]

[0070] In Table 1, conv2d represents two-dimensional convolution operation, PCBAM is the proposed attention module, bneck refers to bottleneck operation, SE indicates whether there is a Squeeze-And-Excite structure in the block (√ indicates that there is a Squeeze-And-Excite structure, otherwise it indicates that there is no Squeeze-And-Excite structure), NL indicates the type of non-linear activation function used (where HS refers to hard-swish and RE refers to ReLU), and GAP represents global average pooling.

[0071] like Figure 3 As shown, the specific structure of the PCBAM attention module introduced in the improved MobileNetV3 network is as follows:

[0072] The connection method of the traditional Convolutional Block Attention Module (CBAM) is changed from serial to parallel to obtain Parallelized Convolutional Block Attention Module (PCBAM). That is, for the input intermediate feature map, attention weights are inferred through independent channel attention mechanism and spatial attention mechanism respectively. Then, they are multiplied with the original intermediate feature map to adaptively adjust the features. Finally, the two feature maps with adaptive feature adjustment are fused with the original feature map to make the network pay more attention to the region of interest.

[0073] The following examples illustrate the specific implementation of this invention using classification instances on the benchmark datasets for face liveness detection, REPLAY ATTACK and CASIA FASD:

[0074] (1) Image preprocessing: Read each frame of the video, use the dlib library to detect facial key points, use affine transformation to align the detected faces, obtain the face image and perform data augmentation;

[0075] (2) Build a network model and use the binary cross-entropy function as the loss function to train the model on the training set;

[0076] (3) Use the trained network model to perform face liveness detection on the test set.

[0077] (4) The method of the present invention is compared with the existing methods in terms of cross-dataset generality. The results are shown in Table 3.

[0078] The method of this invention involves two cross-dataset experiments: one is trained on the CASIA FASD dataset and tested on the REPLAY ATTACK dataset; the other is trained on the REPLAY ATTACK dataset and tested on the CASIA FASD dataset.

[0079] The evaluation metric for cross-dataset experiments was the Half Total Error Rate (HTER). Face liveness detection is a binary classification problem, with four basic metrics: True Positive (TP), which refers to correctly identified positive examples (actually positive); False Positive (FP), which refers to incorrectly identified positive examples (actually negative); True Negative (TN), which refers to correctly identified negative examples (actually negative); and False Negative (FN), which refers to incorrectly identified negative examples (actually positive). The confusion matrix of the classification results is shown in Table 2.

[0080] Table 2

[0081]

[0082]

[0083] The purpose of face liveness detection is to classify captured face images. Therefore, there are two types of classification errors: one is classifying a real face as a fraudulent face, called the False Rejection Rate (FRR); the other is classifying a fraudulent face as a real face, called the False Acceptance Rate (FAR). The Half Total Error Rate (HTER) is actually the average of FAR and FRR. The formulas for FRR, FAR, and HTER are as follows:

[0084]

[0085]

[0086]

[0087] By comparing with other methods, it can be verified that the method proposed in this invention has significant advantages over 15 other methods:

[0088] Table 3

[0089]

[0090]

[0091] On the other hand, the present invention also provides a face liveness detection system based on a multi-scale inversion color method, comprising:

[0092] The preprocessing module is used to read the video, preprocess the video to obtain a set of face images, and divide the set of face images into a training set and a test set.

[0093] The model building module is used to build a network model based on a multi-scale inverse color method and an attention mechanism. The network model uses an improved MobileNetV3 network as its backbone network.

[0094] The training module is used to train the network model on the training set;

[0095] The testing module is used to perform face liveness detection on the test set using the trained network model.

[0096] Other technical details of the detection system are similar to those of the detection method, and will not be repeated here.

[0097] Based on the above embodiments, in this embodiment of the invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the face liveness detection method based on a dual-stream convolutional neural network in any of the above method embodiments.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0103] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A face liveness detection method based on a multi-scale inverse color method, characterized in that, include: (1) Read the video, preprocess the video to obtain a set of face images, and divide the set of face images into a training set and a test set; (2) Construct a network model based on the multi-scale inversion method and attention mechanism. Its overall structure is a two-stream convolutional neural network. One branch converts the input face image to be tested into a grayscale image and then obtains the MSCI image through the multi-scale inversion method. The other branch directly inputs the face image to be tested. The improved MobileNetV3 network is used as the backbone network. The MSCI image and the face image to be tested are respectively input into the improved MobileNetV3 network to extract the reflection features and color features of the face. The features from these two streams are fused and passed to a fully connected network to determine whether the input image is a real human face image; The multi-scale color inversion method includes: (21) The input image S(x,y) is filtered and normalized by a Gaussian function, and then subtracted from the identity matrix I to obtain the illumination image L(x,y), i.e. (22) Subtract log[L(x,y)] from log[S(x,y)] to obtain r(x,y). Each r(x,y) is the output of a single-scale inversion color method. The specific formula is as follows: Where (x,y) represents the pixel coordinates of the input image; G(x,y) is the Gaussian filter function, and different values ​​of the standard deviation in the Gaussian function represent different scales of Gaussian filtering. i (x,y) represents the Gaussian filter function in the i-th single-scale color inversion method; r i (x,y) represents the output of the i-th single-scale inversion color method. The symbol for convolution; (23) The output of the multi-scale inversion method is obtained by weighted fusion of the outputs of multiple single-scale inversion methods. The specific formula is as follows: To ensure that the single-scale inversion method has the advantages of low, medium, and high scales simultaneously, the number of scales is set to K=3, and That is, a multi-scale inversion method is obtained by weighted fusion of the three single-scale inversion methods; (3) Train the network model on the training set; (4) Use the trained network model to perform face liveness detection on the test set.

2. The face liveness detection method based on multi-scale inverse color method according to claim 1, characterized in that, Step (1) of video preprocessing includes the following steps: (11) Read each frame of the video, use the dlib library to detect facial landmarks, and use affine transformation to align the detected faces to obtain the face image; (12) Data augmentation is performed on the obtained face image. The data augmentation methods include random rotation, random translation, random brightness adjustment, random horizontal flipping, and random scaling.

3. The face liveness detection method based on multi-scale inverse color method according to claim 1, characterized in that, In step (3), the binary cross-entropy function is used as the loss function to train the network model.

4. The face liveness detection method based on multi-scale inverse color method according to claim 1, characterized in that, The improved MobileNetV3 network specifically includes: changing the stride of the first layer of the MobileNetV3 network from 2 to 1, meaning that the feature scale remains unchanged during the convolution operation in this layer; adding a parallel CBAM attention module after the convolution of the first layer of the MobileNetV3 network; merging the bottleneck layers of the second and third layers of the MobileNetV3 network in conjunction with the modification of the stride of the first layer of the MobileNetV3 network; deleting the last two layers of the MobileNetV3 network; and changing the number of output channels of the last layer of the MobileNetV3 network to 2.

5. The face liveness detection method based on multi-scale inversion color method according to claim 4, characterized in that, The parallel CBAM attention module changes the connection method of the CBAM attention module from serial to parallel. That is, for the input intermediate feature map, attention weights are inferred through independent channel attention mechanism and spatial attention mechanism respectively. Then, they are multiplied with the original intermediate feature map to perform adaptive adjustment of the features. Finally, the two feature maps after feature adaptive adjustment are fused with the original feature map to make the network pay more attention to the region of interest.

6. A face liveness detection system based on the method of any one of claims 1-5, characterized in that, include: The preprocessing module is used to read the video, preprocess the video to obtain a set of face images, and divide the set of face images into a training set and a test set. The model building module is used to build a network model based on a multi-scale inverse color method and an attention mechanism. The network model uses an improved MobileNetV3 network as its backbone network. The training module is used to train the network model on the training set; The testing module is used to perform face liveness detection on the test set using the trained network model.

7. The face liveness detection system according to claim 6, characterized in that, In the model building module, the overall structure of the network model based on the multi-scale inversion method and attention mechanism is a two-stream convolutional neural network. One branch first converts the input face image to be tested into a grayscale image, and then obtains the MSCI image through the multi-scale inversion method. The other branch directly inputs the face image to be tested. The MSCI image and the face image to be tested are respectively input into the improved MobileNetV3 network to extract the reflection features and color features of the face. The features from these two streams are fused and passed to a fully connected network to determine whether the input image is a real human face image.

8. A computer storage medium storing a computer program thereon, characterized in that: When the computer program is executed by a computer processor, it implements the method described in any one of claims 1 to 5.

Images