Face liveness detection method and system based on five sense organ component segmentation
By using a face liveness detection system based on facial feature segmentation, and leveraging heatmap extraction and multi-branch network to generate pseudo-labels, the system solves the problem of existing methods' difficulty in recognizing faces in mask and photo attacks, achieving higher recognition accuracy and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHENGDU QINGRONG TECH CO LTD
- Filing Date
- 2023-04-18
- Publication Date
- 2026-06-02
AI Technical Summary
Existing face liveness detection methods are difficult to effectively identify 2D and 3D attacks, especially mask and photo attacks. Furthermore, methods that rely on specific hardware are costly, methods using commercial RGB cameras have poor generalization capabilities, and they do not make sufficient use of local and global semantic information.
A face liveness detection system based on facial feature segmentation is adopted. Through a heatmap extraction module and a multi-branch liveness detection module, facial feature heatmaps are extracted using facial key points, and pseudo-labels and fusion features are generated through a multi-branch network for liveness detection.
It significantly reduced the average classification error rate of liveness detection, improved the accuracy of identifying attacked images, and enhanced the generalization ability of the method.
Smart Images
Figure CN116665265B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of biometric recognition technology, specifically to a face liveness detection method and system based on facial feature segmentation. Background Technology
[0002] With the rapid development of biometric technology, facial recognition has been widely used in modern society, such as mobile payment, smart attendance, access control, and financial services.
[0003] The output of a face liveness detection algorithm is a judgment on whether the input image is a real human face. Depending on the dimension of the attack, attacks can be divided into 2D attacks and 3D attacks. 2D attacks include printing and video replay attacks, while 3D attacks include attacks using masks of different materials, plaster, wax figures, etc.
[0004] In liveness detection, traditional feature-based methods extract important facial features, such as liveness and texture features, and analyze these features to determine whether the input is a live person. Deep learning-based liveness detection methods fall into two main categories: those using commercial monocular RGB cameras and those using advanced sensors. The former primarily uses RGB modal data as input, while the latter utilizes multimodal information, including depth and near-infrared sensors.
[0005] Face liveness detection methods using specific hardware may rely on structured light 3D sensors, time-of-flight (ToF) sensors, near-infrared (NIR) sensors, thermal sensors, etc. However, these sensors are still expensive, thus limiting their use to certain applications. In most scenarios, users need to authenticate on their own devices. In such cases, hardware-specific liveness detection methods are unavailable.
[0006] Face liveness detection methods using commercial RGB cameras suffer from limited input image information and highly targeted face presentation attacks, hindering generalization. Some advanced datasets now include attacks involving imposters wearing photos or masks with deliberately cropped facial features, making it difficult for methods relying solely on whole-face information to correctly identify the target. Recent methods have addressed this type of attack by segmenting the entire face into random face patches or feeding them into the network, or by dividing the image into several squares and feeding them into different branches to train different backbone networks. The drawback of this approach is that the selected facial portions are too small to capture local semantic information, or it focuses only on local facial features while ignoring global semantics. Summary of the Invention
[0007] The purpose of this invention is to provide a face liveness detection method and system based on facial feature segmentation, so as to significantly reduce the average classification error rate of liveness detection.
[0008] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:
[0009] This invention provides a face liveness detection system based on facial feature segmentation, the face liveness detection system based on facial feature segmentation includes:
[0010] A heatmap extraction module is used to extract facial feature heatmaps from the original facial image using facial key points;
[0011] A multi-branch liveness detection module, comprising a first-stage sub-module and a second-stage sub-module, wherein the first-stage sub-module is used to obtain pseudo-labels for each facial feature based on the original face image and the facial feature heatmap;
[0012] The second stage submodule is used to obtain the face liveness detection result based on the original face image, the facial feature heatmap, and the pseudo-labels of each facial feature.
[0013] Optionally, the first stage submodule includes a Mask processing unit, an activation function layer, a first classification unit, a sorting unit, and a pseudo-label generation unit arranged sequentially.
[0014] The Mask processing unit is used to perform Mask processing on the original face image and the facial feature heatmap to obtain multiple facial feature component images;
[0015] The activation function layer is used to activate each of the facial features to obtain multiple facial features;
[0016] The first classification unit is used to generate an activity score for each of the multiple facial features based on the facial features;
[0017] The sorting unit is used to sort the activity scores of each facial feature to obtain the sorting result;
[0018] The pseudo-label generation unit is used to generate pseudo-labels for each facial feature based on the sorting results and a preset threshold.
[0019] Optionally, the second stage submodule includes a BCE loss unit, a dot product unit, a fusion unit, a second classification unit, and a liveness detection result generation unit arranged sequentially.
[0020] The BCE loss unit is used to perform loss calculation on each facial feature to obtain multiple loss calculation results corresponding to each facial feature.
[0021] The dot product unit is used to perform dot product calculations on each loss calculation result and each pseudo-label of the facial features respectively, to obtain the dot product calculation result corresponding to each facial feature;
[0022] The fusion unit is used to perform a fusion operation on the dot product calculation results corresponding to all facial features to obtain a fusion result.
[0023] The second classification unit is used to generate a liveness score based on the fusion result;
[0024] The liveness detection result generation unit is used to generate liveness detection results based on the liveness score.
[0025] Alternatively, the facial features include full-face features, eye features, nose features, and mouth features;
[0026] The fusion unit includes a first convolutional layer, an addition layer, a second convolutional layer, a pruning layer, a central difference convolutional layer, a pooling layer, and an aggregation layer;
[0027] The first convolutional layer is used to convolve the eye features, nose features, and mouth features to obtain a unified feature;
[0028] The summation layer is used to add the unified feature and the whole face feature to obtain the summation result;
[0029] The second convolutional layer is used to convolve the summation result to obtain facial features with increased information weights;
[0030] The reduction layer is used to subtract the uniform feature from the whole face feature to obtain the reduction result;
[0031] The central difference convolutional layer is used to capture detailed information in the pruning result to obtain the central difference convolution result;
[0032] The pooling layer is used to perform pooling operations on the central difference convolutional structure to obtain the pooling result.
[0033] The aggregation layer is used to aggregate the pooling result and the facial features with added information weights to obtain a fusion result.
[0034] Optionally, there are two second convolutional layers. The two second convolutional layers are used to strengthen the emphasis on facial features of the whole face and increase the information weight of facial features.
[0035] There are two central difference convolutional layers, which are used to enhance the extraction of deception features.
[0036] Alternatively, the first classification unit and the second classification unit are configured to have the same structure, and the first classification unit and / or the second classification unit includes:
[0037] Six third convolutional layers arranged sequentially, and a first pooling layer located between the third and fourth third convolutional layers.
[0038] The present invention also provides a face liveness detection method based on the above-mentioned face liveness detection system based on facial feature segmentation, the face liveness detection method comprising:
[0039] S1: Extract facial feature heatmap from the original face image using facial key points;
[0040] S2: Based on the original face image and the facial feature heatmap, obtain pseudo-labels for each facial feature, where the pseudo-labels include 0 and 1;
[0041] S3: Determine if all pseudo-labels of each facial feature are 0. If yes, proceed to S4; otherwise, proceed to S5.
[0042] S4: Perform a fusion operation on the dot product calculation results corresponding to all facial features to obtain the face liveness detection results;
[0043] S5: Based on the original face image, the facial feature heatmap, and the pseudo-labels of each facial feature, obtain the face liveness detection result.
[0044] Alternatively, S2 includes:
[0045] The original face image and the facial feature heatmap are processed with a mask to obtain multiple images of facial features.
[0046] Activation processing is performed on each of the aforementioned facial features to obtain multiple facial features;
[0047] An activity score for each facial feature is generated based on the aforementioned multiple facial features;
[0048] The activity scores of each facial feature were ranked to obtain the ranking results;
[0049] Based on the sorting results and preset thresholds, pseudo-labels are generated for each facial feature.
[0050] Alternatively, S5 includes:
[0051] Loss calculations are performed on each facial feature to obtain multiple loss calculation results for each facial feature;
[0052] Dot product calculations are performed on each loss calculation result and each pseudo label of each facial feature to obtain the dot product calculation result corresponding to each facial feature.
[0053] The dot product calculation results corresponding to all facial features are fused to obtain the fused result;
[0054] A liveness score is generated based on the fusion result;
[0055] Liveness detection results are generated based on the liveness score.
[0056] Alternatively, the facial features include full-face features, eye features, nose features, and mouth features;
[0057] The loss for the whole-face feature is calculated as follows:
[0058] Loss=-(y*log(p)+(1-y)*log(1-p))
[0059] The loss L of the eye features, the nose features, and the mouth features β× The calculation is as follows:
[0060] L β× = -β·ylog(p) - (1-y)log(1-p)
[0061] Where β represents the weighting coefficient, y represents the true value, and p represents the predicted value.
[0062] The present invention has the following beneficial effects:
[0063] This invention innovatively proposes a face liveness detection method and system based on facial feature segmentation. The system includes a heatmap extraction module and a multi-branch liveness detection module, which comprises a facial feature fusion unit. This invention aims to more fully utilize the local features of facial features in the input image for liveness detection and their correlation with the overall face using the multi-branch liveness detection module and the facial feature fusion unit. The designed model was validated on the public datasets OULU-NPU and CASIA-SURF. Compared with other liveness detection models, the model designed in this invention significantly reduces the average classification error rate for liveness detection. Attached Figure Description
[0064] Figure 1 This is a schematic diagram of the face liveness detection system based on facial feature segmentation according to the present invention;
[0065] Figure 2 This is a schematic diagram of the fusion unit of the present invention;
[0066] Figure 3 This is a flowchart of the face liveness detection method based on facial feature segmentation according to the present invention. Detailed Implementation
[0067] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.
[0068] This invention provides a face liveness detection system based on facial feature segmentation, referenced Figure 1 The facial liveness detection system based on facial feature segmentation includes:
[0069] A heatmap extraction module is used to extract facial feature heatmaps from the original facial image using facial key points;
[0070] In order to extract facial features from attacks of different types and environments, this invention uses a specialized facial landmark extraction network to train and obtain facial feature heatmaps.
[0071] A multi-branch liveness detection module, comprising a first-stage sub-module and a second-stage sub-module, wherein the first-stage sub-module is used to obtain pseudo-labels for each facial feature based on the original face image and the facial feature heatmap;
[0072] Optionally, the first stage submodule includes a Mask processing unit, an activation function layer, a first classification unit, a sorting unit, and a pseudo-label generation unit arranged sequentially.
[0073] The Mask processing unit is used to perform Mask processing on the original face image and the facial feature heatmap to obtain multiple facial feature component images;
[0074] Since mask processing is a prior art technique, it will not be specifically described here.
[0075] The activation function layer is used to activate each of the facial feature images to obtain multiple facial feature characteristics;
[0076] The first classification unit is used to generate an activity score for each of the multiple facial features based on the facial features;
[0077] The feature is a multidimensional matrix with height H, width W, and number of channels C. The first classification unit changes the length, width, and height through a convolutional layer and performs pooling calculations through a pooling layer to generate an activity score.
[0078] In this invention, the first classification unit includes six third convolutional layers arranged in sequence, and a first pooling layer located between the third and fourth third convolutional layers.
[0079] The sorting unit is used to sort the activity scores of each facial feature to obtain the sorting result;
[0080] The pseudo-label generation unit is used to generate pseudo-labels for each facial feature based on the sorting results and a preset threshold.
[0081] Here, the facial features of this invention include full-face features, eye features, nose features, and mouth features. Based on this, the activity scores of the full-face features, eye features, nose features, and mouth features are ranked, with reference to... Figure 1 As shown, those skilled in the art can determine pseudo-labels for the whole face features, eye features, nose features, and mouth features based on the sorting results as needed.
[0082] As one specific embodiment, refer to Figure 1 As shown, according to the preset threshold, if the current feature (one of the whole face feature, eye feature, nose feature and mouth feature) is greater than the preset threshold, then the current feature is a retained feature and the pseudo label corresponding to the feature is 1; otherwise, it is 0.
[0083] In addition, if all values of the sorting results are less than the preset threshold, facial feature fusion is performed directly.
[0084] The second stage submodule is used to obtain the face liveness detection result based on the original face image, the facial feature heatmap, and the pseudo-labels of each facial feature.
[0085] Optionally, the second stage submodule includes a BCE loss unit, a dot product unit, a fusion unit, a second classification unit, and a liveness detection result generation unit arranged sequentially.
[0086] The BCE loss unit is used to perform loss calculation on each facial feature to obtain multiple loss calculation results corresponding to each facial feature.
[0087] The dot product unit is used to perform dot product calculations on each loss calculation result and each pseudo-label of the facial features respectively, to obtain the dot product calculation result corresponding to each facial feature;
[0088] The fusion unit is used to fuse the dot product calculation results corresponding to all facial features to obtain the fusion result; the fusion unit includes a first convolutional layer, an addition layer, a second convolutional layer, a pruning layer, a central difference convolutional layer, a pooling layer, and an aggregation layer;
[0089] The first convolutional layer is used to convolve the eye features, nose features, and mouth features to obtain a unified feature;
[0090] The summation layer is used to add the unified feature and the whole face feature to obtain the summation result;
[0091] The second convolutional layer is used to convolve the summed result to obtain facial features with increased information weight; in this invention, there are two second convolutional layers, which are used to strengthen the emphasis of the whole face on the facial feature region features and increase the information weight of the facial feature features.
[0092] The reduction layer is used to subtract the uniform feature from the whole face feature to obtain the reduction result;
[0093] The central difference convolutional layer is used to capture detailed information in the pruning result to obtain the central difference convolution result; there are two central difference convolutional layers, which are used to enhance the extraction of deception features.
[0094] Specifically, the central difference convolutional layer operates at the feature level, and its effect is to extract deceptive features at these edges and boundaries.
[0095] The pooling layer is used to perform pooling operations on the central difference convolutional structure to obtain the pooling result.
[0096] The aggregation layer is used to aggregate the pooling result and the facial features with added information weights to obtain a fusion result.
[0097] That is, the fusion unit includes two branches: an addition branch and a deletion branch, see reference. Figure 2 As shown, the summation branch adds the unified features to the whole-face features, and then passes them through two second convolutional layers. This strengthens the emphasis of the whole face on the features of the facial features and increases the information weight of the facial features. The subtraction branch subtracts the unified features of the three facial features from the whole-face features, and then aggregates them with the result of the summation branch (facial features with increased information weight) through two central difference convolutional (CDC) layers and pooling layers. This yields features mainly composed of non-facial feature regions in the whole face. These regions are mainly skin and have significant differences from the eyes, nose, and mouth in terms of texture, reflectivity, and other features. This branch is introduced to utilize the information that is mutually exclusive with the facial features, and then uses the strong deception feature extraction capability of the central difference convolution to distinguish the deception features at the edges of the facial features and at the junction of the face and the background.
[0098] The second classification unit is used to generate a liveness score based on the fusion result;
[0099] The liveness detection result generation unit is used to generate liveness detection results based on the liveness score.
[0100] Alternatively, the first classification unit and the second classification unit are configured to have the same structure, and the first classification unit and / or the second classification unit includes:
[0101] Six third convolutional layers arranged sequentially, and a first pooling layer located between the third and fourth third convolutional layers.
[0102] This invention also provides a face liveness detection method based on the above-mentioned face liveness detection system based on facial feature segmentation, referencing... Figure 3 As shown, the face liveness detection method includes:
[0103] S1: Extract facial feature heatmap from the original face image using facial key points;
[0104] S2: Based on the original face image and the facial feature heatmap, obtain pseudo-labels for each facial feature, where the pseudo-labels include 0 and 1;
[0105] S3: Determine if all pseudo-labels of each facial feature are 0. If yes, proceed to S4; otherwise, proceed to S5.
[0106] S4: Perform a fusion operation on the dot product calculation results corresponding to all facial features to obtain the face liveness detection results;
[0107] S5: Based on the original face image, the facial feature heatmap, and the pseudo-labels of each facial feature, obtain the face liveness detection result.
[0108] Alternatively, S2 includes:
[0109] The original face image and the facial feature heatmap are processed with a mask to obtain multiple images of facial features.
[0110] Activation processing is performed on each of the aforementioned facial features to obtain multiple facial features;
[0111] An activity score for each facial feature is generated based on the aforementioned multiple facial features;
[0112] The activity scores of each facial feature were ranked to obtain the ranking results;
[0113] Based on the sorting results and preset thresholds, pseudo-labels are generated for each facial feature.
[0114] Alternatively, S5 includes:
[0115] Loss calculations are performed on each facial feature to obtain multiple loss calculation results for each facial feature;
[0116] Dot product calculations are performed on each loss calculation result and each pseudo label of each facial feature to obtain the dot product calculation result corresponding to each facial feature.
[0117] The dot product calculation results corresponding to all facial features are fused to obtain the fused result;
[0118] A liveness score is generated based on the fusion result;
[0119] Liveness detection results are generated based on the liveness score.
[0120] Alternatively, the facial features include full-face features, eye features, nose features, and mouth features;
[0121] The loss for the whole-face feature is calculated as follows:
[0122] Loss=-(y*log(p)+(1-y)*log(1-p))
[0123] The loss L of the eye features, the nose features, and the mouth features β× The calculation is as follows:
[0124] L β× = -β·ylog(p) - (1-y)log(1-p)
[0125] Where β represents the weighting coefficient, y represents the true value, and p represents the predicted value.
[0126] It should be noted that the specific operation of S4 is the same as that in S5, "the dot product calculation results corresponding to all facial features are fused to obtain a fusion result; a liveness score is generated based on the fusion result; and a liveness detection result is generated based on the liveness score."
[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A face liveness detection system based on five sense components segmentation, characterized in that, The face liveness detection system based on facial feature segmentation includes: A heatmap extraction module is used to extract facial feature heatmaps from the original facial image using facial key points; A multi-branch liveness detection module, comprising a first-stage sub-module and a second-stage sub-module, wherein the first-stage sub-module is used to obtain pseudo-labels for each facial feature based on the original face image and the facial feature heatmap; The second stage submodule is used to obtain the face liveness detection result based on the original face image, the facial feature heatmap, and the pseudo-labels of each facial feature. The second stage submodule includes, in sequence, a BCE loss unit, a dot product unit, a fusion unit, a second classification unit, and a liveness detection result generation unit; The BCE loss unit is used to perform loss calculation on each facial feature to obtain multiple loss calculation results corresponding to each facial feature. The dot product unit is used to perform dot product calculations on each loss calculation result and each pseudo-label of the facial features respectively, to obtain the dot product calculation result corresponding to each facial feature; The fusion unit is used to perform a fusion operation on the dot product calculation results corresponding to all facial features to obtain a fusion result. The second classification unit is used to generate a liveness score based on the fusion result; The liveness detection result generation unit is used to generate liveness detection results based on the liveness score; Facial features include overall facial features, eye features, nose features, and mouth features; The fusion unit includes a first convolutional layer, an addition layer, a second convolutional layer, a pruning layer, a central difference convolutional layer, a pooling layer, and an aggregation layer; The first convolutional layer is used to convolve the eye features, nose features, and mouth features to obtain a unified feature; The summation layer is used to add the unified feature and the whole face feature to obtain the summation result; The second convolutional layer is used to convolve the summation result to obtain facial features with increased information weights; The reduction layer is used to subtract the uniform feature from the whole face feature to obtain the reduction result; The central difference convolutional layer is used to capture detailed information in the pruning result to obtain the central difference convolution result; The pooling layer is used to perform a pooling operation on the central difference convolution result to obtain the pooling result. The aggregation layer is used to aggregate the pooling result and the facial features with added information weights to obtain a fusion result; There are two second convolutional layers. These two second convolutional layers are used to strengthen the emphasis on facial features of the whole face and increase the information weight of facial features. There are two central difference convolutional layers, which are used to enhance the extraction of deception features.
2. The five-organ part-based segmentation based face liveness detection system of claim 1, wherein, The first stage submodule includes a Mask processing unit, an activation function layer, a first classification unit, a sorting unit, and a pseudo-label generation unit, which are set sequentially. The Mask processing unit is used to perform Mask processing on the original face image and the facial feature heatmap to obtain multiple facial feature component images; The activation function layer is used to activate each of the facial features to obtain multiple facial features; The first classification unit is used to generate an activity score for each of the multiple facial features based on the facial features; The sorting unit is used to sort the activity scores of each facial feature to obtain the sorting result; The pseudo-label generation unit is used to generate pseudo-labels for each facial feature based on the sorting results and a preset threshold.
3. The face liveness detection system based on five sense components segmentation according to claim 2, characterized in that, The first classification unit and the second classification unit are configured to have the same structure, and the first classification unit and / or the second classification unit includes: Six third convolutional layers arranged sequentially, and a first pooling layer located between the third and fourth third convolutional layers.
4. A face living body detection method based on the face living body detection system based on the five sense component segmentation according to any one of claims 1-3, characterized in that, The face liveness detection method includes: S1: Extract facial feature heatmap from the original face image using facial key points; S2: Based on the original face image and the facial feature heatmap, obtain pseudo-labels for each facial feature, where the pseudo-labels include 0 and 1; S3: Based on the original face image, the facial feature heatmap, and the pseudo-labels of each facial feature, obtain the face liveness detection result.
5. The face liveness detection method of claim 4, wherein, S2 includes: The original face image and the facial feature heatmap are processed with a mask to obtain multiple images of facial features. Activation processing is performed on each of the aforementioned facial features to obtain multiple facial features; An activity score for each facial feature is generated based on the aforementioned multiple facial features; The activity scores of each facial feature were ranked to obtain the ranking results; Based on the sorting results and preset thresholds, pseudo-labels are generated for each facial feature.
6. The face liveness detection method of claim 4, wherein, S3 includes: Loss calculations are performed on each facial feature to obtain multiple loss calculation results for each facial feature. Dot product calculations are performed on each loss calculation result and each pseudo label of each facial feature to obtain the dot product calculation result corresponding to each facial feature; The dot product calculation results corresponding to all facial features are fused to obtain the fused result; A liveness score is generated based on the fusion result; Liveness detection results are generated based on the liveness score.
7. The face liveness detection method of claim 6, wherein, Facial features include overall facial features, eye features, nose features, and mouth features; The loss of the whole face features The calculation is as follows: Loss of the eye features, the nose features, and the mouth features The calculation is as follows: in, Indicates the weighting coefficient. Represents the actual value. This represents the predicted value.