A video-based liveness detection method
The liveness detection network constructed through IR binocular infrared equipment and deep learning methods solves the problems of complexity, high cost and poor real-time performance of liveness detection in existing technologies, achieves high-accuracy liveness detection, simplifies the operation process and reduces costs.
Patent Information
- Application Number
- CN202311265731.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-28
- Publication Date
- 2025-09-16
- Estimated Expiration
- 2043-09-28
AI Technical Summary
In the existing technology, video-based liveness detection methods have problems such as complex implementation process, high cost, poor real-time performance and low accuracy, making it difficult to effectively distinguish real faces from forged faces.
An IR binocular infrared device is used to simultaneously capture color images and IR images. A deep learning method is used to construct a liveness detection convolutional neural network and a dual-input classification neural network. By annotating and training image data, non-motion liveness detection is achieved, simplifying operations and improving accuracy and real-time performance.
It achieves high-accuracy liveness detection, simplifies the operation process, eliminates the need for complex equipment, reduces costs, improves real-time performance, and can effectively distinguish between real faces and forged faces.
Smart Images
Figure CN117253294B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of intelligent monitoring and relates to a living body detection method, in particular to a living body detection method based on video. Background Art
[0002] With the development of science and technology, facial recognition technology has become commonplace in many fields, greatly facilitating our work and lives. For example, in finance, travel, security, and other fields, facial recognition is required to capture a person's face for identity authentication.
[0003] However, some unscrupulous traffickers attempt to use forged faces (such as color prints, photos, photos in electronic devices, etc.) to pass facial recognition authentication and conduct some illegal activities.
[0004] While technology facilitates our lives and work, it must also shoulder the responsibility of protecting our safety. By extracting biometric features from a person's face, it can further determine whether it is a genuine or forged face, thus safeguarding facial recognition.
[0005] After searching, the following two documents in the prior art were found:
[0006] Cai Huimin, author of "A Review of Face Anti-Spoofing Detection Based on Machine Learning and Deep Learning," describes the current state of liveness detection. Facial liveness detection methods include those based on traditional texture information, motion information, 3D structure, and deep learning. Traditional texture-based applications require high-quality capture equipment and are costly; those based on motion information suffer from poor real-time performance; and those based on 3D structure require different viewpoints of the target, making them difficult to implement in practice.
[0007] 2. Zhang Fan, author of "Research Progress in Face Recognition Anti-Fraud," introduces currently commonly used deep learning methods, namely binary supervision methods, depth map methods, rPPG methods, spatiotemporal information fusion methods, multimodal methods, and deception trace separation methods. Among them, the depth map method and the multimodal method require depth information, which places high demands on the equipment, requiring equipment that can collect depth maps; the rPPG method is sensitive to noise; the spatiotemporal information fusion method is not advantageous in real-time performance; the binary supervision method is a general two-category classification with low accuracy; and the deception trace separation method improves the generalization ability of the model. The biggest problem with single-modal datasets is the accuracy problem, and the biggest problem with multimodal datasets is the difficulty of data preparation and the high requirements for equipment. For example, using a depth camera to obtain depth information is costly.
[0008] Therefore, how to propose a video-based liveness detection method that is simple to implement, highly accurate, real-time, and low-cost is a technical challenge that those skilled in the art urgently need to solve. Summary of the Invention
[0009] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose a video-based liveness detection method. It uses binocular infrared to perform non-motion liveness detection, that is, the target does not need to perform special actions such as blinking and turning the head. Liveness detection is performed using a deep learning method, and the operation is simple, the accuracy is high, and the real-time performance is high.
[0010] The present invention solves the practical problem by adopting the following technical solutions:
[0011] A method for detecting living bodies based on video, comprising the following steps:
[0012] Step 1: Use IR binocular infrared equipment to simultaneously collect image data of living and non-living objects in pairs under color images and IR images;
[0013] Step 2: label the image data in step 1;
[0014] Step 3: Construct a convolutional neural network for liveness detection in IR images, a convolutional neural network for liveness detection in color images, and a liveness dual-input classification neural network.
[0015] Step 4: Perform model training on the liveness detection convolutional neural network and liveness dual-input classification neural network constructed in step 3;
[0016] Step 5: Use the model trained in step 4 to output the liveness detection results.
[0017] Moreover, the specific method of step 1 is:
[0018] Collect image data of living and non-living objects under color images and IR images respectively;
[0019] Live data refers to real face data, while non-live data refers to fake face data;
[0020] The color image and IR image are collected simultaneously and in pairs at the same time.
[0021] Moreover, the specific method of step 2 is:
[0022] The face target's circumscribed rectangle is used as the boundary for annotation. The annotation range is the left and right boundaries excluding the ears, the lower boundary is the chin position, and the upper boundary is the forehead position excluding the hair. According to this rule, non-living objects and living objects in color images and IR images are annotated separately.
[0023] Among them, non-living targets are marked as FP, and living targets are marked as TP for distinction.
[0024] Moreover, the specific steps of step 3 include:
[0025] (1) Based on the YOLOV3 network, a convolutional neural network NET1 for liveness detection in IR images is constructed: the TP and FP target categories and position information in the IR images are used to train the neural network NET1;
[0026] (2) Based on the YOLOV3 network, a convolutional neural network NET2 for liveness detection in color images is constructed: the TP and FP target categories and position information in the color image are used to train the neural network NET2;
[0027] Among them, the liveness detection convolutional neural network NET1 can provide the target's location information, label information, and target confidence for an IR image in subsequent detection;
[0028] The liveness detection convolutional neural network NET2 can provide the target's location information, label information, and target confidence for a color image in subsequent detections;
[0029] (3) Based on the position information output from step (1) and step (2), the corresponding IR image and the target face image in the color image are deducted in pairs according to the position information, and used as the input information of the living double-input classification neural network NET3 to construct the living double-input classification neural network NET3:
[0030] Moreover, the input of the living dual-input classification neural network NET3 in step (3) of step 3 is dual-channel, channel 1 is the target face image corresponding to the IR image; channel 2 is the target face image corresponding to the color image.
[0031] Moreover, the specific method of step 4 is:
[0032] Paired training images are fed to the living body dual-input classification neural network NET3. The paired training images refer to the IR image and color image of the same target at the same time.
[0033] Moreover, the specific steps of step 5 include:
[0034] (1) The IR image obtained by the binocular liveness detection device is fed into the liveness detection convolutional neural network NET1, which outputs the position of the face target to be confirmed and the confidence information in the image;
[0035] (2) The color image obtained by the binocular liveness detection device is fed into the liveness detection convolutional neural network NET2, which outputs the position of the face target to be confirmed in the image and the confidence information;
[0036] (3) matching the target information under the IR image and the target information under the color image obtained in steps (1) and (2);
[0037] (4) If the category information output in the IR image and the category information output in the color image of the same target are consistent, and both score thresholds are greater than the set threshold, the category is considered credible and the result is output directly;
[0038] If the category information output in the IR image is inconsistent with the category information output in the color image, or the category information is consistent but at least one score threshold is lower than the limit, further confirmation is required and jump to step (5).
[0039] (5) The target image under the corresponding IR image and the target image under the color image obtained in step (3) are simultaneously sent to the living body dual-input classification neural network NET3. The network will integrate the information under the two images, complement each other with the required feature information, and make a comprehensive judgment on whether it is a living body classification result; if the score threshold of the output category information is greater than the set threshold, the result is output, and the category is considered credible, otherwise uncertain information is output.
[0040] Advantages and beneficial effects of the present invention:
[0041] 1. The present invention proposes a video-based liveness detection method with high accuracy and no requirement for complex 3D acquisition equipment. It is easy to operate and has good real-time performance, without the need for the target to blink or turn their head. Data preparation is simple, without the need for complex calibration calculations. The present invention uses liveness detection convolutional neural networks NET1 and NET2 for liveness confirmation, and uses a liveness dual-input classification neural network NET3 to fully integrate IR image information and color image information for judgment, thereby improving accuracy.
[0042] 2. The present invention extracts biometric features from the face and fully integrates IR image information and color image information in the classification neural network NET3 to further determine whether it is a real face or a forged face, thereby providing escort for face recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a processing flow chart of the present invention. DETAILED DESCRIPTION
[0044] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:
[0045] A video-based liveness detection method, such as Figure 1 As shown, the following steps are included:
[0046] Step 1: Use IR binocular infrared equipment to simultaneously collect image data of living and non-living objects in pairs under color images and IR images;
[0047] The specific method of step 1 is:
[0048] In this embodiment, in step 1, image data of living and non-living objects are collected under color images, IR images (i.e., infrared images, IR is the abbreviation of infrared radiation), respectively;
[0049] Among them, IR binocular infrared devices support both color images and IR images; live data refers to real facial data, and non-live data refers to fake facial data (such as color prints, photos, photos in electronic devices, 3D head models, etc.); color images and IR images are collected simultaneously and in pairs at the same time.
[0050] Step 2: label the image data in step 1;
[0051] The specific method of step 2 is:
[0052] The face target's circumscribed rectangle is used as the boundary for annotation. The annotation range is the left and right boundaries excluding the ears, the lower boundary is the chin position, and the upper boundary is the forehead position excluding the hair. According to this rule, non-living objects and living objects in color images and IR images are annotated separately.
[0053] Among them, non-living targets are labeled FP (False Person) and living targets are labeled TP (True Person) for distinction.
[0054] Step 3: Construct a convolutional neural network for liveness detection in IR images, a convolutional neural network for liveness detection in color images, and a liveness dual-input classification neural network.
[0055] The specific method of step 3 is:
[0056] Construct a convolutional neural network for liveness detection in IR images (NET1), a convolutional neural network for liveness detection in color images (NET2), and a liveness dual-input classification neural network (NET3);
[0057] The specific steps include:
[0058] (1) Based on the YOLOV3 network, a convolutional neural network NET1 for liveness detection in IR images is constructed: the TP and FP target categories and position information in the IR images are used to train the neural network NET1;
[0059] (2) Based on the YOLOV3 network, a convolutional neural network NET2 for liveness detection in color images is constructed: the TP and FP target categories and position information in the color image are used to train the neural network NET2;
[0060] In this embodiment, the liveness detection convolutional neural network NET1 can provide the target's location information, label information, and target confidence for an IR image in subsequent detections;
[0061] The liveness detection convolutional neural network NET2 can provide the target's location information, label information, and target confidence for a color image in subsequent detections;
[0062] (3) Based on the position information output from step (1) and step (2), the corresponding IR image and the target face image in the color image are deducted in pairs according to the position information, and used as the input information of the living double-input classification neural network NET3 to construct the living double-input classification neural network NET3:
[0063] The input of the living dual-input classification neural network NET3 in step (3) of step 3 is dual channels, channel 1 is the target face image corresponding to the IR image; channel 2 is the target face image corresponding to the color image.
[0064] In this embodiment, in order to more accurately eliminate the mixed fake facial information, we further construct a living dual-input classification neural network NET3.
[0065] Step 4: Perform model training on the liveness detection convolutional neural network and liveness dual-input classification neural network constructed in step 3;
[0066] The specific method of step 4 is:
[0067] Paired training images are fed to the living body dual-input classification neural network NET3. The paired training images refer to the IR image and color image of the same target at the same time.
[0068] In this embodiment, it is important to note that since the IR lens and color lens of the binocular liveness detection device are basically installed in the same position, with the same height and small left-right deviation, when we send training images to NET, we must send them in pairs. This pair refers to the IR image and color image of the same target at the same time.
[0069] Step 5: Use the model trained in step 4 to output the liveness detection results;
[0070] The specific steps of step 5 include:
[0071] (1) The IR image obtained by the binocular liveness detection device is fed into the liveness detection convolutional neural network NET1, which outputs the position of the face target to be confirmed and the confidence information in the image;
[0072] (2) The color image obtained by the binocular liveness detection device is fed into the liveness detection convolutional neural network NET2, which outputs the position of the face target to be confirmed in the image and the confidence information;
[0073] (3) matching the target information under the IR image and the target information under the color image obtained in steps (1) and (2);
[0074] In this embodiment, since the installation differences of the device lens positions are very small, the same target has the same position in the IR image and the color image, which we confirm using the IOU (intersection over union) strategy.
[0075] (4) If the category information output in the IR image and the category information output in the color image of the same target are consistent, and both score thresholds are greater than the set threshold, the category is considered credible and the result is output directly;
[0076] In this embodiment, the threshold is set to 0.8. If the classification is considered credible, the classification result is directly output. If it is a living TP, further face recognition or other operations are performed. If it is not a living TP, no operation is performed or a prompt is given.
[0077] If the category information output in the IR image is inconsistent with the category information output in the color image, or the category information is consistent but at least one score threshold is lower than the limit, further confirmation is required and jump to step (5).
[0078] The target image under the corresponding IR image and the target image under the color image obtained in step (3) are simultaneously sent to the living body dual-input classification neural network NET3. The network will integrate the information under the two images, complement each other with the required feature information, and make a comprehensive judgment on whether it is a living body classification result; if the score threshold of the output category information is greater than the set threshold, the result is output, and the category is considered credible, otherwise uncertain information is output.
[0079] In this embodiment, the threshold is set to 0.9, and the category is considered to be credible, otherwise uncertain information is output.
[0080] The present invention will be further described below through specific examples:
[0081] A method for detecting living bodies based on video, comprising the following steps:
[0082] A. Collect color images and IR images of living objects at the same moment. The data at the same moment are named by time, with the suffixes _RGB and _IR to distinguish the two image formats. Collect color images and IR images of non-living objects at the same moment. The data at the same moment are named by time (year, month, day, hour, minute, second), with the suffixes _RGB and _IR to distinguish the two image formats. For example, the RGB image and IR image of the same target are named 20230820110425_RGB.jpg and 20230820110425_IR.jpg, respectively. The number of collected RGB-IR image pairs is no less than 10,000.
[0083] B. Label the data from step A, i.e., mark the location information of the face frames in the color image and IR image, with living subjects marked as TP and non-living subjects as FP;
[0084] C. Using the labeled data from step B, construct a convolutional neural network (NET1) for liveness detection in IR images, a convolutional neural network (NET2) for liveness detection in color images, and a dual-input liveness classification neural network (NET3).
[0085] D. Perform model training on the liveness detection convolutional neural network NET1 under IR images, the liveness detection convolutional neural network NET2 under color images, and the liveness dual-input classification neural network NET3 in step C respectively;
[0086] E. Liveness detection applications, such as Figure 1 shown
[0087] A target image to be detected for liveness is placed in front of a binocular camera. The camera collects data at the same time for identification. The color image data enters NET2, and the IR image data enters NET1. NET1 and NET2 networks output the face location information and liveness category information, respectively. The face location information output by NET1 and NET2 is cut out and sent to NET3. NET3 outputs liveness category information. The output results of the three networks are combined to determine the liveness category, thereby completing video-based liveness detection.
[0088] In step A, image data of living and non-living objects are collected under color images, IR images (IR is the abbreviation of infrared radiation), and IR binocular infrared equipment supports both color images and IR images.
[0089] Live data refers to real facial data, and non-live data refers to fake facial data (such as color prints, photos, photos in electronic devices, 3D head models, etc.); color images and IR images are collected simultaneously and in pairs at the same time.
[0090] In step B, the face target's circumscribed rectangle is used as the boundary for annotation. The annotation range is the left and right boundaries excluding the ears, the lower boundary is the chin position, and the upper boundary is the forehead position excluding the hair. According to this rule, the targets in the color image and IR image are annotated separately. Non-living targets are labeled FP (False Person) and living targets are labeled TP (True Person) for distinction.
[0091] Furthermore, in step C, the process of constructing the liveness detection convolutional neural network NET1 under IR images, the liveness detection convolutional neural network NET2 under color images, and the liveness dual-input classification neural network NET3 is as follows:
[0092] D1. Construct the liveness detection convolutional neural networks NET1 and NET2, both using the YOLOV3 network.
[0093] D2. Use the TP and FP target categories and location information in the IR image to train the neural network NET1;
[0094] D3. Use the TP and FP target categories and location information in the color image to train the neural network NET2;
[0095] The liveness detection convolutional neural network NET1 can provide the target's location information, label information, and target confidence for an IR image in subsequent detections;
[0096] The liveness detection convolutional neural network NET2 can provide the target location information, i.e., rectangular box information, label information, i.e., whether it is live TP or non-live FP, and the target confidence, i.e., category score, for a color image in subsequent detection;
[0097] Furthermore, in order to more accurately eliminate the mixed fake facial information, we further constructed a living dual-input classification neural network NET3.
[0098] Among them, the network structure of NET3 is shown in Table 1:
[0099] Among them, we call the Convolution layer, connected to the BatchNorm layer, connected to the Scale layer, connected to the ReLU layer, and this sequential flow is referred to as CBSR, where;
[0100] We call the Convolution layer, connected to the BatchNorm layer, connected to the Scale layer, and this sequential flow is referred to as CBS. In the NET3 network, all convolution layers use 3*3 convolution kernels.
[0101] The input data is dual-channel, which is the color image data and face position information data_1 obtained from NET2, and the IR image data and face position information data_2 obtained from NET1. The input data size is 112*112*3, that is, width*height*channel;
[0102] The data_1 data stream and the data_2 data stream are trained separately in layers 0-34. The operations on the data_1 data stream and the data_2 data stream are consistent in layers 0-34 and have no intersection. The purpose is to fully extract the color image feature information and the IR image feature information separately without interfering with each other.
[0103] The color image branch data flow is as follows:
[0104] The data_1 data stream first performs CBSR-CBSR-Pooling operations through sequence numbers 0-3, with the purpose of performing a preliminary feature extraction and dimensionality reduction;
[0105] Furthermore, the Split operation is performed on sequence 4. One branch of the feature information is directly connected to the Eltwise layer of sequence 7. After the CBSR-CBS operation, the other branch is connected to the Eltwise layer of sequence 7. Through this cross-layer connection method, the feature information of the previous and next contexts can be fully combined to obtain features, while avoiding the problems of gradient vanishing and gradient exploding.
[0106] Furthermore, the ReLU operation of sequence number 8 is performed;
[0107] Furthermore, sequence number 9 performs a Split operation, one branch of the feature information is directly connected to the Eltwise layer of sequence number 12, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of sequence number 12;
[0108] Furthermore, the ReLU operation of sequence number 13 is performed;
[0109] Furthermore, sequence number 14 performs a Split operation, one branch of the feature information is directly connected to the Eltwise layer of sequence number 17, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of sequence number 17;
[0110] Furthermore, the ReLU operation of sequence number 18 is performed;
[0111] Furthermore, sequence number 19 performs a Split operation, one branch of the feature information performs a CBS operation and then connects to the Eltwise layer of sequence number 22, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of sequence number 22;
[0112] Furthermore, the ReLU operation of sequence number 23 is performed;
[0113] Furthermore, the sequence number 24 performs a Split operation, one branch of the feature information is directly connected to the Eltwise layer of the sequence number 27, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of the sequence number 27;
[0114] Furthermore, the ReLU operation of sequence number 28 is performed;
[0115] Furthermore, sequence number 29 performs a Split operation, one branch of the feature information is directly connected to the Eltwise layer of sequence number 32, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of sequence number 32;
[0116] Furthermore, the ReLU operation of sequence number 33 is performed;
[0117] Furthermore, sequence number 34 performs a Split operation. One branch of the feature information performs a CBS operation and then connects to the Eltwise layer of sequence number 37. Another branch performs a CBSR-CBS operation and then connects to the Eltwise layer of sequence number 37. Another branch enters the mixed channel branch and connects to the Concat layer of sequence number 35 to fuse the color image features and IR features.
[0118] Furthermore, the ReLU operation of sequence number 38 is performed;
[0119] Furthermore, sequence number 39 performs a Split operation, one branch of the feature information is directly connected to the Eltwise layer of sequence number 42, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of sequence number 42;
[0120] Furthermore, the ReLU operation of sequence number 43 is performed;
[0121] Furthermore, the Pooling operation of sequence number 44 is performed;
[0122] Further, perform the Reshape operation of sequence number 45;
[0123] Furthermore, the InnerProduct operation of sequence number 46 is performed, and finally the classification result of the color image branch data stream is output;
[0124] Similarly, the processing operations of the IR graph branch data stream are the same as those of the color graph branch data stream;
[0125] The hybrid channel branch data flow is as follows:
[0126] The mixed channel branch obtains the feature data of the color image and the feature data of the IR image at sequence number 35, and combines the Concat operation to complement each other;
[0127] Furthermore, sequence number 36 performs a Split operation, one branch of the feature information performs a CBS operation and then connects to the Eltwise layer of sequence number 39, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of sequence number 39;
[0128] Furthermore, the ReLU operation of sequence number 40 is performed;
[0129] Furthermore, the sequence number 41 performs a Split operation, one branch of the feature information is directly connected to the Eltwise layer of the sequence number 44, and one branch performs a CBSR-CBS operation and then connects to the Eltwise layer of the sequence number 44;
[0130] Furthermore, the ReLU operation of sequence number 45 is performed;
[0131] Furthermore, the Pooling operation of sequence number 46 is performed;
[0132] Further, perform the Reshape operation of sequence number 47;
[0133] Furthermore, the InnerProduct operation of sequence number 48 is performed, and finally the classification result of the mixed channel branch data stream is output;
[0134] The advantage of this network is that it performs feature extraction training on the IR channel, performs feature extraction training on the color image channel, and fully utilizes the fusion information of the two channels for training on the mixed channel (the intermediate features come from the IR channel and the color image channel), which can maximize the use of information from the two channels.
[0135]
[0136]
[0137] D4. The input of the live dual-input classification neural network NET3 is dual-channel: Channel 1 is the target face image corresponding to the IR image; Channel 2 is the target face image corresponding to the color image. It is important to note that in binocular liveness detection equipment, the IR lens and color lens are basically installed in the same position, with high consistency and small left-right deviation. When we feed training images to NET, we must train them in pairs. This pair refers to the IR image and color image of the same target at the same time.
[0138] Furthermore, the liveness detection in step E is applied as follows:
[0139] E1. Feed the IR image obtained by the binocular liveness detection device into the liveness detection convolutional neural network NET1, which outputs the location of the face target to be confirmed in the image, as well as category information and confidence information;
[0140] E2. Feed the color image obtained by the binocular liveness detection device into the liveness detection convolutional neural network NET2, which outputs the location of the face target to be confirmed in the image, as well as category information and confidence information;
[0141] E3. Compare the target information in the IR image obtained in steps E1 and E2 with the target information in the color image. Since the installation position of the device lens is slightly different, the same target is located in the same position in the IR image and the color image. We confirm this using the IOU (Intersection over Union) strategy.
[0142] E4. If the category information of the same target is consistent with that output in the color image, and both score thresholds are greater than the set threshold, the category is considered reliable and the result is directly output; here we set the threshold limit to 0.8;
[0143] If the category information output in the IR image is inconsistent with the category information output in the color image, or the category information is consistent but at least one score threshold is below the limit, further confirmation is required and the process jumps to step E5;
[0144] E5. The target image (IR image) and the target image (color image) obtained in step E3 are simultaneously fed into the in vivo dual-input classification neural network NET3. The network outputs three groups: a color image branch, an IR image branch, and a mixed channel branch. Each group outputs category information (live TP / non-live FP) and a category confidence score. If the category information of the three output groups (color image branch, IR image branch, and mixed channel branch) is consistent and has a high confidence score, we set a threshold of 0.9 here, and the category is considered reliable and the consistent category information is output. Otherwise, the output is uncertain.
[0145] If it is a live TP, further facial recognition or other operations will be performed; if it is not a live TP, no operation will be performed or a prompt will be given;
[0146] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.
Claims
1. A video-based liveness detection method, characterized by: The following steps are involved: Step 1: Use IR binocular infrared equipment to simultaneously collect image data of living and non-living objects in pairs under color images and IR images; Step 2: label the image data in step 1; Step 3: Construct a convolutional neural network for liveness detection in IR images, a convolutional neural network for liveness detection in color images, and a liveness dual-input classification neural network. Step 4: Perform model training on the liveness detection convolutional neural network and liveness dual-input classification neural network constructed in step 3; Step 5: Use the model trained in step 4 to output the liveness detection results; The specific steps of step 3 include: (1) Based on the YOLOV3 network, a convolutional neural network NET1 for liveness detection in IR images is constructed: the TP and FP target categories and position information in the IR images are used to train the neural network NET1; (2) Based on the YOLOV3 network, a convolutional neural network NET2 for liveness detection in color images is constructed: the TP and FP target categories and position information in the color image are used to train the neural network NET2; Among them, the liveness detection convolutional neural network NET1 can provide the target's location information, label information, and target confidence for an IR image in subsequent detection; The liveness detection convolutional neural network NET2 can provide the target's location information, label information, and target confidence for a color image in subsequent detections; Based on the position information output from step (1) and step (2), the corresponding IR image and the target face image in the color image are deducted in pairs according to the position information, and used as the input information of the living double-input classification neural network NET3 to construct the living double-input classification neural network NET3; The NET3 network outputs three groups: color image branch, IR image branch, and mixed channel branch. Each group outputs the category confidence of living TP and non-living FP.
2. The video-based living body detection method according to claim 1, wherein: The specific method of step 1 is: Collect image data of living and non-living objects under color images and IR images respectively; Live data refers to real face data, while non-live data refers to fake face data; The color image and IR image are collected simultaneously and in pairs at the same time.
3. The video-based liveness detection method according to claim 1, wherein: The specific method of step 2 is: The face target's circumscribed rectangle is used as the boundary for annotation. The annotation range is the left and right boundaries excluding the ears, the lower boundary is the chin position, and the upper boundary is the forehead position excluding the hair. According to this rule, non-living objects and living objects in color images and IR images are annotated separately. Among them, non-living targets are marked as FP, and living targets are marked as TP for distinction.
4. The video-based living body detection method according to claim 1, wherein: The input of the living dual-input classification neural network NET3 in step (3) of step 3 is dual channels, channel 1 is the target face image corresponding to the IR image; channel 2 is the target face image corresponding to the color image.
5. The video-based living body detection method according to claim 1, wherein: The specific method of step 4 is: Paired training images are fed to the living body dual-input classification neural network NET3. The paired training images refer to the IR image and color image of the same target at the same time.
6. The video-based living body detection method according to claim 1, wherein: The specific steps of step 5 include: (1) The IR image obtained by the binocular liveness detection device is fed into the liveness detection convolutional neural network NET1, which outputs the position of the face target to be confirmed and the confidence information in the image; (2) The color image obtained by the binocular liveness detection device is fed into the liveness detection convolutional neural network NET2, which outputs the position of the face target to be confirmed in the image and the confidence information; (3) matching the target information under the IR image and the target information under the color image obtained in steps (1) and (2); (4) If the category information output in the IR image and the category information output in the color image of the same target are consistent, and both score thresholds are greater than the set threshold, the category is considered credible and the result is output directly; If the category information output in the IR image is inconsistent with the category information output in the color image, or the category information is consistent but at least one score threshold is lower than the limit, further confirmation is required and jump to step (5); (5) The target image under the corresponding IR image and the target image under the color image obtained in step (3) are simultaneously sent to the living body dual-input classification neural network NET3. The network will integrate the information under the two images, complement each other with the required feature information, and make a comprehensive judgment on whether it is a living body classification result; if the score threshold of the output category information is greater than the set threshold, the result is output, and the category is considered credible, otherwise uncertain information is output.
Citation Information
Patent Citations
Image detection method and device, computer equipment and storage medium
CN112308035A