Static face living body detection method, device and system based on binocular camera

Abstract

A method, apparatus, and system for static face living body detection based on a binocular camera are disclosed. The method for static face living body detection based on the binocular camera according to an embodiment of the present application comprises: intercepting a left eye face image and a right eye face image corresponding to a specific face from a left eye image frame and a right eye image frame captured by a binocular camera; performing face key point detection on the left eye face image and the right eye face image; and judging whether the specific face is a living face based on position information of each face key point of the left eye face image and the right eye face image.

Description

Technical Field

[0001] This invention relates to the field of computers, and more particularly to a static face liveness detection method, apparatus, and system based on a binocular camera. Background Technology

[0002] In recent years, the application areas of facial recognition technology have gradually expanded. However, facial spoofing and fraud techniques have posed a significant challenge to the accuracy and security of facial recognition. Common facial spoofing and fraud techniques include displaying facial photos, playing facial videos, and wearing facial masks or hoods. If facial recognition technology cannot accurately identify whether a face is a real, living person, it could greatly threaten people's property, privacy, and even personal safety. Therefore, facial liveness detection technology has emerged. Summary of the Invention

[0003] According to an embodiment of the present invention, a static face liveness detection method based on a binocular camera includes: extracting a left-eye face image and a right-eye face image corresponding to a specific face from a left-eye image frame and a right-eye image frame captured by the binocular camera; performing facial key point detection on the left-eye face image and the right-eye face image; and determining whether the specific face is a live face based on the position information of each facial key point in the left-eye face image and the right-eye face image.

[0004] The static face liveness detection device based on a binocular camera according to an embodiment of the present invention includes components for implementing the above-described static face liveness detection method based on a binocular camera.

[0005] A static face liveness detection system based on a binocular camera according to an embodiment of the present invention includes: a processor; and a memory storing computer-executable instructions thereon, wherein, when executed by the processor, the computer-executable instructions cause the processor to perform the above-described static face liveness detection method based on a binocular camera. Attached Figure Description

[0006] The invention can be better understood from the following description of specific embodiments of the invention in conjunction with the accompanying drawings, wherein:

[0007] Figure 1 This is an example block diagram of a static face liveness detection device based on a binocular camera according to an embodiment of the present invention;

[0008] Figure 2 This is an example flowchart of a static face liveness detection method based on a binocular camera according to an embodiment of the present invention;

[0009] Figure 3 It is possible. Figure 1 The device and / or shown Figure 2 An example block diagram of a computer system for the method shown. Detailed Implementation

[0010] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific configurations and algorithms presented below, but covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description in order to avoid unnecessarily obscuring the invention.

[0011] Currently, facial liveness detection technology is mainly divided into two categories: dynamic cooperative liveness detection and static silent liveness detection. Dynamic cooperative liveness detection involves the system issuing a series of commands (e.g., closing eyes, opening mouth), and the user performing the actions according to the commands. The system then detects whether the user's actions match the commands to determine if the user is alive. Because dynamic cooperative liveness detection requires user cooperation and often involves judging multiple actions, the detection time is relatively long and the user experience is poor. Static silent liveness detection is mainly divided into two-dimensional (2D) detection and three-dimensional (3D) detection. 2D detection primarily distinguishes between real and fake faces based on differences in features (e.g., moiré patterns, reflectivity, Local Binary Pattern (LBP) features), while 3D detection primarily distinguishes between real and fake faces based on 3D data obtained from the depth information of the face. This method can better defend against planar spoofing attacks such as those using facial photos, and also enhances defense against mask-based attacks, making it the most effective technology currently available.

[0012] Existing 3D liveness detection technologies include three approaches: structured light, time-of-flight (TOF), and binocular cameras. Structured light and TOF solutions are relatively expensive and therefore unsuitable for large-scale commercialization, while binocular camera solutions are relatively inexpensive and widely used. However, existing binocular camera solutions have one or more drawbacks: they are essentially based on 2D liveness detection principles and cannot effectively identify stereoscopic fraud (e.g., distorted photos, masks, etc.); binocular cameras require capturing left and right eye images with corresponding pixel features that are as similar as possible, thus typically collecting RGB features and lacking infrared (IR) spectral features, resulting in complex algorithms, slow processing speeds, long computation times, and poor real-time performance.

[0013] In view of the above, a static face liveness detection method, apparatus, and system according to embodiments of the present invention are proposed, which can detect whether a static face is a live face more accurately, more easily, and in more real time.

[0014] Figure 1 Example block diagrams of a static face liveness detection device based on a binocular camera according to embodiments of the present invention are shown. Figure 2 An example flowchart of a static face liveness detection method based on a binocular camera according to an embodiment of the present invention is shown. The following is in conjunction with... Figure 1 and Figure 2 This paper describes in detail the static face liveness detection device and method based on a binocular camera according to embodiments of the present invention.

[0015] like Figure 1 As shown, the static face liveness detection device 100 based on a binocular camera according to an embodiment of the present invention includes a face image cropping unit 102, a face feature detection unit 104, and a liveness detection unit 106, wherein: the face image cropping unit 102 is configured to crop left-eye face images and right-eye face images corresponding to a specific face from the left-eye image frame and right-eye image frame captured by the binocular camera (i.e., execute step S202); the face feature detection unit 104 is configured to perform face key point detection on the left-eye face image and right-eye face image (i.e., execute step S204); the liveness detection unit 106 is configured to determine whether a specific face is a live face based on the position information of each face key point in the left-eye face image and right-eye face image (i.e., execute step S206).

[0016] In some embodiments, the face image cropping unit 102 may be configured to crop left-eye and right-eye face images corresponding to a specific face from the left-eye and right-eye image frames through the following processes: performing face region detection and face contour point detection on the left-eye and right-eye image frames; performing face alignment on the left-eye and right-eye image frames based on the face regions and face contour points in the left-eye and right-eye image frames; and cropping left-eye and right-eye face images corresponding to a specific face from the face-aligned left-eye and right-eye image frames.

[0017] In some embodiments, the face image cropping unit 102 can be configured to use a lightweight face detection model to detect face regions in the left and right image frames and five facial contour points (e.g., left and right eyes, nose tip, and left and right corners of the mouth) of the detected face regions. The lightweight face detection model is obtained by modifying the weights of each layer and adding a facial contour point detection branch based on, for example, a MobileNet series neural network. It can detect five facial contour points of any face region after detecting it in any image frame. Compared to ordinary face detection models, the lightweight face detection model has reduced size and computational cost, thus improving its running speed. It can be applied to embedded devices with lower computing power, saving costs while maintaining accuracy.

[0018] In some embodiments, the face image cropping unit 102 can be configured to perform face alignment on the left and right image frames using an affine transformation operation. For example, for any face region in either the left or right image frame, the affine transformation operation can be performed as follows: an affine transformation matrix for performing the affine transformation operation on the face region is calculated based on the two-dimensional coordinates of the face contour points and the target transformation position coordinates, and the affine transformation operation is performed on the face region based on the affine transformation matrix. Specifically, the affine transformation operation is a transformation operation between two-dimensional coordinates, which can be represented by equation (1):

[0019]

[0020] Where X represents the original image coordinates, T represents the transformed image coordinates, and A and B represent the affine transformation coefficients. Let For convenience, A and B are often combined into an affine transformation matrix M:

[0021]

[0022] Substituting equation (2) into equation (1), we get:

[0023]

[0024] In this embodiment, it is assumed that the two-dimensional coordinates of the five facial contour points (e.g., left and right eyes, nose tip, left and right corners of mouth) in the face region are X, and their target transformation position coordinates are T. The affine transformation matrix M can be obtained based on X and T. Then, the face region after face alignment can be obtained by performing an affine transformation operation on the face region based on the affine transformation matrix M.

[0025] In some embodiments, the face image cropping unit 102 may be configured to perform at least one of distortion correction and stereo correction on at least one of the left and right image frames before cropping the left and right face images from the left and right image frames, so that the left and right image frames are coplanar and aligned. This ensures that the face positions in the left and right image frames are the same, making subsequent facial landmark detection and liveness detection results more accurate.

[0026] In some embodiments, the face feature detection unit 104 can be configured to perform head pose estimation on the left and right face images while simultaneously detecting facial landmarks on the left and right face images. In this case, when the head poses of both the left and right face images meet predetermined conditions, it is further determined whether the specific face corresponding to them is a live face.

[0027] In some embodiments, the face feature detection unit 104 can be configured to use a Lightweight Practical Facial Pose and Landmark Detector (PFPLD) algorithm to perform facial landmark detection and head pose estimation on the left and right face images. For example, when the three pose angles (e.g., pitch, yaw, and roll angles) characterizing the head pose of the left and right face images are all less than 30 degrees, the system continues to determine whether the specific face corresponding to them is a live face. This prevents misjudgments caused by excessively large pose angles or excessive occlusion of facial landmarks. The PFPLD algorithm is an improvement on the Practical Facial Landmark Detector (PFLD) algorithm. It primarily strengthens the head pose branch and merges the facial landmark and head pose branches, resulting in better regression performance for both facial landmark detection and head pose estimation. Furthermore, it makes facial landmark detection more robust to complex conditions such as closed eyes, varying lighting, facial occlusion, and image blur. Compared to the standard PFPLD algorithm, the lightweight PFPLD algorithm can be implemented with a lightweight model of approximately 1.7MB while maintaining comparable accuracy, making it suitable for embedded devices with limited computing power.

[0028] In some embodiments, the liveness detection unit 106 can be configured to determine whether a specific face is a live face based on the positional information of each facial key point in the left and right face images corresponding to that specific face, through the following processing: calculating the three-dimensional coordinates of each facial key point in at least one face image in the left and right face images based on the two-dimensional coordinates of each facial key point in each face image in the left and right face images; and determining whether the specific face is a live face based on the three-dimensional coordinates of each facial key point in at least one face image in the left and right face images. Compared to traditional 2D liveness detection technology, 3D liveness detection technology based on the three-dimensional coordinates of facial key points has higher accuracy and can defend against more types of face spoofing fraud, thus providing higher security.

[0029] In some embodiments, the live face determination unit 104 can be configured to calculate the three-dimensional coordinates of each facial key point in at least one face image in the left and right eye face images based on the two-dimensional coordinates of each facial key point in each face image in the left and right eye face images through the following process: Based on the two-dimensional coordinates of each facial key point in the left and right eye face images, fine-tune the two-dimensional coordinates of each facial key point in a first face image in the left and right eye face images, wherein the first face image is any one of the left and right eye face images. A face image is used. Based on the finely adjusted 2D coordinates of each facial key point in the first face image and the 2D coordinates of each facial key point in the left and right face images, the disparity between corresponding facial key points in the first and second face images is calculated. The second face image is another face image among the left and right face images. The 3D coordinates of each facial key point in the second face image are also calculated based on the 2D coordinates of each facial key point in the second face image and the disparity between the corresponding facial key points in the first and second face images. This method does not use traditional stereo matching, eliminating the need to search for matching points line by line in the left and right face images. Instead, it directly calculates the disparity between corresponding facial key points in the left and right face images, resulting in faster computation and applicability to real-time detection scenarios.

[0030] In some embodiments, the live face determination unit 106 can be configured to fine-tune the two-dimensional coordinates of each facial key point in the first face image in the left and right face images based on the two-dimensional coordinates of each facial key point in the left and right face images corresponding to a specific face through the following processing: calculating the average difference of the horizontal coordinates between facial key points located on the midline of the face in the left and right face images; dividing the left and right face images into multiple face partitions and calculating the average offset of the horizontal coordinates between facial key points located in the corresponding face partitions in the left and right face images; and calculating the fine-tuned two-dimensional coordinates of each facial key point in the first face image based on the two-dimensional coordinates of each facial key point in the first face image and the average offset of the horizontal coordinates between facial key points located in the corresponding face partitions in the left and right face images.

[0031] For example, if the face feature detection unit 104 detects 106 facial key points in each face image in the left and right face images, the three-dimensional coordinates of each facial key point in the left face image can be calculated based on the two-dimensional coordinates of each facial key point in the left and right face images through the following processing:

[0032] (1) For each of the 106 facial key points in the left and right face images, the facial key points located on the midline of the face are the 17th, 52nd, 53rd, 54th, 55th, 61st, 88th, 94th, 99th, and 103rd facial key points. The average difference between the abscissa (i.e., x-coordinate) and ordinate (i.e., y-coordinate) of these 10 facial key points in the left and right face images can be calculated according to the following equation (4): mean_x, mean_y.

[0033]

[0034] Among them, lx i ly i Let rx represent the x and y coordinates of the i-th facial landmark in the left eye image, respectively. i ry i Let x and y represent the x and y coordinates of the i-th facial key point in the right eye face image, respectively, i∈(17,52,53,54,55,61,88,94,99,103).

[0035] (2) Divide the left and right face images into the following 6 face regions: left contour, right contour, left eyebrow + left eye, right eyebrow + right eye, nose, and mouth. For any face region, calculate the mean_bias of the horizontal coordinates between the facial key points located in the face region in the left and right face images according to the following equation (5).

[0036]

[0037] Among them, lx j rx represents the x-coordinate of the j-th facial keypoint located in the face region of the left eye face image. j This represents the x-coordinate of the j-th facial keypoint located in the face region of the right eye face image. j This indicates the number of facial landmarks located in this face partition.

[0038] (3) For any face partition, based on the average offset of the horizontal coordinates between the facial key points located in the face partition in the left and right face images, calculate the fine-tuned x-coordinate new_rx of each facial key point located in the face partition in the right face image according to the following equation (6). j .

[0039] new_rx j =rx j -mean_bias (6)

[0040] (4) Using the x-coordinate of any facial landmark (e.g., the k-th facial landmark) in the left eye face image and the fine-tuned x-coordinate of the corresponding facial landmark (e.g., the k-th facial landmark) in the right eye face image, calculate the disparity disp between the corresponding facial landmarks in the left eye face image and the right eye face image according to the following equation (7). k .

[0041] disp k =lx k -new_rx k (7)

[0042] (5) Based on the following equation (8) and the disparity obtained in the previous step, calculate the three-dimensional coordinates x of each facial key point in the left eye face image. 3d y 3d depth k .

[0043]

[0044] Where f represents the focal length of the left eye camera in the binocular camera system, b represents the baseline distance of the binocular camera system, and cx and cy represent the principal point coordinates of the left eye camera in the binocular camera system, respectively.

[0045] In some embodiments, the live face determination unit 106 can be configured to use a live face classification model to determine whether a specific face is a live face based on the three-dimensional coordinates of each facial key point in the left and / or right eye face images corresponding to that specific face. For example, the three-dimensional coordinates of 106 facial feature points from a large number of images can be used as features to train the live face classification model.

[0046] In some embodiments, a camera calibration checkerboard and a binocular camera calibration tool (e.g., the SteroCamera Cailbrator tool in Matlab) can be used to calibrate the binocular camera and obtain the intra-camera parameters and extrinsic parameters of the left and right binocular cameras.

[0047] It should be noted that binocular cameras can be in the form of dual IR cameras, dual RGB cameras, or RGB camera + IR camera; that is, there are no restrictions on the selection of binocular cameras.

[0048] Figure 3 It is possible. Figure 1 The device and / or shown Figure 2 An example block diagram of a computer system illustrating the method is shown. It should be understood that... Figure 3 The computer system 300 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0049] like Figure 3 As shown, the computer system 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the computer system 300. The processing device 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0050] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, cameras, accelerometers, gyroscopes, sensors, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, motors, electronic speed controllers, etc.; storage devices 308 including, for example, flash cards; and communication devices 309. Communication device 309 allows computer system 300 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 3 A computer system 300 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0051] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure provide a computer-readable storage medium storing a computer program that includes functions for executing... Figure 2 The program code shown represents processes S202 to S206. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it implements... Figure 2 The processes S202 to S206 are shown.

[0052] It should be noted that the computer-readable medium according to embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. The computer-readable storage medium according to embodiments of the present invention may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Additionally, the computer-readable signal medium according to embodiments of the present invention may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0053] Computer program code for performing operations according to embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0054] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0055] This invention can be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithm described in a particular embodiment can be modified without departing from the basic spirit of the invention. Therefore, the present embodiments are to be regarded as exemplary rather than limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and scope of the claims and their equivalents are thus included within the scope of the invention.

Claims

1. A static face liveness detection method based on a binocular camera, comprising: Extract the left and right eye face images corresponding to a specific face from the left and right eye image frames captured by the binocular camera; Facial landmark detection is performed on the left-eye and right-eye face images; Calculating the three-dimensional coordinates of each facial key point in at least one of the left-eye and right-eye face images includes: Based on the two-dimensional coordinates of each facial key point in the left and right face images, the two-dimensional coordinates of each facial key point in the first face image in the left and right face images are fine-tuned. This fine-tuning includes: calculating the average difference in horizontal coordinates between facial key points located on the midline of the face in the left and right face images; dividing the left and right face images into multiple face partitions; calculating the average offset of horizontal coordinates between facial key points located in the corresponding face partitions in the left and right face images; and calculating the fine-tuned two-dimensional coordinates of each facial key point in the first face image based on the two-dimensional coordinates of each facial key point in the first face image and the average offset of horizontal coordinates between facial key points located in the corresponding face partitions in the left and right face images. The first face image is any face image in either the left or right face image. Based on the finely adjusted two-dimensional coordinates of each facial key point in the first face image and the two-dimensional coordinates of each facial key point in the second face image in the left eye face image and the right eye face image, the disparity between the corresponding facial key points in the first face image and the second face image is calculated, wherein the second face image is another face image in the left eye face image and the right eye face image. Based on the two-dimensional coordinates of each facial key point in the second face image and the disparity between corresponding facial key points in the first and second face images, the three-dimensional coordinates of each facial key point in the second face image are calculated; and Based on the three-dimensional coordinates of each facial key point in at least one of the left-eye and right-eye face images, it is determined whether the specific face is a living face.

2. The binocular camera based static face liveness detection method of claim 1, wherein, The process of extracting the left-eye face image and the right-eye face image from the left-eye image frame and the right-eye image frame includes: Face region detection and face contour point detection are performed on the left and right eye image frames; Face alignment is performed on the left and right image frames based on the face regions and face contour points in the left and right image frames; and The left-eye face image and the right-eye face image are extracted from the left-eye image frame and the right-eye image frame after face alignment.

3. The static face liveness detection method based on a binocular camera according to claim 2, wherein, Face alignment is performed on the left and right image frames using an affine transformation operation.

4. The static face liveness detection method based on a binocular camera according to claim 3, wherein, The process of aligning faces in the left and right image frames based on the face regions and face contour points in the left and right image frames includes: For any face region in either the left or right image frame, an affine transformation matrix is ​​calculated based on the two-dimensional coordinates of the face contour points in the face region and the target transformation position coordinates. Then, an affine transformation operation is performed on the face region based on the affine transformation matrix.

5. The static face liveness detection method based on a binocular camera according to claim 1 further includes: While performing facial landmark detection on the left and right eye face images, head pose estimation is also performed on the left and right eye face images. Specifically, when the head posture of both the left-eye face image and the right-eye face image meets predetermined conditions, it is further determined whether the specific face is a living face.

6. The static face liveness detection method based on a binocular camera according to claim 1 further includes: Before extracting the left-eye face image and the right-eye face image from the left-eye image frame and the right-eye image frame, at least one of the left-eye image frame and the right-eye image frame is subjected to at least one of distortion correction and stereo correction, so that the left-eye image frame and the right-eye image frame are coplanar and aligned.

7. The static face liveness detection method based on a binocular camera according to claim 1 further includes: Obtain the internal and external parameters of the left and right cameras in the binocular camera system.

8. A static face liveness detection device based on a binocular camera, comprising a unit for implementing the static face liveness detection method based on a binocular camera as described in any one of claims 1 to 7.

9. A static face liveness detection system based on a binocular camera, comprising: processor; as well as A memory having stored thereon computer-executable instructions, wherein, when executed by the processor, the computer-executable instructions cause the processor to perform the static face liveness detection method based on a binocular camera as described in any one of claims 1 to 7.

Images