Face living body detection method and device, computer readable storage medium and equipment

Abstract

The application discloses a face living body detection method and device, readable storage medium and equipment, and belongs to the face recognition field. The application obtains face images under ambient light and light compensation, obtains overall regions and local regions of the face images under the ambient light and the light compensation through pre-processing, extracts features through a convolutional neural network, and classifies living face and false face after fusing the extracted features. The application utilizes the feature that the living face changes locally when encountering a light source, analyzes the local change of the face before and after the light compensation, utilizes the overall change feature of the living face before and after the light compensation, analyzes the overall change of the face images before and after the light compensation, and performs living body detection. The application has high accuracy, is simple and convenient, does not need active cooperation of a user, and has good user experience.

Description

Technical Field

[0001] This invention relates to the field of face recognition, and in particular to a method, apparatus, computer-readable storage medium, and device for face liveness detection. Background Technology

[0002] Currently, facial recognition technology is widely used in the financial and security fields. While faces offer advantages such as easy and contactless acquisition, they are also easily exploited by others to breach facial recognition systems using photos or videos. Therefore, facial liveness detection technology, as the first line of defense in facial recognition, is particularly important.

[0003] Currently, there are three main methods for mobile face liveness detection. The first method is face action liveness detection, where the liveness detection system issues random head and face action commands, and the user is judged to be alive if they complete the corresponding actions according to the commands. The second method is to extract features from RGB images to distinguish between real and fake faces, and perform binary classification of real and fake faces. The third method is an integration of the first two methods. In the process of using the first method for face action liveness detection, multiple RGB images with multiple expressions are provided for the second method of face liveness detection to determine whether they are real faces.

[0004] Facial motion liveness detection requires a high degree of user cooperation, and the random head and face motion commands issued are generally single action commands, such as nodding, turning the head, blinking, and opening the mouth. Criminals have a high probability of being able to deceive the liveness detection system by shaking, distorting, or rotating photos, or by recording videos of multiple user actions, which can also easily pass the liveness detection system.

[0005] The second method, silent liveness detection, extracts features from RGB images to distinguish between real and fake faces. This method requires no user intervention and is more readily accepted by users. Deep learning methods are generally used to extract features for distinguishing real and fake faces. Driven by a large amount of live and non-live face data, the system automatically learns to effectively differentiate between real and fake faces, distinguishing the differences in their imaging. However, RGB images are prone to misjudgment due to differences in lighting, resolution, and different phone lenses. Summary of the Invention

[0006] To address the shortcomings of existing face liveness detection methods, which require user cooperation and are prone to misjudgment, this invention provides a face liveness detection method, apparatus, readable storage medium, and device, which improves the accuracy of face liveness detection and does not require active user cooperation, resulting in a better user experience.

[0007] The technical solution provided by this invention is as follows:

[0008] In a first aspect, the present invention provides a method for detecting human face liveness, the method comprising:

[0009] Acquire ambient light face images and supplementary lighting face images of the same user over a period of time, and perform preprocessing to obtain an overall ambient light face image, a partial ambient light face image, an overall supplementary lighting face image, and a partial supplementary lighting face image.

[0010] The first feature and the second feature are obtained by extracting features from the overall image of the face under ambient light and the overall image of the face under supplementary light using a first convolutional neural network; the third feature and the fourth feature are obtained by extracting features from the partial image of the face under ambient light and the partial image of the face under supplementary light using a second convolutional neural network.

[0011] The first feature, the second feature, the third feature, and the fourth feature are fused together to obtain the fused feature;

[0012] The fusion features are used to classify whether the ambient light face image and the supplementary lighting face image are from a live subject.

[0013] Furthermore, the first convolutional neural network and the second convolutional neural network are trained using the following method:

[0014] Construct ambient light overall face image samples, ambient light partial face image samples, supplementary lighting overall face image samples, and supplementary lighting partial face image samples, and assign labels to each;

[0015] The first convolutional neural network is used to extract features from the overall image samples of the face under ambient light and the overall image samples of the face under supplementary light to obtain the fifth and sixth features; the second convolutional neural network is used to extract features from the local image samples of the face under ambient light and the local image samples of the face under supplementary light to obtain the seventh and eighth features.

[0016] Calculate the contrast loss L1 between the fifth and sixth features and the contrast loss L2 between the seventh and eighth features;

[0017] The fifth, sixth, seventh, and eighth features are fused to obtain fused sample features, and the cross-entropy loss L3 of the fused sample features is calculated.

[0018] The weighted sum of L1, L2, and L3 yields the total loss L. All The parameters of the first and second convolutional neural networks are updated through backpropagation.

[0019] Furthermore, the fifth, seventh, sixth, and eighth features are sequentially connected to obtain fused sample features;

[0020] The first feature, the third feature, the second feature, and the fourth feature are sequentially connected to obtain the fused feature.

[0021] Furthermore, the partial image of the face is an image of the eyes.

[0022] Furthermore, the preprocessing of the ambient light face image and the supplementary lighting face image includes:

[0023] Face detection, facial landmark localization, head pose estimation, and face normalization are performed on the ambient light face image / filled light face image to obtain the overall ambient light face image / filled light face image.

[0024] Using the eye coordinates obtained from facial key point localization as the center, expand outwards by a certain number of pixels to obtain the region of interest containing the eyes. The number of pixels expanded outwards is determined according to the distance between the left and right eyes.

[0025] Within the region of interest, the center and radius of the eye are determined using a radial symmetry transformation;

[0026] A partial image of the face under ambient light / partial image of the face under supplementary lighting is obtained by cropping based on the center and radius of the eye.

[0027] Secondly, the present invention provides a face liveness detection device, the device comprising:

[0028] The image acquisition module is used to acquire ambient light face images and supplementary lighting face images of the same user within a certain period of time, and to preprocess them to obtain ambient light overall face image, ambient light partial face image, supplementary lighting overall face image, and supplementary lighting partial face image.

[0029] The image feature extraction module is used to extract features from the overall image of the face under ambient light and the overall image of the face under supplementary lighting using a first convolutional neural network to obtain a first feature and a second feature; and to extract features from the partial image of the face under ambient light and the partial image of the face under supplementary lighting using a second convolutional neural network to obtain a third feature and a fourth feature.

[0030] The image feature fusion module is used to fuse the first feature, the second feature, the third feature and the fourth feature to obtain the fused feature;

[0031] The liveness detection module is used to classify the fused features and determine whether the ambient light face image and the supplementary lighting face image are from a live person.

[0032] Furthermore, the first convolutional neural network and the second convolutional neural network are trained using the following modules:

[0033] The sample construction module is used to construct ambient light face overall image samples, ambient light face partial image samples, supplementary lighting face overall image samples, and supplementary lighting face partial image samples, and set labels for each;

[0034] The sample feature extraction module is used to extract features from the overall ambient light face image sample and the overall supplementary lighting face image sample using a first convolutional neural network to obtain the fifth and sixth features; and to extract features from the local ambient light face image sample and the local supplementary lighting face image sample using a second convolutional neural network to obtain the seventh and eighth features.

[0035] The contrast loss calculation module is used to calculate the contrast loss L1 between the fifth feature and the sixth feature, and the contrast loss L2 between the seventh feature and the eighth feature.

[0036] The cross-entropy loss calculation module is used to fuse the fifth, sixth, seventh and eighth features to obtain fused sample features, and to calculate the cross-entropy loss L3 of the fused sample features.

[0037] The backpropagation module is used to perform a weighted summation of L1, L2, and L3 to obtain the total loss L. All The parameters of the first and second convolutional neural networks are updated through backpropagation.

[0038] Furthermore, the fifth, seventh, sixth, and eighth features are sequentially connected to obtain fused sample features;

[0039] The first feature, the third feature, the second feature, and the fourth feature are sequentially connected to obtain the fused feature.

[0040] Furthermore, the partial image of the face is an image of the eyes.

[0041] Further preprocessing of ambient light face images and supplemental lighting face images includes:

[0042] The face processing unit is used to perform face detection, face key point localization, head pose estimation and face normalization on the ambient light face image / filled light face image to obtain the overall ambient light face image / filled light face image.

[0043] The eye region of interest acquisition unit is used to expand outwards by a number of pixels from the eye coordinates obtained by facial key point localization to obtain the region of interest containing the eye. The number of pixels expanded outwards is determined according to the distance between the left and right eyes.

[0044] A pupil localization unit is used to determine the eye center and radius using radial symmetry transformation within the region of interest;

[0045] The image cropping unit is used to crop a partial image of the face under ambient light or a partial image of the face under supplementary lighting, based on the center and radius of the eyes.

[0046] Thirdly, the present invention provides a computer-readable storage medium for face liveness detection, including a memory for storing processor-executable instructions, which, when executed by the processor, implement the steps of the face liveness detection method described in the first aspect.

[0047] Fourthly, the present invention provides an apparatus for face liveness detection, comprising at least one processor and a memory storing computer-executable instructions, wherein the processor executes the instructions to implement the steps of the face liveness detection method described in the first aspect.

[0048] The present invention has the following beneficial effects:

[0049] This invention acquires facial images under ambient light and supplemental lighting conditions. Through preprocessing, it obtains the overall and local regions of the facial images under both ambient and supplemental lighting conditions. Features are extracted using a convolutional neural network, and the extracted features are fused for classification of live faces and spoofed faces. This invention utilizes the characteristic that live faces undergo local changes when exposed to point light sources to analyze the changes in local facial features before and after supplemental lighting. Furthermore, it leverages the overall imaging changes of live faces before and after supplemental lighting to analyze the overall changes in the facial images before and after supplemental lighting, performing liveness detection with high accuracy. It is simple and convenient, requires no active user cooperation, and provides a good user experience. Attached Figure Description

[0050] Figure 1 This is a flowchart of the face liveness detection method of the present invention;

[0051] Figure 2 A comparison of imaging effects of live and prosthetic eye areas under supplemental lighting and non-supplemental lighting conditions;

[0052] Figure 3 , 4 This is a schematic diagram of the preprocessing process;

[0053] Figure 5 A schematic diagram of face liveness detection using a convolutional neural network;

[0054] Figure 6 This is a schematic diagram of training a convolutional neural network.

[0055] Figure 7 The flowchart shows the training methods for the first and second convolutional neural networks.

[0056] Figure 8 Flowchart of the method for obtaining the training set;

[0057] Figure 9A flowchart illustrating the method for preprocessing ambient light face images and supplemental lighting face images;

[0058] Figure 10 This is a schematic diagram of the face liveness detection device of the present invention. Detailed Implementation

[0059] To make the technical problems, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. The components of the embodiments of this invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0060] Example 1:

[0061] This invention provides a face liveness detection method that can be used in face recognition devices, especially mobile face recognition devices. The mobile face recognition device has a screen or a supplementary light to provide supplementary lighting during face liveness detection. The mobile face recognition device includes, but is not limited to, smartphones, laptops, tablets, PDAs, etc.

[0062] Figure 1 This is a flowchart of the face liveness detection method of the present invention, the method comprising:

[0063] S100: Acquire ambient light face images and supplementary light face images of the same user over a period of time, and perform preprocessing to obtain an overall ambient light face image, a partial ambient light face image, an overall supplementary light face image, and a partial supplementary light face image.

[0064] In this step, an RGB image can be captured under ambient light using a face camera to obtain an ambient light face image. Then, supplementary lighting is applied, and an RGB image is captured under supplementary lighting conditions to obtain a supplementary lighting face image.

[0065] Facial recognition devices can use supplemental lighting. If the facial recognition device is a mobile device such as a smartphone, its built-in screen lighting can also be used, without the need for additional hardware, and this is very convenient. Screen lighting can maximize screen brightness, and except for the face area, the pixels can be set to all white, red, yellow, or blue, or alternately flashing.

[0066] In one example, the specific implementation method for obtaining the ambient light face image and the supplementary lighting face image is as follows:

[0067] Multiple ambient light face images are acquired within the time period t1, and face detection is performed simultaneously. Ambient light face images with detected faces and meeting the image quality requirements are retained. If no face is detected within the time period t1 or the image quality does not meet the requirements, a failure is returned, prompting the user to adjust the face position and re-acquire ambient light face images.

[0068] After successfully acquiring an ambient light face image, turn on the fill light or screen to provide fill light. Acquire multiple fill light face images within time period t2, and perform face detection simultaneously. Retain ambient light face images that detect faces and whose image quality meets the requirements. If no face is detected or the image quality does not meet the requirements within time period t2, return failure and remind the user to adjust the acquisition position or posture and re-acquire fill light face images.

[0069] Head poses of multiple ambient light / supplementary light face images were obtained through keypoint regression, and ambient light / supplementary light face images with frontal head poses were selected.

[0070] Based on image quality, the algorithm selects suitable ambient light / filled light face images from face images with frontal head pose, thereby improving the algorithm's robustness.

[0071] Selecting suitable supplemental lighting / ambient light face images can be done using traditional methods such as image signal-to-noise ratio (SNR) or face image quality assessment algorithms; this invention does not limit the selection. Generally, supplemental lighting / ambient light face images with a quality score higher than a set quality score threshold can be selected.

[0072] Then, the ambient light face image and the supplementary light face image are preprocessed to obtain the overall ambient light face image, the partial ambient light face image, the overall supplementary light face image, and the partial supplementary light face image.

[0073] Prosthetic faces generally include printed facial photos, photographs of faces that have been reproduced and played on a screen, or videos. This invention needs to distinguish between prosthetic faces and live faces to achieve liveness detection.

[0074] Studies have shown that localized areas of the human face, such as the eyes, cheeks, nose, forehead, and mouth, are sensitive to point light sources and change before and after supplemental lighting. Taking the eye as an example, under ambient light, there is essentially no light spot within the pupil. Under supplemental lighting, a light spot forms in the black part of the pupil, and the pupil constricts, containing a brighter light spot. Therefore, a real human eye (i.e., a living organism) shows a significant difference before and after supplemental lighting: before supplemental lighting, there is no light spot within the pupil; after supplemental lighting, the pupil contains a brighter light spot. Figure 2 As shown, under supplemental lighting conditions, areas such as the cheeks, nose, forehead, and mouth exhibit shadows due to unevenness.

[0075] The eyes in a prosthetic face do not possess the characteristic of being sensitive to point light sources. Therefore, there is no difference in the appearance of the eyes before and after illumination; either a light spot is present in the pupil simultaneously before and after illumination, or a light spot is absent simultaneously. Figure 2 As shown, a real face shows a change in the light spot within the pupil before and after illumination, while a prosthetic face shows no such change. Because a prosthetic face is flat, its cheeks, nose, forehead, and mouth show no shadows under both ambient and illumination conditions. These characteristics allow for the differentiation between real and prosthetic faces.

[0076] Therefore, this invention obtains ambient light and supplementary lighting partial images of the face through preprocessing, and utilizes the characteristics of changes in a live face under ambient light and supplementary lighting conditions to distinguish between a live face and a fake face. The smaller the changes in the ambient light and supplementary lighting partial images (i.e., the more similar the ambient light and supplementary lighting partial images are), the more likely it is that the ambient light and supplementary lighting partial images originate from a fake face.

[0077] For example, by utilizing the characteristic that the pupil forms a light spot when it encounters a point light source, we can analyze whether there are changes in the pupil light spot in local images of a face under ambient light and those under supplemental lighting, in order to distinguish between a live face and a fake face. The smaller the change in the pupil light spot in the local images of a face under ambient light and those under supplemental lighting, the more it indicates that the image is from a fake face.

[0078] However, many users currently wear glasses, and the lenses can create reflective spots. These reflective spots can easily obscure the light source within the pupil, leading to a high misjudgment rate when using pupil light to distinguish between live and fake faces when the user is wearing glasses. Furthermore, printing fake faces on smooth, reflective paper like photographic paper, and distorting the image, can also artificially create light spots or shadows within the pupil, thus affecting the accuracy of liveness detection.

[0079] To address this issue, this invention focuses on the entire human face. Compared to a prosthetic face, a real (live) face contains rich 3D information. Under supplemental lighting from the same direction as the lens, areas closer to the lens, such as the tip of the nose and forehead, are brighter, while areas farther from the lens, such as the eye sockets and cheeks, are darker, resulting in a significant difference compared to the image before supplemental lighting under ambient light. Under ambient light, the brightness difference between various parts of a real face is not significant, and the contrast between areas closer to and farther from the lens is not obvious. In other words, due to the rich 3D information of a real face, the image of a real face changes considerably before and after supplemental lighting.

[0080] The surfaces of printed facial photographs and the screens displaying them are essentially flat. From the perspective of light reflection, the reflection angles at various points on a flat surface are roughly the same. Therefore, under supplemental lighting, a flat prosthetic face will not exhibit the characteristics of a real face (i.e., areas closer to the lens, such as the tip of the nose and forehead, are brighter, while areas farther from the lens, such as the eye sockets and cheeks, are darker). This results in minimal changes to the image of the prosthetic face before and after supplemental lighting.

[0081] Even if the surface of a face photo is not flat by distorting the photo, the 3D information of the face constructed by the distorted photo is relatively simple and differs greatly from the 3D information of a real face. Therefore, it cannot achieve the imaging effect of making the parts of the real face, such as the tip of the nose and forehead, which are closer to the lens, brighter and the parts of the face, such as the eye sockets and cheeks, which are farther from the lens, darker.

[0082] It is evident that the image of a real face changes significantly before and after illumination, while the image of a prosthetic face changes less. This characteristic can be used to distinguish between real and prosthetic faces.

[0083] Therefore, this invention obtains an overall image of the face under ambient light and an overall image of the face under supplementary lighting through preprocessing, and analyzes the changes in the overall images of the face under ambient light and supplementary lighting to distinguish between a live face and a fake face. The smaller the changes in the overall images of the face under ambient light and supplementary lighting (i.e., the more similar the overall images of the face under ambient light and supplementary lighting are), the more likely it is that the overall images of the face under ambient light and supplementary lighting originate from a fake face.

[0084] S200: The first feature and the second feature are obtained by extracting features from the overall image of the face under ambient light and the overall image of the face under supplementary light using a first convolutional neural network; the third feature and the fourth feature are obtained by extracting features from the local image of the face under ambient light and the local image of the face under supplementary light using a second convolutional neural network.

[0085] As described above, this invention distinguishes between live faces and artificial faces by analyzing changes in ambient light local images and supplementary light local images of faces (e.g., changes in light spots within the pupils), as well as analyzing changes in ambient light overall images and supplementary light overall images of faces.

[0086] This invention analyzes the changes in local images of a face under ambient light and local images of a face under supplemental lighting using convolutional neural networks, as well as the changes in overall images of a face under ambient light and overall images of a face under supplemental lighting.

[0087] The first and second convolutional neural networks of this invention are lightweight or small networks, such as the base network of the Siamese Network. This invention requires information about the overall and local regions of the face; therefore, two convolutional neural networks (i.e., the first and second convolutional neural networks) are used. The structures of the first and second convolutional neural networks can be the same. Alternatively, considering algorithm efficiency, the second convolutional neural network can be relatively smaller and more lightweight to improve algorithm efficiency.

[0088] An example of the structure of a first convolutional neural network and a second convolutional neural network is as follows: Figure 5 As shown, Net1 is the first convolutional neural network used to extract features from the entire face region, specifically extracting features from the ambient light face image and the supplementary lighting face image. Here, face_b and face_f represent the ambient light face image and the supplementary lighting face image, respectively, and the extracted features are the first feature f1 and the second feature f2, each with a length of 256.

[0089] Net2 is the second convolutional neural network used for feature extraction of local regions, extracting features from both ambient light and supplemental lighting local face images. Here, eye_b and eye_f represent the ambient light and supplemental lighting local face images, respectively, and the extracted features are the third feature f3 and the fourth feature f4, each with a length of 256.

[0090] S300: The first feature, the second feature, the third feature and the fourth feature are fused to obtain the fused feature.

[0091] This invention fuses features from ambient light overall face image, supplementary light overall face image, ambient light partial face image, and supplementary light partial face image to fully extract the 3D information of the overall face and information such as light spots in the eyes, which is used to distinguish between fake faces and live faces with high accuracy.

[0092] The feature fusion method used in this invention can be feature stacking, directly connecting (Concat) the four features for fusion, or it can be fusion using convolution.

[0093] In one example, the first, third, second, and fourth features are concatenated sequentially to obtain the fused feature. That is, they are concatenated in the order f1, f3, f2, f4 to form a 1024-dimensional fused feature fc.

[0094] S400: Classify the images based on the fusion features to determine whether the ambient light face image and the supplementary lighting face image are from a live subject.

[0095] The fusion features include overall information and local information such as eyes in the face image under ambient light, as well as overall information and local information such as eyes in the face image under supplementary lighting. By analyzing the fusion features, the degree of change in the local face images under ambient light and supplementary lighting, as well as the degree of change in the overall face images under ambient light and supplementary lighting, can be obtained to distinguish between live faces and fake faces. The smaller the degree of change, the more similar the ambient light and supplementary lighting face images are, and the more likely they are to originate from a fake face.

[0096] During classification, Softmax can be used to classify the fused feature fc, where a classification result of 1 represents a fake and 0 represents a live.

[0097] This invention acquires facial images under ambient light and supplemental lighting conditions. Through preprocessing, it obtains the overall and local regions of the facial images under both ambient and supplemental lighting conditions. Features are extracted using a convolutional neural network, and the extracted features are fused for classification of live faces and spoofed faces. This invention utilizes the characteristic that live faces undergo local changes when exposed to point light sources to analyze the changes in local facial features before and after supplemental lighting. Furthermore, it leverages the overall imaging changes of live faces before and after supplemental lighting to analyze the changes in facial images before and after supplemental lighting for liveness detection. This method is highly accurate, simple, convenient, requires no active user cooperation, and provides a good user experience.

[0098] The aforementioned first and second convolutional neural networks need to be trained before use, such as... Figure 7 As shown, the training method is as follows:

[0099] S10: Construct ambient light overall face image samples, ambient light partial face image samples, supplementary light overall face image samples, and supplementary light partial face image samples, and set labels for each.

[0100] Its specific implementation methods include:

[0101] Obtain a training set consisting of training sample pairs, wherein the training sample pairs include a pair of spoofed face images consisting of ambient light face images and supplementary light face images of spoofed faces, and a pair of live face images consisting of ambient light face images and supplementary light face images of live faces.

[0102] Different labels are assigned to the prosthetic face image pairs and the live face image pairs respectively.

[0103] For example, the label of a fake face image pair can be set to 1, indicating that it is similar and fake, while the label of a live face image pair can be set to 0, indicating that it is dissimilar and live.

[0104] The training sample pairs are preprocessed to obtain ambient light overall face image samples, ambient light partial face image samples, supplementary lighting overall face image samples, and supplementary lighting partial face image samples.

[0105] S20: The first convolutional neural network is used to extract features from the overall image samples of the face under ambient light and the overall image samples of the face under supplementary light to obtain the fifth and sixth features; the second convolutional neural network is used to extract features from the local image samples of the face under ambient light and the local image samples of the face under supplementary light to obtain the seventh and eighth features.

[0106] The feature extraction method in this step is the same as that in S200. You can refer to the content described in step S200 for understanding, and it will not be repeated here.

[0107] S30: Calculate the contrast loss L1 between the fifth feature and the sixth feature, and the contrast loss L2 between the seventh feature and the eighth feature.

[0108] This invention trains and extracts features from both ambient light and supplemental lighting face images using the same convolutional neural network (i.e., a first convolutional neural network), sharing weights. This first convolutional neural network is a Siamese network. The Siamese network is used to analyze and extract features from pairs of samples, more specifically targeting the imaging differences of faces before and after supplemental lighting, and using these differences for classification. The same approach is applied to ambient light and supplemental lighting face local image samples.

[0109] The extracted features are supervised using Contrastive Loss.

[0110] Contrastive Loss, also known as contrastive loss, is expressed as follows:

[0111]

[0112] Where L is the contrast loss, and d = ||a n -b n ||2, where a is a feature of two samples n b n The Euclidean distance, n = 1, 2, ..., N, where N is the sample feature a n b n The number of samples, y is the label for whether the features of two samples match, y=1 means that the features of two samples are similar or match, y=0 means that they do not match, and margin is the set threshold.

[0113] In this invention, the contrast loss is used to measure whether two samples of the same person's overall face image / partial face image match under ambient light and supplementary lighting. A mismatch represents a real person sample pair, and a match represents a fake person sample pair. f1 and f2, f3 and f4 are used to calculate the contrast loss L1 and L2 in pairs, respectively.

[0114] S40: The fifth, sixth, seventh and eighth features are fused to obtain fused sample features, and the cross-entropy loss L3 of the fused sample features is calculated.

[0115] The method for feature fusion in this step is the same as in S300, and can be understood by referring to the content described in step S300, which will not be repeated here. In one example, the fifth, seventh, sixth, and eighth features are sequentially connected to obtain the fused sample features.

[0116] This invention uses the Cross Entropy Loss function for classification by fusing sample features, calculating the Cross Entropy Loss L3, as follows: Figure 6 As shown.

[0117] S50: Sum the weighted values ​​of L1, L2, and L3 to obtain the total loss L. All The parameters of the first and second convolutional neural networks are updated through backpropagation.

[0118] In one example, the weights of L1, L2, and L3 are 0.25, 0.25, and 0.5, respectively. All =0.25L1+0.25L2+0.5L3.

[0119] In the aforementioned S10, the training set consisting of training sample pairs can be obtained using the following method, such as... Figure 8 As shown:

[0120] S11: Acquire several ambient light face images of a live human face captured by the face camera under ambient light.

[0121] In this step, the face in front of the face camera is a live face. When acquiring face images, the face camera is turned on, and a live face image is acquired under ambient light. After the face detection algorithm detects the face and determines that the image quality meets the requirements, acquisition continues, continuously acquiring n_b face images to obtain the ambient light face image of the live face.

[0122] S12: Acquire several live human face images captured by the face camera under the illumination of a fill light or screen.

[0123] In this step, the face in front of the camera is a live face. The upper-layer application controls the fill light or screen brightness to capture a live face image under this illuminated environment. When a face is detected in the image and the image quality meets the requirements, acquisition continues, continuously acquiring n_f face images. Once the required number of images is reached, the fill light is turned off or the screen returns to normal, resulting in a live face image with fill light.

[0124] S13: Randomly select one image from the ambient light face image and the supplementary light face image of the live face to form a live face image pair.

[0125] This step is used to create live face image pairs. Each live face image pair contains an ambient light face image and a supplementary lighting face image of a live face acquired at the same time. One image is randomly selected multiple times from each of the n_b ambient light face images and the n_f supplementary lighting face images to form a live face image pair.

[0126] S14: Acquire several ambient light face images of the prosthetic face captured by the face camera under ambient light.

[0127] In this step, the face in front of the camera is a fake face. The acquisition method is the same as in S11. You can refer to the content described in step S11 for understanding. It will not be repeated here.

[0128] S15: Acquire several illuminated images of the prosthetic face captured by the face camera under the illumination of a fill light or screen.

[0129] In this step, the face in front of the camera is a fake face. The acquisition method is the same as in S12. You can refer to the content described in step S12 for understanding. It will not be repeated here.

[0130] S16: Randomly select one image from the ambient light face image and the supplementary light face image of the prosthetic face to form a prosthetic face image pair.

[0131] The implementation method of this step is the same as that of S13. You can refer to the content described in step S13 for understanding, and it will not be repeated here.

[0132] The facial features of the present invention can be the eyes, cheeks, nose, forehead, mouth and other facial features, preferably the eyes.

[0133] In this invention, taking the eye region as an example of a human face, it can be achieved through methods such as... Figure 9 The method shown preprocesses the ambient light face image and the supplementary lighting face image:

[0134] S110: Perform face detection, face key point localization, head pose estimation, and face normalization on the ambient light face image / supplementary light face image to obtain the overall ambient light face image / supplementary light face image.

[0135] This invention does not limit the specific methods for face detection, facial landmark localization, head pose estimation, and face normalization. In one example, the MTCNN detection algorithm is used to achieve face detection and facial landmark localization. There are five facial landmarks: left eye, right eye, nose, left corner of mouth, and right corner of mouth. 3D pose information is estimated using the 2D coordinate information of the landmarks. Face normalization is performed using the binocular coordinates, specifically including face alignment and scaling. The eyes are aligned to (38, 52) and (74, 52), and scaled to (112, 112). Figure 3 As shown.

[0136] S120: Using the eye coordinates obtained from facial key point localization as the center, expand outwards by several pixels to obtain the region of interest containing the eyes. The number of pixels expanded outwards is determined according to the distance between the left and right eyes.

[0137] To fully capture the changes in pupil size before and after supplemental lighting, the following fine-grained normalization operation was performed:

[0138] Using the eye coordinates obtained from facial landmark localization as the center, expand outward by several pixels to obtain the region of interest (eye ROI) containing the eye.

[0139] The number of pixels that expand outwards is determined by the distance between the left and right eyes, which is d. iris =|x right -x left |,x right x is the x-coordinate of the right eye. left Let d be the horizontal coordinate of the left eye. In one example, the number of pixels expanding outwards is d. iris / 3.

[0140] S130: Within the region of interest, the center and radius R of the eye are determined using the radial symmetry transform (RST).

[0141] S140: Obtain a partial image of the face under ambient light / partial image of the face under supplementary lighting based on the center and radius of the eye.

[0142] For each eye, left and right, the center of the circle (for the left eye, the center is x) is used as the reference point. l_center y l_centerCentered on a target image, crop an image with a width of 4.4R and a height of 2.2R. Scale the cropped left and right eye images to 112*56 pixels, then stitch and stack the scaled left and right eye images together to obtain an ambient light face local image / filled light face local image, as shown below. Figure 4 As shown.

[0143] In the aforementioned S10, when constructing ambient light face overall image samples, ambient light face partial image samples, supplementary light face overall image samples, and supplementary light face partial image samples, it is also necessary to preprocess the training sample pairs. This preprocessing method is the same as the fish hammer method for ambient light face images and supplementary light face images.

[0144] Example 2:

[0145] This invention provides a face liveness detection device, such as... Figure 10 As shown, the device includes:

[0146] Image acquisition module 1 is used to acquire ambient light face images and supplementary light face images of the same user within a certain period of time, and perform preprocessing to obtain ambient light overall face image, ambient light partial face image, supplementary light overall face image, and supplementary light partial face image.

[0147] Image feature extraction module 2 is used to extract features from the overall image of the face under ambient light and the overall image of the face under supplementary lighting through a first convolutional neural network to obtain the first feature and the second feature; and to extract features from the local image of the face under ambient light and the local image of the face under supplementary lighting through a second convolutional neural network to obtain the third feature and the fourth feature.

[0148] Image feature fusion module 3 is used to fuse the first feature, the second feature, the third feature and the fourth feature to obtain fused features.

[0149] The liveness detection module 4 is used to classify faces by fusing features and determine whether the face images with ambient light and those with supplemental lighting are from live subjects.

[0150] This invention acquires facial images under ambient light and supplemental lighting conditions. Through preprocessing, it obtains the overall and local regions of the facial images under both ambient and supplemental lighting conditions. Features are extracted using a convolutional neural network, and the extracted features are fused for classification of live faces and spoofed faces. This invention utilizes the characteristic that live faces undergo local changes when exposed to point light sources to analyze the changes in local facial features before and after supplemental lighting. Furthermore, it leverages the overall imaging changes of live faces before and after supplemental lighting to analyze the overall changes in the facial images before and after supplemental lighting, performing liveness detection with high accuracy. It is simple and convenient, requires no active user cooperation, and provides a good user experience.

[0151] The aforementioned first and second convolutional neural networks need to be trained before use, and they are obtained through the following modules:

[0152] The sample construction module is used to construct ambient light overall face image samples, ambient light partial face image samples, supplementary lighting overall face image samples, and supplementary lighting partial face image samples, and set labels for each.

[0153] The sample feature extraction module is used to extract features from the overall image samples of the face under ambient light and the overall image samples of the face under supplementary lighting through the first convolutional neural network to obtain the fifth and sixth features; and to extract features from the local image samples of the face under ambient light and the local image samples of the face under supplementary lighting through the second convolutional neural network to obtain the seventh and eighth features.

[0154] The contrast loss calculation module is used to calculate the contrast loss L1 between the fifth and sixth features and the contrast loss L2 between the seventh and eighth features.

[0155] The cross-entropy loss calculation module is used to fuse the fifth, sixth, seventh, and eighth features to obtain fused sample features, and to calculate the cross-entropy loss L3 of the fused sample features.

[0156] The backpropagation module is used to perform a weighted summation of L1, L2, and L3 to obtain the total loss L. All The parameters of the first and second convolutional neural networks are updated through backpropagation.

[0157] In one example, the aforementioned sample construction module includes:

[0158] The first acquisition unit is used to acquire several ambient light face images of live human faces captured by the face camera under ambient light.

[0159] The second acquisition unit is used to acquire several illuminated facial images of live human faces captured by the face lens under the illumination of a fill light or screen.

[0160] The live face image pair acquisition unit is used to randomly extract one image from the ambient light face image and the supplementary light face image of the live face to form a live face image pair.

[0161] The third acquisition unit is used to acquire several ambient light face images of the prosthetic face captured by the face lens under ambient light.

[0162] The fourth acquisition unit is used to acquire several illuminated facial images of the prosthetic face captured by the face lens under the illumination of a fill light or screen.

[0163] The prosthetic face image pair acquisition unit is used to randomly extract one image from each of the ambient light face image and the supplementary light face image of the prosthetic face to form a prosthetic face image pair.

[0164] In this invention, the fusion method of the first feature, the third feature, the second feature and the fourth feature is not limited. In one example, the fifth feature, the seventh feature, the sixth feature and the eighth feature are connected in sequence to obtain the fused sample features.

[0165] Correspondingly, the first feature, third feature, second feature, and fourth feature are sequentially connected to obtain the fused feature.

[0166] The facial partial image of this invention is an eye image. Correspondingly, the preprocessing of the ambient light facial image and the supplementary lighting facial image includes:

[0167] The face processing unit is used to perform face detection, face landmark localization, head pose estimation, and face normalization on ambient light face images / supplementary light face images to obtain ambient light overall face images / supplementary light overall face images.

[0168] The eye region of interest acquisition unit is used to expand outwards by a number of pixels from the eye coordinates obtained by facial key point localization to obtain a region of interest containing the eyes. The number of pixels expanded outwards is determined according to the distance between the left and right eyes.

[0169] The pupil localization unit is used to determine the eye center and radius within the region of interest using radial symmetry transformation.

[0170] The image cropping unit is used to crop a partial image of the face under ambient light or a partial image of the face under supplementary lighting, based on the center and radius of the eyes.

[0171] The device provided in this embodiment of the invention has the same implementation principle and technical effects as the aforementioned method embodiment 1. For the sake of brevity, any parts not mentioned in this device embodiment can be referred to the corresponding content in the aforementioned method embodiment 1. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the aforementioned device and unit can all be referred to the corresponding processes in the aforementioned method embodiment 1, and will not be repeated here.

[0172] Example 3:

[0173] The method described in Embodiment 1 of this invention can implement business logic through a computer program and record it on a storage medium. This storage medium can be read and executed by a computer, achieving the effects of the solution described in Embodiment 1 of this specification. Therefore, this invention also provides a computer-readable storage medium for face liveness detection, including a memory for storing processor-executable instructions. When executed by a processor, the instructions implement the steps of the face liveness detection method of Embodiment 1.

[0174] This invention acquires facial images under ambient light and supplemental lighting conditions. Through preprocessing, it obtains the overall and local regions of the facial images under both ambient and supplemental lighting conditions. Features are extracted using a convolutional neural network, and the extracted features are fused for classification of live faces and spoofed faces. This invention utilizes the characteristic that live faces undergo local changes when exposed to point light sources to analyze the changes in local facial features before and after supplemental lighting. Furthermore, it leverages the overall imaging changes of live faces before and after supplemental lighting to analyze the overall changes in the facial images before and after supplemental lighting, performing liveness detection with high accuracy. It is simple and convenient, requires no active user cooperation, and provides a good user experience.

[0175] The storage medium may include a physical device for storing information, typically digitizing the information and then storing it using electrical, magnetic, or optical methods. The storage medium may include: devices that store information using electrical energy, such as various types of memory, like RAM and ROM; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memory, bubble memory, and USB flash drives; and devices that store information using optical methods, such as CDs or DVDs. Of course, there are other readable storage media, such as quantum memories and graphene memories.

[0176] The storage medium described above may also include other implementation methods according to the description of method embodiment 1. The implementation principle and technical effects of this embodiment are the same as those of the aforementioned method embodiment 1. For details, please refer to the description of the relevant method embodiment 1, which will not be repeated here.

[0177] Example 4:

[0178] The present invention also provides a device for face liveness detection. The device may be a standalone computer, or it may include an actual operating device that uses one or more of the methods or embodiments described in this specification. The face liveness detection device may include at least one processor and a memory storing computer-executable instructions. When the processor executes the instructions, it implements the steps of any one or more of the face liveness detection methods described in Embodiment 1.

[0179] This invention acquires facial images under ambient light and supplemental lighting conditions. Through preprocessing, it obtains the overall and local regions of the facial images under both ambient and supplemental lighting conditions. Features are extracted using a convolutional neural network, and the extracted features are fused for classification of live faces and spoofed faces. This invention utilizes the characteristic that live faces undergo local changes when exposed to point light sources to analyze the changes in local facial features before and after supplemental lighting. Furthermore, it leverages the overall imaging changes of live faces before and after supplemental lighting to analyze the overall changes in the facial images before and after supplemental lighting, performing liveness detection with high accuracy. It is simple and convenient, requires no active user cooperation, and provides a good user experience.

[0180] The device described above may include other implementation methods according to the description of method embodiment 1. The implementation principle and technical effects of this embodiment are the same as those of the aforementioned method embodiment 1. For details, please refer to the description of the relevant method embodiment 1, which will not be repeated here.

[0181] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting human face liveness, characterized in that, The method includes: Acquire ambient light face images and supplementary lighting face images of the same user over a period of time, and perform preprocessing to obtain an overall ambient light face image, a partial ambient light face image, an overall supplementary lighting face image, and a partial supplementary lighting face image. The first feature and the second feature are obtained by extracting features from the overall image of the face under ambient light and the overall image of the face under supplementary light using a first convolutional neural network; the third feature and the fourth feature are obtained by extracting features from the partial image of the face under ambient light and the partial image of the face under supplementary light using a second convolutional neural network. The first feature, the second feature, the third feature, and the fourth feature are fused together to obtain the fused feature; The fusion features are used to classify whether the ambient light face image and the supplementary lighting face image are from a live subject.

2. The face liveness detection method according to claim 1, characterized in that, The first convolutional neural network and the second convolutional neural network are trained using the following method: Construct ambient light overall face image samples, ambient light partial face image samples, supplementary lighting overall face image samples, and supplementary lighting partial face image samples, and assign labels to each; The first convolutional neural network is used to extract features from the overall image samples of the face under ambient light and the overall image samples of the face under supplementary lighting to obtain the fifth and sixth features; the second convolutional neural network is used to extract features from the local image samples of the face under ambient light and the local image samples of the face under supplementary lighting to obtain the seventh and eighth features. Calculate the contrast loss L1 between the fifth and sixth features and the contrast loss L2 between the seventh and eighth features; The fifth, sixth, seventh, and eighth features are fused to obtain fused sample features, and the cross-entropy loss L3 of the fused sample features is calculated. The weighted sum of L1, L2, and L3 yields the total loss L. All The parameters of the first convolutional neural network and the second convolutional neural network are updated through backpropagation.

3. The face liveness detection method according to claim 2, characterized in that, The fifth, seventh, sixth, and eighth features are sequentially connected to obtain the fused sample features; The first feature, the third feature, the second feature, and the fourth feature are sequentially connected to obtain the fused feature.

4. The face liveness detection method according to any one of claims 1-3, characterized in that, The partial image of the face is an image of the eyes.

5. The face liveness detection method according to claim 4, characterized in that, Preprocessing of the ambient light face image and the supplementary lighting face image includes: Face detection, facial landmark localization, head pose estimation, and face normalization are performed on the ambient light face image / filled light face image to obtain the overall ambient light face image / filled light face image. Using the eye coordinates obtained from facial key point localization as the center, expand outwards by a certain number of pixels to obtain the region of interest containing the eyes. The number of pixels expanded outwards is determined according to the distance between the left and right eyes. Within the region of interest, the center and radius of the eye are determined using a radial symmetry transformation; A partial image of the face under ambient light / partial image of the face under supplementary lighting is obtained by cropping based on the center and radius of the eye.

6. A face liveness detection device, characterized in that, The device includes: The image acquisition module is used to acquire ambient light face images and supplementary lighting face images of the same user within a certain period of time, and to preprocess them to obtain ambient light overall face image, ambient light partial face image, supplementary lighting overall face image, and supplementary lighting partial face image. The image feature extraction module is used to extract features from the overall image of the face under ambient light and the overall image of the face under supplementary lighting using a first convolutional neural network to obtain a first feature and a second feature; and to extract features from the partial image of the face under ambient light and the partial image of the face under supplementary lighting using a second convolutional neural network to obtain a third feature and a fourth feature. The image feature fusion module is used to fuse the first feature, the second feature, the third feature and the fourth feature to obtain the fused feature; The liveness detection module is used to classify the fused features and determine whether the ambient light face image and the supplementary lighting face image are from a live person.

7. The face liveness detection device according to claim 6, characterized in that, The first convolutional neural network and the second convolutional neural network are trained using the following modules: The sample construction module is used to construct ambient light face overall image samples, ambient light face partial image samples, supplementary lighting face overall image samples, and supplementary lighting face partial image samples, and set labels for each; The sample feature extraction module is used to extract features from the overall ambient light face image sample and the overall supplementary lighting face image sample using the first convolutional neural network to obtain the fifth feature and the sixth feature; and to extract features from the local ambient light face image sample and the local supplementary lighting face image sample using the second convolutional neural network to obtain the seventh feature and the eighth feature. The contrast loss calculation module is used to calculate the contrast loss L1 between the fifth feature and the sixth feature, and the contrast loss L2 between the seventh feature and the eighth feature. The cross-entropy loss calculation module is used to fuse the fifth, sixth, seventh and eighth features to obtain fused sample features, and to calculate the cross-entropy loss L3 of the fused sample features. The backpropagation module is used to perform a weighted summation of L1, L2, and L3 to obtain the total loss L. All The parameters of the first convolutional neural network and the second convolutional neural network are updated through backpropagation.

8. The face liveness detection device according to claim 6 or 7, characterized in that, The partial image of the face is an image of the eyes.

9. A computer-readable storage medium for face liveness detection, characterized in that, It includes a memory for storing processor-executable instructions, which, when executed by the processor, implement the steps of the face liveness detection method according to any one of claims 1-5.

10. A device for face liveness detection, characterized in that, It includes at least one processor and a memory storing computer-executable instructions, wherein the processor executes the instructions to implement the steps of the face liveness detection method according to any one of claims 1-5.

Images