A face recognition method, device, intelligent door lock and computer program product

Abstract

The application provides a face recognition method and device, an intelligent door lock and a computer program product, and relates to the technical field of face recognition. A face recognition method comprises: acquiring a face color image, a first face infrared image and a face speckle image of a recognition object; performing face detection on the first face infrared image; acquiring a second face infrared image according to the face detection result; generating a face depth image according to the face speckle image; performing living body discrimination based on the face depth image and the second face infrared image; performing feature fusion on at least two images in the face color image, the second face infrared image and the face depth image to acquire a fused feature vector; and performing face recognition on the recognition object through the fused feature vector. According to the embodiment of the application, various prosthesis attacks can be effectively prevented, and the accuracy of the face recognition effect can be ensured.

Description

Technical Field

[0001] This application relates to the field of facial recognition technology, and more specifically, to a facial recognition method, device, smart door lock, and computer program product. Background Technology

[0002] Currently, door locks with facial recognition functionality are increasingly being used in daily life. However, existing facial recognition solutions for door locks are vulnerable to attacks using spoofing, including physical photos, video playback, and 3D head models. Therefore, it is necessary to incorporate liveness detection functionality into the facial recognition solutions for door locks to improve their security.

[0003] Single 2D or deep liveness detection cannot effectively prevent attacks using fake faces of various types, materials, and scenarios. Moreover, existing technologies for integrating facial recognition systems into smart terminals are significantly affected by chip computing power costs.

[0004] The current trend in smart door lock functionality is to have both peephole and biometric recognition functions. Existing technologies require a combination of peephole and biometric recognition devices to achieve this, which is costly and difficult to promote and apply in the industry.

[0005] Therefore, current door lock facial recognition solutions urgently need a technology that combines multimodal data fusion to prevent various spoofing attacks, ensure the security and stability of facial recognition, and improve the efficiency of facial recognition. Summary of the Invention

[0006] This application provides a face recognition method, device, smart door lock, and computer program product. Through multimodal fusion information liveness detection and face recognition scheme, it can solve the problem of attacks on face recognition systems by various non-live face images.

[0007] According to one aspect of this application, a face recognition method is provided, comprising: acquiring a color image of a face of an object to be recognized, a first infrared image of a face, and a speckle image of a face; processing the first infrared image of the face, comprising: performing face detection on the first infrared image of the face; acquiring a second infrared image of the face based on the face detection result; generating a face depth image based on the speckle image of the face; performing liveness detection based on the face depth image and the second infrared image of the face; performing feature fusion on at least two of the color image of the face, the second infrared image of the face, and the face depth image to obtain a fused feature vector; and performing face recognition on the object to be recognized using the fused feature vector.

[0008] According to some embodiments, the first face infrared image processing includes: determining whether a face bounding box is detected in the first face infrared image; if so, generating a second face infrared image based on the face bounding box; obtaining exposure parameters of the second face infrared image that meet a threshold condition, and processing the second face infrared image with the exposure parameters; if not, processing the first face infrared image with preset exposure parameters.

[0009] According to some embodiments, obtaining exposure parameters of the second face infrared image that meet threshold conditions, and processing the second face infrared image with the exposure parameters, includes: determining whether the grayscale of the second face infrared image reaches a preset grayscale threshold; if yes, using the first exposure parameter of the second face infrared image as the exposure parameter of the next frame image; if no, obtaining a second exposure parameter based on the second face infrared image and the first exposure parameter, and using the second exposure parameter as the exposure parameter of the next frame image.

[0010] According to some embodiments, obtaining a second exposure parameter based on the second face infrared image and the first exposure parameter includes: inputting the second face infrared image and the first exposure parameter into an automatic exposure value prediction network; and obtaining the second exposure parameter output by the automatic exposure value prediction network.

[0011] According to some embodiments, generating a face depth image based on the face speckle image includes: inputting the face speckle image and a preset speckle reference image into a depth imaging network; acquiring the face depth image output by the depth imaging network; and performing data normalization processing on the face depth image.

[0012] According to some embodiments, liveness detection based on the second face infrared image of the face depth image includes: performing data normalization processing on the second face infrared image; merging the second face infrared image and the face depth image into arrays along the channel direction after data normalization processing; inputting the data generated by the array merging into a convolutional computing network; obtaining the calculation result output by the convolutional computing network; and performing normalization processing on the calculation result output by the convolutional computing network to obtain the result of liveness detection.

[0013] According to some embodiments, feature fusion is performed on at least two images among the color face image, the second infrared face image, and the face depth image to obtain a fused feature vector, including: obtaining an infrared face-aligned image, a color face-aligned image, and a depth face-aligned image of the second infrared face image; generating a first feature vector based on the infrared face-aligned image; generating a second feature vector based on the color face-aligned image; generating a third feature vector based on the depth face-aligned image; and weightedly fusing at least two of the first feature vector, the second feature vector, and the third feature vector to obtain the fused feature vector.

[0014] According to some embodiments, generating a first feature vector based on the infrared face alignment image includes: inputting the infrared face alignment image into an infrared face feature extraction network to obtain the first feature vector.

[0015] According to some embodiments, generating a second feature vector based on the color face alignment image includes: inputting the color face alignment image into a color face feature extraction network to obtain the second feature vector.

[0016] According to some embodiments, generating a third feature vector based on the deep face alignment image includes: inputting the deep face alignment image into a deep face feature extraction network to obtain the third feature vector.

[0017] According to some embodiments, performing face recognition on the identified object using the fused feature vector includes: comparing the fused feature vector with face feature vectors in a face feature database; and determining whether the identified object is a person in the face feature database based on the comparison result.

[0018] According to one aspect of this application, a face recognition device is provided, comprising: a data acquisition unit for acquiring a color image of a face of an object to be recognized, a first infrared image of a face, and a speckle image of a face; a data processing unit for sequentially performing liveness detection and face recognition on the object to be recognized based on the color image of the face, the first infrared image of the face, and the speckle image of the face, and obtaining liveness detection results and face recognition results; and a data storage unit for storing the image data of the data acquisition unit and the data processing unit.

[0019] According to some embodiments, the data acquisition unit includes: a multi-channel camera, including a wide-angle multi-channel camera, for acquiring the color image of the face of the identified object, the first infrared image of the face, and the speckle image of the face; and a speckle projector for generating the speckle image of the face.

[0020] According to some embodiments, the data processing unit includes: an automatic exposure module, which performs face detection on the first face infrared image to obtain a second face infrared image, and performs exposure processing on the second face infrared image; a liveness detection module, which generates a face depth image based on the face speckle image, and performs liveness detection based on the face depth image and the exposed second face infrared image; and a face recognition module, which performs feature fusion on the face color image and the second face infrared image to obtain a fused feature vector, which is used to compare with the feature vector of a face feature library to generate the face recognition result.

[0021] According to one aspect of this application, a smart lock is provided, the smart lock including the face recognition device as described above.

[0022] According to one aspect of this application, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the method as described above.

[0023] According to the embodiments of this application, under the premise of effectively ensuring the quality of face images by automatically acquiring exposure parameters, a liveness detection scheme and face recognition strategy based on multimodal fusion information are adopted to effectively resist intrusions of various methods and ensure the accuracy of face recognition results.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application.

[0026] Figure 1 A flowchart illustrating a face recognition method according to an example embodiment of this application is shown.

[0027] Figure 2 A flowchart illustrating the infrared image exposure processing of a face according to an example embodiment of this application is shown.

[0028] Figure 3 This diagram illustrates the workflow of an automatic exposure value prediction network model according to an example embodiment of this application.

[0029] Figure 4 A flowchart illustrating the workflow of a depth imaging network model according to an example embodiment of this application is shown.

[0030] Figure 5 A flowchart illustrating the liveness detection process according to an example embodiment of this application is shown.

[0031] Figure 6 A flowchart illustrating the acquisition of a fused feature vector for face recognition according to an example embodiment of this application is shown.

[0032] Figure 7 A block diagram of a face recognition device according to an example embodiment of this application is shown.

[0033] Figure 8 A schematic diagram of a smart door lock according to an example embodiment of this application is shown.

[0034] Figure 9 A block diagram of an electronic device according to an example embodiment of this application is shown. Detailed Implementation

[0035] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this application will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0036] The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of these specific details, or other methods, components, materials, apparatus, or operations may be employed. In these cases, well-known structures, methods, apparatuses, implementations, materials, or operations will not be shown or described in detail.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0039] This application provides a high-performance, high-efficiency, and high-accuracy anti-spoofing face recognition and liveness detection method, device, and smart lock that utilizes a multi-channel camera and speckle projector. Through a multi-modal information fusion scheme, it effectively resists the intrusion of images captured by various non-live faces and ensures the accuracy of face recognition.

[0040] The following will describe in detail, with reference to the accompanying drawings, a face recognition method, device, smart door lock, and computer program product according to embodiments of this application.

[0041] This application uses the following terms:

[0042] All-pass camera: A camera that can receive light in the infrared band, visible light band and more bands is called an all-pass camera.

[0043] Dual-channel camera: A camera that can receive both infrared and visible light wavelengths is called a dual-channel camera.

[0044] Multi-channel cameras: including full-channel and dual-channel cameras, which can simultaneously acquire infrared and color images.

[0045] Figure 1 A flowchart illustrating a face recognition method according to an example embodiment of this application is shown.

[0046] like Figure 1 As shown, in step S110, the face recognition device acquires a color image of the face of the object to be recognized, a first infrared image of the face, and a speckle image of the face.

[0047] For example, a face recognition device includes a multi-channel camera and a speckle projector. The multi-channel camera generates a color image of the face, a first infrared image of the face, and a speckle image of the face. The speckle projector projects speckle light spots into space, and the multi-channel camera receives images of the speckle light spots reflected from the surface of an object to generate the speckle image.

[0048] In step S120, the face recognition device performs face detection on the first face infrared image and obtains the second face infrared image based on the face detection result.

[0049] For example, in step S120, the face recognition device uses a face detection algorithm to perform face detection on the first face infrared image acquired by the multi-channel camera, confirms the face bounding box in the first face infrared image, and generates a second face infrared image based on the face bounding box.

[0050] The face recognition device determines whether the grayscale of the second face infrared image reaches a preset grayscale threshold, and performs exposure processing on the second face infrared image based on the grayscale determination result.

[0051] In step S130, the face recognition device generates a face depth image based on the face speckle image.

[0052] For example, in step S130, the face recognition device acquires a face speckle image generated by the speckle projector and a preset speckle reference image, and inputs the face speckle image and the speckle reference image into the depth imaging network. The face recognition device acquires a face depth image output by the depth imaging network and performs data normalization processing on the face depth image.

[0053] Optionally, the face recognition device can also perform pixel offset calculations on the face speckle image based on the speckle reference image, and calculate the face depth image based on the pixel offset calculation results and camera intrinsic and extrinsic parameters through trigonometric function relationships.

[0054] In step S140, the face recognition device performs liveness detection based on the face depth image and the second face infrared image.

[0055] For example, in step S140, the face recognition device performs data normalization processing on the exposed second face infrared image, and then merges the normalized second face infrared image and the face depth image into arrays along the channel direction. The face recognition device inputs the multi-channel data generated after array merging into a convolutional network, obtains the calculation results output by the convolutional network, and performs normalization processing to obtain the liveness detection result of the identified object.

[0056] In step S150, the face recognition device performs feature fusion on the face color image, the second face infrared image, and the face depth image to obtain a fused feature vector.

[0057] For example, in step S150, the face recognition device obtains facial feature points from the second infrared face image, the color face image, and the depth face image using a facial feature point detection algorithm. The face recognition device selects facial key points from these feature points and performs feature corrections on the second infrared face image, the color face image, and the depth face image based on these key points to obtain an infrared face-aligned image, a color face-aligned image, and a depth face-aligned image.

[0058] The face recognition device generates a first feature vector based on an infrared face alignment image, a second feature vector based on a color face alignment image, and a third feature vector based on a depth face alignment image. It then performs weighted fusion of at least two of the first, second, and third feature vectors to obtain a fused feature vector.

[0059] In step S160, the face recognition device performs face recognition on the object by fusing feature vectors.

[0060] For example, in step S160, the face recognition device pre-sets a face feature database and compares the fused feature vector with the face feature vectors in the database. Based on the feature vector comparison result, the face recognition device determines whether the identified object is a person in the face feature database. If the identified object is not a person in the face feature database, the device saves the image data of the identified object and issues a warning message.

[0061] Optionally, the comparison methods for fusing feature vectors and facial feature vectors in the facial feature database include the cosine distance method, Euclidean distance method, Manhattan distance method, and Chebyshev distance method.

[0062] According to embodiments of this application, the face recognition device can automatically predict the exposure parameters of the next frame of the face image, thereby ensuring the quality of the face image. Furthermore, through the fusion of multimodal data, it ensures the effectiveness of liveness detection and the accuracy of face recognition.

[0063] Figure 2 A flowchart illustrating the infrared image exposure processing of a face according to an example embodiment of this application is shown.

[0064] like Figure 2 As shown, step S120 includes steps S121, S122, S123, and S124. In step S121, the face recognition device acquires a first infrared image of a face through a multi-channel camera.

[0065] In step S122, the face recognition device performs face detection on the first face infrared image to determine whether a face frame has been detected.

[0066] For example, if the face recognition device does not detect a face frame in the first face infrared image, the face recognition device uses the preset exposure parameters as the exposure parameters for the first face infrared image.

[0067] If the face recognition device detects a face frame in the second face infrared image, the face recognition device generates a second face infrared image based on the face frame.

[0068] In step S123, the face recognition device judges the grayscale of the second face infrared image to determine whether the grayscale of the second face infrared image reaches the preset grayscale threshold.

[0069] For example, if the grayscale of the second face infrared image reaches a preset grayscale threshold, the face recognition device uses the current first exposure parameter of the second face infrared image as the exposure parameter of the next frame image.

[0070] If the grayscale of the second face infrared image does not reach the preset grayscale threshold, the face recognition device will input the second face infrared image and the first exposure parameter into the exposure value automatic prediction network to obtain the second exposure parameter output by the exposure value automatic prediction network.

[0071] In step S124, the face recognition device uses the second exposure parameter output by the network to automatically predict the exposure value as the exposure parameter for the next frame image.

[0072] According to embodiments of this application, a face recognition device can automatically predict the exposure of face images, effectively improving image quality and mitigating the problem of face images being too dark or overexposed under various conditions.

[0073] Figure 3 This diagram illustrates the workflow of an automatic exposure value prediction network model according to an example embodiment of this application.

[0074] like Figure 3 As shown, step S123 above includes steps S1231, S1232, and S1233. In step S1231, the face recognition device acquires the second face infrared image and the first exposure parameters.

[0075] For example, in step S1231, the face recognition device uses the second face infrared image and the current first exposure parameter y0 of the second face infrared image as the exposure value to automatically predict the data input to the network.

[0076] In step S1232, the face recognition device extracts the grayscale features of the second face infrared image.

[0077] For example, a face recognition device can automatically predict the grayscale features x of a second face infrared image using an exposure value prediction network model.

[0078] In step S1233, the face recognition device processes the data through a convolutional network and outputs the second exposure parameters.

[0079] For example, the model of the automatic exposure prediction network is trained using face images captured in different scenes and the current first exposure parameter y0 as samples, and the second exposure parameter y1 required for the next frame image corresponding to the pre-determined face image as the sample label. The face recognition device processes the grayscale features x of the second face infrared image and the first exposure parameter y0 together through a convolutional network to obtain a mapping relationship, and then outputs the second exposure parameter y1 required for the next frame image through the automatic exposure prediction network.

[0080] Furthermore, the face recognition device uses the second exposure parameter y1 output by the automatic exposure prediction network to perform exposure processing on the second face infrared image. If the grayscale of the second face infrared image after exposure processing is inconsistent with the current frame image, the face recognition device re-inputs the second face infrared image into the automatic exposure prediction network for prediction.

[0081] According to embodiments of this application, the face recognition device uses an automatic exposure prediction network to perform automatic exposure processing on face images, which can ensure the accuracy of exposure parameters in various situations, improve the quality of face images, and is applicable to a wider range of environments.

[0082] Figure 4 A flowchart illustrating the workflow of a depth imaging network model according to an example embodiment of this application is shown.

[0083] like Figure 4 As shown, step S130 includes steps S131, S132, and S133. In step S131, the face recognition device acquires a face speckle image and a speckle reference image.

[0084] For example, a face recognition device generates a speckle image of a face using a speckle projector, while a speckle reference image is pre-generated by acquiring images at fixed distances. The face recognition device then inputs the face speckle image and the speckle reference image into a depth imaging network as input data for the network.

[0085] In step S132, the face recognition device processes the face speckle image through a depth imaging network to obtain a face depth image and output it.

[0086] For example, in step S132, the face recognition device obtains a face disparity map in the depth imaging network based on the face speckle image and the speckle reference image, and then obtains a face depth image based on the disparity map and the camera intrinsic and extrinsic parameters.

[0087] In step S133, the face recognition device performs data normalization processing on the face depth image.

[0088] For example, in step S133, the face recognition device performs data mapping on the face depth image, transforming the face depth image into (0,255) mapping data, and then normalizing the mapping data.

[0089] According to embodiments of this application, the face recognition device directly uses end-to-end face speckle images, which can make full use of the rich information in the face speckle images while saving the CPU resources consumed by depth calculation.

[0090] Figure 5 A flowchart illustrating the liveness detection process according to an example embodiment of this application is shown.

[0091] like Figure 5 As shown, step S140 includes steps S141, S142, S143, and S144. In step S141, the face recognition device acquires the second face infrared image and the face depth image, respectively.

[0092] For example, in step S141, the face recognition device acquires a second face infrared image that has undergone automatic exposure processing, performs data normalization processing on the second face infrared image, and acquires a face depth image that has undergone data normalization processing.

[0093] In step S142, the face recognition device merges the second face infrared image and the face depth image into arrays in the channel direction.

[0094] For example, in step S142, the face recognition device merges the second face infrared image and face depth image, which have undergone data normalization processing, into arrays in two or more channel directions to generate two-channel or multi-channel data.

[0095] The data distribution format in a convolutional network is NCHW, where N represents the number of data points, C represents the data channels, H represents the data height, and W represents the data width. In the embodiments of this application, the face recognition device merges a second face infrared image of a certain size (e.g., 1*1*80*80) and a face depth image of a certain size (e.g., 1*1*80*80) in the channel direction array of the upper convolutional network to obtain a 2-channel data (1*2*80*80), and then inputs the obtained 2-channel data into the subsequent convolutional network.

[0096] In step S143, the face recognition device processes the merged array data using a convolutional network to obtain the convolutional network calculation results.

[0097] For example, in step S143, the face recognition device inputs the 2-channel data generated after merging the arrays into the subsequent convolutional network. After the convolutional network performs calculations, it outputs the binary classification result.

[0098] In step S144, the face recognition device obtains the liveness detection result based on the calculation result of the convolutional network.

[0099] For example, in step S144, the face recognition device normalizes the binary classification calculation results output by the convolutional network through the softmax layer of the convolutional network to obtain prob[0] and prob[1] for binary classification, and prob[0] + prob[1] = 1. Here, 0 and 1 represent the two categories of the recognition object: fake and live. The preset threshold is used to determine whether prob[1] is determined to be a live object, thereby obtaining the liveness determination result of the recognition object.

[0100] If the liveness detection result of the current identified object's face is a fake, the face recognition device terminates the liveness detection process for the current identified object and begins a new liveness detection process. If the liveness detection result of the current identified object's face is a live object, the face recognition device continues the face recognition process.

[0101] According to the embodiments of this application, by combining facial depth images and facial infrared images, the facial recognition device can effectively prevent two-dimensional planar attacks and three-dimensional attacks such as masks or head models, making up for the deficiencies of single infrared or single depth liveness detection data.

[0102] Figure 6 A flowchart illustrating the acquisition of a fused feature vector for face recognition according to an example embodiment of this application is shown.

[0103] like Figure 6 As shown, step S150 includes steps S151, S152, S153, S154, S155, S156, S157, S158, S159, S15A, and S15B. In step S151, the face recognition device acquires a second face infrared image, a face color image, and a face depth image, respectively.

[0104] For example, in step S151, a second infrared image of the face, a color image of the face, and a depth image of the face of the identified object are acquired.

[0105] In step S152, the face recognition device acquires the facial feature points of the second face infrared image.

[0106] For example, in step S152, the face recognition device obtains the positions of multiple face feature points in the second face infrared image through a face feature point detection algorithm, and selects multiple face key points from the multiple face feature points in the second face infrared image.

[0107] According to some embodiments, facial key points can be selected based on the condition that the smallest bounding rectangle between multiple facial key points can cover more than half of the face. For example, if it is necessary to select 5 facial key points, the midpoint of each of the two eyes, the tip of the nose, and the points on both sides of the corners of the mouth can be selected.

[0108] In step S153, the face recognition device performs feature correction on the second face infrared image to obtain an infrared face aligned image.

[0109] For example, in step S153, the face recognition device selects N (e.g., more than 5) facial key points from multiple facial feature points in the second face infrared image, and aligns the facial key points of the second face infrared image with a standard template face. The face recognition device performs matrix calculations based on the coordinates of the facial key points in the second face infrared image, and performs similarity transformations on the second face infrared image using methods such as rotation, translation, and proportional scaling according to the correspondence of the matrix. The transformed image is then matched with the standard template face to obtain an infrared face-aligned image.

[0110] In step S154, the face recognition device acquires the facial feature points of the color face image.

[0111] For example, in step S154, if the first infrared face image and the color face image are acquired at the same time, the face recognition device synchronizes multiple facial feature points from the second infrared face image to the color face image. If the first infrared face image and the color face image are acquired at different times, the face recognition device obtains the positions of multiple facial feature points in the color face image using a facial feature point detection algorithm, and selects multiple facial key points from the multiple facial feature points in the color face image.

[0112] In step S155, the face recognition device performs feature correction on the color face image to obtain a color face aligned image.

[0113] For example, in step S155, the face recognition device selects N (e.g., more than 5) facial key points from multiple facial feature points in the color face image, and aligns the color face image and the facial key points of the color face image to a standard template face. The face recognition device performs matrix calculations based on the coordinates of the facial key points in the color face image, and performs similarity transformations on the color face image using methods such as rotation, translation, and proportional scaling according to the correspondence of the matrix. The transformed image is then matched with the standard template face to obtain a color face aligned image.

[0114] In step S156, the face recognition device acquires the facial feature points of the face depth image.

[0115] For example, in step S156, the face recognition device obtains the positions of multiple face feature points in the face depth image through a face feature point detection algorithm, and selects multiple face key points in the face depth image from the multiple face feature points in the face depth image.

[0116] In step S157, the face recognition device performs feature correction on the face depth image to obtain a depth-aligned face image.

[0117] For example, in step S157, the face recognition device selects N (e.g., more than 5) facial key points from multiple facial feature points in the face depth image, and aligns the face depth image and the facial key points of the face depth image to a standard template face. The face recognition device performs matrix calculations based on the coordinates of the facial key points in the face depth image, and performs similarity transformations on the face depth image using methods such as rotation, translation, and proportional scaling according to the correspondence of the matrix. The transformed image is then matched with the standard template face to obtain a depth-aligned face image.

[0118] In step S158, the face recognition device generates a first feature vector based on the infrared face alignment image.

[0119] For example, in step S158, the face recognition device inputs the infrared face alignment image into the infrared face feature extraction network to obtain the first feature vector A extracted by the infrared face feature extraction network.

[0120] In step S159, the face recognition device generates a second feature vector based on the color face alignment image.

[0121] For example, in step S159, the face recognition device inputs the color face alignment image into the color face feature extraction network to obtain the second feature vector B extracted by the color face feature extraction network.

[0122] In step S15A, the face recognition device generates a third feature vector based on the depth face alignment image.

[0123] For example, in step S15A, the face recognition device inputs the deep face alignment image into the deep face feature extraction network to obtain the third feature vector C extracted by the deep face feature extraction network.

[0124] In step S15B, the face recognition device performs weighted fusion of at least two of the first feature vector, the second feature vector, and the third feature vector to obtain a fused feature vector.

[0125] For example, in step S15B, the face recognition device employs different fusion schemes based on the data quality of the second face infrared image, face color image, and face depth image, including:

[0126] The first fusion scheme involves weighted fusion of the first feature vector A of the second infrared face image and the second feature vector B of the face color image, i.e., A and B.

[0127] The second fusion scheme selects the image with better data quality from the second face infrared image and the face color image, and performs weighted fusion of the feature vector A of the second face infrared image or the feature vector B of the face color image with the third feature vector C of the face depth image, i.e. (A or B) and C.

[0128] The third fusion scheme involves weighted fusion of the first feature vector A of the second face infrared image, the second feature vector B of the face color image, and the third feature vector C of the face depth image, i.e., A, B, and C.

[0129] For example, according to the first fusion scheme, the first feature vector A with a length of m*n (e.g., 1*512) and the second feature vector B with a length of m*n (e.g., 1*512) are weighted and fused. The fusion formula can be (αA, βB) to obtain a fused feature vector with a length of m*2n (e.g., 1*1024), where α is the weight of the first feature vector A and β is the weight of the second feature vector B.

[0130] According to some embodiments, the weights of the feature vectors can be automatically adjusted based on the data quality of the face infrared image, face color image, and face depth image. For example, if the data quality of the face infrared image reaches a preset quality threshold, the weight of the first feature vector A is α = 1; if the data quality of the face color image reaches a preset quality threshold, the weight of the second feature vector B is β = 1; and the fused feature vector is a (αA, βB) vector. The weights of the first feature vector A and the second feature vector B are dynamically adjusted according to their respective image quality.

[0131] According to the embodiments of this application, the face recognition device effectively utilizes the modal data of color face images and infrared face images, extracts features through their respective network models, and adopts a multimodal feature weight fusion strategy to ensure the accuracy of face recognition.

[0132] Figure 7 A block diagram of a face recognition device according to an example embodiment of this application is shown.

[0133] like Figure 7 As shown, the face recognition device 200 includes a data acquisition unit 210, a data processing unit 220, and a data storage unit 230.

[0134] The data acquisition unit 210 includes a multi-channel camera and a speckle projector, and the multi-channel camera can be a wide-angle multi-channel camera. The data acquisition unit 210 acquires a color image of the face of the object to be identified and a first infrared image of the face through the multi-channel camera, and acquires a speckle image of the face through the speckle projector and the multi-channel camera.

[0135] The data processing unit 220 includes an automatic exposure module, which incorporates an automatic exposure value prediction network model. The automatic exposure module performs face detection on the first face infrared image to obtain a second face infrared image, and then performs exposure processing on the second face infrared image using the automatic exposure value prediction network.

[0136] The data processing unit 220 includes a liveness detection module, which incorporates a depth imaging network model and a liveness detection convolutional network model. The liveness detection module acquires a speckle image of a face through the data acquisition unit 210, inputs the speckle image and a preset speckle reference image into the depth imaging network to generate a face depth image, and performs data normalization on the face depth image. The liveness detection module acquires a second face infrared image that has already undergone exposure processing by the automatic exposure module, performs data normalization on the second face infrared image, and inputs the normalized second face infrared image and the face depth image into the liveness detection convolutional network to obtain the convolutional network calculation result. The liveness detection module obtains the liveness detection result based on the convolutional network calculation result.

[0137] The data processing unit 220 includes a face recognition module, which is equipped with an infrared face feature extraction network model, a color face feature extraction network model, and a depth face feature extraction network model. The face recognition module acquires a second infrared face image, a color face image, and a depth face image, and obtains face feature points based on these images. The face recognition module performs feature correction on the second infrared face image, the color face image, and the depth face image based on the facial key points selected from the feature points, generating an infrared face-aligned image, a color face-aligned image, and a depth face-aligned image. The face recognition module inputs the infrared face-aligned image, the color face-aligned image, and the depth face-aligned image into the infrared face feature extraction network, the color face feature extraction network, and the depth face feature extraction network, respectively, to obtain a first feature vector, a second feature vector, and a third feature vector. At least two of the first feature vector, the second feature vector, and the third feature vector are weighted and fused to obtain a fused feature vector. The face recognition module compares the fused feature vector with the face feature vector in the preset face feature database to determine whether the object being recognized is a person in the face feature database.

[0138] The data storage unit 230 is used to store the data acquired and generated during the operation of the data acquisition unit 210 and the data processing unit 220, including data such as face speckle images, face color images, first face infrared images, second face infrared images, face depth images, infrared face aligned images, color face aligned images, and depth face aligned images.

[0139] Figure 8A schematic diagram of a smart door lock according to an example embodiment of this application is shown.

[0140] like Figure 8 As shown, the smart door lock includes a face recognition device 200, a main control device 300, and a door lock device 400.

[0141] The main control device 300 includes a proximity light sensor 310. When an object approaches the smart door lock, the proximity light sensor 310 detects the human body, transmits information to the main control device 300, and triggers the main control device 300.

[0142] The face recognition device 200 includes a multi-channel camera 211 and a speckle projector 212. The multi-channel camera 211 is a wide-angle multi-channel camera. Because it can acquire color image information, it can realize face recognition and liveness detection functions, as well as a smart doorbell function, making the hardware structure of the smart door lock highly integrated and simplified, and reducing hardware costs.

[0143] After the main control device 300 is triggered, the face recognition device 200 is activated. The face recognition device 200 can detect the target when it is at a distance (within the field of view) or at the edge of the field of view using a multi-channel camera 211. When the target enters the field of view of the multi-channel camera 211, the face recognition device 200 performs face detection and tracking, acquiring a first infrared image and a color image of the face. If a face is detected, the face recognition device 200 acquires a second infrared image of the face after automatic exposure processing. When the target approaches the smart lock and enters the field of view of the speckle projector 212 of the face recognition device 200, the face recognition device 200 acquires a speckle image of the face. The face recognition device 200 processes the second infrared image and the speckle image of the face, performing liveness detection and face recognition sequentially.

[0144] After the face recognition device 200 obtains the liveness detection result and the face recognition result, assuming the liveness detection result is a real person and the person is in the category database, it sends an unlockable result to the main control device 300; otherwise, it sends an unlockable result. Based on the unlocking result, if unlockable, the main control device 300 sends a command to the door lock device 400 and controls the door lock device 400 to complete the unlocking operation.

[0145] The smart door lock according to the embodiments of this application not only integrates the peephole and face recognition device, reducing hardware costs, but also enables face detection and facial key point detection when the object is at a distance. When someone approaches, the smart door lock can complete or partially complete the face recognition work in advance, effectively reducing the computing power required at the same time and effectively shortening the time for liveness detection and face recognition, thereby speeding up unlocking.

[0146] Figure 9 A block diagram of an electronic device according to an example embodiment of this application is shown.

[0147] like Figure 9 As shown, the electronic device 600 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0148] like Figure 9 As shown, the electronic device 600 is manifested in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc. The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the methods described in this specification according to the various exemplary embodiments of this application. For example, the processing unit 610 can perform, for example... Figure 1 The method shown.

[0149] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0150] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0151] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0152] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0153] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. The technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0154] Software products may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0155] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0156] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0157] The aforementioned computer-readable medium carries one or more programs, which, when executed by a device, cause the computer-readable medium to perform the aforementioned functions.

[0158] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0159] According to some embodiments of this application, the technical solution of this application can significantly improve the quality of face images and the speed of liveness detection and face recognition. Through the multimodal fusion information scheme, it can effectively resist the intrusion of various non-live face images and ensure the accuracy of face recognition.

[0160] The embodiments of this application have been described in detail above. These descriptions are solely for the purpose of helping to understand the method and core ideas of this application. Furthermore, any changes or modifications made by those skilled in the art based on the ideas of this application, its specific implementation methods, and its application scope, are all within the scope of protection of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A face recognition method, characterized in that, include: Acquire the color image of the face of the object to be identified, the infrared image of the first face, and the speckle image of the face; The first facial infrared image processing includes: Perform face detection on the first infrared image of the face; Obtain the second face infrared image based on the face detection results; A facial depth image is generated based on the facial speckle image; Liveness detection is performed based on the facial depth image and the second facial infrared image; At least two of the face color image, the second face infrared image, and the face depth image are fused to obtain a fused feature vector; Face recognition of the identified object is performed using the fused feature vector; The first facial infrared image processing includes: Determine whether a face bounding box is detected in the first face infrared image; If so, generate the second face infrared image based on the face bounding box; Obtain the exposure parameters of the second face infrared image that meet the threshold conditions, and process the second face infrared image with the exposure parameters; If not, process the first face infrared image with preset exposure parameters; The process of obtaining exposure parameters that satisfy a threshold condition from the second face infrared image and processing the second face infrared image using the exposure parameters includes: Determine whether the grayscale of the second face infrared image reaches a preset grayscale threshold; If so, the first exposure parameters of the second face infrared image are used as the exposure parameters of the next frame image; If not, a second exposure parameter is obtained based on the second face infrared image and the first exposure parameter, and the second exposure parameter is used as the exposure parameter for the next frame image.

2. The method according to claim 1, characterized in that, The second exposure parameters are obtained based on the second face infrared image and the first exposure parameters, including: The second face infrared image and the first exposure parameters are input into the exposure value automatic prediction network; The second exposure parameter is obtained from the exposure value and automatically predicted by the network.

3. The method according to claim 1, characterized in that, Generating a face depth image based on the face speckle image includes: The face speckle image and the preset speckle reference image are input into the depth imaging network; Acquire the face depth image output by the depth imaging network; The facial depth image is subjected to data normalization processing.

4. The method according to claim 1, characterized in that, Liveness detection based on the facial depth image and the second facial infrared image includes: The second face infrared image is subjected to data normalization processing; The second face infrared image and the face depth image, after data normalization processing, are merged into arrays along the channel direction. The data generated by merging the arrays is input into the convolutional computation network; Obtain the calculation results output by the convolutional computation network; The calculation results output by the convolutional computation network are normalized to obtain the liveness detection result.

5. The method according to claim 1, characterized in that, At least two images from the color face image, the second infrared face image, and the face depth image are fused to obtain a fused feature vector, including: Obtain the infrared face alignment image, color face alignment image, and depth face alignment image of the second face infrared image; A first feature vector is generated based on the infrared face alignment image; A second feature vector is generated based on the color face alignment image; Based on the depth-aligned face image, a third feature vector is generated; At least two of the first feature vector, the second feature vector, and the third feature vector are weighted and fused to obtain the fused feature vector.

6. The method according to claim 5, characterized in that, Based on the infrared face alignment image, a first feature vector is generated, including: The infrared face alignment image is input into an infrared face feature extraction network to obtain the first feature vector.

7. The method according to claim 5, characterized in that, Based on the color face alignment image, a second feature vector is generated, including: The color face alignment image is input into a color face feature extraction network to obtain the second feature vector.

8. The method according to claim 5, characterized in that, Based on the deep face alignment image, a third feature vector is generated, including: The deep face alignment image is input into a deep face feature extraction network to obtain the third feature vector.

9. The method according to claim 1, characterized in that, Facial recognition of the identified object using the fused feature vector includes: The fused feature vector is compared with the facial feature vector in the facial feature database; Based on the comparison results, it is determined whether the identified object is a person in the facial feature database.

10. A face recognition device, characterized in that, include: The data acquisition unit acquires a color image of the face of the object to be identified, an infrared image of the first face, and a speckle image of the face; The data processing unit performs liveness detection and face recognition on the identified object sequentially based on the color face image, the first infrared face image, and the face speckle image, and obtains the liveness detection result and the face recognition result. as well as A data storage unit stores the image data from the data acquisition unit and the data processing unit. The data processing unit includes: An automatic exposure module performs face detection on the first face infrared image to obtain a second face infrared image, and performs exposure processing on the first face infrared image or the second face infrared image. The liveness detection module generates a face depth image based on the face speckle image, and performs liveness detection based on the face depth image and the second face infrared image after exposure processing. The face recognition module performs feature fusion on at least two images among the face color image, the second face infrared image, and the face depth image to obtain a fused feature vector, which is used to compare with the feature vector of the face feature library to generate the face recognition result. The automatic exposure module is also used for Determine whether a face bounding box is detected in the first face infrared image; If so, generate the second face infrared image based on the face bounding box; Obtain the exposure parameters of the second face infrared image that meet the threshold conditions, and process the second face infrared image with the exposure parameters; If not, process the first face infrared image with preset exposure parameters; The process of obtaining exposure parameters that satisfy a threshold condition from the second face infrared image and processing the second face infrared image using the exposure parameters includes: Determine whether the grayscale of the second face infrared image reaches a preset grayscale threshold; If so, the first exposure parameters of the second face infrared image are used as the exposure parameters of the next frame image; If not, a second exposure parameter is obtained based on the second face infrared image and the first exposure parameter, and the second exposure parameter is used as the exposure parameter for the next frame image.

11. The apparatus according to claim 10, characterized in that, The data acquisition unit includes: A multi-channel camera, including a wide-angle multi-channel camera, is used to acquire the color image of the face of the identified object, the first infrared image of the face, and the speckle image of the face; A speckle projector is used to generate the speckle image of the face.

12. A smart door lock, characterized in that, The smart door lock includes the face recognition device as described in claim 10 or 11.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the method as described in any one of claims 1-9.

Images