A multimodal face recognition method, device and smart door lock

Abstract

The present application provides a method, device and smart door lock for multimodal face recognition, the method comprising: using the multi-channel camera and the infrared camera to respectively capture images of an object to be recognized within the field of view to obtain a first image and a second image, wherein the first image is captured by the multi-channel camera and includes color image information and infrared image information; the second image is captured by the infrared camera; based on the first image and the second image, key point detection is performed to obtain the facial key point positions of the first image and the second image respectively; based on the obtained facial key point positions, key point matching is performed to obtain a depth image; quality judgment is performed on the first image, the second image and the depth image to obtain a captured image quality judgment result; and multimodal face recognition is performed based on the captured image quality judgment result.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and device for multimodal face recognition and a smart door lock. Background Art

[0002] The development of the smart door lock market is inseparable from the advancement of two core technologies: communications and biometrics. Communications technology influences the efficiency and stability of remote interactions, while biometrics determines the reliability and security of unlocking performance. In the early days of facial recognition smart door locks, most products used 2D facial recognition technology. In recent years, smart door lock manufacturers have turned to another thermal imaging facial recognition technology, namely 3D facial recognition. Summary of the Invention

[0003] According to one aspect of the present application, a multimodal face recognition method is proposed. The face recognition device includes a multi-channel camera, an infrared camera, and a data processing module. The method includes:

[0004] The multi-channel camera and the infrared camera simultaneously capture images of the object to be identified within the field of view to obtain a first image and a second image, wherein the first image is captured by the multi-channel camera and includes color image information and infrared image information; the second image is captured by the infrared camera; based on the first image and the second image, key point detection is performed respectively to obtain the facial key point positions of the first image and the second image; based on the obtained facial key point positions, key point matching is performed to obtain a depth image; quality judgment is performed on the first image, the second image and the depth image to obtain a captured image quality judgment result; and multimodal face recognition is performed based on the captured image quality judgment result.

[0005] According to some embodiments, the method further includes: saving the color image information, infrared image information, and depth image information included in the facial data into a registration library respectively.

[0006] According to some embodiments, in the method, multimodal face recognition is performed based on the quality judgment result of the acquired image, including: taking the first image, the second image and the depth image as feature extraction objects; setting confidence levels based on the quality judgment result of the acquired image; and allocating corresponding feature participation ratios to the first image, the second image and the depth image based on the confidence levels to perform feature extraction on the feature extraction objects.

[0007] According to some embodiments, in the method, performing multimodal face recognition based on the collected image quality judgment result includes: performing light detection on the application environment; and selecting the first image and the second image, or selecting the second image and the depth image as feature extraction objects based on the result of the light detection, to perform feature extraction.

[0008] According to some embodiments, in the method, performing multimodal face recognition based on the collected image quality judgment result also includes: inputting the feature extraction object into a color recognition network, an infrared recognition network and a 3D recognition network respectively to perform feature extraction; wherein the color feature extracted by the color recognition network is the first recognition feature, the infrared feature extracted by the infrared recognition network is the second recognition feature, and the 3D feature extracted by the 3D recognition network is the third recognition feature.

[0009] According to some embodiments, in the method, the registration library stores color image information, infrared image information, and depth image information contained in the facial data, and performing multimodal face recognition based on the collected image quality judgment result further includes:

[0010] If the collected image quality judgment result meets the first threshold condition, the first identification feature is selected to be compared with the color image information in the registration library to obtain a first comparison value, and the second identification feature is selected to be compared with the infrared image information in the registration library to obtain a second comparison value; the first comparison value and the second comparison value are fused and calculated to obtain an identification comparison value; if the collected image quality judgment result meets the second threshold condition, the second identification feature is selected to be compared with the infrared image information in the registration library to obtain a third comparison value, and the third identification feature is selected to be compared with the depth image information in the registration library to obtain a fourth comparison value; the third comparison value and the fourth comparison value are fused and calculated to obtain an identification comparison value; if the identification comparison value is less than the threshold, it is determined that the identification is passed.

[0011] According to some embodiments, in the method, the registration library stores color image information, infrared image information and depth image information contained in the facial data. After the determination is that the recognition is successful, the method further includes: using the first recognition feature to update the color image information in the registration library; using the second recognition feature to update the infrared image information in the registration library; and using the third recognition feature to update the depth image information in the registration library.

[0012] According to some embodiments, in the method, performing multimodal face recognition based on the collected image quality judgment result also includes: inputting the first image and the second image into an infrared color recognition network for feature extraction; inputting the second image and the depth image into an infrared 3D recognition network for feature extraction; wherein the fusion feature extracted by the infrared color recognition network is the fourth recognition feature, and the fusion feature extracted by the infrared 3D network is the fifth recognition feature.

[0013] According to some embodiments, in the method, the registration library stores color image information, infrared image information and depth image information contained in the facial data, and the multimodal face recognition is performed based on the collected image quality judgment result, and also includes: if the collected image quality judgment result meets the third threshold condition, selecting the fourth identification feature and the fourth identification feature in the registration library for comparison to obtain an identification comparison value; if the collected image quality judgment result meets the fourth threshold condition, selecting the fifth identification feature and the fifth identification feature in the registration library for comparison to obtain an identification comparison value; if the identification comparison value is less than the threshold, it is determined that the recognition is passed.

[0014] According to some embodiments, in the method, the registration library stores color image information, infrared image information and depth image information contained in the facial data. After the determination is that the recognition is successful, the method further includes: updating the fourth recognition feature and the fifth recognition feature to the registration library.

[0015] According to another aspect of the present application, a device for multimodal face recognition is provided, comprising: a multi-channel camera for acquiring a multi-channel image of a face, the multi-channel image comprising color image information and infrared image information; an infrared camera for acquiring an infrared image of the face; and a data processing module for processing the full-image image and the infrared image according to any of the above methods to perform multimodal face recognition. The multi-channel camera comprises a wide-angle multi-channel camera.

[0016] According to another aspect of the present application, a smart door lock is provided, comprising the device as described above.

[0017] According to another aspect of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the above methods when executing the computer program.

[0018] According to another aspect of the present application, a computer program product is provided, comprising a computer program or instructions, which implement any one of the above methods when executed by a processor.

[0019] According to an example embodiment of the present application, through an adaptive multimodal recognition scheme, when there is sufficient light and the color information is valid, face recognition is performed using color information and infrared information images; when there is insufficient light, face recognition is performed using infrared information and 3D information.

[0020] It should be understood that the foregoing general description and the following detailed description are merely illustrative and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0022] Figure 1 A block diagram of an apparatus for multimodal face recognition according to an exemplary embodiment of the present application is shown.

[0023] Figure 2 A flowchart of a method for multimodal face recognition according to an exemplary embodiment of the present application is shown.

[0024] Figure 3 A flowchart of a method for adaptive multimodal face recognition according to an exemplary embodiment of the present application is shown.

[0025] Figure 4 A flowchart of a method for adaptive multimodal face recognition according to another example embodiment of the present application is shown.

[0026] Figure 5 A flow chart of a registration library updating method according to an exemplary embodiment of the present application is shown.

[0027] Figure 6 The following is a general flow chart of liveness detection and face recognition according to an exemplary embodiment of the present application.

[0028] Figure 7 A block diagram of an electronic device according to an exemplary embodiment is shown. DETAILED DESCRIPTION

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0030] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0033] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below could be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.

[0034] Those skilled in the art will understand that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.

[0035] Currently on the market, mature facial recognition door locks mainly include three methods: structured light solution, TOF solution and binocular solution.

[0036] Structured light solution: It is mainly based on a light projector to obtain speckle images to calculate the 3D information of the face.

[0037] TOF (Time of Flight) solution: It mainly collects multi-phase infrared images within a period and uses phase difference to calculate the 3D information of the face.

[0038] The above two solutions obtain 3D facial information through calculation and then use it for face liveness detection and recognition. However, the inventor believes that both solutions are relatively costly.

[0039] The binocular solution uses a high-definition color camera and a high-definition infrared camera to perform camera calibration and depth calculation to achieve face recognition. However, the inventors discovered that this solution has difficulty obtaining effective information for depth calculation during nighttime recognition due to insufficient information from the color camera.

[0040] To address at least one of the aforementioned issues, the inventors propose a multimodal face recognition method and device, as well as a smart door lock. The recognition device utilizes a multi-channel camera and a conventional single-infrared camera. The multi-channel camera captures a multi-channel image (color information + infrared information), while the infrared camera captures an infrared image. Based on the multi-channel and infrared images, the recognition device calculates local 3D information, combining the infrared, color, and 3D information to perform multimodal face recognition.

[0041] This application uses a single infrared camera and a multi-channel camera to capture images and calculate 3D information about the facial region based on facial key point detection. It also proposes adaptive multimodal recognition using color, infrared, and 3D information. This application reduces the cost of traditional recognition solutions and achieves a uniform recognition range, speed, and accuracy.

[0042] The following describes exemplary embodiments of the present application with reference to the accompanying drawings.

[0043] Figure 1 A block diagram of an apparatus for multimodal face recognition according to an exemplary embodiment of the present application is shown.

[0044] According to some embodiments, a multimodal face recognition apparatus includes: a multi-channel camera 101 , an infrared camera 102 , and a data processing module 103 .

[0045] The multi-channel camera 101 can obtain color image information and infrared image information. The infrared camera 102 can obtain infrared image information. The multi-channel camera 101 can be a full-channel camera or a dual-channel camera.

[0046] A full-pass camera can receive all bands, and a dual-pass camera can receive visible light and infrared bands for imaging. The received bands may include: narrowband single band (infrared band, such as 850nm / 940nm, etc.) and wide-area band (visible light band).

[0047] Multi-channel camera 101 and infrared camera 102 perform binocular matching to obtain local 3D information of the target. During face recognition, the information obtained by multi-channel camera 101 and infrared camera 102 is used to detect facial key points, then perform facial key point matching and calculate 3D information of the facial area, thereby obtaining 3D information of the local face.

[0048] The data processing module 103 is used to process the information obtained by the multi-channel camera 101 and the infrared camera 102. The data processing module 103 performs face recognition by analyzing the images obtained by the multi-channel camera 101 and the infrared camera 102.

[0049] The device can perform multimodal face recognition and liveness detection based on the obtained color information, infrared information and 3D information.

[0050] Figure 2 A flowchart of a method for multimodal face recognition according to an exemplary embodiment of the present application is shown.

[0051] See also Figure 2 In step S201, the multi-channel camera and the infrared camera simultaneously capture images of the object to be identified within the field of view to obtain a first image and a second image. The first image is captured by the multi-channel camera and includes color image information and infrared image information; the second image is captured by the infrared camera.

[0052] According to some embodiments, if the multi-channel camera has a normal field of view, when the object to be identified (i.e., a human face) enters the field of view of the multi-channel camera and the single infrared camera (the field of view can generally be determined by whether it is close to the light sensor to sense whether the object to be identified enters the set range), in step S201, the multi-channel camera and the single infrared camera respectively obtain images.

[0053] For another example, if the multi-channel camera has a large field of view (the field of view is larger than that of a single infrared camera, and the following description takes a wide-angle multi-channel camera as an example), when the object to be identified enters the field of view of the wide-angle multi-channel camera (relatively long distance), in step S201, the wide-angle multi-channel camera works to perform face detection, tracking and key point detection; the wide-angle multi-channel camera has a large field of view, so the recognition range is wide.

[0054] When an object to be identified approaches, the single infrared camera activates and captures an infrared image. The wide-angle, multi-channel camera captures an image containing both color and infrared information as the first image. The single infrared camera captures the infrared image as the second image.

[0055] In step S203, based on the first image and the second image obtained in step S201, key point detection is performed to obtain the facial key point positions of the first image and the facial key point positions of the second image.

[0056] The recognition device detects facial key points based on images captured by a multi-channel camera, and also detects facial key points based on images captured by a common infrared camera (i.e., a single infrared camera). The recognition device detects feature points in the facial region captured by the multi-channel camera and the single infrared camera, respectively. Based on the feature point information, the recognition device performs local calibration of the facial region and calculates 3D information to obtain 3D information of the facial region, which can be subsequently used for face recognition and liveness detection.

[0057] According to some embodiments, in step S203, the recognition device applies a deep learning neural network to perform face detection on the collected color image information and infrared image information respectively; if a face is detected, the image containing the face part can be cropped from the infrared image as infrared face area information, in preparation for further face recognition.

[0058] In step S205 , the recognition device performs key point matching based on the positions of facial key points obtained in step S203 to obtain a depth image.

[0059] The recognition device matches key points based on the positions of key points of the face detected by the multi-channel camera and the single infrared camera images. In addition, the recognition device calculates the local depth 3D information of the face area to obtain the depth information 3D information of the face area.

[0060] When the multi-channel camera has a large field of view, the multi-channel camera and the single infrared camera can perform face detection and facial key point detection at different times, which can reduce the computing power in the same period.

[0061] In step S207 , quality judgment is performed on the first image, the second image, and the depth image to obtain a captured image quality judgment result.

[0062] According to some embodiments, in step S207, an image quality detection model is applied to perform quality judgment on the first image, the second image, and the depth image to obtain a captured image quality judgment result.

[0063] For example, the recognition device detects the light in the application environment and makes an adaptive judgment.

[0064] When the ambient light is sufficient (such as during the day or when the lighting conditions are good), the color image and infrared image quality are good, so the color information and infrared information are preferentially selected and sent to the multimodal face recognition module for feature extraction.

[0065] When the ambient light is not ideal (for example, at night), the color image quality is not high, but the infrared image and depth image are less affected by light and have better quality. Therefore, the recognition device selects the infrared information and depth information and sends them to the multimodal face recognition module for feature extraction.

[0066] The purpose of image quality detection is to determine the quality of color information. Ambient light detection is one way to determine this. This can be achieved by determining the brightness of the RGB channels in the first image to see if it is within a set threshold, thereby determining whether the ambient light is ideal and the quality of the color information. Another way is to use an image quality detection network to make the determination.

[0067] Adaptive image quality assessment can also be implemented in another way. Each time the camera is activated, the recognition device simultaneously collects three modal data types: color, infrared, and 3D information. The recognition device then assesses the quality of the input image information and assigns different confidence levels to each modal data type. Based on the confidence level, the modal data is assigned different feature contributions, adaptively prioritizing the two modal data types for recognition.

[0068] The confidence level determines the contribution of each image element to the subsequent multimodal face recognition process. A higher confidence level results in a higher contribution, which in turn improves face recognition accuracy.

[0069] For example, when the quality of the color information is good, the recognition device can determine that it is daytime or the door lock environment is in good lighting, which is considered condition one; when the quality of the color information is poor, the recognition device can determine that it is nighttime or the door lock environment is in unideal lighting, which is considered condition two.

[0070] In conditions with sufficient light and valid color information, the recognition device uses both color information and infrared images for face recognition. In conditions with insufficient light, the recognition device uses infrared information and 3D information for face recognition. This adaptive multimodal recognition solution can improve recognition accuracy.

[0071] In step S209, multimodal face recognition is performed based on the collected image quality judgment result.

[0072] The registration database stores the color image information, infrared image information, and depth image information contained in the face data. The recognition device performs adaptive multimodal face recognition based on the collected image quality judgment result, which can be implemented in two specific ways.

[0073] The first implementation method includes the following contents.

[0074] The recognition device inputs the feature extraction object into the color recognition network, infrared recognition network, and 3D recognition network for feature extraction. The color feature extracted by the recognition device via the color recognition network is the first recognition feature. The infrared feature extracted by the recognition device via the infrared recognition network is the second recognition feature. The 3D feature extracted by the recognition device via the 3D recognition network is the third recognition feature. Recognition features are the results extracted by the learning network and are generally represented as feature vectors of a certain dimension.

[0075] If the recognition device determines that the collected image quality judgment result meets the first threshold condition (for example, meets condition one previously judged by the light judgment module), it selects to compare the first recognition feature with the color image information in the registration library to obtain a first comparison value; compare the second recognition feature with the infrared image information in the registration library to obtain a second comparison value; and fuse the first comparison value and the second comparison value to obtain an identification comparison value.

[0076] If the recognition device determines that the collected image quality judgment result meets the second threshold condition (for example, it meets the second condition judged by the light judgment module before), it chooses to compare the second recognition feature with the infrared image information in the registration library to obtain a third comparison value; compare the third recognition feature with the depth image information in the registration library to obtain a fourth comparison value; and fuse the third comparison value and the fourth comparison value to obtain the recognition comparison value.

[0077] If the recognition device determines that the recognition comparison value is less than the threshold, it is determined that the recognition is successful.

[0078] The second implementation method is as follows.

[0079] The recognition device inputs the first and second images into an infrared color recognition network for feature extraction; and inputs the second image and depth image into an infrared 3D recognition network for feature extraction. The fused features extracted by the recognition device through the infrared color recognition network serve as the fourth recognition feature. The fused features extracted by the recognition device through the infrared 3D network serve as the fifth recognition feature.

[0080] If the recognition device determines that the collected image quality judgment result meets the third threshold condition, the fourth recognition feature is selected to be compared with the fourth recognition feature in the registration library to obtain a recognition comparison value.

[0081] If the recognition device determines that the collected image quality judgment result meets the fourth threshold condition, the fifth recognition feature is selected to be compared with the fifth recognition feature in the registration library to obtain a recognition comparison value.

[0082] If the recognition device determines that the recognition comparison value is less than the threshold, it is determined that the recognition is successful.

[0083] The following Figure 3 and Figure 4 Two implementation methods of adaptive multimodal face recognition are described as examples.

[0084] Figure 3 A flowchart of a method for adaptive multimodal face recognition according to an exemplary embodiment of the present application is shown.

[0085] According to some embodiments, the application side deploys three recognition networks: an infrared recognition network, a color recognition network, and a 3D recognition network. When a user registers, for each registered user, if the recognition device determines that the three modal data are all good, the facial image data of each registered user is sent to the corresponding infrared recognition network, color recognition network, and 3D recognition network for feature extraction. The extracted color feature is the first recognition feature of this person in the registration library; the extracted infrared feature is the second recognition feature of this person in the registration library; and the extracted 3D feature is the third recognition feature of this person in the registration library. The recognition device stores the extracted recognition feature one, recognition feature two, and recognition feature three in the registration library.

[0086] See also Figure 3 In step S301, during face recognition, during daytime or when the ambient light is sufficient (i.e., condition one), the recognition device inputs the color information of the subject to be recognized into the color recognition network and extracts the first recognition feature. The extracted first recognition feature is then compared with all first recognition features in the registration database to obtain a comparison value of one. The registration database stores the first recognition features of all registered users. When a user is identified, the recognition device compares the extracted first recognition feature with the first recognition features of all users in the registration database, finds the one with the highest similarity, and calculates the comparison value of one.

[0087] Similarly, the recognition device inputs the infrared information of the object to be recognized into the infrared recognition network and extracts the second recognition feature; compares the extracted second recognition feature with all the second recognition features in the registration library to obtain a comparison value two.

[0088] In step S303, the recognition device inputs the first and second comparison values ​​into a comparison fusion module. Based on the image quality detection module's detection and judgment of each modality data, for example, the quality detection module scores each modality data to obtain a quality score for each modality data. Based on the quality score for each modality data, the comparison values ​​are fused and the final output is the recognition comparison value. If the output recognition comparison value meets the corresponding threshold condition, the recognition device deems the recognition object to have passed recognition.

[0089] During the face recognition process, at night or when the ambient light is not ideal, that is, condition two (not shown in the figure), the process is similar to condition one of this implementation method. The recognition device inputs the infrared information of the object to be identified into the infrared recognition network and extracts the second recognition feature; the extracted second recognition feature is compared with all the second recognition features in the registration library to obtain a comparison value two. The recognition device performs a 3D recognition network on the 3D information of the object to be identified and extracts the third recognition feature; the extracted third recognition feature is compared with all the third recognition features in the registration library to obtain a comparison value three. The recognition device inputs the second comparison value and the third comparison value into the comparison fusion module, and performs comparison value fusion on the quality score of each modal data according to the image quality detection module, and finally outputs the comparison value as the recognition comparison value. If the output recognition comparison value meets the corresponding threshold condition, the recognition device deems that the object to be identified has passed the recognition.

[0090] Figure 4 A flowchart of a method for adaptive multimodal face recognition according to another example embodiment of the present application is shown.

[0091] According to some embodiments, the application deploys two fusion recognition networks: an infrared color recognition network and an infrared 3D recognition network. During user registration, for each registered user, if the recognition device determines that both modal data are good, the infrared and color facial data of each registered user are fed into the infrared color recognition network for feature extraction. The extracted fusion feature becomes Feature 4 of the person in the registration database. The infrared and 3D facial data are fed into the infrared 3D recognition network for feature extraction. The extracted fusion feature becomes Feature 5 of the person in the registration database. The recognition device stores the extracted recognition features 4 and 5 in the registration database.

[0092] See also Figure 4 In step S401, during the face recognition process, when it is daytime or the ambient light is sufficient (i.e., condition 1 in step S207), the recognition device inputs the infrared and color modal data of the object to be recognized into the infrared and color recognition network for feature extraction.

[0093] In step S403, the extracted fourth fusion feature is compared with all fourth identification features in the registration library to obtain a comparison value of four; if the comparison value of four meets the corresponding threshold condition, the recognition device regards that the identification object has been identified.

[0094] During face recognition at night or in low-light conditions, i.e., condition 2 (not shown), the process is similar to condition 1. The recognition device inputs the infrared and 3D modal data of the object to be recognized into the infrared 3D recognition network for feature extraction. The extracted fusion feature, i.e., the fifth recognition feature, is compared with all fifth recognition features in the registration database to obtain a comparison value of five. If the comparison value of five meets the corresponding threshold condition, the recognition device deems the object to be recognized as having been recognized.

[0095] Adaptive multimodal face recognition can also be implemented by sending color information, infrared information and 3D information to the trimodal face recognition network at the same time for feature extraction, and then matching and recognizing with the face modality information in the registration library. The specific method is the same as Figure 3 and Figure 4 The method shown is similar and will not be repeated here.

[0096] Figure 5 A flow chart of a registration and update method for a registry according to an exemplary embodiment of the present application is shown.

[0097] See also Figure 5 The registration library can adopt the client program registration method. The client can be a smart phone, smart tablet, etc. For example, a mobile phone is used as the client, and the color camera of the mobile phone client is used to collect the image of the registered person. After the image quality is detected and judged, the image is transmitted to the application end (such as the door lock end, which includes the recognition device described in this article) to perform data preprocessing operations such as face detection and face key point detection. Finally, the application end extracts the facial area feature information and saves the feature information in the whitelist registration library.

[0098] After a user registers in the registration library, if the user performs an identification action on the application side, the application side can dynamically update the registration library based on the identification result. The dynamic update of the registration library includes the following situations.

[0099] Scenario 1) Face registration is performed using the client. Considering nighttime scenarios, color information may be invalid, or there may be differences between the color information collected by the application and the color information collected by the client (for example, the color information collected by the client is of poor quality, while the color information collected by the application is of better quality). If the application recognizes reliably, the infrared face features, color face features, or 3D face features used by the application during recognition will be updated to the registration database, making the face information in the registration database more consistent with the actual usage scenario. Applications during recognition include, but are not limited to, color face feature matching, infrared face feature matching, 3D feature matching, or fused feature matching, thereby achieving multimodal and cross-modal recognition.

[0100] Scenario 2) When using the client for face registration, a quality score is added to each modal data information during the registration process. If the three modal data cannot all meet the good quality requirements, the modal data with better quality will be registered first. During the next recognition, if the recognition on the application side is reliable, the modal data with lower quality scores in the face modal information in the registration library will be updated to expand the face modal information in the registration library and continuously improve and optimize the face registration library.

[0101] To identify a person, the client collects color information for registration. Upon entry, the client's color image undergoes facial image quality assessment and is then transmitted to the lock for preprocessing, where facial region information is extracted and stored. When a person is identified at the lock, a dynamic registration mechanism can be set up based on the results of adaptive multimodal recognition. When certain conditions are met, the photo in the registration database is replaced with one or more of the following: color information, infrared information, and calculated 3D information captured by the lock's multi-channel wide-angle or infrared camera. This ensures that the feature information in the database is more consistent with the identification scenario, improving recognition rates and enabling cross-modal and multimodal recognition. Furthermore, the wide-angle, multi-channel camera allows for nighttime recognition.

[0102] Figure 6 The following is a general flow chart of liveness detection and face recognition according to an exemplary embodiment of the present application.

[0103] See also Figure 6 In step S601, when a face enters the field of view, the multi-channel camera and the ordinary infrared camera respectively acquire images.

[0104] In step S603, the recognition device performs face detection and key point detection based on the image captured by the multi-channel camera, and performs face detection and key point detection based on the image captured by the ordinary infrared camera.

[0105] In step S605, based on the positions of facial key points detected by the multi-channel camera and the ordinary infrared camera images, the recognition device performs key point matching and calculates the local depth 3D information of the facial area to obtain the depth information 3D information of the facial area and the infrared facial area information.

[0106] In step S607, the recognition device judges the quality of the image, and if it is judged that the recognition threshold condition is met, it enters the adaptive multimodal face recognition module to perform face recognition.

[0107] In step S609, the recognition device performs multimodal liveness detection, and can perform feature fusion on the images captured by the multi-channel camera and the images captured by the infrared camera through a deep learning neural network to obtain a feature fusion result; the feature fusion result is transmitted to the classification network of the neural network for liveness detection.

[0108] If the liveness detection determines that the person is alive, the recognition device will then determine whether he is in the whitelist and perform subsequent operations based on the whitelist determination result.

[0109] According to some embodiments, the above face recognition solution can perform face detection and tracking at a long distance and liveness detection at a close distance.

[0110] According to some embodiments, when the application scenario is a door lock, the door lock may include the above-mentioned face recognition and liveness detection device, and realize the smart door lock function according to the above-mentioned module workflows.

[0111] For smart door lock recognition, you can choose a single multi-channel camera with a larger field of view than a single infrared camera, or a single wide-angle multi-channel camera. The latter can implement a peephole function, combining the functions of the peephole and the door lock.

[0112] The wide-angle, multi-channel camera can detect, track, and identify objects at long distances (within the field of view) and at the edges of the field of view. Once a face enters the field of view, face detection and tracking are performed, along with facial key point detection. As a person approaches the door lock and enters the field of view of the single infrared camera, infrared images are captured for face detection and facial key point detection. Key point matching is performed based on the locations of facial key points detected by the multi-channel camera and the single infrared camera images, and local 3D depth information of the facial area is calculated to obtain 3D depth information for the facial area.

[0113] The use of wide-angle multi-channel cameras can not only realize the integration of cat-eye door locks, but also perform face detection and facial key point detection at a long distance, which can effectively reduce the computing power in the same period of time.

[0114] Using a wide-angle, multi-channel camera as a smart door lock solution effectively leverages the advantages of both color and infrared information. Compared to standard multi-channel cameras, it offers a wider field of view and enables behavioral recognition and analysis. It can also replace traditional peephole cameras, fully utilizing images captured by the wide-angle, multi-channel camera for facial recognition, with the option to detect and upload images of people opening and closing doors or suspicious individuals. If a person not on the whitelist is detected lingering at the door for an extended period, the door lock camera will capture their image and upload it to the mobile client for early warning.

[0115] It should be clearly understood that this application describes how to form and use specific examples, but this application is not limited to any details of these examples. On the contrary, based on the teaching of the content disclosed in this application, these principles can be applied to many other embodiments.

[0116] Those skilled in the art will appreciate that all or part of the steps for implementing the above embodiments may be implemented as a computer program executed by a CPU. When the computer program is executed by the CPU, the program for performing the functions defined in the above method provided herein may be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0117] Furthermore, it should be noted that the aforementioned figures are merely illustrative of the processes included in the methods according to exemplary embodiments of the present application and are not intended to be limiting. It is readily understood that the processes illustrated in the aforementioned figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0118] Through the description of the example embodiments, it is easy for those skilled in the art to understand that the method, device and smart door lock of multimodal face recognition according to the embodiments of the present application have at least one or more of the following advantages.

[0119] According to the example embodiment, images are collected by a single infrared camera and a multi-channel camera, and 3D information of the facial area is calculated based on facial key point detection; it is proposed to use color information, infrared information and 3D information for adaptive multimodal recognition, achieving the unification of recognition range, speed and accuracy.

[0120] According to an example embodiment, a dynamic registration mechanism is set up according to the results of adaptive multimodal recognition during identification at the door lock end. When certain conditions are met, the photos in the registration library are replaced with one or more of the color information, infrared information and calculated 3D information collected by the multi-channel wide-angle camera or infrared camera at the door lock end, so that the feature information in the library is more consistent with the identification scene, the recognition rate is improved, and cross-modal and multimodal recognition is realized. The wide-angle multi-channel camera can also perform identification at night.

[0121] Figure 7 A block diagram of an electronic device according to an exemplary embodiment is shown.

[0122] Refer to the following Figure 7 hereinafter, an electronic device 200 according to this embodiment of the present application is described. Figure 7 The electronic device 200 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0123] like Figure 7 As shown, electronic device 200 is implemented as a general-purpose computing device. Components of electronic device 200 may include, but are not limited to, at least one processing unit 210, at least one storage unit 220, a bus 230 connecting various system components (including storage unit 220 and processing unit 210), a display unit 240, and the like.

[0124] The storage unit stores program codes, and the program codes can be executed by the processing unit 210, so that the processing unit 210 executes the methods described in this specification according to various exemplary embodiments of the present application.

[0125] The storage unit 220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 2201 and / or a cache memory unit 2202 , and may further include a read-only memory unit (ROM) 2203 .

[0126] The storage unit 220 may also include a program / utility 2204 having a set (at least one) of program modules 2205, such program modules 2205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0127] Bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0128] The electronic device 200 can also communicate with one or more external devices 300 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 200, and / or any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 250. Furthermore, the electronic device 200 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 260. The network adapter 260 can communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0129] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software or by combining software with necessary hardware. The technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the embodiment of the present application.

[0130] The software product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0131] Computer-readable storage media may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0132] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0133] Those skilled in the art will appreciate that the modules described above can be distributed in the device according to the description of the embodiment, or can be modified accordingly to be used in one or more devices that are different from the embodiment. The modules of the above embodiment can be combined into one module or further divided into multiple submodules.

[0134] While the exemplary embodiments of the present application have been specifically illustrated and described above, it should be understood that the present application is not limited to the detailed structures, configurations, or implementations described herein; rather, the present application is intended to encompass various modifications and equivalent configurations within the spirit and scope of the appended claims.

Claims

1. A multimodal face recognition method, characterized in that: The face recognition device includes a multi-channel camera, an infrared camera and a data processing module. The method includes: The multi-channel camera and the infrared camera respectively capture images of the object to be identified within the field of view to obtain a first image and a second image, wherein the first image is captured by the multi-channel camera and includes color image information and infrared image information; the second image is captured by the infrared camera; Based on the first image and the second image, key point detection is performed respectively to obtain facial key point positions of the first image and the second image; According to the obtained facial key point positions, key point matching is performed to obtain a depth image; Performing quality judgment on the first image, the second image, and the depth image to obtain a captured image quality judgment result; Multimodal face recognition is performed based on the collected image quality judgment result.

2. The method according to claim 1, characterized in that The method further comprises: The color image information, infrared image information and depth image information contained in the face data are saved in the registration library respectively.

3. The method according to claim 2, characterized in that The performing of multimodal face recognition according to the collected image quality judgment result includes: Taking the first image, the second image and the depth image as feature extraction objects; Setting a confidence level according to the collected image quality judgment result; According to the confidence level, corresponding feature participation ratios are allocated to the first image, the second image, and the depth image to perform feature extraction on the feature extraction object.

4. The method according to claim 2, characterized in that The performing of multimodal face recognition according to the collected image quality judgment result includes: Conduct light detection on the application environment; According to the result of the light detection, the first image and the second image, or the second image and the depth image are selected as feature extraction objects to perform feature extraction.

5. The method according to claim 3 or 4, characterized in that The method of performing multimodal face recognition according to the collected image quality judgment result further includes: Inputting the feature extraction object into a color recognition network, an infrared recognition network, and a 3D recognition network respectively to perform feature extraction; Among them, the color feature extracted by the color recognition network is the first recognition feature, the infrared feature extracted by the infrared recognition network is the second recognition feature, and the 3D feature extracted by the 3D recognition network is the third recognition feature.

6. The method according to claim 5, characterized in that The registration database stores color image information, infrared image information, and depth image information contained in the facial data. The multimodal face recognition is performed based on the collected image quality judgment result, further comprising: If the collected image quality judgment result meets a first threshold condition, selecting to compare the first identification feature with the color image information in the registration library to obtain a first comparison value, and selecting to compare the second identification feature with the infrared image information in the registration library to obtain a second comparison value; Performing a fusion calculation on the first comparison value and the second comparison value to obtain an identification comparison value; If the collected image quality judgment result meets the second threshold condition, selecting to compare the second identification feature with the infrared image information in the registration library to obtain a third comparison value, and comparing the third identification feature with the depth image information in the registration library to obtain a fourth comparison value; Fusing the third comparison value and the fourth comparison value to obtain an identification comparison value; If the recognition comparison value is less than the threshold, it is determined that the recognition is successful.

7. The method according to claim 6, characterized in that The registration database stores color image information, infrared image information, and depth image information included in the face data. After the recognition is determined to be successful, the method further includes: updating the color image information in the registration database using the first identification feature; Using the second identification feature to update the infrared image information in the registration database; The depth image information in the registration library is updated using the third identification feature.

8. The method according to claim 5, characterized in that The method of performing multimodal face recognition according to the collected image quality judgment result further includes: Inputting the first image and the second image into an infrared color recognition network for feature extraction; Inputting the second image and the depth image into an infrared 3D recognition network for feature extraction; Among them, the fusion feature extracted by the infrared color recognition network is the fourth recognition feature, and the fusion feature extracted by the infrared 3D network is the fifth recognition feature.

9. The method according to claim 8, characterized in that The registration database stores color image information, infrared image information, and depth image information contained in the face data, and the multimodal face recognition is performed according to the quality judgment result of the collected image, further comprising: If the collected image quality judgment result meets the third threshold condition, selecting to compare the fourth identification feature with the fourth identification feature in the registration database to obtain an identification comparison value; If the collected image quality judgment result meets the fourth threshold condition, selecting to compare the fifth identification feature with the fifth identification feature in the registration database to obtain an identification comparison value; If the recognition comparison value is less than the threshold, it is determined that the recognition is successful.

10. The method according to claim 9, characterized in that The registration database stores color image information, infrared image information, and depth image information included in the face data. After the recognition is determined to be successful, the method further includes: The fourth identification feature and the fifth identification feature are updated into the registration library.

11. A multimodal face recognition device, characterized in that: include: A multi-channel camera is used to obtain a multi-channel image of a human face, wherein the multi-channel image includes color image information and infrared image information; Infrared camera, used to obtain infrared images of human faces; A data processing module, configured to process the multi-channel image and the infrared image according to the method according to any one of claims 1 to 10, so as to perform multimodal face recognition.

12. The device according to claim 11, characterized in that The multi-channel camera includes a wide-angle multi-channel camera.

13. A smart door lock, characterized in that: Comprising the apparatus of claim 11.

14. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 10 when executing the computer program.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images