Face recognition system and face recognition method

Abstract

This invention discloses a facial recognition system and method. The system is a biosafety system based on multispectral sensing. It uses an imaging module including an infrared thermal sensor unit and a convolutional neural network-based module to detect faces and body temperature. The system authenticates faces and generates an alarm as a warning when excessively high temperatures are detected.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a facial recognition system and a facial recognition method. Background Technology

[0002] Traditionally, with the rapid development of smart devices, the Internet of Things (IoT), and artificial intelligence, facial recognition technology has become an indispensable function on smart devices due to its security and contactless advantages. However, facial recognition systems are highly vulnerable to malicious attacks, posing a significant threat to system security. In response to these attacks, facial recognition spoofing detection has become a crucial component of facial recognition systems.

[0003] Currently, common methods for deceiving facial recognition systems include photo attacks, video attacks, and 3D mask attacks. In recent years, research on face and liveness detection has developed rapidly, yielding many valuable results. Near-infrared cameras, structured light / time-of-flight (ToF) sensors, and light field cameras are also used as auxiliary hardware for facial recognition and anti-spoofing. In recent years, several large-scale influenza and viral outbreaks have spread worldwide. The massive H1N1 influenza outbreak in 2009 and the large-scale COVID-19 outbreak in 2020 have caused incalculable losses to human safety. Fever is one of the symptoms of infection, but in densely populated areas, most people conduct temperature checks on people entering and leaving to screen for infected individuals. Traditional thermometers have significant shortcomings when there is a need for rapid and accurate temperature measurement during sudden outbreaks of influenza or viruses.

[0004] Published U.S. Patent 20070153871, attributed to Jacob Fraden, discloses a system for non-contact rapid screening of febrile human subjects using a thermal imaging camera. Furthermore, the patent also discloses a device for automatically screening febrile individuals using a thermal imaging device. Because this invention provides non-contact screening via a thermal imaging camera, the system is quite expensive.

[0005] Another WIPO patent, 2018058554, belonging to Intel, discloses a technique for detecting facial images, including using a spatial convolutional neural network (CNN) to generate scores for one or more facial images detected by a facial image detector. While this invention is more advanced than previous inventions in using a CNN model for facial image detection, it still cannot authenticate faces under all challenging conditions due to the lack of a temperature sensor for scanning and verifying the face.

[0006] This invention aims to provide an improvement in the field of facial recognition, and more specifically, an improvement in the field of CNN-based facial recognition, but is not limited thereto. Furthermore, this invention proposes a multispectral fusion security system with functions for facial recognition anti-spoofing and human body temperature measurement.

[0007] Therefore, to overcome the shortcomings of existing technologies, a multispectral fusion security system for facial recognition is needed. Moreover, this system primarily uses an RGB camera with an infrared thermal sensor and a CNN-based anti-spoofing module. In view of the foregoing invention, there is a need in the art for a system that overcomes or reduces the aforementioned shortcomings of existing technologies.

[0008] Clearly, many methods and systems have been developed in the prior art to achieve various purposes. Furthermore, even if these inventions may be suitable for the specific purposes stated herein, they may not be suitable for the purposes of this invention as described above. Summary of the Invention

[0009] A facial recognition system and method using an RGB camera and an infrared thermal sensor is disclosed. The infrared thermal sensor is used to measure the temperature distribution of the facial area, and then the system determines whether the face being measured is a fake face or a real face.

[0010] This invention proposes a facial recognition system with functions including facial recognition anti-spoofing and human body temperature measurement. The main functions and modules of the system are shown below:

[0011] 1. Calibration of RGB cameras to infrared thermal sensors (online and offline);

[0012] 2. CNN-based temperature measurement module;

[0013] 3. Anti-spoofing module based on CNN.

[0014] Combining an RGB camera and an infrared thermal sensor for facial recognition anti-spoofing detection is a relatively novel method. Unlike other facial recognition anti-spoofing methods, this method combines facial information from RGB and thermal images and utilizes the analysis of temperature distribution in facial regions. Since different objects have different emissivity of infrared thermal radiation, it is difficult to mimic the temperature distribution of a real face in photos, videos, or 3D masks, which greatly reduces the difficulty of detecting fake faces. Simultaneously, the system provided by this invention can accurately measure the temperature of a human face. During pandemic outbreaks, it can be used for rapid, non-contact fever detection in some areas.

[0015] The main objective of this invention is to provide a bio-safe facial recognition system for face identification. This facial recognition system includes an imaging module and a CNN-based module. The imaging module includes an RGB camera unit and an infrared thermal sensor unit.

[0016] The RGB camera unit includes an offline calibrator and an online calibrator. The offline calibrator uses a calibration board to calibrate the face to generate a first calibrated face. The online calibrator corrects for errors in the first calibrated face to generate a second calibrated face.

[0017] The infrared thermal sensor unit includes a thermal extractor, multiple filters, and a calibrator. The thermal extractor extracts a first thermal image of the face. The multiple filters include temporal and spatial filters. The temporal filter removes noise from the first thermal image to generate a second thermal image. The spatial filter divides the second thermal image into multiple grid regions. The spatial filter identifies the temperature distribution across the multiple grid regions. The calibrator identifies the grid region with the highest temperature among the multiple grid regions and eliminates temperature drift in the highest temperature grid region to generate a calibrated grid region.

[0018] The CNN-based module includes a temperature measurement unit and an anti-spoofing unit. The temperature measurement unit compares the temperature distribution of multiple grid regions with a predefined template to verify the face. The anti-spoofing unit analyzes the calibrated grid regions and a second calibrated face to verify the face.

[0019] Another object of the present invention is to provide a method and system for measuring the temperature of a facial region, comprising the following modules and steps: applying a facial detector to an RGB image to obtain facial bounding box information; then, mapping the facial bounding box to a thermal image using a calibration function; eliminating noise using a temporal filter, a Kalman filter, and a rolling average method; then obtaining a compensated temperature using a spatial filter, the Otsu's algorithm, and a median filter; and finally, compensating the compensated temperature to the core human body temperature, wherein blackbody calibration is used to address the time drift problem, i.e., a blackbody is used in daily life to calibrate the compensation factor.

[0020] Another object of the present invention is to provide a facial recognition system that further verifies a face based on one or more biometric attributes. Further, the biometric attributes include any one of the forehead, lacrimal glands, and neck.

[0021] Another object of the present invention is to provide a facial recognition system including an RGB camera unit to detect one or more features from a face, including gender, glasses, posture, hair cover or mask.

[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings, so that other objects and aspects of the present invention will become clearer.

[0023] To achieve the above and related objectives, the present invention may be implemented in the form shown in the accompanying drawings. However, please note that the drawings are merely illustrative and changes may be made to the specific structure of the drawings and descriptions within the scope of the appended claims.

[0024] Although the present invention has been described above through various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functions described in one or more individual embodiments are not limited to the specific embodiments described above, but can be applied individually or in various combinations to one or more other embodiments of the present invention, whether or not such embodiments are described or whether such features are presented as part of the described implementation. Therefore, the breadth and scope of the present invention should not be limited by any of the exemplary embodiments described above.

[0025] In some cases, the presence of extended terms such as “one or more,” “at least,” “but not limited to,” or other similar phrases should not be interpreted as implying a narrower scope in the absence of such extended terms. Attached Figure Description

[0026] The claims will now be described in conjunction with the accompanying drawings, which will make the object and features of the invention clearer. It should be understood that these drawings depict only typical embodiments of the invention and should not be considered as limiting the scope of the invention. The invention will be described and explained below using the accompanying drawings in conjunction with additional features and details.

[0027] Figure 1 A facial recognition system according to the present invention is shown;

[0028] Figure 2 An imaging module of a facial recognition system is shown;

[0029] Figure 3 The workflow of a human body temperature measurement system for a facial recognition system is shown.

[0030] Figure 4The workflow for online / offline calibration of a sensor in an imaging module is shown;

[0031] Figure 5A The workflow for online / offline calibration of an imaging module is shown;

[0032] Figure 5B A multi-mode calibration board is shown;

[0033] Figure 6 This invention illustrates a rapidly expandable user interface.

[0034] Figure 7 A module based on a convolutional neural network (CNN) is shown;

[0035] Figure 8 A facial recognition method is shown;

[0036] Figure 9 A face recognition method based on multispectral sensing is shown. Detailed Implementation

[0037] In recent years, with the rapid development of smart devices, the Internet of Things (IoT), and artificial intelligence (AI), facial recognition technology has become an indispensable function on smart devices due to its security and contactless advantages. However, facial recognition systems are highly vulnerable to malicious attacks, posing a significant threat to system security. In response to these attacks, facial recognition deception detection has become a crucial component of facial recognition systems.

[0038] Currently, common methods for deceiving facial recognition systems include photo attacks, video attacks, and 3D mask attacks. In recent years, research on face and liveness detection has developed rapidly, yielding many valuable results. Near-infrared cameras, structured light / time-of-flight (ToF) sensors, and light field cameras are also used as auxiliary hardware for facial recognition and anti-spoofing. In this paper, we present a novel facial recognition anti-spoofing method that combines an RGB camera with an infrared thermal sensor. The infrared thermal sensor's temperature measurement function is used to obtain the temperature distribution of the facial area, and a facial temperature distribution model is used to determine whether the tested face is a forged face.

[0039] In recent years, several large-scale influenza and viral outbreaks have spread worldwide. The massive H1N1 influenza outbreak in 2009 and the large-scale COVID-19 outbreak in 2020 have caused incalculable losses to human safety. Fever is one of the symptoms of infection, but in densely populated areas, most people have their temperature checked when entering and leaving to screen for infected individuals. Traditional thermometers have significant shortcomings when there is a need for rapid and accurate temperature measurement during sudden outbreaks of influenza or viruses. In addition to facial recognition anti-spoofing methods, this invention also provides a method for measuring human body temperature that can accurately measure facial temperature by combining an infrared thermal sensor and an RGB camera.

[0040] This invention relates to a facial recognition system, which is a multispectral system using an RGB camera and an infrared thermal sensor. This system can verify user identity, defend against spoofing attacks, and measure the temperature of biometric attributes of specific body parts, including the face, forehead, tear glands, and neck. A system and method for facial recognition using an RGB camera unit and an infrared thermal sensor unit utilizes the temperature measurement function of the infrared thermal sensor to obtain the temperature distribution of the facial area, and then uses a facial temperature distribution model to determine whether the measured face is a fake or a real face. Specifically, the facial temperature distribution model can be used to determine whether the measured face is a fake or a real face.

[0041] This invention proposes a multispectral fusion security system with facial recognition anti-spoofing and human body temperature measurement functions. The main functions and modules of this system include:

[0042] 1. Calibration of RGB Camera to Infrared Thermal Sensor (Online and Offline)

[0043] 2. CNN-based modules

[0044] Combining RGB cameras and infrared thermal sensors for facial recognition and anti-spoofing detection is a relatively novel approach. Unlike other facial recognition anti-spoofing methods, this method combines facial information from RGB and thermal images and utilizes the facial temperature distribution model of this invention to analyze the temperature distribution of the facial region. Since different objects have different infrared thermal emissivity, it is difficult to simulate the temperature distribution of a real face in photos, videos, or 3D masks, which greatly reduces the difficulty of detecting fake faces. Simultaneously, the system provided by this invention can accurately measure the temperature of a human face. During pandemics, it can be used for rapid, non-contact fever detection in some areas.

[0045] Figure 1 A facial recognition system according to the present invention is illustrated. The facial recognition system 100 is a multispectral biosafety-based facial recognition system. The facial recognition system includes an imaging module 102 and a CNN-based module 104. The imaging module 102 includes an RGB camera unit and an infrared thermal sensor unit.

[0046] The RGB camera unit consists of an offline calibrator and an online calibrator. The offline calibrator uses a calibration board to calibrate the face to generate a first calibrated face. The online calibrator corrects for errors in the first calibrated face to generate a second calibrated face.

[0047] The infrared thermal sensor unit includes a thermal extractor, multiple filters, and a calibrator. The thermal extractor extracts a first thermal image of the face. The multiple filters include temporal and spatial filters. The temporal filter removes noise from the first thermal image to generate a second thermal image. The spatial filter divides the second thermal image into multiple grid regions and identifies the temperature distribution within these grid regions. The calibrator identifies the grid region with the highest temperature from the multiple grid regions and eliminates temperature drift within the grid region to generate a calibrated grid region.

[0048] Module 104, based on a CNN, includes a temperature measurement unit and an anti-spoofing unit. The temperature measurement unit compares the temperature distribution of multiple grid regions with a predefined template to verify the face. The anti-spoofing unit analyzes the calibrated grid regions and the second calibrated face to verify the face.

[0049] The facial recognition system in this embodiment is a multispectral fusion security system. Furthermore, the functionality of the multispectral fusion security system is based on facial recognition anti-spoofing and facial temperature measurement.

[0050] In addition, the facial recognition system in this embodiment further verifies the face based on one or more biometric attributes. These biometric attributes primarily include the forehead, but may also include the lacrimal glands or neck.

[0051] The calibration board in the offline calibrator is a multi-mode calibration board. Furthermore, the multi-mode calibration board is a black-and-white calibration board that includes one or more heating elements. These heating elements are used to regulate the temperature of multiple grid areas. Additionally, the multi-mode calibration board forms one or more calibration patterns. These calibration patterns include any one of rectangular, circular, and checkerboard patterns.

[0052] The RGB camera unit is used to examine one or more facial features. These features include gender, glasses, posture, hair veil, or mask. The facial recognition system generates an alarm and verifies the face when the temperature in a calibrated grid area exceeds a predefined threshold.

[0053] Figure 2An imaging module for a facial recognition system is shown. The imaging module 102 includes an RGB camera unit 106 and an infrared thermal sensor unit 112. The RGB camera unit 106 includes an offline calibrator 108 and an online calibrator 110. The offline calibrator 108 calibrates a face using a calibration plate to generate a first calibrated face. The online calibrator 110 corrects errors in the first calibrated face to generate a second calibrated face.

[0054] The calibration board is a multi-mode calibration board. In addition to the traditional monochrome calibration board for the RGB camera unit 106, a new type of calibration board including a heating element is provided. This heating element can regulate the temperature to ensure that the thermal image has significant characteristics in feature detection, matching, and stereo calibration. The calibration pattern can be a square pattern, a rectangular pattern, a circular pattern, or other randomly designed patterns.

[0055] The infrared thermal sensor unit 112 includes a thermal extractor, multiple filters, and a calibrator. The thermal extractor 114 extracts a first thermal image of the face. The multiple filters 116 include a temporal filter 118 and a spatial filter 120. The temporal filter 118 is used to remove noise from the first thermal image to generate a second thermal image. The spatial filter 120 is used to divide the second thermal image into multiple grid regions.

[0056] In addition, the spatial filter 120 identifies the temperature distribution of multiple grid regions. The calibrator 122 identifies the grid region with the highest temperature from the multiple grid regions, thereby eliminating temperature drift in the grid regions to generate calibrated grid regions.

[0057] Online calibration is used to correct errors present in offline calibration, such as white borders in thermal images. The size of the face is modeled as being proportional to the height of the face border. Therefore, the border height is converted to a real-world distance by estimating a linear mapping for reprojection. The RGB camera unit includes an RGB camera. The infrared thermal sensor unit includes a thermal camera.

[0058] This setup allows for online adjustment of three rotation parameters, three translation parameters, the focal length of the RGB camera and the thermal camera, and the depth fitting coefficient. Therefore, to effectively achieve this goal, this embodiment features a user-friendly interface that allows for continuous parameter adjustment via visual image features and a tracking bar. Users can smoothly adjust all parameters without worrying about the actual positional relationship of the target camera (i.e., the RGB camera or the thermal camera).

[0059] This facial recognition system includes an RGB camera unit and an infrared thermal sensor unit. The system can verify user identity, defend against spoofing attacks, and measure the temperature of specific body parts such as the face, forehead, tear ducts, and neck.

[0060] Figure 3 This paper illustrates the workflow of a human body temperature measurement system within a facial recognition system. To measure the temperature of the facial region, the system in this embodiment includes the following modules and steps: Applying a facial detector to an RGB image using an RGB camera unit to obtain facial bounding box information. Mapping the bounding box of the facial region of interest (ROI) to a first thermal image using a calibration function. Removing noise using a temporal filter, Kalman filter, and rolling average method. Obtaining a compensated temperature distribution using a spatial filter, Otsu's algorithm, and median filter. Compensating the temperature using a computational strategy to obtain the core human body temperature. Blackbody calibration is used to address the time drift problem; blackbodies are routinely used for temperature calibration to adjust the compensation factor.

[0061] Temperature compensation is used as a regression method to obtain distance-related compensation coefficients. The input consists of facial information size from an RGB image and a compensated temperature map from a thermal sensor, with the output being the body temperature measured by a forehead thermometer. Curve fitting is performed on a large amount of collected data to obtain the compensation coefficients. Another compensation method is applied to situations where facial attributes such as gender, glasses, hairpiece, posture, and face mask are identified. The figure shows the temperature measurement and alarm system.

[0062] Temperature measurements from infrared thermal sensors are influenced by numerous factors, including their inherent characteristics, temperature drift, noise levels, and the user's own characteristics such as occlusion, posture, age, and gender. External environmental factors, such as ambient temperature and light intensity, also play a role. The inherent characteristics of the sensors primarily contribute to temperature drift and noise levels. To address these factors, temporal and spatial filtering are employed to reduce their impact on the algorithm. Kalman filtering can, to some extent, eliminate temperature drift caused by measurement noise. On the hardware side, blackbody timing is used to calibrate the thermal sensor's compensation coefficient, significantly reducing temperature measurement errors caused by temperature drift over time.

[0063] Regarding factors influencing the tester's own behavior, the main influencing factors are facial occlusion and facial posture. To eliminate the influence of facial occlusion (including glasses, bangs, masks, etc.), we first attempted to use facial key point detection to track the forehead area and obtain the temperature of the forehead. This method eliminated the influence of glasses and masks on temperature measurement, but in the presence of bangs, it can sometimes lead to significant temperature measurement errors. Therefore, we tried dividing the facial area into M x N grid regions and measuring the temperature of each grid separately. This method uses the grid with the highest temperature value as the overall temperature measurement result, which not only eliminates the influence of glasses, masks, and bangs, but also reduces the temperature measurement error to a certain extent.

[0064] Figure 4 This paper illustrates a workflow for online / offline calibration of an imaging module's sensor. It showcases novel concepts for both offline and online calibration. The use of a single calibration board and optimization of the online method make the installation of new devices easier and faster.

[0065] A facial recognition system may include an RGB camera unit and an infrared thermal sensor unit. This system can verify user identity, perform facial detection, defend against spoofing attacks, and measure the temperature distribution of biometric attributes, including specific body parts such as the forehead, tear glands, and neck. This figure illustrates the main modules and workflow of the system in this embodiment.

[0066] According to the workflow, a thermal sensor is used for calibration while the RGB camera detects the face. The detected face is then mapped to the region of interest in the thermal image.

[0067] Figure 5A A workflow for online / offline calibration of an imaging module is illustrated. Multiple methods for extracting single-camera or multi-camera calibration parameters are proposed for the calibration of RGB camera units and infrared thermal sensor units. The RGB camera unit includes an RGB camera, and the infrared thermal sensor unit includes a thermal camera. This invention innovates a multi-mode calibration board. In addition to the traditional black-and-white calibration board for RGB cameras, this invention also provides a novel calibration board including a heating device. This device can regulate the temperature to ensure that thermal data (e.g., a first or second calibration face) or image data (e.g., a first or second thermal image) has significant characteristics in feature detection, matching, and stereo calibration. The calibration can be a square pattern, rectangular pattern, circular pattern, or other randomly designed patterns. Figure 5B Only one pattern formed by a typical calibration plate is shown as an example.

[0068] Online calibration is used to correct errors present in offline calibration, such as white borders in thermal images.

[0069] This method projects two-dimensional points on an RGB image (e.g., a first or second calibrated face) into three-dimensional world coordinates with intrinsic parameters and an estimated depth. The three-dimensional points are then reprojected onto a two-dimensional display plane using extrinsic parameters between the two cameras and intrinsic parameters of the thermal camera. During this process, depth can be estimated using features characterizing the target depth. Features in real-world coordinates and their corresponding projections onto the image plane can be encoded into a regression model. For example, in this case, the size of the face is modeled as being proportional to the height of the face border. Therefore, the border height is converted to a real-world distance by estimating a linear mapping for reprojection.

[0070] This setup allows online adjustment of three rotation parameters, three translation parameters, the focal lengths of the RGB camera and the thermal camera, and the depth fitting coefficients. Therefore, to effectively achieve this, a user-friendly interface was designed, allowing for continuous parameter adjustment through visualizing image features and a tracking bar. Users can smoothly adjust all parameters without worrying about the actual positional relationships of the target cameras in the configuration.

[0071] Figure 5B A multi-mode calibration board is shown. This multi-mode calibration board is an innovative calibration board. In addition to the traditional black-and-white calibration board for RGB cameras, the present invention provides a novel calibration board including a heating device. This device is capable of temperature regulation to ensure that the thermal image has significant characteristics in feature detection, matching, and stereo calibration. The calibration pattern can be a square pattern, a rectangular pattern, a circular pattern, or other randomly designed patterns, such as a checkerboard 158.

[0072] Figure 6 This invention illustrates a rapidly expandable user interface. Thermal images and RGB three-channel data are combined into four-channel data, which is used as input to a convolutional neural network to output facial test information and spoofing test results. The method for preventing facial recognition spoofing involves an RGB camera unit and an infrared thermal sensor unit displaying red 160, green 162, and blue 164 regions. The temperature distribution of the facial region is obtained using the temperature measurement function of the infrared thermal sensor unit, and a facial temperature distribution model is used to determine whether the tested face is a fake face.

[0073] Figure 7 A CNN-based module is shown. The CNN-based module includes a temperature measurement unit 166 and an anti-spoofing unit 168. The temperature measurement unit 166 compares the temperature distribution of multiple grid regions with a predefined template to verify the face. The anti-spoofing unit 168 analyzes the calibrated grid regions and the second calibrated face to verify the face.

[0074] Temperature measurements from infrared thermal sensors are influenced by numerous factors, including their inherent characteristics, temperature drift, noise levels, and the user's own characteristics such as occlusion, posture, age, and gender. External environmental factors, such as ambient temperature and light intensity, also play a role. The inherent characteristics of the sensors primarily contribute to temperature drift and noise levels. To address these factors, temporal and spatial filtering are employed to reduce their impact on the algorithm. Kalman filtering can, to some extent, eliminate temperature drift caused by measurement noise. On the hardware side, blackbody timing is used to calibrate the thermal sensor's compensation coefficient, significantly reducing temperature measurement errors caused by temperature drift over time.

[0075] Regarding factors influencing the tester's own behavior, the main influencing factors are facial occlusion and facial posture. To eliminate the influence of facial occlusion (including glasses, bangs, masks, etc.), we first attempted to use facial key point detection to track the forehead area and obtain the temperature of the forehead. This method eliminated the influence of glasses and masks on temperature measurement, but in the presence of bangs, it can sometimes lead to significant temperature measurement errors. Therefore, we tried dividing the facial area into M x N grid regions and measuring the temperature of each grid separately. This method uses the grid with the highest temperature value as the overall temperature measurement result, which not only eliminates the influence of glasses, masks, and bangs, but also reduces the temperature measurement error to a certain extent.

[0076] This invention proposes two deception detection methods. The first method (e.g.) Figure 8 As shown, deception detection and face detection are performed separately, using a second thermal image of the facial region as input, and a two-level model is trained using a deep learning convolutional neural network. The second method implements face detection and deception detection functions in the same neural network. The thermal image (e.g., a first or second thermal image) and RGB three channels are combined into four-channel data, which is used as input to the convolutional neural network, and outputs face test information and deception test results.

[0077] Figure 8 A face recognition method 200 is illustrated. The method includes: Step 202, detecting a face using an RGB camera unit; Step 204, calibrating the face based on one or more facial features to generate a first calibrated face; Step 206, correcting errors in the first calibrated face to generate a second calibrated face; Step 208, extracting a first thermal image of the face using a thermal extractor; Step 210, removing noise from the first thermal image to generate a second thermal image; Step 212, dividing the second thermal image into multiple grid regions; Step 214, identifying the temperature distribution of the multiple grid regions; Step 216, identifying the grid region with the highest temperature among the multiple grid regions; Step 218, eliminating temperature drift in the grid region with the highest temperature to generate calibrated grid regions; Step 220, comparing the temperature distribution of the multiple grid regions with a predefined template to verify the face; Step 222, analyzing the calibrated grid regions and the second calibrated face to verify the face.

[0078] Figure 9A face recognition method 300 based on multispectral sensing is illustrated. Step 302: Detect a face using an RGB camera unit. Step 304: Calibrate the face based on one or more facial features and correct for calibration errors to generate a calibrated face. Step 306: Extract a first thermal image of the face using a thermal extractor. Step 308: Remove noise from the first thermal image to generate a second thermal image. Step 310: Divide the second thermal image into multiple grid regions and identify the temperature distribution of each grid region. Step 312: Identify the grid region with the highest temperature among the multiple grid regions. Step 314: Eliminate temperature drift in the grid region with the highest temperature to generate a calibrated grid region. Step 316: Compare the temperature distribution of the calibrated grid region with a predefined template to verify the face. Step 318: Analyze the calibrated grid region and the calibrated face to verify the face.

[0079] While various embodiments of the invention have been described above, it should be understood that they are merely exemplary and not limiting. Similarly, the accompanying drawings may depict exemplary architectures or other configurations of the invention, done to aid in understanding the features and functions included in the invention. The invention is not limited to the illustrated exemplary architectures or configurations, but various alternative architectures and configurations can be used to achieve the desired features.

[0080] Although the present invention has been described above through various exemplary embodiments and implementations, it should be understood that the various features, aspects, and functions described in one or more individual embodiments are not limited to the specific embodiments described above, but can be applied individually or in various combinations to one or more other embodiments of the present invention, whether or not such embodiments are described or whether such features are presented as part of the described implementations. Therefore, the breadth and scope of the present invention should not be limited by any of the exemplary embodiments described above.

[0081] In some cases, the presence of extended terms such as “one or more,” “at least,” “but not limited to,” or other similar phrases should not be interpreted as implying a narrower scope in the absence of such extended terms.

Claims

1. A face recognition system for recognizing a face, characterized by, The face recognition system comprises an imaging module and a convolutional neural network based module; The imaging module comprises an RGB camera unit and an infrared thermal sensor unit for detecting the face; The RGB camera unit comprises an offline calibrator for calibrating the face using a calibration plate to generate a first calibrated face and an online calibrator for correcting errors in the first calibrated face to generate a second calibrated face; The infrared thermal sensor unit comprises a thermal extractor for extracting a first thermal image of the face, a plurality of filters including a temporal filter for removing noise from the first thermal image to generate a second thermal image and a spatial filter for dividing the second thermal image into a plurality of grid regions, wherein the spatial filter is configured to identify temperature distribution of the plurality of grid regions, and a calibrator for identifying a grid region with highest temperature among the plurality of grid regions and removing temperature drift of the grid region with highest temperature to generate a calibrated grid region; The convolutional neural network based module comprises a temperature measurement unit for comparing the temperature distribution of the plurality of grid regions with a predefined template to verify the face and a spoofing prevention unit for analyzing the calibrated grid region and the second calibrated face to verify the face.

2. The facial recognition system of claim 1, wherein, The face recognition system comprises a multispectral fusion security system.

3. The facial recognition system of claim 2, wherein, The multispectral fusion security system is configured to implement spoofing prevention and temperature measurement of the face.

4. The facial recognition system of any one of claims 1 to 3, wherein, The face recognition system is further configured to verify the face based on one or more biometric attributes.

5. The facial recognition system of claim 4, wherein, The biometric attributes include any one of a forehead, a tear duct, and a neck.

6. The facial recognition system of any one of claims 1 to 3, wherein, The calibration plate is a multi-modal calibration plate.

7. The facial recognition system of claim 6, wherein, The multi-modal calibration plate is a black and white calibration plate comprising one or more heating devices.

8. The facial recognition system of claim 7, wherein, The one or more heating devices are configured to adjust temperature of the plurality of grid regions.

9. The facial recognition system of claim 7, wherein, The multi-modal calibration plate is configured to form one or more calibration patterns.

10. The facial recognition system of claim 9, wherein, The one or more calibration patterns include any one of a rectangular pattern, a circular pattern, and a chessboard pattern.

11. The facial recognition system of any one of claims 1 to 3, wherein, The RGB camera unit is configured to detect one or more features of the face, the features including one of a gender, glasses, a posture, a hair cover, and a face cover.

12. The facial recognition system of any one of claims 1 to 3, wherein, The face recognition system is configured to generate an alert when temperature of the calibrated grid region is higher than a predefined threshold.

13. The facial recognition system of any one of claims 1 to 3, wherein, The face recognition system is configured to verify a human face.

14. A face recognition system for recognizing a human face, characterized by, The face recognition system comprises an imaging module and a convolutional neural network based module; The imaging module comprises an RGB camera unit for detecting the human face and an infrared thermal sensor unit for extracting a first thermal image of the human face, The RGB camera unit comprises an offline calibrator for calibrating the human face based on one or more features of the human face to generate a first calibrated face and an online calibrator for correcting errors in the first calibrated face to generate a second calibrated face; The infrared thermal sensor unit includes a pair of filters and calibrators, the pair of filters including a temporal filter and a spatial filter, wherein the temporal filter is used to eliminate noise of the first thermal image to generate a second thermal image, and the spatial filter is used to divide the second thermal image into a plurality of grid regions and identify temperature distribution of the plurality of grid regions; the calibrator is used to identify a grid region with the highest temperature in the plurality of grid regions, and eliminate temperature drift of the grid region with the highest temperature to generate a calibrated grid region; The convolutional neural network-based module includes a temperature measurement unit and an anti-fraud unit, the temperature measurement unit is used to compare the temperature distribution of the plurality of grid regions with a predefined template to verify the face; and the anti-fraud unit is used to analyze the calibrated grid region and the second calibrated face to verify the face.

15. A face recognition method characterized by, The method includes: detecting the face by using an RGB camera unit; calibrating the face based on one or more features of the face to generate a first calibrated face; correcting errors in the first calibrated face to generate a second calibrated face; extracting a first thermal image of the face by using a thermal extractor; eliminating noise of the first thermal image to generate a second thermal image; dividing the second thermal image into a plurality of grid regions; identifying temperature distribution of the plurality of grid regions; identifying a grid region with the highest temperature in the plurality of grid regions; eliminating temperature drift of the grid region with the highest temperature to generate a calibrated grid region; comparing the temperature distribution of the plurality of grid regions with a predefined template to verify the face; and analyzing the calibrated grid region and the second calibrated face to verify the face.

16. A face recognition method, characterized by, The method includes: detecting the face by using an RGB camera unit; calibrating the face based on one or more features of the face, and correcting errors in face calibration to generate a calibrated face; extracting a first thermal image of the face by using a thermal extractor; eliminating noise of the first thermal image to generate a second thermal image; dividing the second thermal image into a plurality of grid regions, and identifying temperature distribution of the plurality of grid regions; identifying a grid region with the highest temperature in the plurality of grid regions; eliminating temperature drift of the grid region with the highest temperature to generate a calibrated grid region; analyzing the calibrated grid region and the calibrated face to verify the face; and comparing the temperature distribution of the plurality of grid regions with a predefined template to verify the face. The method includes: detecting the face by using an RGB camera unit; calibrating the face based on one or more features of the face, and correcting errors in face calibration to generate a calibrated face; extracting a first thermal image of the face by using a thermal extractor; eliminating noise of the first thermal image to generate a second thermal image; dividing the second thermal image into a plurality of grid regions, and identifying temperature distribution of the plurality of grid regions; identifying a grid region with the highest temperature in the plurality of grid regions; eliminating temperature drift of the grid region with the highest temperature to generate a calibrated grid region; analyzing the calibrated grid region and the calibrated face to verify the face; and comparing the temperature distribution of the plurality of grid regions with a predefined template to verify the face.

Images