Multispectral face recognition module and method

Abstract

This invention discloses a multispectral face recognition module and method, including a camera assembly and a control assembly. The camera assembly includes an IR camera, a fill light, and an RGB camera. The control assembly includes a main controller, a power management unit, and a storage unit. The IR camera and the RGB camera are connected to the main controller. The IR and RGB images of the captured face are processed by the image processing unit and then sent to the main controller. The main controller performs infrared image liveness detection and color image liveness detection on the IR and RGB images. If both the IR and RGB images meet the liveness criteria, they are then compared with a database for recognition, and the recognition result is output. This invention utilizes the structure of the IR and RGB cameras to capture IR and RGB images of the face, which are then used by the main controller for liveness detection and face recognition, improving recognition accuracy.

Description

Technical Field

[0001] This invention relates to a multispectral face recognition module and method, belonging to the field of face recognition technology. Background Technology

[0002] Facial recognition technology is now widely used across various industries. However, in practical applications, existing facial recognition technologies generally suffer from poor environmental adaptability and unsatisfactory recognition results. Summary of the Invention

[0003] In order to overcome the shortcomings of the existing technology, the present invention provides a multispectral face recognition module and method.

[0004] This invention can be achieved by adopting the following technical solutions:

[0005] A multispectral face recognition module includes a camera assembly and a control assembly. The camera assembly includes an IR camera, a fill light, and an RGB camera. The control assembly includes a main controller, a power management unit, and a storage unit.

[0006] The IR camera and the RGB camera are connected to the main controller, and the captured IR and RGB images of the face are processed by the image processing unit and then sent to the main controller.

[0007] The fill light includes white LED beads and infrared LED beads arranged vertically and packaged together. The fill light is integrated with the IR camera and the RGB camera to provide fill light to the IR camera and the RGB camera according to the ambient light conditions.

[0008] The main controller performs infrared image liveness detection and color image liveness detection on IR and RGB images. If both the IR and RGB images meet the liveness criteria, the IR and RGB images are then compared with the database for identification, and the identification results are output.

[0009] The power management unit is connected to the main controller and provides power to the main controller;

[0010] The storage unit is connected to the main controller and is used to store data and programs.

[0011] Preferably, the IR camera is connected to the main controller via a DVP interface, and the RGB camera is connected to the main controller via a MIPI interface.

[0012] Preferably, the control component further includes an encryption IC, which is connected to the main controller.

[0013] Preferably, the control component further includes a voice interface, a USB interface, a UART interface, an LCD interface, and a TP interface connected to the main controller.

[0014] A multispectral face recognition method includes the following steps:

[0015] (1) The IR camera and the RGB camera take pictures of the face. The IR camera and the RGB camera respectively perform light metering, detect the ambient light conditions, and make intelligent dynamic exposure adjustment according to the data measured by each of them. The IR image and RGB image are captured and sent to the main controller after being processed by the image processing unit.

[0016] (2) The IR image liveness detection unit and the RGB image liveness detection unit of the main controller perform liveness detection on the IR image and the RGB image respectively. If the detection results of the IR image liveness detection unit and the RGB image liveness detection unit are both live, then proceed to the next step; if either the detection result of the IR image liveness detection unit or the RGB image liveness detection unit is not live, then output the detection failure result.

[0017] (3) The IR image recognition unit and RGB image recognition unit of the main controller perform face recognition on the IR image and RGB image according to the face information in the database. If no face information is matched in the database, the output is "no face information matched"; if a face information is matched in the database, the corresponding matched face information is output.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention utilizes the structure of an IR camera and an RGB camera to capture the face and obtain IR and RGB images, which are then used by the main controller for liveness detection and face recognition, effectively improving the overall recognition accuracy. At the same time, the IR camera and RGB camera detection can adapt to different lighting environments, effectively improving the efficiency of recognition and expanding the scope of application. Attached Figure Description

[0019] Figure 1 This is a structural block diagram of the multispectral face recognition module of the present invention;

[0020] Figure 2 This is a schematic diagram of the camera assembly of the present invention;

[0021] Figure 3 This is a flowchart of the multispectral face recognition method of the present invention. Detailed Implementation

[0022] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Example 1

[0024] like Figures 1 to 3 As shown, the multispectral face recognition module of this embodiment includes a camera assembly 1 and a control assembly 2. The camera assembly includes an IR camera 13, an RGB camera 11, and a fill light 12. The control assembly 2 includes a main controller 21, a power management unit 27, and a storage unit 26.

[0025] The IR camera 13 and the RGB camera 11 are connected to the main controller 21, and the captured IR and RGB images of the face are processed by the image processing unit and then sent to the main controller 21.

[0026] The fill light 12 is integrated with the IR camera 13 and the RGB camera 11 and is connected to the main controller 21 respectively, providing fill light to the IR camera 13 and the RGB camera 11 according to the ambient light conditions.

[0027] The RGB camera 11 and IR camera 13 are placed at both ends of the module bracket 14, and the supplementary light 12 is placed in the middle of the module bracket 14. The supplementary light 12 encapsulates the infrared LED bead 121 and the white LED bead 122 together, preferably using an upper and lower structure encapsulation, with the white LED bead 122 preferably placed on top, which can effectively ensure that the white light and infrared light can be evenly covered in the center position; the angle of the two LED beads covers the viewing window range of the camera, preferably using 120-degree large-angle LED beads; preferably, the supplementary light 12 is on the same horizontal plane as the IR camera 13 and the RGB camera 11; the camera assembly 1 is preferably connected to the core board via an FPC cable.

[0028] The main controller 21 performs infrared image liveness detection and color image liveness detection on the IR image and RGB image. If both the IR image and RGB image meet the liveness standard, the IR image and RGB image are then compared with the database for identification, and the identification result is output.

[0029] The power management unit 27 is connected to the main controller 21 and provides power to the main controller 21;

[0030] The storage unit 26 is connected to the main controller 21 and is used to store data and programs.

[0031] In this embodiment,

[0032] The IR camera 13 is connected to the main controller 21 via a DVP interface, and the RGB camera 11 is connected to the main controller 21 via a MIPI interface, which facilitates the main controller 21 to connect to the IR+RGB camera to acquire image information.

[0033] The control component also includes an encryption IC 28, which is connected to the main controller 21 and is used to protect the face recognition module from malicious cracking.

[0034] The control component 2 also includes a voice interface 29, a USB interface 22, a UART interface 23, an LCD interface 24, and a TP interface 25 connected to the main controller 21. The LCD interface 24 is used to connect to a display screen to show image information and menu information acquired by the camera. The USB interface 22 is used to communicate with the doorbell module, transmitting images via the UVC protocol. The UART interface 23 is used to communicate with the lock control module, transmitting information related to face recognition results. The TP interface 25 is used to connect an external touchscreen, enabling human-computer interaction input. The voice interface 29 is used to connect an external speaker to output voice information.

[0035] All of the above components are common components, and the connections between them use common circuit connection methods.

[0036] Example 2

[0037] Based on Example 1, such as Figure 3 As shown, the multispectral face recognition method in this embodiment includes the following steps:

[0038] (1) The IR camera 13 and the RGB camera 11 take pictures of the face. The IR camera 13 and the RGB camera 11 respectively perform light metering, detect the ambient light conditions, and make intelligent dynamic exposure adjustment according to the data they measure. The IR image and RGB image are captured and sent to the main controller 21 after being processed by the image processing unit.

[0039] (2) The IR image liveness detection unit and the RGB image liveness detection unit of the main controller 21 perform liveness detection on the IR image and the RGB image respectively. If the detection results of the IR image liveness detection unit and the RGB image liveness detection unit are both live, then proceed to the next step; if either the detection result of the IR image liveness detection unit or the RGB image liveness detection unit is not live, then output the detection failure result.

[0040] (3) The IR image recognition unit and RGB image recognition unit of the main controller 21 perform face recognition on the IR image and RGB image according to the face information in the database. If no face information is matched in the database, the output is "no face information is matched"; if a face information is matched in the database, the corresponding matched face information is output.

[0041] The specific steps in the above method are as follows:

[0042] 1. After the module is powered on, turn off the fill light 12, use the RGB camera 11 to capture an image, calculate the corresponding average brightness X of the image, and when X is greater than the set threshold, determine that the ambient light is a strong light environment. Initialize the exposure value and other parameters of the IR sensor of the IR camera 13 to the corresponding preset values ​​through IIC, and at the same time adjust the exposure value and other parameters of the RGB sensor of the RGB camera 11.

[0043] 2. The IR sensor is initialized with an exposure value of 1. Before the initialization takes effect, a frame of RGB image Trgb1 is acquired and the face position is located. If the face is located, the RGB anti-counterfeiting model is called to perform RGB liveness detection. If the face information is not located (at this time, it may be in a dark environment or the face is in a backlit state, and the acquired face image is very dark), then wait for the IR sensor to obtain the face location information.

[0044] 3. When the IR sensor is activated and receives the frame synchronization (vsync) signal to start image acquisition, the infrared LED is turned on instantly and the brightness is adjusted to 100%. The IR sensor acquires the first valid image Tir1. The preferred CMOS of this IR sensor uses a global exposure device. If a non-global exposure CMOS is used, the image acquisition sequence must be strictly followed to keep the brightness of the LED consistent during the CMOS exposure period.

[0045] 4. Quickly adjust the brightness of the infrared LED beads to 30%, and initialize the IR sensor with an exposure value of 2;

[0046] 5. During the initialization of the IR sensor, the face location information is located based on the IR image Tir1. If no face is located, the loop continues to step 3 after acquiring the next frame. If face information is located, the face location information located by Tir1 is mapped onto the TRGb1 image, the brightness of the face area is judged, and the exposure value and other parameters of the RGB sensor are readjusted to effectively perform individual exposure for the face area.

[0047] Traditional automatic exposure adjusts the entire image. In situations with strong ambient light interference, because faces occupy a relatively small proportion of the image, the sensor often darkens the background, making the face appear even darker and causing face localization failure. This new method, however, can achieve effective localized imaging even with ordinary IR CMOS sensors, thereby improving face localization and liveness detection in strong light.

[0048] 6. When the brightness of the infrared LED beads is 30%, infrared image Tir2 is acquired. Based on Tir1 and Tir2, liveness detection is performed. The main process is as follows:

[0049] Locate the face region in both images with the nose as the center, and divide it into a 9-grid layout;

[0050] Calculate the mean brightness Lxy of the corresponding grid cells in the two images respectively;

[0051] Perform variance calculation on corresponding grid cells of the two images, using the corresponding formula. The root mean square error s(σ)y of the corresponding grid is obtained;

[0052] The mean square error of each cell is comprehensively judged. If the mean square error s(σ)y is greater than the set threshold, it is judged as a fake; otherwise, it is a live cell.

[0053] The actual results are better when the images of Tir1 and Tir2 are sampled extensively and then trained into a neural network algorithm.

[0054] 7. Based on the Trgb1 image, call the neural network RGB anti-counterfeiting model for judgment. If the image quality is poor and cannot be judged, turn on the white LED beads for instantaneous supplementary lighting and acquire a new image Trgbn. Call the RGB anti-counterfeiting model again for liveness detection. If it is a fake, return to step 3 and repeat the process.

[0055] 8. If both the IR and RGB images pass liveness detection, the image recognition process begins. The IR and RGB images are compared and recognized separately. If the recognition passes, the process ends.

[0056] The steps of IR image recognition are as follows: a registration image processing step, generating registration data based on the input registration face image; and a recognition image processing step, recognizing the acquired face image. The registration image processing step includes: a registration face image feature extraction step, extracting features from multiple registration face images to generate registration data; the recognition image processing step includes: a conversion image generation step, obtaining a conversion image from the IR image of the face to be recognized based on pre-set learning information; an occlusion region determination step, determining the difference caused by occlusion based on the difference image between the IR image of the face to be recognized and the conversion image; a face image feature extraction step, extracting features from the acquired face image to be recognized; and a similarity calculation and evaluation step, evaluating the similarity based on the features of the registered face image and the... The features of the face image to be identified are described, and the difference portion identified in the occlusion region determination step is discarded. The similarity between the registered face image and the face image to be identified is calculated. The learning information is a conversion formula generated from the sample image space to the feature space of the registered face image, based on multiple learning sample images taken under unoccluded conditions. In the converted image generation step, the image is converted from the face image to be identified to obtain the features of the unoccluded face image space according to the conversion formula, and then the inverse operation of the conversion formula is used to convert it back to the original space to obtain a converted image of a similar unoccluded face. In the occlusion region determination step, the converted image is subtracted from the face image to be identified to generate the difference image. After binarizing the difference image, the difference portion is generated based on the size and distribution of pixels in the difference image.

[0057] The steps for RGB image recognition are the same as those for IR image recognition, and will not be described here.

[0058] The aforementioned IR image liveness detection, RGB image liveness detection, IR image recognition, and RGB image recognition can also employ other commonly used methods, as long as they can identify live objects and human face images.

[0059] The present invention has been described above with reference to preferred embodiments, but the present invention is not limited to the embodiments disclosed above, but should cover various modifications and equivalent combinations made in accordance with the essence of the present invention.

Claims

1. A multispectral face recognition module, characterized in that: The system includes a camera assembly and a control assembly. The camera assembly includes an IR camera, a fill light, and an RGB camera. The control assembly includes a main controller, a power management unit, and a storage unit. The IR and RGB cameras are connected to the main controller, and the IR and RGB images of the captured face are processed by the image processing unit and then sent to the main controller. The fill light includes white LED beads and infrared LED beads arranged vertically and packaged together. The fill light is integrated with the IR and RGB cameras to provide supplementary lighting for the IR and RGB cameras according to the ambient light conditions. The main controller performs infrared image liveness detection and color image liveness detection on the IR and RGB images. If both the IR and RGB images meet the liveness criteria, the IR and RGB images are compared with a database for identification, and the identification result is output. The power management unit is connected to the main controller and provides power to the main controller. The storage unit is connected to the main controller and is used to store data and programs; The working steps of the multispectral face recognition module include: (1) After the module is powered on, turn off the fill light, use the RGB camera to capture an image, calculate the average brightness X of the image, and when X is greater than the set threshold, determine that the ambient light is a strong light environment. Initialize the exposure value and other parameters of the IR sensor of the IR camera to the corresponding preset values ​​through IIC, and at the same time adjust the exposure value and other parameters of the RGB sensor of the RGB camera. (2) The IR sensor is initialized with an exposure value of 1. Before the initialization takes effect, a frame of RGB image Trgb1 is acquired and the face position is located. If the face is located, the RGB anti-counterfeiting model is called to perform RGB liveness detection. If the face information is not located, the IR sensor waits for the face location information. (3) When the IR sensor is activated, the infrared LED is turned on instantly when the frame synchronization signal is received and the image acquisition begins. The brightness is adjusted to 100%. The IR sensor acquires the first valid image Tir1. The IR sensor CMOS uses a global exposure device. If a non-global exposure CMOS is used, the image acquisition sequence must be strictly followed to keep the LED brightness consistent during CMOS exposure. (4) Quickly adjust the brightness of the infrared LED beads to 30%, and initialize the IR sensor with an exposure value of 2. (5) During the initialization of the IR sensor, the face position information is located according to the IR image Tir1. If the face is not located, the loop continues after the next frame image is acquired. If the face information is located, the face position information located by Tir1 is mapped onto the Tragb1 image, the brightness of the face area is judged, and the exposure value and other parameters of the RGB sensor are readjusted to effectively expose the face area separately. (6) When the brightness of the infrared LED beads is 30%, infrared image Tir2 is acquired. Based on Tir1 and Tir2, liveness detection is performed. The main process is as follows: Locate the face region centered on the nose in both images and divide it into a 9-grid; calculate the mean brightness Lxy of the corresponding grid in both images; perform variance calculation on the corresponding grid in both images, using the corresponding formula. The mean squared error s(σ)y of the corresponding cell is obtained; the mean squared error of each cell is judged comprehensively. When the mean squared error s(σ)y is greater than the set threshold, it is judged as a fake; otherwise, it is a live cell; the images of Tir1 and Tir2 are sent to the neural network algorithm for training after a large number of samples. (7) Call the neural network RGB anti-counterfeiting model based on the Trgb1 image to make a judgment. If the image quality is poor and cannot be judged, turn on the white LED lamp beads for instantaneous supplementary light and collect a new image Trgbn. Call the RGB anti-counterfeiting model again to make a liveness detection judgment. If it is a fake, return to (3) to repeat the process. (8) If the liveness detection of the IR image and the RGB image passes simultaneously, the image recognition process will begin. The IR and RGB images will be compared and recognized separately. If the recognition passes, the process will end.

2. The multispectral face recognition module according to claim 1, characterized in that: The IR camera is connected to the main controller via a DVP interface, and the RGB camera is connected to the main controller via a MIPI interface.

3. The multispectral face recognition module according to claim 1 or 2, characterized in that: The control component also includes an encryption IC, which is connected to the main controller.

4. The multispectral face recognition module according to claim 1 or 2, characterized in that: The control components also include a voice interface, a USB interface, a UART interface, an LCD interface, and a TP interface connected to the main controller.

5. A multispectral face recognition method, characterized in that: Includes the following steps: (1) The IR camera and the RGB camera take pictures of the face. The IR camera and the RGB camera respectively perform light metering, detect the ambient light conditions, and make intelligent dynamic exposure adjustment according to the data measured by each of them. The IR image and RGB image are captured and sent to the main controller after being processed by the image processing unit. (2) The IR image liveness detection unit and the RGB image liveness detection unit of the main controller perform liveness detection on the IR image and the RGB image respectively. If the detection results of the IR image liveness detection unit and the RGB image liveness detection unit are both live, then proceed to the next step; if either the detection result of the IR image liveness detection unit or the RGB image liveness detection unit is not live, then output the detection failure result. (3) The IR image recognition unit and RGB image recognition unit of the main controller perform face recognition on the IR image and RGB image according to the face information in the database. If no face information is matched in the database, the output is "no face information matched"; if a face information is matched in the database, the corresponding matched face information is output. The specific steps of this multispectral face recognition module are as follows: 1) After the module is powered on, turn off the fill light, use the RGB camera to capture an image, calculate the corresponding average brightness X of the image, and when X is greater than the set threshold, determine that the ambient light is a strong light environment. Initialize the exposure value and other parameters of the IR sensor of the IR camera to the corresponding preset values ​​through IIC, and at the same time adjust the exposure value and other parameters of the RGB sensor of the RGB camera. 2) The IR sensor is initialized with an exposure value of 1. Before the initialization takes effect, a frame of RGB image Trgb1 is acquired and the face position is located. If the face is located, the RGB anti-counterfeiting model is called to perform RGB liveness detection. If the face information is not located, the sensor waits for the face location information from the IR sensor. 3) When the IR sensor is activated and the frame synchronization signal is received to start image acquisition, the infrared LED is turned on instantly and the brightness is adjusted to 100%. The IR sensor acquires the first valid image Tir1. The IR sensor CMOS uses a global exposure device. If a non-global exposure CMOS is used, the image acquisition sequence must be strictly followed to keep the LED brightness consistent during CMOS exposure. 4) Quickly adjust the brightness of the infrared LED beads to 30%, and initialize the IR sensor with an exposure value of 2; 5) During the initialization of the IR sensor, the face position information is located based on the IR image Tir1. If the face is not located, the loop continues to step 3) after the next frame image is acquired. If the face information is located, the face position information located by Tir1 is mapped onto the TRGb1 image, the brightness of the face area is judged, and the exposure value and other parameters of the RGB sensor are readjusted to effectively perform separate exposure for the face area. 6) When the infrared LED bead brightness is 30%, infrared image Tir2 is acquired. Liveness detection is performed based on Tir1 and Tir2. The main process is as follows: The face region is located centered on the nose in both images and divided into a 9-grid layout; the mean brightness Lxy of the corresponding grid in each of the two images is calculated; the variance of the corresponding grid in the two images is calculated using the corresponding formula. The mean squared error s(σ)y of the corresponding cell is obtained; the mean squared error of each cell is judged comprehensively. When the mean squared error s(σ)y is greater than the set threshold, it is judged as a fake; otherwise, it is a live cell; the images of Tir1 and Tir2 are sent to the neural network algorithm for training after a large number of samples. 7) Based on the Trgb1 image, call the neural network RGB anti-counterfeiting model for judgment. If the image quality is poor and cannot be judged, turn on the white LED beads for instantaneous supplementary lighting, and collect a new image Trgbn. Call the RGB anti-counterfeiting model again for liveness detection. If it is a fake, return to 3) and repeat the process. 8) If both the IR and RGB images pass the liveness detection, the image recognition process begins; the IR and RGB images are compared and recognized separately, and the process ends if the recognition passes.

Images