Binocular facial recognition device and calibration method of binocular facial recognition device

Abstract

The embodiment of the present application provides a binocular facial recognition device and a calibration method for the binocular facial recognition device. The binocular facial recognition device includes a visible light lens module, an infrared lens module and a processor; the visible light lens module is used to shoot multiple first visible light images; the infrared lens module is used to shoot multiple first infrared images; the processor is used to detect facial key points to obtain the first facial key point and the second facial key point; calculate the first distance between the first coordinate of the first facial key point in the target coordinate system and the second coordinate of the second facial key point in the target coordinate system, adjust the model parameters of the first coordinate transformation model in the direction of reducing the first distance, obtain the second coordinate transformation model, iterate to obtain the third coordinate transformation model; and use the third coordinate transformation model to perform facial liveness detection to obtain the facial liveness detection result. It can reduce the instability caused by bumps and vibrations to the binocular facial recognition device.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a binocular facial recognition device and a calibration method for the binocular facial recognition device. Background Art

[0002] In order to prevent unauthorized persons from entering a specific area, an access control machine with a facial recognition function can be set at the entrance of the specific area. When a person requests to enter the specific area, the access control machine will perform facial recognition on the person requesting to enter to obtain the identity of the person. If the identity of the person has the authority to enter the specific area, the access control machine will control the access control to open to release the person. If the identity of the person does not have the authority to enter the specific area, the access control machine will control the access control to remain closed to prevent the person. However, person A who does not have the authority to enter the specific area can wear a mask printed with the facial image of person B (person B has the authority to enter the specific area), so that the access control machine mistakenly recognizes the identity of person A as person B, and then releases person A.

[0003] Based on this, in the related art, a visible light lens module and an infrared lens module can be set on the access control machine at the same time. The visible light lens module is used to capture visible light images for facial recognition, and the infrared lens module is used to capture infrared images for facial liveness detection. The access control machine will only release the person if the facial recognition result indicates that the person has the authority to enter the specific area, and the liveness detection result indicates that the person's face is alive, and the visible light lens module and the infrared lens module capture the same person. In this way, even if person A wears a mask, he will not be released by the access control machine because the facial liveness detection result is not alive.

[0004] In the process of determining whether the visible light lens module and the infrared lens module have captured the same person, the conversion relationship between the image coordinate system of the visible light lens module and the image coordinate system of the infrared lens module is required. In the related art, the manufacturer of the access control machine often calibrates the visible light lens module and the infrared lens module before leaving the factory to determine the conversion relationship.

[0005] However, during the use of the access control machine, the internal and external parameters of the visible light lens module and the infrared lens module may change due to collisions by people or debris, and the conversion relationship calibrated at the factory is not accurate enough. The maintenance party of the access control machine, such as the community property, usually does not have the professional knowledge required for calibration. At this time, the access control machine manufacturer must be requested to assist in recalibrating the visible light lens module and the infrared lens module. It often takes a long time from the maintenance party making a request to the manufacturer to the manufacturer sending technicians to the site for recalibration. During this period, the access control machine will not work properly. Summary of the invention

[0006] The purpose of the embodiments of the present application is to provide a binocular facial recognition device and a calibration method for the binocular facial recognition device, so as to improve the stability of the binocular facial recognition device. The specific technical solution is as follows:

[0007] The embodiment of the present application provides a binocular facial recognition device, which includes a visible light lens module, an infrared lens module and a processor;

[0008] The visible light lens module is used to capture a plurality of first visible light images and send them to the processor in response to the calibration instruction;

[0009] The infrared lens module is used to take a plurality of first infrared images in response to the calibration instruction and send them to the processor, wherein each first infrared image is taken synchronously with a first visible light image, and different first infrared images are taken synchronously with different first visible light images;

[0010] The processor is used to perform facial key point detection on the second visible light image to obtain first facial key points; and perform facial key point detection on the second infrared image to obtain second facial key points; perform coordinate transformation on the first facial key points and / or the second facial key points using a first coordinate transformation model to obtain first coordinates of the first facial key points in a target coordinate system and second coordinates of the second facial key points in the target coordinate system, and calculate a first distance between the first coordinate and the second coordinate, and adjust the model parameters of the first coordinate transformation model in a direction in which the first distance decreases to obtain a second coordinate transformation model; use the second coordinate transformation model as a new first coordinate transformation model and a visible light image in the first visible light image that has not been used as a second visible light image as a new second visible light image, return to perform the step of performing facial key point detection on the second visible light image to obtain the first facial key points, until a preset correction end condition is met, and save the latest second coordinate transformation model as a third coordinate transformation model;

[0011] Wherein, the second visible light image is initially any first visible light image, the second infrared image is a first infrared image captured synchronously with the second visible light image, and the model parameters of the first coordinate transformation model are initially internal parameters and external parameters pre-set for the visible light lens module and the infrared lens module;

[0012] The visible light lens module is further used to capture a third visible light image and send it to the processor in response to the recognition instruction;

[0013] The infrared lens module is further used to take a third infrared image in response to the recognition instruction and send it to the processor, wherein the third infrared image is taken synchronously with the third visible light image;

[0014] The processor is further used to perform facial key point detection on the third visible light image to obtain third facial key points; and perform facial key point detection on the third infrared image to obtain fourth facial key points; read and use the third coordinate transformation model to perform coordinate transformation on the third facial key points and / or the fourth facial key points to obtain third coordinates of the third facial key points in the target coordinate system and fourth coordinates of the fourth facial key points in the target coordinate system, and calculate a second distance between the third coordinates and the fourth coordinates. If the second distance meets a preset matching condition, perform facial liveness detection on the third infrared image to obtain a facial liveness detection result; if the facial liveness detection result is liveness, perform facial recognition on the third visible light image to obtain a facial recognition result.

[0015] In a possible embodiment, the processor is further used to record a matching failure if the second distance does not meet the preset matching condition; send a correction instruction to the visible light lens module and the infrared lens module if the record of matching failure since the last execution of saving the latest second coordinate transformation model as the third coordinate transformation model meets the preset re-calibration condition; and use the third coordinate transformation model as the first coordinate transformation model, return to execute the step of performing facial key point detection on the second visible light image to obtain the first facial key point.

[0016] In a possible embodiment, the processor is also used to read the third coordinate transformation model in response to the binocular facial recognition device being powered on; if the reading fails, a correction instruction is sent to the visible light lens module and the infrared lens module, and the step of detecting facial key points on the second visible light image to obtain the first facial key points is executed.

[0017] In a possible embodiment, the internal parameters include the focal length of the lens module, the pixel size of the lens module, the position of the imaging center of the lens module, and the distortion coefficient of the lens module;

[0018] The external parameters include an offset of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module, and a deflection angle of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module;

[0019] Wherein, the focal length, the pixel size, and the position of the imaging center are initially set to the focal length, pixel size, and the position of the imaging center of the lens module rated by the manufacturer of the lens module; and the distortion coefficient is initially set to 0;

[0020] The offset is initially set to a rated offset between the optical axis of the visible light lens module and the optical axis of the infrared lens module; and the deflection angle is initially set to 0.

[0021] The present application also provides a binocular facial recognition device calibration method, the method comprising:

[0022] Acquire a plurality of first visible light images captured by a visible light lens module in a binocular facial recognition device, and acquire a plurality of first infrared images captured by an infrared lens module in the binocular facial recognition device, wherein each first infrared image is captured synchronously with a first visible light image, and different first infrared images are captured synchronously with different first visible light images;

[0023] Performing facial key point detection on the second visible light image to obtain first facial key points; and performing facial key point detection on the second infrared image to obtain second facial key points;

[0024] The first facial key point and / or the second facial key point are transformed by using a first coordinate transformation model to obtain a first coordinate of the first facial key point in a target coordinate system and a second coordinate of the second facial key point in the target coordinate system, wherein the second visible light image is initially an arbitrary first visible light image, the second infrared image is a first infrared image obtained by synchronously shooting with the second visible light image, and the model parameters of the first coordinate transformation model are initially internal parameters and external parameters pre-set for the visible light lens module and the infrared lens module;

[0025] calculating a first distance between the first coordinate and the second coordinate;

[0026] adjusting the model parameters of the first coordinate transformation model in a direction in which the first distance decreases, to obtain a second coordinate transformation model;

[0027] Use the second coordinate transformation model as the new first coordinate transformation model and the visible light image in the first visible light image that has not been used as the second visible light image as the new second visible light image, return to execute the step of performing facial key point detection on the second visible light image to obtain the first facial key points, until the preset correction end condition is met, and save the latest second coordinate transformation model as the third coordinate transformation model.

[0028] In a possible embodiment, the method further includes:

[0029] In response to the binocular facial recognition device being powered on, reading a third coordinate conversion model;

[0030] If the reading fails, the steps of acquiring a plurality of first visible light images captured by the visible light lens module in the binocular facial recognition device and acquiring a plurality of first infrared images captured by the infrared lens module in the binocular facial recognition device are performed.

[0031] In a possible embodiment, the internal parameters include the focal length of the lens module, the pixel size of the lens module, the position of the imaging center of the lens module, and the distortion coefficient of the lens module;

[0032] The external parameters include an offset of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module, and a deflection angle of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module;

[0033] Wherein, the focal length, the pixel size, and the position of the imaging center are initially set to the focal length, pixel size, and the position of the imaging center of the lens module rated by the manufacturer of the lens module; and the distortion coefficient is initially set to 0;

[0034] The offset is initially set to a rated offset between the optical axis of the visible light lens module and the optical axis of the infrared lens module; and the deflection angle is initially set to 0.

[0035] The present application also provides a binocular face recognition method, the method comprising:

[0036] Acquire a third visible light image captured by a visible light lens module in a binocular facial recognition device, and acquire a third infrared image captured by an infrared lens module in the binocular facial recognition device, wherein the third infrared image is captured synchronously with the third visible light image;

[0037] Reading a third coordinate transformation model obtained according to any of the above binocular facial recognition device calibration methods;

[0038] Performing facial key point detection on the third visible light image to obtain third facial key points;

[0039] Performing facial key point detection on the third infrared image to obtain fourth facial key points;

[0040] Performing coordinate transformation on the third facial key point and / or the fourth facial key point using the third coordinate transformation model to obtain a third coordinate of the third facial key point in the target coordinate system and a fourth coordinate of the fourth facial key point in the target coordinate system;

[0041] calculating a second distance between the third coordinate and the fourth coordinate;

[0042] If the second distance satisfies a preset matching condition, performing facial liveness detection on the third infrared image to obtain a facial liveness detection result;

[0043] If the facial liveness detection result is liveness, facial recognition is performed on the third visible light image to obtain a facial recognition result.

[0044] In a possible embodiment, the method further includes:

[0045] If the second distance does not meet the preset matching condition, a matching failure is recorded;

[0046] If the records of failed matching since the most recent execution of any of the above-mentioned binocular facial recognition device calibration methods meet the preset recalibration conditions, the third coordinate transformation model is used as the first coordinate transformation model, and any of the above-mentioned binocular facial recognition device calibration methods is executed to obtain a new third coordinate transformation model.

[0047] The present application also provides an electronic device, including:

[0048] Memory, used to store computer programs;

[0049] The processor is used to implement any of the above-mentioned binocular facial recognition device calibration methods or binocular facial recognition methods when executing the program stored in the memory.

[0050] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-described binocular facial recognition device calibration methods or binocular facial recognition methods.

[0051] Beneficial effects of the embodiments of the present application:

[0052] The binocular facial recognition device and the calibration method of the binocular facial recognition device provided in the embodiments of the present application can verify whether the first coordinate transformation model is accurate based on the first infrared image taken by the visible light lens module and the second infrared image taken synchronously by the infrared lens module, and learn the inaccuracy of the first coordinate transformation model, thereby adjusting the first coordinate transformation model in the direction of making the first coordinate transformation model more accurate, obtaining a more accurate second coordinate transformation model, and then continuing to learn with the second coordinate transformation model as the new first coordinate transformation model. By repeatedly iterating in this way, a sufficiently accurate second coordinate transformation model is obtained as the third coordinate transformation model, and coordinate transformation is performed based on the obtained third coordinate transformation model to determine whether the persons in the third visible light image and the third infrared image are the same, and liveness detection and facial recognition are performed only when the persons are the same. It can be seen that the binocular facial recognition device provided by the present application can learn the target coordinate transformation relationship in a machine learning manner based on the visible light image and infrared image taken by itself before performing liveness detection and facial recognition, so as to automatically calibrate the target coordinate transformation relationship. Even if the target transformation relationship changes significantly due to bumps, shocks, etc., the binocular facial recognition device can automatically calibrate the target coordinate transformation relationship and perform coordinate transformation according to the calibrated coordinate transformation relationship, so as to accurately determine whether the person in the visible light image and the infrared image is the same, reduce the possibility of inaccurate liveness detection and facial recognition results due to different people, and thus improve the stability of the binocular facial recognition device. In other words, the stability of the binocular facial recognition device can be effectively improved without the need for personnel with relevant knowledge to go to the site for recalibration.

[0053] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0055] Figure 1a A schematic diagram of the structure of a binocular facial recognition device provided by this application;

[0056] Figure 1b Another structural schematic diagram of the binocular facial recognition device provided by this application;

[0057] Figure 2 A flowchart of the process performed during the calibration phase of a binocular facial recognition device provided in this application;

[0058] Figure 3 A flowchart of the process performed during the authentication phase of a binocular facial recognition device provided for this application;

[0059] Figure 4a A flowchart of the process performed by the key point acquisition sub-stage of the binocular facial recognition device provided in this application;

[0060] Figure 4b A schematic diagram of a facial recognition frame and facial key points detected by a binocular facial recognition device provided in this application;

[0061] Figure 5 A flowchart of the process performed by the binocular facial recognition device model adjustment sub-stage provided in this application;

[0062] Figure 6 A flowchart of the process performed in the calibration phase + authentication phase of the binocular facial recognition device provided in this application;

[0063] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0065] In order to more clearly illustrate the binocular facial recognition device provided by the present application, the following will take the two scenarios of access control and attendance as examples to exemplify the application scenarios of the binocular facial recognition device provided by the application. It can be understood that the two scenarios of access control and attendance in the following examples are only two possible application scenarios of the binocular facial recognition device provided by the present application. In other possible embodiments, the binocular facial recognition device provided by the present application can also be applied to other possible application scenarios, and the following examples do not impose any limitations on this.

[0066] For access control scenarios:

[0067] There is an access control system at the entrance of the community, which includes a binocular facial recognition device and a gate. The gate is closed by default to prevent people from passing through. When a person needs to enter the community, he needs to enter the field of view of the binocular facial recognition device first. At this time, the binocular facial recognition device will synchronously capture the visible light image and infrared image, and detect the facial key points in the visible light image (hereinafter referred to as visible light key points) and the facial key points in the infrared image (hereinafter referred to as infrared key points). The visible light key points are projected into the image coordinate system of the infrared image using the infrared image and the conversion relationship between the image coordinate system of the visible light image and the image coordinate system of the infrared image (hereinafter referred to as the target conversion relationship) to obtain the projected key points, and the distance between the projected key points and the infrared key points is calculated. If the distance is less than the preset distance threshold, it can be considered that the faces of the people in the visible light image and the infrared image are the faces of the same person. Then, facial liveness detection is performed based on the infrared image. If the liveness detection is passed, facial recognition is performed using the visible light image to determine whether the person is on the preset whitelist. If the recognition result indicates that the person is on the whitelist, the binocular facial recognition device drives the gate to switch to the open state to allow the person to pass. If the recognition result indicates that the person is not on the whitelist, or has not passed the liveness detection, or the distance is not less than the preset distance threshold, the binocular facial recognition device controls the gate to continue to remain closed.

[0068] The target conversion relationship is usually calibrated by the manufacturer of the binocular facial recognition device when the binocular facial recognition device leaves the factory and is pre-stored in the binocular facial recognition device, so that the binocular facial recognition device can perform liveness detection according to the pre-stored target conversion relationship during operation.

[0069] It is understandable that binocular facial recognition equipment is often set near the path where people pass through the gate, so that people can enter the field of view of the binocular facial recognition equipment without having to take a detour. However, this also results in the distance between people and the binocular facial recognition equipment being often small during the process of passing through the gate, which is prone to collision. Once a collision occurs, the target conversion relationship will change. For the convenience of description, the target conversion relationship when no collision occurs is referred to as the first target conversion relationship, and the target conversion relationship after the collision is referred to as the second target conversion relationship. The aforementioned pre-stored target conversion relationship is obviously the first target conversion relationship.

[0070] When the number of collisions is small, it can be considered that the collisions will only slightly change the target conversion relationship, so it can be approximately considered that the first target conversion relationship is the same as the second target conversion relationship. Therefore, at this time, the binocular facial recognition device can still use the pre-stored first target conversion relationship to accurately project the visible light key points. However, as the binocular facial recognition device is used for an increasing period of time, the number of collisions continues to accumulate. At this time, it can be considered that the collisions will greatly change the target conversion relationship, that is, the first target conversion relationship is quite different from the second target conversion relationship. Therefore, if the binocular facial recognition device continues to use the pre-stored first target conversion relationship to accurately project the visible light key points, the following situations 1 and 2 will occur:

[0071] Case 1: The person requesting to pass is not wearing a mask and is on the whitelist. Even if the binocular facial recognition device correctly captures the visible light image and infrared image of the person, the calculated distance should be less than the preset distance threshold. However, since the first target conversion relationship is not accurate enough, the distance calculated exceeds the preset distance threshold, which prevents the person from passing normally.

[0072] Case 2: The person requesting to pass is person A who is wearing a mask and is not on the whitelist, and when the person requests to pass, person B who is not wearing a mask is standing behind him. The binocular facial recognition device correctly captures the visible light image of person A, but mistakenly captures the infrared image of person B. At this time, the calculated distance should exceed the preset distance threshold, but because the first target conversion relationship is not accurate enough, the distance calculated when calculating the distance is less than the preset distance threshold, which triggers the liveness detection of the infrared image. Since the infrared image is obtained by capturing person B and person B is not wearing a mask, the liveness detection will pass, and then the visible light image will be recognized for facial recognition. Since person A is wearing a mask, although person A is not on the whitelist, he can pass the facial recognition and pass through the access control.

[0073] It can be seen that an inaccurate first target conversion relationship may result in the inability of people who should pass normally to pass normally, and may also result in the ability of people who should not pass normally to pass normally, which will also cause the access control machine to fail to work properly.

[0074] Moreover, even if the community property finds that the access control cannot work properly due to too many collisions, it is difficult to restore the binocular facial recognition device to a state without collisions, and can only recalibrate the target conversion relationship. However, the calibration conversion relationship requires the person who performs the calibration to have relevant knowledge, and the property often does not have personnel with relevant knowledge, so it is necessary to contact the manufacturer of the binocular facial recognition device to send engineers to assist. It often takes several days from the time the property contacts the manufacturer to the time the manufacturer sends engineers to the community and calibrate. During this period, the access control will be in a state of not being able to work properly, affecting the safety of the community.

[0075] For attendance scenarios:

[0076] There is an attendance machine at the entrance of the company. In this example, the attendance machine is a binocular facial recognition device.

[0077] First, enter the field of view of the attendance machine. At this time, the attendance machine will capture the visible light image and infrared image of the employee's face, detect the visible light key points and infrared key points, and use the target transformation relationship to project the visible light key points into the image coordinate system of the infrared image to obtain the projected key points. The distance between the projected key points and the infrared key points is calculated. If the distance is less than the preset distance threshold, it can be considered that the faces of the people in the visible light image and the infrared image are the faces of the same employee. Then, the infrared image is used for facial liveness detection. If the liveness detection is passed, the visible light image is used for facial recognition to identify the employee's work number, and the system time of the attendance machine and the recognized work number are recorded to generate an attendance record.

[0078] Assume that the attendance machine in this example is installed on the wall, and the company has been renovated at least once after the installation of the attendance machine. During the renovation process, the wall will vibrate, which will cause the attendance machine to vibrate. This vibration will cause a significant change in the target conversion relationship. Similarly, in the aforementioned access control scenario, the attendance machine will not be able to record attendance normally after the vibration occurs.

[0079] In addition, most companies often do not have employees with the relevant knowledge to calibrate binocular facial recognition equipment. Therefore, they need to contact the manufacturer of the binocular facial recognition equipment to dispatch engineers to assist. It often takes several days from the time the company contacts the manufacturer to the time the manufacturer dispatches engineers to the company and calibrate. During this period, the attendance machine will be unable to work normally, affecting the normal management of the company.

[0080] Based on this, the present application provides a binocular facial recognition device, such as Figure 1a As shown, including:

[0081] A visible light lens module 110 , an infrared lens module 120 and a processor 130 .

[0082] For example, the binocular facial recognition device may refer to a binocular facial recognition camera, and the structure of the binocular facial recognition camera may be as follows: Figure 1b As shown. The binocular facial recognition camera includes: a visible light lens module 110, an infrared lens module 120 and a processor 130. It can be understood that, Figure 1b This is just a possible schematic diagram of the structure of a binocular facial recognition camera. Figure 1b Only the visible light lens module 110 and the infrared lens module 120 are shown, and the processor 130 is not shown.

[0083] The following will describe the steps performed by the visible light lens module 110, the infrared lens module 120 and the processor 130 in the calibration stage and the recognition stage respectively:

[0084] For the calibration phase, the execution process can be found in Figure 2 ,include:

[0085] In step S201 , the visible light lens module responds to a calibration instruction, captures a plurality of first visible light images, and sends the images to a processor.

[0086] Step S202: the infrared lens module responds to the calibration instruction, captures a plurality of first infrared images and sends them to the processor.

[0087] Among them, each first infrared image is taken synchronously with a first visible light image, and different first infrared images are taken synchronously with different first visible light images. For the convenience of description below, it is assumed that in step S201, the visible light lens module takes a total of 10 first visible light images, which are respectively recorded as the first visible light image 1, the first visible light image 2, ..., the first visible light image 10, and the infrared lens module takes a total of 10 first infrared images in step S202, which are respectively recorded as the first infrared image 1, the first infrared image 2, ..., the first infrared image 10. And the first visible light image 1 is taken synchronously with the first infrared image 1, the first visible light image 2 is taken synchronously with the first infrared image 2, and so on.

[0088] In step S203, the processor performs facial key point detection on the second visible light image to obtain first facial key points, and performs facial key point detection on the second infrared image to obtain second facial key points.

[0089] The second visible light image is initially any first visible light image, and the second infrared image is a first infrared image taken synchronously with the second visible light image. Exemplarily, assuming that the second visible light image is the first visible light image 1 when step S203 is performed for the first time (i.e., initially), the second infrared image is the first infrared image 1.

[0090] It can be understood that since the second visible light image and the second infrared image are taken synchronously, if there is only one person in the field of view of the binocular facial recognition device at the same time during the authentication stage, it can be considered that the second visible light image and the second infrared image are obtained by taking the same person, and therefore the first facial key points and the second facial key points can be considered to be the projections of the facial key points of the same person in different image coordinate systems.

[0091] Step S204: the processor uses the first coordinate transformation model to perform coordinate transformation on the first facial key point and / or the second facial key point to obtain a first coordinate of the first facial key point in a target coordinate system and a second coordinate of the second facial key point in the target coordinate system.

[0092] Depending on the target coordinate system, the coordinate conversion in step S204 is performed in different ways, which are described in three cases below:

[0093] Case A: The target coordinate system is the image coordinate system of the visible light lens module. In this case, since the first facial key point is originally in the target coordinate system, there is no need to perform coordinate transformation on the first facial key point to obtain the first coordinate. Therefore, in this case, only the second facial key point needs to be transformed to obtain the second coordinate.

[0094] Case B: The target coordinate system is the image coordinate system of the infrared lens module. In this case, since the second facial key point is originally in the target coordinate system, there is no need to perform coordinate transformation on the second facial key point to obtain the second coordinate. Therefore, in this case, only the first facial key point needs to be transformed to obtain the first coordinate.

[0095] Case C: The target coordinate system is neither the image coordinate system of the visible light lens module nor the image coordinate system of the infrared lens module. In this case, it is necessary to perform coordinate transformation on the first facial key point to obtain the first coordinate, and also to perform coordinate transformation on the second facial key point to obtain the second coordinate.

[0096] The model parameters of the first coordinate transformation model mentioned in this step are initially the internal and external parameters pre-set for the visible light lens module and the infrared lens module. It can be understood that the internal and external parameters are not the real external and real internal parameters of the visible light lens module and the infrared lens module, so the first coordinate transformation model with the internal and external parameters as model parameters cannot accurately obtain the first coordinate and the second coordinate.

[0097] Furthermore, as described in the aforementioned step S203, the first facial key point and the second facial key point can be regarded as projections of the facial key points of the same person in different image coordinate systems. Therefore, in theory, the projections of the first facial key point and the second facial key point in the same coordinate system are exactly the same. In other words, in theory, the first coordinate and the second coordinate should be the same, but because the first coordinate transformation model cannot accurately obtain the first coordinate and the second coordinate, there is a certain distance between the first coordinate and the second coordinate actually calculated.

[0098] Step S205: the processor calculates a first distance between the first coordinate and the second coordinate.

[0099] Step S206: the processor adjusts the model parameters of the first coordinate transformation model in a direction in which the first distance decreases, to obtain a second coordinate transformation model.

[0100] As described in the aforementioned step S204, the first distance is caused by the inaccuracy of the first coordinate transformation model. Therefore, adjusting the model parameters of the first coordinate transformation model in the direction of reducing the first distance can make the coordinate transformation model more accurate. Therefore, it can be considered that the second coordinate transformation model can realize coordinate transformation more accurately than the first coordinate transformation model.

[0101] In step S207, the processor uses the second coordinate transformation model as the new first coordinate transformation model and the visible light image in the first visible light image that has not been used as the second visible light image as the new second visible light image, and returns to execute step S203 until the preset correction end condition is met, and saves the latest second coordinate transformation model as the third coordinate transformation model.

[0102] As described in the aforementioned step S206, it can be considered that each time step S206 is executed, a more accurate coordinate transformation model will be obtained. Before the preset correction end condition is met, it can be considered that the newly obtained coordinate transformation model is not accurate enough. When the preset correction end condition is met, it can be considered that the newly obtained coordinate transformation model is accurate enough and can accurately calculate the first coordinate and the second coordinate, so there is no need to further generate a new second coordinate transformation model.

[0103] The preset correction end condition in this article can be that the convergence of the latest coordinate transformation model is higher than the preset convergence threshold, or the number of times step S206 is executed reaches the preset number threshold, or all first visible light images are used as second visible light images. This application does not impose any restrictions on this.

[0104] For the authentication phase, the execution process can be found in Figure 3 ,include:

[0105] Step S301: the visible light lens module responds to the recognition instruction, captures a third visible light image and sends it to the processor.

[0106] Step S302: the infrared lens module responds to the recognition instruction, captures a third infrared image and sends it to the processor.

[0107] The third infrared image is taken synchronously with the third visible light image.

[0108] In step S303, the processor performs facial key point detection on the third visible light image to obtain third facial key points, and performs facial key point detection on the third infrared image to obtain fourth facial key points.

[0109] Although the third infrared image is taken synchronously with the third visible light image, unlike the correction phase, the authentication phase often lasts longer, so it is often difficult to ensure that there is only one person in the field of view of the binocular facial recognition device at the same time. Therefore, the third facial key point and the fourth facial key point may also be the facial key points of different persons.

[0110] Step S304: the processor reads and uses the third coordinate transformation model to perform coordinate transformation on the third facial key point and / or the fourth facial key point to obtain the third coordinate of the third facial key point in the target coordinate system and the fourth coordinate of the fourth facial key point in the target coordinate system.

[0111] This step is similar to the aforementioned step S204 and is divided into three cases, namely:

[0112] Case A': the target coordinate system is the image coordinate system of the visible light lens module. In this case, the third coordinate can be obtained without performing coordinate transformation on the third facial key point. Therefore, in this case, only the fourth facial key point needs to be transformed to obtain the fourth coordinate.

[0113] Case B': the target coordinate system is the image coordinate system of the infrared lens module. In this case, the fourth coordinate can be obtained without performing coordinate transformation on the fourth facial key point. Therefore, in this case, only the third facial key point needs to be transformed to obtain the third coordinate.

[0114] Case C': the target coordinate system is neither the image coordinate system of the visible light lens module nor the image coordinate system of the infrared lens module. In this case, both the third facial key point needs to be transformed to obtain the third coordinate and the fourth facial key point needs to be transformed to obtain the fourth coordinate.

[0115] Step S305, the processor calculates the second distance between the third coordinate and the fourth coordinate. If the second distance meets the preset matching condition, facial liveness detection is performed on the third infrared image to obtain a facial liveness detection result. If the facial liveness detection result is liveness, facial recognition is performed on the third visible light image to obtain a facial recognition result.

[0116] As described in the aforementioned step S207, it can be considered that the third coordinate transformation model can accurately realize the coordinate transformation, that is, it can be considered that the obtained third coordinate and fourth coordinate are accurate. Therefore, if the second distance between the third coordinate and the fourth coordinate does not meet the preset matching condition, it can be considered that the spatial positions of the third facial key point and the fourth facial key point are different, that is, the third visible light image and the third infrared image are not obtained by shooting the same person. At this time, even if the third infrared image is performed with liveness detection, since the person in the third infrared image is different from the person in the third visible light image, the liveness detection result cannot reflect whether the person identified by facial recognition (that is, the person in the third visible light image) is alive or non-living, and therefore, there is no need to further perform liveness detection and facial recognition.

[0117] If the second distance between the third coordinate and the fourth coordinate meets the preset matching condition, it can be considered that the spatial positions of the third facial key point and the fourth facial key point are the same, and only one person can exist in the same spatial position at the same time. Therefore, it can be considered that the third visible light image and the third infrared image are obtained by shooting the same person. At this time, liveness detection is performed on the third infrared image. If the liveness detection result is live, the person in the third visible light image can be considered to be live, and then facial recognition can be performed on the third visible light image. This can reduce the possibility of inaccurate facial recognition results caused by wearing a mask, etc.

[0118] Depend on Figure 2 The calibration phase flow shown and Figure 3It can be seen from the authentication stage process shown that when the binocular facial recognition device provided by the present application is selected, the binocular facial recognition device can verify whether the first coordinate transformation model is accurate based on the first infrared image taken by the visible light lens module and the second infrared image taken synchronously by the infrared lens module, and learn the inaccuracy of the first coordinate transformation model, thereby adjusting the first coordinate transformation model in the direction of making the first coordinate transformation model more accurate, obtaining a more accurate second coordinate transformation model, and then continuing to learn with the second coordinate transformation model as the new first coordinate transformation model. By repeatedly iterating in this way, a sufficiently accurate second coordinate transformation model is obtained as the third coordinate transformation model, and coordinate transformation is performed according to the obtained third coordinate transformation model to determine whether the persons in the third visible light image and the third infrared image are the same, and liveness detection and facial recognition are performed only when the persons are the same. It can be seen that the binocular facial recognition device provided by the present application can learn the target coordinate transformation relationship in a machine learning manner based on the visible light image and infrared image taken by itself before performing liveness detection and facial recognition, so as to automatically calibrate the target coordinate transformation relationship. Even if the target transformation relationship changes significantly due to bumps, shocks, etc., the binocular facial recognition device can automatically calibrate the target coordinate transformation relationship and perform coordinate transformation according to the calibrated coordinate transformation relationship, so as to accurately determine whether the person in the visible light image and the infrared image is the same, reduce the possibility of inaccurate liveness detection and facial recognition results due to different people, and thus improve the stability of the binocular facial recognition device. In other words, the stability of the binocular facial recognition device can be effectively improved without the need for personnel with relevant knowledge to go to the site for recalibration.

[0119] The above has explained the processes of the correction stage and the authentication stage respectively. The following will explain how to construct the initial first coordinate transformation model involved in the correction stage. In order to better explain the construction, the imaging principle of the binocular facial recognition device will be briefly explained below.

[0120] The light emitted (or reflected) from a point P (x, y, z) in the real space propagates to the first object point O1 (x1, y1) under the action of the optical element in the visible light lens module 110, and is incident on the first photosensitive element on the photosensitive component in the visible light lens module 110. Since the photosensitive element corresponds to the pixel point in the visible light image one by one, the first photosensitive element can be represented by the pixel coordinate of the corresponding pixel point in the visible light image, which is recorded as S1 (x1', y1 ’), similarly, the light emitted (or reflected) from the point P (x, y) propagates to the second object point O2 (x2, y2) under the action of the optical element in the infrared lens module 120, and is incident on the second photosensitive element on the photosensitive component in the visible light lens module 120. The second photosensitive element can also be represented by the pixel coordinates of the corresponding pixel point in the infrared image, which is recorded as S2 (x2 ’ , y2 ’ ).

[0121] Furthermore, considering that the visible light lens module 110 and the infrared lens module 120 are distorted, the light emitted (or reflected) by the point P (x, y, z) will not appear as S1 (x1 ’ ,y1 ’ ), but at other locations. This paper assumes that the light emitted (or reflected) by the point P (x, y) affected by the distortion is actually presented at R1 (x1 ’’ ,y1 ’’ ) and assume that the light emitted (or reflected) by the point P (x, y, z) affected by the distortion is actually presented at R2 (x2 ’’ , y2 ’’ ) place.

[0122] R1 (x1 ’’ ,y1 ’’ ) and R2 (x2 ’’ , y2 ’’ ) can be regarded as two projection points of the spatial point P (x, y, z) in different coordinate systems. Therefore, if R1 (x1 ’’ ,y1 ’’ ) and R2 (x2 ’’ , y2 ’’ ) are projected into the same coordinate system, then R1(x1 ’’ ,y1 ’’ ) and R2 (x2 ’’ , y2 ’’ ) will have the same projection points in this same coordinate system.

[0123] R1 (x1 ’’ ,y1 ’’ ) belongs to the coordinate system of the image plane coordinate system of the visible light lens module 110 affected by the distortion (hereinafter referred to as the coordinate system vis), R2 (x2 ’’ , y2 ’’ ) is the image plane coordinate system of the infrared lens module 120 affected by the distortion (hereinafter referred to as the coordinate system inf). If the conversion relationship between the coordinate system vis and the coordinate system inf is recorded as f vis，inf , then due to R1 (x1 ’’ ,y1 ’’) is projected to the second coordinate system at R2 (x2 ’’ , y2 ’’ ), so there is formula (1):

[0124] R2 (x2 ’’ , y2 ’’ ) = f vis，inf [R1 (x1 ’’ ,y1 ’’ )] … (1)

[0125] S1(x1',y1 ’ ) belongs to the coordinate system of the image plane coordinate system of the visible light lens module 110 that is not affected by distortion (hereinafter referred to as the coordinate system vis_undistorted), S2 (x2 ’ , y2 ’ ) is the image plane coordinate system of the infrared lens module 120 affected by the distortion (hereinafter referred to as the coordinate system inf_undistorted). If the conversion relationship between the coordinate system vis and the coordinate system vis_undistorted is recorded as f D_vis , and the transformation relationship between the coordinate system inf and the coordinate system inf_undistorted is recorded as f D_inf , and the conversion relationship between the coordinate system vis_undistorted and the coordinate system inf_undistorted is recorded as f vis，inf,undis , then formula (1) can be rewritten as formula (2):

[0126] R2 (x2 ’’ , y2 ’’ )=f D_inf -1 [f vis，inf,undis [f D_vis [R1 (x1 ’’ ,y1 ’’ )]]] … (2)

[0127] According to the principle of optical imaging, f vis，inf,undis The internal parameter matrix K of the visible light lens module 110 can be used vis , the internal parameter matrix K of the infrared lens module 120 inf And the translation matrix T and rotation matrix R between the visible light lens module 110 and the infrared lens module 120 are expressed, so formula (2) can be rewritten as formula (3):

[0128] R2 (x2 ’’ , y2 ’’ )=f D_inf -1 [K inf [RKvis -1 f D_vis [R1 (x1 ’’ , y1 ’’ )]+T / d]] … (3)

[0129] Wherein, d is the depth of the spatial point P. If the optical lens module 110 and the infrared lens module 120 can be both arranged in the xy plane of the spatial coordinate system and facing the z direction, then d is the coordinate of the point P (x, y, z) in the z direction, that is, z. At this time, formula (3) can be rewritten as formula (4):

[0130] R2 (x2 ’’ , y2 ’’ )=f D_inf -1 [K inf [RK vis -1 f D_vis [R1 (x1 ’’ , y1 ’’ )]+T / z]] … (4)

[0131] It can be understood that according to formula (4), the key point of visible light can be projected into the image coordinate system of the infrared image. Referring to the above description of case B and case B', it can be seen that formula (4) can be used as a third coordinate transformation model. In other words, in a possible embodiment, the correction process can be regarded as a process of obtaining formula (4).

[0132] Since the visible light lens module 110 and the infrared lens module 120 may collide, f in formula (4) D_inf , K inf , R, f D_vis , K vis and T are unknown, so formula (4) cannot be obtained directly. Therefore, we can first construct formula (5):

[0133] R 2_p (x2 ’’ , y2 ’’ )=f1 -1 [K2[R x K1 -1 f2[R1(x1 ’’ , y1 ’’ )]+T x / z]] … (5)

[0134] f1, K2, R in formula (5) x , K1, f2, T xThey can all be regarded as independent variables. Subtracting formula (4) from formula (5) yields formula (6):

[0135] R 2_p (x2 ’’ , y2 ’’ ) - R2 (x2 ’’ , y2 ’’ )=f1 -1 [K2[R x K1 -1 f2[R1(x1 ’’ ,y1 ’’ )]+T x / z]]- f D_inf -1 [K inf [RK vis -1 f D_vis [R1 (x1 ’’ ,y1 ’’ )]+T / z]] … (6)

[0136] The right side of the equal sign in formula (6) can be regarded as a x , K1, f2, T x is a function of the independent variable, so if R 2_p (x2 ’’ , y2 ’’ ) - R2 (x2 ’’ , y2 ’’ ) is recorded as Loss, then formula (6) can be rewritten as formula (7):

[0137] Loss = f(f1, K2, R x , K1, f2, T x ) … (7)

[0138] Here, it is only to illustrate the principle of how to construct the first coordinate transformation relationship, and it does not limit the Loss in this application to be R. 2_p (x2 ’’ , y2 ’’ ) - R2 (x2 ’’ , y2 ’’ ), in other possible embodiments, R 2_p (x2 ’’ , y2 ’’ ) 2 - R2 (x2 ’’ , y2 ’’ ) 2 Denoted as Loss, for these cases, it can be proved that formula (7) is valid.

[0139] And it is understandable that if f1= f D_inf , K2= K inf , R x =R,K1= K vis ,f2= f D_vis , T x =T, then Loss is equal to 0, so it can be considered that: if f1, K2, R x , K1, f2, T x With f D_inf , K inf , R, K vis , f D_vis , T is close enough, then adjust f1, K2, R in the direction where Loss tends to 0. x , K1, f2, T x , which will make f1, K2, R x , K1, f2, T x Converges to f D_inf , K inf , R, K vis , f D_vis , T, at this time, formula (5) is equivalent to formula (4), that is, formula (4) is obtained.

[0140] It can be seen that formula (4) can be obtained from formula (5) by adjustment, and the third coordinate transformation model in this application is obtained by adjusting the first coordinate transformation model. As described above, formula (4) can be regarded as the third coordinate transformation model, so formula (5) can be regarded as the first coordinate transformation model. It is only necessary to make f1, K2, R x , K1, f2, Tx and f D_inf , K inf , R, K vis , f D_vis , T is close enough.

[0141] Therefore, the first coordinate transformation model can be constructed according to formula (8):

[0142] R 2_p (x2 ’’ , y2 ’’ )=f D_inf_def -1 [K inf_def [R def K vis_def -1 f D_vis_def [R1 (x1 ’’ , y1 ’’ )]+T def / z]]… (8)

[0143] Among them, f D_inf_def is the estimated f D_inf , K inf_def is the estimated K inf , R def is the estimated R, K vis_def is the estimated K vis , f D_vis_def is the estimated f D_vis , T def is the estimated T. The following will discuss how to estimate f D_inf_def , K inf_def , R def , K vis_def , f D_vis_def , T def To explain:

[0144] For f D_inf_def , it is understandable that f D_inf It can be expressed in the form of formula (9) and formula (10):

[0145] x corrected =x(1+k1r 2 +k2r 4 )+2p1xy+p2(r 2 +2x 2 ) … (9)

[0146] y corrected =y(1+k1r 2 +k2r 4 )+p1(r 2 +2y 2 )+2p2xy … (10)

[0147] Wherein, k1, k2, p1, p2 are the distortion coefficients of the infrared lens module 120, and x is f D_inf The horizontal component of the input, y is f D_inf The vertical component of the input, r 2 =x 2 +y 2 , x corrected f D_inf The horizontal component of the output, y corrected f D_inf The vertical component of the output of . Therefore, the estimated f D_inf It can be regarded as the estimated distortion coefficient D inf In one possible embodiment, k1, k2, p1, and p2 may all be estimated to be 0. In another possible embodiment, D may also be estimated according to the technical manual provided by the infrared lens module 120 manufacturer. inf .

[0148] The same is true for fD_vis_def , estimate f D_vis It can be regarded as estimating the distortion coefficient of the visible light lens module 110. In one possible embodiment, the distortion coefficient of the visible light lens module 110 can be estimated to be 0. In another possible embodiment, the distortion coefficient of the visible light lens module 110 can also be estimated according to the technical manual provided by the manufacturer of the infrared lens module 120.

[0149] For K inf_def , it is understandable that K inf It can be written in the form of formula (11):

[0150] … (11)

[0151] Among them, f inf is the focal length of the infrared lens module 120, dx inf is the width of the pixel size of the infrared lens module 120, dy inf is the height of the pixel size of the infrared lens module 120, u inf is the horizontal component of the pixel offset of the imaging center of the infrared lens module 120 relative to the left vertex of the imaging plane, v inf It is the vertical component of the pixel offset of the imaging center of the infrared lens module 120 relative to the left vertex of the imaging plane.

[0152] From formula (11), we can see that the estimated K inf can be regarded as an estimate of f inf 、dx inf ,dy inf 、u inf 、v inf In a possible embodiment, f can be estimated according to the technical manual provided by the infrared lens module 120 manufacturer. inf 、dx inf ,dy inf 、u inf 、v inf In another possible embodiment, f can also be estimated based on the user's actual experience. inf 、dx inf ,dy inf 、u inf 、v inf .

[0153] The same applies to K vis_def , it is understandable that K vis It can be written in the form of formula (12):

[0154] … (12)

[0155] Among them, fvis is the focal length of the visible light lens module 110, dx vis is the width of the pixel size of the visible light lens module 110, dy vis is the height of the pixel size of the visible light lens module 110, u vis is the horizontal component of the pixel offset of the imaging center of the visible light lens module 110 relative to the left vertex of the imaging plane, v vis It is the vertical component of the pixel offset of the imaging center of the visible light lens module 110 relative to the left vertex of the imaging plane.

[0156] From formula (12), we can see that the estimated K inf can be regarded as an estimate of f vis 、dx vis ,dy vis 、u vis 、v vis In a possible embodiment, f can be estimated based on the technical manual provided by the manufacturer of the visible light lens module 110. vis 、dx vis ,dy vis 、u vis 、v vis In another possible embodiment, f can also be estimated based on the user's actual experience. vis 、dx vis ,dy vis 、u vis 、v vis .

[0157] For R def , it can be understood that R can be written in the form of formula (13):

[0158] … (13)

[0159] Among them, r1, r2, and r3 are the three coefficients of R, so the estimated R def It can be regarded as an estimation of r1, r2, and r3. Moreover, it can be considered that even if there is a relative rotation between the visible light lens module 110 and the infrared lens module 120, the relative rotation is small, so it can be estimated that r1=r2=r3=1.

[0160] In another possible embodiment, it can also be considered that there is no relative rotation between the visible light lens module 110 and the infrared lens module 120. In this embodiment, it can be directly considered that r1=r2=r3=1, thereby obtaining R, without estimating R. def In this embodiment, the first coordinate transformation model is not expressed in the form of formula (8) but in the form of formula (14):

[0161] R 2_p(x2 ’’ , y2 ’’ )=f D_inf_def -1 [K inf_def [RK vis_def -1 f D_vis_def [R1 (x1 ’’ , y1 ’’ )]+T def / z]] … (14)

[0162] For T def , it can be understood that T can be written in the form of formula (15):

[0163] … (15)

[0164] Wherein, b is the distance between the centers of the visible light lens module 110 and the infrared lens module 120, so it is estimated that T def It can be regarded as estimating b. b can be estimated as a rated value of the distance between the centers of the visible light lens module 110 and the infrared lens module 120.

[0165] The above has provided an exemplary explanation of how to construct the first coordinate transformation model. It can be understood that the above examples correspond to the aforementioned cases B and B', while the principles for cases A and A' as well as cases C and C' are the same. The only difference is that the form of the first coordinate transformation model is different, so it will not be repeated here.

[0166] The following still takes the above examples for case B and case B' as an example to exemplify how to adjust the model parameters in the first coordinate transformation model to obtain the second coordinate transformation model:

[0167] In this example, the first coordinate transformation model initially constructed is as shown in the above formula (8), assuming that there are a total of n first facial key points, the first coordinates of these first facial key points in the target coordinate system (the target coordinate system in this example is the image coordinate system of the infrared lens module 120) are respectively recorded as first coordinate 1, first coordinate 2, ..., first coordinate n, where first coordinate 1 is the first coordinate of the first first facial key point, first coordinate 2 is the first coordinate of the second first facial key point, and so on.

[0168] Correspondingly, there should also be n second facial key points in this example, and the second coordinates of these second facial key points in the target coordinate system are recorded as second coordinate 1, second coordinate 2, ..., second coordinate n, where second coordinate 1 is the second coordinate of the first second facial key point, second coordinate 2 is the second coordinate of the second second facial key point, and so on. It can be understood that for any positive integer i in the range of [1,n], the i-th first facial key point and the i-th second facial key point are the same facial key point. For example, if the first first facial key point is the center point of the left eye of person A, then the first second facial key point should also be the center point of the left eye of person A.

[0169] In this example, the difference between the first coordinate and the second coordinate can be calculated according to formula (16):

[0170] … (16)

[0171] Among them, x i is the horizontal component of the first coordinate i, y i is the vertical component of the first coordinate i, x i ’ is the horizontal component of the second coordinate i, y i ’ is the vertical component of the second coordinate i.

[0172] As can be seen from the above explanation of formula (7), Loss can be regarded as a function of f D_inf_def , K inf_def , R def , K vis_def , f D_vis_def , T def function, so formula (16) can be rewritten as formula (17):

[0173] Loss = f(f D_inf_def , K inf_def , R def , K vis_def , f D_vis_def , T def ) … (17)

[0174] As mentioned above, how to estimate f D_inf_def , K inf_def , R def , K vis_def , f D_vis_def , T def From the relevant explanation of , we can know that formula (17) can be rewritten as formula (18):

[0175] Loss = f(f inf , dxinf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , r1, r2, r3, k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b) … (18)

[0176] Among them, k1 ’ , k2 ’ , p1 ’ , p2 ’ is the distortion coefficient of the visible light lens module.

[0177] For the aforementioned scenario where the visible light lens module and the infrared lens module do not rotate relative to each other, formula (18) can be rewritten as formula (19):

[0178] Loss = f(f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b) … (19)

[0179] For the case of formula (18), both sides of the equal sign are inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , r1, r2, r3, k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b is derived to get the gradient direction, and f is adjusted according to the gradient directioninf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , r1, r2, r3, k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b, and adjust the f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , r1, r2, r3, k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b is brought back into formula (8) to obtain the second coordinate transformation model.

[0180] For the above formula (19), the same principle can be applied to both sides of the equal sign. inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b is derived to get the gradient direction, and f is adjusted according to the gradient direction inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , k1, k2, p1, p2, k1 ’ , k2’ , p1 ’ , p2 ’ , b, and adjust the f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b is brought back into formula (14) to obtain the second coordinate transformation model.

[0181] The above has respectively provided exemplary explanations of how to construct the first coordinate transformation model and how to adjust the model parameters of the first coordinate transformation model. In order to more clearly explain the aforementioned correction stage, the correction stage will be fully explained below:

[0182] The correction phase can be divided into two sub-phases, namely the key point acquisition sub-phase and the model adjustment sub-phase.

[0183] The key point acquisition sub-stage is the aforementioned process of acquiring the first facial key point and the second facial key point, corresponding to the aforementioned steps S201 to S203. In a possible embodiment, it can be as follows Figure 4a As shown, including:

[0184] In step S401 , the visible light lens module responds to a calibration instruction, captures a plurality of first visible light images, and sends the images to a processor.

[0185] The multiple first visible light images can be regarded as a visible light video stream.

[0186] Step S402: the infrared lens module responds to the calibration instruction, captures a plurality of first infrared images and sends them to the processor.

[0187] The multiple first infrared images can be regarded as an infrared video stream.

[0188] Step S403: The processor performs facial key point detection on all first visible light images to obtain facial detection frames and facial key points.

[0189] Step S404: the processor performs facial key point detection on all first infrared images to obtain facial detection frames and facial key points.

[0190] Step S405: The processor performs data screening based on the detected facial detection frame and facial key points to screen out the second visible light image and the second infrared image. The screened second visible light image and the second infrared image are used in the subsequent model adjustment sub-stage, and the screened data are discarded, that is, the images other than the second visible light image and the second infrared image in the plurality of first visible light images and the plurality of first infrared images are discarded.

[0191] Facial key points can be Figure 4b As shown, it includes 5 key points, among which key point 21 is the top left corner vertex of the face detection frame, and the facial key points include: key point 22, key point 23, key point 24, key point 25, and key point 26. Among them, key point 22 corresponds to the left eye, key point 23 corresponds to the right eye, key point 24 corresponds to the nose, key point 25 corresponds to the left corner of the mouth, and key point 26 corresponds to the right corner of the mouth. In other possible embodiments, the facial key points may also include Figure 4b Other key points besides the 5 key points shown may also include Figure 4b This application does not impose any limitation on some or all of the 5 key points shown.

[0192] The screening principles may vary according to actual needs, but the following conditions should be met:

[0193] 1. The clarity of the person's face in the screened second visible light image and the second infrared image should be higher than a preset clarity threshold.

[0194] 2. The screened second visible light image needs to have an infrared image that is taken synchronously. Similarly, the screened second infrared image also needs to have a visible light image that is taken synchronously. The first visible light image and the second visible light image that are taken synchronously are either both screened as the second visible light image and the second infrared image, respectively, or neither of them is screened as the second visible light image and the second infrared image.

[0195] 3. The face of the same person is detected in the second infrared image and the second visible light image captured synchronously.

[0196] Exemplarily, assume that there is a first visible light image A, and the first visible light image A and the first infrared image B are taken synchronously. If the face of person C exists in the first visible light image A, but not in the first infrared image B, the first visible light image A will not be used as the second visible light image, nor will the first infrared image B be used as the second infrared image. If the face of person C exists in both the first visible light image A and the first infrared image B, but the clarity of the first infrared image B does not reach the preset clarity threshold, then even if the clarity of the first visible light image A reaches the preset clarity threshold, the first visible light image A will not be used as the second visible light image, nor will the first infrared image B be used as the second infrared image.

[0197] After screening, a second visible light image and a second infrared image are obtained. The facial key points detected from the second visible light image are the first facial key points, and the facial key points detected from the second infrared image are the second facial key points. The subsequent model adjustment sub-stage can be performed based on the first facial key points and the second facial key points.

[0198] The model adjustment sub-stage is the process of adjusting the first coordinate transformation model, corresponding to the process of steps S204 to S207. In a possible embodiment, it can be as follows Figure 5 As shown, including:

[0199] Step S501, substituting the initial value of the model into the coordinate transformation model to obtain a first coordinate transformation model.

[0200] For the example of formula (8), the initial value of the model is the estimated f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , r1, r2, r3, k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b, and for the example of formula (14), the initial value of the model is the estimated f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis, v vis , k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b.

[0201] Step S502: using a first coordinate conversion model to predict and obtain a first coordinate.

[0202] Step S503, calculating the deviation between the first coordinate and the second coordinate.

[0203] This example is the aforementioned situation B, so as described above for situation B, the first coordinate is obtained by converting the first facial key point to the image coordinate system of the infrared lens module through the first coordinate transformation model, and the second coordinate is the coordinate of the second facial key point itself.

[0204] The calculation of the deviation can refer to the above formula (16).

[0205] Step S504: adjusting the first coordinate transformation model in a deviation gradient descent manner to obtain a second coordinate transformation model.

[0206] The adjustment method can refer to the above description on how to adjust the first coordinate transformation model, which will not be repeated here. After obtaining the second coordinate transformation model, the second coordinate transformation model is used as the new first coordinate transformation model, and step S502 is returned to execute until the preset calibration end condition is met.

[0207] The above has been exemplified from the overall and phased process of the calibration phase. The following will be combined with the calibration phase and the authentication phase to exemplify the process performed by the binocular facial recognition device provided by the present application. Figure 6 ,include:

[0208] Step S601: the processor reads the third coordinate transformation model after powering on.

[0209] As described above, in case B and case B', the third coordinate transformation model can be based on f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , r1, r2, r3, k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’, b these model parameters can also be expressed in terms of f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b are represented by these model parameters, so reading the third coordinate transformation model can be regarded as reading these model parameters.

[0210] Different from the estimated f inf , dx inf ,dy inf ,u inf , v inf , f vis , dx vis ,dy vis ,u vis , v vis , r1, r2, r3, k1, k2, p1, p2, k1 ’ , k2 ’ , p1 ’ , p2 ’ , b These model parameters, the model parameters of the third coordinate transformation model are obtained through adjustment in the calibration stage, so they can be regarded as calibrated model parameters, so these model parameters can also be called calibration parameters.

[0211] If the calibration parameters are read successfully, it can be considered that the third coordinate transformation model has been obtained before, and step S602 is executed. If the calibration parameters are not read successfully, it can be considered that the third coordinate transformation model has not been obtained before, so the correction process (ie step S605) will be executed to obtain the third coordinate transformation model.

[0212] Step S602: Use the third coordinate transformation model to perform face matching.

[0213] That is, the process of calculating the second distance between the third coordinate and the fourth coordinate in the aforementioned steps S301 to S304 and step S305 is executed.

[0214] If the second distance meets the preset matching condition, that is, the match is successful, then it can be considered that the faces of the people in the third infrared image and the third visible light image are the faces of the same person, and then step S603 can be executed, that is, the business process of face recognition and liveness detection. If the second distance does not meet the preset matching condition, that is, the match fails, then it can be considered that the faces of the people in the third infrared image and the third visible light image are not the faces of the same person, so step S603 will not be executed, that is, the business process of face recognition and liveness detection will not be executed, and step S604 will be executed instead.

[0215] Step S603, business process of face recognition and liveness detection.

[0216] That is, in step S305, facial liveness detection is performed on the third infrared image to obtain a facial liveness detection result. If the facial liveness detection result is liveness, facial recognition is performed on the third visible light image to obtain a facial recognition result.

[0217] Step S604, record the matching failures, and determine whether the matching failure records since the last calibration process was performed meet the preset recalibration conditions.

[0218] The preset recalibration condition can be set according to the user's needs and / or experience. For example, the preset recalibration condition can be that the frequency of records of matching failures since the most recent execution of the calibration process is higher than a preset frequency threshold, or that the number of consecutively recorded matching failures since the most recent execution of the calibration process exceeds a preset number threshold. In this article, consecutively recorded matching failures means that no matching success occurs between the two recorded matching failures.

[0219] The matching failure may be because the faces of the people in the third infrared image and the third visible light image are not the faces of the same person, or it may be because the third coordinate transformation model is not accurate enough due to bumps, vibrations, etc. of the binocular facial recognition device since the last calibration process was executed.

[0220] If the preset re-calibration conditions are met, it can be considered that the matching failures occur too frequently. It is understandable that the fact that the faces of the people in the third infrared image and the third visible light image are not the faces of the same person is a low-probability event in itself, and therefore does not occur frequently. Therefore, if the matching failures occur frequently, it can be considered that the reason for the matching failure is that the third coordinate transformation model is not accurate enough, so the calibration process needs to be re-executed at this time.

[0221] If the preset recalibration conditions are not met, there is no need to discard the recorded calibration parameters.

[0222] Step S605, calibration process.

[0223] Step S606, after the calibration process is completed, the latest third coordinate transformation relationship is recorded.

[0224] As described above, recording the latest third coordinate transformation relationship can be regarded as recording the aforementioned calibration parameters.

[0225] For the convenience of description, the above examples are all described under case B and case B'. The principles of the above cases A and A' and C and C' are the same, so detailed examples are not given here. Only the binocular facial recognition device provided by the present application is described without loss of generality:

[0226] In a possible embodiment, the processor is further used to record a matching failure if the second distance does not meet a preset matching condition; send a correction instruction to the visible light lens module and the infrared lens module if the record of matching failure since the most recent execution of saving the latest second coordinate transformation model as the third coordinate transformation model meets a preset re-calibration condition; and use the third coordinate transformation model as the first coordinate transformation model, return to execute the step of performing facial key point detection on the second visible light image to obtain the first facial key point.

[0227] In a possible embodiment, the processor is also used to read the third coordinate transformation model in response to the binocular facial recognition device being powered on; if the reading fails, a correction instruction is sent to the visible light lens module and the infrared lens module, and facial key point detection is performed on the second visible light image to obtain the first facial key point.

[0228] Corresponding to the binocular facial recognition device provided in the present application, the present application also provides a binocular facial recognition device calibration method, including:

[0229] Step A: acquiring a plurality of first visible light images captured by a visible light lens module in a binocular facial recognition device, and acquiring a plurality of first infrared images captured by an infrared lens module in a binocular facial recognition device.

[0230] Wherein, each first infrared image is taken synchronously with a first visible light image, and different first infrared images are taken synchronously with different first visible light images;

[0231] Step B, performing facial key point detection on the second visible light image to obtain first facial key points; and performing facial key point detection on the second infrared image to obtain second facial key points;

[0232] Step C, using the first coordinate transformation model to transform the first facial key point and / or the second facial key point to obtain the first coordinate of the first facial key point in the target coordinate system and the second coordinate of the second facial key point in the target coordinate system.

[0233] The second visible light image is initially any first visible light image, the second infrared image is a first infrared image captured synchronously with the second visible light image, and the model parameters of the first coordinate transformation model are initially internal parameters and external parameters pre-set for the visible light lens module and the infrared lens module;

[0234] Step D, calculating a first distance between the first coordinate and the second coordinate;

[0235] Step E, adjusting the model parameters of the first coordinate transformation model in a direction in which the first distance decreases, to obtain a second coordinate transformation model;

[0236] Step F, using the second coordinate transformation model as the new first coordinate transformation model and the visible light image in the first visible light image that has not been used as the second visible light image as the new second visible light image, returning to execute the step of performing facial key point detection on the second visible light image to obtain the first facial key points, until the preset correction end condition is met, and saving the latest second coordinate transformation model as the third coordinate transformation model.

[0237] In a possible embodiment, the method further includes:

[0238] In response to the binocular facial recognition device being powered on, reading a third coordinate conversion model;

[0239] If the reading fails, the steps of acquiring a plurality of first visible light images captured by the visible light lens module in the binocular facial recognition device and acquiring a plurality of first infrared images captured by the infrared lens module in the binocular facial recognition device are performed.

[0240] In a possible embodiment, the internal parameters include the focal length of the lens module, the pixel size of the lens module, the position of the imaging center of the lens module, and the distortion coefficient of the lens module;

[0241] The external parameters include the offset of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module, and the deflection angle of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module;

[0242] Among them, the focal length, pixel size, and imaging center position are initially set to the focal length, pixel size, and imaging center position of the lens module rated by the manufacturer of the lens module; the distortion coefficient is initially set to 0;

[0243] The offset is initially set to a rated offset between the optical axis of the visible light lens module and the optical axis of the infrared lens module; the deflection angle is initially set to 0.

[0244] Corresponding to the binocular facial recognition device provided in the present application, the present application also provides a binocular facial recognition method, including:

[0245] Step A', obtaining a third visible light image captured by the visible light lens module in the binocular facial recognition device, and obtaining a third infrared image captured by the infrared lens module in the binocular facial recognition device.

[0246] The third infrared image is taken synchronously with the third visible light image.

[0247] Step B', reading the third coordinate transformation model obtained according to any of the above-mentioned binocular facial recognition device calibration methods.

[0248] Step C', performing facial key point detection on the third visible light image to obtain third facial key points.

[0249] Step D', performing facial key point detection on the third infrared image to obtain fourth facial key points.

[0250] Step E', using the third coordinate transformation model to transform the third facial key point and / or the fourth facial key point to obtain the third coordinate of the third facial key point in the target coordinate system and the fourth coordinate of the fourth facial key point in the target coordinate system.

[0251] Step F', calculating a second distance between the third coordinate and the fourth coordinate.

[0252] Step G': if the second distance meets the preset matching condition, perform facial liveness detection on the third infrared image to obtain a facial liveness detection result.

[0253] Step H': if the facial liveness detection result is liveness, perform facial recognition on the third visible light image to obtain a facial recognition result.

[0254] In a possible embodiment, the method further includes:

[0255] If the second distance does not meet the preset matching condition, the matching failure is recorded;

[0256] If the records of failed matching since the most recent execution of any of the aforementioned binocular facial recognition device calibration methods meet the preset recalibration conditions, the third coordinate transformation model is used as the first coordinate transformation model, and any of the aforementioned binocular facial recognition device calibration methods is executed to obtain a new third coordinate transformation model.

[0257] The present application also provides an electronic device, such as Figure 7 As shown, including:

[0258] Memory 701, used for storing computer programs;

[0259] The processor 702 is used to execute the program stored in the memory 701, and implements the following steps:

[0260] Acquire a plurality of first visible light images captured by a visible light lens module in a binocular facial recognition device, and acquire a plurality of first infrared images captured by an infrared lens module in the binocular facial recognition device, wherein each first infrared image is captured synchronously with a first visible light image, and different first infrared images are captured synchronously with different first visible light images;

[0261] Performing facial key point detection on the second visible light image to obtain first facial key points; and performing facial key point detection on the second infrared image to obtain second facial key points;

[0262] The first facial key point and / or the second facial key point are transformed by using a first coordinate transformation model to obtain a first coordinate of the first facial key point in a target coordinate system and a second coordinate of the second facial key point in the target coordinate system, wherein the second visible light image is initially an arbitrary first visible light image, the second infrared image is a first infrared image obtained by synchronously shooting with the second visible light image, and the model parameters of the first coordinate transformation model are initially internal parameters and external parameters pre-set for the visible light lens module and the infrared lens module;

[0263] Calculate a first distance between the first coordinate and the second coordinate;

[0264] Adjusting the model parameters of the first coordinate transformation model in a direction in which the first distance decreases to obtain a second coordinate transformation model;

[0265] The second coordinate transformation model is used as the new first coordinate transformation model, and the visible light image in the first visible light image that has not been used as the second visible light image is used as the new second visible light image. The step of performing facial key point detection on the second visible light image to obtain the first facial key points is returned to the step until the preset correction end condition is met, and the latest second coordinate transformation model is saved as the third coordinate transformation model.

[0266] Alternatively, implement the following steps:

[0267] Acquire a third visible light image captured by a visible light lens module in the binocular facial recognition device, and acquire a third infrared image captured by an infrared lens module in the binocular facial recognition device, wherein the third infrared image is captured synchronously with the third visible light image;

[0268] Reading a third coordinate transformation model obtained according to any of the aforementioned binocular facial recognition device calibration methods;

[0269] Performing facial key point detection on the third visible light image to obtain third facial key points;

[0270] Performing facial key point detection on the third infrared image to obtain fourth facial key points;

[0271] Performing coordinate transformation on the third facial key point and / or the fourth facial key point using the third coordinate transformation model to obtain a third coordinate of the third facial key point in the target coordinate system and a fourth coordinate of the fourth facial key point in the target coordinate system;

[0272] calculating a second distance between the third coordinate and the fourth coordinate;

[0273] If the second distance satisfies a preset matching condition, performing facial liveness detection on the third infrared image to obtain a facial liveness detection result;

[0274] If the facial liveness detection result is liveness, facial recognition is performed on the third visible light image to obtain a facial recognition result.

[0275] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 702, the communication interface, and the memory 701 communicate with each other via the communication bus.

[0276] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0277] The communication interface is used for communication between the above electronic device and other devices.

[0278] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0279] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0280] In another embodiment provided by the present application, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned binocular facial recognition device calibration methods or binocular facial recognition methods are implemented.

[0281] In another embodiment provided by the present application, a computer program product including instructions is also provided. When the computer is run on a computer, the computer executes any binocular facial recognition device calibration method or binocular facial recognition method in the above embodiments.

[0282] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0283] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0284] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the binocular facial recognition device calibration method, binocular facial recognition method, and electronic device, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0285] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A binocular facial recognition device, characterized in that: The binocular facial recognition device includes a visible light lens module, an infrared lens module and a processor; The visible light lens module is used to capture a plurality of first visible light images and send them to the processor in response to the calibration instruction; The infrared lens module is used to take a plurality of first infrared images in response to the calibration instruction and send them to the processor, wherein each first infrared image is taken synchronously with a first visible light image, and different first infrared images are taken synchronously with different first visible light images; The processor is used to perform facial key point detection on the second visible light image to obtain first facial key points; and perform facial key point detection on the second infrared image to obtain second facial key points; perform coordinate transformation on the first facial key points and / or the second facial key points using a first coordinate transformation model to obtain first coordinates of the first facial key points in a target coordinate system and second coordinates of the second facial key points in the target coordinate system, and calculate a first distance between the first coordinate and the second coordinate, and adjust the model parameters of the first coordinate transformation model in a direction in which the first distance decreases to obtain a second coordinate transformation model; use the second coordinate transformation model as a new first coordinate transformation model and a visible light image in the first visible light image that has not been used as a second visible light image as a new second visible light image, return to perform the step of performing facial key point detection on the second visible light image to obtain the first facial key points, until a preset correction end condition is met, and save the latest second coordinate transformation model as a third coordinate transformation model; Wherein, the second visible light image is initially any first visible light image, the second infrared image is a first infrared image captured synchronously with the second visible light image, and the model parameters of the first coordinate transformation model are initially internal parameters and external parameters pre-set for the visible light lens module and the infrared lens module; The visible light lens module is further used to capture a third visible light image and send it to the processor in response to the recognition instruction; The infrared lens module is further used to take a third infrared image in response to the recognition instruction and send it to the processor, wherein the third infrared image is taken synchronously with the third visible light image; The processor is further used to perform facial key point detection on the third visible light image to obtain third facial key points; and perform facial key point detection on the third infrared image to obtain fourth facial key points; read and use the third coordinate transformation model to perform coordinate transformation on the third facial key points and / or the fourth facial key points to obtain third coordinates of the third facial key points in the target coordinate system and fourth coordinates of the fourth facial key points in the target coordinate system, and calculate a second distance between the third coordinates and the fourth coordinates. If the second distance meets a preset matching condition, perform facial liveness detection on the third infrared image to obtain a facial liveness detection result; if the facial liveness detection result is liveness, perform facial recognition on the third visible light image to obtain a facial recognition result.

2. The binocular facial recognition device according to claim 1, characterized in that: The processor is also used to record a matching failure if the second distance does not meet the preset matching condition; send a correction instruction to the visible light lens module and the infrared lens module if the record of matching failure since the last execution of saving the latest second coordinate transformation model as the third coordinate transformation model meets the preset re-calibration condition; and use the third coordinate transformation model as the first coordinate transformation model, return to execute the step of performing facial key point detection on the second visible light image to obtain the first facial key point.

3. The binocular facial recognition device according to claim 1, characterized in that: The processor is also used to read the third coordinate transformation model in response to the binocular facial recognition device being powered on; if the reading fails, a correction instruction is sent to the visible light lens module and the infrared lens module, and the step of detecting facial key points on the second visible light image to obtain the first facial key points is executed.

4. The binocular facial recognition device according to any one of claims 1 to 3, characterized in that: The internal parameters include the focal length of the lens module, the pixel size of the lens module, the position of the imaging center of the lens module, and the distortion coefficient of the lens module; The external parameters include an offset of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module, and a deflection angle of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module; Wherein, the focal length, the pixel size, and the position of the imaging center are initially set to the focal length, pixel size, and the position of the imaging center of the lens module rated by the manufacturer of the lens module; and the distortion coefficient is initially set to 0; The offset is initially set to a rated offset between the optical axis of the visible light lens module and the optical axis of the infrared lens module; and the deflection angle is initially set to 0.

5. A binocular facial recognition device calibration method, characterized in that: The method comprises: Acquire a plurality of first visible light images captured by a visible light lens module in a binocular facial recognition device, and acquire a plurality of first infrared images captured by an infrared lens module in the binocular facial recognition device, wherein each first infrared image is captured synchronously with a first visible light image, and different first infrared images are captured synchronously with different first visible light images; Performing facial key point detection on the second visible light image to obtain first facial key points; and performing facial key point detection on the second infrared image to obtain second facial key points; The first facial key point and / or the second facial key point are transformed by using a first coordinate transformation model to obtain a first coordinate of the first facial key point in a target coordinate system and a second coordinate of the second facial key point in the target coordinate system, wherein the second visible light image is initially an arbitrary first visible light image, the second infrared image is a first infrared image obtained by synchronously shooting with the second visible light image, and the model parameters of the first coordinate transformation model are initially internal parameters and external parameters pre-set for the visible light lens module and the infrared lens module; calculating a first distance between the first coordinate and the second coordinate; adjusting the model parameters of the first coordinate transformation model in a direction in which the first distance decreases, to obtain a second coordinate transformation model; Use the second coordinate transformation model as the new first coordinate transformation model and the visible light image in the first visible light image that has not been used as the second visible light image as the new second visible light image, return to execute the step of performing facial key point detection on the second visible light image to obtain the first facial key points, until the preset correction end condition is met, and save the latest second coordinate transformation model as the third coordinate transformation model.

6. The method according to claim 5, characterized in that The method further comprises: In response to the binocular facial recognition device being powered on, reading a third coordinate conversion model; If the reading fails, the steps of acquiring a plurality of first visible light images captured by the visible light lens module in the binocular facial recognition device and acquiring a plurality of first infrared images captured by the infrared lens module in the binocular facial recognition device are performed.

7. The method according to claim 5, characterized in that The internal parameters include the focal length of the lens module, the pixel size of the lens module, the position of the imaging center of the lens module, and the distortion coefficient of the lens module; The external parameters include an offset of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module, and a deflection angle of the optical axis of the visible light lens module relative to the optical axis of the infrared lens module; Wherein, the focal length, the pixel size, and the position of the imaging center are initially set to the focal length, pixel size, and the position of the imaging center of the lens module rated by the manufacturer of the lens module; and the distortion coefficient is initially set to 0; The offset is initially set to a rated offset between the optical axis of the visible light lens module and the optical axis of the infrared lens module; and the deflection angle is initially set to 0.

8. A binocular face recognition method, characterized in that: The method comprises: Acquire a third visible light image captured by a visible light lens module in a binocular facial recognition device, and acquire a third infrared image captured by an infrared lens module in the binocular facial recognition device, wherein the third infrared image is captured synchronously with the third visible light image; Reading a third coordinate transformation model obtained by the binocular facial recognition device calibration method according to any one of claims 5 to 7; Performing facial key point detection on the third visible light image to obtain third facial key points; Performing facial key point detection on the third infrared image to obtain fourth facial key points; Performing coordinate transformation on the third facial key point and / or the fourth facial key point using the third coordinate transformation model to obtain a third coordinate of the third facial key point in the target coordinate system and a fourth coordinate of the fourth facial key point in the target coordinate system; calculating a second distance between the third coordinate and the fourth coordinate; If the second distance satisfies a preset matching condition, performing facial liveness detection on the third infrared image to obtain a facial liveness detection result; If the facial liveness detection result is liveness, facial recognition is performed on the third visible light image to obtain a facial recognition result.

9. The method according to claim 8, characterized in that The method further comprises: If the second distance does not meet the preset matching condition, a matching failure is recorded; If the records of failed matching since the last execution of the binocular facial recognition device calibration method described in any one of claims 5-7 meet the preset re-calibration conditions, the third coordinate transformation model is used as the first coordinate transformation model, and the binocular facial recognition device calibration method described in any one of claims 5-7 is executed to obtain a new third coordinate transformation model.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 5-7 or 8-9 when executing a program stored in a memory.

Images