Living face recognition method based on infrared thermal image and RGB image

Abstract

The present invention discloses a method for live face recognition based on infrared thermal images and RGB images, comprising the following steps: performing face recognition on the infrared thermal image and the RGB image respectively to obtain face recognition results; in the face recognition results of S2, mutually projecting the face recognition results using the proportional relationship, and judging whether it is a live face based on the projection results. The method for live face recognition based on infrared thermal images and RGB images of the present invention first compares the infrared thermal image and the RGB image corresponding to the face to be recognized, and then calculates the proportional relationship between the infrared thermal image and the RGB image; then, performing face recognition on the infrared thermal image and the RGB image respectively to obtain face recognition results; mutually projecting the face recognition results using the proportional relationship, and thus judging whether it is a live face based on the projection results, thereby reducing the influence of the surrounding environment during face recognition and improving the reliability of face recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of face recognition, and in particular relates to a living face recognition method based on infrared thermal images and RGB images. Background Art

[0002] Facial recognition is a key research direction in the field of artificial intelligence, and it is widely used in identity authentication, facial payment, and human-computer interaction.

[0003] Currently, face recognition mostly uses traditional two-dimensional face recognition methods. However, changes in the lighting environment will cause changes in the reflection of light on the face surface, thereby changing the texture of the two-dimensional face image. Different lighting environments will produce different shadows on the two-dimensional face image, causing some features in the face image to disappear. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a living face recognition method based on infrared thermal images and RGB images, which reduces the influence of the surrounding environment during face recognition and improves the reliability of face recognition.

[0005] The technical solution adopted in the present invention is:

[0006] A method for live face recognition based on infrared thermal images and RGB images, comprising the following steps:

[0007] S1. Compare the infrared thermal image and RGB image corresponding to the face to be identified, and then calculate the ratio between the infrared thermal image and the RGB image;

[0008] S2. Performing face recognition on the infrared thermal image and the RGB image respectively to obtain face recognition results;

[0009] S3. In the face recognition results of S2, the face recognition results are mutually projected using the proportional relationship, and whether it is a living face is determined according to the projection results.

[0010] Preferably, the method further includes: S4, detecting whether there is a blinking action on the infrared thermal image and the RGB image within a fixed time, and then doubly judging whether it is a living human face.

[0011] Preferably, the proportional relationship includes a magnification scale coefficient and a regional center of gravity offset, and the magnification scale coefficient includes a length magnification scale coefficient and a width magnification scale coefficient.

[0012] Preferably, the S3 specifically includes the following steps:

[0013] S31, multiplying the length and width of the infrared thermal image by corresponding magnification coefficients respectively to obtain an amplified infrared thermal image;

[0014] S32, after binarizing the facial region identified in the RGB image, calculating the centroid coordinates of the RGB image projected onto the amplified infrared thermal image using the corresponding regional centroid offset, and determining whether there are points exceeding a preset ratio projected onto the amplified infrared thermal image;

[0015] After binarizing the facial region identified in the amplified infrared thermal image, the centroid coordinates of the amplified infrared thermal image projected onto the RGB image are calculated using the corresponding regional centroid offset, and it is determined whether there are points exceeding a preset ratio projected onto the RGB image;

[0016] S33. If more than a preset proportion of points in the RGB image are projected onto the magnified infrared thermal image, and more than a preset proportion of points in the magnified infrared thermal image are projected onto the RGB image, then it is determined to be a living face; otherwise, it is not a living face.

[0017] Preferably, the S4 specifically includes the following steps:

[0018] S41, detecting the positions of facial features in the face recognition result;

[0019] S42, using a fixed time window to sequentially slide and collect facial images;

[0020] S43. Determine whether there are a first fixed number of consecutive images of an open-eye state and a second fixed number of consecutive images of a closed-eye state. If so, determine that the face has blinking behavior and is a living face. Otherwise, determine that the face has no blinking behavior and is not a living face.

[0021] Preferably, the facial features are eyes.

[0022] Preferably, in S41, an open source computer vision function library is used to detect the positions of facial features based on the face recognition result.

[0023] Preferably, in S2, face recognition is performed on the infrared thermal image and the RGB image using a support vector machine.

[0024] Preferably, in S2, face recognition is performed on the infrared thermal image and RGB image using a recursive convolutional neural network algorithm.

[0025] Preferably, the infrared thermal imager and the RGB camera are placed in parallel, and the coordinate system of the thermal imager and the coordinate system of the RGB camera contain collinear coordinate axes.

[0026] Compared with the prior art, the live face recognition method based on infrared thermal images and RGB images of the present invention first compares the infrared thermal image and RGB image corresponding to the face to be recognized, and then calculates the proportional relationship between the infrared thermal image and the RGB image. Then, face recognition is performed on the infrared thermal image and the RGB image respectively to obtain the face recognition result. The face recognition results are projected onto each other using the proportional relationship, so as to determine whether it is a live face based on the projection result, reduce the influence of the surrounding environment during face recognition, and improve the reliability of face recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a method for live face recognition based on infrared thermal images and RGB images provided by an embodiment of the present invention;

[0028] Figure 2 This is a specific flow chart of a living face recognition method based on infrared thermal images and RGB images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] The embodiment of the present invention provides a living face recognition method based on infrared thermal image and RGB image, such as Figure 1-2 As shown, the following steps are included:

[0031] S1. Compare the infrared thermal image and RGB image corresponding to the face to be identified, and then calculate the ratio between the infrared thermal image and the RGB image;

[0032] S2. Performing face recognition on the infrared thermal image and the RGB image respectively to obtain face recognition results;

[0033] S3. In the face recognition results of S2, the face recognition results are mutually projected using the proportional relationship, and whether it is a living face is determined according to the projection results.

[0034] In this way, by first comparing the infrared thermal image and the RGB image corresponding to the face to be identified, and then calculating the proportional relationship between the infrared thermal image and the RGB image, and then performing face recognition on the infrared thermal image and the RGB image respectively, obtaining the face recognition result, and using the proportional relationship to project the face recognition results onto each other, it is determined whether it is a living face based on the projection result, thereby reducing the influence of the surrounding environment during face recognition and improving the reliability of face recognition.

[0035] The infrared thermal imager and RGB camera are placed in parallel, and the coordinate system of the thermal imager and the coordinate system of the RGB camera contain collinear coordinate axes. That is, the infrared thermal imager and the RGB camera constitute a binocular stereo vision system. During installation, the infrared thermal imager and the RGB camera are placed in parallel, that is, the optical axes of the two cameras are parallel to each other, and the coordinate system of the thermal imager and the coordinate system of the visible light camera contain collinear coordinate axes. In this way, the imaging planes of the two cameras are as coplanar as possible, and the distance between the two cameras is a fixed value. It is assumed here that there is no rotation angle between the two installed cameras (i.e., the rotation angle of the three axes is 0), and only a translation vector, i.e., the translation vector T.

[0036] Among them, the RGB color model is a color standard in the industry. It obtains various colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other. RGB represents the colors of the three channels of red, green, and blue. This standard covers almost all colors that human vision can perceive and is one of the most widely used color systems. RGB images refer to images displayed using the RGB color model. Such images can only be viewed through media that use light to represent colors, such as televisions and computers.

[0037] The OpenCV library is a cross-platform computer vision library released under the BSD license (open source), running on Linux, Windows, Android, and Mac OS. It is lightweight and efficient—consisting of a series of C functions and a small number of C++ classes. It also provides interfaces for languages like Python, Ruby, and MATLAB, implementing many common algorithms for image processing and computer vision.

[0038] The method further includes: S4, detecting whether there is a blinking action in the infrared thermal image and the RGB image within a fixed time, and then dually judging whether it is a living human face.

[0039] In this way, when there is an error in the face recognition method in S3, it is also possible to detect whether there is a blinking action on the infrared thermal image and RGB image within a fixed time, and then double-judge whether it is a living face, thereby improving the accuracy of face recognition.

[0040] The proportional relationship includes an enlargement proportional coefficient and an area center of gravity offset, and the enlargement proportional coefficient includes a length enlargement proportional coefficient and a width enlargement proportional coefficient.

[0041] The S3 specifically includes the following steps:

[0042] S31, multiplying the length and width of the infrared thermal image by corresponding magnification coefficients respectively to obtain an amplified infrared thermal image;

[0043] S32, after binarizing the facial region identified in the RGB image, calculating the centroid coordinates of the RGB image projected onto the amplified infrared thermal image using the corresponding regional centroid offset, and determining whether there are points exceeding a preset ratio projected onto the amplified infrared thermal image;

[0044] After binarizing the facial region identified in the amplified infrared thermal image, the centroid coordinates of the amplified infrared thermal image projected onto the RGB image are calculated using the corresponding regional centroid offset, and it is determined whether there are points exceeding a preset ratio projected onto the RGB image;

[0045] S33. If more than a preset proportion of points in the RGB image are projected onto the magnified infrared thermal image, and more than a preset proportion of points in the magnified infrared thermal image are projected onto the RGB image, then it is determined to be a living face; otherwise, it is not a living face.

[0046] In this way, through S31, the length and width of the infrared image are multiplied by k respectively. x and k y ;

[0047] S32, the face identified on the RGB image is binarized, the background is represented by 0, and the face area is represented by 255, so as to calculate the centroid coordinates (c xv ,c yv ), projecting each point on the face onto the image captured by the infrared imager;

[0048] That is, calculate the center of gravity offset Δx = c vr -c xv , Δy=c yr -c yv , project each point on the face onto the image of the infrared imager, and let the coordinates on the RGB image be (x v ,y v ), the image formed on the thermal imager is the image with coordinates (x r ,y r ), the coordinates of the point projected from the RGB image to the infrared imager area are (x′ r ,y′ r ); the corresponding relationship is:

[0049]

[0050] If more than 60% of the points in the face area detected from the RGB image are projected onto the infrared imager area, it is T vr .

[0051] The face area extracted from the image on the infrared imager is binarized, with the background represented by 0 and the face area represented by 255, so as to calculate the centroid coordinates of the face area (c xr ,c yr ), projecting the face area on the infrared imager image onto the RGB image;

[0052] That is, any coordinate point in the face area detected from the image obtained by the infrared imager is projected onto the RGB image, and the coordinates are expressed as (x′ v ,y′ v ), the correspondence between the face area detected on the infrared imager and the RGB image:

[0053]

[0054] If more than 60% of the points in the face area detected by the infrared imager are projected onto the RGB detection area, it is T rv ;

[0055] S33. If A=T vr &T rv If it is true, it means that it can be judged as a face with temperature, and the result is represented by A.

[0056] In the S2, face recognition is performed on the infrared thermal image and RGB image respectively. The recursive convolutional neural network algorithm (i.e., FasterR-CNN) can be used to replace the traditional target extraction method with network training to achieve this, which greatly improves the detection and classification speed of the entire process. The basic structure is as follows: ① Feature extraction part: Use a series of convolutions + pooling to extract the feature map from the original image; ② RPN part: Get the approximate position of the target from the feature map through network training; ③ Proposal Layer part: Use the approximate position obtained by RPN to continue training to obtain a more accurate position; ④ ROI Pooling part: Use the precise position obtained previously to extract the target to be used for classification from the feature map and pool it into fixed-length data; Among them, face recognition can also be performed on the infrared thermal image and RGB image through support vector machine (i.e., OpenCV function library).

[0057] Among them, network training refers to providing enough samples to the network, adjusting the network structure (mainly adjusting the weights) through a certain algorithm, so that the network output is consistent with the expected value. This process is neural network training.

[0058] The S4 specifically includes the following steps:

[0059] S41, detecting the positions of facial features in the face recognition result;

[0060] S42, using a fixed time window to sequentially slide and collect facial images;

[0061] S43. Determine whether there are a first fixed number of consecutive images of an open-eye state and a second fixed number of consecutive images of a closed-eye state. If so, determine that the face has blinking behavior and is a living face. Otherwise, determine that the face has no blinking behavior and is not a living face.

[0062] In this way, the face with temperature judged may lead to judgment errors. The blinking of the person is judged again. First, the open source computer vision function library is used to detect the position of the facial features in the face recognition result, and a fixed time window is used to slide and collect facial images in sequence. Then, it is judged whether there are a first fixed number of frames of continuous images in the open-eye state and a second fixed number of frames of continuous images in the closed-eye state. If so, it is judged that the person has blinking behavior and is a living face. Otherwise, it is judged that the person has no blinking behavior and is not a living face.

[0063] Among them, the five senses are specifically eyes.

[0064] In this way, the corresponding face is detected through the five senses including the two eyes. At the same time, in the later more accurate face recognition process, other facial features such as the nose, mouth, eyebrows, ears, etc. can also be detected.

[0065] The live face recognition method based on infrared thermal images and RGB images of the present invention first compares the infrared thermal image and RGB image corresponding to the face to be recognized, and then calculates the proportional relationship between the infrared thermal image and the RGB image. Then, face recognition is performed on the infrared thermal image and the RGB image respectively to obtain the face recognition result. The face recognition results are projected onto each other using the proportional relationship, so as to determine whether it is a live face based on the projection result, reduce the influence of the surrounding environment during face recognition, and improve the reliability of face recognition.

[0066] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for live face recognition based on infrared thermal images and RGB images, characterized in that: The steps include: S1. Compare the infrared thermal image and RGB image corresponding to the face to be identified, and then calculate the ratio between the infrared thermal image and the RGB image; S2. Performing face recognition on the infrared thermal image and the RGB image respectively to obtain face recognition results; S3. In the face recognition results of S2, project the face recognition results onto each other using the proportional relationship, and determine whether the face is a living face based on the projection results; The S3 specifically includes the following steps: S31, multiplying the length and width of the infrared thermal image by corresponding magnification coefficients respectively to obtain an amplified infrared thermal image; S32, after binarizing the facial region identified in the RGB image, calculating the centroid coordinates of the RGB image projected onto the amplified infrared thermal image using the corresponding region centroid offset, and determining whether there are points exceeding a preset ratio projected onto the amplified infrared thermal image; After binarizing the facial region identified in the amplified infrared thermal image, the centroid coordinates of the amplified infrared thermal image projected onto the RGB image are calculated using the corresponding regional centroid offset, and it is determined whether there are points exceeding a preset ratio projected onto the RGB image; S33. If more than a preset proportion of points in the RGB image are projected onto the magnified infrared thermal image, and more than a preset proportion of points in the magnified infrared thermal image are projected onto the RGB image, then it is determined to be a living face; otherwise, it is not a living face.

2. The method for live face recognition based on infrared thermal images and RGB images according to claim 1, characterized in that: Also includes: S4. Detect whether there is blinking action in the infrared thermal image and RGB image within a fixed time, and then double-judge whether it is a living face.

3. The method for live face recognition based on infrared thermal images and RGB images according to claim 1, characterized in that: The proportional relationship includes an enlargement proportional coefficient and an area center of gravity offset, and the enlargement proportional coefficient includes a length enlargement proportional coefficient and a width enlargement proportional coefficient.

4. The method for live face recognition based on infrared thermal images and RGB images according to claim 2, characterized in that: The S4 specifically includes the following steps: S41, detecting the positions of facial features in the face recognition result; S42, using a fixed time window to sequentially slide and collect facial images; S43. Determine whether there are a first fixed number of continuous images of an open-eye state and a second fixed number of continuous images of a closed-eye state. If so, determine that the face has blinking behavior and is a living face. Otherwise, determine that the face has no blinking behavior and is not a living face.

5. The method for live face recognition based on infrared thermal images and RGB images according to claim 4, characterized in that: The five sense organs are specifically eyes.

6. The method for live face recognition based on infrared thermal images and RGB images according to claim 4, characterized in that: In the S41, an open source computer vision function library is used to detect the positions of the facial features based on the face recognition result.

7. The method for live face recognition based on infrared thermal images and RGB images according to any one of claims 1 to 6, characterized in that: In S2, face recognition is performed on the infrared thermal image and the RGB image using a support vector machine.

8. The method for live face recognition based on infrared thermal images and RGB images according to any one of claims 1 to 6, characterized in that: In S2, face recognition is performed on the infrared thermal image and the RGB image using a recursive convolutional neural network algorithm.

9. The method for live face recognition based on infrared thermal images and RGB images according to any one of claims 1 to 6, characterized in that: The infrared thermal imager and the RGB camera are placed in parallel, and the coordinate system of the thermal imager and the coordinate system of the RGB camera contain collinear coordinate axes.

Images