A 3D face recognition method based on deep learning

Abstract

A three-dimensional face recognition method based on deep learning includes: constructing a three-dimensional face deep learning network, calculating the high-frequency-depth map of each training sample, and then simultaneously inputting the color face two-dimensional image and the high-frequency-depth map of each training sample into the face three-dimensional deep learning network for training; using a color camera to obtain a color face two-dimensional image of a face to be tested, and then inputting the color face two-dimensional image of the face to be tested into the trained face three-dimensional deep learning network, and outputting the three-dimensional point cloud coordinates of the face to be tested; according to the color face two-dimensional image and the three-dimensional point cloud coordinates of the face to be tested, calculating a three-dimensional face feature vector, and then comparing the three-dimensional face feature vector of the face to be tested with the three-dimensional face feature vector of the registered face in the registration library, so as to identify the personnel information of the face to be tested. The present invention belongs to the field of information technology, and can make full use of the depth information of the face to effectively improve the reliability of the face recognition result.

Description

Technical Field

[0001] The invention relates to a three-dimensional face recognition method based on deep learning, and belongs to the field of information technology. Background Art

[0002] Face recognition refers to computer technology that uses analysis of facial visual feature information to authenticate identity. It is a non-contact recognition method with high concealment. It is widely used in areas with high safety requirements such as offices and construction sites.

[0003] In recent years, with the development of deep learning technology, face recognition technology using neural networks has become increasingly mature, and related applications have gradually been implemented. However, due to the loss of depth information in the face imaging process, the reliability of all recognition results is still insufficient.

[0004] Therefore, how to make full use of the depth information of the face to effectively improve the reliability of face recognition results has become one of the technical problems that need to be urgently solved in the existing technology. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a three-dimensional face recognition method based on deep learning, which can make full use of the depth information of the face to effectively improve the reliability of the face recognition result.

[0006] In order to achieve the above object, the present invention provides a three-dimensional face recognition method based on deep learning, comprising:

[0007] Step 1: construct a 3D deep learning network for faces, calculate the high-frequency-depth map of each training sample, and then simultaneously input the color 2D face image and the high-frequency-depth map of each training sample into the 3D deep learning network for training;

[0008] Step 2: Use a color camera to obtain a color 2D face image of the face to be tested, then input the color 2D face image of the face to be tested into the trained 3D face deep learning network, and output the 3D point cloud coordinates of the face to be tested;

[0009] Step 3: Calculate the 3D facial feature vector based on the color 2D facial image and 3D point cloud coordinates of the face to be tested, and then compare the 3D facial feature vector of the face to be tested with the 3D facial feature vectors of the registered faces in the registration library to identify the person information of the face to be tested.

[0010] Calculating the high frequency-depth map of any training sample X further includes:

[0011] Step 11: Perform Fourier spectrum transform on the color face two-dimensional image of the training sample X, so as to obtain a spectrum diagram of the training sample X;

[0012] Step 12: extract several key points from the color face two-dimensional image of the training sample X, then read the spectrum value of each key point from the spectrum graph of the training sample X, and calculate the cutoff value D of the high-pass filter function based on it. 0 , D 0 is the average of the spectrum values ​​of all key points;

[0013] Step 13: Set a high-pass filter function, and pass the spectrum of the training sample X through the high-pass filter function to obtain a filtered high-frequency image. The high-pass filter function is set as follows: Where D(u, v) is the spectrum value of the coordinate (u, v) on the spectrum graph of the training sample X, n is the order constant, which is 2 or 4, and H(u, v) is the spectrum value obtained after filtering;

[0014] Step 14: Perform inverse Fourier transform on the high-frequency image of the training sample X to obtain a high-frequency color face two-dimensional image;

[0015] Step 15: Compare the brightness value of each point in the high-frequency color face two-dimensional image with the threshold, and replace the brightness value above the threshold with the depth value. After all points are compared, the obtained high-frequency color face two-dimensional image is the high-frequency-depth map of the training sample X.

[0016] Compared with the prior art, the beneficial effects of the present invention are: the present invention adds facial depth information to the training process of the neural network, making up for the lack of depth information, thereby effectively improving the reliability of face recognition results; the present invention only uses one camera, only needs to obtain a single image of the face to be tested, and combines deep learning and machine learning technologies, which greatly improves the speed, pass rate and anti-counterfeiting rate of face liveness detection, and has low cost and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of a three-dimensional face recognition method based on deep learning of the present invention.

[0018] Figure 2 It is a specific flow chart for calculating the high-frequency-depth map of any training sample X.

[0019] Figure 3 yes Figure 1 In step three, a specific flow chart of calculating the three-dimensional face feature vector based on the color two-dimensional face image and the three-dimensional point cloud coordinates of the face to be tested. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0021] like Figure 1As shown, the present invention provides a three-dimensional face recognition method based on deep learning, comprising:

[0022] Step 1: construct a 3D deep learning network for faces, calculate the high-frequency-depth map of each training sample, and then simultaneously input the color 2D face image and the high-frequency-depth map of each training sample into the 3D deep learning network for training;

[0023] Step 2: Use a color camera to obtain a color 2D face image of the face to be tested, then input the color 2D face image of the face to be tested into the trained 3D face deep learning network, and output the 3D point cloud coordinates of the face to be tested;

[0024] Step 3: Calculate the 3D facial feature vector based on the color 2D facial image and 3D point cloud coordinates of the face to be tested, and then compare the 3D facial feature vector of the face to be tested with the 3D facial feature vectors of the registered faces in the registration library to identify the person information of the face to be tested.

[0025] When registering a face, a 3D sensor can be used to accurately record its three-dimensional information and generate a three-dimensional face feature vector, and finally the three-dimensional face feature vector of the registered face is stored in the registration library.

[0026] The specific structure of the face 3D deep learning network in step 1 is as follows:

[0027] It is implemented based on the tensorflow framework and adopts the encoding-decoding (i.e. encoder-decoder) structure: in the encoder part, it contains 1 convolution layer and 7 residual layers, so that the input 250*250*3 two-dimensional face image can be converted into a 16*16*1024 feature map; in the decoder part, 9 deconvolution layers are set, so that the feature map can be converted into a 250*250*1 three-dimensional depth map, and then through the fixed conversion relationship between the three-dimensional depth map and the point cloud, the three-dimensional depth map is converted into a three-dimensional point cloud output. The kernel size of the convolution layer and the deconvolution layer is 4, and the activation function uses Softplus. In this way, when a 250*250*3 color face two-dimensional image is input, the output of the face three-dimensional deep learning network is a 250*250 three-dimensional point cloud coordinate (a total of 62,500).

[0028] It is worth mentioning that when training the three-dimensional deep learning network of the face, the present invention can calculate the high-frequency-depth map of each training sample, and simultaneously input the color two-dimensional face image and the high-frequency-depth map of each training sample into the three-dimensional deep learning network of the face for training, so that in the model parameters obtained after training, the weight of the general features in the face image (i.e., the features that are more common in the face and difficult to use for identification) is reduced, and the weight of the individual features (i.e., the features that are more individual in the face and help to accurately identify) is increased, which significantly improves the accuracy of face recognition. Figure 2 As shown, calculating the high frequency-depth map of any training sample X may further include:

[0029] Step 11: Perform Fourier spectrum transform on the color face two-dimensional image of the training sample X, so as to obtain a spectrum diagram of the training sample X;

[0030] Step 12: extract several key points from the color two-dimensional face image of the training sample X. The key points may be eyebrows, eyes, nose, mouth, facial contour, etc., and then read the spectrum value of each key point from the spectrum graph of the training sample X, and calculate the cutoff value D of the high-pass filter function accordingly. 0 , D 0 is the average of the spectrum values ​​of all key points;

[0031] Step 13: Set a high-pass filter function, and pass the spectrum of the training sample X through the high-pass filter function to obtain a filtered high-frequency image. The high-pass filter function is set as follows: Where D(u, v) is the spectrum value of the coordinate (u, v) on the spectrum graph of the training sample X, n is the order constant, which is 2 or 4, and H(u, v) is the spectrum value obtained after filtering; this can effectively remove the low-frequency information and obtain the required high-frequency information;

[0032] Step 14: Perform inverse Fourier transform on the high-frequency image of the training sample X to obtain a high-frequency color face two-dimensional image;

[0033] Step 15, compare the brightness value of each point in the high-frequency color face two-dimensional image with the threshold, and replace the brightness value above the threshold with the depth value: determine whether the brightness value of each point is higher than the threshold, if yes, read the three-dimensional coordinates of the point from the three-dimensional point cloud of the training sample X, and convert the read three-dimensional coordinates into a depth value, and then replace the brightness value of the point with the depth value; if no, continue to determine the next point, when all points are compared, the obtained high-frequency color face two-dimensional image is the high-frequency-depth map of the training sample X. The three-dimensional point cloud of the training sample X can be obtained by collecting the training sample using a depth camera, and the threshold can be set according to actual business needs, for example, 20.

[0034] like Figure 3 As shown, in step 3, calculating the 3D face feature vector according to the color 2D face image and 3D point cloud coordinates of the face to be tested may further include:

[0035] Step 31, after performing face detection, cropping, and alignment on the two-dimensional image of the face to be tested, the image is converted into a floating-point matrix, and then the corresponding two-dimensional image feature vector is calculated, where the two-dimensional image feature vector is a one-dimensional vector with a size of 512;

[0036] Step 32: According to the three-dimensional point cloud coordinates of the face to be tested, a Cartesian xyz rectangular coordinate system is established with the nose tip as the coordinate origin, the line connecting the two eyes as the horizontal axis direction x, and the line connecting the nose tip and the middle of the lips as the vertical axis direction y, so as to obtain the three-dimensional point cloud matrix of the face, and then calculate the corresponding three-dimensional point cloud feature vector, which is a three-dimensional vector with a size of 3*512;

[0037] Step 33, set the weight factor, fuse the two-dimensional image feature vector and the three-dimensional point cloud feature vector, the feature vector obtained after fusion is the three-dimensional face feature vector, among which the weight factors of the two-dimensional face feature, x-direction feature, y-direction feature, and z-direction feature can be set to: 0.5, 0.17, 0.17, and 0.16, respectively.

[0038] In step three, comparing the three-dimensional facial feature vector of the face to be tested with the three-dimensional facial feature vector of any registered face in the registration library may further include:

[0039] Step A1: Calculate the cosine similarity between the three-dimensional facial feature vector A of the face to be tested and the three-dimensional facial feature vector B of the registered face. The calculation formula may be: Where A·B represents the dot product of vectors A and B, ||A|| 2 represents the L2 norm of vector A, ||B|| 2 represents the L2 norm of vector B;

[0040] Step A2, calculate the cosine distance between the face to be tested and the registered face: dist(A, B)=1-cos(A, B). The larger the cosine distance, the lower the similarity between the two faces, and judge whether the face to be tested is a registered face based on this.

[0041] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional face recognition method based on deep learning, characterized in that: Included are: Step 1: construct a 3D deep learning network for faces, calculate the high-frequency-depth map of each training sample, and then simultaneously input the color 2D face image and the high-frequency-depth map of each training sample into the 3D deep learning network for training; Step 2: Use a color camera to obtain a color 2D face image of the face to be tested, then input the color 2D face image of the face to be tested into the trained 3D face deep learning network, and output the 3D point cloud coordinates of the face to be tested; Step 3: Calculate the 3D facial feature vector based on the color 2D facial image and 3D point cloud coordinates of the face to be tested, and then compare the 3D facial feature vector of the face to be tested with the 3D facial feature vectors of the registered faces in the registration library to identify the person information of the face to be tested. Calculating the high frequency-depth map of any training sample X further includes: Step 11: Perform Fourier spectrum transform on the color face two-dimensional image of the training sample X, so as to obtain a spectrum diagram of the training sample X; Step 12: extract several key points from the color two-dimensional face image of the training sample X, then read the spectrum value of each key point from the spectrum diagram of the training sample X, and calculate the cutoff value D0 of the high-pass filter function based on it, where D0 is the average value of the spectrum values ​​of all key points; Step 13: Set a high-pass filter function, and pass the spectrum of the training sample X through the high-pass filter function to obtain a filtered high-frequency image. The high-pass filter function is set as follows: Where D(u,v) is the spectrum value of the coordinate (u,v) on the spectrum graph of the training sample X, n is the order constant, which is 2 or 4, and H(u,v) is the spectrum value obtained after filtering; Step 14: Perform inverse Fourier transform on the high-frequency image of the training sample X to obtain a high-frequency color face two-dimensional image; Step 15: Compare the brightness value of each point in the high-frequency color face two-dimensional image with the threshold, and replace the brightness value above the threshold with the depth value. After all points are compared, the obtained high-frequency color face two-dimensional image is the high-frequency-depth map of the training sample X.

2. The method according to claim 1, characterized in that The specific structure of the face 3D deep learning network in step 1 is as follows: It is implemented based on the tensorflow framework and adopts an encoder-decoder structure: in the encoder part, it contains 1 convolution layer and 7 residual layers, which converts the input 250*250*3 two-dimensional face image into a 16*16*1024 feature map; in the decoder part, 9 deconvolution layers are set to convert the feature map into a 250*250*1 three-dimensional depth map, and then through the fixed conversion relationship between the three-dimensional depth map and the point cloud, the three-dimensional depth map is converted into a three-dimensional point cloud output. The kernel size of the convolution layer and the deconvolution layer is 4, and the activation function uses Softplus.

3. The method according to claim 1, characterized in that Step 15 further includes: Determine whether the brightness value of each point is higher than the threshold. If yes, read the 3D coordinates of the point from the 3D point cloud of training sample X, convert the read 3D coordinates into depth values, and then replace the brightness value of the point with the depth value; if no, continue to determine the next point.

4. The method according to claim 1, characterized in that: In step three, the three-dimensional face feature vector is calculated according to the color two-dimensional face image and the three-dimensional point cloud coordinates of the face to be tested, which further includes: Step 31, after face detection, cropping and alignment, the color two-dimensional face image of the face to be tested is converted into a floating-point matrix, and then the corresponding two-dimensional image feature vector is calculated, and the two-dimensional image feature vector is a one-dimensional vector with a size of 512; Step 32: According to the three-dimensional point cloud coordinates of the face to be tested, a Cartesian xyz rectangular coordinate system is established with the nose tip as the coordinate origin, the line connecting the two eyes as the horizontal axis direction x, and the line connecting the nose tip and the middle of the lips as the vertical axis direction y, so as to obtain the three-dimensional point cloud matrix of the face, and then calculate the corresponding three-dimensional point cloud feature vector, which is a three-dimensional vector with a size of 3*512; Step 33: Set a weight factor to fuse the two-dimensional image feature vector and the three-dimensional point cloud feature vector. The feature vector obtained after fusion is the three-dimensional face feature vector.

5. The method according to claim 4, characterized in that In step 33, the weight factors of the two-dimensional facial features, x-direction features, y-direction features, and z-direction features are set to 0.5, 0.17, 0.17, and 0.16, respectively.

6. The method according to claim 1, characterized in that In step three, the three-dimensional facial feature vector of the face to be tested is compared with the three-dimensional facial feature vector of any registered face in the registration library, further comprising: Step A1: Calculate the cosine similarity between the three-dimensional facial feature vector A of the face to be tested and the three-dimensional facial feature vector B of the registered face. The calculation formula is: Among them, A·B represents the dot product of vectors A and B, ‖A‖2 represents the L2 norm of vector A, and ‖B‖2 represents the L2 norm of vector B; Step A2: Calculate the cosine distance between the face to be tested and the registered face: dist(A, B) = 1-cos(A, B), and determine whether the face to be tested is a registered face based on this.

Images