Multi-angle face recognition method based on 3D intelligent reconstruction

Through three-dimensional intelligent reconstruction and deep learning technology, the problem of imperfect data sets and reconstruction complexity in multi-angle face recognition is solved, and high accuracy and fast multi-angle face recognition are achieved, which is suitable for a wide range of application scenarios.

CN114882545BActive Publication Date: 2024-11-08NANJING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210336825.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-31
Publication Date
2024-11-08
Estimated Expiration
2042-03-31

AI Technical Summary

Technical Problem

The existing multi-angle facial recognition technology has problems such as incomplete data sets, high lighting requirements for traditional three-dimensional reconstruction and unsuitable for real scenarios, and the need for human manipulation, resulting in limited application.

Method used

Using a multi-angle face recognition method based on three-dimensional intelligent reconstruction, the front face feature point extraction and ASM algorithm are used to reconstruct the shape and texture vectors of the front face feature point, and the cascaded residual network are predicted, and the three-dimensional face model is constructed and rotated. Match, and finally compare with the external database for identification.

Benefits of technology

It achieves high accuracy of side face recognition with large angle deflection, and improves operating speed through deep learning, which is suitable for a wide range of application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114882545B_ABST
    Figure CN114882545B_ABST
Patent Text Reader

Abstract

The present invention discloses a multi-angle face recognition method based on three-dimensional intelligent reconstruction. A face picture is input, the face feature points in the picture are extracted, the face deflection angle is calculated and the front face feature points are reconstructed in combination with the ASM algorithm; the shape vector and texture vector in the deformation model are predicted by using a cascade residual network, a three-dimensional face gray model is established according to the predicted shape vector, and it is matched with the front face feature points at the same time; then coloring is performed according to the texture vector to construct a complete three-dimensional face model; the three-dimensional face model is rotated according to the face deflection angle, and the similarity is compared with the face of the input picture to obtain the final three-dimensional face model and complete face recognition. The present invention realizes multi-angle face recognition based on three-dimensional reconstruction, and also uses a cascade residual network to accelerate the calculation process of the deformation model, and forms a self-feedback network to perform self-feedback training and punishment on the deformation model, which has a very broad application scenario.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a multi-angle face recognition method based on three-dimensional intelligent reconstruction. Background Art

[0002] With the rapid development of Internet technology, the ability of algorithms to process data has also been greatly improved. With the rapid development of big data technology, there will be hidden dangers such as information security, making related technical means such as information identification and detection particularly important. The emergence of face recognition technology solves these problems. Compared with the traditional means of using magnetic cards or ID cards with magnetic passwords, face recognition, as an inherent feature of the human body, is fixed for a single person and cannot be repeated for different people. The face is a biological feature determined by each person's unique and unchanging genes. It is both simple and fast to identify identity through the face. Therefore, the rapid development of face recognition is a necessary trend for social progress.

[0003] With the rapid development of deep learning, the field of face recognition has also achieved great success, but it is still not perfect. Compared with frontal static face recognition, the development of multi-angle face recognition technology in social scenes has encountered many bottlenecks. One of the main reasons why the development of multi-angle face recognition is not as good as frontal face recognition is that the data set is not perfect. Various face data sets led by Google all use frontal faces, but faces are three-dimensional features, and the deflection of face angles should also be a major research area. Dynamic face recognition is crucial to realizing the full potential of face recognition in life, because multi-angle face recognition is essentially a passive recognition technology that recognizes non-cooperative objects. In recent years, face 3D reconstruction technology has further developed, and various 3D reconstruction technologies have been widely used. Traditional 3D reconstruction based on picture depth technology has too high requirements for lighting effects and is not suitable for real scenes. Although the 3D reconstruction based on multi-view stereo vision has a good reconstruction effect, it is not an automated process and requires human control in the background. Therefore, the use of deformation models for 3D face reconstruction is still one of the main research directions.

[0004] It can be seen that the multi-angle face recognition technology in the existing technology has the problem of incomplete data collection; the traditional three-dimensional reconstruction based on picture depth technology is not suitable for real scenes because of its high requirements on lighting; the traditional three-dimensional reconstruction based on multi-eye stereo vision requires background control, which is too troublesome to implement. Summary of the invention

[0005] In view of the above problems, the present invention proposes a multi-angle face recognition method based on three-dimensional intelligent reconstruction.

[0006] To achieve the purpose of the present invention, a multi-angle face recognition method based on three-dimensional intelligent reconstruction is provided, comprising the following steps:

[0007] s1: input a face picture, extract facial feature points from the face picture, calculate facial feature quantities of each facial area, and then calculate facial deflection angles by scene based on the number of extracted facial feature points;

[0008] s2: Based on the facial feature values ​​of each facial region and the facial deflection angle, the extracted facial feature points are reconstructed into frontal facial feature points using an ASM algorithm;

[0009] s3: constructing an internal face database and a deformation model, using a cascade residual network trained with samples to predict the shape vector and texture vector of the deformation model based on the internal face database, wherein the internal face database stores a training data set of three-dimensional face images used for prediction;

[0010] s4: construct a three-dimensional face gray model according to the predicted shape vector, and then use the reconstructed front face feature points to match the three-dimensional face gray model; if the match is successful, the three-dimensional face gray model is colored and supplemented with details based on the texture vector to establish a three-dimensional face model; if the match fails, return to step s3 to perform self-feedback penalty training on the deformation model;

[0011] s5: after rotating the three-dimensional face model according to the face deflection angle, performing face similarity matching between the input face image and the rotated three-dimensional face model, and obtaining a face similarity SIM, and then performing scene-by-scene operations according to the face similarity SIM, and obtaining a final three-dimensional face model;

[0012] s6: Compare the final 3D face model with an external face database to complete the final face recognition.

[0013] Furthermore, in the step s1, there are 68 facial feature points in total, and the facial feature values ​​of each facial region include: eye feature value, nose bridge feature value, mouth feature value and jaw feature value;

[0014] The calculation formula of the human eye feature quantity is:

[0015] Among them, D eye Represents the characteristic quantity of human eye, and They represent the sum of the distances between the two feature points of the upper eyelid and the corresponding two feature points of the lower eyelid. Indicates the horizontal distance between the feature points of the left and right eye corners;

[0016] The calculation formula of the nose bridge feature is:

[0017] Among them, D nose represents the characteristic value of the nose bridge, Indicates the vertical distance between the highest feature point and the lowest feature point of the middle bridge of the nose, Indicates the horizontal line distance between the lowest feature point of the middle nose bridge and the rightmost feature point of the nose;

[0018] The calculation formula of the mouth feature is:

[0019]

[0020] Among them, D mouth represents the mouth feature quantity, Respectively represent the vertical distances between the three groups of upper and lower feature points of the upper lip, Respectively represent the vertical distances between the three groups of upper and lower feature points of the lower lip. and Respectively represent the horizontal connection distance between the left and right corners of the upper and lower lips;

[0021] The calculation formula of the mandibular feature value is:

[0022] Among them, D mandible represents the mandibular feature quantity, represents the vertical distance between the lowest mandibular feature point and the highest mandibular feature point, Represents the lateral distance between the lowest mandibular feature point and the highest mandibular feature point.

[0023] Furthermore, in step s1, the scene-based calculation formula of the face deflection angle is as follows:

[0024]

[0025] in, represents the face deflection angle, Ω represents the number of extracted facial feature points, Represents the distance between the two eyes of the face after rotation, Represents the distance from the mouth to the center of the two eyes after deflection, R 31 , R 32 , R 33 They represent the variables in the rotation matrix, Λ is a constant, θ1 represents the oblique deflection angle, and θ2 represents the horizontal deflection angle;

[0026] If 65≤Ω≤68, use the first formula to calculate the face deflection angle; if 48≤Ω≤64, use the second formula to calculate the face deflection angle; if Ω<48, use the third formula to calculate the face deflection angle.

[0027] Furthermore, in step s2, the reconstruction of the frontal facial feature points is expressed by the following model:

[0028]

[0029] in, represents the average shape vector, Π represents the new face sample feature points composed of these principal component shape vectors; b * represents a vector that controls the weight of feature points, represents the covariance matrix obtained after extracting the principal component shape vector, represents the average human eye feature, represents the average nose bridge feature, represents the average mouth feature quantity, Represents the average mandibular feature value.

[0030] Furthermore, in step s3, the structure of the sample-trained cascade residual network includes the following:

[0031] The three-dimensional face is uniformly represented in vector form:

[0032]

[0033] in, and represent the shape vector and texture vector of the i-th face in the internal face database respectively, represents the gray model of a face composed of shape vectors, Represents the facial details composed of texture vectors, a i and b i Respectively represent the shape vector and texture vector of the i-th face, Represents the number of faces in the internal face database; assuming that the rendered model face image is I model (x mat ,y mat ), the target face image is I input (x mat ,y mat ), then the objective function is:

[0034]

[0035] Among them, ||·|| 2 represents the square of the 1-norm, (x mat ,y mat ) represents the pixel coordinates used for 3D face model matching, E I represents the variance between the model face and the target face, I R,model (xmat ,y mat ), I G,model (x mat ,y mat ), I B,model (x mat ,y mat ) represent the three model face images corresponding to the three primary color RGB channels, and T represents the transposed matrix;

[0036] In order to implement the cascaded residual network with mathematical formula, is the activation function, is the convolution function, and the output of the mth cascade residual block is defined as And after two convolutional layers, the residuals are added to get the following formula:

[0037]

[0038] Among them, O m-1 represents the output of the previous 1*1 convolutional layer connected to the mth cascaded residual block, W m represents the convolutional layer connected to the mth cascaded residual block, W m,1 represents the first convolutional layer connected to the mth cascaded residual block, W m ,2 represents the second convolutional layer connected to the mth residual block, and “⊙” represents the cascade operation;

[0039] Use the loss function:

[0040]

[0041] Among them, l MSE represents the loss function, r o represents the scaling factor, I LR represents a low-resolution face image, I HR represents a high-resolution face image, W id Indicates the width of the image, H ig Indicates the height of the image. Represents the generated frontal face image.

[0042] Furthermore, in the step s5, after rotating the three-dimensional face model according to the face deflection angle, the face image is matched with the rotated three-dimensional face model for face similarity, and the specific process of obtaining the face similarity SIM is as follows:

[0043] First, convert the RGB color space to the Lup color space through the XYZ color space. The conversion formula is as follows:

[0044]

[0045] Among them, (X col ,Y col ,Z col ) represents the coordinates of the XYZ color space, (R col ,G col ,B col ) represents the coordinates in RGB color space;

[0046] Then, the XYZ color space is converted to the Lup color space, and the conversion formula is as follows:

[0047]

[0048]

[0049] Among them, (L c ,u c ,p c ) represents the coordinates of the Lup color space, t c and All represent intermediate variables;

[0050] The face comparison process is as follows:

[0051]

[0052] Among them, Pic1 and Pic2 represent the two pictures to be compared. and Represent the pixel value variance of the two images respectively, and Represent the pixel mean of the two images respectively, Represents the covariance of two images, LU represents brightness similarity, CON represents contrast similarity, and STR represents structural similarity;

[0053] The calculation formula of face similarity SIM is:

[0054]

[0055] Among them, SIM represents the final face similarity, and its range is [0,1].

[0056] Furthermore, in step s5, the specific process of performing scene operations according to the face similarity SIM and obtaining the final three-dimensional face model includes the following:

[0057] When the face similarity SIM is less than or equal to 60%, the number of three-dimensional face images in the internal face database is first expanded, and then the step s3 is repeated to perform self-feedback penalty training on the deformation model;

[0058] When the face similarity is 60%≤SIM≤85%, repeating step s3 to perform self-feedback penalty training on the deformation model;

[0059] When the face similarity SIM is ≥85%, the 3D face model at this time is the final 3D face model.

[0060] Compared with the prior art, the present invention has the following beneficial technical effects:

[0061] The present invention can use the method of three-dimensional intelligent reconstruction of the face to perform multi-angle face recognition, and integrates the method of reconstructing the feature points of the frontal face. It has high recognition accuracy for side faces with large angle deflection, and uses deep learning methods to improve the operating speed, and has a very broad application scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a schematic flow chart of a multi-angle face recognition method based on three-dimensional intelligent reconstruction according to an embodiment;

[0063] Figure 2 is a schematic diagram of 68 facial feature points in one embodiment;

[0064] Figure 3 is a diagram representing the distance of human eyes according to an embodiment;

[0065] Figure 4 is a nose bridge distance representation diagram of an embodiment;

[0066] Figure 5 A mouth distance representation diagram for one embodiment;

[0067] Figure 6 A diagram representing the mandibular distance according to an embodiment;

[0068] Figure 7 A schematic diagram of a face deflection angle classification scenario according to an embodiment;

[0069] Figure 8 A schematic diagram of a cascaded residual network structure according to an embodiment. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0071] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] refer to Figure 1 As shown, Figure 1 The flowchart of a multi-angle face recognition method based on three-dimensional intelligent reconstruction according to an embodiment includes the following steps:

[0073] s1: input a face picture, extract facial feature points from the face picture, calculate facial feature quantities of each facial area, and then calculate facial deflection angles by scene based on the number of extracted facial feature points;

[0074] The face image can be a frontal image or a non-frontal image.

[0075] s2: Based on the facial feature values ​​of each facial region and the facial deflection angle, the extracted facial feature points are reconstructed into frontal facial feature points using an ASM algorithm;

[0076] s3: constructing an internal face database and a deformation model, using a cascade residual network trained with samples to predict the shape vector and texture vector of the deformation model based on the internal face database, wherein the internal face database stores a training data set of three-dimensional face images used for prediction;

[0077] s4: construct a three-dimensional face gray model according to the predicted shape vector, and then use the reconstructed front face feature points to match the three-dimensional face gray model; if the match is successful, the three-dimensional face gray model is colored and supplemented with details based on the texture vector to establish a three-dimensional face model; if the match fails, return to step s3 to perform self-feedback penalty training on the deformation model;

[0078] s5: after rotating the three-dimensional face model according to the face deflection angle, performing face similarity matching between the input face image and the rotated three-dimensional face model, and obtaining a face similarity SIM, and then performing scene-by-scene operations according to the face similarity SIM, and obtaining a final three-dimensional face model;

[0079] s6: Compare the final 3D face model with an external face database to complete the final face recognition.

[0080] In one embodiment, in step s1, the following is selected: Figure 2The 68 feature points shown are used as facial feature points, and the facial feature quantities of each facial area include: Figure 3 The human eye features shown in Figure 4 The nose bridge feature quantity shown in Figure 5 The mouth features shown and Figure 6 Mandibular feature quantities shown;

[0081] The calculation formula of the human eye feature quantity is:

[0082] Among them, D eye Represents the characteristic quantity of human eye, and They represent the sum of the distances between the two feature points of the upper eyelid and the corresponding two feature points of the lower eyelid. Indicates the horizontal distance between the feature points of the left and right eye corners;

[0083] The calculation formula of the nose bridge feature is:

[0084] Among them, D nose represents the characteristic value of the nose bridge, Indicates the vertical distance between the highest feature point and the lowest feature point of the middle bridge of the nose, Indicates the horizontal line distance between the lowest feature point of the middle nose bridge and the rightmost feature point of the nose;

[0085] The calculation formula of the mouth feature is:

[0086]

[0087] Among them, D mouth represents the mouth feature quantity, Respectively represent the vertical distances between the three groups of upper and lower feature points of the upper lip, Respectively represent the vertical distances between the three groups of upper and lower feature points of the lower lip. and Respectively represent the horizontal connection distance between the left and right corners of the upper and lower lips;

[0088] The calculation formula of the mandibular feature value is:

[0089] Among them, D mandible represents the mandibular feature quantity, represents the vertical distance between the lowest mandibular feature point and the highest mandibular feature point, Represents the lateral distance between the lowest mandibular feature point and the highest mandibular feature point.

[0090] In one embodiment, Figure 7As shown in the figure, according to the number of detected facial feature points, it is divided into three scenes. When 65 to 68 facial feature points are detected, it is classified as scene (1), and the corresponding face deflection angle range is 0° to 23°; when 48 to 64 facial feature points are detected, it is classified as scene (2), and the corresponding face deflection angle range is 23° to 67°; when less than 48 feature points are detected, it is classified as scene (3), and the corresponding face deflection angle range is 67°-90°.

[0091] Assume that the angle of rotation around the x-axis is α, the angle of rotation around the y-axis is β, and the angle of rotation around the z-axis is γ. The corresponding total rotation matrix is:

[0092]

[0093] Let the vertices of the three-dimensional model be (X mod ,Y mod ,Z mod ), the translation vector is (T x ,T y ,T z ) T , the image feature point is pi(u s ,v s ),

[0094] have:

[0095]

[0096] Among them, s r is a constant, K r is the adjustment vector, denoted as:

[0097] s r pi = HI;

[0098] The matrix H is the conversion factor from two-dimensional to three-dimensional, called the unit matrix. Pi is the eigenvector of the image feature point, and I is the eigenvector of the three-dimensional model vertex. The internal structure of the unit matrix is:

[0099]

[0100] Among them, fu is the camera parameter, and the least squares method is used to solve the camera parameter fu:

[0101]

[0102] Given a nonlinear system of equations to be solved:

[0103]

[0104] The scene-based calculation formula for the face deflection angle is as follows:

[0105]

[0106] in, represents the face deflection angle, Ω represents the number of extracted facial feature points, Represents the distance between the two eyes of the face after rotation, Represents the distance from the mouth to the center of the two eyes after deflection, R 31 , R 32 , R 33 They represent the variables in the rotation matrix, Λ is a constant, θ1 represents the oblique deflection angle, and θ2 represents the horizontal deflection angle;

[0107] If 65≤Ω≤68, use the first formula to calculate the face deflection angle; if 48≤Ω≤64, use the second formula to calculate the face deflection angle; if Ω<48, use the third formula to calculate the face deflection angle.

[0108] In one embodiment, in step s2, the reconstruction of the frontal facial feature points is expressed by the following model:

[0109]

[0110] in, represents the average shape vector, Π represents the new face sample feature points composed of these principal component shape vectors; b * represents a vector that controls the weight of feature points, represents the covariance matrix obtained after extracting the principal component shape vector, represents the average human eye feature, represents the average nose bridge feature, represents the average mouth feature quantity, Represents the average mandibular feature value.

[0111] Use the constructed mapping model to test the new point set of the new image and match it, that is, use the model area to describe the feature point set in the image

[0112]

[0113]

[0114] in, and Represents the translation change of each feature point, represents the transformation matrix, It represents a mapping method that converts the feature points of the original image into the feature points of the frontal face. Represents the feature point set of the new image, s u represents the zoom scale, Represents the face deflection angle, corresponding to the calculation methods in the three different scenarios in the previous section, x ASM and ASM Represents each feature point that needs to be transformed.

[0115] To describe and The matching similarity between them is calculated by establishing an objective function to represent the matching degree:

[0116]

[0117] Among them, |·| 2 represents the square of the modulus, is the newly established objective function, parameter b * The initialization value is set to 0. At this time, the face model is the average face model. The transformation matrix is ​​solved by rotation and translation transformation. Then Through the transformation matrix Mapped to the face model to get a point set

[0118]

[0119] in, express The inverse matrix of the new point set Mapping to the average point set In the tangent plane, make the following scale transformation:

[0120]

[0121] in, Represents the new point set after scale transformation, and the new shape parameters are obtained through the shape model

[0122]

[0123] in, Represents the covariance matrix The transposed matrix of If it converges to the given range, it means that the positioning is successful. If it does not converge, continue to search for the feature point set in the face according to the new Π solved. Until the shape parameter convergence.

[0124] When judging the face measurement standard, a distance error between the target object shape model point cloud and the neighboring boundary point cloud is introduced as the matching judgment criterion. The vector Υ represents the shape model point cloud, Υ' represents the neighboring edge point cloud, and the distance error is:

[0125]

[0126] in, represents the distance error, and |·| represents the modulus. Assuming a certain facial feature point is input, sample points from the left and right sides of the feature point and construct a vector g face , then we can get the series set of frontal facial feature points (ordered) and {g face}.

[0127] In one embodiment, a cascaded residual network is used to predict the shape vector and texture vector of the deformable model, and the algorithm flow of the deformable model for calculating the shape vector and texture vector is abandoned. In step s3, the structure of the cascaded residual network trained by the sample includes the following:

[0128] The three-dimensional face is uniformly represented in vector form:

[0129]

[0130] in, and represent the shape vector and texture vector of the i-th face in the internal face database respectively, represents the gray model of a face composed of shape vectors, Represents the facial details composed of texture vectors, a i and b i Respectively represent the shape vector and texture vector of the i-th face, Represents the number of faces in the internal face database; assuming that the rendered model face image is I model (x mat ,y mat ), the target face image is I input (x mat ,y mat ), then the objective function is:

[0131]

[0132] Among them, ||·|| 2 represents the square of the 1-norm, (x mat ,y mat ) represents the pixel coordinates used for 3D face model matching, E I represents the variance between the model face and the target face, I R,model (x mat ,y mat ), I G,model (x mat ,y mat ), I B,model (x mat ,ymat ) represent the three model face images corresponding to the three primary color RGB channels, and T represents the transposed matrix.

[0133] like Figure 8 As shown, in order to implement the cascade residual network with mathematical formula, let is the activation function, is the convolution function, and the output of the mth cascade residual block is defined as And after two convolutional layers, the residuals are added to get the following formula:

[0134]

[0135] Among them, O m-1 represents the output of the previous 1*1 convolutional layer connected to the mth cascaded residual block, W m represents the convolutional layer connected to the mth cascaded residual block, W m,1 represents the first convolutional layer connected to the mth cascaded residual block, W m ,2 represents the second convolutional layer connected to the mth residual block, and “⊙” represents the cascade operation;

[0136] Use the loss function:

[0137]

[0138] Among them, l MSE represents the loss function, r o represents the scaling factor, I LR represents a low-resolution face image, I HR represents a high-resolution face image, W id Indicates the width of the image, H ig Indicates the height of the image. Represents the generated frontal face image.

[0139] In one embodiment, in step s5, after rotating the three-dimensional face model according to the face deflection angle, the face image is matched with the rotated three-dimensional face model for face similarity, and the specific process of obtaining the face similarity SIM is as follows:

[0140] First, convert the RGB color space to the Lup color space through the XYZ color space. The conversion formula is as follows:

[0141]

[0142] Among them, (X col ,Y col ,Z col ) represents the coordinates of the XYZ color space, (Rcol ,G col ,B col ) represents the coordinates in RGB color space;

[0143] Then, the XYZ color space is converted to the Lup color space, and the conversion formula is as follows:

[0144]

[0145]

[0146] Among them, (L c ,u c ,p c ) represents the coordinates of the Lup color space, t c and All represent intermediate variables;

[0147] The face comparison process is as follows:

[0148]

[0149] Among them, Pic1 and Pic2 represent the two pictures to be compared. and Represent the pixel value variance of the two images respectively, and Represent the pixel mean of the two images respectively, Represents the covariance of two images, LU represents brightness similarity, CON represents contrast similarity, and STR represents structural similarity;

[0150] The calculation formula of face similarity SIM is:

[0151]

[0152] Among them, SIM represents the final face similarity, and its range is [0,1].

[0153] In one embodiment, in step s5, the specific process of performing scene operations according to the face similarity SIM and obtaining the final three-dimensional face model includes the following:

[0154] When the face similarity SIM is less than or equal to 60%, the number of three-dimensional face images in the internal face database is first expanded, and then the step s3 is repeated to perform self-feedback penalty training on the deformation model;

[0155] When the face similarity is 60%≤SIM≤85%, repeating step s3 to perform self-feedback penalty training on the deformation model;

[0156] When the face similarity SIM is ≥85%, the 3D face model at this time is the final 3D face model.

[0157] When the face similarity is 85%≤SIM, it means that the deep learning prediction result is accurate and no self-feedback training penalty is required; when the face similarity is SIM≤60%, it means that the face 3D reconstruction result is seriously inconsistent with the input image. It is speculated that the main reason is that the 3D face database data set is insufficient and the number of faces in the internal face database needs to be expanded; when the face similarity satisfies 60%≤SIM≤85%, it means that the deep learning prediction result is inaccurate and self-feedback training penalty is required:

[0158] From the above, we can see that the matching of 3D face model is a multi-iteration process. The iterative form of the k-th layer 3D face model error is expressed as:

[0159]

[0160] Among them, (x mat ,y mat ) represents the pixel value coordinates used for 3D face model matching, represents the target face image after k iterations, represents the 3D model face image after k iterations, represents the variance after k iterations;

[0161] The combined face vector after k iterations is as follows:

[0162]

[0163] in, represents the face shape vector of the 3D model after k iterations, represents the face texture vector of the 3D model after k iterations, represents the average face shape after k iterations, represents the average face texture after k iterations, a k represents the shape vector after k iterations, b k represents the texture vector after k iterations, represents the principal components of the shape vector after k iterations, Represents the principal component of the texture vector after k iterations;

[0164] The iterative relationship between texture vectors and shape vectors from low to high levels is as follows:

[0165]

[0166] in, represents the transformation matrix of the shape vector after k iterations, represents the transformation matrix of the texture vector after k iterations, and express and The transposed matrix of

[0167] The number of iterations is determined by the face similarity SIM. The lower the face similarity SIM, the more iterations; the higher the face similarity SIM, the fewer iterations;

[0168] The formula for calculating the number of iterations is as follows:

[0169]

[0170] Among them, S ta represents the sample standard deviation of the internal face database, λ cap represents the sample size, SIM represents the face similarity, ω represents the significance level, ξ represents a constant, represents the confidence level, represents the historical iteration mean, k up represents the upper limit of the number of iterations, k down represents the lower limit of the number of iterations, k final Indicates the final number of iterations.

[0171] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0172] It should be noted that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects, and do not represent a specific order for the objects. It is understandable that the specific order or sequence of "first\second\third" can be interchanged where permitted. It should be understood that the objects distinguished by "first\second\third" can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0173] The terms "including" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products, or equipment.

[0174] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-angle face recognition method based on three-dimensional intelligent reconstruction, characterized in that: The steps include: s1: input a face picture, extract facial feature points from the face picture, calculate facial feature quantities of each facial area, and then calculate facial deflection angles by scene based on the number of extracted facial feature points; s2: Based on the facial feature values ​​of each facial region and the facial deflection angle, the extracted facial feature points are reconstructed into frontal facial feature points using an ASM algorithm; s3: constructing an internal face database and a deformation model, using a cascade residual network trained with samples to predict the shape vector and texture vector of the deformation model based on the internal face database, wherein the internal face database stores a training data set of three-dimensional face images used for prediction; s4: constructing a three-dimensional face gray model according to the predicted shape vector, and then matching the reconstructed frontal face feature points with the three-dimensional face gray model; If the match is successful, the three-dimensional face gray model is colored and detailed based on the texture vector, thereby establishing a three-dimensional face model; if the match fails, the process returns to step s3 to perform self-feedback penalty training on the deformable model; s5: after rotating the three-dimensional face model according to the face deflection angle, performing face similarity matching between the input face image and the rotated three-dimensional face model, and obtaining a face similarity SIM, and then performing scene-by-scene operations according to the face similarity SIM, and obtaining a final three-dimensional face model; s6: Compare the final 3D face model with an external face database to complete the final face recognition.

2. The multi-angle face recognition method based on three-dimensional intelligent reconstruction according to claim 1 is characterized in that: In the step s1, there are 68 facial feature points in total, and the facial feature values ​​of each facial region include: eye feature value, nose bridge feature value, mouth feature value and jaw feature value; The calculation formula of the human eye feature quantity is: Among them, D eye Represents the characteristic quantity of human eye, and They represent the sum of the distances between the two feature points of the upper eyelid and the corresponding two feature points of the lower eyelid. Indicates the horizontal distance between the feature points of the left and right eye corners; The calculation formula of the nose bridge feature is: Among them, D nose represents the characteristic value of the bridge of nose, Indicates the vertical distance between the highest feature point and the lowest feature point of the middle bridge of the nose, Indicates the horizontal line distance between the lowest feature point of the middle nose bridge and the rightmost feature point of the nose; The calculation formula of the mouth feature is: Among them, D mouth represents the mouth feature quantity, Respectively represent the vertical distances between the three groups of upper and lower feature points of the upper lip, Respectively represent the vertical distances between the three groups of upper and lower feature points of the lower lip. and Respectively represent the horizontal connection distance between the left and right corners of the upper and lower lips; The calculation formula of the mandibular feature value is: Among them, D mandible represents the mandibular feature quantity, represents the vertical distance between the lowest mandibular feature point and the highest mandibular feature point, Represents the lateral distance between the lowest mandibular feature point and the highest mandibular feature point.

3. The multi-angle face recognition method based on three-dimensional intelligent reconstruction according to claim 2 is characterized in that: In step s1, the scene-based calculation formula for the face deflection angle is as follows: in, represents the face deflection angle, Ω represents the number of extracted facial feature points, Represents the distance between the two eyes of the face after rotation, Represents the distance from the mouth to the center of the two eyes after deflection, R 31 , R 32 , R 33 They represent the variables in the rotation matrix, Λ is a constant, θ1 represents the oblique deflection angle, and θ2 represents the horizontal deflection angle; If 65≤Ω≤68, use the first formula to calculate the face deflection angle; if 48≤Ω≤64, use the second formula to calculate the face deflection angle; if Ω<48, use the third formula to calculate the face deflection angle.

4. The multi-angle face recognition method based on three-dimensional intelligent reconstruction according to claim 3 is characterized in that: In step s2, the reconstruction of the frontal facial feature points is expressed by the following model: in, represents the average shape vector, Π represents the new face sample feature points composed of these principal component shape vectors; b * represents a vector that controls the weight of feature points, represents the covariance matrix obtained after extracting the principal component shape vector, represents the average human eye feature, represents the average nose bridge feature, represents the average mouth feature quantity, Represents the average mandibular feature value.

5. The multi-angle face recognition method based on three-dimensional intelligent reconstruction according to claim 4 is characterized in that: In step s3, the structure of the cascaded residual network trained with samples includes the following: The three-dimensional face is uniformly represented in vector form: in, and represent the shape vector and texture vector of the i-th face in the internal face database respectively, represents the gray model of a face composed of shape vectors, Represents the facial details composed of texture vectors, a i and b i Respectively represent the shape vector and texture vector of the i-th face, Represents the number of faces in the internal face database; assuming that the rendered model face image is I model (x mat ,y mat ), the target face image is I input (x mat ,y mat ), then the objective function is: Among them, ||·|| 2 represents the square of the 1-norm, (x mat ,y mat ) represents the pixel coordinates used for 3D face model matching, E I represents the variance between the model face and the target face, I R,model (x mat ,y mat ), I G,model (x mat ,y mat ), I B,model (x mat ,y mat ) represent the three model face images corresponding to the three primary color RGB channels, and T represents the transposed matrix; In order to implement the cascaded residual network with mathematical formula, is the activation function, is the convolution function, and the output of the mth cascade residual block is defined as And after two convolutional layers, the residuals are added to get the following formula: Among them, O m-1 represents the output of the previous 1*1 convolutional layer connected to the mth cascaded residual block, W m represents the convolutional layer connected to the mth cascaded residual block, W m,1 represents the first convolutional layer connected to the mth cascaded residual block, W m,2 represents the second convolutional layer connected to the mth residual block, and "⊙" represents the cascade operation; Use the loss function: Among them, l MSE represents the loss function, r o represents the scaling factor, I LR represents a low-resolution face image, I HR represents a high-resolution face image, W id Indicates the width of the image, H ig Indicates the height of the image. Represents the generated frontal face image.

6. The multi-angle face recognition method based on three-dimensional intelligent reconstruction according to claim 5 is characterized in that: In the step s5, after rotating the three-dimensional face model according to the face deflection angle, the input face image is matched with the rotated three-dimensional face model for face similarity, and the specific process of obtaining the face similarity SIM is as follows: First, convert the RGB color space to the Lup color space through the XYZ color space. The conversion formula is as follows: Among them, (X col ,Y col ,Z col ) represents the coordinates of the XYZ color space, (R col ,G col ,B col ) represents the coordinates in RGB color space; Then, the XYZ color space is converted to the Lup color space, and the conversion formula is as follows: Among them, (L c ,u c ,p c ) represents the coordinates of the Lup color space, t c and All represent intermediate variables; The face comparison process is as follows: Among them, Pic1 and Pic2 represent the two pictures to be compared. and Represent the pixel value variance of the two images respectively, and Represent the pixel mean of the two images respectively, Represents the covariance of two images, LU represents brightness similarity, CON represents contrast similarity, and STR represents structural similarity; The calculation formula of face similarity SIM is: Among them, SIM represents the final face similarity, and its range is [0,1].

7. The multi-angle face recognition method based on three-dimensional intelligent reconstruction according to claim 6 is characterized in that: In step s5, the specific process of performing scene operations according to the face similarity SIM and obtaining the final three-dimensional face model includes the following: When the face similarity SIM is less than or equal to 60%, the number of three-dimensional face images in the internal face database is first expanded, and then the step s3 is repeated to perform self-feedback penalty training on the deformation model; When the face similarity is 60%≤SIM≤85%, repeating step s3 to perform self-feedback penalty training on the deformation model; When the face similarity SIM is ≥85%, the 3D face model at this time is the final 3D face model.