Face recognition method and device based on three-dimensional mixed deformation model

Abstract

The invention relates to a face recognition method and device based on a three-dimensional mixed deformation model, and belongs to the field of computer image processing. A hair style matching step and an image matching step. According to the invention, accurate feature migration can be carried out between face models of different styles, so that the features of a real face can be accurately reflected in a cartoon style model.

Description

Technical Field

[0001] The present invention relates to the field of computer image processing, and in particular to a face recognition method and device based on a three-dimensional hybrid deformation model. Background Art

[0002] 3DMM is a technology for face modeling. It uses 3D models to represent and analyze faces, and can capture the shape and texture information of faces. The technology is mainly based on two methods: Gaussian process for modeling shape changes and Markov chain Monte Carlo sampling for fitting models. The combination of these two methods produces a completely probabilistic shape modeling method. This probabilistic deformable model is the core of 3DMM.

[0003] FaceParse is a technology for facial parsing that can decompose facial images into multiple parts, such as eyes, nose, mouth, etc. This technology uses a bilateral segmentation network to solve the problem of sacrificing resolution to improve inference. The technology first designs a spatial path with a smaller stride to retain spatial information and generate high-resolution features. At the same time, a context path with a fast downsampling strategy is used to obtain sufficient receptive fields. Finally, on top of the two paths, the technology introduces a new feature fusion module to effectively combine features.

[0004] US Patent US20160260183A1: "Method and system for creating apersonalized 3D model of a user's face". This patent describes a method and system for creating a personalized 3D model of a user's face. The technology involves capturing a set of images of the user's face and then using these images to create a 3D face model. The core of the technology is marching cubes, sampling distance fields. Image data format conversion, such as converting polar coordinates to rectangular coordinates or converting IGES to combined geometric descriptions. This model can be used in various applications such as virtual reality, augmented reality, games, etc.

[0005] The main problem with 3D face reconstruction technology is that it mainly supports realistic style reconstruction, which means that the generated 3D face model is mainly an accurate reflection of the real world. However, this method encounters some difficulties when migrating to different artistic styles, such as cartoons, anime or abstract art. These styles usually distort or simplify reality in some way, and existing 3D face reconstruction technology is not designed to handle this situation. In addition, these styles also introduce some unique features and elements, such as exaggerated proportions or specific color schemes, which are beyond the capabilities of existing 3D face reconstruction technology. Summary of the invention

[0006] The present invention is intended to provide a method and device for face recognition based on a three-dimensional hybrid deformation model to solve the deficiencies in the prior art. The technical problem to be solved by the present invention is achieved through the following technical solutions. The first aspect of the present invention provides a method for face recognition based on a three-dimensional hybrid deformation model, the method comprising:

[0007] Image preprocessing steps: input a face photo and standardize the face through preprocessing; in order to standardize the face, that is, to make the position and size of the input face the same, use MTCNN to detect five key points of the face, including the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth; next, calculate the midpoint of the two eyes as the eyebrow center, rotate the entire image around the eyebrow center until the horizontal coordinates of the two eyes are the same, and then take the eyebrow center as the center and half the eye distance, that is, the distance d from the eyebrow center to one eye as the unit length, and take a new box of 4d length from the top, bottom, left and right of the eyebrow center. The length and width of the box are 8d respectively, which completely contains the face, and the face position is the same and the size is similar, which is convenient for subsequent operations;

[0008] Hairstyle matching step: performing shape matching on the hairstyles in the hair area, taking out the hair area from the result based on the faceparse algorithm; performing shape matching on the hair area and various hairstyles in the asset library;

[0009] Image matching steps: extract the facial features and cheek area from the faceparse algorithm, and then use 3DMM to perform 3D reconstruction of the face:

[0010] Using the difference fitting method, the benchmark model selects a set of key points The cartoon-style model selects the corresponding point set at this time and It is a corresponding relationship;

[0011] Use 3DMM to reconstruct the face. 3DMM reconstruction is based on an optimized algorithm that fits the benchmark model to the face in the two-dimensional image. Here we define the benchmark model M:

[0012]

[0013] in and are the average shape and texture, S i ,e i and t i Represent shape parameters, expression parameters and texture parameters respectively, α i ,β i ,γ i Represent their weights respectively, n, l and k are natural numbers;

[0014] Several pre-set feature point positions are taken from the three-dimensional face generated by 3DMM, and the key point coordinate 2 is detected using the algorithm provided by Mediapipe; the pre-set feature point positions of the corresponding style are taken from the style asset library as key point coordinate 1; key point coordinates 1 and 2 need to be aligned because of different coordinate systems and scales, and after alignment, key point coordinates 3 and 4 are obtained respectively; based on the values ​​of key point coordinates 3 and 4, the final shapekey value is obtained through the fitting algorithm, so that the deformed result matches the feature points of the person in the photo.

[0015] In combination with the first aspect, the above-mentioned image preprocessing step further includes preprocessing the eyebrow part, and the specific steps are as follows:

[0016] Get the feature points of the eyebrows on the model, store their x-axis and z-axis coordinates, and find the corresponding 2D points on the image with the help of the faceparse algorithm;

[0017] Use the faceparse algorithm to segment each part and output the segmented image;

[0018] Calculate the center points of the nose and mouth based on the segmented image, and rotate it to a vertical direction to obtain a frontal face image;

[0019] According to the x-coordinate ratio of the points stored on the model, find the points corresponding to the eye part in the base image, compare the offset values ​​of these points when inputting a new image, and calculate the offset on the base model through the scale transformation between the image pixels and the model coordinates;

[0020] According to the ShapeKey calculation method, the optimal representation of the ShapeKey value of the eyebrow part within a specific range is calculated;

[0021] The standardized face is analyzed using the faceparse algorithm to obtain the segmentation results of each facial area.

[0022] In combination with the first aspect, the hairstyle matching step further includes:

[0023] Model the hair by referring to different styles of face pictures, match the input picture with the real face, and then use the hair model corresponding to the matched real face as the hairstyle corresponding to the input picture. The specific matching method is:

[0024] First, the faceparse algorithm is used to obtain the hairstyle mask of the reference image set to obtain the hairstyle mask set. In the preprocessing part, the faces are standardized, and the faces of all images can be regarded as being in the same position and of the same size.

[0025] Then, the faceparse algorithm is used to obtain the input hairstyle mask for the input face, and the matching degree between the input hairstyle mask and each picture in the hairstyle mask set is calculated respectively. The matching algorithm is: the weighted sum of the number of overlapping pixels between the input hairstyle mask and the reference hairstyle mask / the number of pixels of the input hairstyle mask and the number of overlapping pixels between the input hairstyle mask and the reference hairstyle mask / the number of pixels of the reference hairstyle mask. The photo with the highest score using this method is considered the best match, and the hairstyle with the highest matching degree is output as the matching result, where the symbol " / " represents division.

[0026] In combination with the first aspect, the fitting algorithm further comprises: using the cropped image as an input image for 3DMM reconstruction, first performing facial key point detection to obtain a number of key points in the two-dimensional face image in order to match the reference model with the face in the input image, wherein the key points correspond to the key points in the reference model;

[0027] Next, define the loss function:

[0028]

[0029] in Represents the mean square loss of the key points of the two-dimensional face image and the key points of the face model projection, represents the mean square loss of the rendered image and the input image, The L2 regularization of shape parameters, expression parameters, and texture parameters is used to optimize the model parameters by calculating the error, and then the fitted face model can be obtained.

[0030] In the 3DMM fitting process,

[0031]

[0032] Where S is the reconstructed 3D face shape, S i is the i-th main shape mode, α i is the weight of the i-th shape mode, which needs to be determined through an optimization process, n is a natural number, S, It is the fitting target, which solves the problem of difficult direct fitting due to stylization and visually preserves the facial feature information.

[0033] In combination with the first aspect, the calculation process of the ShapeKey value further includes:

[0034] Select a set of feature points Sa on the benchmark model and select a set of corresponding feature points A on the cartoon model;

[0035] The input photo is processed to obtain the unique model point Sb of the photo. Since the topology is the same, the offset of the feature point ΔS=S is calculated b -S a ;

[0036] The deformation matrix C can be calculated by step-by-step feature point positions. The offset displayed on the model after deformation is ΔA=CΔS.

[0037] Get the target vector corresponding to the image

[0038] The unit ShapeKey value A can be calculated based on the ShapeKey value of the model * ={A1, A2...A n};

[0039] The optimal representation of ShapeKey value in a specific range is calculated by SLSQP optimization algorithm. * ,α * ={α1, α2...α n};

[0040] Since the reference model itself has a vector to record the head orientation, the head orientation vector after fitting is calculated and compared with the rotation angle of the reference model vector on the three axes of X, Y, and Z. The fitting model is reversely rotated in the three directions in turn to rotate the fitting model to the positive direction;

[0041] After fitting, the rotation angles of the head orientation vector compared with the baseline model vector in three directions are (-a, -b, -c), and the X-axis rotation matrix is ​​obtained:

[0042]

[0043] Similarly, the Y-axis rotation matrix is:

[0044]

[0045] Z-axis rotation matrix:

[0046]

[0047] The parameters sin(a), sin(b), sin(c), cos(a), cos(b), and cos(c) represent the sine and cosine values ​​of angles a, b, and c, respectively;

[0048] Finally, the fitted model is multiplied by the rotation matrix to straighten the face.

[0049] A second aspect of the present invention provides a face recognition device based on a three-dimensional hybrid deformation model, the device comprising:

[0050] Image preprocessing module: input a face photo and standardize the face through preprocessing; in order to standardize the face, that is, to make the position and size of the input face the same, use MTCNN to detect the five key points of the face, including the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth; then calculate the midpoint of the two eyes as the eyebrow center, rotate the image as a whole around the eyebrow center until the horizontal coordinates of the two eyes are the same, and then take the eyebrow center as the center and half the eye distance, that is, the distance d from the eyebrow center to one eye as the unit length, and take a new box of 4d length from the top, bottom, left and right of the eyebrow center. The length and width of the box are 8d respectively, which completely contains the face, and the face position is the same and the size is similar, which is convenient for subsequent operations;

[0051] Hairstyle matching module: performs shape matching on the hairstyles in the hair area, extracts the hair area from the result based on the faceparse algorithm, and performs shape matching on the hair area and various hairstyles in the asset library;

[0052] Image matching module: extract the facial features and cheek area from the faceparse algorithm, and then use 3DMM to perform 3D reconstruction of the face:

[0053] Using the difference fitting method, the benchmark model selects a set of key points The cartoon-style model selects the corresponding point set at this time and It is a corresponding relationship;

[0054] Use 3DMM to reconstruct the face. 3DMM reconstruction is based on an optimized algorithm that fits the benchmark model to the face in the two-dimensional image. Here we define the benchmark model M:

[0055]

[0056] in and are the average shape and texture, S i ,e i and t i Represent shape parameters, expression parameters and texture parameters respectively, α i ,β i ,γ i Represent their weights respectively, n, l and k are natural numbers;

[0057] Several pre-set feature point positions are taken from the three-dimensional face generated by 3DMM, and the key point coordinate 2 is detected using the algorithm provided by Mediapipe; the pre-set feature point positions of the corresponding style are taken from the style asset library as key point coordinate 1; key point coordinates 1 and 2 need to be aligned because of different coordinate systems and scales, and after alignment, key point coordinates 3 and 4 are obtained respectively; based on the values ​​of key point coordinates 3 and 4, the final shapekey value is obtained through the fitting algorithm, so that the deformed result matches the feature points of the person in the photo.

[0058] The embodiment of the present invention has the following advantages: first, a three-dimensional face reconstruction is performed based on a single face photo. Then, a number of pre-defined key points are selected on the reconstructed three-dimensional face model and the cartoon reference model, and these key points correspond to the same facial features; then, a difference fitting method is applied to each facial feature to ensure that the features of the real face are retained to the greatest extent in the cartoon style model. Through this method, we can accurately transfer features between face models of different styles, so as to accurately reflect the features of the real face in the cartoon style model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of a computer device in one embodiment of a facial recognition method based on a three-dimensional hybrid deformation model of the present invention;

[0060] Figure 2 It is a diagram of the internal structure of a computer device in one embodiment of a facial recognition method based on a three-dimensional hybrid deformation model of the present invention. DETAILED DESCRIPTION

[0061] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] Before describing the present invention in detail, several basic concepts are first introduced:

[0063] Key points, face key points refer to some specific points in a face image or model that can identify the contour and characteristic parts of the face, such as eyes, nose, mouth and chin. Face key point detection is the basis of many computer vision tasks, such as face recognition and expression analysis.

[0064] Face parsing refers to the process of segmenting or identifying different parts of a face, such as eyes, nose, mouth, etc. It is a technology in computer vision and is widely used in fields such as expression analysis, face reconstruction, and augmented reality.

[0065] The Gaussian process is a nonparametric Bayesian machine learning model used to process continuous data and regression problems. It assumes that all data points in the data set satisfy the multivariate Gaussian distribution, can effectively predict the value of unobserved points, and estimate the uncertainty of the model.

[0066] Markov chain Monte Carlo sampling is a method for estimating complex probability distributions by constructing a Markov chain and performing random sampling. It is used in fields such as Bayesian statistics and machine learning to approximate distributions that are difficult to calculate directly, and is often used for model optimization and inference in high-dimensional parameter spaces.

[0067] Blendshape is a 3D modeling technique used to generate complex facial expressions and animations by linearly combining a series of predefined deformations or poses. It is widely used in facial animation and expression generation, and can create realistic dynamic expressions through weight adjustment.

[0068] In order to achieve style transfer that matches the characteristics of the photo in face reconstruction, the method adopted by the present invention is as follows: First, a 3D face reconstruction is performed based on a single face photo. Then, key points are selected on the 3D face benchmark model and the cartoon benchmark model, respectively, and these key points correspond to the same facial features. Then, the difference fitting method is applied to each facial feature to ensure that the characteristics of the real face are retained to the greatest extent in the cartoon style model. Through this method, we can accurately transfer features between face models of different styles, so as to accurately reflect the characteristics of the real face in the cartoon style model.

[0069] The overall technical solution of this patent is as follows Figure 1 As shown:

[0070] 1. Input a face photo and standardize the face through preprocessing.

[0071] In order to standardize the face of the input image, that is, the position and size of the input face are the same, first, use MTCNN to detect the five key points of the face, including the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth. Next, calculate the midpoint of the two eyes as the eyebrow center, and rotate the entire image around the eyebrow center until the horizontal coordinates of the two eyes are the same. We believe that the two eyes are on the same horizontal plane. Then, with the eyebrow center as the center and half the eye distance, that is, the distance d from the eyebrow center to one eye as the unit length, take a 4d length to the top, bottom, left and right of the eyebrow center to intercept a new box. The length and width of the box are 8d respectively, which completely contains the face. The faces intercepted in this way are in the same position and similar in size, which is convenient for subsequent operations.

[0072] The specific pre-processing process of the eyebrows is as follows:

[0073] (1) Obtain the feature points of the eyebrows on the model and store their x-axis and z-axis coordinates, hoping to find the corresponding 2D points on the image with the help of face-parsing.

[0074] (2) Segment each part through face-parsing and output the segmented image.

[0075] (3) Calculate the center points of the nose and mouth based on the segmented image and rotate it to the vertical direction to obtain a frontal face image.

[0076] (4) According to the x-coordinate ratio of the points stored on the model, find the points corresponding to the eye part in the basic image. When a new image is input, the offset values ​​of these points are compared. The offset on the basic model can be calculated by scaling the image pixels and the model coordinates.

[0077] (5) Based on ShapeKey, calculate the optimal representation of the eyebrow ShapeKey within a specific range.

[0078] The standardized face is analyzed by the faceparse algorithm, which obtains the segmentation results of each facial area.

[0079] 2. Shape matching of hairstyles in the hair area.

[0080] Get the hair area from the result of faceparse;

[0081] The hair area of ​​the photo is shape matched to various hairstyles in the asset library.

[0082] Take hair as an example: First, we model the hair with reference to different styles of face pictures. In order to improve the matching accuracy, we need to match the input picture with the real face, and then use the hair model corresponding to the matched real face as the hairstyle corresponding to the input picture. The specific matching method is: first, use faceParse to obtain the hairstyle mask for a series of reference picture sets to obtain the hairstyle mask set. Since the face is standardized in the preprocessing part, it can be regarded that the faces of all pictures are in the same position and have the same size. Next, use faceParse to obtain the input hairstyle mask for the input face. Calculate the matching degree of the input hairstyle mask and each picture in the hairstyle mask set respectively. The matching algorithm is: the weighted sum of the number of overlapping pixels between the input hairstyle mask and the reference hairstyle mask / the number of pixels of the input hairstyle mask and the number of overlapping pixels between the input hairstyle mask and the reference hairstyle mask / the number of pixels of the reference hairstyle mask. The photo with the highest score in this method is considered the best match.

[0083] Output the hairstyle with the highest matching degree as the matching result.

[0084] 3. Image matching, extract the facial features and cheek area from faceparse, and then use 3DMM to reconstruct the face in 3D;

[0085] In order to avoid the problem that the face shape and facial features of realistic style and cartoon style are too different to fit due to the deformation caused by stylization, we adopted the difference fitting method. First, we selected a set of key points in the realistic baseline model. At the same time, the corresponding point set is selected on the cartoon-style model We believe that at this time and It is a corresponding relationship.

[0086] Then, use 3DMM to reconstruct the face. 3DMM reconstruction is based on an optimized algorithm that fits the benchmark model to the face in the two-dimensional image. Here we define the benchmark model M:

[0087]

[0088] in and are the average shape and texture, S i ,e i and T i Represent shape parameters, expression parameters and texture parameters respectively, α i ,β i ,γ i Represent their weights respectively.

[0089] Several pre-set feature point positions are taken out from the three-dimensional face generated by 3DMM, and the key point 2 is directly detected using the algorithm provided by Mediapipe as the key point coordinate 2.

[0090] The cropped image is used as the input image for 3DMM reconstruction. In order to match the benchmark model with the face in the input image, we first perform facial key point detection to obtain several key points in the two-dimensional face image, which correspond to the key points in the benchmark model. Next, we define the loss function

[0091]

[0092] in Represents the mean square loss of the key points of the two-dimensional face image and the key points of the face model projection, represents the mean square loss of the rendered image and the input image, It represents the L2 regularization of shape parameters, expression parameters, and texture parameters. By calculating the error to optimize the model parameters, the fitted face model can be obtained.

[0093] In the 3DMM fitting process, we can abstract it as:

[0094]

[0095] Where S is the reconstructed 3D face shape, α i is the weight of the i-th shape mode, which needs to be determined through the optimization process. i is the i-th principal shape mode (also called principal component or shape key). S, This is our fitting goal. This method solves the problem of being difficult to fit directly due to stylization while retaining the facial feature information as much as possible visually.

[0096] 4. Image matching step 3. Take the pre-set feature point positions of the corresponding style from the style asset library as the key point coordinates 1.

[0097] 5. Image matching step 4. Key point coordinates 1 and 2 need to be aligned because of different coordinate systems and scales. After alignment, key point coordinates 3 and 4 are obtained respectively.

[0098] 6. Image matching step 5. Based on the values ​​of key point coordinates 3 and 4, the final shapekey is obtained through the fitting algorithm. Shapekey is a set of coefficients that can be used to deform the model in the style asset library. Our algorithm can obtain a set of shapekeys, so that the deformed result matches the feature points of the person in the photo.

[0099] ShapeKey calculation process:

[0100] 1. Select a set of feature points S on the benchmark model and a corresponding set of feature points A on the cartoon model;

[0101] 2. The input photo is processed to obtain the model point Sb unique to the photo. Since the topology is the same, the offset of the feature point ΔS=S is calculated b -S a ;

[0102] 3. The deformation matrix C can be calculated by step-by-step feature point positions. The offset displayed on the model after deformation is ΔA = CΔS;

[0103] 4. Get the target vector corresponding to the image

[0104] 5. According to the ShapeKey of the model, the value A of the unit ShapeKey can be calculated * ={A1, A2...A n};

[0105] 6. Based on such a set of bases, we can use the SLSQP optimization algorithm to calculate the optimal representation of ShapeKey in a specific range α * ,α * ={α1, α2...α n};

[0106] 7. Since the benchmark model itself has a vector to record the head orientation, we only need to calculate the rotation angle of the head orientation vector after fitting and the benchmark model vector on the X, Y, and Z axes, and perform reverse rotations on the fitting model in the three directions in turn to rotate the fitting model to the positive direction;

[0107] 8. After fitting, the rotation angles of the head orientation vector compared with the baseline model vector in three directions are (-a, -b, -c), and the X-axis rotation matrix is ​​obtained:

[0108]

[0109] 9. Similarly, the Y-axis rotation matrix is ​​obtained:

[0110]

[0111] 10.Z-axis rotation matrix:

[0112]

[0113] Finally, the fitted model is multiplied by the rotation matrix to straighten the face.

[0114] The present invention has been planned to be applied to China Unicom cloud disk products. By realizing style transfer that conforms to the characteristics of photos, new possibilities are provided for the fields of virtual reality (VR), augmented reality (AR), film and television special effects, and personalized content creation. It can reduce computing costs and is easy to integrate into existing platforms or software, so it is more likely to be implemented. In the current market, there is an increasing demand for technologies that can accurately capture and reproduce individual facial details and dynamic expressions, especially in social media applications. In addition, with the development of e-commerce, consumers have higher and higher requirements for online fitting experience, which makes the character clothing recognition technology based on image processing have broad market prospects.

[0115] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, such as Figure 2As shown, the computer program may be stored in a non-volatile computer-readable storage medium, and the computer program, when executed, may include the process of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical storage, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0116] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0117] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A face recognition method based on a three-dimensional hybrid deformation model, characterized in that: The method comprises: Image preprocessing steps: input a face photo and standardize the face through preprocessing; in order to standardize the face, that is, to make the position and size of the input face the same, use MTCNN to detect five key points of the face, including the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth; next, calculate the midpoint of the two eyes as the eyebrow center, rotate the entire image around the eyebrow center until the horizontal coordinates of the two eyes are the same, and then take the eyebrow center as the center and half the eye distance, that is, the distance d from the eyebrow center to one eye as the unit length, and take a new box of 4d length from the top, bottom, left and right of the eyebrow center. The length and width of the box are 8d respectively, which completely contains the face, and the face position is the same and the size is similar, which is convenient for subsequent operations; Hairstyle matching step: performing shape matching on the hairstyles in the hair area, taking out the hair area from the result based on the faceparse algorithm; performing shape matching on the hair area and various hairstyles in the asset library; Image matching steps: extract the facial features and cheek area from the faceparse algorithm, and then use 3DMM to perform 3D reconstruction of the face: Using the difference fitting method, the benchmark model selects a set of key points The cartoon-style model selects the corresponding point set at this time and It is a corresponding relationship; Use 3DMM to reconstruct the face. 3DMM reconstruction is based on an optimized algorithm that fits the benchmark model to the face in the two-dimensional image. Here we define the benchmark model M: in and are the average shape and texture, S i ,e i and t i Represent shape parameters, expression parameters and texture parameters respectively, α i ,β i ,γ i Represent their weights respectively, n, l and k are natural numbers; Several pre-set feature point positions are taken from the three-dimensional face generated by 3DMM, and the key point coordinate 2 is detected using the algorithm provided by Mediapipe; the pre-set feature point positions of the corresponding style are taken from the style asset library as key point coordinate 1; key point coordinates 1 and 2 need to be aligned because of different coordinate systems and scales, and after alignment, key point coordinates 3 and 4 are obtained respectively; based on the values ​​of key point coordinates 3 and 4, the final shapekey value is obtained through the fitting algorithm, so that the deformed result matches the feature points of the person in the photo.

2. A face recognition method based on a three-dimensional hybrid deformation model as claimed in claim 1, characterized in that: The image preprocessing step further includes preprocessing the eyebrow part, and the specific steps are as follows: Get the feature points of the eyebrows on the model, store their x-axis and z-axis coordinates, and find the corresponding 2D points on the image with the help of the faceparse algorithm; Use the faceparse algorithm to segment each part and output the segmented image; Calculate the center points of the nose and mouth based on the segmented image, and rotate it to a vertical direction to obtain a frontal face image; According to the x-coordinate ratio of the points stored on the model, find the points corresponding to the eye part in the base image, compare the offset values ​​of these points when inputting a new image, and calculate the offset on the base model through the scale transformation between the image pixels and the model coordinates; According to the ShapeKey calculation method, the optimal representation of the ShapeKey value of the eyebrow part within a specific range is calculated; The standardized face is analyzed using the faceparse algorithm to obtain the segmentation results of each facial area.

3. A face recognition method based on a three-dimensional hybrid deformation model as claimed in claim 2, characterized in that: The hairstyle matching step further includes: modeling the hair with reference to different styles of face pictures, matching the input picture with the real face, and then using the hair model corresponding to the matched real face as the hairstyle corresponding to the input picture. The specific matching method is: First, the faceparse algorithm is used to obtain the hairstyle mask of the reference image set to obtain the hairstyle mask set. In the preprocessing part, the faces are standardized, and the faces of all images can be regarded as being in the same position and of the same size. Then, the faceparse algorithm is used to obtain the input hairstyle mask for the input face, and the matching degree between the input hairstyle mask and each picture in the hairstyle mask set is calculated respectively. The matching algorithm is: the weighted sum of the number of overlapping pixels between the input hairstyle mask and the reference hairstyle mask / the number of pixels of the input hairstyle mask and the number of overlapping pixels between the input hairstyle mask and the reference hairstyle mask / the number of pixels of the reference hairstyle mask. The photo with the highest score using this method is considered the best match, and the hairstyle with the highest matching degree is output as the matching result, where the symbol " / " represents division.

4. A face recognition method based on a three-dimensional hybrid deformation model as claimed in claim 3, characterized in that: The fitting algorithm further comprises: The cropped image is used as the input image for 3DMM reconstruction. In order to match the benchmark model with the face in the input image, facial key point detection is first performed to obtain several key points in the two-dimensional face image, which correspond to the key points in the benchmark model. Next, define the loss function: in Represents the mean square loss of the key points of the two-dimensional face image and the key points of the face model projection, represents the mean square loss of the rendered image and the input image, The L2 regularization of shape parameters, expression parameters, and texture parameters is used to optimize the model parameters by calculating the error, and then the fitted face model can be obtained. In the 3DMM fitting process, Where S is the reconstructed 3D face shape, S i is the i-th main shape mode, α i is the weight of the i-th shape mode, which needs to be determined through an optimization process, n is a natural number, S, It is the fitting target, which solves the problem of difficult direct fitting due to stylization and visually preserves the facial feature information.

5. A face recognition method based on a three-dimensional hybrid deformation model as claimed in claim 4, characterized in that: The calculation process of the ShapeKey value further includes: Select a set of feature points Sa on the benchmark model and select a set of corresponding feature points A on the cartoon model; The input photo is processed to obtain the unique model point Sb of the photo. Since the topology is the same, the offset of the feature point ΔS=S is calculated b -S a ; The deformation matrix C can be calculated by step-by-step feature point positions. The offset displayed on the model after deformation is ΔA=CΔS. Get the target vector corresponding to the image The unit ShapeKey value A can be calculated based on the ShapeKey value of the model * ={A1, A2…A n }; Calculate the optimal representation of ShapeKey value in a specific range through SLSQP optimization algorithm α * , α * ={α1, α2...α n }; Since the reference model itself has a vector to record the head orientation, the head orientation vector after fitting is calculated and compared with the rotation angle of the reference model vector on the three axes of X, Y, and Z. The fitting model is reversely rotated in the three directions in turn to rotate the fitting model to the positive direction; The rotation angles of the head orientation vector after fitting and the reference model vector in three directions are (-a, -b, -c), and the X-axis rotation matrix is ​​obtained: Similarly, the Y-axis rotation matrix is: Z-axis rotation matrix: The parameters sin(a), sin(b), sin(c), cos(a), cos(b), and cos(c) represent the sine and cosine values ​​of angles a, b, and c, respectively; Finally, the fitted model is multiplied by the rotation matrix to straighten the face.

6. A face recognition device based on a three-dimensional hybrid deformation model, characterized in that: The device comprises: Image preprocessing module: input a face photo and standardize the face through preprocessing; in order to standardize the face, that is, to make the position and size of the input face the same, use MTCNN to detect the five key points of the face, including the left eye, right eye, nose, left corner of the mouth, and right corner of the mouth; then calculate the midpoint of the two eyes as the eyebrow center, rotate the image as a whole around the eyebrow center until the horizontal coordinates of the two eyes are the same, and then take the eyebrow center as the center and half the eye distance, that is, the distance d from the eyebrow center to one eye as the unit length, and take a new box of 4d length from the top, bottom, left and right of the eyebrow center. The length and width of the box are 8d respectively, which completely contains the face, and the face position is the same and the size is similar, which is convenient for subsequent operations; Hairstyle matching module: performs shape matching on the hairstyles in the hair area, extracts the hair area from the result based on the faceparse algorithm, and performs shape matching on the hair area and various hairstyles in the asset library; Image matching module: extract the facial features and cheek area from the faceparse algorithm, and then use 3DMM to perform 3D reconstruction of the face: Using the difference fitting method, the benchmark model selects a set of key points The cartoon-style model selects the corresponding point set at this time and It is a corresponding relationship; Use 3DMM to reconstruct the face. 3DMM reconstruction is based on an optimized algorithm that fits the benchmark model to the face in the two-dimensional image. Here we define the benchmark model M: in and are the average shape and texture, S i ,e i and t i Represent shape parameters, expression parameters and texture parameters respectively, α i ,β i ,γ i Represent their weights respectively, n, l and k are natural numbers; Several pre-set feature point positions are taken from the three-dimensional face generated by 3DMM, and the key point coordinate 2 is detected using the algorithm provided by Mediapipe; the pre-set feature point positions of the corresponding style are taken from the style asset library as key point coordinate 1; key point coordinates 1 and 2 need to be aligned because of different coordinate systems and scales, and after alignment, key point coordinates 3 and 4 are obtained respectively; based on the values ​​of key point coordinates 3 and 4, the final shapekey value is obtained through the fitting algorithm, so that the deformed result matches the feature points of the person in the photo.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the face recognition method based on a three-dimensional hybrid deformation model described in any one of claims 1 to 5 are implemented.

8. A computer device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program to implement the steps of the face recognition method based on the three-dimensional hybrid deformation model as described in any one of claims 1 to 5.

Images