A 3D Reconstruction Method for Face Models Based on Multiple Views
By using a multi-view approach for 3D face reconstruction, multiple images are used for initialization and projection matrix adjustment, which solves the ambiguity problem in single-image reconstruction, improves the realism and efficiency of the model, and generates a model that better matches the real shape of the modeled object.
Patent Information
- Application Number
- CN202211173087.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-26
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-09-26
AI Technical Summary
Existing technologies lack reliable constraints when reconstructing 3D models from single face images, resulting in limited model performance and a dependence on deep learning parameters in the depth direction. This makes it impossible to guarantee the realism of the face model, and ambiguity issues are prone to occur when reconstructing from a single image.
A multi-view-based approach is adopted, which involves taking multiple facial images for modeling, generating a facial model using a single image for initialization, extracting the two-dimensional image coordinates of facial feature points, calculating the projection matrix, adjusting the three-dimensional vertex positions, and combining the topological structure to complete the shape modeling of the facial model and generate texture information.
It improves the efficiency and fidelity of face modeling, solves the ambiguity problem in single image reconstruction, generates models that better match the real shape of the modeled object, and reduces the dependence on deep learning parameters.
Smart Images

Figure CN115496856B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of 3D model reconstruction and relates to a method for 3D reconstruction of a 3DMM face model based on multiple views. Background Technology
[0002] In the application of virtual reality (VR) and augmented reality (AR) technologies, the reconstruction of realistic 3D face models from 2D face images is playing an increasingly important role. In the traditional computer animation industry, a relatively complete technical framework and industrial chain have been formed to generate high-fidelity character images. Currently, the most common face reconstruction method is based on 3D Morphable Models (3DMMs). 3D Morphable Models (3DMMs) are currently the most widely used type of 3D face model. They are built upon a 3D face database, using face model statistics as constraints, and representing faces with the same number of model points and the same topological structure. Typically, a triangular network with a fixed number of vertices is defined as a template. Face models from the 3D face database are registered onto the template mesh, thus aligning the face models and template to the same reference space and with the same topological structure. Then, principal component analysis is used to reduce the dimensionality of the face models in the database.
[0003] 3DMM model is represented as
[0004]
[0005] in s represents the average three-dimensional human face shape. i This represents the principal component of PCA corresponding to the shape, often referred to as the face orthogonal basis, α. i This represents the weight corresponding to the shape. Any human face can be represented by the above expression, and different 3D face models have different shape weights.
[0006] There are two main methods for generating 3D face models from 2D facial images and 3DMM models: traditional fitting methods and deep learning methods. These methods obtain the shape weights of the orthogonal bases for each face, thus yielding the 3D face model. Currently, the most common method is to reconstruct the 3DMM model from a single image using deep learning. Several professionals have already open-sourced related deep networks and their code. A single facial image is input into the deep network, and the shape weights of the orthogonal bases for each face can be obtained from the network's output.
[0007] However, the face models obtained by the above methods have the following problems: the model's performance is limited, and the face models obtained from a single face image rely too much on the parameters in the deep learning training process in the depth direction, which cannot guarantee the realism of the face models.
[0008] The problem with single-image face reconstruction is the lack of reliable constraints, which can lead to unsolvable ambiguities, such as difficulty in distinguishing the height of the nose and cheekbones from a frontal view. Summary of the Invention
[0009] This invention addresses the problem of unreliable constraints in reconstructing 3D models from single face images by providing a multi-view face reconstruction method, specifically including the following steps:
[0010] S1: Take several facial images while keeping the modeling object relatively still, ensuring that the facial area is captured completely and that the lighting intensity remains consistent;
[0011] S2: Using a method to generate a face model from a single image, each face image is independently modeled to obtain a set of three-dimensional face models. The initialization of the vertex positions of the three-dimensional face models is completed, and the value range of each three-dimensional vertex is determined.
[0012] S3: Extract the two-dimensional image coordinates of a preset number of facial feature points in each face image, calculate the coordinates of the facial feature points in three-dimensional space using interpolation, and calculate the projection matrix of each face image using the correspondence of the facial feature points.
[0013] S4: Adjust the three-dimensional coordinates within the range of values of each three-dimensional vertex, map the three-dimensional coordinates onto each face image using the projection matrix, determine the position of each vertex by using the color similarity of the same vertex in different face images, and complete the face model shape modeling by combining the topological structure between vertices.
[0014] S5: Using the face image and the corresponding projection matrix, generate the texture information of the 3D face model to complete the 3D face reconstruction.
[0015] After step S1, n face images of the modeling object have been obtained, denoted as I = {I1, I2, ..., I...} n The faces in each image have no obvious shadows, maintaining consistent lighting.
[0016] Furthermore, in step S2, a face model is generated using a single image to obtain a set of 3D face models, thereby completing the face model initialization and determining the value range of each 3D vertex. The specific algorithm flow is as follows:
[0017] S2-1: After step S1, obtain n face images of the modeling object, denoted as I = {I1, I2, ..., I...} n}, for image I in set I iThe weight coefficients of each face orthogonal basis are obtained by generating a 3DMM model from a single image. The number of face orthogonal basis bases is denoted as m, and the weights of the face orthogonal basis bases can be expressed as A. i = {a (i,1) ,a (i,2) ,...,a (i,j) ,…,a (i,m)};
[0018] S2-2: Through weighting coefficient A i Image I can be calculated i The corresponding 3D model vertex data V i That is, V i ={p i,1 ,p i,2 ,...,p i,t ,...,p i,s}, where p i,t =(x i,t ,y i,t ,z i,t Let ) represent the 3D coordinates of the vertex, and s represent the number of vertices in each face model; let V be the set of 3D models generated from the images in image set I, then V = {V1, V2, ..., V...} n This step yields an initial set of face models; for any image I i with I j The corresponding three-dimensional model V i With V j They all have p i,t and p j,t Correspondingly, p i,t and p j,t Corresponding to the same vertex t in the face model;
[0019] S2-3: Using the set V = {V1, V2, ..., V...} n The data of each vertex contained in} is used to initialize each vertex of the 3D face model, and the result of the initialization is the mean of the corresponding vertex.
[0020] S2-4: Using the data in set V, determine the reasonable value range for each vertex of the 3D face model.
[0021] At this point, the initial values of each 3D vertex of the face model have been calculated, and the value range of each 3D vertex has been obtained.
[0022] Further, step S3 extracts the two-dimensional pixel coordinates of 68 facial feature points on the face image, calculates the three-dimensional spatial coordinates of the facial feature points in the face model using an interpolation algorithm, and the specific algorithm flow for calculating the projection matrix of each face image using the three-dimensional spatial coordinates of the feature points and the two-dimensional pixel coordinates is as follows:
[0023] S3-1: Extract the pixel coordinates of facial feature points from the image. Using a feature point matching algorithm, calculate the descriptor for each feature point. Use these descriptors to match facial feature points with the same meaning across different images. For a given feature point p with the same meaning, find it in the image set... In each pair, I is matched. p The set of 3D models corresponding to the images in the image is denoted as .
[0024] S3-2: Since the face models generated by 3DMM have the same topological structure, each feature point has a unique correspondence in the 3D model. An interpolation algorithm is used to obtain the feature point p in the set V. p The three-dimensional spatial coordinates of the l models are denoted as {p}. s1 ,p s2 ,…,p sl};
[0025] S3-3: If set {p s1 ,p s2 ,...,p sl If there exists a subset such that the distance between all three-dimensional feature points is less than a given threshold ε, then the average value of the three-dimensional feature points contained in the subset is considered reliable, and the average value of the three-dimensional feature points contained in the subset is selected as reliable three-dimensional information.
[0026] If there is more than one subset, the average value of the three-dimensional feature points calculated from the subset with the largest number of three-dimensional feature points is selected as the reliable three-dimensional information.
[0027] Record the three-dimensional position coordinates and two-dimensional image coordinates of reliable feature points in each image;
[0028] S3-4: Using the spatial coordinates corresponding to the reliable three-dimensional information and the two-dimensional pixel coordinates corresponding to the reliable three-dimensional information, calculate the projection matrix of each image. The projection matrix is a 3×4 matrix. Calculate the pixel coordinates (u,v) of the matrix on the image using the three-dimensional point positions (x,y,z). The solution process for the projection matrix uses the method of solving an overdetermined system of equations to obtain the least squares solution of this equation.
[0029] Now that we have obtained the projection matrix of the image, we can calculate the two-dimensional pixel coordinates of the vertices in the image using their three-dimensional spatial coordinates.
[0030] Further, step S4 uses the projection matrix to obtain the similarity of the RGB values of each 3D vertex coordinate in multiple images, and adjusts the position of each 3D vertex. The algorithm flow is as follows:
[0031] S4-1: The homogeneous coordinates of any vertex t on the face model in 3D space are (x...). t ,y t ,z t ,1) T The homogeneous pixel coordinates (u, v, 1) of the vertex in each image are calculated using the projection matrices. T In image I i The RGB information of a pixel can be obtained by using its pixel coordinates, denoted as R. i (u t,i ,v t,i ), G i (u t,i ,v t,i B) i (u t,i ,v t,i );
[0032] S4-2: In the data set {R0(u0,v0),R1(u1,v1),...,R...} i (u i ,v i In the context of )}, the similarity s of data R is calculated. R Similarly, calculate the similarity s between data G and data B respectively. G With s B ;
[0033] S4-3: If the data similarity is less than the given threshold ψ, it means that the three-dimensional position of vertex t does not need to be adjusted. If the condition that the similarity is less than the given threshold ψ is not met, the three-dimensional position of t needs to be adjusted. Within the range of vertex t, the space is evenly divided into γ×γ×γ subspaces. The distance between the center point of each subspace and the initial position is calculated. The reciprocal of the distance is used as the weight of each space. The spaces are sorted according to the size of the weight. The center point of each subspace is used as the adjustment position of the three-dimensional point. The operation of steps S4-1 and S4-2 is continued. The similarity corresponding to the adjusted position is calculated. If the similarity is less than the given threshold ψ, the adjustment of the vertex ends. If the calculated similarity is greater than the given threshold ψ, the vertex with the highest similarity is used as the adjusted result.
[0034] S4-4: Based on the process from S4-1 to S4-3, the three-dimensional position adjustment of each vertex on the face model is completed sequentially.
[0035] At this point, the three-dimensional position of each vertex of the face model has been obtained. Combined with the topological structure between the vertices, the shape of the face model has been adjusted.
[0036] Furthermore, step S5 utilizes multiple images and their corresponding projection coordinates to generate texture information for a 3D face model. The algorithm flow is as follows:
[0037] S5-1: Calculate the image fragments in different images corresponding to each facet in the face model using the projection matrix;
[0038] S5-2: Calculate the cost value of an image fragment to a patch. The smaller the cost value, the higher the correlation between the image fragment and the patch. The patch is denoted as F. i The image fragment is m ij The value is then represented as C(F). i ,m ij )
[0039] S5-3: Calculate the visibility of the gap between adjacent patches, if F i With F k Adjacent, E ik F represents i With F k Given the common edge, calculate the integral difference C between the edge colors of adjacent patches. Eik (F i ,F k ,m ij ,m kj );
[0040] S5-4: Calculate appropriate image fragments for each grid using the principle of hidden Markov random fields;
[0041] S5-5: Using the Poisson editing method, the various image fragments are fused together to obtain the final face model texture map.
[0042] This invention provides a method for generating a 3D facial model using multiple facial images. The facial model generated using this method can be used for facial modeling in industries such as games and animation, and can also be applied in VR and AR fields. Compared with traditional facial modeling methods, the advantages of this invention are:
[0043] (1) Compared with the traditional method of face modeling by hand, this method does not require a lot of manual operation, and greatly improves the efficiency of face modeling while ensuring the modeling effect and saving modeling costs.
[0044] (2) Compared with the method of face reconstruction using a single image, this method can effectively form constraints on the face model using multiple images, thus solving the problem of unsolvable ambiguity encountered when modeling a face model using a single image.
[0045] (3) Compared with the traditional linear 3DMM face model reconstruction algorithm, the 3DMM model is obtained by linearly adding several orthogonal face baselines. This model has limited representation ability and cannot completely and realistically model each modeling object. Based on 3DMM, this method adjusts the position of each vertex to make the model built by this method fit the real shape of the modeling object as closely as possible, thereby improving the fidelity of the modeling. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly described below. Referring to the accompanying drawings will provide a clearer understanding of the features and advantages of the present invention. The drawings are illustrative and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. Wherein:
[0047] Figure 1 This is a flowchart of a method for generating face models based on multiple images and a 3DMM model;
[0048] Figure 2 Flowchart for initializing face model vertices and calculating vertex value ranges
[0049] Figure 3 A diagram illustrating the specific meanings of the 68 feature points of a face;
[0050] Figure 4 Flowchart for calculating the image projection matrix using facial feature points;
[0051] Figure 5 Flowchart for adjusting vertex positions using vertex color similarity;
[0052] Figure 6 Flowchart for generating face model textures; Detailed Implementation
[0053] To make the technical solution of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings. The overall flowchart is as follows. Figure 1 As shown.
[0054] Specifically, a 3D face model reconstruction method based on multiple views is characterized by the following steps:
[0055] S1: Take several facial images while keeping the modeling object relatively still, ensuring that the facial area is captured completely and that the lighting intensity remains consistent;
[0056] S2: Using a method to generate a face model from a single image, each face image is independently modeled to obtain a set of three-dimensional face models. The initialization of the vertex positions of the three-dimensional face models is completed, and the value range of each three-dimensional vertex is determined.
[0057] S3: Extract the two-dimensional image coordinates of a preset number of facial feature points in each face image, calculate the coordinates of the facial feature points in three-dimensional space using interpolation, and calculate the projection matrix of each face image using the correspondence of the facial feature points.
[0058] S4: Adjust the three-dimensional coordinates within the range of values of each three-dimensional vertex, map the three-dimensional coordinates onto each face image using the projection matrix, determine the position of each vertex by using the color similarity of the same vertex in different face images, and complete the face model shape modeling by combining the topological structure between vertices.
[0059] S5: Using the face image and the corresponding projection matrix, generate the texture information of the 3D face model to complete the 3D face reconstruction.
[0060] To ensure the final effect of the generated 3D face model and maintain the realism and reliability of the modeling, the modeling object in step S1 needs to be kept relatively still to take several face images, ensuring the integrity of the facial area. The environment with good lighting conditions should be selected to ensure that the lighting conditions are consistent during the shooting process and to avoid the local shadows caused by direct light sources, which would affect the final modeling effect.
[0061] After step S1, n face images of the modeling object have been obtained, and the image set is denoted as I = {I1, I2, ..., I...} n}
[0062] Further, in step S2, the method of generating face models using a single image is used to model each image independently to obtain the weight coefficients corresponding to the face orthogonal basis of the 3DMM model. According to formula (1), the three-dimensional spatial position of each vertex of the three-dimensional face model is calculated using the weight coefficients to obtain a set of three-dimensional face models. The data in the three-dimensional model set is used to initialize the vertex position of the face model and determine the value range of each three-dimensional vertex.
[0063] like Figure 2 As shown in the flowchart, the specific algorithm flow for step S2 is as follows:
[0064] S2-1: The idea behind 3DMM modeling is that the face model can be obtained by linearly weighting the orthogonal lines of the face. Therefore, for images I in set I... iThe initial face model is obtained by deriving the weight coefficients of the face orthogonal basis using existing methods for generating 3DMM models from single images. The number of face orthogonal basis points is denoted as m, and the weights of the face orthogonal basis points can be expressed as A. i ={a (i,1) a (i,2) , ..., a (i,j) , ..., a (i,m)}; Given n images in set I, we can obtain n sets of weight values, denoted as {A1, A2, ..., A...} i A n};
[0065] S2-2: For image I i Through the weighting coefficient {a (i,1) a (i,2) , ..., a (i,j) , ..., a (i,m) Image I can be calculated using formula (1). i The corresponding 3D model vertex data V i ={p i,1 p i,2 , ..., p i,t , ..., p i,s}, where p i,t =(x i,t y i,t , z i,t ), where s represents the number of vertices. Each image in the image set I can yield corresponding 3D model vertex data, denoted as V = {V1, V2, ..., V...}. n}; for any image I i with I j The corresponding vertex set V i With V j They all have p i,t and p j,t Matching, P i,t and P j,t Corresponding to the same vertex t in the face model;
[0066] S2-3: Using the set V = {V1, V2, ..., V...} n The vertices of the 3D face model are initialized. For a vertex t in the initial face model, in image I... i The 3D coordinates of the model are represented as p i,t =(x i,t y i,t , z i,t The initial value of vertex t is p. t0 It can be represented as:
[0067]
[0068] Where p t0 The coordinates are represented as
[0069] S2-4: Determine the reasonable value range of each vertex in the 3D face model. For vertex t in the face model, in the set {p 1,t p 2,t ...p n,t In the diagram, the minimum value in the x-direction is denoted as x. min,t The maximum value in the x-direction is denoted as x. max,t Similarly, the minimum value in the y-direction is denoted as y0. min,t The maximum value in the y-direction is denoted as y. max,t The minimum value in the z-direction is denoted as z. min,t The maximum value in the z-direction is denoted as z. max,t Then the range of values for point t along the x-axis can be expressed as (x min,t -d x x max,t +d x ), where d x Based on the set value of the model's 3D data in the x-axis direction, similarly, the range of values for t in the y-axis direction is expressed as (y min,t -d y y ma,t +d y The range of values for t in the z-axis direction is expressed as (z min,t -d z , z max,t +d z ), where d y and d z These represent the set values in the y-axis and z-axis directions, respectively. x d y d z The calculations are performed using formulas (3), (4), and (5) respectively:
[0070] d x =λ(x) ma,t -x min,t (3)
[0071] d y =λ(y max,t -y min,t (4)
[0072] d z =λ(z) max,t -z min,t (5)
[0073] In the experiment, λ = 2 was chosen to determine the range of values for each three-dimensional vertex.
[0074] After step S2, the initial value of each vertex of the 3D face model and the value range of each vertex have been obtained.
[0075] Further, step S3 calculates the three-dimensional spatial coordinates of 68 facial feature points in the image and the two-dimensional pixel coordinates in the image, and uses the three-dimensional spatial coordinates and two-dimensional pixel coordinates to calculate the projection matrix of each image. Currently, mature algorithms exist that can extract 68 feature points with definite meaning from face images, such as... Figure 3 As shown, the facial feature point extraction algorithm has been integrated into the commonly used third-party library Dlib.
[0076] like Figure 4 As shown in the flowchart, the specific process of step S3 is as follows:
[0077] S3-1: Extract the pixel coordinates of 68 facial feature points from the image. Using a feature point matching algorithm, calculate the descriptor for each feature point. Use these descriptors to match facial feature points with the same meaning across different images, and record the successfully matched two-dimensional feature points {s1, s2, ..., s...} in each image. l};
[0078] S3-2: The face models generated by 3DMM have the same topological structure, and each facial feature point can have a unique correspondence in the three-dimensional model. The three-dimensional position of the successfully matched feature points in the three-dimensional face model corresponding to each image is obtained by using an interpolation algorithm.
[0079] S3-3: A set of successfully matched two-dimensional facial feature points {s1, s2, ..., s...} l}, using the correspondence, obtain the three-dimensional position coordinates of its feature points {p s1 p s2 , ..., p sl If set {p} s1 p s2 , ..., p sl In the set}, there exists a subset such that the distance between the three-dimensional feature points is d. si,sj If all values are less than a given threshold ε, the average value of the 3D feature points in this subset is considered reliable, representing the true location of the feature point in 3D space. If there is more than one subset, the average value of the 3D feature points in the subset with the largest number of elements is selected as the reliable 3D information. The 3D spatial coordinates and corresponding 2D image coordinates of the reliable feature points in each image are recorded, where d... si,sj The value is calculated using formula (6):
[0080] d si,Sj =||p si -p sj|| 2 =(x si -x sj ) 2 +(y si -y sj ) 2 +(z si -z sj ) 2 (6)
[0081] S3-4: Calculate the projection matrix M for each image using reliable two-dimensional pixel coordinates and three-dimensional spatial coordinates. project The projection matrix is a 3×4 matrix, which can be used to calculate the pixel coordinates (u, v) and (x, y, z, 1) of a point on the image from its three-dimensional position (x, y, z). T Represents the homogeneous coordinates of a 3D point, (u, v, 1). T This represents the second coordinate of a two-dimensional pixel.
[0082] M project It can be represented as:
[0083]
[0084] Where a 34 =1, requiring at least 4 pairs of correspondences between 3D coordinates and 2D pixels to solve. The formula for solving this problem is denoted as:
[0085]
[0086] Where l > 4, U and C are represented as follows:
[0087]
[0088]
[0089] If the number of reliable correspondences between three-dimensional spatial coordinates and two-dimensional pixels in the image is less than 4, the image is discarded and no further steps will be performed.
[0090] Regarding the image projection matrix M project By using methods for solving overdetermined systems of equations, the least-squares solution to this equation can be obtained. The specific implementation can be achieved using functions from the Python third-party library NumPy; the detailed solution process will not be elaborated here. At this point, the projection matrices of each image have been obtained, and the correspondence between 3D coordinates and 2D pixel coordinates can be calculated.
[0091] Further, step S4 uses the projection matrix to obtain the similarity of the RGB values of each vertex in multiple images, and adjusts the position of the 3D vertices, such as... Figure 5The algorithm flow shown in the flowchart is as follows:
[0092] S4-1: The homogeneous coordinates of any vertex t on the face model in 3D space are (x...). t y t , z t ,1) T The homogeneous pixel coordinates (u, v, 1) of the vertex in each image are calculated using the projection matrix. T The calculation is shown in formula (11):
[0093]
[0094] For image I i In other words, the RGB color information of a pixel can be obtained through its pixel coordinates, denoted as R... i (u t,i v t,i ), G i (u t,i v t,i B i (u t,i v t,i );
[0095] S4-2: In the data set {R1(u t,1 v t,1 ), R1(u t,2 v t,2 ), ..., R n (u t,n v t,n In the context of )}, the similarity s of data R is calculated. R As shown in formula (12),
[0096]
[0097] in As shown in formula (13):
[0098]
[0099] Similarly, calculate the similarity s between data G and data B. G Similarity s B As shown in formulas (14) and (15) respectively:
[0100]
[0101]
[0102] in, and As shown in formulas (16) and (17) respectively:
[0103]
[0104]
[0105] S4-3: If the data similarity is less than the given threshold ψ, then the three-dimensional position of vertex t is (x, y, z); if the condition that it is less than the given threshold ψ is not met, then the three-dimensional position of the vertex needs to be adjusted; within the range of vertex t, the space is evenly divided into γ×γ×γ subspaces (in this method, γ=7), the distance between the center point of each subspace and the initial position is calculated, and the reciprocal of the distance is used as the weight of each space. The spaces are sorted according to the size of the weights, and the center point of each subspace is used as the adjustment position of the three-dimensional point. The operation of steps S4-1 and S4-2 is continued, and the corresponding similarity is calculated. If the similarity is less than the threshold ψ, then the adjustment of the vertex ends. If the calculated similarity is greater than the threshold ψ, then the position of the vertex with the highest similarity is used as the adjusted result.
[0106] S4-4: Based on the process from S4-1 to S4-3, the three-dimensional position adjustment of each vertex on the face model is completed sequentially.
[0107] Having obtained the positional information of each vertex, the mesh model of the face of the modeling object can be obtained by utilizing the inherent topological structure of the model.
[0108] Furthermore, using multiple images and their corresponding projection matrices, texture information for a 3D facial model is generated. This texture is then projected onto a facial mesh model to complete the 3D facial reconstruction. Figure 6 The specific process shown in the flowchart is as follows:
[0109] S5-1: Calculate the image fragments in different images corresponding to each facet in the face model using projected coordinates. Each facet in the face mesh model is denoted as {F1, F2, ... F...}. k}, F i The three vertices are denoted as o, p, and q, in the image I j The corresponding pixel coordinates are denoted as u. j,o u j,p u j,q Typically in image I j middle u j,o u j,p u j,q The triangle formed by these elements is called the original image I. j China F i The corresponding image fragments, {F1, F2, ... F k Each facet in} corresponds to an image fragment in several original images, Fi The corresponding image fragment is denoted as {m i1 m i2 …, m ij , ..., m il};
[0110] S5-2: Use cost C(F) i m ij ) represents image fragment m ij With sheet F i Relevance, the lower the cost value, the more likely the image fragment is to be related (m). ij With F i The higher the correlation, the better C(F) i m ij As shown in formula (18):
[0111]
[0112] This indicates that F is computed using the Sobel operator. i The corresponding m ij The magnitude of the gradient, calculate m ij The sum of the gradients of each pixel;
[0113] S5-3: Calculate the visibility of the gap between adjacent patches, i.e., F i With F k Adjacent, E ik F represents i With F k The common edge is used to calculate the integral value of the color difference between the edges of adjacent patches, as shown in formula (19).
[0114]
[0115] Among them, Pr m It is a projection operator that projects a patch F onto an image fragment m, where d(,) represents the Euclidean distance in RGB space;
[0116] S5-4: Use the principle of hidden Markov random fields to solve for the minimum value of E in formula (20) and calculate the appropriate image fragments for each grid.
[0117]
[0118] S5-5: Using the Poisson editing method, the various image fragments are fused together to obtain the final face model texture map.
[0119] By following the steps described above, a 3D model of a face can be obtained from multiple face images.
Claims
1. A method for three-dimensional reconstruction of a 3DMM face model based on multiple views, characterized in that, The method includes the following steps: S1: Take several facial images while keeping the modeling object relatively still, ensuring that the facial area is captured completely and that the lighting intensity remains consistent; S2: Using a method to generate a face model from a single image, each face image is independently modeled to obtain a set of three-dimensional face models. The initialization of the vertex positions of the three-dimensional face models is completed, and the value range of each three-dimensional vertex is determined. S2 includes the following steps: S2-1: After step S1, n face images of the modeling object are obtained, denoted as... For the images in set I The weight coefficients of each face orthogonal basis are obtained by generating a 3DMM model from a single image. The number of face orthogonal bases is denoted as m, and the weights of the face orthogonal bases can be expressed as... ; S2-2: Through weighting coefficients The image can be calculated Corresponding 3D model vertex data That is, ,in Let s represent the 3D coordinates of the vertices, and s represent the number of vertices in each face model; the set of 3D models generated from the images in image set I is denoted as . Then there is This step yields an initial set of face models; for any image and Corresponding 3D model and They all and Correspondingly, and Corresponding to the same vertex t in the face model; S2-3: Using sets The data of each vertex contained therein is used to initialize each vertex of the 3D face model, and the result of the initialization is the mean value of the corresponding vertex. S2-4: Using sets The data in the model is used to determine the reasonable value range for each vertex of the 3D face model; S3: Extract the two-dimensional image coordinates of a preset number of facial feature points in each face image, calculate the coordinates of the facial feature points in three-dimensional space using interpolation, and calculate the projection matrix of each face image using the correspondence of the facial feature points. S4: Adjust the three-dimensional coordinates within the range of values of each three-dimensional vertex, map the three-dimensional coordinates onto each face image using the projection matrix, determine the position of each vertex by using the color similarity of the same vertex in different face images, and complete the face model shape modeling by combining the topological structure between vertices. S5: Using the face image and the corresponding projection matrix, generate the texture information of the 3D face model to complete the 3D face reconstruction.
2. The method according to claim 1, characterized in that, S3 includes the following steps: S3-1: Extract the pixel coordinates of facial feature points from the image. Using a feature point matching algorithm, calculate the descriptor for each feature point. Use these descriptors to match facial feature points with the same meaning across different images. For a given feature point p with the same meaning, find it in the image set... They are matched pairwise. The set of 3D models corresponding to the images in the image is denoted as . ; S3-2: Since the face models generated by 3DMM have the same topological structure, each feature point has a unique correspondence in the 3D model. An interpolation algorithm is used to obtain the feature point p in the set... The three-dimensional spatial coordinates of the m models are denoted as follows: ; S3-3: If set There exists a subset such that the distance between all three-dimensional feature points is less than a given threshold. If the average value of the three-dimensional feature points contained in the subset is considered reliable, then the average value of the three-dimensional feature points contained in the subset is selected as reliable three-dimensional information. If there is more than one subset, the average value of the three-dimensional feature points calculated from the subset with the largest number of three-dimensional feature points is selected as the reliable three-dimensional information. Record the three-dimensional position coordinates and two-dimensional image coordinates of reliable feature points in each image; S3-4: Using the spatial coordinates corresponding to the reliable 3D information and the 2D pixel coordinates corresponding to the reliable 3D information, calculate the projection matrix for each image. The projection matrix is a 3×4 matrix, obtained through the 3D point positions. Calculate the pixel coordinates of the matrix on the image. The process of solving the projection matrix involves using the method of solving overdetermined equations to obtain the least-squares solution of the equation.
3. The method according to claim 1, characterized in that, S4 includes the following steps: S4-1: The homogeneous coordinates of any vertex t on the face model in 3D space are: The homogeneous pixel coordinates of the vertex in each image are calculated using each projection matrix. In the image The RGB color information of a pixel can be obtained through its pixel coordinates, denoted as follows: , , ; S4-2: In the data set In the process, the similarity of data R is calculated. Similarly, calculate the similarity between data G and data B respectively. and ; S4-3: If the data similarity is less than a given threshold, then the 3D position of vertex t does not need to be adjusted. If the condition that the similarity is less than the given threshold is not met, then the 3D position of t needs to be adjusted further. Within the range of vertex t values, the space is uniformly divided into... For each subspace, the distance between the center point of each subspace and the initial position is calculated. The reciprocal of the distance is used as the weight of each space, and the subspaces are sorted according to the weight. The center point of each subspace is used as the adjustment position of the 3D point. The calculation of steps S4-1 and S4-2 is continued. The similarity corresponding to the adjusted position is calculated. If the similarity is less than the given threshold, the adjustment of the vertex ends. If the calculated similarity is greater than the given threshold, the vertex with the highest similarity is taken as the adjusted result. S4-4: Based on the process from S4-1 to S4-3, the three-dimensional position adjustment of each vertex on the face model is completed sequentially.
4. The method according to claim 1, characterized in that, Includes the following steps: S5-1: Calculate the image fragments in different images corresponding to each facet in the face model using the projection matrix; S5-2: Calculate the cost value of an image fragment to a patch. The smaller the cost value, the higher the correlation between the image fragment and the patch. The patch is denoted as... Image fragments are Then the value is expressed as S5-3: Calculate the visibility of the gap between adjacent patches, if and Adjacent, express and Given a common edge, calculate the integral difference in edge colors between adjacent facets. ; S5-4: Calculate appropriate image fragments for each grid using the principle of hidden Markov random fields; S5-5: Using the Poisson editing method, the various image fragments are fused together to obtain the final face model texture map.
Citation Information
Patent Citations
Face reconstruction method, device and equipment and storage medium
CN109685873A
Method and system for the three-dimensional reconstruction of a human head from of a plurality of images
EP3657440A1
