Combined three-dimensional deformation model

Abstract

The present invention relates to generating a three-dimensional morphable model (3DMM) by combining two known three-dimensional deformation models. According to one aspect of the present invention, a computer-implemented method for generating a new 3DMM by combining a first three-dimensional morphable model (3DMM) with a second 3DMM is described, the method comprising: generating a plurality of first shapes using the first 3DMM; calculating a mapping from a plurality of second parameters of the second 3DMM to a plurality of first parameters of the first 3DMM; for each of a plurality of second shapes generated using the second 3DMM, generating a corresponding first shape; forming a plurality of merged shapes by merging each second shape with the corresponding first shape; performing principal component analysis on the plurality of merged shapes to generate the new 3DMM.

Description

Technical Field

[0001] The present invention relates to generating a three - dimensional morphable model (3DMM) by combining two known three - dimensional deformation models. Background Art

[0002] A three - dimensional morphable model (3DMM) is a powerful statistical tool for representing three - dimensional surfaces of object classes. Due to the ability of 3DMM to infer and represent three - dimensional surfaces, there are many applications in computer vision, computer graphics, biometrics, and medical imaging.

[0003] The creation of a 3DMM usually requires a large amount of work because many registered three - dimensional scans are needed to correctly train the model. In addition, the dataset used to generate the model may subsequently not be publicly available, which limits the possibility of refining the 3DMM. Summary of the Invention

[0004] According to one aspect of the present invention, a computer - implemented method for generating a new 3DMM by combining a first three - dimensional morphable model (3DMM) with a second 3DMM is described. The method includes: generating a plurality of first shapes using the first 3DMM; calculating a mapping from a plurality of second parameters of the second 3DMM to a plurality of first parameters of the first 3DMM; for each second shape among the plurality of second shapes generated using the second 3DMM, generating a corresponding first shape; forming a plurality of merged shapes by merging each second shape with the corresponding first shape; and performing principal component analysis on the plurality of merged shapes to generate the new 3DMM.

[0005] The first 3DMM can be used to generate shapes for a first object, and the second 3DMM is used to generate shapes for a second object, where at least a part of the first object overlaps with at least a part of the second object. The first object can be a complete head shape, and the second object can be a face or an ear. The shapes generated by the second 3DMM have a higher resolution than the shapes generated by the first 3DMM.

[0006] Calculating the mapping may include: for each first shape among the plurality of first shapes: determining a set of second parameters that describe a shape corresponding to at least a part of the first shape; determining a set of first parameters that describe the part of the first shape. Calculating the mapping may also include calculating a regression matrix for mapping multiple sets of second parameters to corresponding multiple sets of first parameters.

[0007] Calculating the regression matrix may include: constructing a first shape matrix including the multiple sets of first parameters; constructing a second shape matrix including the multiple sets of second parameters; calculating the regression matrix by minimizing the following formula:

[0008] ||C h -W h，f C f || 2

[0009] where C h represents the first shape matrix, C f represents the second shape matrix, and W h，f represents the regression matrix. Determining the set of second parameters for each first shape may include registering the first shape with the average shape of the second 3DMM.

[0010] Merging each second shape with the corresponding first shape may include: discarding the region of the first shape corresponding to the second shape; applying non-rigid registration between the second shape and the first shape. Applying the non-rigid registration may include assigning stiffness weights to points on the second shape according to the distance from the center point of the second shape.

[0011] According to another aspect of the present invention, a method for generating a 3D object is described, the method including using a new 3DMM of any method described herein.

[0012] According to another aspect of the present invention, a computer-implemented method for generating a Gaussian process morphable model (GPMM) by combining a first three-dimensional morphable model (3DMM) with a second 3DMM is described, the method including: registering the average shape of the first 3DMM with the average shape of the second 3DMM and a template shape; projecting multiple points of the template shape onto the average shape of the first 3DMM and / or the average shape of the second 3DMM; determining a general covariance matrix of the GPMM according to the projection points of the template shape onto the average shape of the first 3DMM and / or the average shape of the second 3DMM, the covariance matrix of the first 3DMM, and the covariance matrix of the second 3DMM; defining the GPMM according to the general covariance matrix and a predefined average deformation.

[0013] Determining the general covariance matrix of the GPMM may include: determining a local general covariance matrix for each projection point in the projection point pair according to the position of each projection point in the projection point pair in the average shape of the first 3DMM and / or the average shape of the second 3DMM; and determining the general covariance matrix according to the local covariance matrix.

[0014] Determining a local covariance matrix for each projection point in the projection point pair according to the position of each projection point in the projection point pair in the average shape of the first 3DMM and / or the average shape of the second 3DMM includes, for each projection point pair: determining whether any projection point in the projection point pair is located outside the overlapping region of the first 3DMM and the second 3DMM; after an affirmative determination, determining the local general covariance matrix according to the position of the projection point in the projection point pair in the first average shape and according to the local covariance matrix of the first 3DMM. Determining the local general covariance matrix may include taking a weighted sum of the local covariance matrices, where the weights in the weighted sum are based on the position of the projection point in the projection point pair in the first average shape.

[0015] The method may further include: after a negative determination, determining the local general covariance matrix according to the positions of the projection points in the projection point pair in the first average shape and the second average shape, and according to the local covariance matrix of the first 3DMM and the local covariance matrix of the second 3DMM. Determining the local general covariance matrix may include taking a weighted sum of the local covariance matrices, where the weights in the weighted sum are based on the position of the projection point in the projection point pair in the first average shape and the position of the projection point in the projection point pair in the second average shape.

[0016] Projecting a plurality of points of the template shape onto the average shape of the first 3DMM and / or the average shape of the second 3DMM may include: for each point, determining the barycentric coordinates with respect to the grid point triangles in the first average shape and / or the second average shape. Determining the general covariance matrix is based on the barycentric coordinates of each projection point.

[0017] The method may further include refining the GPMM using Gaussian process regression for a plurality of sample objects corresponding to the subject of the first 3DMM.

[0018] According to another aspect of the present invention, a method for generating a 3D object is described, the method including using a GPMM generated by any method described herein.

[0019] According to another aspect of the present invention, a device is described, the device comprising: one or more processors; a memory including computer-readable instructions that, when executed by the one or more processors, cause the device to perform one or more of the methods described herein.

[0020] According to another aspect of the present invention, a computer program product is described, the computer program product including computer-readable instructions that, when executed by a computer, cause the computer to perform one or more of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Embodiments are now described by way of non-limiting examples with reference to the accompanying drawings, in which:

[0022] Figure 1 An overview of a method for generating a three-dimensional morphable model by combining two known three-dimensional morphable models is shown;

[0023] Figure 2 A flowchart of an exemplary method for generating a three-dimensional morphable model by combining two known three-dimensional morphable models is shown;

[0024] Figure 3 An overview of another method for generating a three-dimensional morphable model by combining two known three-dimensional morphable models is shown;

[0025] Figure 4 A flowchart of another exemplary method for generating a three-dimensional morphable model by combining two known three-dimensional morphable models is shown;

[0026] Figure 5 An overview of a method for refining the generated three-dimensional morphable model is shown;

[0027] Figure 6 A schematic example of a system / apparatus for performing any of the methods described herein is shown. DETAILED DESCRIPTION

[0028] Generating a three-dimensional morphable model (3DMM) by combining 3DMMs that are constructed using different templates, have different representational capabilities, and / or have been trained on different training data can enable the generated model to have a combination of the desired properties of the original 3DMMs. For example, the representation of a particular region of the model can be enhanced by combining the model with another model that specifically models that particular region.

[0029] However, the dataset used to generate the original 3DMM may not be available for training the combined model. Instead, only the original 3DMM is available. This specification describes a method for generating a 3DMM by combining the original 3DMM without knowledge of the dataset on which the original 3DMM was trained.

[0030] Compared to the original 3DMM that retains the same number of major components, the combined 3DMM generated using the method described herein can enhance the "compactness" of the model. Compactness is defined by the variance of the training data explained by the model. Additionally, the combined 3DMM can exhibit better generality than the original 3DMM.

[0031] Figure 1 An overview of a method for generating a 3DMM by combining two known 3DMMs is shown. Method 100 includes three stages: a mapping stage 102, a model combination stage 104, and a model registration stage 106. The method takes a first 3DMM and a second 3DMM to generate a new 3DMM therefrom. The first 3DMM and the second 3DMM may have different properties, such as representing an object at different resolutions.

[0032] Each 3DMM can represent a 3D object as a mesh S in three-dimensional space. The mesh includes a plurality of points x in 3D space i , and can be represented as an N-dimensional vector, where N is the number of points in the mesh:

[0033]

[0034] The first 3DMM can generate a first mesh S1 according to a first set of parameters 114p1 (i.e., S1 = S1(p1)). The second 3DMM can generate a second mesh S2 according to a second set of parameters 112p2 (i.e., S2 = S2(p2)). The parameters of the i-th 3DMM can be represented as n i component vectors:

[0035]

[0036] Generally, the number of points in the first mesh and the second mesh does not need to be equal (i.e., in some embodiments, N1 is not equal to N2, where N i is the number of points in the mesh output by the i-th model). The first 3DMM and the second 3DMM can also each use a different number of parameters (i.e., in some embodiments, p1 has a different number of components than p2).

[0037] In the example shown, the first 3DMM is a complete head model described by a first parameter p h , and the second 3DMM is a second parameter p fThe described face model. However, it should be understood that the method can equally be applied to any type of 3DMM that at least partially overlaps. The first 3DMM can correspond to an object of a specific class (e.g., the head), and the second 3DMM corresponds to the features of the object of that class (e.g., the face, facial features, ears).

[0038] In the mapping stage 102, multiple shapes 108 are generated using the first 3DMM. These shapes 108 may be referred to herein as "first shapes". Multiple shapes can be randomly generated from the first model using a predetermined distribution (e.g., Gaussian distribution).

[0039] Then, the mapping from the second model to the first model is determined using the multiple first shapes 108 and the attributes of the second model. The mapping 110 includes mapping multiple parameters 112 of the second model (referred to herein as "second parameters") to multiple parameters 114 of the first model (referred to herein as "first parameters"), and / or vice versa. In some embodiments, the reference shape 116 (e.g., the average shape) of the second model can be registered with each of the multiple first shapes in order to generate a set of second parameters representing the first shape (i.e., representing the overlapping region of the first 3DMM and the second 3DMM) for each of the multiple first shapes. Multiple sets of second parameters 112 represent multiple shapes 118 of the second model, each shape corresponding to one of the multiple first shapes 108. The multiple sets of second parameters 112 and the multiple sets of first parameters 114 are compared to determine the mapping.

[0040] In the model combination stage 104, the determined mapping is used to form multiple combined shapes 120 based on multiple shapes (referred to herein as "second shapes") generated by the second model 122. For each second shape in the second shapes 122 (e.g., the face), a corresponding first shape 124 (e.g., the complete head) is generated using the mapping determined in the first stage. Then, each of the generated first shapes is combined with the corresponding second shape to generate the combined shapes 120.

[0041] In some embodiments, the combination includes, for each of these generated first shapes 124, removing one or more regions corresponding to the second shape used to generate the first shape to generate a partial / reduced shape 126. Then, each second shape is registered with the corresponding partial / reduced shape 126 to create the combined shapes 120.

[0042] In the model registration stage 106, multiple combined shapes 120 are used to generate a new 3DMM. Principal component analysis 128 can be used on the multiple combined shapes to determine the combined 3DMM.

[0043] In some embodiments, the model registration phase 106 further includes registering 130 each merged shape 120 in the merged shapes 120 with a reference template 132 before performing a principal component analysis 128 on the merged shapes. This can reduce the distortion of the merged shapes 120 that occurs during the model combination phase 104. For example, when combining a head shape 3DMM with a face shape 3DMM, the neck of the merged head may deform during the model combination phase 104, and registering the merged head with a head template can make the combined 3DMM more accurately represent the neck region.

[0044] Figure 2 A flowchart illustrating an exemplary method of generating a three-dimensional deformation model by combining two known three-dimensional deformation models is shown.

[0045] Method 200 can be implemented on a computer. Method 200 generates a new 3DMM by combining a first 3DMM and a second 3DMM.

[0046] In operation 2.1, a plurality (n r of) first shapes 108 are generated using the first 3DMM. A plurality of first shapes 108 can be randomly generated from the first model using a predetermined distribution (e.g., a Gaussian distribution).

[0047] The first 3DMM can be a principal component analysis generation model having N1 points in a mesh S1. The first 3DMM can be described using an orthonormal basis formed by a plurality of principal components of the model. In some embodiments, the first n1 principal components corresponding to the n1 eigenvalues with the largest magnitudes are used to form the basis. In some embodiments, the number of eigenvalues n1 used can be greater than 20. For example, the number can be in the range of 20 to 60 components. Then, the first 3DMM can generate the first shape using the following formula:

[0048] S1(p1) = m1 + U1p1 (3)

[0049] where m1 is the first reference shape. An example of the first reference shape is the average shape of the first model, but other reference shapes can also be used. It should be understood that this example is not the only type of first 3DMM that can be used.

[0050] Data in the latent feature space of the first 3DMM can be synthesized by sampling from a predefined statistical distribution, so that a plurality of first shapes 108 are generated by the first 3DMM. For example, a plurality of first shapes 108 can be generated by sampling the first 3DMM using a Gaussian distribution defined by the principal eigenvalues of the first 3DMM. The standard deviation of the distribution can be based on the eigenvalues of the corresponding eigenvectors of each parameter. For example, the standard deviation can be equal to the square root of the eigenvalue. Other statistical distributions can also be used.

[0051] A total of n r first shapes can be generated to form a plurality of first shapes 108.

[0052] In embodiments involving a combined head 3DMM and a face 3DMM, the first 3DMM can be a head shape model that models the shape of the head. The shape of the head can include the cranial shape. The head shape model can also include facial details (i.e., is a craniofacial head model). An example of such a head shape model is the Liverpool Yorkhead model (LYHM). In this model, the facial region is described at the same spatial resolution as the rest of the skull.

[0053] In operation 2.2, a mapping from a plurality of second parameters p2 of the second 3DMM to a plurality of first parameters p1 of the first 3DMM is determined. The reference shape 116 (e.g., the average shape) of the second model can be registered with each of the plurality of first shapes generated in operation 2.1 to generate a set of second parameters representing the first shape for each of the plurality of first shapes. In some embodiments, the parameters p2 of the second 3DMM that describe the shape corresponding to at least a portion of the first shape are found and a set of first parameters that describe the portion of the first shape are determined.

[0054] The second parameters can be determined by registering the first shape 108 with the reference shape 116 (e.g., the average shape) of the second model. For example, non-rigid registration such as non-rigid iterative closest point (NICP) registration can be used for registration. After registering each of the first shapes in the first shape 108 with the reference shape 116, the portion of the first shape 108 corresponding to the second 3DMM (e.g., the overlapping portion of the first 3DMM and the second 3DMM) is determined. These corresponding portions of the first shape 108 are projected onto the subspace of the second model to determine the corresponding second parameters p2.

[0055] Thus, for each of the first shapes in the first shape 108, there is a pair of parameter sets (p1, p2) corresponding to the first 3DMM and the second 3DMM. In some embodiments, the first element of the pair corresponds to an object of the first 3DMM, while the second element of the pair corresponds to a feature of an object generated by the second 3DMM. For example, the pair (p1, p2) can correspond to a full head representation and a face representation, respectively.

[0056] For example, in an embodiment where the first 3DMM is a full head model and the second 3DMM is a face model, the facial region of each full head of the first shape 108 is used to determine the parameters of the face model that encodes the face.

[0057] In some embodiments, the second 3DMM may be a principal component analysis generative model with N2 points in the mesh S2. The second 3DMM may be described using an orthogonal basis formed by multiple principal components of the model. In some embodiments, the first n2 principal components corresponding to the n2 eigenvalues with the largest magnitudes are employed to form the basis. In some embodiments, the number of principal components n2 used may be greater than 20. For example, the number may be in the range of 20 to 60 components. Then, the second 3DMM may generate a second shape using the following formula:

[0058] S2(p2) = m2 + U2p2 (4)

[0059] where m2 is the second reference shape. An example of the second reference shape is the average shape of the second model, but other reference shapes may also be used. It should be understood that this example is not the only type of second 3DMM that can be used.

[0060] The second 3DMM may include a feature model that generates a feature model that at least partially overlaps with the model generated by the first 3DMM. In embodiments involving a combined head 3DMM and a face 3DMM, the second 3DMM may be a face model / face feature model that models one or more sub-features of the face (such as the nose, eyes, ears, and / or mouth). An example of such a head shape model is a large scale facial model (LSFM). This model combines custom patterns in terms of age, gender, and ethnicity.

[0061] After determining the corresponding pairs of parameter sets, the mapping from p1 to p2 (and / or p2 to p1) is determined. In some embodiments, determining the mapping includes: calculating a regression matrix for mapping multiple sets of second parameters to the corresponding multiple sets of first parameters.

[0062] In some embodiments, a first matrix including all the parameters p1 of the first shape 108 in the first 3DMM may be constructed. A corresponding second matrix including all the corresponding parameters p2 of the first shape 108 in the second 3DMM may be constructed. Using these two matrices, a weight matrix is determined, which describes the mapping between the parameters of the first 3DMM and the parameters of the second 3DMM.

[0063] For example, the problem of finding the weight matrix can be formulated as a least squares problem of minimizing an objective function:

[0064] ||C1 - W 1，2 C2|| 2 (5)

[0065] This can be solved in a variety of ways. For example, by using the normal equations, the solution can be given by the following formula:

[0066]

[0067] where, is the right pseudo-inverse of C2.

[0068] In operation 2.3, for each of the plurality of second shapes generated using the second 3DMM, a corresponding first shape is generated. The plurality of second shapes can be generated by the second 3DMM by randomly sampling the second 3DMM or the like. The method for sampling the first 3DMM described above with respect to operation 2.1 can be used to sample the second 3DMM. Alternatively, pre-existing second shapes can be used, such as second shapes obtained from a database of second shapes. When using a facial 3DMM as the second 3DMM, an example of such a database is the MeIn3D database. However, other databases can alternatively or additionally be used.

[0069] The mapping determined in operation 2.2 can be used to generate the first shape S1′ from each second shape S2′. A weighting matrix can be used to create the generated first shape from each second shape. For example, the generated first shape can be generated using the following equation:

[0070]

[0071] In some embodiments, the second shape is a face generated by a face model. According to the face model, a complete head model is generated using the mapping determined in operation 2.2. For example, the LSFM of the face can be used to generate the LYHM of the complete head.

[0072] In operation 2.4, a plurality of merged shapes are formed by merging each second shape with the corresponding first shape. The merged shapes include elements of the generated first shape and elements of the second shape. The elements of the second shape can replace the corresponding elements of the generated first shape.

[0073] In some embodiments, the elements of the generated first shape that correspond to the elements of the second shape are discarded from the generated first shape to produce a reduced shape (i.e., a partial first shape). For example, if the second shape is a face and the generated first shape is a complete head, the facial region of the complete head can be discarded.

[0074] Then, replace the discarded elements of the first shape generated with the second shape. A registration process can be used to merge these two shapes. The NCIP framework can be used to merge the meshes of the first shape and the second shape. To preserve the features of the second shape while providing a smooth combination of the two meshes, the deformation during registration can be restricted to the outer part of the mesh of the second shape. For example, this can be achieved by introducing a higher stiffness weight into the inner mesh than the outer mesh during the registration process. These weights can be calculated based on the Euclidean distance from a reference point in the second mesh (such as the nose in a face model). For example, the farther a point is from the reference point, the smaller the weight may be.

[0075] In operation 2.5, perform principal component analysis on the multiple merged shapes to generate a new 3DMM. The new 3DMM S n with N n points in the mesh can be described by the following equation:

[0076] S n (p n ) = m n + U n p n (8)

[0077] where, m n is the new reference shape, is the new principal component matrix including the first n n principal components, p n is the vector of new parameters corresponding to the first n n principal components of the new model. An example of the new reference shape is the average shape of the new 3DMM, but other reference shapes can also be used. It should be understood that this example is not the only type of new 3DMM that can be used. In some embodiments, the number n n of principal components used can be greater than 20. For example, this number can be in the range of 20 to 60 components.

[0078] Performing principal component analysis on the multiple merged shapes results in a newly generated model that exhibits a combination of the attributes of the first model and the second model.

[0079] In some embodiments, before performing principal component analysis, the merged shapes are each registered with a template shape. Registering the merged shapes with the template can reduce the inconsistencies in the merged models that may occur during the model combination phase. A registration process such as NICP can be applied between the merged shapes and the template shape to register each merged shape with the template shape. A weighting scheme can be used during the registration process. Weights can be assigned to each point based on the position of each point in the merged shape (and / or template shape) relative to a reference point in the shape. For example, the reference point can be the centroid of the shape or a prominent feature of the shape. Weights can be assigned based on the Euclidean distance from the reference point. Additional landmarks can be added around one or more prominent features of the merged shape and / or template shape to preserve the shape of these landmarks during registration.

[0080] For example, in an embodiment related to a 3DMM of the head, weights can be assigned to the points of the merged head based on the Euclidean distance of the points of the merged head from the center of mass of the head. For example, this can reduce any inconsistencies in the neck region that may arise from the regression scheme. For the region around the ears, a set of additional landmarks, such as an additional 50 landmarks, are introduced to control the registration and preserve the overall shape of the ear region.

[0081] Figure 3 An overview of another method for generating a three-dimensional morphable model by combining two known three-dimensional morphable models is shown.

[0082] In some embodiments, the 3DMM can be a Gaussian process morphable model (GPMM). The GPMM is a generalization of the classical point distribution model that combines Gaussian processes (e.g., a model constructed using principal component analysis). The shape S is modeled as a deformation of a reference shape S R as follows:

[0083] S = {x + u(x)|x ∈ S R} (9)

[0084] where x is a point in the reference shape and u is the deformation function where the deformation is modeled as a Gaussian process where is the mean deformation and is the covariance function or kernel. The covariance function / kernel can be discretized into a covariance matrix. The covariance matrix K is a discrete approximation of the covariance function / kernel.

[0085] Given at least partially overlapping first and second GPMMs, a common covariance matrix K can be constructed from the covariance matrix K1 of the first GPMM and the covariance matrix K2 of the second GPMM UCombine the first GPMM and the second GPMM to form a new GPMM. In the example shown, the first GPMM is a head model and the second GPMM is a face model, but it is understood that the method is applicable to the combination of 3DMMs of any shape type.

[0086] Method 300 uses the average shape 302 (first average shape) of the first GPMM and the average shape 304 (second average shape) of the second GPMM. Register the average shape 302 of the first GPMM and the average shape 304 of the second GPMM using a registration process. An example of such a process is NICP registration. In some embodiments, the first average shape 302 and the second average shape 304 can be transformed into the same scale space before registration. During registration, a weighting scheme can be applied, such as the weighting scheme described above with respect to Figure 2 the described weighting scheme.

[0087] Register the template shape 306 with the first average shape 302 and the second average shape 304. This can be performed as a separate step after registering the first average shape 302 and the second average shape 304, or as part of the same registration process. NICP registration can be used to register the head template 306 with the first average shape 302 and the second average shape 304. Registering the template shape, registering the first average shape, and registering the second average shape together form a registered model 308.

[0088] After creating the registered model 308, determine the general covariance matrix K from the covariance matrices of the first GPMM and the second GPMM U . Determine the general covariance matrix by projecting the point pairs of the template shape 306 onto the average shape 302 of the first GPMM and / or the average shape 304 of the second GPMM using the registered model 308. A weighted sum of the components of the covariance matrix of the first GPMM and / or the components of the covariance matrix of the second GPMM is used to determine the general covariance matrix.

[0089] In some embodiments, determine the general covariance matrix from a plurality of local general covariance matrices 310 For each of the plurality of point pairs (i, j) in the registered template shape, a local general covariance matrix can be calculated The local general covariance matrix can be calculated by projecting the points of each pair onto the registered average shape 302 of the first GPMM and / or the registered average shape 304 of the second GPMM and mixing the local covariance matrix of the first GPMM 302 and / or the local covariance matrix of the second GPMM 304 according to the projection.

[0090] In some embodiments, the projection involves determining the barycentric coordinates 312, 314 of points i and j in the registered first average shape and / or the registered second average shape. The barycentric coordinates can be taken with respect to the mesh point triangles 316, 318 in the registered first average shape and / or the registered second average shape. The barycentric coordinates can be used to mix the local covariance matrix associated with the mesh point triangles 316, 318 with the local general covariance matrix, as described below with respect to Figure 4 as described.

[0091] The general covariance matrix is used to define the combined GPMM. In some embodiments, the combined GPMM undergoes a refinement process after the combination of the first GPMM and the second GPMM, as described below in connection with Figure 5 and 6 as described.

[0092] Figure 4 A flowchart of another exemplary method for generating a three-dimensional deformation model by combining two known three-dimensional deformation models is shown. Method 400 can be implemented on a computer. A Gaussian process morphable model (GPMM) is generated by combining a first 3DMM and a second 3DMM. Hereinafter, for convenience, it is assumed that the second 3DMM corresponds to a sub-region of the first 3DMM (e.g., the second 3DMM is a face model and the first 3DMM is a full head model). However, it should be understood that the situation may be reversed.

[0093] The first 3DMM and / or the second 3DMM can be a GPMM defined by an average deformation and a covariance function / kernel. In some embodiments, the first 3DMM and / or the second 3DMM are not in the form of a GPMM, but are converted to a GPMM by determining a covariance matrix corresponding to the covariance function / kernel.

[0094] For example, the principal orthonormal basis and eigenvalues of the first 3DMM and / or the second 3DMM can be used to determine the covariance matrix K of the 3DMM using the following equation i :

[0095]

[0096] where, is the covariance matrix of the i-th model, is the diagonal matrix of the first n i eigenvalues of the model, is the principal component matrix including the first n i principal components of the i-th model corresponding to the n i eigenvalues in Λ n principal components of the i-th model corresponding to the n

[0097] In operation 4.1, the average shape 302 of the first 3DMM, the average shape 304 of the second 3DMM, and the template shape 306 are registered with each other. Registration methods such as NICP are used to register these elements with each other.

[0098] In Figure 3 the example shown, the first 3DMM is a head model, the second 3DMM is a face model, and the template shape is a template head. For example, the head model can be LYHM. For example, the face model can be LSFM. The head template can be a mesh of the reference head shape.

[0099] In operation 4.2, a plurality of points of the template shape are projected onto the average shape of the first 3DMM and / or the average shape of the second 3DMM.

[0100] The points in the template shape are projected onto the mesh of the average shape of the first 3DMM. If the points of the template shape are projected onto the region of the average shape of the first 3DMM that overlaps with the average shape of the second 3DMM, then these points can alternatively or additionally be projected onto the average shape of the second 3DMM. Any projection method known in the art can be used to perform these projections.

[0101] In some embodiments, the exact position of the projected points in the average shape of the first 3DMM and / or the average shape of the second 3DMM is based on the barycentric coordinates of the triangle of the mesh points in the corresponding mesh with respect to the relevant average shape. For the i-th projected point, the barycentric coordinates in the average shape of the first 3DMM and / or the average shape of the second 3DMM with respect to the corresponding mesh triangle are identified, where are the vertices of the mesh triangle.

[0102] In operation 4.3, a general covariance matrix of the GPMM is determined based on the projected point pairs of the template shape onto the average shape of the first 3DMM and / or the average shape of the second 3DMM, the covariance matrix of the first 3DMM, and the covariance matrix of the second 3DMM.

[0103] Given a point pair (i,j) in the template shape, a local general covariance matrix can be determined based on the projection of the point (i,j) onto the average shape of the first 3DMM and / or the average shape of the second 3DMM. The local general covariance matrices for each point pair in the template shape are determined based on the covariance matrix of the first 3DMM and / or the second 3DMM. The local general covariance matrices are combined to form a general covariance matrix K that characterizes the new / merged GPMM U .

[0104] In some embodiments, for each point pair (i, j) in the template shape, it is determined whether the projection of the first point i and / or the second point j lies within the region where the average shapes of the first 3DMM and the second 3DMM overlap.

[0105] If at least one point lies outside the region where the average shapes of the two 3DMMs overlap, the local general covariance matrix is determined based on the positions of the two points in the point pair in the average shape of the first model. In embodiments where the point positions are described in terms of the barycentric coordinates of the triangles in the mesh relative to the average shape, the local general covariance matrix can be determined by taking the weighted sum of the local covariance matrices between the vertices of two triangles. The weights can be based on the respective barycentric coordinates of the two points (i, j).

[0106] For example, let be the barycentric coordinate of point i relative to triangle and let be the barycentric coordinate of point j relative to triangle Each vertex pair (v, k) between the two triangles is associated with a local vertex covariance matrix (where ). The local covariance matrix for the point pair can be determined by mixing the local vertex covariance matrices to create a mixed local covariance matrix. For example, the weighted sum of the local vertex covariance matrices can be taken as follows:

[0107]

[0108] where, are the weights. The weights can be a function of the barycentric coordinates of points i and j. For example:

[0109]

[0110] If at least one point lies within the region where the average shapes of the two 3DMMs overlap, the local general covariance matrix is determined based on the positions of the two points in the point pair in the average shape of the first model and the average shape of the second model. In embodiments where the point positions are described in terms of the barycentric coordinates of the triangles in the mesh relative to the first average shape and the second average shape, the local general covariance matrix can be determined by taking the weighted sum of the mixed local covariance matrix of the first 3DMM and the mixed local covariance matrix of the second 3DMM, as follows:

[0111]

[0112] where, ρ ij =(ρ i +ρ j ) / 2 is based on the position of the point (i, j) relative to the reference point of the registration mesh (ρ i, ρ j )'s normalized weight. and are the mixed covariance matrices of the first 3DMM and the second 3DMM respectively. In the case where at least one point is outside the region where the average shapes of the two 3DMMs overlap, these mixed covariance matrices can each be determined using the method described above.

[0113] Local general covariance matrices can be determined for each pair of points in the template shape. These local covariance matrices are combined to form a complete general covariance matrix characterizing the merged GPMM.

[0114] In operation 4.4, the merged GPMM is defined using the general covariance matrix. For example, using the template shape S t and the general covariance matrix K U , the covariance function / kernel for points x and y can be defined as:

[0115]

[0116] where CP(S t , x) is a function that returns the index of the point on the template shape closest to x. The average deformation μ U can be set to a predefined value, such as a zero vector. Thus, the merged GPMM is defined as:

[0117] [[ID=--30]]

[0118] Any other method for defining and / or approximating a GPMM based on the general covariance matrix can also be used.

[0119] Figure 5 shows an overview of the method for refining the generated GPMM. After generating the merged GPMM using the Figure 3 and Figure 4 methods, a refinement process 500 can be applied to refine the merged GPMM. The refinement process includes using Gaussian process regression to determine the refined GPMM based on the merged GPMM and one or more original scans.

[0120] The merged GPMM can be used to generate multiple shapes corresponding to the theme of the first 3DMM (e.g., a complete head shape) based on multiple shapes corresponding to the theme of the second 3DMM (e.g., a face). Gaussian process regression can be used to generate multiple first shapes. Given a set of observed deformations X affected by Gaussian noise , the posterior model is calculated by Gaussian process regression according to the current model :

[0121]

[0122] Average value μ p And covariance k p Can be calculated using the following equation:

[0123] μ p (x) = μ(x) + K X (x) T (K XX + σ 2 I) -1 x(17)

[0124] And

[0125] k p (x, x′) = k(x, x′) - K X (x) T (K XX + σ 2 I) -1 K X (x). (18)

[0126] Wherein,

[0127]

[0128] And

[0129]

[0130] The refinement method 500 uses one or more scans of the 3D object (also referred to herein as "original scan") 502 and the reference shape GPMM 504 as inputs. The input scan 502 corresponds to a subset of the reference shape 504. For example, in Figure 5 The example shown, the reference shape 504 is the complete head and the input scan 502 is the face.

[0131] Each input scan 502S includes a plurality of points. One or more point sets can be used to define one or more landmarks L of the input scan s = {l1,... l n}。The reference shape 504S t Includes a plurality of points. One or more point sets can be used to define one or more landmarks L of the reference shape st = {l1,... l m}。For example, the reference shape 504 can be the average shape of the combined GPMM.

[0132] Given one or more input scans 502 and the reference shape 504, an initial posterior model is determined according to the sparse deformation defined by the corresponding landmarks in the average shape 504 and the input scan 502. For example, the initial posterior model can be defined using the following equation:

[0133]

[0134] Among them, is the merged GPMM. The posterior GPMM defines the new average shape 506 of the GPMM. The new average shape 506 can be used to define a new average deformation from the reference model.

[0135] Then, for example, the posterior model is iteratively refined using the iterative closest point algorithm. In each iteration i, the current registration result S is determined based on the reference shape and the average deformation of the posterior model in the previous iteration i reg .

[0136] For each point x in the reference shape, the current registration can be determined using the following equation:

[0137]

[0138] Among them, is the average deformation of the posterior model at the previous iteration. This can be considered that the reference shape 504 contains the average deformation of the posterior model of the refined previous iteration.

[0139] For each point of the current registration result, the closest point U on the input scan 502 is determined i . The difference between these points and the corresponding points on the current registration result is used to determine the updated GPMM. For example, the updated GPMM can be determined using the following equation:

[0140]

[0141] After a threshold number of iterations, the final registration result 508S is obtained reg .

[0142] Then, the region of the final registration result 508 corresponding to the input scan 502 (for example, if the input scan 502 is a complete head, it is the facial region of the final registration result 508) is non-rigidly aligned with the input scan 502 to obtain the reconstructed shape 510. Then, the reconstructed shape 510 is used to determine the new sample covariance matrix 512.

[0143] In some embodiments, the reconstruction obtained by the above method may produce unrealistic 3D shapes. The covariance matrix can be modified before Gaussian process regression to reduce this effect. The modification includes calculating the principal components by decomposing the covariance matrix. The modification also includes reconstructing the covariance matrix with fewer statistical components, for example, using Equation 10, but reconstructing the covariance matrix with fewer statistical components.

[0144] In some embodiments, the process may be performed with multiple input scans 502 to determine multiple reconstructed scans 510. Statistical modeling 514 may be used to determine a new sample covariance matrix 512.

[0145] Gaussian process regression may be repeated using the new sample covariance matrix to refine the reconstructed shape. The refinement model may be determined by performing principal component analysis on the refined reconstructed shape.

[0146] Figure 6 A schematic example of a system / apparatus for performing any of the methods described herein is shown. The system / apparatus shown is an example of a computing device. Those skilled in the art will understand that other types of computing devices / systems may also be used to implement the methods described herein, such as a distributed computing system.

[0147] The apparatus (or system) 600 includes one or more processors 602. The one or more processors control the operation of the other components of the system / apparatus 600. For example, the one or more processors 602 may include a general-purpose processor. The one or more processors 602 may be a single-core device or a multi-core device. The one or more processors 602 may include a central processing unit (CPU) or a graphical processing unit (GPU). Alternatively, the one or more processors 602 may include dedicated processing hardware, such as a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0148] The system / apparatus includes working or volatile memory 604. The one or more processors may access the volatile memory 604 to process data and may control the storage of data in the memory. The volatile memory 604 may include any type of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), or it may include flash memory, such as an SD-Card.

[0149] The system / apparatus includes non-volatile memory 606. The non-volatile memory 606 stores a set of operation instructions 608 for controlling the operation of the processor 602 in the form of computer-readable instructions. The non-volatile memory 606 may be any type of memory, such as read only memory (ROM), flash memory, or magnetic drive memory.

[0150] One or more processors 602 are used to execute the operation instructions 408 to cause the system / apparatus to perform any method described herein. The operation instructions 608 may include code related to the hardware components of the system / apparatus 600 (i.e., drivers), as well as code related to the basic operation of the system / apparatus 600. Generally, one or more processors 602 use volatile memory 604 to temporarily store data generated during the execution of the operation instructions 608, thereby executing one or more instructions of the operation instructions 608 that are permanently or semi-permanently stored in non-volatile memory 606.

[0151] The implementation of the methods described herein may be implemented in digital electronic circuits, integrated circuits, application specific integrated circuits (ASICs) specifically designed, computer hardware, firmware, software, and / or combinations thereof. These may include computer program products (e.g., software stored on a disk, optical disk, memory, programmable logic device, etc.) that include computer-readable instructions that, when executed by a computer (e.g., in combination with Figure 6 the computer described), cause the computer to perform one or more of the methods described herein.

[0152] Any system feature described herein may also be provided as a method feature, and vice versa. As used herein, module plus function features may be represented according to their corresponding structures. Specifically, method aspects may be applied to system aspects, and vice versa.

[0153] In addition, any, some, and / or all features in one aspect may be applied in any suitable combination to any, some, and / or all features in any other aspect. It should also be understood that the specific combinations of the various features described and defined in any aspect of the present invention may be implemented and / or provided and / or used independently.

[0154] Although several embodiments have been shown and described, those skilled in the art will understand that changes may be made in these embodiments without departing from the principles of the present invention, the scope of which is defined in the claims.

Claims

1. A computer-implemented method for generating a new 3DMM by combining a first 3DMM and a second 3DMM, where the 3DMM is a three-dimensional deformation model, characterized in that, The method includes: Generating a plurality of first shapes using the first 3DMM; Calculating a mapping from a plurality of second parameters of the second 3DMM to a plurality of first parameters of the first 3DMM; For each second shape among the plurality of second shapes generated using the second 3DMM, generating a corresponding first shape; Forming a plurality of merged shapes by merging each second shape with the corresponding first shape; Performing principal component analysis on the plurality of merged shapes to generate the new 3DMM.

2. The method according to claim 1, characterized in that, The first 3DMM is used to generate shapes for a first object, and the second 3DMM is used to generate shapes for a second object, where at least a part of the first object overlaps with at least a part of the second object.

3. The method according to claim 2, wherein The first object is a head, and the second object is a face or an ear.

4. The method according to claim 2 or 3, characterized in that, The shapes generated by the second 3DMM have a higher resolution than the shapes generated by the first 3DMM.

5. The method according to any one of claims 1-3, characterized in that, Calculating the mapping includes: For each first shape among the plurality of first shapes: Determining a set of second parameters that describe a shape corresponding to at least a part of the first shape; Determining a set of first parameters that describe the part of the first shape; Calculating a regression matrix for mapping the sets of second parameters to the corresponding sets of first parameters.

6. The method according to claim 5, characterized in that Calculating the regression matrix includes: Constructing a first shape matrix including the sets of first parameters; Constructing a second shape matrix including the sets of second parameters; Calculating the regression matrix by minimizing the following formula: ||C h -W h，f C f || 2 Among them, C h represents the first shape matrix, C f represents the second shape matrix, W h,f represents the regression matrix.

7. The method according to claim 5 or 6, characterized in that, Determining the set of second parameters for each first shape includes registering the first shape with the average shape of the second 3DMM.

8. The method according to any one of claims 1 to 3, characterized in that Merging each second shape with the corresponding first shape includes: Discarding the region of the first shape corresponding to the second shape; Applying non-rigid registration between the second shape and the first shape.

9. The method according to claim 8, wherein Applying the non-rigid registration includes assigning stiffness weights to points on the second shape according to the distance from the center point of the second shape.

10. A device, characterized in that, Includes: One or more processors; A memory including computer-readable instructions that, when executed by the one or more processors, cause the device to perform the method according to any one of claims 1 to 9.

11. A computer program product, characterized in that, Includes computer-readable instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

Images