Modeling method for creating virtual model of head of user
By acquiring and processing multiple depth images of user heads, combining specific algorithms and technical means, the problems of insufficient accuracy and high computational cost in the existing technology are solved, and a virtual model with high accuracy and low computational cost are achieved. Production.
Patent Information
- Application Number
- CN202380050305.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-08-03
- Filing Date
- 2023-07-31
- Publication Date
- 2025-02-14
AI Technical Summary
The prior art is difficult to achieve high accuracy when making virtual models of user heads, especially in head areas involving the use of glasses, and at the same time, the calculation cost is high.
By obtaining multiple depth images of the user's head under different postures, using correction processing and specific marking point positioning, estimating the head posture and transforming it, combining rigid sequential iteration of the nearest point algorithm and the cost function minimization algorithm, gradually constructing and aligning Synthesize point clouds and statistical head models.
A virtual model of the user's head is achieved with high accuracy in the head area where glasses are used, while reducing computational costs.
Smart Images
Figure CN119452392A_ABST
Abstract
Description
[0001] The present invention relates to a modeling method for making a three-dimensional virtual model of a user's head, which can be implemented using a terminal such as a personal computer, a smart phone, a laptop computer, etc.
[0002] This modeling approach can be used in applications that enable display accessories (such as glasses) to be virtually worn on a virtual 3D model of a user's head. This is particularly useful for promoting online sales of products, as potential buyers typically try on these products to assess their aesthetics before purchasing.
[0003] In order to obtain the most realistic trial effect possible, a three-dimensional virtual model of the user's head must be as accurate as possible.
[0004] Known modeling methods first involve acquiring multiple depth images of the user's head in different poses. A depth image is defined as a two-dimensional or 2D image in which each pixel is associated with a corresponding value of the distance between the point represented by the pixel and the point where the image was observed and captured. Depth image acquisition systems are known and may include either a stereo vision system or a so-called depth camera. If the depth image acquisition system is properly calibrated, then the value of the distance associated with the pixel can be determined directly in actual metric units (such as meters).
[0005] In any case, the depth image acquisition system is configured to capture a two-dimensional image, determine a two-dimensional map of data representing the distance of each pixel from the observation point, and associate the two-dimensional map with the acquired two-dimensional image in a manner such that each pixel in the acquired two-dimensional image is associated with the data of the two-dimensional map. This association is performed with the aid of a synchronization algorithm.
[0006] Therefore, each pixel of each depth image can be represented by a point in a three-dimensional Cartesian reference system, with three spatial coordinates derived from the two-dimensional map. Each depth image is then associated with a corresponding three-dimensional point cloud.
[0007] Nowadays, more and more terminals (such as smartphones, tablets, etc.) are equipped with depth cameras for detection in several application contexts (for example, in the field of architecture or glasses).
[0008] After acquiring a plurality of depth images, known modeling methods generally involve a merging step, in which the three-dimensional point clouds of the acquired depth images are merged together to form a composite point cloud.
[0009] After the merging step, a step of aligning the synthetic point cloud with a statistical head model, such as the so-called Basel head, is provided. A statistical head model is a geometric model that describes a collection of semantically similar objects of compact shape. The statistical head model consists of an average head obtained from the average of a large number of three-dimensional heads and a deformation described by a linear combination of a set of eigenvalues on the corresponding eigenvectors. By varying the eigenvalues, different anatomically coherent heads are obtained. Finally, the known modeling method comprises a so-called deformation step, in which the statistical head model is subjected to a non-rigid deformation process by means of a non-rigid iterative closest point or ICP algorithm, aiming to minimize the distance in a way that the corresponding points in the synthetic point cloud are as close as possible. In practice, the statistical head model is deformed to resemble the synthetic point cloud without losing the typical human proportions included in the statistical head model. The deformed statistical head model creates the final virtual model of the user's head.
[0010] Known modeling methods suffer from certain disadvantages.
[0011] The first drawback is that the deformation is performed on all points of the synthetic point cloud using the same algorithm and objective; the accuracy of the obtained model requires a lot of computational effort to reach a high level.
[0012] A second disadvantage is that the accuracy of the model may not be sufficient for trial use of a pair of virtual glasses, which involves virtually positioning a three-dimensional virtual model of the glasses on a final virtual model of the head, requiring high accuracy particularly in the contact area between the two virtual models.
[0013] The purpose of the present invention is to overcome the above-mentioned drawbacks and in particular to devise a modeling method for making a virtual model of a user's head that is able to achieve high accuracy in the areas of the head that relate to the use of glasses while maintaining low computational costs compared to the prior art.
[0014] This and other objects of the present invention are achieved by making a modeling method for making a virtual model of a user's head as set forth in claim 1.
[0015] Further characteristics of the modeling method for making a virtual model of a user's head are the objects of the dependent claims.
[0016] Another object of the present invention is to overcome the above-mentioned disadvantages and in particular to design a computer program that can be loaded into the memory of an electronic computer, which computer program includes instructions that enable the electronic computer to implement a modeling method for making a virtual model of a user's head that is more accurate and faster than the prior art.
[0017] This further object according to the invention is achieved by making a computer program as set out in claim 9.
[0018] The features and advantages of the modeling method for making a virtual model of a user's head according to the present invention will become more apparent from the following exemplary but non-limiting description with reference to the accompanying drawings, in which:
[0019] Figure 1 is a flow chart showing a modeling method according to a first embodiment of the present invention;
[0020] Figure 2 is a flow chart showing a modeling method according to a second embodiment of the present invention.
[0021] Referring to the drawings, a modeling method, generally referred to as 100, for making a virtual model of a user's head is shown.
[0022] Such a modeling method 100 can be implemented by an electronic calculator provided with a memory for data storage. In particular, the modeling method 100 can be implemented by means of a modeling or deformation program or software loaded into a computer memory.
[0023] Therefore, such a modeling or deformation program includes instructions that cause an electronic computer to implement the modeling method 100 when the electronic computer runs the program.
[0024] The modeling program 100 can be combined with an e-commerce platform for glasses in such a way that each time a user purchases glasses on the glasses e-commerce platform, the modeling or deformation program runs.
[0025] More generally, the modeling or morphing program may also be run by the operator's electronic terminal before the user purchases a pair of glasses.
[0026] The electronic terminal may be a desktop personal computer, a smart phone, a tablet computer, a PDA, or the like.
[0027] The electronic terminal comprises a depth camera and an electronic computer, wherein the electronic computer is provided with a memory for storing data, wherein the modeling or deformation program or software according to the present invention is loaded.
[0028] The modeling method 100 comprises the following steps:
[0029] - acquiring 110 a plurality of depth images of the user's head in different poses, wherein each of the depth images comprises a two-dimensional or 2D image and a 2D image of data representing the distance of each pixel from a viewing point coinciding with the position of the depth camera
[0030] a map, the two-dimensional map being aligned with the two-dimensional or 2D image such that each pixel of the acquired two-dimensional image is associated with data of the two-dimensional map;
[0031] - for each of said acquired depth images, performing 120 processing of the acquired depth images by correcting the alignment between the two-dimensional data map and the two-dimensional or 2D image;
[0032] - for each of said two-dimensional images in said acquired and processed depth images, locating 130 a first marker point on the two-dimensional image corresponding to a specific feature of the head;
[0033] - for each of said two-dimensional images of said acquired depth images, estimating 140 a pose of the head by estimating the yaw, pitch and roll angles of the head and the positioning of the head along three axes X, Y, Z of a first Cartesian axis reference system XYZ;
[0034] - for each of said two-dimensional images in said depth image, performing a transformation 150 of the pose of the head by modifying the previously estimated yaw, pitch and roll angles of the head and the positioning of the head so that they are referenced to a second Cartesian axis reference system having a predefined relationship with the first Cartesian axis reference system XYZ
[0035] X'Y'Z';
[0036] - for each of said acquired depth images, projecting 160 said first marker point located in the corresponding point cloud of said depth image so as to obtain a corresponding first three-dimensional marker point in a second Cartesian axis reference system X'Y'Z';
[0037] - iteratively associating 170 the point clouds of the depth images in a pairwise manner by means of a rigid sequential iterative closest point algorithm or ICP and processing them until a synthetic point cloud is obtained that represents a synthesis of all the selected poses and also includes the first three-dimensional marker point;
[0038] - Aligning the synthetic point cloud by 180° with the statistical head model by means of a rigid iterative closest point type alignment algorithm or nearest neighbor ICP;
[0039] - deforming 190 the statistical head model by means of an algorithm for minimizing a predefined cost function so that it comes as close as possible to the synthetic point cloud of the model of the user's head.
[0040] The step of acquiring 110 multiple depth images is particularly performed by taking pictures using a depth camera or a camera system of a mobile terminal or a desktop PC suitable for use as a depth camera.
[0041] The step of locating 130 first marker points corresponding to specific head features (such as eyes, nose, cheekbones, ears, etc.) is performed by a first algorithm. The term "locating" means to indicate the position or spatial coordinates of a pixel in the space of a two-dimensional image. For example, the first algorithm can be a machine learning algorithm, which will not be described in detail as it is known in the prior art.
[0042] If the first algorithm is of the machine learning type, it detects and locates first marker points corresponding to specific head features based on a computational model trained on a collection of images for which the positions of these points are known.
[0043] The estimation 140 of the pose is performed with the aid of an algorithm called "Solve PnP", which stands for "Solve Perspective n-Point". Given as input (i) first two-dimensional markers, (ii) three-dimensional coordinates of points of a statistical head model (semantically corresponding to the first two-dimensional markers), (iii) distortion coefficients and the intrinsic matrix of the camera, the "Solve PnP" algorithm estimates the pose of the head in the XYZ Cartesian axis reference system by generating as output a pair of vectors, one of which has as components the three yaw, roll, pitch angles and the other vector having as components the coordinates of the head point along the three X, Y, Z axes of the reference system. Thus, each point of the head is associated with a pair of relative vectors generated by the pose estimation algorithm; such a pair of vectors represents the position and orientation of each point of the head in the first Cartesian axis reference system XYZ. This first reference system is the reference system of the camera and can be different from or equal to a second Cartesian axis reference system X'Y'Z', which is the reference system of the pose estimation algorithm (i.e. the "Solve PnP" algorithm). This second reference system coincides with the reference system of the statistical head model and has a predefined and therefore known relationship with the first reference system. This predefined and known relationship is the initial setting.
[0044] The semantic corresponding point means a point indicating the same feature (such as nose tip, earlobe, etc.) of the reference head.
[0045] The transformation 150 of the pose of the head is intended to obtain, for each point of the head, a pair of new vectors representing the position and orientation of the point in the second Cartesian axis reference system X'Y'Z', which vectors are consistent with the position and orientation of the statistical head model as reported above. Therefore, from a computational point of view, the transformation of the pose is used to facilitate (i.e., speed up) the step of aligning 180 the synthetic point cloud with the statistical head model.
[0046] Depth image processing 120 refers to processing that involves looking up the distortion coefficients and intrinsic matrix of the camera as provided by the manufacturer, and then processing the image based on these distortion coefficients and intrinsic matrix. This processing modifies the alignment between the two-dimensional data map and the two-dimensional or 2D image, that is, the association between each pixel of the acquired two-dimensional image and the data of the two-dimensional map. This modification involves correction of the initial pixel-data association of the two-dimensional map set when the depth image was acquired.
[0047] In an association step 170, the point clouds of the successively acquired and processed depth images are pairwise associated by a rigid sequential iterative closest point or ICP algorithm. This association occurs in an iterative manner; thus, for example, after associating the point cloud of the first image with the point cloud of the second image to obtain a first merged point cloud and associating the point cloud of the third image with the point cloud of the fourth image to obtain a second merged point cloud, the first merged point cloud is associated with the second merged point cloud. This rigid sequential iterative closest point algorithm considers the points in the two closest point clouds to point to the same head point and associates them. In particular, the rigid ICP algorithm is iterative and receives two point clouds as input: one point cloud as a reference, which remains fixed, and one point cloud as a source, which is rigidly transformed based on a predefined criterion.
[0048] This algorithm is called rigid because the transformations involved are of the rigid type, i.e., they preserve the Euclidean distance between pairs of points.
[0049] Thus, after a series of iterations, a synthetic point cloud is obtained.
[0050] For example, the alignment step 180 is performed by considering a so-called Basel head as a statistical head model.
[0051] Rigid Iterative Closest Point Alignment or Nearest Neighbor ICP algorithm is a rigid ICP algorithm that provides the following steps:
[0052] a) For each point in the source point cloud, determine the nearest neighbor point in the reference point cloud;
[0053] b) estimating a rotation-translation matrix that minimizes a point-to-point distance measure (such as a mean squared error value) between corresponding points found in the previous step;
[0054] c) transform the vertices of the source point cloud using the rotation and translation matrix estimated in the previous step;
[0055] d) Repeating steps a)-c) for a predetermined number of iterations, and in any case making sure that the error is smaller than a predefined threshold.
[0056] The deformation step 170 is performed with the aid of a minimization algorithm of a predefined cost function.
[0057] The cost function L minimized in the deformation step 190 is a linear combination of multiple components, including at least components named loss_nn and loss_coeff, weighted by corresponding always positive coefficients w_nn and w_coeff, which can be modified to change the importance of individual components in the final calculation of the cost function itself.
[0058] Specifically, the loss_nn component is proportional to the distance between a point of the statistical head model to be deformed and the nearest neighbor point belonging to the synthetic point cloud determined in the alignment step 160 .
[0059] The nearest neighbor point is specifically determined as the point in the synthetic point cloud having the shortest distance to the corresponding point in the statistical head model.
[0060] The loss_coeff component is equal to the norm of the eigenvalues of the statistical head model.
[0061] In the first embodiment, the cost function L is defined by the following expression:
[0062] L=w_nn*loss_nn+w_coef*loss_coef
[0063] The deformation process of the statistical head model can ultimately be understood as an optimization process aiming at identifying the minimum of the above-mentioned cost function which defines as outputs the rotation and translation matrix to be applied to the statistical head model and the eigenvalues of the statistical head model.
[0064] Preferably, the modeling method 100 comprises the following steps before the associating step 170:
[0065] - identifying 310 extreme two-dimensional images among the two-dimensional images of the depth image, ie two-dimensional images whose yaw angles are at the ends of a predefined range;
[0066] - For each extreme two-dimensional image, identifying 320 second marker points on the two-dimensional image corresponding to specific features of the head manually selected and positioned by said user.
[0067] In this step, a user observing the extreme two-dimensional image manually selects second marker points corresponding to, for example, ears, nose, etc.
[0068] In particular, the extreme 2D images have the highest yaw angle and the lowest yaw angle of all 2D images. For example, the acceptable range may be [-40°, 40°].
[0069] Therefore, it can be considered that the second marking point is more reliable than the first marking point identified by the first algorithm.
[0070] To this end, in this case, the alignment algorithm advantageously imposes an exact correspondence between the second marker points and the relatively principal points of the statistical head model (i.e., three-dimensional points of the statistical head model that are predefined and that can correspond to different specific points of the head, such as the nose tip or the tragus). This makes the alignment step 160 more accurate, since the alignment algorithm combines the nearest neighbor logic and the exact correspondence between the second marker points and the principal points of the statistical head model.
[0071] In that case, the cost function L is defined by the following expression:
[0072] L=w_nn*loss_nn+w_fixed*loss_fixed+w_coef*loss_coef
[0073] where the term loss_fixed is proportional to the distance between the points in the synthetic point cloud and the points in the statistical head model that are expected to be exact matches.
[0074] Preferably, the alignment algorithm is performed only for points in the statistical head model that belong to a preset region of interest of the head, which includes at least the eyes and ears. This makes the alignment step 180 faster and more accurate to precisely align the region of the head involved in using the glasses.
[0075] Pre-set regions of interest on the head may be identified by bands encircling the user's head at eye and ear level.
[0076] Preferably, before the step of iteratively associating 170 the acquired depth images in pairs and after a possible step of identifying the second marker point 320, the modeling method 100 provides a filtering step 400, wherein for each of the acquired depth images, the yaw angle of the corresponding transformed posture is compared with an acceptable range 410; if the yaw angle is included in the acceptable range, then the depth image is deemed acceptable 420, and if the yaw angle is not included in the acceptable range, then the depth image is deemed unacceptable 430.
[0077] Subsequently, the acceptable angle range may be uniformly subdivided into a number of sub-ranges equal to the number of depth images used for 3D reconstruction. From the depth images that have passed the filtering step (i.e., the images deemed acceptable in step 420), the image with the yaw angle closest to the above-mentioned sub-range is selected, and the other images are discarded and deemed to have passed the verification.
[0078] The step of iteratively associating 170 the acquired depth images in pairs is performed only for images deemed acceptable or, if provided, for images deemed acceptable and verified.
[0079] This is done to ensure that the image shows all the most important details of the user's head.
[0080] On the one hand, this leads to a reduction in the time and computational cost for modeling the user's head, and on the other hand, ensures that we represent the user's head as completely as possible.
[0081] Preferably, before the step of iteratively associating 170 the acquired depth images in pairs, the modeling method further comprises a preparation step 200, wherein for each of the acquired depth images or for each of the acceptable acquired depth images, the following steps are provided:
[0082] -Locate 210 the user's head;
[0083] - removing 220 the background to remove unwanted noise components;
[0084] - Applying 230 DBSCAN type clustering algorithms etc. to remove further outliers,
[0085] Thus a prepared depth image is obtained.
[0086] The DBSCAN or similar algorithms are selected from algorithms well known in the art.
[0087] The prepared depth image consists only of the points that form the head, and this helps to further reduce the cost and computational time of making the second model of the user's head.
[0088] From the description given, the characteristics of the modeling method, the objects of the invention and the relative advantages are clear.
[0089] In fact, employing a cost function minimization algorithm in the deformation step allows for a more accurate and flexible control of the deformation process, since the weight coefficients of the cost function can be varied as desired.
[0090] Furthermore, if manual selection of a second marker point is provided, the modeling approach is even more accurate, since a more precise alignment step is possible.
[0091] The possible presence of a predefined head region of interest, the filtering steps and the preparation steps lead to a significant reduction in time and computational costs compared to known modeling methods.
[0092] Finally, it is clear that the modeling system thus conceived is susceptible of numerous modifications and variations, all of which are within the scope of the invention; moreover, all the details may be replaced by technically equivalent elements. In practice, the materials used and their dimensions may be of any type according to technical requirements.
Claims
1. A modeling method (100) for making a virtual model of a user's head, comprising the following steps: - acquiring (110) a plurality of depth images of the user's head in different postures, wherein each of the depth images comprises a two-dimensional or 2D image and a two-dimensional map of data representing the distance of each pixel from a viewpoint coinciding with the position of the depth camera, the two-dimensional map being aligned with the two-dimensional or 2D image, Associating each pixel of the acquired two-dimensional image with data of the two-dimensional map; - for each of said acquired depth images, performing by correcting the alignment between said two-dimensional data map and said two-dimensional or 2D image (120) processing the acquired depth image; - for each of said two-dimensional images of said acquired and processed depth images, locating (130) a first marker point on the two-dimensional image corresponding to a specific feature of said head; - for each of the two-dimensional images of the acquired and processed depth images, by estimating the yaw, pitch and roll angles of the head and the position of the head along the three axes X, Y, Z of a first Cartesian axis reference system XYZ to estimate (140) the posture of the head; - for each of said two-dimensional images of said depth image, performing a transformation 150 of the pose of the head by modifying the previously estimated yaw, pitch and roll angles of the head and the positioning of the head so that they are referenced to a second Cartesian axis reference system having a predefined relationship with the first Cartesian axis reference system XYZ X'Y'Z'; - for each of the acquired depth images, projecting (160) The first marker point is located in the corresponding point cloud of the depth image so as to obtain the corresponding first three-dimensional marker in the second Cartesian axis reference system X'Y'Z' point; - performing iterative pairwise association (170) of the point clouds of the acquired and processed depth images by means of a sequential iterative closest point algorithm until a synthetic point cloud including a first three-dimensional marker point is obtained; - Align the The synthesized point cloud is aligned with the statistical head model (180); - deforming (190) the statistical head model by means of an algorithm for minimizing a predefined cost function so that the statistical head model is as close as possible to a synthetic point cloud of a model of the user's head.
2. The modeling method (100) according to claim 1, further comprising the following steps before the associating step (170): - identifying (310) extreme two-dimensional images in the two-dimensional image of the depth image picture; - for each of the extreme two-dimensional images, identifying (320) second marker points on the two-dimensional image corresponding to specific features of the head that have been manually selected and positioned by the user.
3. The modeling method (100) according to claim 2, wherein the alignment algorithm imposes an exact match between second marker points and relative main points of the statistical head model.
4. A modelling method (100) according to one or more of the preceding claims, wherein the alignment algorithm is performed only on points of the statistical head model belonging to a preset region of interest comprising a band of the head at eye level.
5. The modeling method (100) according to one or more of the preceding claims, comprises a filtering step (400) before the step of performing iterative pairwise association of the acquired depth images, wherein for each of the acquired depth images, the yaw angle of the corresponding transformed posture is compared with an acceptable range (410), if the yaw angle is included in the acceptable range, then the depth image is considered acceptable (320), if the yaw angle is not included in the acceptable range, then the depth image is considered unacceptable (330), and the step of performing iterative pairwise association of the acquired depth images is performed only on images that are considered acceptable.
6. The modeling method (100) according to one or more of the preceding claims, comprising a preparatory step (200) prior to the step of iteratively associating the acquired depth images in pairs, wherein for each of the acquired depth images or for each of the acceptable acquired depth images: - positioning (210) the user's head; - removing (220) the background; - Apply (230) a DBSCAN type clustering algorithm or the like to obtain a prepared depth image.
7. Modeling method (100) according to one or more of the preceding claims, wherein the cost function is a linear combination of a plurality of components, including at least components called loss_nn and loss_coeff, weighted by respective coefficients w_nn and w_coef, which are always positive, wherein the loss_nn component is proportional to the distance between a point of the statistical head model to be deformed and the nearest neighbor point belonging to the synthetic point cloud determined in the alignment step (160), and the loss_coeff component is equal to the norm of the eigenvalues of the statistical head model, the cost function being defined by the following expression: L=w_nn*loss_nn+w_coef*loss_coef.
8. The modeling method (100) according to claim 7, wherein the cost function is defined by the following expression: L=w_nn*loss_nn+w_fixed*loss_fixed+w_coef*loss_coef where the term loss_fixed is proportional to the distance between the points in the synthetic point cloud and the points in the statistical head model that are expected to be exact matches.
9. A computer program loadable in a memory of an electronic computer and comprising instructions which, when the electronic computer runs the program, cause the electronic computer to implement the modeling method (100) according to one or more of the preceding claims. 10 . An electronic terminal comprising a depth camera and an electronic computer, wherein the electronic computer is provided with a memory for data storage, wherein the program according to claim 9 is loaded into the memory.