Method for head image registration and head model generation and corresponding device

Abstract

A method and apparatus for head image registration using multiple combined 2D and depth images (e.g., RGBD images) of a head recorded from different positions is provided. Landmark points are determined for each of these combined images. A coarse registration is performed based on these landmark points, followed by a fine registration based on the complete point cloud. Based on this registration, a head model can be generated.

Description

[0001] The present application relates to methods for head image registration and head model generation, and to corresponding devices for performing these methods.

[0002] A 3D model of a person's head (herein referred to as a head model) can be used to represent the person's head as a so-called avatar in various applications. One example of such an application is the virtual fitting and trying on of eyeglass frames, as described, for example, in WO 2019 / 008087 A1.

[0003] For such applications, it is desirable that the 3D model of the head used reflects and matches the person's actual head well in order to give a correct visual impression of how the spectacle frames will look like when worn by the person.

[0004] One method for generating a head model, such as that implemented in the applicant's Visufit system, uses multiple fixed cameras arranged around a person's head, which can capture images of the person's head substantially simultaneously. Because the arrangement is fixed, the relative positions and orientations of the cameras are known, and a 3D model of the head can be generated based on techniques similar to triangulation. In the relevant technical field, the combination of position and orientation in space is also referred to as a "pose," as defined, for example, in ISO 8373:2012 4.5. However, this method requires a fixed camera arrangement, which can be set up, for example, at an optician's office, but is rarely feasible for private use.

[0005] Therefore, methods have been proposed in which a person can use a mobile device such as a smartphone or tablet PC to capture images of his or her head from various angles, and based on these images, a 3D model of the entire body, and optionally also of the person's head, is generated. A disadvantage of these methods is that the pose of the mobile device when the image was captured is not known a priori and must be determined.

[0006] For example, Tanskanen, Petri et al., "Life metric 3D reconstruction on mobile phones", Proceedings of the IEEE International Conference on Computer Vision, 2013 discloses a life metric 3D reconstruction of a statue, such as one in a museum, in which an accelerometer and / or gyroscope provided in a mobile phone uses inertial tracking to perform pose estimation of the mobile phone.

[0007] Kolev, Kalin et al., "Turning mobile phones into 3D scanners", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014. A camera motion tracking system is used to provide the camera pose.

[0008] Muratov, Oleg et al., "3D Capture: 3D Reconstruction for a Smartphone" also uses an inertial measurement unit to track camera pose.

[0009] These methods rely on the accuracy of acceleration or similar measurements to provide camera poses from which a 3D model can be computed.

[0010] US 8,126,261 B2 does not specifically address smartphones and similar mobile devices, but uses prior knowledge based on generic faces for processing.

[0011] WO 2019 / 164502 A1 discloses using a smartphone to generate a 3D model of a head based on a 2D image represented using a 3D mesh. This method is computationally expensive, and the document proposes the possibility of performing the calculation on a remote server.

[0012] Recently, smartphones have been equipped with depth sensors in addition to traditional cameras. These sensors can measure the distance from the smartphone to an object (e.g., a face) with a certain spatial resolution. The combination of a traditional 2D camera and this depth sensor is also known as an RGBD camera (RGB stands for red, green, and blue, the primary colors used by 2D image sensors, and D stands for depth). In other approaches, the depth camera can be provided separately from the smartphone.

[0013] US10,157,477B2 discloses the use of such an RGBD camera to generate a 3D model of the head. In the introduction, the document discusses the use of facial landmark tracking and argues that this method is inaccurate and disadvantageous. Other methods are also discussed. The document then proposes a method that relies on calculating at least two head meshes from different perspectives and determining the translation and rotation between the head meshes, or in other words, the registration between the head meshes. On this basis, the posture of the camera relative to the person's head when capturing the image can be determined. Registration generally refers to a transformation process between two entities (meshes, RGBD images, point clouds, etc.), which transforms one of the entities into another entity in the entity. Registration is necessary in order to be able to compare or integrate data obtained from different measurements.

[0014] In this regard, it should be noted that, given the way a camera transforms a 3D object into an image, which in turn is given by the camera's known physical properties (such as focal length and resolution), providing a registration between RGBD images or corresponding head meshes acquired at different camera poses is essentially equivalent to determining the camera pose itself. This is easy to understand, for example, when the camera pose is rotated relative to the face, the image will also rotate. When the camera position moves toward or away from the head, the head will become smaller or larger in the corresponding RGBD image, and the corresponding magnification will be greater or less than 1, etc.

[0015] However, matching meshes comprising many vertices (as in US 10,157,477 B2) can be computationally expensive.

[0016] Vyas Katha et al., "An Efficient 3D Synthetic Model Generation Pipeline for Human Pose Data Augmentation," 2021 IEEE / CVF conference on computer vision and pattern recognition workshops (CVPRW), present a two-step registration process between facial images, with a coarse registration using surface normals and a fine registration using an iterative closest point algorithm.

[0017] Xiaoguang Lu et al., "Matching 2.5D Face Scans to 3D Models", IEEE transactions on pattern analysis and machine intelligence, IEEE Computer Society, USA, Vol. 28, No. 1, 2006, CN 111 414 798 and Bagchi Parama et al., "Reg3DFacePtCd: Registration of 3D Point Clouds Using a Common Set of Landmarks for Alignment of Human Face Images", KI. Künstliche Intelligenz, Arendtap Desktop Publishing, Bremen, Vol. 33, No. 4, 2019, each discloses a method in which a point cloud of a face, in particular an RGBD image, is registered to a 3D model of a head using a two-step process in which landmarks are used for coarse registration and ICP is used for fine registration.

[0018] Therefore, starting from US 10,157,477 B2, the object is to provide an improved method for image registration of images acquired by an RGBD camera and, on this basis, a corresponding method and device for generating a 3D model of a human head.

[0019] According to several aspects of the present invention, a method for head image registration is provided, the method comprising:

[0020] Provides multiple combined 2D images and depth images of the head captured from different positions,

[0021] Determine landmark points for each of these combined 2D and depth images, and

[0022] A pairwise coarse registration is performed between the combined 2D image and depth image pairs based on these landmark points.

[0023] The method is characterized in that, after the pairwise coarse registration, the coarsely registered combined 2D images and depth images are finely registered based on the complete point cloud represented by the combined 2D images and depth images.

[0024] Coarse registration using only landmarks can advantageously be performed with relatively low computational complexity. Furthermore, since a coarse registration has already been performed, fine registration using the full point cloud can also be performed with lower computational complexity and / or higher accuracy than, for example, the case of using only a full head mesh for registration as in US Pat. No. 10,157,477 B2 discussed above.

[0025] The terms used above are explained below.

[0026] A combined 2D image and depth image refers to a 2D image and depth image acquired from corresponding locations at substantially the same time. The 2D image can be a color image, such as an RGB (red, green, blue) image, or a grayscale image. The depth image provides a distance map from the camera to the object (in this case, the head). An example of a combined 2D image and depth image is an RGBD image. To capture the 2D portion of the combined image, any conventional image sensor and corresponding camera optics can be used. To capture the depth image, any conventional depth sensor, such as a time-of-flight sensor, can also be used. The combined image can include two separate files or other data entities, one of which contains a grayscale or color value for each 2D coordinate (e.g., pixel), and the other contains a depth value for each 2D coordinate. The combined image can also include a single data entity, containing both grayscale / color information and depth information for each 2D coordinate. In other words, as long as both grayscale / color information and depth information are available for the scene captured in the image, the manner in which the information is stored in the data entity, such as a file, is unimportant. A camera that is suitable for capturing a combined 2D image and a depth image (in this case a color image) is also called an RGBD (Red, Green, Blue, Depth) camera. As mentioned in the introduction, some modern smartphones or other mobile devices are equipped with such an RGBD camera. In other cases, as also explained in the introduction, an RGBD camera or a depth sensor (which is then used together with the smartphone's built-in camera) can also be attached to the smartphone. It should be noted that the depth image does not need to have the same resolution as the 2D image. In this case, a scaling operation (reduction or enlargement) can be performed to adapt the resolutions of the 2D image and the depth image to each other. The result is basically a point cloud, where each point has a 3D coordinate based on the 2D coordinate in the image and a depth coordinate from the depth sensor, as well as a pixel value (color value or grayscale value).

[0027] The phrase "captured from different positions" indicates that the relative position of the head and the device used to capture the combined 2D image and depth image may change between different images. For example, when using a smartphone, the smartphone may be held in a fixed position and the head may be moved (e.g., rotated), or the smartphone may be moved around the head. The smartphone or other mobile device used (e.g., a tablet PC) may provide corresponding instructions to the person, i.e., instructions to move the head relative to the smartphone. As with any method that relies on human-computer interaction, errors may result if the person does not follow the instructions.

[0028] Image registration generally involves the process of finding a transformation that transforms one of the combined 2D image and depth image into the other of the combined 2D image and depth image. This transformation can include a rotation component, a translation component, and a magnification component (magnification greater or less than 1), and can be written in matrix form. As initially mentioned, because a specific combined image of the head is produced in a specific pose due to the characteristics of the camera (such as the focal length and resolution of the optics), registration by determining the transformation mentioned above is essentially equivalent to determining the (relative) camera pose (position and orientation) that captured the combined image.

[0029] Landmark points are predefined points on the head. Such landmark points may include, for example, the tip of the nose, points on the bridge of the nose, corners of the mouth or eyes, points forming the shape of eyebrows, etc. Such landmark points in the combined 2D image and depth image can be determined by various conventional methods. For example, trained machine learning logic such as a neural network can be used to determine the landmark points. In this case, for training, multiple combined 2D images and depth images from different locations and for multiple different heads are used as training data, wherein the landmark points can be manually annotated. After training, the trained machine learning logic determines the landmark points. Details can be found, for example, in Wu, Y., Hassner, T., Kim, K., Medioni, G., and Natarajan, P. (2017), Facial landmark detection with tweaked convolutional neural networks, IEEE transactions on pattern analysis and machine intelligence, 40(12), 3067–3074, Perakis, P., Passalis, G., Theoharis, T. & Kakadiaris, I.A. (2012), 3D facial landmark detection under large yaw and expression variations, IEEE transactions on pattern analysis and machine intelligence, 35(7), 1552–1564, or Wu, Y. & Ji, Q. (2019), Facial landmark detection: A literature survey, International Journal of Computer Vision [International Journal of Computer Vision], 127(2), 115-142 (2018). Landmark detection can be preceded by segmentation, where, for example, the background is separated from the face and the face is segmented to provide face segments. Landmarks can then be extracted from the face segments.

[0030] Pairwise coarse registration provides a coarse alignment between pairs of 3D landmarks. In an embodiment, this pairwise coarse registration estimates a transformation matrix between the landmarks of the two combined images that aligns the landmarks in a least squares sense, i.e., such that the error Minimize, where is the i-th landmark of the j-th image, and is the transformation matrix from the second image in the corresponding pair (j=2) to the first image in the corresponding pair (j=1). This coarse registration can be performed by a method called point-to-point ICP ("Iterative Closest Point"), which is described, for example, in Besl, Pazl J. and McKay, Neil D, "Method for registration of 3D shapes", Sensor fusion IV: control paradigms and data structures, vol. 1611, International Society for Optics and Photonics, 1992. Preferably, to eliminate potential outliers that may be generated during the landmark determination step, a random sampling consensus procedure can be used, as described in the following literature: Fischler, Martin A. and Bolles, Robert C., "Random sampling consensus: a paradigm for model fitting with applications to image analysis and automated cartography", Communications of the ACM, 24.6 (1981): 381-2395. The above and other formulas presented herein use so-called homogeneous coordinates, as is often the case in computer vision applications. This means that the transformation T is represented as a 4×4 matrix [R t; 00 0 1], where R is a 3×3 rotation matrix, t is a translation vector, and the last row is 0 0 0 1. The 3D point (x, y, z) is augmented with a homogeneous component w, i.e., (x, y, z, w), where typically w = 1. This allows translation and rotation to be included in a single matrix multiplication, i.e., instead of x2=Rx1+t (where vector x2 and vector x1 are in Cartesian coordinates), the matrix multiplication can be written as x2w=Tx1w (where x1w and x1w are corresponding vectors in homogeneous coordinates). However, this is merely a matter of notation, and the same calculation can be performed in Cartesian or other coordinates.

[0031] Fine registration refines the transformation mentioned above, i.e., makes it more precise. For this fine registration, the complete point cloud represented by the combined 2D and depth images (i.e., the complete RGBD image or a point cloud derived from these images) can be used. In particular, color information can also be used. Because a coarse registration has already been performed, fine registration can be performed more efficiently than when only the point cloud is used for registration or when a corresponding mesh is used.

[0032] Different methods can be used for fine registration. In a first specific aspect, and preferably also in other aspects, the method selected can depend on the geometric difference, i.e., the error remaining after the coarse registration, such as the error e mentioned above, or other error quantities (such as the angular inverse difference between the marker points of the transformed combined 2D image and the depth image pair determined based on the coarse registration). For example, angular difference or inverse difference can also be used. If such a deviation occurs (for example, the error e is less than a certain value, the angular difference between the marker points is less than a certain threshold angle (such as 5°), or their position difference is less than a certain position (such as 5 cm, for example as the average value of the marker points)), RGBD odometry can be used for fine registration, in which not only the depth coordinates but also the color of each point in the point cloud are taken into account. For example, Park, Jaesik, Zhou, Quian-Yi and Koltun, Vladlen, "Colored point cloud registration revisited", Proceedings of the IEEE International Conference on Computer Vision 2017, describe RGBD odometry. For cases with large discrepancies, the images can be registered using point-to-plane ICP on the point cloud, as described, for example, in Rusinkiewicz, Szymon and Levoy, Marc, "Efficient variants of the ICP algorithm", Proceedings of the third international conference on 3D digital imaging and modeling, IEEE 2001.

[0033] For both alternatives, it is preferred to perform two registrations, one to estimate the transformation from the first combined image in the corresponding pair to the second combined image in the corresponding pair, and one to estimate the transformation from the second combined image to the first combined image, with slightly different starting values ​​for the algorithms. This can help to improve the overall accuracy. In other words, determine and Determine the error between two registrations If the registration is stable, the error should be close to the identity matrix I4 (i.e., a diagonal matrix with only 1 values). If the error (i.e., the deviation from the identity matrix) is less than a certain threshold, the corresponding transformation can be added as an adjective between the corresponding combined images, which is a so-called pose graph. In some embodiments, the covariance of the transformation can also be determined.

[0034] In a second specific aspect, and preferably also in other aspects, for both coarse and fine registration, instead of using all possible combined 2D image and depth image pairs, pairs to be registered can be determined based on a classification of the combined 2D image and depth image with respect to their orientation relative to the head from which the images were captured (e.g., based on a so-called putative matchmap indicating pairs that can be registered). This can be accomplished by using approximate pose data or other approximate information to determine combined images that are captured in poses that are sufficiently similar so that registration can be reasonably performed. For example, if one combined image is acquired from the left side of the head and the other combined image is acquired from the right side of the head, it will be difficult to obtain any common landmarks from the two combined images (e.g., the left eye is only visible on the left side, the right eye is only visible on the right side, and the same is true for the left and right corners of the mouth). Therefore, classification can be performed starting from the frontal image to divide the combined images into categories such as left, right, top, and bottom. In each category, pairwise coarse and fine registration as described above is performed.

[0035] In some embodiments, the classification can be based on metric data from the image recording device itself (e.g., an ARKit tool if the device is based on iOS, or an ARCore tool if the device used to capture the combined image is based on Android). In other cases, an inferred matching graph can be obtained from the above markings via 2D / 3D correspondences and a perspective n-point solver, as described, for example, in Urban, Steffen, Leitloff, Jens, and Hinz, Stefan, "MIpnp - a real-time maximum likelihood solution to the perspective n-point problem", arXiv preprint, arXiv:1607.08112 (2016).

[0036] In this way, the method can avoid attempting to register combined images where the lack of common landmarks to improve robustness makes such registration difficult or impossible.

[0037] Based on the registration (i.e. the above transformation), a pose can then be estimated for each combined image in the global reference frame. The pose can be either the pose of the head represented by the combined image or the pose of the camera. As explained above, the pose of the camera when capturing the combined image is directly related to the registration of the images and therefore to the pose of the head in the combined image, so if the camera pose is known, the head pose can be determined and vice versa. This pose graph is then preferably optimized. Choi, Sungjoon, Zhou, Qian-Yi and Vladlen Koltun, "Robust reconstruction of indoor scenes", proceedings of the IEEE conference on computer vision and pattern recognition, 2015, describe a possible approach for pose graph optimization, which involves generating a pose M for each combined image based on the registration. j .

[0038] Based on these postures M j , providing a posture graph p = {M, e}, which consists of a node M (i.e., a posture M j ) and edge E (i.e., transformation and possible covariances (if these transformations and covariances are determined)) composition.

[0039] Based on this pose graph, in the third specific aspect, and preferably also in other aspects, so-called edge pruning is performed for further optimization. This can be used to remove to determine whether the pose is estimated based on valid edges (i.e., valid transformations obtained in the registration). To this end, one of the poses is used as a reference. The shortest path along the edges in the pose graph is then connected from the unoptimized pose graph to another node. This obtains a further pose estimate of the node to be tested. This further estimate is then compared with the pose from the optimized pose graph. In the case where the deviation is large in this comparison (i.e., the deviation is above a threshold), the corresponding edge can be identified as an erroneous edge. These edges can then be removed. Afterwards, the pose graph optimization mentioned above is repeated without removing the edges until no erroneous edges remain in the graph. In other words, in a third aspect, and preferably also in other aspects, pruning comprises optimizing the pose graph and comparing the pose of the optimized pose graph with the odometry-based pose generated based on one or more edges of the pose graph.

[0040] According to a further aspect, there is provided a method for head model generation, the method comprising:

[0041] performing the method for head image registration as described above, and

[0042] A head model is generated based on the registration.

[0043] In this context, a head model refers to a three-dimensional representation of the head, which can be in the form of a voxel cloud or a 3D mesh consisting of points linked by edges.

[0044] The generation of such a head model may include fusing the point clouds represented by the combined images into a final point cloud by transforming them in the corresponding pose. Can be achieved through To determine the fused point X fused , where U is an operator that combines all individually transformed point clouds. This point cloud can then be thinned by leaving only one point per voxel, where each voxel has a predefined side length of ν, for example, ν = 1 mm. That is, if each voxel has more than one point from different transformed point clouds, all but one are removed. This reduces the number of points and thus makes further processing of the fused point cloud easier.

[0045] Preferably, mesh extraction is then performed based on the fused point cloud to obtain the head model, i.e., a mesh is generated based on the point cloud. This can be performed using Poisson reconstruction, as described, for example, in Kazhdan, Michael, Bolitho, Matthew, and Hoppe, Hugues, "Poisson surface reconstruction," Proceedings of the fourth Eurographics symposium on Geometry processing, Vol. 7, 2006. Due to noise, the mesh may have some uneven surface elements. In an embodiment, to remove this effect, the mesh is smoothed using, for example, Laplacian smoothing or Taubin smoothing over several iterations, as described, for example, in Belyaev, Alexander, and Yutaka, Ohtake, "A comparison of mesh smoothing methods," Israel-Korea Binational conference on geometric modeling and computer graphics, Vol. 2, 2003.

[0046] Optionally, in some embodiments, the aforementioned landmark points are applied to the grid, as having 3D landmarks on the head (such as eyes, ear points, or nose points) is helpful for some applications, for example, for virtual fitting of eyeglass frames. In some embodiments, the landmarks can be obtained from one of the combined images determined above, with corresponding coordinated translations performed according to the applied registration. In other embodiments, for higher accuracy, the 3D landmarks can be obtained by triangulation based on the estimated camera pose.

[0047] In addition, in some embodiments, a normalized coordinate system can be provided for the model, for example, where the z-axis points vertically upwards. To this end, the 3D landmarks mentioned above can be used, because generally for a head, various landmarks (such as nose point, eyes, mouth, etc.) have a certain directional relationship.

[0048] Finally, the head model is textured based on the image. One possible approach to this is described in Waechter, Michael, Moehrle, Nils, and Goesele, Michael, "Let there be color! Large-scale texturing of 3D reconstructions," European conference on computer vision, Springer, Kahn, 2014.

[0049] The head model can then be used as a stand-in for virtually trying on eyeglass frames, or for any other purpose for which head models are traditionally used.

[0050] A corresponding device (e.g., a smartphone or tablet PC) is configured accordingly, and a computer program is also provided. For example, a computer program for a mobile device comprising a processor, a 2D camera, and a depth sensor is provided, wherein the computer program, when executed on the processor, causes any of the above methods to be performed. The computer program may be provided on a tangible storage medium such as a memory, a hard disk, a DVD, or the like, or may be transmitted as a data carrier signal.

[0051] Furthermore, according to another aspect, a device is provided, comprising a processor configured to:

[0052] receiving a plurality of combined 2D images and depth images of the head captured from different positions,

[0053] Determine landmark points for each of these combined 2D and depth images, and

[0054] Perform pairwise coarse registration between combined 2D and depth image pairs based on landmark points,

[0055] Characterized in that the processor is further configured to:

[0056] After the coarse registration, the coarsely registered combined 2D image and depth image are finely registered based on the complete point cloud represented by the combined 2D image and depth image.

[0057] The device may further include a camera and a depth sensor configured to capture the combined 2D image and depth image.

[0058] The above explanation of the method also applies to the device, and the device can be configured to perform any of the above methods by configuring (e.g., programming) the processor accordingly, and is configured to perform a method according to one of the first specific aspects to the third specific aspects discussed above.

[0059] The embodiments will be discussed with reference to the accompanying drawings, in which:

[0060] Figure 1 is a schematic block diagram of a device according to an embodiment;

[0061] Figure 2 It is used to demonstrate the use Figure 1 a device for capturing a plurality of combined images;

[0062] Figure 3 is a flowchart illustrating a method for head model generation according to an embodiment;

[0063] Figure 4 is a diagram showing the embodiment of the Figure 3 A flowchart of the image registration used in step 39;

[0064] Figure 5 An example of a gesture diagram for illustrating an embodiment is shown;

[0065] Figure 6A An example of a landmark point is shown, and Figure 6B shows an example of when the landmark points are applied to the head; and

[0066] Figure 7 An example of face segmentation is shown.

[0067] Hereinafter, embodiments related to generating a 3D model of a head using a device such as a smartphone or a tablet PC will be described. Figure 1 A block diagram of a device 10 that can be used in some embodiments is shown. The device 10 can be a smartphone or tablet PC, but can also be a dedicated device. In the case of a smartphone or tablet PC, the device is programmed accordingly to perform the methods discussed herein.

[0068] Device 10 includes a camera 12 and a depth sensor 11. Depth sensor 11 and camera 12 form an RGBD camera as discussed above. In addition, device 10 includes a touch screen 13, a processor 15, a memory 14, and (multiple) input / output interfaces 16. Touch screen 13 can be used to control device 10 and can also be used to output instructions to a person, for example, outputting instructions to capture an image, as described below with respect to Figure 2The processor 15 executes the instructions stored in the memory 14 to implement the method as discussed below. The input / output interface 16 can provide communication to a network (such as the Internet) (for example, to transmit the generated head model to other computer devices), and may also include other devices for communicating with the user, such as a speaker or microphone. In general, Figure 1 Only some of the components that may be used with the methods as discussed herein are shown, and other components may be provided by the phone as in a conventional smartphone or tablet PC.

[0069] In order to generate a head model of a person's head, for example to be used as an avatar, a plurality of combined depth images and 2D images are first acquired from the head at a plurality of different positions of the device 10 relative to the head. Figure 2 It was displayed in . Figure 2 A head 20 is shown for which a model is to be generated.

[0070] For image generation, device 10 outputs an instruction to the person to move his or her head relative to device 10. For example, at the beginning of image recording, the person can look directly at device 10, thereby capturing a frontal image, and then, as indicated by arrow 21, he / she can turn his / her head first to the right and then to the left. Up and down movement is also possible. During this movement, device 10 records multiple combined 2D images (using camera 12) and depth images (using depth sensor 11). In other embodiments, device 10 can move around head 20 instead of rotating head 20. Based on multiple images of the combined depth image and 2D image thus generated, device 10 then generates a 3D model of head 20 using processor 15. Figure 3 The method employed by apparatus 10 is shown, and Figure 4 The method of image registration is shown. Image registration is Figure 3 One of the core aspects of the method.

[0071] Figure 3The method receives as input data 30 a 2D image recorded by the camera 12 and a corresponding depth image recorded by the depth sensor 11, i.e. a combined image (e.g. as an RGB-D image). Furthermore, the method uses as input data internal camera data, in particular camera calibration data of the smartphone used for the method. The camera calibration data comprises information on how the camera 12 images an object. As already mentioned initially, for a specific camera (a specific image sensor, a specific lens with a specific focal length), an object located at a certain position relative to the camera is imaged onto the image sensor in a manner determined by the camera. This information is determined by the physical properties of the camera and possible built-in software image processing (e.g. rectification) and can be provided, for example, by the manufacturer of the camera. The camera calibration information and the corresponding pre-processed (e.g. reduced) camera calibration information 34 can be written into a matrix K which maps a 3D point x in real space to a pixel value. R, y R ,z R (Using the coordinate system relative to the camera) to convert to 2D image point x, y, that is, (x, y, 1) = K(x R ,y R ,z R ) T .

[0072] The input data is then pre-processed in step 31 to provide pre-processed camera and image data 32. For example, in some smartphones and similar devices, the depth sensor 11 may have a lower resolution than the camera 12. Therefore, data pre-processing 31 provides a downscaled image with the same resolution as the depth sensor 11 and corresponding downscaled camera calibration information. As a simple numerical example, if the original 2D image 31 has a resolution of 4000×3000 pixels and the depth image 37 has a resolution of 400×300 pixels, the original image is also downscaled to 400×300 pixels, so that the resolution of the downscaled image is also 400×300 pixels. Any resampling technique conventionally used for downscaling can be used. The smartphone's camera calibration information is downscaled accordingly, so that it now indicates how the object is recorded as the downscaled image. Other pre-processing may, for example, include denoising or deblurring the image.

[0073] At 33, the pre-processed image data 32 is processed to provide segmentation and landmark extraction. For example, the background is separated from the face and the face is segmented to provide face segments 34. Any conventional face segmentation algorithm may be used. Figure 7 An example segmentation is shown in . Here, the background is separated from the person and segmented into hair 80, face 81, eyebrows 82, 83, eyes 84, 85, nose 86, mouth (with upper and lower lips) 87. In addition, the neck region 88 and shoulder region 89 are provided as separate regions.

[0074] Additionally, facial landmarks are identified in the 2D image based on the downscaled image. For example, Figure 6A A model using 68 points as landmarks is presented, which specifically characterizes the eyebrows, eyes, nose, mouth, and lower face edge (including the chin area). Figure 6B Shown Figure 6A The use of such landmarks is described, for example, in Guo, Jianzhu et al., "Towards fast, accurate and stable 3D dense face alignment", arXiv preprint, arXiv:2009.09960 (2020).

[0075] Figure 6A The landmarks in the model also give "pseudo-depth" information corresponding to a predetermined depth assigned to each landmark. In other words, the model regresses the 2D coordinates of each landmark in the 2D image and the pseudo-depth, which can be based on the average spatial relationship of the landmark points to each other (e.g., the tip of the nose protruding from the face, etc.). Therefore, the pseudo-depth is relatively inaccurate and can only be expressed in proportion, that is, not in units of measurement, but in some arbitrary units, because the landmarks are regressed based on the 2D image (i.e., the absolute measurement scale is lost when transforming from a 3D object to a 2D image). However, based on the pseudo-depth, it can be determined which landmarks were farther away from the camera and which landmarks were closer to the camera when the image was captured. In an embodiment, if each landmark is visible in the current image being processed, the probability of the landmark can be obtained based on the pseudo-depth. For example, some landmark points (e.g., the landmark points associated with the right eye in the side view of the left) may not be visible in the side view of the face. Similarly in Figure 6B In , some landmarks on the right side of the face are not visible. The probability p of landmark i = 1…I (I is the number of visible landmarks) i (i.e., visibility probability) can be calculated according to the following formula:

[0076]

[0077] Among them, p min is an adjustable parameter, d i is the pseudo depth of the i-th marker, d min is the minimum pseudo depth of the landmark point, and d max is the maximum pseudo depth of the landmark points. Thus, this landmark extraction at 33 results in the facial landmarks 35 and their pseudo depths in the reduced image.

[0078] At 36 , the method then includes obtaining 3D marker points 310 based on the 2D facial landmarks 35 and the corresponding depth image (part of the pre-processed data 32 ).

[0079] To obtain the 3D landmarks at 36, based on the above probabilities, only facial landmark points with probabilities exceeding a threshold are used. Based on the depth image, each of these 2D landmark points of the facial landmark 35 is assigned a corresponding landmark depth Where i is the number of landmarks and j is the image. Then the 3D point corresponding to image j can be obtained according to the following formula

[0080]

[0081] where K is the (reduced) camera calibration 34 mentioned above, written in matrix form as described above, and are the 2D marker coordinates (z coordinate is set to 1). This results in 3D marker points 310 for each image. Due to the use of depth images, these markers now have a metric scale, i.e., their positions can be expressed not only in arbitrary units but also in metric units.

[0082] Furthermore, at 37, a point cloud is obtained for the face segment 34. In particular, using the face segment 34, the point cloud can be restricted to certain image regions, which can also be referred to as masking. For example, the valid 3D points can be restricted to only the head, excluding the neck region 88 and the shoulder region 89 as well as Figure 7 In this way, when the holding device 10 is substantially stationary (e.g. Figure 2 (as depicted), registration can be performed by rotating the head. If the shoulder region and background are included in such an analysis, these shoulder regions and background will remain fixed between images, while only the head will rotate, which will make registration at least significantly more difficult (if not impossible). In embodiments where the device 10 moves around the head to obtain images, for example, the neck region 88 and shoulder region 89 may be included. For the selected segment, the point cloud X for each image is obtained according to the following formula j :

[0083]

[0084] The formula is similar to the one explained above for 3D marker points. However, here, the index i is iterated over all image points within the valid masking region (e.g., the head region). A point cloud with J number of images is formed

[0085] This results in a point cloud 38 for each image. Based on the 3D markers 310 and the point cloud 38, a two-step registration process 39 is then performed. As initially explained, performing a registration amounts to determining the camera pose associated with the combined image.

[0086] This registration process 39 corresponds to the process already described above in a more general sense. In short, a coarse registration 311 based on 3D landmarks 310 produces an approximate registration and, therefore, an initial metric camera pose 312, which is then refined in a fine registration 313 using the point cloud 38 to produce a final registration corresponding to the final metric camera pose. Reference will now be made to Figure 4 Explain the registration process in more detail.

[0087] Figure 4 The registration process of receives a combined image (RGBD image) at 50 (ie, J images (corresponding to Figure 3 RGBD image at 30)), point cloud ( Figure 3 38) and 3D logo ( Figure 3 310) as input. In addition, Figure 4 The registration process receives as input an inferred matching graph G 51. As explained above, the inferred matching graph uses approximate pose data to identify potential matching pairs, which can be given as metric data from the sensor itself (e.g. from ARKit or ARCore) in the form of head pose estimates, or can be obtained from facial landmarks (3D landmarks) via a 3D correspondence and perspective n-point solver. Therefore, as also explained above, starting with the frontal image, the images are classified into categories such as left, right, top and bottom. In each category (left, right, top and bottom), a coarse registration at 52 is performed (corresponding to Figure 3 311 in) and fine registration at 54 (corresponding to Figure 3 313 in ).

[0088] In the coarse registration at 52, 3D landmarks are aligned between image pairs by point-to-point ICP. Perform a coarse alignment as mentioned above. Thus, estimate the transformation matrix from image 1 to image 2 This transformation matrix aligns the two landmark point clouds in the least squares sense, i.e., the error Minimize. Also as mentioned, apply the random sampling consensus procedure. This will produce the initial relative transformation between the image pairs 53.

[0089] Then, using the point cloud Alternatively, the RGBD images (combined 2D and depth images), which also form a point cloud, are fine-registered 54 to refine these relative transformations 53. As also explained previously, the point cloud can be used based on the position difference between the two images. The images are registered using the RGBD odometry or point-to-plane ICP method. In addition, two registrations can be performed, namely, from the first image in the image pair to the second image and from the second image in the image pair to the first image, which corresponds to the registration of and Estimation of . Based on the error between the registrations, i.e.,

[0090]

[0091] The error should be close to the identity matrix I4, and the relative transformation can be added as an edge between image 1 and image 2 to the pose graph to be generated, i.e., if the error is less than a predefined threshold, the transformation is added. Optionally, the covariance matrix can also be estimated for the registration quality The covariance matrix contains information about the estimated accuracy of the ICP or RGBD odometry alignment. On the main diagonal of the covariance matrix, the variance values ​​diag(var_r1,var_r2,var_r3,var_t1,var_t2,var_t3) of the rotation (3 elements) and translation (3 elements) are provided, where r1 to r3 are the rotation values ​​and t1 to t3 describe the translation vectors. On the off-diagonal elements, covariance values ​​that explain the correlation between different elements of the transformation can be provided. But the most important elements are usually the elements on the main diagonal.

[0092] This covariance matrix can then also be added to the pose graph. The variance value can then be used to weight the different transformations in the pose graph optimization described below. A larger variance means greater uncertainty. Therefore, such a transformation may have less impact on the pose graph optimization. For example, the error can be weighted using the following formula: 1 / variance × error.

[0093] Based on fine registration (optimized transformation by fine registration), the pose M of each combined image (RGBD image) in the global reference system is then used j to generate the initial pose graph 55. As mentioned, this pose graph optimization can be performed by traditional methods (e.g., by the method proposed by Choi, Sungioon et al. cited above) or using a state-of-the-art optimization library (e.g., the "ceres solver" by Agarwal, Sameer and Keir Mierle et al.), which is available at: http: / / ceres-solver.org (2015). For illustration, Figure 5 An example posture diagram is shown in . As an example, Figure 5 The posture graph of FIG shows four nodes 60 to 63 corresponding to postures M1 to M4 of four images. As more images are formed, the posture graph will include more nodes. The nodes are linked by edges 64 to 69, each of which is assigned a function that transforms one posture into another (e.g., T3 of edge 66). 4 The corresponding transformation transforms the posture M3 of node 62 into the posture M4 of node 63 If determined, each edge also has an assigned covariance Based on the initial pose graph 55, at 56, the method includes performing a pose graph optimization, for example, as disclosed in Choi, Sungioon et al. A poor estimate of ), thus forming a further optimization, a pruning step is performed at 57. To this end, the optimized pose M is taken as a reference. j is compared with the pose obtained by traveling along the edges of the unoptimized pose graph to a node along the shortest path in the graph. For example, with M1 as a reference (e.g., as the identity matrix I4), M4 can be estimated by odometry as Compare this to the optimized M4 of Figure 63 which follows edges 64, 65. middle, At this point, it should be close to the identity matrix I4. Since pose M4 is optimized by the pose graph optimization at 56 mentioned above, there may be slight differences. However, if the deviation from the identity matrix I4 is greater than a predefined threshold, this indicates the presence of bad edges and possible nodes, which can be removed at 57. Afterwards, the pose graph optimization at 56 can be repeated. In many cases, a single iteration is sufficient.

[0094] Therefore, the result is the final optimized pose graph P 58.

[0095] Figure 3 This optimized pose graph in corresponds to Figure 3 The final metric camera pose in 314. As mentioned, the pose of the pose graph is directly associated with the pose of device 10, and therefore the pose of the camera when the image was captured.

[0096] Based on the metric pose 314 and the point cloud 38 , at 315 , Figure 3 The method comprises, for example, transforming each single point cloud into a common reference frame according to the following equation Fusion into a final point cloud X fused :

[0097]

[0098] These fused point clouds can be thinned by leaving only one point per voxel, where each voxel has a predefined side length of ν, for example, ν = 1 mm. Other fusion methods can also be used, such as view-based methods (as disclosed in S. Galliani, K. Lasinger, and K. Schindler, Massively Parallel Multiview Stereopsis by Surface Normal Diffusion, ICCV 2015) or machine learning-based methods (such as Choe, Jaesung et al., "VolumeFusion: DeepDepth Fusion for 3D Scene Reconstruction", arXiv preprint, arXiv:2108.08623 (2021)).

[0099] Then, based on the fused point cloud, at 317 a metric 3D model of the head 318 is generated, which can be used as an avatar.

[0100] In step 317, the method includes computing a mesh based on the fused point cloud using Poisson reconstruction as discussed above. Due to noise, the mesh may have some uneven surface elements. To reduce this effect, the mesh may be smoothed using Laplace smoothing or Taubin smoothing over one or more iterations. In addition, to clean the mesh and remove unwanted artifacts, the connected components are extracted and only the largest components are retained, resulting in a cleaned mesh that corresponds to the final geometry of the model. This means that smaller non-connected meshes that may be "floating" will be discarded. Such non-connected meshes may be caused, for example, by a person having sparse hair that is not close enough to the head to be connected. The mesh can be transformed into a world coordinate system of standard size (e.g., with the z-axis pointing upwards).

[0101] To finally generate a 3D model of the head 318, the cleaned mesh in the world coordinate system is textured at 317. The eye region is textured from the central image to avoid providing textures associated with eyes taken from different perspectives. For texturing, as mentioned above, the method discussed in Waechter, Michael et al., cited above, can be used. To find the central image, for example, Figure 7 The facial segments shown in FIG are analyzed, and the largest eye segments 84 and 85 of the left eye and the right eye can be taken as the central image.

[0102] The 3D model thus created can then be used, for example, as a stand-in for virtually trying on spectacle frames, thereby determining the frame's centering parameters, among other things.

Claims

1. A method for head image registration, the method comprising: providing a plurality of combined 2D images and depth images of the head (20) captured from different positions, determining, for each of the combined 2D and depth images, a landmark point (310) that is a predefined point on the head, and Perform pairwise coarse registration between the combined 2D image and depth image pair based on these landmark points (310), Characterized in that the method further comprises: After the pairwise coarse registration, the coarsely registered combined 2D images and depth images are finely registered based on the complete point cloud represented by the combined 2D images and depth images. The fine registration is performed optionally based on odometry or iterative closest point (ICP) of a point to plane according to the geometric differences between the combined 2D image and the depth image in the corresponding pairs after the coarse registration, and determining the marker points (310) includes determining a visibility probability of the marker points based on a predetermined depth assigned to each marker point and a 2D image in the combined 2D images, Selecting landmarks based on the visibility probability, and 3D marker points are provided for the selected marker points based on these combined 2D images and depth images.

2. The method according to claim 1, wherein Pairs for the pairwise coarse registration are determined based on classification of the combined 2D and depth images with respect to orientation relative to the head on which the images were captured.

3. The method according to claim 1 or 2, wherein: Further comprising, after the coarse registration, generating a pose graph based on the fine registration.

4. The method according to claim 3, wherein The pose graph is pruned to remove incorrect edges in the pose graph.

5. The method according to claim 4, wherein The pruning includes optimizing the pose graph and comparing poses of the optimized pose graph to odometry-based poses generated based on one or more edges of the pose graph.

6. The method according to any one of claims 1 to 5, characterized in that The method is performed using a device including a 2D image sensor and a depth sensor, the device being selected from the group consisting of a smartphone and a tablet PC.

7. A method for generating a head model, the method comprising: performing the method for head image registration according to any one of claims 1 to 6, and A head model is generated based on the registration.

8. The method according to claim 7, wherein Camera poses are determined based on the registration, and the head model is generated based on the camera poses.

9. The method according to claim 7 or 8, wherein Generating the head model includes: generating a fused point cloud based on the plurality of combined 2D images and depth images and the registration, Generate a mesh based on the fused point cloud, and A texture is added to the mesh based on the 2D image in the combined 2D image and the depth image.

10. A computer program product comprising a computer program for a mobile device (10) comprising a processor, a 2D camera and a depth sensor, wherein The computer program, when run on the processor, causes the method of any one of claims 1 to 9 to be performed.

11. A computer-readable storage medium having stored thereon a computer program for a mobile device (10) comprising a processor, a 2D camera, and a depth sensor, wherein: The computer program, when run on the processor, causes the method of any one of claims 1 to 9 to be performed.

12. A device comprising a processor configured to: receiving a plurality of combined 2D images and depth images of a head (20) captured from different positions, determining, for each of the combined 2D and depth images, a landmark point (310) that is a predefined point on the head, and Perform pairwise coarse registration between the combined 2D image and depth image pair based on these landmark points (310), It is characterized in that The processor is further configured to: After the coarse registration, the coarsely registered combined 2D images and depth images are finely registered based on the complete point cloud represented by the combined 2D images and depth images. The fine registration is performed optionally based on odometry or iterative closest point (ICP) of a point to plane according to the geometric differences between the combined 2D image and the depth image in the corresponding pair after the coarse registration, and determining the marker points (310) includes determining a visibility probability of the marker points based on a predetermined depth assigned to each marker point and a 2D image in the combined 2D images, Selecting landmarks based on the visibility probability, and 3D marker points are provided for the selected marker points based on these combined 2D images and depth images.

13. The device according to claim 12, characterized in that The device further includes a camera and a depth sensor configured to capture the combined 2D images and depth images.

Images