Method and apparatus for face pose estimation and centering parameter determination

Abstract

A computer implemented method is provided that is suitable for pose estimation of a human face. The method comprises: identifying (10) a plurality of landmarks in a combined 2D image and depth image of a face; and estimating (11) a pose of the face based on a principal component analysis of the plurality of landmarks. The estimated pose can be used to determine centring parameters. Corresponding apparatus and computer programs are also discussed.

Description

Technical Field

[0001] This application relates to methods (particularly computer-implemented methods) and corresponding apparatus for facial pose estimation, as well as methods and apparatus suitable for determining centering parameters. The process of determining such parameters and applying them to spectacle lenses is commonly referred to as centering. Centering parameters are the parameters required to correctly arrange spectacle lenses in a frame (i.e., to center them) so that the spectacle lenses are worn in the correct position relative to the human eye. Background Technology

[0002] Examples of such centering parameters include interpupillary distance, vertex distance, tilt angle, y-coordinates of the left and right centering points (also known as fitting point height), face curvature of the lens rim, distance point, and other parameters defined in Section 5 of DIN EN ISO 13666:2012, as well as the tilt of the frame.

[0003] Currently, centering parameters are typically determined using a computer-based program that also includes capturing images of a person's head. In one approach, the person is wearing eyeglasses when the images are captured. In some approaches, a model of the person's head is determined based on multiple images, including a model of the eyeglasses frames if the person is wearing them. In other approaches, a model of the eyeglasses frames is provided separately and then fitted onto a model of the person's head.

[0004] Some methods for determining centering parameters use fixed devices, such as the Zeiss Visufit 1000. This device includes a calibration camera system comprising multiple cameras.

[0005] In recent years, centering methods have been developed that do not require such a fixed device but instead use mobile devices (such as smartphones or tablet PCs) for centering. For example, such a method is described in WO2021 / 233718A1.

[0006] For proper centering, the pose of the face, and therefore the head, relative to the camera used to capture the image must be correct or at least known, so that corrections can be made based on that pose. As used herein, pose refers to a combination of position and orientation in space, corresponding, for example, to the use of the term in the field of robotics according to ISO 8373:2012 4.5. A pose can be given, for example, in six coordinates, with three coordinates (e.g., Cartesian coordinates) used to represent position and three coordinates (e.g., angular coordinates) used to represent orientation. Therefore, techniques for estimating the pose of a face (also referred to herein as facial pose) are desirable.

[0007] EP 3 262 617 A1 discloses a method for fitting a pair of virtual glasses onto a real user's face, wherein the pose is estimated using a user's facial model and an image. Here, a user's facial model is required to estimate the pose.

[0008] WO 2021 / 257 406 A1 discloses a method for pupil localization that uses facial mesh markers that fuse 2D images and corresponding 3D depth maps for pupil localization. These facial mesh markers are used to generate a 3D facial mesh, which can be used to estimate interpupillary distance (PD).

[0009] US2017 / 0316582 A1 discloses the use of a depth camera to estimate the pose of a human head. Here, an image sequence is used; that is, multiple images showing the rotation of the human head must be captured.

[0010] Other methods are known from: Bazarevsky, Valentin et al., “Blazeface: Sub-millisecond neural face detection on mobile gpus”, arXiv preprint, arXiv:1907.05047 (2019); Ruiz, Nataniel, Eunji Chong and James M. Rehg, “Fine-grained head pose estimation without key points”, Proceedings of the IEEE conference on computer vision and pattern recognition workshops, 2018; or Yang, Heng et al., “Face alignment assisted by head pose estimation”, arXiv preprint, arXiv:1507.03148 (2015).

[0011] US 2016 / 086 017 discloses another method for head pose estimation, in which a 2D image and depth map of the head are captured and matched with a 3D head model having a known orientation. In other words, the transformation from the image (including the depth map) to the 3D model is calculated, and this transformation (translation and rotation) yields the head pose. This method requires a corresponding 3D head model and the computation of the corresponding transformations.

[0012] This document aims to provide a pose estimation method that does not require a head model and can operate on a single image including a depth map.

[0013] Non-patent literature—Zhang J, Gao K, Fu K, Cheng P, “Deep 3D facial landmark localization on position maps,” Neurocomputing, September 17, 2020, Vol. 406: pp. 89-98—proposes a novel 3D facial landmark localization network (3DLLN) that is robust to challenges such as pose changes, facial expression changes, and self-occlusion. 3DLLN utilizes position maps as an intermediate representation from which it detects 3D landmark coordinates. Furthermore, a deep regression architecture is employed to improve the accuracy and robustness for a large number of landmarks.

[0014] Non-patent literature—Grupp M, Kopp P, Huber P, Rätsch M, “A 3D face modelling approach for pose-invariant face recognition in a human-robot environment,” included in RoboCup 2016: RobotWorld Cup XX 20, 2017 (pp. 121-134), Springer International Publishing—discloses a 3D face modeling framework with real-time processing capabilities, applicable to 2D field images and suitable for robotics. The fitting of the 3D deformable model is entirely based on automatically detected landmarks. After fitting, the face can be pose-corrected and transformed back into a more suitable frontal 2D representation for face recognition.

[0015] Non-patent literature—González-Jiménez D and Alba-Castro JL, “Toward pose-invariant 2-d face recognition through point distribution models and facial symmetry,” IEEE Transactions on Information Forensics and Security, August 20, 2007, Vol. 2, No. 3: pp. 413-429—proposes a technique for handling pose variations in 2D face recognition scenarios. A point distribution model is constructed using a training set of sparse facial meshes, and the parameters (i.e., pose parameters) responsible for controlling the apparent deformation caused by head rotation and nodding are identified. Based on this, two methods for pose correction are disclosed: 1) a method of setting pose parameters from two meshes to typical values ​​for a frontal face, and 2) a method of using the pose parameters of one mesh with those of the other. Finally, a pose-corrected mesh is obtained, and a virtual view is synthesized using a plate spline-based deformation technique leveraging facial symmetry.

[0016] U.S. Patent 11,126,016 B2 discloses a method for determining eyeglass fitting parameters. The method includes: using a depth information detection unit to detect depth information related to a user's head; and determining eyeglass fitting parameters based on the depth information. The method further includes: using a 2D camera to record a 2D image of the user's head. The 2D camera and the depth information detection unit each have a corresponding optical axis, and the optical axis of the 2D camera and the optical axis of the depth information detection unit are combined. The 2D camera is different from the depth information detection unit. The method also includes correcting the 2D image based on the depth information, specifically by performing at least one of the following operations: aligning or correcting the 2D image to remove distortions in the 2D image.

[0017] Based on the background technology, the purpose of this disclosure is to provide an improved method for estimating the pose of a human face. Summary of the Invention

[0018] According to a first aspect, a computer-implemented method suitable for estimating the pose of a human face is provided, the method comprising identifying multiple landmarks in a combined 2D image and a depth image of the face. The method is characterized by estimating the facial pose based on principal component analysis of the multiple landmarks. Facial pose refers to a combination of the position and orientation of the face in space. Estimating the facial pose based on the principal component analysis of the identified landmarks includes: estimating the position of the face based on the average position of the multiple landmarks; and estimating the orientation of the face based on at least one unit vector generated by the principal component analysis. The at least one unit vector includes a first unit vector corresponding to the upright direction and a second unit vector and a third unit vector forming a plane perpendicular to the upright direction. The facial pose is estimated based on a coordinate system with the average position of the multiple landmarks as the origin and the first, second, and third unit vectors as axes.

[0019] By using principal component analysis, pose determination does not require a head model and can be based on a single composite 2D image and depth image. Furthermore, facial pose (referring to the combination of facial position and orientation) can be easily determined or estimated using principal component analysis.

[0020] The characteristics of the above methods will be further explained and defined below.

[0021] As explained in the introduction, pose refers to a combination of position and orientation. Pose is typically determined relative to a reference coordinate system, for example, in the case of the methods described above, relative to the apparatus used to capture a combined 2D image and a depth image. The face refers to a portion of the head that includes at least the eyes, and typically also the nose and mouth. For applications like centering, the entire head is not required; only the eye portion of the head is needed, including the eyes and possibly the nose (for mounting eyeglasses). When the pose of the face is known, the pose of the head is also known, as the face is a fixed part of the head.

[0022] A combined 2D image and depth image (referred to as a combined image) refers to a 2D image and a depth image captured essentially from the same position relative to the head or from positions that are fixed relative to each other. The 2D image can be a color image such as an RGB image (red, green, blue) or it can be a grayscale image or an infrared image. The depth image provides a map of the distance from the device used to capture the depth image to the object (in this case, the face). In the case of a color image, the combined 2D image and depth image are also referred to as an RGBD image. To capture the 2D portion of the combined 2D image and depth image, any conventional image sensor can be used in conjunction with corresponding camera optics. To capture the depth image, any conventional depth sensor, such as a time-of-flight sensor, can also be used. A combined 2D image and depth image can include two separate files or other data entities, where one data entity provides a grayscale or color value for each 2D coordinate (i.e., pixel), and another data entity provides a depth value for each 2D coordinate. A combined image can also include only a single data entity, where both grayscale / color information and depth information are provided for each 2D coordinate. In other words, the way information is stored in a data entity such as a file is irrelevant, as long as the grayscale / color information and depth information of the face are available. Cameras suitable for capturing a combined 2D image and a depth image (in this case, a color image) are also called RGBD cameras. Some modern smartphones or other mobile devices are equipped with such RGBD cameras, or with separate cameras and depth sensors. It should be noted that the depth image does not need to have the same resolution as the 2D image. In this case, scaling operations (reducing or enlarging) can be performed to adapt the resolution of the 2D image and the depth image to each other. Furthermore, for example, in devices such as smartphones, the depth sensor and the RGB camera may be slightly offset from each other, and this offset can be numerically corrected in any conventional way. The result is essentially a point cloud, where each point has a 3D coordinate based on its 2D coordinates in the image and its depth coordinates from the depth sensor, as well as a color value or grayscale value.

[0023] As used herein, markers are predefined points on the head. Markers, as used herein, can specifically be facial markers, i.e., specific points on the face. Such markers may include, for example, the tip of the nose, the bridge of the nose, the corners of the mouth or eyes, eyebrow contours, cheeks, chin, etc. Such markers can be determined from composite 2D and depth images of the face using various conventional methods. For example, trained machine learning logic, such as a neural network, can be used to determine marker points. In this case, for training, several composite 2D and depth images of multiple different facial types are used as training data, where markers can be manually labeled. After training, the trained machine learning logic determines the marker points. For details, please refer to, for example: Wu Y, Hassner T, Kim K, Medioni G, and Natarajan P, “Facial landmark detection with tweaked convolutional neural networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, Vol. 40, No. 12, pp. 3067-3074; Perakis P, Passalis G, Theoharis T, and Kakadiaris IA, “3D facial landmark detection under large yaw and expression variations,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, Vol. 35, No. 7, pp. 1552-1564; or Wu Y and Ji Q, “Facial landmark detection: A literature survey,” International Journal of Computer Vision. [International Journal of Computer Vision], 2019, Vol. 127, No. 2, pp. 115-142 (2018). Before marker detection, segmentation can be performed, where, for example, the background is separated from the face, and the face is segmented to provide facial segments. The marker can then be extracted from the facial segments. In other methods, the marker can be determined using conventional image processing techniques without employing machine learning logic.

[0024] Therefore, the markers essentially form a point cloud of 3D points. This pose is estimated based on principal component analysis (PCA) of multiple markers (i.e., the point cloud). Principal component analysis (PCA) is a known technique for analyzing datasets containing a large number of features (in this case, markers). The principal components of a set of points in real coordinate space (such as markers) are a series of unit vectors, where the i-th unit vector is the direction of the line that best fits the data and is orthogonal to the preceding i-1 vectors. The best fit can be defined as the fit that minimizes the average perpendicular distance from a point to a line with its corresponding unit vector. The 0th component, generated by PCA, is the mean of the points, calculated by summing the corresponding coordinates (X, Y, Z) of each point and dividing by the total number of points. The unit vectors are the eigenvectors of the covariance matrix of the data (in this case, markers).

[0025] Preferably, the 0th component (i.e., the average of the coordinates of the marker) is considered the position of the head, and the orientation is taken from the first few (e.g., the first to the third) unit vectors. For example, a face is typically elongated in the vertical direction, and the first unit vector would correspond to the vertical direction, also known as the upward vector. The next two unit vectors then form a plane perpendicular to the vertical direction, so the pose is known. In some cases, it is not possible to directly find the correct mapping between the unit feature vector and the left / right and forward directions relative to the camera. This occurs if the shape of the head differs from that of a typical face. In this case, the method can therefore select a vector pointing to the device used to capture the composite image as the forward vector. The left / right directions are then calculated using the cross product of the upward vector and this forward vector.

[0026] In this way, principal component analysis can be used to easily determine the location and orientation.

[0027] Signs can be identified in the aforementioned 2D image composed of a combined 2D image and a depth image, and the 3D coordinates of multiple signs can be provided based on the signs in the 2D image and associated depth information from the depth image. In this way, 3D signs can be generated based on the combined 2D image and the depth image.

[0028] Preferably, the estimated pose can be filtered by applying an averaging filter to the estimated pose and a previously estimated pose. This is particularly feasible if the method is performed multiple times consecutively on the same face, in which case it can be assumed that only minor changes occur in the facial pose. The previously estimated pose is then the pose estimated when the method was previously performed. The averaging filter can be, for example, an exponentially moving average (EMA) filter, which is a first-order infinite impulse response filter that applies exponentially decreasing weighting factors, such that the more distant the previously estimated pose, the lower its weight in the filtering.

[0029] This can improve the repeatability and reproducibility of pose estimation.

[0030] Although the method described above can be repeated, it also works effectively when the combined 2D image and depth image are a single combined 2D image and depth image, i.e., pose estimation only requires a single combined 2D image and depth image.

[0031] Additionally, a computer is provided, comprising a processor configured to perform any of the methods described above, namely, at least the following steps: identifying multiple landmarks in a combined 2D image and a depth image of the face; and estimating the facial pose based on principal component analysis of the multiple landmarks. A data processing system is also proposed, comprising a processor and a storage medium coupled to the processor, wherein the processor is adapted to perform the steps of the methods described above based on a computer program stored in the storage medium, namely, at least the following steps: identifying multiple landmarks in a combined 2D image and a depth image of the face; and estimating the facial pose based on principal component analysis of the multiple landmarks. The term "computer" does not necessarily mean a single computer, but can also refer to a computer network for data transfer between computers. For example, landmark identification can be performed in one computer, and then the data can be transferred to another computer for pose estimation.

[0032] A computer program including instructions is also provided, which, when executed by a computer, cause the computer to perform any of the methods described above. Furthermore, a computer-readable storage medium having the computer program stored thereon and a data carrier signal carrying the computer program are also provided. Additionally, a computer program stored on a non-transitory tangible computer-readable storage medium is also provided, the computer program including instructions that, when executed by a computer, cause the computer to perform any of the methods described above.

[0033] According to a second aspect, a method suitable for determining at least one centering parameter for fitting spectacle lenses into a person's eyeglass frame is provided, wherein the method is characterized by: capturing a combined 2D image and a depth image of a person's face; estimating the pose of the face based on the combined 2D image and the depth image; and determining at least one centering parameter based on the combined 2D image and the depth image and the estimated pose. Determining at least one centering parameter based on the combined 2D image and the depth image and the estimated pose includes: determining a deviation between the estimated pose and a reference pose; and at least partially correcting the deviation when determining the at least one centering parameter. The deviation between the estimated pose and the reference pose is determined based on a coordinate system given by the estimated pose of the head and another reference coordinate system of the means for capturing the combined 2D image and the depth image. In this way, the centering parameter can be determined based on a single combined 2D image and a depth image.

[0034] Generally, facial pose can be advantageously estimated using any of the methods described above.

[0035] In a preferred embodiment, the facial pose can be estimated using principal component analysis. Estimating the facial pose based on a combined 2D image and a depth image may include: identifying multiple landmarks in the combined 2D image and depth image of the face; and estimating the facial pose based on principal component analysis of these multiple landmarks.

[0036] By using principal component analysis, pose determination does not require a head model and can be based on a single composite 2D image and depth image. Furthermore, facial pose (referring to the combination of facial position and orientation) can be easily determined or estimated using principal component analysis.

[0037] In an embodiment, facial pose refers to the combination of facial position and orientation in space. Estimating facial pose based on principal component analysis of identified landmarks may include: estimating facial position based on the average position of multiple landmarks; and estimating facial orientation based on at least one unit vector generated by the principal component analysis. This at least one unit vector may include a first unit vector corresponding to the upright direction and a second and third unit vector forming a plane perpendicular to the upright direction. Facial pose can be estimated based on a coordinate system with the average position of multiple landmarks as its origin and the first, second, and third unit vectors as its axes. That is, the coordinate system given by the estimated pose of the head may correspond to a coordinate system with the average position of multiple landmarks as its origin and the first, second, and third unit vectors as its axes. Therefore, improved or alternative methods can be used to estimate facial pose.

[0038] Preferably, identifying multiple markers may include: identifying multiple markers in a 2D image of a combined 2D image and a depth image; and providing 3D coordinates of the multiple markers based on the multiple markers in the 2D image and depth information from the depth map of the combined 2D image and the depth image.

[0039] In an embodiment, the estimated pose can be filtered by using an averaging filter on the estimated pose and the previously estimated pose.

[0040] The general approach for determining centering parameters is similar to conventional computer-based centering methods. The combined 2D image and depth image are essentially treated as a 3D model of at least a portion of the face. A frame model is provided to this 3D model, based on a frame found in the image or a fitted frame model. Centering parameters are then determined, for example, based on the pupil position in the combined 2D image and depth image, and the position of the frame. However, by additionally using the estimated pose, the centering parameters can be determined more precisely. Several methods for achieving this, which can be used in combination or individually, will be described below.

[0041] In some embodiments, the deviation between the estimated pose and a reference pose can be determined. Based on this deviation, information can then be output to a person to correct the facial pose, and the capture and estimation can be repeated. For example, the output can be given via a speaker or a display, and instructions can be given to the person on how to correct the pose (e.g., "move your face to a more upright position," "tilt your head more to the right / left," etc.). This can be repeated until the estimated pose matches the reference pose sufficiently, for example, until the deviation is below a threshold.

[0042] Additionally or alternatively, this deviation can be mathematically compensated for when centering parameters are determined. For example, a tilt of the head relative to an upright position may cause a corresponding change in some centering parameters (such as fitting height or pupillary distance), which can be corrected based on simple geometric principles. These two methods can be combined; for example, the person can be instructed to correct his / her facial pose, and then the remaining deviation can be mathematically addressed.

[0043] In this way, the correct pose can be ensured to determine the centering parameters. In this way, pose deviations can be corrected.

[0044] In other embodiments, a pose can be used to fit an eyeglass frame model (also called an eyeglass frame template) to a composite 2D image and a depth image. The depth image may be affected by low image resolution or sensor noise. Due to the resolution limitations of the projected pattern, structured light-based sensors often struggle to capture small objects. For example, even if a person is wearing eyeglass frames in a composite image, the depth sensor may not capture the frames, instead interpolating depth values ​​from the background, resulting in incorrect depth information at the location where the frames are worn. This is especially true for thin-rimmed frames.

[0045] In this scenario, a frame template composed of the planes of the left and right lenses can be fitted. The origin of the planes can be located at the center of the bridge of the frame, for example, by detecting the center of the bridge of the frame in a 2D image using feature detection or a deep learning marker model. Preferably, the method uses a fixed marker located in the nose region from the markers extracted for pose estimation, referred to as the bridge of the frame point. A region of interest is cropped from the image around this bridge of the frame point. The method then uses classical image processing techniques (such as edge detection) to detect the point with the highest response amplitude within this region of interest as the bridge of the frame. This 2D location (i.e., the bridge of the frame detected in the 2D image) is backprojected using the depth value at the corresponding location in the depth map and camera intrinsics, which describe how the scene (3D world coordinates) is converted to 2D image coordinates. Camera intrinsics are typically represented as matrices. The inverse of this matrix can be used to backproject the 2D pixel location in the 2D image back to its 3D location relative to the camera along with its corresponding z-value in the depth map. The 3D position of the nose bridge of the eyeglasses in the camera coordinate system (i.e., generated as from a combined 2D image and a depth image) is then transformed to the face pose coordinate system (up vector, forward vector, left / right vector) as determined above. The position in the face pose coordinate system can then be filtered using an averaging filter, similar to the face pose processing described above, to reduce detection noise. The template eyeglasses plane is then positioned with the nose bridge point in the face pose coordinate system as the origin, and with rotation derived from the wrap angle and tilt angle, which have been empirically estimated as default angles. For example, the wrap angle can be set to 5°, and the tilt angle can be set to 8°. The rotation is calculated based on these angles in a face coordinate system aligned with the ground plane. This coordinate system is derived from the face coordinate system and assumes that the capture device is positioned in an upright manner, with the optical axis parallel to the camera. This adjustment eliminates the problem that the forward axis of the face pose is not always aligned with the camera's optical axis.

[0046] The 2D segmentation of the frame in the RGB color space can be projected onto these planes to upscale the 2D model to 3D, or a generic frame model can be positioned on these planes. Then, in its usual manner, this projection of the frame onto the planes is used to estimate centering values ​​such as fitting height, frame size, distance from the frame center to the eye point, and distance to the back vertex. On devices with higher-resolution depth sensors, the forward tilt and frame curvature angles can be calculated directly from the data because the depth of the frame in the 2D image can be obtained more accurately. Then, 3D point cloud segmentation can be used instead of 2D image segmentation to separate the frame points from the head points. Finally, RANSAC and least-squares fitting can be used to directly fit the planes to the left and right frame point clouds. This method is described, for example, in: Martin A. Fischler and Robert C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography” (PDF), Comm. ACM, June 1981, Vol. 24, No. 6: pp. 381-395, doi:10.1145 / 358669.358692. S2CID 972888. The centering can then be estimated conventionally, as described in the previous section.

[0047] The above centering process can be performed on a frontal facial image (i.e., an image taken from the front). The computer can be a mobile device, such as a smartphone.

[0048] The method described above for determining at least one centering parameter can be repeated multiple times using different combined 2D images and depth images. The final centering parameter can then be determined by averaging the centering parameters determined by each method.

[0049] A method for providing spectacle lenses is also provided, the method comprising: determining at least one centering parameter as described above; and manufacturing spectacle lenses based on the at least one centering parameter.

[0050] Similarly, for the second aspect, a corresponding device is provided. This device includes: a camera and a depth sensor for capturing a combined 2D image and a depth image of a human face; and a computer including a processor configured to perform the remaining steps of the methods described above, namely, at least the following steps: estimating the pose of the face; and determining at least one centering parameter. This device can be a mobile device with a corresponding configuration, such as a smartphone or tablet PC. Another device is provided, including: a camera device as described above; and a data processing system including a processor and a storage medium coupled to the processor, wherein the processor is adapted to perform the remaining steps besides capturing an image, namely, estimating the pose and determining at least one centering parameter. A computer program for a device including a camera device is also provided, wherein the computer program, when executed on the processor of the device, causes any of the methods described above in the second aspect to be performed. This computer program can be provided, for example, as an App (application program) to a mobile device (such as a smartphone).

[0051] A computer-readable storage medium having the aforementioned computer program stored thereon is also provided, as well as a data signal carrying the computer program. The computer program can be stored on a non-transitory tangible computer-readable storage medium, and the computer program includes instructions that, when executed by a device including a camera device, cause the device to perform the method of the second aspect described above.

[0052] The methods and apparatus described above for determining at least one centering parameter can also be used in conjunction with other methods for estimating pose, different from the principal component analysis-based pose estimation methods described above. For example, the methods for determining pose mentioned in the introduction can be used. In other embodiments, conventional toolkits suitable for determining pose based on markers can be used. For example, in iOS systems, ARKit provides the possibility of determining facial pose based on markers. In other embodiments, conventional deep learning methods can be used, wherein a deep learning neural network is trained using multiple sets of markers and associated poses so as to output a pose based on these markers after training. Thus, for example, according to a third aspect, a method suitable for determining at least one centering parameter for fitting eyeglass lenses into a person's eyeglass frame is provided, characterized by: capturing a combined 2D image and a depth image of a person's face; estimating the facial pose based on the combined 2D image and the depth image; and determining at least one centering parameter based on the combined image and the estimated pose. A corresponding apparatus is also provided.

[0053] Other features, devices, computer programs, etc., described above in relation to the second aspect also apply to this third aspect.

[0054] In addition, a method suitable for providing spectacle lenses is provided, the method comprising: determining at least one centering parameter using any of the above methods; and centering the spectacle lens based on the at least one centering parameter. Attached Figure Description

[0055] Preferred embodiments will now be described with reference to the accompanying drawings, in which:

[0056] Figure 1 This is a flowchart illustrating a method according to an embodiment.

[0057] Figure 2 This is a flowchart illustrating a method according to an embodiment.

[0058] Figure 3 This is a block diagram of the device according to an embodiment.

[0059] Figures 4A to 4D , Figures 5A to 5C as well as Figure 6A and Figure 6B It is used for explanation Figure 1 and Figure 2 Various types of schematic diagrams of the methods. Detailed Implementation

[0060] Figure 1 This is a flowchart illustrating a computer-implemented method for pose estimation according to an embodiment. In step 10, the method includes identifying markers in a combined 2D image and a depth image, as discussed above. For illustration, Figure 4A A 2D image 40 of a human head 41 (including the face) is shown, in which the person is wearing glasses 42. Figure 4B A corresponding depth map 43 is shown. For each pixel of the face 41 in the 2D image 40, the corresponding depth can be obtained from the depth map 43. In the example shown, the resolution of the depth map 43 is lower than that of the 2D image 40, causing several pixels in the depth map to be assigned the same depth. As can be seen, because of this, the depth of the frame 42 is not captured correctly in this example, and the black areas represent areas where the correct depth cannot be assigned. In other embodiments, with higher resolution, the depth can be captured correctly.

[0061] Figure 4C An example is shown of identifying multiple landmarks 34 in a 2D image, wherein a depth is assigned to each landmark identified in the 2D image according to a depth map 43, thereby generating 3D landmarks.

[0062] Return to Figure 1 In step 12, the facial pose is estimated based on principal component analysis, as described above. Figure 4DThe corresponding results are shown, where the position is represented by the origin 45 of the coordinate system, which corresponds to... Figure 4C The average position of marker 44, as described above, and arrows 46 and 47 indicate the pose (arrow 46 indicates the upright direction of the head, and arrow 47 indicates the left and right direction), these arrows correspond to the unit vector.

[0063] Figure 2 A method suitable for determining at least one centering parameter according to an embodiment is shown.

[0064] In step 20, the method includes capturing a composite image of a person's head, for example, such as Figure 4A and Figure 4B As shown.

[0065] In step 21, Figure 2 Methods include estimating pose, for example using Figure 1 The method may be used, or any of the other methods described above. In step 22, the method includes determining at least one centering parameter, as described above. Steps 23 through 25 illustrate several possibilities that the centering parameter can be determined using the pose estimation from step 21, as explained above. In step 23, if the pose does not match the reference pose, feedback is given to the person so that they can correct their pose and the method can be restarted. In step 24, mathematical correction may be performed. The reference... Figures 5A to 5C Show steps 23 and 24.

[0066] Ideally, the head pose should be at least approximately as follows: Figure 4A As shown, the figure is upright, with the eyes looking almost directly forward. Figures 5A to 5C Various deviations from this ideal pose are shown.

[0067] exist Figures 5A to 5C In the figures, reference numeral 51 denotes a combined 2D image and a depth image, wherein each pixel has 3D coordinates. Reference numeral 50 denotes the reference coordinate system of the apparatus for capturing the combined 2D image and the depth image (an example of such an apparatus can be found below). Figure 3 (Description of the figures). In their respective cases, reference numeral 52 indicates a coordinate system given by the pose of the head (e.g., Figure 4D The origin 45, according to Figure 4D (The axes of arrows 46 and 47, and the third axis perpendicular to them).

[0068] Figure 5A The diagram shows a person looking at the camera at an angle instead of straight ahead. In this case, the person can be instructed to look more directly forward, and this process can be repeated. Additionally or alternatively, in step 24, this deviation can be mathematically corrected. For example, in... Figure 5A In the attached diagram, reference numeral 53 indicates the lens plane of an eyeglass frame worn by a person. From the camera's perspective (i.e., from coordinate system 50), the point where the line from the pupil to the camera intersects plane 53 differs from the case of looking directly forward. This is due to the posture (especially the head and...) Figure 4A The angle of rotation (the frontal position) can be known through pose estimation, and this difference can be mathematically corrected using simple geometric principles.

[0069] Figure 5B An example of the head tilting to the side is shown. Reference numeral 54 denotes an eyeglass frame. Here, for example in step 23, an instruction can be given to keep the head more upright, or mathematical corrections can be used since the relative angle between the pose indicated by coordinate system 52 and coordinate system 50 is known. It should be noted that... Figures 5A to 5C The deviation from the upright pose is drawn in an exaggerated manner to provide better understanding, but in practice, the deviation may be smaller.

[0070] Figure 5C This shows a side view with the head tilted backward. This also changes the point from which the person sees through the glasses when viewed from coordinate system 50. Alternatively, in step 23, the person could be instructed to keep their head more upright, or similarly in step 24, mathematical corrections could be performed.

[0071] Return to Figure 2 Additionally or alternatively, in step 24, a frame template (which is then used to determine the centering parameters) can be fitted to a facial point cloud defined by the combined 2D image and depth image. This will refer to Figure 6A and Figure 6B To explain. Figure 6A A frontal image of a facial point cloud 51 with eyeglasses frame 53 is shown. Although the eyeglasses frame can be identified in the 2D image (see...). Figure 4A However, depth information may be unreliable (see [link]). Figure 4B Therefore, as in Figure 6B As can be seen in the top view, the frame template 53 is positioned based on 2D segmentation of the frame in the 2D image, constrained by the face pose and fixed frame bending and tilt angles. Specifically, for example, according to... Figure 4C The markers indicate the point of the nose, and the orientation can be known based on the pose estimation indicated by coordinate system 52, so that plane 53 can be positioned in the correct location and orientation.

[0072] Figure 1 The method can be implemented using any computer device. Figure 2The methods can be implemented in any computer device incorporating a camera, which may be external to the device. However, these methods are preferably implemented in mobile devices such as smartphones or tablet PCs. Figure 3 This is implemented in the mobile device 30 shown schematically. The mobile device 30 includes a touchscreen 32, which serves as an input / output device, for example, for user input commands and / or output. Figure 2 The instructions in step 23. Additionally or alternatively, such instructions may also be output via speaker 35 or, for example, via a headset connected to mobile device 30 in a conventional manner (e.g., via Bluetooth).

[0073] The mobile device 30 also includes a camera 33 and a depth sensor 34, which are used to capture combined 2D images and depth images. As mentioned above, some current smartphones already include such a camera device including camera 33 and depth sensor 34.

[0074] Furthermore, the mobile device 30 includes a processor 31 that controls the mobile device 30 to perform actions such as... Figure 1 and Figure 2 The described method (i.e., controlling the camera 33 and depth sensor 34 to capture a combined 2D image and a depth image) is executed. Figure 1 and Figure 2 The calculations required for the other steps.

[0075] In summary, the embodiments discussed above can utilize principal component analysis (PCA) to analyze landmarks in composite 2D images and depth images of a human face to estimate facial pose. For example, PCA can be directly applied to landmarks in individual images of a human face to determine the facial pose in each image.

[0076] Some embodiments are defined by the following terms.

[0077] Clause 1. A computer-implemented method suitable for pose estimation of a human face (41), the method comprising:

[0078] Multiple markers (44) were identified (10) in the combined 2D image and depth image (40, 43) of the face.

[0079] Its features are,

[0080] The pose of the face (41) is estimated (11) based on principal component analysis of the multiple markers (44).

[0081] Clause 2. The method as described in Clause 1, characterized in that estimating (11) the pose of the face (41) based on principal component analysis of the plurality of signs (44) includes estimating the position of the face (41) based on the average position of the plurality of signs (44).

[0082] Clause 3. The method as described in Clause 1 or 2, characterized in that estimating (11) the pose of the face (41) based on principal component analysis of the identified marker (44) includes estimating the orientation of the face (41) based on at least one unit vector generated by the principal component analysis.

[0083] Clause 4. The method of any one of Clauses 1 to 3, wherein identifying (10) the plurality of marks (44) comprises: identifying the plurality of marks (44) in a 2D image (40) of the combined 2D image and depth image (40, 43); and providing 3D coordinates of the plurality of marks (44) based on the plurality of marks (44) in the 2D image (40) and depth information from a depth map (43) of the combined 2D image and depth image (40, 43).

[0084] Clause 5. The method of any one of Clauses 1 to 4, characterized in that the estimated pose is filtered by using an averaging filter on the estimated pose and a previously estimated pose.

[0085] Clause 6. The method as described in any one of Clauses 1 to 5, characterized in that the combined 2D image and depth image (40, 43) are a single combined 2D image and depth image.

[0086] Clause 7. A method suitable for determining at least one centering parameter for fitting spectacle lenses into a person's spectacle frame, characterized in that:

[0087] Capture (20) a combined 2D image of the person’s face (41) and a depth image (40, 43).

[0088] The pose of the face (41) is estimated (21) by any one of the methods described in any one of Clauses 1 to 6, and

[0089] The at least one centering parameter is determined based on the combined 2D image and depth image (40, 43) and the estimated pose (22).

[0090] Clause 8. The method as described in Clause 7, characterized in that the determination (22) of the at least one centering parameter based on the combined 2D image and depth image (40, 43) and the estimated pose comprises:

[0091] Determine the deviation between the estimated pose and the reference pose.

[0092] Based on this deviation, information is output to the person to correct the pose, and

[0093] Repeat the capture (20) and the estimate (21).

[0094] Clause 9. The method as described in Clause 7 or 8, characterized in that the determination (22) of the at least one centering parameter based on the combined 2D image and depth image (40, 43) and the estimated pose comprises:

[0095] Determine the deviation between the estimated pose and the reference pose, and

[0096] When determining the at least one centering parameter, the deviation is at least partially corrected (24).

[0097] Clause 10. The method as described in any one of Clauses 7 to 9, characterized in that the determination (22) of the at least one centering parameter based on the combined 2D image and depth image (40, 43) and the estimated pose comprises:

[0098] Based on this pose, the eyeglass frame template (53) is fitted to the combined 2D image and depth image (40, 43), and

[0099] The at least one centering parameter is determined based on the fitted eyeglass frame template (53).

[0100] Clause 11. A method for providing an eyeglass lens, characterized in that it comprises determining at least one centering parameter by a method as described in any one of Clauses 7 to 10; and manufacturing the eyeglass lens based on the at least one centering parameter.

[0101] Clause 12. A computer program comprising instructions which, when executed by a computer, cause the computer to perform the method as described in any one of Clauses 1 to 6.

[0102] Clause 13. A computer program for an apparatus (30) including a camera (33), a depth sensor (34) and at least one processor (31), the computer program including instructions that, when executed by the at least one processor (31), cause to perform the method as described in any one of Clauses 7 to 10.

[0103] Clause 14. A computer including a processor configured to perform the steps of: identifying (10) a plurality of signs (44) in a combined 2D image and depth image (40, 43) of the face, and

[0104] The pose of the face (41) is estimated (11) based on principal component analysis of the multiple markers (44).

[0105] Clause 15. An apparatus comprising a camera (33), a depth sensor (34), and at least one processor (31) configured to cause the apparatus to perform the steps of: capturing (20) a combined 2D image of a person’s face (41) and a depth image (40, 43).

[0106] Multiple markers (44) were identified (10) in the combined 2D image and depth image (40, 43) of the face, and

[0107] The pose of the face (41) is estimated (11) based on principal component analysis of the multiple markers (44), and

[0108] The at least one centering parameter is determined based on the combined 2D image and depth image (40, 43) and the estimated pose (22).

Claims

1. A computer-implemented method suitable for pose estimation of a human face (41), the method comprising: Multiple markers (44) are identified (10) from a combination of 2D images (40) and depth images of the face, taken from the same position relative to the head or from positions that are fixed to each other. The pose of the face (41) is estimated (11) based on principal component analysis of the multiple markers (44). Its features are: The pose of the face (41) refers to the combination of the position and orientation of the face in space; Among them, the estimation (11) of the pose of the face (41) based on principal component analysis of the identified marker (44) includes: The position of the face (41) is estimated based on the average of the coordinates of the multiple markers (44), wherein the average of the coordinates of the multiple markers is the sum of the corresponding coordinates of the multiple markers and divided by the number of the multiple markers; The orientation of the face (41) is estimated based on at least one unit vector generated by the principal component analysis, wherein the at least one unit vector is an eigenvector of the covariance matrix of the plurality of signs; The at least one unit vector includes a first unit vector corresponding to the upright direction and a second unit vector and a third unit vector forming a plane perpendicular to the upright direction. The pose of the face is estimated based on a coordinate system with the average of the coordinates of the plurality of signs (44) as the origin (45) and the first unit vector, the second unit vector and the third unit vector as the axes.

2. The method as described in claim 1, characterized in that, Identifying (10) the plurality of signs (44) includes: identifying the plurality of signs (44) in the 2D image (40) of the composite image; and providing the 3D coordinates of the plurality of signs (44) based on the plurality of signs (44) in the 2D image (40) and depth information from the depth map of the composite image.

3. The method as described in claim 1 or 2, characterized in that, The estimated pose is filtered by applying an averaging filter to the estimated pose and the previously estimated pose.

4. The method according to any one of claims 1 to 3, characterized in that, This composite image is a single composite image.

5. A method suitable for determining at least one centering parameter for fitting spectacle lenses into a person's spectacle frame, the method comprising: Capture (20) a composite image of the person's face (41), Based on this composite image, the pose of the face (41) is estimated (21), and Based on the combined image and the estimated pose, (22) the at least one centering parameter is determined. The feature is that the determination of (22) the at least one centering parameter based on the combined image and the estimated pose includes: Determine the deviation between the estimated pose and the reference pose, and When determining the at least one centering parameter, the deviation is at least partially corrected (24); The deviation between the estimated pose and the reference pose is determined based on a coordinate system given by the estimated pose of the head and another reference coordinate system of the device used to capture the composite image. The estimation (21) of the pose of the face (41) based on the combined image is performed by the method according to any one of claims 1 to 4.

6. The method as described in claim 5, characterized in that, The determination of (22) at least one centering parameter based on the combined image and the estimated pose includes: Based on this deviation, information is output to the person to correct the pose, and Repeat the capture (20) and the estimate (21).

7. The method as described in claim 5 or 6, characterized in that, The determination of (22) at least one centering parameter based on the combined image and the estimated pose includes: Based on this pose, the eyeglass frame template (53) is fitted to the composite image, and The at least one centering parameter is determined based on the fitted eyeglass frame template (53).

8. A method for providing spectacle lenses, characterized in that, This includes determining at least one centering parameter by the method described in any one of claims 5 to 7; and manufacturing the spectacle lens based on the at least one centering parameter.

9. A computer-readable storage medium having a computer program stored thereon comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon for an apparatus (30) including a camera (33), a depth sensor (34) and at least one processor (31), the computer program including instructions that, when executed by the at least one processor (31), cause to perform the method as claimed in any one of claims 5 to 7.

11. A computer comprising a processor configured to perform the steps of: identifying (10) a plurality of markers (44) in a combined image of a face comprising 2D images (40) taken from the same position relative to the head or from positions having a fixed relationship with each other, and... The pose of the face (41) is estimated (11) based on principal component analysis of the multiple markers (44). Its features are: The pose of the face (41) refers to the combination of the position and orientation of the face in space; Among them, the estimation (11) of the pose of the face (41) based on principal component analysis of the identified marker (44) includes: - The position of the face (41) is estimated based on the average of the coordinates of the multiple markers (44), wherein the average of the coordinates of the multiple markers is the sum of the corresponding coordinates of the multiple markers and divided by the number of the multiple markers; - The orientation of the face (41) is estimated based on at least one unit vector generated by the principal component analysis, wherein the at least one unit vector is an eigenvector of the covariance matrix of the plurality of signs; The at least one unit vector includes a first unit vector corresponding to the upright direction and a second unit vector and a third unit vector forming a plane perpendicular to the upright direction. The pose of the face is estimated based on a coordinate system with the average of the coordinates of the plurality of signs (44) as the origin (45) and the first unit vector, the second unit vector and the third unit vector as the axes.

12. The computer of claim 11, wherein, The processor is configured to perform the method as described in any one of claims 2 to 4.

13. An apparatus suitable for determining at least one centering parameter for fitting spectacle lenses into a person's eyeglass frame, the apparatus comprising a camera (33), a depth sensor (34), and at least one processor (31), the at least one processor being configured to cause the apparatus to perform the following steps: Capture (20) a composite image of a person's face (41), Multiple markers (44) were identified (10) in the composite image of the face. The pose of the face (41) is estimated (11) based on principal component analysis of the multiple markers (44), and Based on the combined image and the estimated pose, (22) the at least one centering parameter is determined. Its features are, The determination of (22) at least one centering parameter based on the combined image and the estimated pose includes: - Determine the deviation between the estimated pose and the reference pose, and - In determining the at least one centering parameter, at least partially correct (24) the deviation; The deviation between the estimated pose and the reference pose is determined based on a coordinate system given by the estimated pose of the head and another reference coordinate system of the device used to capture the composite image. The estimation (11) of the pose of the face (41) is performed by the method according to any one of claims 1 to 4.

14. The device as claimed in claim 13, wherein, The at least one processor (31) is configured to cause the device to perform the method as described in claim 6 or 7.

15. A computer including a processor configured to perform the following steps: The pose of the face (41) is estimated (21) based on the composite image of the person, and Based on the combined image and the estimated pose, at least one centering parameter is determined (22). Its features are, The determination of (22) at least one centering parameter based on the combined image and the estimated pose includes: Determine the deviation between the estimated pose and the reference pose, and When determining the at least one centering parameter, the deviation is at least partially corrected (24); The deviation between the estimated pose and the reference pose is determined based on a coordinate system given by the estimated pose of the head and another reference coordinate system of the device used to capture the composite image. The estimation (21) of the pose of the face (41) based on the combined image is performed by the method according to any one of claims 1 to 4.

16. The computer of claim 15, wherein, The processor is configured to perform the method as described in claim 6 or 7.

Images