Monocular camera defocus face measurement

Abstract

A device for measuring a user's face size (referred to as face zoom) using a monocular camera. And calculating the depth according to the sparse feature points. And the estimation accuracy is improved by using a surface grid. The processing pipeline detects facial features by applying facial landmark detection algorithms to find important facial feature points (such as eyes, nose, and mouth). The processing pipeline estimates the feature point depth using the depth obtained by image defocus. The processing pipeline further scales the face using the estimated depth of the facial feature.

Description

Technical Field

[0001] Examples set forth in this disclosure relate to a device having a monocular camera. Background Art

[0002] In virtual try-on and augmented reality (AR) shopping, it is important to understand the actual size of the product so that the user knows which size they should purchase. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The features of the various embodiments disclosed will be readily understood from the following detailed description, with reference to the accompanying drawings. In this specification and in the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements, with an added lowercase letter indicating a specific element.

[0004] Unless otherwise indicated, the various elements shown in the drawings are not drawn to scale. For clarity, the sizes of various elements may be exaggerated or reduced. Several drawings depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following:

[0005] Figure 1 is a high-level functional block diagram of an example device including a monocular camera;

[0006] Figure 2 are images of a user wearing products of different sizes (shown as glasses) covering the user's face;

[0007] Figure 3A and Figure 3B is a diagram depicting the image scale ambiguity when a single front-facing monocular camera images a user's face;

[0008] Figure 4 is a diagram depicting a pipeline of a CPU for determining scaling of a user's face using a single monocular front-facing camera;

[0009] Figure 5 is a flow chart of the steps of an auto focus mode for determining the true proportions of a user's face; and

[0010] Figure 6 is a flow chart of the manual focus mode steps for determining the true proportions of a user's face. DETAILED DESCRIPTION

[0011] A device that uses a monocular camera to measure the size of a user's face (known as face scaling). A dense depth map is not required because the depth calculated from sparse feature points is sufficient. The depth is used to obtain an accurate scale from one or several initial image frames captured by a monocular camera (such as the front-facing monocular camera of a smartphone). The scale is then calculated by the device's processor using face tracking. A facial mesh is used to improve the estimation accuracy. A processing pipeline detects facial features by applying a face landmark detection algorithm to find important facial feature points (such as eyes, nose, and mouth). The processing pipeline estimates the feature point depth using the depth obtained by defocusing the image. The processing pipeline further uses the estimated depth of the facial features to scale the face.

[0012] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link through which an electrical or magnetic signal generated or provided by one system element is imparted to another coupled or connected system element, etc. Unless otherwise described, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element or indirectly supported by an element through another element integrated into or supported by the element.

[0013] Additional objects, advantages and novel features provided in the following description will become apparent to those skilled in the art upon examination of the specification and drawings, or may be learned through production or operation of the examples. The objects and advantages of the subject matter may be realized and obtained using the methods, means and combinations particularly noted herein.

[0014] Reference is now made in detail to the examples illustrated in the accompanying drawings and discussed below.

[0015] Figure 1 1 shows a high-level functional block diagram of an example mobile device in a sample configuration. As shown, the smartphone 100 includes a flash memory 110 that stores programming executed by a CPU 120 to perform all or a subset of the functions described herein. Figure 1As shown, the CPU 120 of the smart phone 100 includes a mobile display driver 130, a user input layer 140 (e.g., a touch screen) of a front image display 145, a display controller 150, a front visible light camera 155, and one or more rear visible light cameras 160 with substantially overlapping fields of view. In such a configuration, the flash memory 110 may further include a plurality of images or videos generated via the camera.

[0016] like Figure 1 As shown, the smart phone 100 may further include at least one digital transceiver (XCVR) 165, shown as a WWAN (Wireless Wide Area Network) XCVR, for digital wireless communication via a wide area wireless mobile communication network. The smart phone 100 may also include additional digital or analog transceivers, such as a short-range transceiver (XCVR) 170 for short-range network communication, such as via NFC (Near-fieldcommunication), VLC (Visible light communication), DECT (Digital Enhanced Cordless Telecommunications), ZigBee, or For example, the short-range XCVR 170 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard protocols for communications implemented in wireless local area networks, such as those under IEEE 802.11. In certain configurations, the XCVR 170 may also be configured to communicate with a global event database.

[0017] In order to generate location coordinates for locating the smart phone 100, the smart phone 100 may also include a global positioning system (GPS) receiver. Alternatively or additionally, the smart phone 100 may utilize one or both of the short-range XCVR 170 and the WWAN XCVR 165 to generate location coordinates for positioning. For example, based on a cellular network, or Positioning systems can generate very accurate location coordinates, especially when used in combination.

[0018] The transceivers 165, 170 (i.e., network communication interfaces) may conform to one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 165 include, but are not limited to, transceivers configured to operate in accordance with Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies, including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE (Long Term Evolution), sometimes referred to as "4G", or 5G New Radio, referred to as "5G". For example, transceivers 165, 170 provide two-way wireless communication of information, including digitized audio signals, still images and video signals, web page information for display and network-related input, and various types of mobile message communications to / from smartphone 100.

[0019] The smartphone 100 further includes a microprocessor that functions as a central processing unit (CPU). Figure 1 120 in FIG. A microprocessor is a circuit having elements constructed and arranged to perform one or more processing functions (typically various data processing functions). Although discrete logic components may be used, the example utilizes components that form a programmable CPU. A microprocessor includes, for example, one or more integrated circuit (IC) chips that incorporate electronic components to perform the functions of the CPU 120. For example, the CPU 120 may be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM (Advanced RISC Machine) architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of microprocessor circuits may be used to form the CPU 120 or microprocessor hardware in smart watches, smart phones, laptop computers, and tablet computers.

[0020] The CPU 120 functions as a programmable host controller for the smartphone 100 by configuring the smartphone 100 to perform various operations according to instructions or programming executable by the CPU 120. Such operations may include, for example, various general operations of the smartphone 100, and operations related to the programming of applications on the smartphone 100. Although a microprocessor may be configured through the use of hard-wired logic, a typical microprocessor in a mobile device is a general-purpose processing circuit that is configured through the execution of programming.

[0021] Smartphone 100 further includes a memory or storage system for storing programming and data. Figure 1 In the example shown, the memory system may include flash memory 110, random-access memory (RAM) 180, a local event database, and other memory components as needed. RAM 180 may be used as short-term storage for instructions and data processed by CPU 120, such as working data processing memory, while flash memory 110 generally provides long-term storage.

[0022] Thus, in the example of a smartphone 100, the flash memory 110 is used to store programming or instructions that are executed by the CPU 120. Depending on the type of device, the smartphone 100 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include: Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS, RIM Blackberry OS, etc.

[0023] In a sample configuration, the CPU 120 may construct a map of the environment surrounding the smartphone 100, determine the location of the smartphone 100 within the mapped environment, and determine the relative position of the smartphone 100 to one or more objects in the mapped environment. The CPU 120 may construct the map and determine the location and position information using a simultaneous localization and mapping (SLAM) algorithm applied to data received from one or more sensors.

[0024] The sensor data may include images received from cameras 155 and 160, distances received from a laser rangefinder, location information received from a GPS unit, motion and acceleration data received from an inertial measurement unit (IMU) 190, or a combination of data from such sensors or from other sensors that provide data useful in determining location information. In the context of augmented reality, SLAM algorithms are used to build and update a map of the environment while tracking and updating the location of the device (or user) in the mapped environment. The mathematical solution can be approximated using various statistical methods (e.g., particle filters, Kalman filters, extended Kalman filters, and covariance intersection). In a system containing a high-definition (HD) camera that captures video at a high frame rate (e.g., thirty frames per second), the SLAM algorithm updates the map and the location of the object at least as frequently as the frame rate, in other words, the map building and positioning are calculated and updated thirty times per second. The methods described herein can be used in any computing device for zooming faces, such as laptops and tablet computers, and are not limited to smart phones.

[0025] Knowing the real size of a user's face enables a virtual try-on experience like glasses, jewelry, etc. However, measuring the real size of a user's face using a single front-facing camera of a smartphone is an unsolved problem. The present disclosure estimates the real size of the face using the amount of defocus in a facial image captured using a front-facing camera. In virtual try-on and augmented reality (AR) shopping, it is important to know the real size of a product so that the user knows which size product they should buy. However, due to image scale ambiguity, it is difficult to find the correct scale of the face without knowing the depth of the face from the camera.

[0026] Currently, monocular cameras on mobile devices (such as front cameras) support adjustable focal length by adjusting the distance of the lens from the image sensor. In principle, based on the camera focal length and the lens position, the actual depth of the focal plane can be calculated, where all objects at this focal plane are in focus. The present disclosure includes two methods. The first method relies on the autofocus function in the camera. The camera automatically focuses on facial features using textures such as eyes, and then the processor of the smartphone obtains the focal length from the camera application programming interface (API) for each feature area. The second method is to capture a focal stack - a set of images captured at different focal lengths. For each facial feature, the smartphone processor selects the image with the clearest facial feature, and the corresponding focal length is the approximate depth of the facial feature. The depth can be further improved based on the interpolation of several images with similar focal lengths. To further improve the accuracy, a facial mesh is applied. The facial mesh is a solution for estimating three-dimensional (3D) facial landmarks in real time on a mobile device. It uses machine learning (ML) to infer the 3D facial surface, requiring only a single camera input without a dedicated depth sensor. For eye distance, instead of using the regular pupil distance, we calculate the distance between the left corner of the left eye and the right corner of the right eye, because eye distance is more robust to facial motion and expressions. Then, we use the eye distance as a ratio to calculate the actual size of all facial parts.

[0027] refer to Figure 2 , image 10 shows a user 12 wearing a product 14 of different sizes (shown as glasses as an example), with the product 14 overlaid on a scaled user's face 16. The CPU 120 calculates the scale of the user's face 16 using a single front-facing monocular camera 155 of the smart phone 100, and then displays the image 10 of the user 12 wearing the product 14 on the display 145.

[0028] Figure 3A and Figure 3B The image scale ambiguity is shown when a single front-facing monocular camera 155 images a user's face 16. Without knowing the distance between the camera 155 and the user's face 16, the scale of the user's face 16 is unknown. Two faces (a closer but smaller face and a farther but larger face) appear the same size in the image captured by the camera 155.

[0029] Figure 4The pipeline 400 of the CPU 120 is shown to determine the scale of the user's face 16 using a single monocular front camera 155. The CPU 120 first detects sparse features, including facial features 402 of the user's face 16, such as eyes, nose, and mouth, as shown in image 404. The CPU 120 performs depth estimation of the facial features 402 as shown in image 406, and then determines the scale of the user's face 16 by measuring the distance between the facial features 402 (e.g., the eye distance (ED) between the left edge of the left eye and the right edge of the right eye, as shown in image 408).

[0030] To measure the true proportions of the face 16, according to one example, the user 12 holds the smartphone 100 about 15-20 cm in front of the face 16 for about 1 second, with the entire face 16 within the field of view of the front camera 155 to capture the entire face. To improve accuracy, it helps if the face 16 and the front camera 155 are relatively still. During this process, the CPU 120 executes one of the methods described below (auto focus mode or manual focus mode) to determine the true proportions of the face 16. In one example, the selection of the method is automatically determined based on the smartphone model.

[0031] Auto focus mode

[0032] Figure 5 1 is a flow chart of a method 500 of auto focus mode steps for determining the true proportions of a user's face. The method 500 is performed by the CPU 120 as follows.

[0033] The CPU 120 defines a set of focusing features (denoted as F S ) to include a set of landmarks on the user's face corresponding to the left and right corners of the eyes and the left and right corners of the mouth.

[0034] In step 502, an empty set S={} of images is defined.

[0035] For one or more focusing features f s ∈F S :

[0036] The image 404 is captured using the camera 155 and two dimensional (2D) facial landmark detection is performed.

[0037] Find the value corresponding to feature f s The pixel p s .

[0038] The front camera 155 is configured to focus on pixel p s On the small window around.

[0039] After the camera 155 auto-focuses, the image is captured. s And read the focal length z of the camera s . Focal length z s Represents facial features f s Distance from camera 155.

[0040] Will (I s ,z s ) is added to S.

[0041] In step 504, the CPU 120 uses the image {I s} to jointly estimate the facial mesh and its pose in each image. Note that this canonical grid only represents the shape of the user's face and does not have the real size. Let d′ be the interocular distance between the left eye outer corner feature point and the right eye outer corner feature point on the estimated canonical grid.

[0042] In step 506, let the estimated eye distance D = {}.

[0043] For each (I s ,z s ):

[0044] For feature f s , using the fitted mesh and its pose, the canonical 3D point (X′) of the feature point is obtained in space relative to the camera 155 s ,Y′ s ,Z′ s ).

[0045] Using perspective scaling, the eye distance based on this image is d s =z s *d′ / Z′ s . s Add to D.

[0046] At step 508, the final eye distance ED estimate is a robust mean of the values ​​in D. The CPU 120 uses a mean, median, or Hodges-Lehmann estimator.

[0047] Manual focus mode

[0048] Figure 6 is a flow chart of a method 600 of manual focus mode steps performed by the CPU 120 for determining the true proportions of a user's face.

[0049] The CPU 120 determines the face ratio based on the focal stack as follows:

[0050] At step 602, a focal stack is captured using camera 155 by moving the camera lens from closest to the image sensor to farthest away. It has a set of N images with different focal lengths. For each image I, the corresponding focus depth z is also saved. In this stack, various parts of the face are in focus and then out of focus.

[0051] At step 604, for the focus stack For each image in , run 2D facial landmark detection. Register the images so that the 2D landmark features are aligned in each image. This creates a registered poly stack

[0052] At step 606, the registered focal stack Each image in J i Apply the convolution operator (eg, Laplacian operator, annular difference filter.) The response of these operators at each pixel is related to the degree of focus at that pixel and is called the focus measure.

[0053] For i = {1,…,N} Among them, * is the convolution operator.

[0054] At step 608, an empty set S = {} of images is defined.

[0055] For one or more focusing features f s ∈F S , estimate its most focused image and the corresponding focal length.

[0056] (I s ,z s )=(I i ,z i ), so that Among them, K i (s) is the feature f corresponding to image i s The value of the focus measurement at the pixel.

[0057] Will (I s ,z s ) is added to S.

[0058] In step 610, the CPU 120 uses the image {I s} to jointly estimate the facial mesh and its pose in each image. Note that this canonical grid only represents the shape of the user's face and does not have the real size. Let d′ be the interocular distance between the left eye outer corner feature point and the right eye outer corner feature point on the estimated canonical grid.

[0059] Assume that the interocular distance estimation D = {}.

[0060] For each (I s ,z s ):

[0061] For feature f s , using the fitted mesh and its pose, the canonical 3D point (X′) of the feature point is obtained in space relative to the camera 155 s ,Y′ s ,Z′ s ).

[0062] Using perspective scaling, the eye distance based on this image is d s =Z s *d′ / z′ s . s Add to D.

[0063] At step 612, the final eye distance ED estimate is the robust mean of the values ​​in D. The CPU 120 uses a mean, median, or Hodges-Lehmann estimator.

[0064] Examples as described herein may include or may operate on a processor, logic or multiple components, modules or mechanisms (herein "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in some manner. In an example, a circuit may be arranged in a specified manner (e.g., internally or relative to an external entity, such as other circuits) as a module. In an example, all or part of one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application parts or applications) to operate to perform a specified operation. In an example, the software may reside on a machine-readable medium. When executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.

[0065] Thus, the term "module" is understood to include at least one of a tangible hardware or software entity, which is an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., transiently) configured (e.g., programmed) to operate in a specified manner or to perform part or all of any of the operations described herein. Considering an example in which a module is temporarily configured, each of the modules need not be instantiated at any one time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured into corresponding different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one time and to constitute different modules at different times.

[0066] The features and flow charts described herein may be embodied as method steps in one or more methods or in one or more applications as previously described. According to some configurations, one or more "applications" are programs that perform the functions defined in the program. Different programming languages ​​may be used to generate one or more of the differently structured applications, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application hosted by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems (such as IOS TM ANDROID TM , The invention relates to mobile software running on a third-party operating system such as a mobile phone or another mobile operating system. In this example, the third-party application may invoke API calls provided by the operating system to facilitate the functionality described herein. The application may be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein may alternatively be embodied in dedicated computer hardware or an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0067] The program aspects of the present technology can be considered as "products" or "articles of manufacture", typically carried or embodied on a type of machine-readable medium in the form of at least one of executable code or associated data. For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memories of computers, processors, etc. or their related modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time. All or part of the software can sometimes communicate over the Internet or different other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another computer or processor, for example, loading software from a service provider's server system or host computer to the computer platform of the smart phone 100 or other portable electronic device. Thus, another type of medium that can carry programs, media content or metadata files includes optical waves, radio waves and electromagnetic waves, such as through wired and optical landline networks and through different air links across the physical interface between local devices. Physical elements that carry such waves (such as wired or wireless links, optical links, etc.) can also be considered as media that carry software. As used herein, unless restricted to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0068] Thus, machine-readable media can take many forms of tangible storage media. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer, etc., such as can be used to implement the client devices, media gateways, transcoders, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including the wires that comprise a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or sound or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs (compact disc read-only memory), DVDs (Digital Video Disks) or DVD-ROMs (Digital Video Disk-Read Only Memory), any other optical media, punched card stock tape, any other physical storage medium with a pattern of holes, RAM, PROMs (Programmable Read-Only Memory) and EPROMs (Erasable Programmable Read-Only Memory), FLASH-EPROMs (flash memory), any other memory chip or cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read at least one of a programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0069] It should be understood that the terms and expressions used herein, except for the specific meanings otherwise provided herein, have the general meaning of the terms and expressions corresponding to their respective fields of study. Relational terms such as first and second, etc., may be used only to distinguish one entity or action from another entity or action, without requiring or implying any actual such relationship or order between these entities or actions. The terms "comprises", "comprising", "includes", "including" or any other variation thereof are intended to cover non-exclusive inclusions, so that a process, method, article or device that includes or contains a series of elements or steps includes not only those elements or steps, but also other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. Without further limitation, an element preceded by "a" or "an" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0070] In addition, in the above detailed description, it can be seen that different features are grouped together in different examples for the purpose of simplifying the present disclosure. This method of the present disclosure should not be interpreted as reflecting the intention that these examples require the same features to be explicitly described. On the contrary, the protectable subject matter lies in the features that are less than all the features of any single disclosed example.

[0071] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in different forms and examples and that they may be applied to many applications, only some of which are described herein.

Claims

1. A device comprising: a monocular camera configured to capture an image of a face, the captured image including a plurality of facial features; a processor configured to determine a distance between the monocular camera and the face, the processor configured to: determining a plurality of facial features in the captured image; and A proportion of the face is determined based on the determined plurality of facial features.

2. The device according to claim 1, wherein: The processor is configured to determine the scale of the face using an autofocus feature of the device, wherein the autofocus feature is configured to apply a two-dimensional (2D) facial landmark detection algorithm to a focused image in the captured images to detect 2D facial landmarks and to determine an autofocus distance between the camera and the face.

3. The device according to claim 2, wherein: The processor is configured to estimate a three-dimensional 3D mesh of the face of a unit size corresponding to the detected 2D facial landmarks.

4. The device according to claim 3, wherein: The processor is configured to determine a distance between two of the plurality of facial features by combining the estimated 3D facial mesh with the autofocus distance to determine a scale of the face.

5. The device according to claim 1, wherein: The processor is configured to process a focus stack of the captured images using a manual focus feature of the device and to determine a scale of the face based on the captured images.

6. The device according to claim 5, wherein: Each of the captured images in the focus stack has a focal plane, wherein the processor is configured to process the focal planes to determine a scale of the face.

7. The device according to claim 6, wherein: The processor is configured to apply a two-dimensional (2D) facial landmark detection algorithm to a focused image in the captured images to detect 2D facial landmarks and to determine a focal distance between the camera and the face.

8. The device according to claim 7, wherein: The processor is configured to estimate a three-dimensional 3D mesh of a unit-sized face corresponding to the detected 2D facial landmarks.

9. The device according to claim 8, wherein: The processor is configured to determine a distance between two of the plurality of facial features by combining the estimated 3D facial mesh with the focal length to determine a scale of the face.

10. The device according to claim 1, wherein: The device further comprises a display, and wherein the device is configured to generate and display a product image associated with the face on the display according to the determined proportions of the face.

11. The device according to claim 10, wherein: The processor is configured to present products of different sizes on the display based on the determined proportions of the face.

12. A method for scaling a face, comprising: Capture images using a monocular camera; determining a plurality of facial features in the captured image; Determining a distance between the monocular camera and the face; as well as A proportion of the face is determined based on the determined plurality of facial features.

13. The method according to claim 12, wherein: Determining the proportions of the face includes: applying a two-dimensional 2D facial landmark detection algorithm to the in-focus image of the captured images to detect 2D facial landmarks; and An autofocus distance between the camera and the face is determined.

14. The method according to claim 13, further comprising: A 3D mesh of a unit-sized face is estimated corresponding to the detected 2D facial landmarks.

15. The method according to claim 14, wherein: Determining the proportions of the face includes: A distance between two of the plurality of facial features is determined by combining the estimated 3D facial mesh with the autofocus distance to determine a scale of the face.

16. The method according to claim 12, wherein: Determining the proportions of the face includes: processing the focal stack of the captured images using a manual focus feature; and A proportion of the face is determined based on the captured image.

17. The method according to claim 16, wherein: Each of the captured images in the focus stack has a focal plane, wherein determining the scale of the face comprises processing the focal plane to determine the scale of the face.

18. The method according to claim 17, further comprising: A 2D facial landmark detection algorithm is applied to a focused image among the captured images to detect 2D facial landmarks and determine a focal distance between the camera and the face.

19. The method according to claim 18, wherein: Determining the scale of the face includes estimating a 3D mesh of the face of a unit size corresponding to the detected 2D facial landmarks, and determining a distance between two of the plurality of facial features by combining the estimated 3D facial mesh with the focal length to determine the scale of the face.

20. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor of a device, are operable to determine a plurality of facial features in a captured image and determine a proportion of a face based on the determined plurality of facial features, the device comprising a monocular camera configured to capture an image of the face, the captured image comprising a plurality of facial features for determining a distance between the camera and the face.

Images