System and method for measuring pupillary distance and use thereof

Abstract

A method of operating a pupil distance system is disclosed. The method includes the steps of capturing a 2D image and a corresponding 3D depth map of a subject's face with at least one camera of the pupil distance system. Pupil positioning information is determined using the 2D image and the corresponding 3D depth map. The pupil positions are further refined based on the pupil positioning information. Pupil center coordinates are determined and the subject's pupil distance between the centers of each pupil is calculated. Processes and uses thereof are also disclosed.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit and priority of U.S. Serial No. 63 / 040,184, filed June 17, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to the technical field of optometry. More specifically, this disclosure relates to systems and methods for measuring pupillary distance, and their use. Background Technology

[0004] The following includes information that may be used to understand the invention. This is not an admission that any information specifically or implicitly referenced herein is prior art or essential to the described or claimed invention. All patents, patent applications, publications, and products mentioned herein are incorporated herein by reference in their entirety.

[0005] Pupillary distance (“PD”) is the distance between the pupils of a subject, such as a human. The accuracy of this distance is a determining factor in the comfort and fit of eyeglasses, sunglasses, or virtual reality headsets. Incorrectly measured PD can result in lenses with a prismatic effect, causing headaches, blurred vision, eye strain, nausea, dizziness, lightheadedness, disorientation, and other problems in the subject. Traditionally, PD measurements are not present in eyeglasses or sunglasses prescriptions but are required to fulfill custom prescription eyeglasses or sunglasses orders. The average PD for adults is approximately 63 mm. However, PD measurements can vary widely between 51 mm and 74.5 mm for women and between 53 mm and 77 mm for men. PD is critical when fitting eyeglasses or sunglasses with progressive lenses because precise lens-to-pupil alignment is required to ensure comfortable vision at all distances.

[0006] Currently, there are various methods for measuring pupillary dysplasia (PD). For example, there are two main methods for PD measurement: single PD and double PD. Single PD (or binocular PD) is the measurement from pupil to pupil between each eye. Double PD (or monocular PD) is the measurement from the bridge of your nose to each eye.

[0007] Traditional methods for obtaining PD measurements may include using a ruler. For this type of measurement, the subject stands approximately 8 inches from a mirror, and the ruler is aligned across the subject's eyebrows from one pupil to the other to obtain the PD measurement. This type of measurement is inaccurate because facial contours and ruler alignment can vary.

[0008] The subject can obtain his / her PD measurement with the assistance of another individual (possibly in a commercial setting). In this example, a corneal reflector pupil meter can be used to measure PD. Alternatively, e-commerce customers can submit a photo of their face while holding a standard-sized reference (e.g., a credit card, etc.), and the photo can be processed remotely to obtain PD.

[0009] Each of the aforementioned traditional examples represents a cumbersome and / or resource-intensive way of obtaining PD measurements. Furthermore, some traditional examples, such as the use of a corneal reflector pupillometer, may require the subject to be less than 6 feet away from the person performing the measurement, leading to violations of social distancing rules during pandemics such as COVID-19. Additionally, corneal reflector pupillometers are not always reliable measures of PD, as they are susceptible to human error and can be miscalibrated. Since PD is essential for fulfilling orders for custom-made prescription eyeglasses or sunglasses, developing seamless, convenient, and contactless methods to obtain PD measurements enhances the customer experience. Moreover, developing easier and more accurate methods to obtain PD measurements facilitates e-commerce and expands customer choices in obtaining comfortable and accurate eyeglasses. Finally, retailers can use imagery and / or statistics related to PD measurements to improve manufacturing processes and / or provide eyeglass recommendations based on user satisfaction and purchase history associated with PD measurements.

[0010] Therefore, there is a need to develop an improved system or method for measuring PD, which will improve accuracy, efficiency, reliability, convenience and use, while reducing or eliminating human intervention and / or human error. Summary of the Invention

[0011] The inventions described and claimed herein have many attributes and aspects, including, but not limited to, those set forth, described, or referenced in this content. It is not intended to be all-encompassing, and the inventions described and claimed herein are not limited to or not construed as being limited by the features or embodiments identified in this content, but are included for illustrative purposes only and not for limitation.

[0012] In various embodiments of this disclosure, systems and methods are provided for acquiring and using a series of image processing algorithms to locate pupils in two-dimensional (“2D”) images. The pupils are then located in world coordinates using three-dimensional (“3D”) data provided by depth-sensing images. Using world coordinates, the physical distance between pupils can be calculated. The systems and methods described herein advantageously provide an accurate and more convenient way to obtain PD measurements. Furthermore, the obtained PD measurements can be used with eyeglasses (or glasses), sunglasses, virtual reality headsets, goggles, safety glasses, smart glasses (including but not limited to augmented reality glasses), and other eyewear. Finally, facial measurements of a subject including PD can be associated with a user account, enabling product recommendations to be provided based on the user's facial measurements and historical customer satisfaction data and purchase history associated with those facial measurements.

[0013] In various embodiments of this disclosure, the systems and methods described herein can obtain facial measurements for use in virtual fitting or try-on systems or applications. In some embodiments, the virtual fitting or try-on system may provide a user with an interface to virtually try on and / or purchase a pair of glasses or sunglasses using facial measurements (including PD measurements) obtained from the disclosures described herein. The systems and methods described herein improve the user's e-commerce experience when incorporated into a virtual fitting or try-on system because they allow the user to try on virtual potential glasses or sunglasses, obtain PD measurements, and select and purchase custom prescription glasses or sunglasses online without visiting a physical store. The systems and methods described herein also improve the virtual fitting or try-on system by obtaining and storing additional facial measurements that can lead to better product recommendations. The systems and methods of this disclosure can advantageously interface with the virtual try-on system and methods described in U.S. Patent Application No. 16 / 550,614, filed August 26, 2019 (titled "Virtual Fitting Systems and Methods for Spectacles"). In various embodiments of this disclosure, a method for operating a pupillary distance system is described. In some embodiments, the method includes the following steps: capturing a first 2D image of a subject's face and a corresponding 3D depth map using at least one camera of the pupil distance system; using the first 2D image and the corresponding 3D depth map to determine pupil positioning information; refining one or more pupil positions based on the pupil positioning information; determining the coordinates of one or more pupil centers; and calculating the subject's PD between the centers of each pupil.

[0014] In some embodiments of this disclosure, multiple facial mesh vertices near the center of an open eye are generated using multiple facial mesh landmarks to obtain one or more initial pupil positions on the subject, thereby determining the pupil positioning information.

[0015] In some embodiments of this disclosure, convolutions with one or more kernels and one or more 2D center-around filters are used to refine the one or more pupil positions. In some embodiments, the one or more kernels use a pupil size estimate of approximately 12 mm.

[0016] In some embodiments of this disclosure, the PD is calculated using depth map values ​​corresponding to one or more pupil locations in the refined 2D image.

[0017] In some embodiments of this disclosure, the PD is determined by using points on a 3D facial mesh corresponding to the center of each pupil.

[0018] In some embodiments of this disclosure, the method further includes the step of performing correction on the PD calculated using the distance from the first 2D image captured by the at least one camera.

[0019] In various embodiments of this disclosure, a non-transitory computer-readable medium embodying computer-executable instructions is provided. When executed by a processor, the computer-executable instructions cause the processor to: acquire a first 2D image of a subject's face and a corresponding 3D depth map from at least one camera; use the first 2D image and the corresponding 3D depth map to determine pupil location information; refine the position of one or more pupils based on the pupil location information; determine the coordinates of the center of one or more pupils; and calculate the physical distribution (PD) of the subject between the centers of each pupil.

[0020] In some embodiments of this disclosure, multiple facial mesh vertices near the center of an open eye are generated using multiple facial mesh landmarks to obtain one or more initial pupil positions of the subject, thereby determining the pupil positioning information.

[0021] In some embodiments of this disclosure, convolutions with one or more kernels and one or more 2D center-around filters are used to refine the one or more pupil positions. In some embodiments, the one or more kernels use a pupil size estimate of approximately 12 mm.

[0022] In some embodiments of this disclosure, the PD is calculated using depth map values ​​corresponding to one or more pupil locations in the refined 2D image.

[0023] In some embodiments of this disclosure, the PD is determined by using points on a 3D facial mesh corresponding to the center of each pupil.

[0024] In some embodiments of this disclosure, the computer-executable instructions also cause the processor to perform correction on the PD calculated using distances from the first 2D image captured by the at least one camera.

[0025] In various embodiments of this disclosure, a pupil distance system is provided, comprising one or more mobile devices. In some embodiments, the one or more mobile devices include a mobile device comprising at least one camera, a memory storing information associated with images and information acquired from the at least one camera, and a processor. The processor is configured to: acquire a 2D image of a subject's face and a corresponding 3D depth map from the at least one camera; use the 2D image and the corresponding 3D depth map to determine pupil positioning information; refine the position of one or more pupils based on the pupil positioning information; determine the coordinates of the center of one or more pupils; and calculate the pupil distance (PD) between the centers of each pupil of the subject.

[0026] In some embodiments of this disclosure, multiple facial mesh vertices near the center of an open eye are generated using multiple facial mesh landmarks to obtain one or more initial pupil positions of the subject, thereby determining the pupil localization.

[0027] In some embodiments of this disclosure, convolutions with one or more kernels and one or more 2D center-around filters are used to refine the one or more pupil positions. In some embodiments, the one or more kernels use a pupil size estimate of approximately 12 mm.

[0028] In some embodiments of this disclosure, the PD is calculated using depth map values ​​corresponding to one or more pupil locations in the refined 2D image.

[0029] In some embodiments of this disclosure, the PD is determined by using points on a 3D facial mesh corresponding to the center of each pupil.

[0030] In some embodiments of this disclosure, correction is performed on the PD calculated using the distance from the first 2D image captured by the at least one camera. Attached Figure Description

[0031] The accompanying drawings, which are incorporated herein and form a part of this specification, illustrate some aspects of this disclosure and, together with the specification, further serve to explain the principles of these aspects and enable those skilled in the art to make and use them. The drawings are for illustrative purposes only, illustrating exemplary and non-limiting embodiments, and are not necessarily drawn to scale.

[0032] Figure 1A An example of a system according to some embodiments of the present disclosure is shown.

[0033] Figure 1B An example of the architecture of a mobile device according to some embodiments of the present disclosure is shown.

[0034] Figure 2 This is a flowchart illustrating an exemplary PD measurement process using depth maps according to some embodiments of this disclosure.

[0035] Figure 3 This is an exemplary image of a user’s pupil region overlaid with a facial mesh 905 (depicted by a plurality of lines 910 connected to a plurality of vertices 920) according to some embodiments of the present disclosure, the facial mesh showing key vertices 950 and centroids 951 of the key vertices.

[0036] Figures 4A-4C An exemplary illustrative representation of refining the initial pupil position using convolution with a 2D image, according to some embodiments of the present disclosure, is shown. Figure 4A A filter kernel (or core) with a center-surround structure is shown; Figure 4B The image patch with circle 990 marked with a cross symbol shows the location with the maximum response to the kernel; and Figure 4C The image response to a convolution with a kernel is shown.

[0037] Figure 5 This is an exemplary illustration of an iterative method for random sample consistency (“RANSAC”) on pupil region images according to some embodiments of the present disclosure. Figure 5 In the diagram, the solid circle 2000 indicates the initial estimate of the iris boundary; the solid cross symbol 2001 indicates the initial estimate of the iris center; the closed point 3010 indicates the intra-iris boundary point; the hollow point 3020 indicates the extra-iris boundary point; the dashed circle 3000 indicates the final estimate of the iris boundary; and the dashed cross symbol 3001 indicates the final estimate of the iris center.

[0038] Figure 6 This is an exemplary illustration of a process for measuring or estimating a remote PD according to some embodiments of the present disclosure.

[0039] Figure 7 This is an example of a PD computing interface 800 according to some embodiments of this disclosure. "Method A"

[0040] "Method B" refers to the depth map method, which is described further below. "Method B" refers to the face mesh method, which is also described further below.

[0041] Figure 8-11 These are exemplary interfaces 801, 802, 803, and 804 for measuring PD according to some embodiments of this disclosure.

[0042] Figure 12This is another example of a PD computing interface 805 according to some embodiments of the present disclosure. Detailed Implementation

[0043] The description of exemplary embodiments is intended to be read in conjunction with the accompanying drawings, which are considered an integral part of the entire written description. Unless otherwise specified, the use of the singular includes the plural. Unless otherwise specified, the use of "or" means "and / or". Furthermore, the use of the term "including" and other forms such as "includes" and "included" is not limiting. Additionally, unless otherwise specified, terms such as "element" or "part" cover elements and parts that include one unit as well as elements and parts that include more than one subunit. Furthermore, the section headings used herein are for organizational purposes only and should not be construed as limiting the subject matter described.

[0044] The following description is provided as an example of a representative set of enabling teachings. Many changes can be made to the embodiments described herein while still obtaining beneficial results. Some of the desired benefits discussed below can be obtained by selecting some of the features discussed herein without utilizing others. Therefore, many modifications and adaptations, as well as subsets, of the features described herein are possible and may even be desirable in some cases. Thus, the following description is provided illustratively and not restrictively.

[0045] As used in this article, the use of singular articles such as “a,” “an,” and “the” is not intended to exclude multiple objects of the article unless the context clearly and explicitly specifies otherwise.

[0046] This disclosure provides a system and method for determining the physical characteristics (PD) of a subject. The subject is a mammal or human, wherein the mammal or human is male, female, non-binary mammal or human (or other gender identity), adult, or child. As discussed throughout this specification, the system and method advantageously provide the subject with accurate and efficient real-time facial measurements. The system and method advantageously provide users with facial measurements that can be stored in an account associated with the respective user, enabling the user to receive product recommendations based on facial measurements and / or customer satisfaction data and / or purchase history associated with historical facial measurements. Such users are humans, wherein the human is male, female, non-binary (or other gender identity), adult, or child. Finally, the system and method can advantageously use customer satisfaction data to improve facial measurement calculations using artificial intelligence (such as machine learning and deep learning).

[0047] System Overview

[0048] In various embodiments, the PD measurement system can interact with client devices to exchange information. Figure 1A An example of system 100 is depicted, in which multiple client devices 110-1, 110-2, and 110-3 (collectively referred to as "client devices 110") are connected to one or more computer system networks 50-1, 50-2 ("computer networks 50") and a management server 130 via a communication network 142. The communication network 142 may be a wide area network ("WAN"), a local area network ("LAN"), a personal area network ("PAN"), etc. In one embodiment, the communication network 142 is the Internet, and the client devices 110 are online. "Online" can mean connected to or accessing source data or information from a location remote from other devices or networks connected to the communication network 142.

[0049] Management server 130 includes a processing unit 24 connected to one or more data storage units 150-1, 150-2 (collectively referred to as "Database Management System 150" or "DBMS 150"). In some embodiments, processing unit 24 is configured to provide a front-end graphical user interface ("GUI") (e.g., a PD measurement GUI 28 and a client user GUI 30) and a back-end or management graphical user interface or portal 32 to one or more remote computers 54 or one or more local computers 34. In some embodiments, a PD measurement interface, described in further detail below, is provided that accesses management server 130 via GUI 28. The GUI may take the form of a webpage displayed, for example, using a browser program native to the remote computer 54 or one or more local computers 34. It should be understood that system 100 may be implemented on one or more computers, servers, or other computing devices. In some embodiments, the GUI may be displayed on client device 110 via a software application. For example, system 100 may include an additional server programmed or partitioned based on permitted access to data stored in DBMS 150. As used herein, "portal" is not limited to general Internet portals, such as... or It also includes a GUI that is of interest to a specific, limited audience, and that provides the party with access to a variety of relevant or irrelevant information, links, and tools, as described below. The terms "webpage" and "website" are used interchangeably herein.

[0050] Remote computer 54 may be part of computer system networks 50-1, 50-2 and obtains access to communication network 142 through Internet service providers (“ISPs”) 52-1, 52-2 (“ISP 52”). As those skilled in the art will understand, client device 110 may obtain access to communication network 142 via a wireless cellular communication network, a WAN hotspot, or via a wired or wireless connection to a computer. As described below, client users and administrators may use remote computer 54 and / or client device 110 to obtain access to system 100. Computer system networks 50-1, 50-2 may include one or more data storage units 56-1, 56-2.

[0051] In one embodiment, client device 110 includes any mobile device capable of sending and receiving wireless signals. Examples of mobile devices include, but are not limited to, mobile or cellular phones, smartphones, personal digital assistants (“PDAs”), laptop computers, tablet computers, music players, and e-readers, to name just a few possible devices.

[0052] Figure 1B This is a block diagram illustrating an example of the architecture of client device 110. (As shown...) Figure 1B As shown, client device 110 includes one or more processors, such as processor(s) 102. Processor(s) 102 can be any central processing unit (“CPU”), microprocessor, microcontroller, or computing device or circuitry for executing instructions. The processor(s) are connected to communication infrastructure 104 (e.g., a communication bus, crossbar, or network). Various software embodiments are described with reference to this exemplary client device 110. After reading this specification, it will be apparent to those skilled in the art how to implement this method using client device 110, including other systems or architectures. Those skilled in the art will understand that computers 34, 54 can have an architecture similar to and / or the same as that of client device 110. In other words, computers 34, 54 can include, for example, Figure 1B Some, all, or additional functional components of the client device 110 shown.

[0053] Client device 110 includes a display 168 that displays graphics, video, text, and other data received from communication infrastructure 104 (or from a frame buffer, not shown) to a user (e.g., a subscriber, business user, backend user, or other user). Examples of such a display 168 include, but are not limited to, LCD screens, OLED displays, capacitive touchscreens, and plasma displays; only a few possible displays are listed. Client device 110 also includes main memory 108, such as random access memory (“RAM”), and may also include secondary memory 110. Secondary memory 121 may include more persistent memory, such as a hard disk drive (“HDD”) 112 and / or a removable storage drive (“RSD”) 114, representing a tape drive, optical disk drive, solid-state drive (“SSD”), etc. In some embodiments, the removable storage drive 114 reads from and / or writes to a removable storage unit (“RSU”) 116 in a manner understood by those skilled in the art. Removable storage unit 116 refers to magnetic tape, optical disc, etc., which can be read from and written to by removable storage drive 114. As will be understood by those skilled in the art, removable storage unit 116 may include a tangible and non-transitory machine-readable storage medium in which computer software and / or data are stored.

[0054] In some embodiments, auxiliary storage 110 may include other means for allowing computer programs or other instructions to be loaded into client device 110. Such means may include, for example, a removable storage unit (“RSU”) 118 and a corresponding interface (“RSP”) 120. Examples of such units 118 and interfaces 120 may include removable memory chips (such as erasable programmable read-only memory (“EPROM”), programmable read-only memory (“PROM”), secure digital storage (“SD”) cards and associated slots, as well as other removable storage units 118 and interfaces 120 that allow software and data to be transferred from removable storage unit 118 to client device 110.

[0055] Client device 110 may also include a speaker 122, an oscillator 123, a camera 124, a light-emitting diode (“LED”) 125, a microphone 126, an input device 128, an accelerometer (not shown), and a global positioning system (“GPS”) module 129. Examples of features of camera 124 include, but are not limited to, optical image stabilization (“OIS”), a larger sensor, a bright lens, 4K video, optical zoom plus raw image and HDR, a “bokeh mode” with multiple lenses, and a multi-lens night mode. Camera 124 may include one or more lenses with different functions. As an example, camera 124 may include an ultra-wide sensor, a telephoto sensor, a time-of-flight sensor, a macro sensor, a megapixel (“MP”) sensor, and / or a depth sensor. As described herein, camera 124 is not limited to a single camera. Camera 124 may include a camera system that includes various different types of cameras, sensors, etc. As an example, Published The camera system includes a 7MP front-facing "selfie" camera, an infrared emitter, an infrared camera, a proximity sensor, an ambient light sensor, a flood illuminator, and a dot projector, which work together to obtain depth maps and associated images. In other words, the camera 124 of the client device 110 may have multiple sensors, cameras, emitters, or other associated components that work as a system to obtain image information for use by the client device 110.

[0056] Examples of input device 128 include, but are not limited to, a keyboard, buttons, a trackball, or any other interface or device through which a user can input data. In some embodiments, input device 128 and display 168 are integrated into the same device. For example, display 168 and input device 128 may be a touchscreen through which a user inputs data into client device 110 using a finger, pen, and / or stylus.

[0057] Client device 110 also includes one or more communication interfaces 169 that allow software and data to be transferred between client device 110 and external devices (such as another client device 110, computers 34, 54, and other devices that can be locally or remotely connected to system 100). Examples of the one or more communication interfaces 169 may include, but are not limited to, modems, network interfaces (such as Ethernet cards or wireless cards), communication ports, PCMCIA (“PCMCIA”) slots and cards, one or more PCI (“PCI”) Express slots and cards, or any combination thereof. The one or more communication interfaces 169 may also include wireless interfaces configured for short-range communication, such as near field communication (“NFC”), Bluetooth, or other interfaces for communication via another wireless communication protocol. As briefly noted above, those skilled in the art will understand that various parts of computers 34, 54, and system 100 may include some or all of the components of client device 110.

[0058] Software and data transmitted via one or more communication interfaces 169 are in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interface 169. These signals are provided to the communication interface 169 via a communication path or channel. The channel may be implemented using wires or cables, optical fibers, telephone lines, cellular links, radio frequency (“RF”) links, or other communication channels.

[0059] In this application, the terms "non-transitory computer program medium" and "non-transitory computer-readable medium" refer to media such as removable storage units 116, 118 or hard disks installed in hard disk drive 112. These computer program products provide software to client device 110. The computer program (also referred to as "computer control logic") may be stored in main memory 108 and / or secondary memory 110. The computer program may also be received via one or more communication interfaces 169. When executed by processor(s) 102(s), such a computer program enables client device 110 to perform the features of the methods and systems discussed herein.

[0060] In various embodiments, such as Figure 1A and 1BAs shown, client device 110 may include computing devices such as hash computers, personal computers, laptop computers, tablet computers, notebook computers, handheld computers, personal digital assistants, portable navigation devices, mobile phones, smartphones, wearable computing devices (e.g., smartwatches, wearable activity monitors, wearable smart jewelry and glasses, and other optical devices including optical head-mounted displays (“OHMDs”), embedded computing devices (e.g., communicating with smart textiles or electronic fabrics), or any other suitable computing device configured to store data and software instructions, execute software instructions to perform operations, and / or display information on a display device. Client device 110 may be associated with one or more users (not shown). For example, a user operates client device 110 to perform one or more operations according to various embodiments.

[0061] Client device 110 includes one or more tangible, non-transitory memories for storing data and / or software instructions, and one or more processors configured to execute the software instructions. Client device 110 may include one or more display devices for displaying information to a user and one or more input devices (e.g., keypad, keyboard, touchscreen, voice-activated control technology, or any other suitable type of known input device) allowing the user to input information into the client device. The processor(s) of client device 110 may be any central processing unit (“CPU”), microprocessor, microcontroller, or computing device or circuitry for executing instructions. The processor(s) are connected to a communication infrastructure (e.g., a communication bus, crossbar, or network). Various software embodiments have been described with reference to this exemplary client device 110. After reading this specification, it will be apparent to those skilled in the art how to implement this method using client device 110, including other systems or architectures. Those skilled in the art will understand that a computer may have an architecture similar to and / or the same as that of client device 110. In other words, a computer may include, for example, Figure 1A and 1B Some, all, or additional functional components of the client device 110 shown.

[0062] Client device 110 also includes one or more communication interfaces 169 that allow software and data to be transferred between client device 110 and external devices (e.g., another client device 110 and other devices that can be connected locally or remotely to client device 110). Examples of the one or more communication interfaces may include, but are not limited to, modems, network interfaces (e.g., communication interface 169, such as Ethernet cards or wireless cards), communication ports, PCMCIA slots and cards, one or more PCI Express slots and cards, or any combination thereof. The one or more communication interfaces 169 may also include wireless interfaces configured for short-range communication, such as NFC, Bluetooth, or other interfaces for communication via another wireless communication protocol.

[0063] Software and data transmitted via one or more communication interfaces 169 are in the form of signals, which may be electronic, electromagnetic, optical, or other signals that can be received by the communication interfaces. These signals are provided to the communication interface 169 via a communication path or channel. The channel may be implemented using wires or cables, optical fibers, telephone lines, cellular links, radio frequency (“RF”) links, or other communication channels.

[0064] In embodiments where system 100 or the method is implemented partially or entirely in software, the software may be stored in a computer program product and loaded into client device 110 using a removable storage drive, hard disk drive, and / or communication interface. When executed by one or more processors, the software causes the processors to perform the functions of the methods described herein. In another embodiment, the method is implemented in hardware primarily using hardware components such as application-specific integrated circuits (“ASICs”). Those skilled in the art will understand that a hardware state machine is implemented to perform the functions described herein. In yet another embodiment, the method is implemented using a combination of hardware and software.

[0065] Embodiments of the subject matter described in this specification can be implemented in system 100, which includes backend components (e.g., as a data server), or middleware components (e.g., an application server), or frontend components (e.g., a client device 110) having a graphical user interface or a web browser through which a user can interact with the implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. Components of the system can be interconnected via any form or medium of digital data communication (e.g., communication network 142). Communication network 142 can include one or more communication networks or media for digital data communication. Examples of communication network 142 include local area networks (“LANs”), wireless LANs, RF networks, NFC networks (e.g., “WiFi” networks), wireless metropolitan area networks (“MANs”) connecting multiple wireless LANs, one or more NFC communication links, and wide area networks (“WANs”) (e.g., the Internet), and combinations thereof. According to various embodiments of this disclosure, communication network 142 may include the Internet and any one or more publicly accessible networks interconnected via one or more communication protocols, including but not limited to Hypertext Transfer Protocol (“HTTP”) and Hypertext Transfer Protocol Security (“HTTPS”) and Secure Sockets Layer / Transport Layer Security (“SSL / TLS”) and Transmission Control Protocol / Internet Protocol (“TCP / IP”). Communication protocols according to various embodiments also include protocols that facilitate data transmission using radio frequency identification (“RFID”) communication and / or NFC. Furthermore, communication network 142 may also include one or more mobile device networks, such as GSM or LTE networks or PCS networks, allowing client devices to send and receive data via applicable communication protocols, including those described herein. For ease of illustration, communication network 142 is shown as an extension of management server 130.

[0066] Client device 110 and server 130 are typically geographically separated and interact via communication network 142. The relationship between client device 110 and management server 130 is established by means of computer programs running on respective system components and having a client-server relationship with each other. In an embodiment, system 100 may include a web / application server (not shown) for obtaining access to a number of services provided by management server 130.

[0067] In one aspect, client device 110 stores in memory one or more software applications that run on the client device and are executed by one or more processors. In some cases, according to various embodiments, each client device stores a software application that, when executed by one or more processors, performs operations to establish communication with management server 130 (e.g., across communication network 142 via communication interface 169) and to obtain information or data from management server 130 via database management system 150.

[0068] In various embodiments, client device 110 may execute one or more stored software applications to interact with management server 130 via a network connection. The executed software applications may enable client device 110 to transmit information (e.g., facial measurements (e.g., PD), user profile information, etc.). As described below, the executed software applications may be configured to allow a user associated with client device 110 to obtain PD measurements using camera 124. As those skilled in the art will understand, the software applications stored on client device 110 may be configured to access the Internet or other suitable web-based communications capable of interacting with communications network 142. For example, a user may access a user account on management server 130 via an Internet webpage. In this example, management server 130 is configured to present Internet webpages to the user on client device 110. Alternatively, management server 130 may provide information to one or more software applications stored on client device 110 via communications network 142. In this example, client device 110 will use the graphical user interface display of the stored software applications to display the information provided by management server 130. In the example above, as will be understood by those skilled in the art and described below, the corresponding user account may be associated with the developer, client user, or supervisor / monitoring authority.

[0069] According to various embodiments, system 100 includes a database management system / storage device 150 for managing and storing data, such as facial measurement information (e.g., PD, etc.), user account authentication information, and other data maintained by management server 130. For convenience, the database management system and / or storage device are simply referred to herein as DBMS 150. DBMS 150 can be communicatively coupled to various modules and engines (not shown).

[0070] It should be understood that various forms of data storage or repositories accessible by a computing system can be used in System 100, such as hard disk drives, tape drives, flash memory, random access memory, read-only memory, EEPROM storage devices, in-memory databases such as SAP HANA, and any combination thereof. The stored data can be formatted within the data storage in one or more formats, such as flat text file storage, relational databases, non-relational databases, XML, comma-separated values, Microsoft Excel files, or any other format known to those skilled in the art, and any combination thereof suitable for a particular use. The data storage can provide access to the stored data in various forms, such as through file system access, network access, SQL protocols (e.g., ODBC), HTTP, FTP, NES, CIFS, and any combination thereof.

[0071] According to various embodiments, client device 110 is configured to access DBMS 150 via management server 130. In various embodiments, DBMS 150 is configured to maintain a database schema. For example, the database schema may be arranged to maintain identifiers in columns within DBMS 150 associated with facial measurements or user account information. In this regard, identifiers refer to specific information related to the aforementioned categories. The database schema within DBMS 150 can be arranged or organized in any suitable manner within the system. Although the above examples identify category identifiers, any number of suitable identifiers can be used to maintain records associated with the system described herein. Additionally, the database schema may include additional categories and identifiers not described above for maintaining record data in system 100. The database may also provide statistical and marketing information associated with users of system 100.

[0072] The database schema described above advantageously organizes identifiers in a manner that allows the system to operate more efficiently. In some embodiments, the categories of identifiers in the database schema are improved by grouping identifiers using an association management model of the management server 130.

[0073] In various embodiments, management server 130 includes computing components configured to store, maintain, and generate data and software instructions. For example, management server 130 may include or access one or more processors 24, one or more servers (not shown), and tangible, non-transitory storage devices (e.g., local data storage (in addition to DBMS 150)) and / or additional data storage for storing software or code to be executed. Servers may include one or more computing devices configured to execute software instructions stored thereon to perform one or more processes according to various embodiments. In some embodiments, DBMS 150 includes a server that executes software instructions to perform operations that provide information to at least one other component of computing environment 100, such as providing data to another data storage device or a third-party recipient (e.g., a banking system, a third-party vendor, an information collection agency, etc.) via a network such as communications network 142.

[0074] Management server 130 may be configured to provide one or more websites, digital portals, or any other suitable services configured to perform various functions of management server 130 components. In some embodiments, management server 130 maintains an application programming interface (“API”) through which one or more applications executed by client device 110 can access the functions and services provided by server 130. In various embodiments, management server 130 may provide information to one or more software applications on client device 110 for display on graphical user interface 168.

[0075] In some embodiments, management server 130 provides information to client device 110 (e.g., via an API associated with an executed application). Client device 110 presents multiple portions of the information to the corresponding user through a corresponding graphical user interface 168 or webpage.

[0076] In various embodiments, management server 130 is configured to provide or receive information associated with services provided by management server 110 to client device 110. For example, client device 110 may receive information via communication network 142 and store multiple portions of the information in locally accessible storage and / or network-accessible storage and data storage (e.g., cloud-based storage). For example, client device 110 executes stored instructions (e.g., applications, web browsers, and / or mobile applications) to process multiple portions of the stored data and render multiple portions of the stored data to present to the corresponding user, or user management server 130 may include additional servers (not shown) that may be incorporated as corresponding nodes in a distributed network or as corresponding networked servers in a cloud computing environment. Furthermore, the server may communicate with one or more additional servers (not shown) via communication network 142, which can facilitate the distribution of processes executed in parallel by the additional servers.

[0077] PD process management

[0078] In various embodiments of this disclosure, a process for measuring or estimating the PD of a user on client device 110 is provided. Reference is now made to... Figure 2 This document illustrates a flowchart of an exemplary PD measurement process 200 using depth maps according to some embodiments of the present disclosure. Initially, in step 202, the user captures an image on the client device 110 via at least one camera (such as camera 124). In some embodiments, instructions for using the PD system are provided to the user before using the PD system (e.g., as shown in the image). Figure 9 (See instruction 900 shown). In some embodiments, feedback is provided to the user regarding whether his / her head is properly positioned for accurate measurement (i.e., whether the head is in position for accurate image capture). This feedback, conveyed in, for example, text, verbal, and / or symbolic formats, guides the user to move his / her head and / or client device 110 in various directions, such as further, closer, up, down, right, left, diagonally, clockwise, or counterclockwise, and / or to view a point or area on or off the screen of client device 110. The feedback may include one or more indicators on the screen of client device 110 that the user can follow with his / her head and / or eyes. As the user follows the feedback, the user eventually positions his / her head for accurate measurement. As an example, Figure 10 and 11An exemplary indicator 1000 is shown, which has textual guidance instructing the user to "Focus on this dot. When it moves, follow it." As the indicator moves from a starting position to an ending position on the screen of the client device 110, the user will move his / her head and / or eyes according to the indicator. Additional feedback can be conveyed to the user before image capture, such as... Figure 10 The "Move farther away" gesture, as shown, refers to further moving the user's head and / or eyes and / or the client device 110 equipped with at least one camera (such as camera 124). Once the user's head is properly positioned for accurate measurement, the user captures an image on the client device 110 via at least one camera (such as camera 124). The aforementioned feedback can occur in conjunction with step 202. According to various embodiments, the captured image is a 2D image and a corresponding depth map. In some embodiments, using... The camera acquires this depth map. As an example, The camera emits an array of infrared dots (e.g., 30,000 dots) in a known pattern onto the subject, and the infrared sensor records how the dots interact with the scene, and generates a 3D structure based on that information. The camera includes a proximity sensor for activation and an ambient light sensor for setting the output light level. 3D information can be used to determine the physical dimensions of the subject being photographed. Although disclosed in conjunction with this embodiment... A camera is used, but those skilled in the art will understand that various other depth-sensing cameras can be used to perform the procedures in the systems and methods described herein (e.g., The Point Cloud DepthCamera). TM Intel Depth camera, Depth+Multi-Spectral Camera TM wait).

[0079] In various embodiments, a 2D image and a corresponding 3D depth map can be obtained from a storage unit on the client device 110. In some embodiments, the 2D image and corresponding 3D depth map information can be received by the client device 110 to perform the processes described herein. It should be understood that the 2D image and corresponding 3D depth map can be obtained by different devices and / or generated in a manner not simultaneous with the steps described herein.

[0080] In various embodiments, in step 203, multiple facial mesh landmarks are used to obtain a coarse pupil location. According to various embodiments, the 3D information identified in step 202 is used to generate coordinates of facial mesh vertices near the center of the open eye (e.g., 0-5 mm from the pupil) to obtain an approximate pupil position. As an example, vertices 1095, 1094, 1107, and 1108 are near the right pupil, and vertices 1076, 1075, 1062, and 1063 are near the left pupil. Those skilled in the art will understand that the vertices identified in the above examples may differ from the actual vertices determined and will vary based on coordinates generated for the individual. Figure 3 This is an exemplary image of a user's pupil region overlaid with a facial mesh 905 (depicted by multiple lines 910 connected to multiple vertices 920) according to some embodiments of this disclosure. The facial mesh shows key vertices 950 and centroids 951 of the key vertices. In some embodiments, for each pupil, the 3D centroid of the vertex corresponding to each pupil is calculated to determine the corresponding 2D image coordinates of the centroid. In various embodiments, through... The ProjectPoint(_:) method or other methods that project one or more points from the scene's 3D world coordinate system to the renderer's 2D pixel coordinate system perform this calculation. Those skilled in the art will appreciate that the calculation of the corresponding 2D image coordinates for determining the 3D centroid can be performed by any suitable application or system.

[0081] In step 204, according to various embodiments, convolution with a 2D center-around filter is used to refine the initial pupil position obtained in step 203. In this context, convolution can be considered a method for finding a region in an image that best matches the appearance of a test pattern called a kernel. In some embodiments, a kernel with a dark circle surrounded by a white ring is used to find kernel-like regions in the image. In various embodiments, if the pattern to be located has an unknown size, the convolution method can be run multiple times with kernels of different sizes. In some embodiments, pupil size estimation (e.g., approximately 12 mm in diameter, approximately = + / - 2 mm) can be used to determine the approximate distance of the eye from the camera using an associated facial mesh. Using the pupil size estimation and the following formula, the size of the iris that will appear in the image can be predicted:

[0082] The pixel diameter D of the iris is given by the following:

[0083]

[0084] Where f is the focal length of the 2D camera, z is the distance between the camera and the pupil, and d is the diameter of the iris, which is assumed to be approximately 12 mm.

[0085] According to various embodiments, the kernel is constructed as follows: it has a value of The central circular region of diameter D and its magnitude A circular region with a diameter of 1.5D is defined. In this example, the magnitude is chosen such that the kernel will produce a zero response on a uniform image. Convolution is applied to a 3D×3D region around the initial estimate, and the location within the region that gives the strongest response is considered as the refined estimate.

[0086] Figures 4(a)-4(c) are exemplary illustrative representations of refining the initial pupil position using convolution of a 2D image with step 204 according to some embodiments of the present disclosure, wherein... Figure 4A A filter kernel (or core) with a center-surround structure is shown; Figure 4B The image patch with circle 990 marked with a cross symbol shows the location with the maximum response to the kernel; and Figure 4C The image response to a convolution with a kernel is shown.

[0087] In step 205, during the final pupil position refinement step, the precise boundary of the iris is estimated, and a robust fit of the boundary is performed using circles. According to various embodiments, a 1.5D pixel-wide region centered on the result of the previous localization step at point 204 is considered. For each region, a horizontal gradient kernel is applied to the left and right halves of the row. In the left half of the row, a [+1, 0, -1] kernel is applied, which elicits a strong response to the light-to-dark transition. In the right half of the row, a [-1, 0, +1] kernel is applied, which elicits a strong response to the dark-to-light transition. For each half-row, the column with the strongest response is considered a candidate boundary point. To illustrate the fact that the entire iris is not visible and that the top of the iris is generally more obscured than the bottom, only rows corresponding to less than 30° upwards from the center of the region and less than 45° downwards from the center of the region are considered.

[0088] Figure 5 This is an exemplary illustration of a RANSAC iterative method on a pupil region image according to some embodiments of this disclosure. Figure 5In the diagram, solid circle 2000 indicates the initial estimate of the iris boundary; solid cross symbol 2001 indicates the initial estimate of the iris center; closed point 3010 indicates an intra-iris boundary point; hollow point 3020 indicates an extra-iris boundary point; dashed circle 3000 indicates the final estimate of the iris boundary; and dashed cross symbol 3001 indicates the final estimate of the iris center. According to various embodiments and continuing to step 205, once candidate boundary points are identified, it is necessary to determine the circle that best fits these points. Because false boundary points often exist due to corneal reflection or other irregularities, it is necessary to find a way to use only true edge points while eliminating false positives. In various embodiments, this is achieved using the RANSAC technique. According to some embodiments, RANSAC works by fitting a model to a randomly selected subset of points and selecting the subset that gives the lowest error. Points that cannot be well fitted by the final circle model are considered extra-iris points, and the remaining points are considered "intra-iris points."

[0089] In step 206 (collectively referred to as 206a and 206b), the coordinates of the pupil centers are correlated with a 3D coordinate system to calculate the physical distance between the pupil centers. In various embodiments, the physical distance between the pupil centers can be calculated in step 206a using a depth map method. In this embodiment, camera calibration is used for image frames provided by software to obtain the z-coordinate. In various embodiments, the software may be Apple iOS. TM Alternatively, similar operating software may be used for the client device 110. In this embodiment, a value corresponding to the refined position of the pupil in the 2D image can be determined in the depth map.

[0090] Alternatively, according to various embodiments, in step 206b, a program can be used to determine a point on the 3D facial mesh corresponding to the center of the pupil. In this embodiment, light rays are emitted corresponding to image coordinates, and the 3D world coordinates of the point where the light rays intersect with the facial mesh are returned.

[0091] Referring to step 206 (collectively referred to as 206a and 206b), according to various embodiments, using Apple iOS TM The provided transformation allows these 3D pupil points to be represented in a facial center coordinate system, where x = 0 corresponds to the facial midline and the x-axis corresponds to the horizontal axis of the head. The binocular PD is then the x-distance between pupil positions, and the monocular PD is the x-distance between the pupil position and zero.

[0092] Figure 6This is an exemplary illustration of a process for measuring or estimating distance PD according to some embodiments of this disclosure. In step 207, PD correction is performed based on the distance from which the image is captured from camera 124. For example, a client device 110 (such as a mobile device) with camera 124 is held at arm's length, which corresponds to approximately 400 mm. If an individual is looking at the screen / display 168 of client device 110, their PD will be lower than their PD when looking at a distance, which is the value most relevant to accurate PD determination. Typically, near PD is converted to far PD by adding a fixed value. However, in this example, the distance from the subject's gaze point is known, and a more principled correction can be performed. In some embodiments, the lateral radius of the eyeball can be assumed to be, for example, 12 mm. The eyeball center coordinates can then be calculated by extending the light rays defined by the camera coordinates and the pupil center coordinates by an additional 12 mm. In some embodiments, the lateral radius of the eyeball is in the range of 10 mm to 14 mm, or any intermediate value within that range. In some embodiments, the lateral radius of the eyeball is 10 mm; 11 mm; 12 mm; 13 mm; or 14 mm. In some embodiments, the lateral radius of the eyeball is approximately 12 mm (where "approximately" means + / - 2 mm). The process is in... Figure 6 As shown in the image.

[0093] In various embodiments, it may be desirable to aggregate multiple measurements to discard potential outliers. Using a median binocular PD measurement is a straightforward way to do this. For monocular measurements, taking the median of the Oculus Sinister (left eye) (“OS”) and Oculus Dexter (right eye) (“OD”) values ​​would likely prevent them from adding up to a median binocular value. More suitable is to determine the proportion of binocular PD assigned to the OS and OD PD values, calculate the median proportion, and then multiply it by the median binocular PD value. According to various embodiments, PD measurements are within 0.5 mm of the pupillometer measurement. In some embodiments, PD measurements are within 1.0 mm of the pupillometer measurement.

[0094] Figure 7 These are examples of a PD calculation interface 800 according to some embodiments of this disclosure. "Method A" refers to the depth map method, which is described herein. "Method B" refers to the face mesh method, which is also described herein.

[0095] Figure 8-11 These are exemplary interfaces 801, 802, 803, and 804 for measuring PD according to some embodiments of this disclosure, and... Figure 12Another example of a PD computing interface 805 according to some embodiments of the present disclosure is shown. Such an interface can be seen on the screen / display 168 of the client device 110 and may also include one or more hyperlinks, web page links, image links and / or service buttons (e.g., "Add to your order" button 1010 or other purchase-related buttons) to guide the user to view, select and / or purchase products that require PD information, such as glasses.

[0096] In various embodiments of this disclosure, system 100 may be configured to store facial measurement data (e.g., PD, etc.) of a specific user of client device 110. In various embodiments, the facial measurement data of a given user may be associated with a user account. In some embodiments, client device 110 may transmit facial measurement data to management server 130 for storage in a DBMS 150 associated with the user account. In various embodiments, facial measurement data may be aggregated with customer satisfaction scores or ratings to improve the facial measurement process. In some embodiments, product recommendations are provided to users of client device 110 based on facial measurement data associated with customer satisfaction scores, purchase history, or other identifying characteristics such as the size, thickness, and / or dimensions of a particular product.

[0097] This disclosure may be embodied in the form of methods and apparatus for practicing those methods. This disclosure may also be embodied in the form of program code in a tangible medium, such as a Security Digital (“SD”) card, USB flash drive, disk, CD-ROM, DVD-ROM, Blu-ray disc, hard disk drive, or any other non-transitory machine-readable storage medium, wherein the machine becomes an apparatus for practicing this disclosure when the program code is loaded into and executed by the machine. This disclosure may also be embodied in the form of program code, for example, whether stored in a storage medium, loaded into and / or executed by a machine, or transmitted via some transmission medium, such as by wire or cable, by optical fiber, or via electromagnetic radiation, wherein the machine becomes an apparatus for practicing this disclosure when the program code is loaded into and executed by the machine (e.g., a computer). When implemented on a general-purpose processor, the program code segments are combined with the processor to provide a unique means of operation similar to that of a particular logic circuit.

[0098] It should be emphasized that the above embodiments are merely possible examples of implementation and only illustrate a clear understanding of the principles of this disclosure. Many variations and modifications can be made to the above embodiments of this disclosure without substantially departing from the spirit and principles of this disclosure. All such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims.

[0099] Although this specification contains numerous details, these should not be construed as limiting the scope of any disclosure or potentially claimed content, but rather as descriptions of features that may be specific to particular embodiments of a particular disclosure. Some features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may involve sub-combinations or variations of sub-combinations.

[0100] Although various embodiments have been described, it should be understood that the described embodiments are merely illustrative, and the scope of this subject matter should be accorded to the full range of equivalents, many variations and modifications that would naturally occur to those skilled in the art upon reading it.

Claims

1. A method of operating a pupillary distance ("PD") system, the method comprising the steps of: capturing, with at least one camera of the pupillary distance system, a first 2D image and a corresponding 3D depth map of a face of a subject; determining pupillary positioning information using the first 2D image and corresponding 3D depth map; refining one or more pupil locations based on the pupillary positioning information using convolution with one or more kernels and one or more 2D center-surround filters to find one or more pupil regions; identifying a set of boundary points for each pupil region; determining a respective iris boundary circle for each set of boundary points based on in-circle boundary points; determining one or more pupil center coordinates as centers of the respective iris boundary circles; and calculating a pupillary distance of the subject between centers of each pupil. determining the pupillary positioning information using a plurality of face mesh landmarks to generate a plurality of face mesh vertices near centers of the open eyes to obtain one or more initial pupil locations on the subject.

2. The method of claim 1, wherein, the one or more kernels use a pupil estimate size of approximately 12 mm.

3. The method of claim 1, wherein, the calculating the pupillary distance uses depth map values corresponding to the refined one or more pupil locations in the 2D image.

4. The method of claim 1, wherein, the pupillary distance is determined by using points on a 3D face mesh corresponding to centers of each pupil.

5. The method of claim 1, wherein, 6. The method of claim 1, further comprising the steps of: performing a correction to the pupillary distance calculated using a distance from which the first 2D image was taken by the at least one camera. when executed by a processor, the computer executable instructions cause the processor to:

7. A non-transitory computer-readable medium having computer-executable instructions embodied thereon, wherein, obtain, from at least one camera, a first 2D image and a corresponding 3D depth map of a face of a subject; determine pupillary positioning information using the first 2D image and corresponding 3D depth map; refine one or more pupil locations based on the pupillary positioning information using convolution with one or more kernels and one or more 2D center-surround filters to find one or more pupil regions; identify a set of boundary points for each pupil region; determine a respective iris boundary circle for each set of boundary points based on in-circle boundary points; determine one or more pupil center coordinates as centers of the respective iris boundary circles; and calculate a pupillary distance of the subject between centers of each pupil. determine the pupillary positioning information using a plurality of face mesh landmarks to generate a plurality of face mesh vertices near centers of the open eyes to obtain one or more initial pupil locations on the subject. the one or more kernels use a pupil estimate size of approximately 12 mm.

8. The non-transitory computer-readable medium of claim 7, wherein, the calculating the pupillary distance uses depth map values corresponding to the refined one or more pupil locations in the 2D image.

9. The non-transitory computer-readable medium of claim 7, wherein, the pupillary distance is determined by using points on a 3D face mesh corresponding to centers of each pupil.

10. The non-transitory computer-readable medium of claim 7, wherein, the computer executable instructions further cause the processor to perform a correction to the pupillary distance calculated using a distance from which the first 2D image was taken by the at least one camera.

11. The non-transitory computer-readable medium of claim 7, wherein, 13. A pupillary distance ("PD") system, comprising:

12. The non-transitory computer-readable medium of claim 7, wherein, a mobile device, the mobile device comprising: at least one camera; ​ ​ a memory storing information associated with images and information obtained from the at least one camera; and a processor configured to: obtain a 2D image and a corresponding 3D depth map of a subject's face from the at least one camera; determine pupil localization information using the 2D image and corresponding 3D depth map; refine one or more pupil locations based on the pupil localization information using convolution with one or more kernels and one or more 3D center-surround filters to find one or more pupil regions; identify a set of boundary points for each pupil region; determine a respective iris boundary circle based on the inlier boundary points; determine one or more pupil center coordinates as the center of the respective iris boundary circle; and calculate a pupil distance between the center of each pupil of the subject.

14. The pupillary distance system of claim 13, wherein, determine pupil localization using a plurality of face mesh landmarks to generate a plurality of face mesh vertices near the center of the open eye to obtain one or more initial pupil locations of the subject.

15. The pupillary distance system of claim 13, wherein, The one or more kernels use a pupil estimation size of about 12 mm.

16. The pupillary distance system of claim 13, wherein, The calculation of the pupil distance uses depth map values corresponding to the refined one or more pupil locations in the 2D image.

17. The pupillary distance system of claim 13, wherein, The pupil distance is determined by using points on a 3D face mesh that correspond to the center of each pupil. The one or more kernels use a pupil estimation size of about 12 mm. The calculation of the pupil distance uses depth map values corresponding to the refined one or more pupil locations in the 2D image. The pupil distance is determined by using points on a 3D face mesh that correspond to the center of each pupil.

Images