Computer-implemented method, non-transitory computer-readable medium, and system

Abstract

A system and method are provided for fitting a head-mounted wearable device to a user based on a single two-dimensional image. The image may include a user's face / head captured by an image sensor of a computing device via an application running on the computing device. Therion nodes of a plurality of nodes of a reference mesh may be mapped to therion nodes of a plurality of nodes of a user mesh. The reference mesh may represent a generic head mesh based on data collected from numerous users. The user mesh may be generated from the two-dimensional image. The positioning of a virtual frame on the two-dimensional image of the user may be adjusted based on a difference in position between therion nodes of the reference mesh and therion nodes of the user mesh.

Description

[Technical Field]

[0001] This description relates generally to sizing and / or fitting wearable devices, and more particularly to sizing and / or fitting head-mounted wearable devices. [Background technology]

[0002] Wearable devices may include, for example, head-mounted wearable devices, wrist-worn wearable devices, hand-worn wearable devices, pendants, fitness trackers, body sensors, and other such devices. Head-mounted wearable devices may include, for example, smart glasses, headsets, goggles, earphones, etc. Wrist / hand-worn wearable devices may include, for example, smart watches, smart bracelets, smart rings, etc. In some situations, a user may want to select and / or customize a wearable device for fit and / or functionality. For example, a user may desire to select and / or customize eyewear to include frame selection, the incorporation of prescription lenses, and other such features. Summary of the Invention

[0003] Described herein are systems and methods for selecting, sizing, and / or fitting a head-mounted wearable device based on a two-dimensional image of a user captured via an application running on a computing device operated by the user. A user mesh is generated to represent a portion of the head, such as the user's face, based on one or more facial landmarks detected in the user's image. A virtual frame is positioned on the reference mesh, for example, at a position corresponding to a therion of the reference mesh. The reference mesh may be generated based on data collected from a relatively large number of subjects and represent a generic face. A rigid transformation may be performed to project the reference mesh and the virtual frame onto the user mesh. The position of the virtual frame may be shifted or adjusted so that a bridge portion of the virtual frame is positioned corresponding to the therion of the user mesh, thereby positioning the virtual frame as a corresponding physical frame that the user would likely wear. In general, the user mesh and / or the reference mesh may represent the user's head.

[0004] The proposed solution particularly relates to a (computer-implemented) method, in particular for partially or fully automated selection, sizing, and / or fitting of a head-mounted wearable device to a user's specific requirements, the method comprising: capturing, via an application running on a computing device operated by the user, image data of an initial image including the user's face; generating a user mesh representing the user's face based on the image data; identifying an index node in the user mesh corresponding to a converged portion of the user's face captured in the image data; and identifying an index node in a reference mesh, wherein the index node of the reference mesh corresponds to a converged portion of the reference mesh, and the converged portion of the reference mesh corresponds to a converged portion of the user mesh; the method further comprises: positioning a virtual frame of the head-mounted wearable device on the reference mesh at a position corresponding to the index node of the reference mesh; projecting the reference mesh and the virtual frame onto the user mesh; and adjusting the position of the virtual frame to correspond to the index node of the user mesh. Based on the virtual frame whose position has been adjusted with respect to the user mesh, components of the head-mounted wearable device and / or a model of the head-mounted wearable device are selected or manufactured for the user for whom the user mesh was generated. For example, the user may be presented with an image including a virtual rendering of a frame positioned on the user's face from an initial image captured by the user. The rendering of the frame on the user's face may represent the actual fit of the user's face and the corresponding physical frame on their head, allowing the user to make a relatively accurate assessment of the fit and appearance of the frame, which may facilitate and / or accelerate partially or fully automated selection, sizing, and / or fitting of a head-mounted wearable device to the user's specific requirements, particularly the user's specific facial characteristics, based on the two-dimensional image of the user.

[0005] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0006] [Figure 1A] FIG. 1A shows an exemplary head-mounted wearable device worn by a user. [Figure 1B] FIG. 1B is a front view of the exemplary head-mounted wearable device shown in FIG. 1A. [Figure 1C] FIG. 1C is a rear view of the exemplary head-mounted wearable device shown in FIG. 1A. [Figure 2A] FIG. 2A shows exemplary ophthalmic fit measurements. [Figure 2B] FIG. 2B shows exemplary ophthalmic fit measurements. [Figure 2C] FIG. 2C shows exemplary ophthalmic fit measurements. [Figure 3] FIG. 3 is a block diagram of a system according to embodiments described herein. [Figure 4A] FIG. 4A is a front view of a user showing exemplary facial and / or cranial landmarks. [Figure 4B] FIG. 4B is a front view of a user showing exemplary facial and / or cranial landmarks. [Figure 4C] FIG. 4C is a side view of a user showing exemplary facial and / or cranial landmarks. [Figure 5A] FIG. 5A illustrates a process for sizing and / or fitting a frame of a head-mounted wearable device according to embodiments described herein. [Figure 5B] FIG. 5B illustrates a process for sizing and / or fitting a frame of a head-mounted wearable device according to embodiments described herein. [Figure 5C]FIG. 5C illustrates a process for sizing and / or fitting a frame of a head-mounted wearable device according to embodiments described herein. [Figure 5D] FIG. 5D illustrates a process for sizing and / or fitting a frame of a head-mounted wearable device according to embodiments described herein. [Figure 5E] FIG. 5E illustrates a process for sizing and / or fitting a frame of a head-mounted wearable device according to embodiments described herein. [Figure 6] FIG. 6 is an exemplary sizing and / or fitting image according to embodiments described herein. [Figure 7] FIG. 7 is a flowchart of an exemplary method according to embodiments described herein. DETAILED DESCRIPTION OF THE INVENTION

[0007] The selection of a wearable device, such as a head-mounted wearable device in the form of eyewear or eyeglasses, can depend on determining a physical or wearable fit to ensure that the eyewear is comfortable when worn by the user and / or aesthetically complementary to the user. The incorporation of corrective lenses into a head-mounted wearable device can depend on determining an ophthalmic fit to ensure that the head-mounted wearable device can provide the desired vision correction. For example, in the case of a head-mounted wearable computing device in the form of smart glasses that include computing / processing and display capabilities, the selection can also depend on determining a display fit to ensure that the visual content is visible to the user. Existing systems for obtaining these types of wearable devices do not provide accurate fitting and customization, especially without access to a retail facility. That is, accurate sizing and / or fitting often depend on the user having access to a retail facility where samples are available for physical try-ons and where an optical technician can facilitate a wearable fit and / or ophthalmic fit and / or aesthetic fit determination based on the physical try-ons and measurements collected using specialized equipment. In some situations, existing virtual systems that provide online selection of wearable devices such as eyewear or eyeglasses may simply superimpose an image of the selected frame onto an image of the user. The virtual placement of the image of the selected frame on the image of the user does not take into account the user's facial features, which may affect the fitting of the physical frame to the user. For example, variations in the height of the bridge of the nose may affect how the physical frame is positioned on the user's face / head, thus affecting the fit and functionality of the head-mounted wearable device when worn by the user. Therefore, these types of systems may produce inaccurate results in selecting eyewear in this manner.

[0008] According to embodiments described herein, systems and methods provide virtual fitting of a wearable device based on one or more features detected in the image data. The systems and methods according to embodiments described herein utilize a reference mesh, or canonical mesh, which may be a three-dimensional representation of the body part on which the wearable device will be worn. For example, in selecting, sizing, and / or fitting a head-mounted wearable device, the reference mesh, or average mesh, or canonical mesh represents an average or typical face and / or head and is generated based on data previously collected for a relatively large number of users. According to embodiments described herein, the systems and methods can generate a user mesh, which may be a representation of the body part of the user on which the wearable device will be worn. In selecting, sizing, and / or fitting a head-mounted wearable device, the user mesh may represent the user's face / head.

[0009] In some examples, the image data includes a two-dimensional image captured by a user via an application running on the user computing device. In some examples, a selected feature (i.e., one of one or more features detected in the image data) is mapped to a keypoint in a reference mesh, or a canonical mesh. A rigid transformation may be applied to the keypoint of the reference mesh to project the keypoint onto the user mesh. The distance between the keypoint of the reference mesh and the keypoint of the user mesh may be used to match or align the reference mesh and the user mesh. In some examples, the distance between the keypoint in the reference mesh and the keypoint in the user mesh may be used to determine vertical and horizontal distances, e.g., in pixels, for projection onto the two-dimensional image captured by the user. This may provide more accurate placement of the wearable device on the two-dimensional image captured by the user operating the computing device. In some examples, the systems and methods described herein provide for fitting a head-mounted wearable device in the form of smart glasses that include processing / computing and display capabilities and / or corrective lenses. Systems and methods according to embodiments described herein may facilitate detection of one or more features by a user and capture of image data for fitting of a wearable device in a self-directed or unsupervised or unmonitored manner, without access to a retail facility and / or without a face-to-face or virtual encounter with a technician or sales representative.

[0010] Hereinafter, for purposes of explanation and illustration only, the systems and methods are described with respect to selecting, sizing, and / or fitting a head-mounted wearable device. Among features detectable in a two-dimensional image captured by a user, therion points are used, for purposes of explanation and illustration only, to match a rigid transformation between a reference mesh and key points on the user's face. The principles described herein may be applied to sizing and / or fitting other types of wearable devices, including, for example, eyeglasses that may or may not include processing / computing / display capabilities and / or corrective lenses or other types of wearable devices. Similarly, the principles described herein may utilize other features detected in image data in addition to or in place of therion points.

[0011] FIG. 1A illustrates a user wearing an exemplary head-mounted wearable device 100 in the form of smart glasses or augmented reality glasses that includes display, eye / gaze tracking, and computing / processing capabilities. FIG. 1B illustrates a front view of the exemplary head-mounted wearable device 100 shown in FIG. 1A , and FIG. 1C illustrates a rear view of the exemplary head-mounted wearable device 100 shown in FIG. 1A . The exemplary head-mounted wearable device 100 includes a frame 110. The frame 110 includes a front frame portion 120 and a pair of arm portions 130 rotatably coupled to the front frame portion 120 by respective hinge portions 140. The front frame portion 120 includes edge portions 123 that surround respective optics in the form of lenses 127, and a bridge portion 129 connects the edge portions 123. The arm portions 130 are coupled, e.g., pivotally or rotatably, to the front frame portion 120 at peripheral portions of the respective edge portions 123. In some examples, lens 127 may be a corrective / prescription lens. In some examples, lens 127 is an optical material that includes glass and / or plastic portions that do not necessarily incorporate corrective / prescription parameters.

[0012] In some examples, the wearable device 100 includes a display device 104 that can output visual content, for example, at an output coupler 105, so that the visual content is visible to the user. In the example shown in FIGS. 1B and 1C, the display device 104 is provided on one of the two arm portions 130 for purposes of explanation and illustration only. A display device 104 may be provided on each of the two arm portions 130 to provide binocular output of content. In some examples, the display device 104 may be a see-through near-eye display. In some examples, the display device 104 may be configured to project light from a display light source onto a portion of teleprompter glass that acts as a beam splitter mounted at an angle (e.g., 30-45 degrees). The beam splitter may allow for reflection and transmission values ​​that allow light from the display light source to be partially reflected while the remaining light is transmitted. Such an optical design may enable a user to view both physical items in the world, for example, through lens 127, next to content (e.g., digital images, user interface elements, virtual content, etc.) output by display device 104. In some implementations, waveguide optics may be used to render content on display device 104.

[0013] In some examples, the head-mounted wearable device 100 includes one or more of an audio output device 106 (e.g., one or more speakers, etc.), a lighting device 108, a sensing system 111, a control system 112, at least one processor 114, and an outward-facing image sensor 116 (e.g., a camera). In some examples, the sensing system 111 may include various sensing devices, and the control system 112 may include various control system devices, including, for example, one or more processors 114 operably coupled to components of the control system 112. In some examples, the control system 112 may include a communications module that provides for communication and exchange of information between the wearable device 100 and other external devices. In some examples, the head-mounted wearable device 100 includes an eye-tracking device 115 that detects and tracks the direction and movement of eye gaze. Data captured by the eye-tracking device 115 may be processed to detect and track the direction and movement of eye gaze as user input. 1B and 1C, the eye-tracking device 115 is provided on one of the two arm portions 130 for purposes of explanation and illustration only. In the exemplary arrangement shown in FIGS. 1B and 1C, the eye-tracking device 115 is provided in the same arm portion 130 as the display device 104, so that the gaze of the user's eyes can be tracked not only with respect to objects in the physical environment, but also with respect to content output for display by the display device 104. In some examples, an eye-tracking device 115 can be provided on each of the two arm portions 130 to provide eye-tracking for each of the user's two eyes. In some examples, the display device 104 can be provided on each of the two arm portions 130 to provide binocular display of visual content.

[0014] A number of different sizing and fitting measurements and / or parameters may be considered when selecting and / or sizing and / or fitting a wearable device, such as the exemplary head-mounted wearable device 100 shown in FIGS. 1A-1C , for a particular user. This may include, for example, wearable fit parameters or wearable fit measurements. The wearable fit parameters / measurements may take into account how a particular frame 110 fits a particular user and / or how a particular user looks and / or feels. The wearable fit parameters / measurements may take into account many factors, such as, for example, whether the rim portion 123 and bridge portion 129 are shaped and / or sized so that the bridge portion 129 rests comfortably on the bridge of the user's nose, whether the frame 110 is wide enough to be comfortable around the temples but not so wide that the frame 110 cannot remain relatively stationary when worn by the user, whether the arm portions 130 are sized to rest comfortably over the user's ears, and other such comfort-related considerations. Wearable fit parameters / measurements may take into account other wearing considerations, including how the frame 110 may be positioned based on the user's natural head posture / where the user naturally tends to wear their glasses. In some examples, aesthetic fit measurements or parameters may take into account whether the frame 110 is aesthetically pleasing to the user / compatible with the user's facial features, etc.

[0015] For head-mounted wearable devices that include display functionality, display fit parameters, or display fit measurements, may be taken into account when selecting and / or sizing and / or fitting the head-mounted wearable device 100 for a particular user. The display fit parameters / measurements may be used to configure the display device 104 in a selected frame 110 for a particular user, such that content output by the display device 104 is viewable by the user. For example, the display fit parameters / measurements may facilitate calibration of the display device 104 so that visual content is output within at least a set portion of the user's field of view. For example, the display fit parameters / measurements may be used to configure the display device 104 to provide at least a set level of visibility corresponding to an amount, portion, or percentage of visual content that the user can see in the periphery (e.g., corners with the least visibility) of the user's field of view.

[0016] In examples where the head-mounted wearable device 100 includes corrective lenses, ophthalmic fit parameters, or ophthalmic fit measurements, may be considered in the selection and / or sizing and / or fitting process. Some exemplary ophthalmic fit measurements are shown in FIGS. 2A-2C . Ophthalmic fit measurements may include, for example, pupillary height PH (the distance from the center of the pupil to the bottom of each lens 127). Ophthalmic fit measurements may also include interpupillary distance IPD (the distance between the pupils). IPD may be characterized by monocular pupillary distances, such as left pupillary distance LPD (the distance from the center of the nose bridge to the left pupil) and right pupillary distance RPD (the distance from the center of the nose bridge to the right pupil). Ophthalmic fit measurements may include pantoscope angle PA (the angle defined by the tilt of the lens 127 relative to vertical). Ophthalmic fit measurements may include vertex distance V (the distance from the cornea to each lens 127). The ophthalmic fit measurements may include other such parameters, or measurements that provide for the selection and / or sizing and / or fitting of head-mounted wearable device 100, including corrective lenses, with or without display device 104 as described above. In some examples, the ophthalmic fit measurements, together with the display fit measurements, can provide for the output of visual content by display device 104 within a defined three-dimensional volume such that the content is within the user's corrected field of view and is therefore visible to the user.

[0017] FIG. 3 is a block diagram of an example system that predicts sizing and / or fitting of a wearable device from at least one keypoint, landmark, or feature detected in at least one image, e.g., a two-dimensional image captured by a computing device operated by a user. The system may utilize at least one three-dimensional reference mesh, or canonical mesh, in determining sizing and / or fitting of the wearable device. In examples where the wearable device is a head-mounted wearable device, the reference mesh may represent a generic head generated based on data previously collected from a relatively large number of subjects. Wearable devices that can be sized and / or adapted by the system in this manner include the various wearable computing devices described above. Hereinafter, sizing and / or fitting of a head-mounted wearable device, such as the exemplary head-mounted wearable device 100, by the system is described solely for purposes of explanation and illustration.

[0018] The system may include one or more computing devices 300. The computing devices 300 may be operated by a user to whom a wearable device is to be sized and / or fitted. The computing devices 300 may be, for example, handheld devices such as smartphones or tablet computing devices, desktop or laptop computing devices, and other such computing devices that can be operated by a user to capture an image of the user. The computing devices 300 may access additional resources 302 to facilitate sizing and / or fitting of the wearable device. In some examples, the additional resources 302 may be available locally on the computing device 300. In some examples, the additional resources 302 are available on the computing device 300 via a network 306. In some examples, some of the additional resources 302 may be available locally on the computing device 300 and some of the additional resources 302 may be available to the computing device 300 via the network 306. The additional resources 302 may include, for example, a server computer system, a processor, a database, a machine learning module, memory storage, etc. In some examples, the processor(s) 390 may provide various processing functions, for example, via object recognition engine(s), pattern recognition engine(s), simulation engine(s), fitting engine(s), and other processors. In some examples, the additional resources 302 include machine learning models and / or algorithms that support sizing and / or fitting of the wearable device.

[0019] The computing device 300 can operate under the control of a control system 370. The computing device 300 can communicate with one or more external devices 304 (such as another wearable computing device, another mobile computing device, etc.) either directly (via wired and / or wireless communication) or via a network 306. In some examples, the computing device 300 includes a communications module 380 to facilitate external communications. In some examples, the computing device 300 includes a sensing system 320 that includes various sensing system components, including, for example, one or more image sensors 322, one or more position / orientation sensors 324 (including, for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer, etc.), one or more audio sensors 326 capable of detecting audio input, one or more touch input sensors 328 capable of detecting touch input, and other such sensors. The computing device 300 can include more or fewer sensing devices and / or combinations of sensing devices.

[0020] In some examples, the image sensor(s) 322 may include a camera, such as a forward-facing, outward-facing, or world-facing camera, that can capture still images and / or video of the environment outside the computing device 300. The still images and / or video may be displayed by a display device of the output system 340, and / or transmitted externally via the communications module 380 and the network 306, and / or stored in the memory 330 of the computing device 300 and / or in a memory device available in the additional resources 302.

[0021] The computing device 300 may include one or more processors 390. The processor 390 may include various modules or engines configured to perform various functions. In some examples, the processor(s) 390 may include object recognition module(s), pattern recognition module(s), configuration identification module(s), and other such processors. The processor(s) 390 may be formed on a substrate configured to execute one or more machine-executable instructions or portions of software, firmware, or a combination thereof. The processor(s) 390 may be semiconductor-based, including semiconductor materials capable of executing digital logic. The memory 330 may include any type of storage device that stores information in a format that can be read and / or executed by the processor(s) 390. The memory 330 may store applications and modules that perform certain operations when executed by the processor(s) 390. In some examples, the applications and modules may be stored on an external storage device and loaded into the memory 330.

[0022] As described above, systems and methods provide for frame selection and / or sizing and / or fitting of a head-mounted wearable device according to embodiments described herein. In some examples, one or more key points, landmarks, or features may be detected in image data of a user's face and / or head. The image data may be a two-dimensional image captured via an application running on a computing device operated by the user. In some examples, the one or more key points, landmarks, or features include a therion or therion point. The therion may be defined at the root of the user's nose or the midline of the bridge of the nose. The therion may be positioned at the point of maximum curvature of the nasal contour, the tip of the bridge of the nose, or the transition point between the bridge of the nose and the forehead. The therion may represent the deepest depression of the nasal bone.

[0023] FIG. 4A is an exemplary two-dimensional image 400 that may be captured via an application running on a computing device operated by a user, such as the exemplary computing device 300 described above with respect to FIG. 3. The exemplary two-dimensional image 400 provides a generally frontal view of the user. FIG. 4B shows multiple exemplary keypoints, features, or landmarks 410 that may be detected in the two-dimensional image 400 of the user, including a serion 420 at the base of the user's nose bridge. More or fewer keypoints, features, or landmarks 410 than those shown in FIG. 4B may be detected in the two-dimensional image 400 captured via an application running on a computing device operated by the user. FIG. 4C is a side view of the user, provided merely to further illustrate the location of the serion 420 relative to the user's nose bridge.

[0024] A two-dimensional image 400 of a user's face may be captured via an application running on a computing device, such as computing device 300, operated by a user, as described above. Object recognition engine(s) and / or pattern recognition engine(s) available via additional resources 302 may analyze the image 400 to detect one or more landmarks 410, including therions 420. In some examples, either the landmarks 410 or groups of landmarks 410 can be used to predict the position and / or fit of a frame on the user's face and / or head. However, the physical positioning of the frame on the user's face / head can vary and be significantly affected based on the height of the nose bridge, nose shape, etc. For example, a relatively high or low nose bridge can cause a noticeable change in the vertical positioning of the frame on the user's face. This can lead to inaccurate virtual sizing and / or fitting of the frame on the head-mounted wearable device. Thus, by using therion 420 to predict the virtual placement of the frame on the user's face / head image 400, a more accurate prediction of the sizing and / or fitting of the frame of a head-mounted wearable device in a virtual fitting situation may be provided.

[0025] Below, systems and methods are described with respect to using therions 420 detected within the two-dimensional image 400 to predict the placement of a virtual frame on the user's image 400 for purposes of virtual fitting. In some examples, for example, a simulation engine available to the computing device 300 via the additional resources 302 may access one or more machine learning models available via the additional resources to generate a user mesh 500, as shown in FIG. 5A . The user mesh 500 may be generated based on data obtained through analysis of the two-dimensional image 400 by an object recognition engine(s) and / or a pattern recognition engine(s). The user mesh 500 may represent the user's face / head based on the two-dimensional image 400 captured by the user. The user mesh 500 may include multiple interconnected nodes, some of which are labeled with reference numeral 505 in FIG. 5A .

[0026] In some examples, a reference mesh 550 such as that shown in FIG. 5B may be accessible to the computing device 300, for example, via one of the database(s) of the additional resource 302. The reference mesh 550 may represent a generic face / head created based on data collected from a relatively large number of subjects. The reference mesh 550 may include multiple interconnected nodes, some of which are labeled with reference numeral 555 in FIG. 5B. Because the reference mesh 550 is a somewhat generic mesh representing a generic face / head created based on data collected from a relatively large number of subjects, the reference mesh 550 is not specific to the user mesh 500. That is, there is not a one-to-one correspondence between the nodes 555 of the reference mesh 550 and the nodes 505 of the user mesh 500.

[0027] In some examples, one of the plurality of nodes 555 of the reference mesh 550 may be identified as an index node. In the examples described herein, the index node of the reference mesh 550 may be one of the plurality of nodes 555 that most closely maps to the location of a therion within the reference mesh 550. In this example, the index node may be identified as the therion node 552, as shown in FIG. 5B. As shown in FIG. 5C, a virtual frame 590 may be positioned on the reference mesh 550, with a bridge portion 598 of the virtual frame 590 positioned corresponding to the therion node 552 to simulate where a corresponding physical frame would naturally be worn by a user with a face / head that matches the reference mesh 550.

[0028] A rigid transformation of the reference mesh 550 (on which the virtual frame 590 is positioned) may be performed to project the reference mesh 550 (and virtual frame 590) onto the user mesh 500, as shown in FIG. 5D . In some examples, the rigid transformation may include rotating, translating, and scaling some number of nodes 555 of the reference mesh 550, or a subset of the nodes 555 of the reference mesh 550, to associate them with a corresponding / respective subset of nodes 505 of the user mesh 500. This rigid transformation does not generate a correspondence between each node of the reference mesh 550 and the user mesh 500. However, this approach may provide an approximation that can be adjusted to predict the sizing and / or fitting of a selected frame of a head-mounted wearable device.

[0029] FIG. 5D shows the initial placement position of the virtual frame 590 on the user's face / head based on the projection of the reference mesh 550 (and virtual frame 590) onto the user mesh 500. As shown in FIG. 5D, there is a positional difference (vertical difference in the direction shown in FIG. 5D) between the index node of the reference mesh 550, i.e., therion node 552 (and the associated initial placement position of the bridge portion 598 of the virtual frame 590 from the reference mesh 550) and the index node of the user mesh 500, i.e., therion node 502 identified within the user mesh 500. FIG. 5E shows the adjusted virtual placement position of the virtual frame 590 on the user's head / face. In FIG. 5E, the position of the virtual frame 590 has been adjusted or shifted so that the position of the bridge portion 598 corresponds to therion node 502 of the user mesh 500. Shifting the virtual frame 590 from the initial virtual placement position shown in Figure 5D to the adjusted virtual position shown in Figure 5E may position the virtual frame 590 on the user's head / face in a position that more closely simulates how the corresponding physical frame would be worn by the user, which may provide a more representative representation of the sizing and / or fitting of the selected frame of the head-mounted wearable device on the user's face / head.

[0030] FIG. 6 shows a two-dimensional image 600 of a virtual frame 590 positioned on a user's head / face in an adjusted virtual position. The two-dimensional image 600 may be presented to a user, for example, via an application running on a computing device operated by the user. In some examples, the virtual frame 590 may be overlaid on the initial image 400 shown in FIG. 4A to generate the image 600 shown in FIG. 6. The position of the virtual frame 590 in the image 600 shown in FIG. 6 represents how a corresponding physical frame would be worn by the user, how the corresponding physical frame would appear on the user's face, and how the corresponding physical frame fits the user. Thus, the user may evaluate the image 600 of the virtual frame 590 positioned on the user's head / face in an adjusted virtual position to confirm the sizing and / or fitting of the frame selected for the head-mounted wearable device.

[0031] As described above, therions 420 can be relatively reliably detected in the two-dimensional image 400 captured by the user. Therefore, therions 420 detected in the image 400 can provide relatively reliable reference points for positioning the virtual frame 590 on the user's face / head in the image 400, thereby providing relatively reliable virtual sizing and / or fitting of the head-mounted wearable device for the user. In the above example, a single general reference mesh is used. As described above, the reference mesh 550 is generated based on data collected from a relatively large number of subjects. Rigidly transforming a subset of the nodes 555 of the reference mesh 550 to a corresponding subset of the nodes 505 of the user mesh 500 can provide a relatively reliable reference for the initial virtual placement of the virtual frame 590 on the user's face / head in the two-dimensional image 400. The identification of therion node 502 in the user mesh 500 (based on detecting therion 420 in the user's image 400) and the identification of therion node 552 in the reference mesh 550 can provide a reference for shifting the virtual frame 590 from an initial virtual placement position to an adjusted virtual placement position that is proximate therion node 502 in the user mesh 500 (corresponding to the identified therion 420 in the user's image 400). The adjusted virtual placement position can represent where the corresponding physical frame would naturally be worn by the user.

[0032] Because the placement position of the virtual frame 590 is adjusted based on the position of the therion, the relatively low computational burden associated with performing rigidity transformations in this manner may enable these processes to be performed locally on the user device rather than relying on the user for external computing resources. This may facilitate virtual sizing and / or fitting of a user's head-mounted wearable device without the need to visit a retail facility and / or without the assistance of an optical engineer or sales representative (either virtual or in-person) and / or without the use of specialized equipment.

[0033] The above examples utilize a single reference mesh in determining the placement location of the virtual frame 590 on the user's face / head for virtual sizing and / or fitting of the head-mounted wearable device. In some implementations, multiple reference meshes may be available to perform the sizing and / or fitting operations as described above.

[0034] For example, as described above, nose bridge height (e.g., detectable based on identification of therion 420 in the user's image 400) can affect how and where the physical frame of the head-mounted wearable device is worn by the user. The above-described systems and methods are implemented using a single reference mesh that is projected onto a user mesh via a rigid transformation. In some examples, multiple reference meshes based on nose bridge height may be available to facilitate sizing and / or fitting of the head-mounted wearable device. In some examples, the system may select from the multiple reference meshes the reference mesh that is most suitable for sizing and / or fitting the frame of the head-mounted wearable device for a particular user. For example, the system may select a first reference mesh in response to determining that the user has an average nose bridge height. Similarly, the system may select a second reference mesh in response to determining that the user has a relatively high nose bridge height, and a third reference mesh in response to determining that the user has a relatively low nose bridge height. The first reference mesh may be generated based on data collected from a relatively large number of subjects, all of whom are determined to have average nose bridge heights. Similarly, the second reference mesh may be generated based on data collected from a relatively large number of subjects who are all determined to have a relatively high nose bridge height, and the third reference mesh may be generated based on data collected from a relatively large number of subjects who are all determined to have a relatively low nose bridge height. Implementing a reference mesh that more closely matches the characteristics of a user's face may improve the accuracy of the virtual placement of the virtual frame on the user's face / head (compared to wearing a corresponding physical frame) and / or reduce the computational load associated with the virtual placement of the frame on an image of the user's face / head.

[0035] In some examples, the system may determine that a user has an average nose bridge height, a relatively high nose bridge height, or a relatively low nose bridge height based on the detected location of the therion 420 relative to other facial landmarks 410 detected within the two-dimensional image 400. In some examples, the object / pattern recognition engine(s), simulation engine(s), and / or machine learning model(s) described above may facilitate the detection of the facial landmarks 410 and therion 420 and the determination of whether the user falls into a first category of users having an average nose bridge height, a second category of users having a relatively high nose bridge height, or a third category of users having a relatively low nose bridge height. In some examples, a threshold distance between various facial landmarks 410 and / or between the facial landmarks 410 and therion 420 may be used to determine whether the user falls into the first category, the second category, or the third category. Based on which category is associated with the user, the system may select a reference mesh from multiple reference meshes available to the system for virtual sizing and / or fitting of the frame of the head-mounted wearable device, for example, The selected reference mesh may then be applied in a similar manner as described above to position the virtual frame 590 on the user's face / head.

[0036] The nose bridge height is just one example of how using multiple reference meshes can further facilitate sizing and / or fitting of a head-mounted wearable device for a particular user. Other reference meshes can similarly be performed based on other characteristics, for example, characteristics and / or features that may be detectable in the image 400 captured via an application running on a computing device operated by the user. This may include, for example, characteristics related to the width of the user's nose, such as one or more widths taken at designated portions of the nose, ratios of widths taken at designated portions of the nose, etc. Other characteristics may include, for example, detected nose contours and / or changes in nose contours that may affect where the bridge portion of the frame is positioned on the user's nose, and other such characteristics and / or features.

[0037] FIG. 7 is a flowchart of an exemplary method 700 according to embodiments described herein. A user operating a computing device (e.g., the computing device 300 described above, or another computing device) may initiate an image capture function of the computing device (block 710). The image capture function may be accessed through an application running on the computing device operated by the user. Initiating the image capture function causes an image sensor (e.g., an image sensor of the front-facing camera of the computing device 300 described above) to capture two-dimensional image data including the user's face and / or head (block 715). One or more fixed features or landmarks may be detected within the image (block 720). The one or more fixed features or landmarks may include facial landmarks that remain substantially static, such as a selion defined at the midline of the user's nose bridge or nose bridge, and other such fixed facial landmarks. A user mesh may be generated based on an analysis of the image and the detected one or more facial landmarks (block 725). In some examples, the user mesh may be generated by one or more machine learning models accessible to the computing device. The system may access or obtain a reference mesh (block 730). The reference mesh may be obtained from a user-accessible database. In some examples, a single reference mesh is available. In some examples, multiple reference meshes may be available representing multiple different facial characteristics, such as different nose bridge height categorizations. A virtual frame associated with sizing and / or fitting the user's head-mounted wearable device may be positioned on the reference mesh (block 735). The virtual frame may be positioned on the reference mesh based on identification of one of multiple nodes of the reference mesh, and in particular, a therion node of the reference mesh that corresponds to a therion region of the reference mesh. A transformation may be performed to project the reference mesh and the virtual frame onto the user mesh (block 740).The transformation may be a rigid transformation including rotation, translation, and scaling that fits at least a portion of the nodes of the reference mesh to the user mesh. In response to determining (block 745) that the therion nodes of the reference mesh are aligned with the corresponding therion nodes of the user mesh, a resized / fitted image may be output, for example, via an application running on a computing device. In response to determining (block 745) that the therion nodes of the reference mesh are offset from or misaligned with the corresponding therion nodes of the user mesh, the system may shift the placement position of the virtual frame from an initial placement position (where the bridge portion of the virtual frame is positioned at the therion nodes of the reference mesh offset from the therion nodes of the user mesh) to an adjusted position (where the bridge portion of the virtual frame is positioned corresponding to the therion nodes of the user mesh) (block 755) before outputting the resized / fitted image (block 750).

[0038] Below, some examples are provided.

[0039] and generating a user mesh, the user mesh representing the user's face based on the image data; the method further comprising: identifying an index node in the user mesh corresponding to a converged portion of the face of the user captured in the image data; and identifying an index node in a reference mesh, the index node of the reference mesh corresponding to a converged portion of the reference mesh, the converged portion of the reference mesh corresponding to the converged portion of the user mesh; the method further comprising: positioning a virtual frame of a head-mounted wearable device on the reference mesh at a location corresponding to the index node of the reference mesh; projecting the reference mesh and the virtual frame onto the user mesh; and adjusting the position of the virtual frame to correspond to the index node of the user mesh.

[0040] Example 2: The computer-implemented method of Example 1, wherein identifying the index node in the user mesh includes identifying a therion node in the user mesh, the therion node corresponding to a position of a therion portion of the face of the user captured in the image data, and identifying the index node in the reference mesh includes identifying a therion node in the reference mesh, the therion node corresponding to a position of a therion portion of the face represented by the reference mesh.

[0041] Example 3: The computer-implemented method of example 1 or example 2, wherein projecting the reference mesh and the virtual frame onto the user mesh includes performing a rigid transformation of the reference mesh and the virtual frame onto the user mesh.

[0042] Example 4: The computer-implemented method of example 3, wherein the reference mesh includes a plurality of nodes, and wherein performing the rigid transformation includes performing rotation, translation, and scaling operations on a subset of the plurality of nodes of the reference mesh to fit the reference mesh to the user mesh.

[0043] Example 5: The computer-implemented method of any one of the preceding examples, wherein generating the user mesh includes detecting one or more facial landmarks in the image data; and generating the user mesh based on the one or more facial landmarks with a machine learning model.

[0044] Example 6: The computer-implemented method of any one of the preceding examples, also including outputting a fitting image, the fitting image including a rendering of the virtual frame superimposed on the initial image of the face of the user generated based on the image data at the location corresponding to the index node of the user mesh.

[0045] Example 7: The computer-implemented method of Example 6, wherein adjusting the position of the virtual frame includes comparing a position of the index node of the reference mesh with a position of the index node of the user mesh, wherein the comparing includes detecting a distance between the index node in the reference mesh and the index node in the user mesh, determining a corresponding pixel distance between the index node of the reference mesh and the index node of the user mesh, and adjusting the position of the virtual frame in the fitting image based on the pixel distance.

[0046] Example 8: The computer-implemented method of any one of the preceding examples, wherein capturing the image data includes capturing a two-dimensional image of the face of the user, the user mesh is a three-dimensional mesh corresponding to the face of the user, and the reference mesh is a three-dimensional mesh generated based on previously collected data representing multiple subjects.

[0047] Example 9: The computer-implemented method of any one of the preceding examples, further comprising selecting a reference mesh from a plurality of reference meshes, wherein the selecting comprises: detecting at least one facial landmark in the image data; mapping the at least one facial landmark to a corresponding node of the user mesh; and selecting the reference mesh from the plurality of reference meshes based on a relative position of an index node of the user mesh and the node in the user mesh that corresponds to the at least one facial landmark.

[0048] Example 10: A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a computing device, are configured to: capture image data including an initial image of a user's face with an image sensor of the computing device; generate a user mesh representing the user's face based on the image data; identify an index node in the user mesh that corresponds to a converged portion of the face of the user captured in the image data; and identify an index node in a reference mesh, the index node of the reference mesh corresponding to a converged portion of the reference mesh, and the converged portion of the reference mesh corresponding to the converged portion of the user mesh; the instructions are further configured to: position a virtual frame of a head-mounted wearable device on the reference mesh at a position corresponding to the index node of the reference mesh; project the reference mesh and the virtual frame onto the user mesh; and adjust the position of the virtual frame to correspond to the index node of the user mesh.

[0049] Example 11: The non-transitory computer-readable medium of Example 10, wherein the instructions cause the at least one processor to identify the index node in the user mesh, wherein the identifying includes identifying a therion node in the user mesh, the therion node corresponding to a position of a therion portion of the face of the user captured in the image data, and the instructions further cause the at least one processor to identify the index node in the reference mesh, wherein the identifying includes identifying a therion node in the reference mesh, the therion node corresponding to a position of a therion portion of the face represented by the reference mesh.

[0050] Example 12: The non-transitory computer-readable medium of Example 10 or Example 11, wherein the instructions cause the at least one processor to perform a rigid transformation of the reference mesh and the virtual frame onto the user mesh so as to project the reference mesh and the virtual frame onto the user mesh.

[0051] Example 13: The non-transitory computer-readable medium of Example 12, wherein the reference mesh includes a plurality of nodes, and the instructions cause the at least one processor to perform the rigid transformation, including a rotation operation, a translation operation, and a scaling operation, on a subset of the plurality of nodes of the reference mesh to fit the reference mesh to the user mesh.

[0052] Example 14: A non-transitory computer-readable medium described in any one of Examples 10 to 13, wherein the instructions cause the at least one processor to detect one or more facial landmarks in the image data and generate the user mesh based on the one or more facial landmarks using a machine learning model.

[0053] Example 15: A non-transitory computer-readable medium described in any one of Examples 10 to 14, wherein the instructions cause the at least one processor to output a fitting image, the fitting image including a rendering of the virtual frame superimposed on the initial image of the user's face generated based on the image data at the location corresponding to the index node of the user mesh.

[0054] Example 16: The non-transitory computer-readable medium of Example 15, wherein the instructions cause the one or more processors to compare a position of the index node of the reference mesh with a position of the index node of the user mesh, and the comparing includes: detecting a distance between the index node in the reference mesh and the index node in the user mesh; determining a corresponding pixel distance between the index node of the reference mesh and the index node of the user mesh; and adjusting a position of the virtual frame in the fitting image based on the pixel distance.

[0055] Example 17: A non-transitory computer-readable medium described in any one of Examples 10 to 16, wherein the instructions cause the at least one processor to capture a two-dimensional image of the user's face, the user mesh is a three-dimensional mesh corresponding to the user's face, and the reference mesh is a three-dimensional mesh generated based on previously collected data representing multiple subjects.

[0056] Example 18: A non-transitory computer-readable medium described in any one of Examples 10 to 17, wherein the instructions cause the at least one processor to select a reference mesh from a plurality of reference meshes, and the selecting includes: detecting at least one facial landmark in the image data; mapping the at least one facial landmark to a corresponding node of the user mesh; and selecting the reference mesh from the plurality of reference meshes based on a relative position of an index node of the user mesh and the node in the user mesh that corresponds to the at least one facial landmark.

[0057] Example 19: A system comprising a computing device, the computing device including an image sensor, at least one processor, and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to capture image data including an initial image of a user's face, generate a user mesh representing the user's face based on the image data, identify a therion node in the user mesh corresponding to a therion portion of the face of the user captured in the image data, identify a therion node of a reference mesh, the therion node of the reference mesh corresponding to the therion portion of the reference mesh, and the therion portion of the reference mesh corresponding to the therion portion of the user mesh, and the instructions further cause the at least one processor to position a virtual frame of a head-mounted wearable device on the reference mesh at a position corresponding to the therion node of the reference mesh, project the reference mesh and the virtual frame onto the user mesh, and adjust the position of the virtual frame to correspond to the therion node of the user mesh.

[0058] Although several embodiments have been described, it will nevertheless be understood that various modifications may be made without departing from the spirit and scope of the invention.

[0059] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Moreover, other steps may be provided or certain steps may be deleted from the described flows, and further, other components may be added to or deleted from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0060] In addition to the above, the system, program, or functionality described herein may provide users with controls that allow them to choose both whether and when collection of user information (e.g., information regarding the user's social network, social actions or activities, occupation, user preferences, or the user's current location) may be enabled, and whether content or communications are sent from the server to the user. Furthermore, certain data may be processed in one or more ways so that personally identifiable information is removed before it is stored or used. For example, the user's identifying information may be processed so that personally identifiable information about the user cannot be determined, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized so that the user's specific location cannot be identified. Thus, users may control what information is collected about them, how that information is used, and what information is provided to them.

[0061] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the embodiments. They are presented by way of example only, not limitation, and it is to be understood that various changes in form and detail may be made. Any portion of the apparatus and / or methods described herein may be combined in any combination except mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

Claims

1. 1. A computer-implemented method comprising: capturing image data including an initial image of a face of the user via an application executing on a computing device operated by the user; generating a user mesh, the user mesh representing a face of the user based on the image data, the method further comprising: identifying an index node in the user mesh that corresponds to a convergent portion of the face of the user captured in the image data; identifying an index node in a reference mesh, the index node of the reference mesh corresponding to a collective portion of the reference mesh, the collective portion of the reference mesh corresponding to the collective portion of the user mesh, the method further comprising: positioning a virtual frame of a head-mounted wearable device on the reference mesh at a location corresponding to the index node of the reference mesh; projecting the reference mesh and the virtual frame onto the user mesh; and adjusting a position of the virtual frame to correspond to the index node of the user mesh.

2. identifying the index node within the user mesh includes identifying a therion node within the user mesh, the therion node corresponding to a location of a therion portion of the face of the user captured in the image data; 2. The computer-implemented method of claim 1, wherein identifying the index node in the reference mesh includes identifying a therion node in the reference mesh, the therion node corresponding to a location of a therion portion of a face represented by the reference mesh.

3. 3. The computer-implemented method of claim 1, wherein projecting the reference mesh and the virtual frame onto the user mesh comprises performing a rigid transformation of the reference mesh and the virtual frame onto the user mesh.

4. 4. The computer-implemented method of claim 3, wherein the reference mesh includes a plurality of nodes, and performing the rigid transformation includes performing rotation, translation, and scaling operations on a subset of the plurality of nodes of the reference mesh to fit the reference mesh to the user mesh.

5. generating the user mesh Detecting one or more facial landmarks within the image data; and generating the user mesh based on the one or more facial landmarks by a machine learning model.

6. 10. The computer-implemented method of claim 1, further comprising outputting a fitting image, the fitting image comprising a rendering of the virtual frame superimposed on the initial image of the face of the user generated based on the image data at the position corresponding to the index node of the user mesh.

7. Adjusting the position of the virtual frame may include: comparing a position of the index node of the reference mesh with a position of the index node of the user mesh, wherein the comparing Detecting a distance between the index node in the reference mesh and the index node in the user mesh; determining corresponding pixel distances between the index nodes of the reference mesh and the index nodes of the user mesh; and adjusting a position of the virtual frame within the fitting image based on the pixel distance.

8. capturing the image data includes capturing a two-dimensional image of the face of the user; 10. The computer-implemented method of claim 1, wherein the user mesh is a three-dimensional mesh corresponding to the face of the user, and the reference mesh is a three-dimensional mesh generated based on previously collected data representing multiple subjects.

9. further comprising selecting a reference mesh from a plurality of reference meshes, said selecting comprising: Detecting at least one facial landmark within the image data; mapping said at least one facial landmark to a corresponding node of said user mesh; and selecting the reference mesh from the plurality of reference meshes based on a relative position of an index node of the user mesh and the node in the user mesh corresponding to the at least one facial landmark.

10. 1. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a computing device, cause the at least one processor to: causing an image sensor of the computing device to capture image data including an initial image of a user's face; generating a user mesh representing a face of the user based on the image data; identifying an index node in the user mesh that corresponds to a portion of the face of the user captured in the image data; and configured to identify an index node in a reference mesh, the index node of the reference mesh corresponding to a collective portion of the reference mesh, the collective portion of the reference mesh corresponding to the collective portion of the user mesh, the instructions further causing the at least one processor to: positioning a virtual frame of a head-mounted wearable device on the reference mesh at a location corresponding to the index node of the reference mesh; projecting the reference mesh and the virtual frame onto the user mesh; A non-transitory computer-readable medium configured to adjust a position of the virtual frame to correspond to the index node of the user mesh.

11. The instructions cause the at least one processor to: wherein identifying the index node within the user mesh includes identifying a therion node within the user mesh, the therion node corresponding to a location of a therion portion of the face of the user captured in the image data, the instructions further causing the at least one processor to:

11. The non-transitory computer-readable medium of claim 10, wherein identifying the index node in the reference mesh includes identifying a therion node in the reference mesh, the therion node corresponding to a location of a therion portion of a face represented by the reference mesh.

12. The instructions cause the at least one processor to:

12. The non-transitory computer-readable medium of claim 10 or 11, further comprising: performing a rigid transformation of the reference mesh and the virtual frame onto the user mesh such that the reference mesh and the virtual frame are projected onto the user mesh.

13. 13. The non-transitory computer-readable medium of claim 12, wherein the reference mesh includes a plurality of nodes, and the instructions cause the at least one processor to perform the rigid transformation, including rotation, translation, and scaling operations, on a subset of the plurality of nodes of the reference mesh to fit the reference mesh to the user mesh.

14. The instructions cause the at least one processor to: detecting one or more facial landmarks within the image data; The non-transitory computer-readable medium of any one of claims 10 to 13, wherein the user mesh is generated based on the one or more facial landmarks by a machine learning model.

15. The instructions cause the at least one processor to:

15. The non-transitory computer-readable medium of claim 10, further comprising: outputting a fitting image, the fitting image comprising a rendering of the virtual frame superimposed on the initial image of the face of the user generated based on the image data at the location corresponding to the index node of the user mesh.

16. The instructions cause the one or more processors to: Comparing the positions of the index nodes of the reference mesh with the positions of the index nodes of the user mesh, and the comparing step includes: Detecting a distance between the index node in the reference mesh and the index node in the user mesh; determining corresponding pixel distances between the index nodes of the reference mesh and the index nodes of the user mesh; and adjusting a position of the virtual frame within the fitting image based on the pixel distance.

17. 17. The non-transitory computer-readable medium of claim 10, wherein the instructions cause the at least one processor to capture a two-dimensional image of the face of the user, the user mesh being a three-dimensional mesh corresponding to the face of the user, and the reference mesh being a three-dimensional mesh generated based on previously collected data representing multiple subjects.

18. The instructions cause the at least one processor to: Selecting a reference mesh from a plurality of reference meshes, said selecting comprising: detecting at least one facial landmark within the image data; mapping said at least one facial landmark to a corresponding node of said user mesh; and selecting the reference mesh from the plurality of reference meshes based on a relative position of an index node of the user mesh and the node in the user mesh corresponding to the at least one facial landmark.

19. 1. A system comprising: a computing device, teeth, an image sensor; at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to: capturing image data including an initial image of the user's face; generating a user mesh representing a face of the user based on the image data; identifying therion nodes in the user mesh corresponding to therion portions of the face of the user captured in the image data; and identifying a therion node in a reference mesh, the therion node of the reference mesh corresponding to a therion portion of the reference mesh, the therion portion of the reference mesh corresponding to the therion portion of the user mesh, the instructions further causing the at least one processor to: positioning a virtual frame of a head-mounted wearable device on the reference mesh at a location corresponding to the therion node of the reference mesh; projecting the reference mesh and the virtual frame onto the user mesh; The system adjusts the position of the virtual frame to correspond to the therion node of the user mesh.

20. The reference mesh includes a plurality of nodes, and the user mesh includes a plurality of nodes, and the instructions cause the at least one processor to: Projecting the reference mesh and the virtual frame onto the user mesh, and the projecting includes:

20. The system of claim 19, further comprising: performing a rigid transformation of the reference mesh and the virtual frame onto the user mesh, wherein performing the rigid transformation comprises performing rotation, translation, and scaling operations on a subset of the plurality of nodes of the reference mesh to fit the reference mesh to the user mesh.

Images