Image-based Detection of Fit for Head-Mounted Wearable Computing Devices

Abstract

A system and method are provided for detecting display fit measurements and / or ophthalmic measurements for a head-mounted wearable computing device including a display device. An image of a fitting frame worn by a user of the computing device is captured by the user via an application running on the computing device. One or more key points and / or features and / or landmarks are detected in the image including the fitting frame. A three-dimensional pose of the fitting frame is determined based on the detected key points and / or features and / or landmarks and configuration information associated with the fitting frame. A display device of the head-mounted wearable computing device may then be configured based on the three-dimensional pose of the fitting frame captured in the image.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application is a continuation of U.S. Application No. 17 / 444,963, filed on August 12, 2021, the disclosure of which is hereby incorporated by reference in its entirety, and claims the benefit thereof.

[0002] Technical Field This description generally relates to predicting the wearable fit and / or display fit and / or ophthalmic fit of wearable devices, and in particular to predicting the display fit and ophthalmic fit of head - mounted computing devices including a display function.

Background Art

[0003] Background Wearable devices may include, for example, head - mounted devices such as smart glasses, headsets, earbuds, etc., wrist and / or hand - worn devices such as smart watches, smart bracelets, smart rings, etc., smart pendants, fitness trackers, cameras, body sensors, and other such devices. In some instances, a user may want to select and / or customize a wearable device for fit and / or functionality. For example, a user may wish to select and / or customize smart glasses to include frame selection, incorporation of prescription lenses, and other such features. Existing systems for procurement of these types of wearable devices do not provide accurate fitting and customization, especially without access to a retail store.

Summary of the Invention

[0004] Summary In one general aspect, a method for detecting display fit measurements for a head-mounted wearable computing device including a display device includes detecting, by an application executed on the computing device, at least one key point on a fitting frame worn by a user from an image captured by the computing device; accessing configuration information associated with the fitting frame detected in the image; determining a three-dimensional pose of the fitting frame captured in the image based on the detection of the at least one key point and the configuration information associated with the fitting frame; and determining a configuration of the display device of the head-mounted wearable computing device based on the three-dimensional pose of the fitting frame captured in the image.

[0005] In some implementations, detecting the at least one key point includes detecting at least one of a bridge portion of the fitting frame, a hinge location between an edge portion and an arm portion of the fitting frame, a peripheral portion of a lens of the fitting frame, or a saddle portion of an arm portion of the fitting frame.

[0006] In some implementations, determining the three-dimensional pose of a fitting frame includes accessing a three-dimensional model of the fitting frame captured in an image and performing a comparison that includes comparing the known position and known orientation of at least one keypoint detected in the image to the corresponding position and corresponding orientation of at least one keypoint in the three-dimensional model of the fitting frame. Detecting at least one keypoint may include detecting a plurality of keypoints on the fitting frame in an image captured by a computing device, and performing the comparison may include, for each of the plurality of keypoints, comparing the known position and known orientation of the keypoint detected in the image to the corresponding position and corresponding orientation of the keypoint in the three-dimensional model of the fitting frame. Determining the three-dimensional pose of the fitting frame may include performing an association of the two-dimensional position of each keypoint detected in the image with the corresponding three-dimensional position of each keypoint in the three-dimensional model of the fitting frame.

[0007] In some implementations, the method may further include detecting a plurality of facial landmarks in the captured image and determining ophthalmic fit measurements for a head-mounted wearable computing device based on the detected plurality of facial landmarks. Detecting the plurality of facial landmarks may include detecting the pupil height in an image of a fitting frame worn by the user, detecting at least one of the interpupillary distance or the monocular pupil distance in an image of a fitting frame worn by the user, and determining the forward tilt angle of the fitting frame worn by the user based on the determined three-dimensional pose of the fitting frame and the detected plurality of facial landmarks. Determining the configuration of the display device of the head-mounted wearable computing device based on the three-dimensional pose of the fitting frame captured in the image may include adapting the configuration of the display device of the head-mounted wearable computing device to correspond to the ophthalmic fit measurements. Determining the configuration of the display device of the head-mounted wearable computing device may include determining the user's field of view based on the ophthalmic fit measurements and the three-dimensional pose of the fitting frame, and configuring the output coupler of the display device so that the content output by the display device is displayed within the determined field of view.

[0008] In another general aspect, a non-transitory computer-readable medium stores executable instructions that, when executed by at least one processor, cause the at least one processor to capture an image of a fitting frame worn by a user, detect at least one keypoint on the fitting frame from the captured image, access configuration information associated with the fitting frame, determine a three-dimensional pose of the fitting frame captured in the image based on the detection of at least one keypoint and the configuration information associated with the fitting frame, and determine the configuration of a display device of a head-mounted wearable computing device based on the three-dimensional pose of the fitting frame captured in the image.

[0009] In some implementations, the instructions cause the at least one processor to detect at least one keypoint on the fitting frame in the captured image, including detecting a first keypoint including a bridge portion of the fitting frame, detecting a second keypoint including a hinge portion between an edge portion and an arm portion of the fitting frame, detecting a third keypoint including a peripheral portion of a lens of the fitting frame, or detecting a fourth keypoint including a saddle portion of an arm portion of the fitting frame. In some implementations, the instructions cause the at least one processor to determine a three-dimensional pose of the fitting frame, access a three-dimensional model of the fitting frame captured in the image, and perform a comparison between a known position and a known orientation of at least one detected keypoint in the image and a corresponding position and a corresponding orientation of at least one keypoint in the three-dimensional model of the fitting frame.

[0010] In some implementations, the instructions cause at least one processor to detect multiple key points and perform a comparison, and for each of the multiple key points, compare the known position and known orientation of the detected key points in the image with the corresponding position and corresponding orientation of the key points in the three-dimensional model of the fitting frame. In some implementations, the instructions cause at least one processor to determine a three-dimensional pose and perform an association of the two-dimensional position of each detected key point in the image with the corresponding three-dimensional position of each key point in the three-dimensional model of the fitting frame.

[0011] In some implementations, the instructions also cause at least one processor to detect multiple face landmarks in the captured image and determine ophthalmic fit measurements for a head-mounted wearable computing device based on the detected multiple face landmarks. The instructions can cause at least one processor to detect multiple face landmarks, detect the pupil height in an image of the fitting frame worn by the user, detect at least one of the interpupillary distance or the monocular pupil distance in an image of the fitting frame worn by the user, and determine the forward tilt angle of the fitting frame worn by the user based on the determined three-dimensional pose of the fitting frame and the detected multiple face landmarks. In some implementations, the instructions cause at least one processor to adapt the configuration of the display device of the head-mounted wearable computing device to correspond to the ophthalmic fit measurements, determine the user's field of view based on the ophthalmic fit measurements and the three-dimensional pose of the fitting frame, and configure the output coupler of the display device such that the content output by the display device is displayed within the determined field of view.

[0012] Details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0013]

Figure 1A

Figure 1B

Figure 1C

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3

Figure 4A

Figure 4B

Figure 5A

Figure 5B

Figure 5C

Figure 6A

Figure 6B

Figure 6C

Figure 6D

Figure 7

Figure 8

Figure 9

DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION The present disclosure relates to systems and methods for the selection and fitting of wearable computing devices, including determination of wearable fit and / or display fit and / or ophthalmic fit parameters related to wearable computing devices. The systems and methods according to implementations described herein provide determination of wearable fit and / or display fit and / or ophthalmic fit for wearable computing devices based on measurements detected within image data. In some implementations, the systems and methods according to implementations described herein provide detection of wearable fit and / or display fit and / or ophthalmic fit based on known parameters associated with a sample frame of a wearable computing device detected within image data. In some examples, the systems and methods described herein provide detection of wearable fit and / or display fit and / or ophthalmic fit from image data for a wearable computing device in the form of smart glasses including a display and / or corrective / prescription lenses customized for the particular physical characteristics, needs, and preferences of a particular user. The systems and methods according to implementations described herein may facilitate capture of image data for detection of measurement data by a user in an autonomous, or unmonitored, or unsupervised manner without access to a retail store and / or without an in-person or virtual appointment with a sales agent.

[0015] The systems and methods according to the implementations described herein are described with respect to the fitting of wearable computing devices in the form of head-mounted display devices, such as smart glasses including a display device, such that the content displayed by the display device can be seen by a user wearing the wearable computing device. In situations where a user benefits from a head-mounted wearable computing device including corrective lenses, the systems and methods described herein may provide for the determination of ophthalmic fit measurements and enable the incorporation of corrective / prescription lenses into the head-mounted wearable computing device while taking into account the characteristics of the corrective / prescription lenses in the configuration of the display device for the head-mounted wearable computing device. That is, the systems and methods according to the implementations described herein may use an image-based capture system and method for providing the fitting of the display device and the corrective / prescription lenses such that the content displayed by the display device can be seen by a user wearing the head-mounted wearable computing device.

[0016] Figure 1 shows a user operating a mobile computing device in a system, where in this system, the mobile computing device can access one or more external resources 1100 via network 1200. Figure 1A provides a third-person perspective of a user in the surrounding environment 1000, including various examples of mobile computing devices 100, 180, 190, 200. The exemplary mobile computing devices shown in Figure 1A include a first head-mounted wearable computing device 100, a second head-mounted wearable computing device 180, a wrist-mounted computing device 190, and a handheld computing device 200. In some examples, the first head-mounted computing device 100 can include various components such as, for example, a display, a camera capable of capturing an image of the surrounding environment, audio input / output functions, user input functions, etc., in a head-mounted display device in the form of smart glasses or a headset. In some examples, the second head-mounted computing device 180 can be an ear-mounted computing device such as headphones or earbuds, and can include audio input / output functions, a camera capable of capturing an image of the surrounding environment, user input functions, etc. In some examples, the wrist-mounted computing device 190 can include a display, a camera capable of capturing an image of the surrounding environment, audio input / output functions, user input functions, etc., in a smartwatch or a wristband. In some examples, the handheld computing device 200 can include a display, a camera capable of capturing an image of the surrounding environment, audio input / output functions, user input functions, etc., in a smartphone. Figure 1A shows only some examples of mobile computing devices. The principles described herein may also be applied to other types of mobile computing devices not explicitly shown in Figure 1A.In some examples, a mobile computing device, including the exemplary computing device shown in FIG. 1A, can communicate with each other and / or with an external resource 1100 to exchange information, receive and transmit inputs and / or outputs, etc.

[0017] FIG. 1B is a front view of the exemplary head-mounted wearable computing device 100 shown in FIG. 1A, and FIG. 1C is a rear view. In some implementations, the exemplary head-mounted wearable computing device 100 may take the form of, for example, smart glasses or augmented reality glasses, or an augmented reality and / or virtual reality headset or goggles as shown in the examples of FIGS. 1B and 1C. Hereinafter, the systems and methods according to the implementations described herein will be described with respect to the wearable computing device 100 in the form of smart glasses merely to facilitate discussion and illustration. The principles described herein can be applied to other types of wearable computing devices and / or combinations of collaborating mobile / wearable computing devices.

[0018] As shown in FIG. 1B, an exemplary head-mounted wearable computing device 100 includes a frame 102. In the example shown in FIGS. 1B and 1C, the frame 102 includes an edge portion 103 that surrounds a glass portion 107 or lens 107. Arm portions 105 are coupled, e.g., pivotably or rotatably coupled, to respective edge portions 103 of the frame 102 by respective hinge portions 110. In some examples, the lens 107 may be a corrective / prescription lens. In some examples, the lens 107 may be a glass portion that does not necessarily incorporate corrective / prescription parameters. In some examples, the bridge portion 109 may connect the edge portions 103 of the frame 102. A display device 104 may be coupled to a portion of the frame 102. In the example shown in FIGS. 1B and 1C, the display device 104 is coupled at an arm portion 105 of the frame 102. Together with the display device 104 coupled at the arm portion 105, an eyebox 104 extends toward the lens 107 for output of content at an output coupler 144, where content output by the display device 104 may be visible to a user. In some examples, the output coupler 144 may substantially coincide with the lens 107. The head-mounted wearable computing device 100 may also include an audio output device 106 (e.g., one or more speakers, etc.), an illumination device 108, a sensing system 111, a control system 112, at least one processor 114, and an outward-facing image sensor 116, or camera 116. In some implementations, the display device 104 may include a see-through near-eye display. For example, the display device 104 may be configured to project light from a display source onto a portion of a teleprompter glass that functions as a beam splitter seated at a predetermined angle (e.g., 30 - 45 degrees). The beam splitter may allow reflection and transmission values such that light from the display source is partially reflected while the remaining light is transmissible.Such an optical design can enable a user to view both the content (e.g., digital images, user interface elements, virtual content, etc.) generated by the display device 104 and physical items in the real world, for example, through the lens 107, adjacent to each other. In some implementations, waveguide optics may be used to depict the content on the display device 104.

[0019] In some implementations, the wearable computing device 100 may include a gaze tracking device 120 that includes, for example, one or more sensors 125 to detect and track the gaze direction and movement. The data captured by the sensors 125 may be processed to detect and track the gaze direction and movement as user input. In some implementations, the sensing system 111 may include various sensing devices, and the control system 112 may include various control system devices including, for example, one or more processors 114 operably coupled to components of the control system 112. In some implementations, the control system 112 may include a communication module that provides communication and exchange of information between the wearable computing device 100 and other external devices.

[0020] When sizing and fitting a wearable computing device 100, such as the exemplary smart glasses shown in FIGS. 1A-1C, for a particular user, many different sizing and fitting measurements and / or parameters may be considered. This may include, for example, wearable fit parameters or wearable fit measurements. Wearable fit parameters / measurements may consider how a particular frame 102 fits and / or looks and / or feels on a particular user. Wearable fit parameters / measurements may consider many factors, such as whether the frame 102 is wide enough to be comfortable with respect to the temples while not being so wide that the frame 102 cannot remain relatively stationary when worn by the user (i.e., too large or too wide). Wearable fit parameters / measurements may consider other factors, such as whether the rim portion 103 and the bridge portion 109 are sized such that the bridge portion 109 can rest comfortably on the user's nasal bridge of the appropriate width, whether the arm portion 105 is sized to rest comfortably on the user's ears, and other such comfort-related considerations. Wearable fit parameters / measurements may consider donning considerations, including, for example, how the user naturally dons the frame 102, such as head posture, how the user naturally holds their head, and how the user positions the frame relative to their face. Wearable fit parameters / measurements may consider whether the size and / or shape and / or contour of the frame 102 is aesthetically pleasing to the user and whether it matches the user's facial features. In some examples, wearable fit parameters / measurements may consider whether the fit associated with a particular frame configuration can provide for the incorporation of a display device such that the display area can be harmonized with the user's natural field of view. In some examples, wearable fit parameters may consider whether an eye-tracking camera array associated with a particular frame configuration can effectively track the user's line of sight and movement.

[0021] Display fit parameters or display fit measurements may be considered when sizing and fitting the wearable computing device 100 for a particular user. The display fit parameters / measurements may be used to configure the display device 104 for a set of frames 102 for a particular user such that the content displayed by the display device 104 is visible to the user. For example, the display fit parameters / measurements may provide an indication of the placement of the display device 104 such that the content displayed by the display device 104 is captured within at least a set portion of the user's field of view. For example, the display fit parameters / measurements may be used to configure the display device 104 to provide at least a set level of gazability corresponding to the amount, or portion, or percentage of content visible to the user at a set luminance level and set pupil size at the periphery of the user's field of view (e.g., at least the visible corners). The display fit parameters / measurements may be used to configure the display device 104 to provide the best possible placement of content display for the user for a particular set of frames 102.

[0022] In some examples, ophthalmic fit parameters or ophthalmic fit measurements may be considered when sizing and fitting a wearable computing device 100 that includes a prescription or corrective lens 107. Some exemplary ophthalmic fit measurements are shown in FIGS. 2A-2D. Ophthalmic fit measurements may include, for example, pupil height PH (the distance from the center of the pupil to the bottom of the lens 107), including, for example, left pupil height and right pupil height. Ophthalmic fit measurements may include interpupillary distance IPD (the distance between the pupils). The IPD may be characterized by monocular pupil distances, for example, left pupil distance LPD (the distance from the central portion of the bridge of the nose to the left pupil) and right pupil distance RPD (the distance from the central portion of the bridge of the nose to the right pupil). Ophthalmic fit measurements may include pantoscopic angle PA (the angle defined by the tilt of the lens 107 relative to vertical). Ophthalmic fit measurements may include vertex distance V (the distance from the cornea to the lens 107), including, for example, left vertex distance and right vertex distance. Ophthalmic fit measurements may include other such parameters or measurements that provide sizing and / or fitting of the head-mounted wearable computing device 100 that includes the display device 104 as described above. FIG. 2C shows vertex distance V associated with a relatively lower power lens 107. FIG. 2D shows vertex distance V associated with a relatively higher power lens 107. Ophthalmic fit measurements may be considered for the user when fitting a wearable computing device 100 that includes the display device 104. For example, ophthalmic fit measurements (along with display fit measurements) may provide for the placement of content display by the display device 104 within an eyebox defined by a three-dimensional volume extending between the lens 107 and the user's eye such that the content is within the user's field of view and within the user's corrected field of view and thus visible to the user.

[0023] In the systems and methods according to the implementations described herein, image data may be captured via an application executed on a computing device operated by a user, such as the computing device 200 described above with respect to FIG. 1A or other computing devices operated by the user. Wearable fit measurements and / or display fit measurements and / or ophthalmic fit measurements may be detected from image data acquired in this format to size and fit a head-mounted wearable computing device, such as the wearable computing device 100 described above, including a display and / or corrective lenses for a particular user. Image data may be captured via an application executed on a computing device 200 operated by a user in an unsupervised or unmonitored format, such as without the need to access a retail store or make an appointment with a fitting / sales professional. In some implementations, the detection of wearable fit and / or display fit and / or ophthalmic fit measurements may be based on the detection of physical characteristics or traits of a sample frame worn by the user included in the image data. A three-dimensional model of the user's head may be generated, or created, or defined based on measurements detected within the captured image data (e.g., measurements of the user's physical characteristics). In some implementations, the detection of physical characteristics and / or traits in the image data may be provided using the image sensor or camera of the computing device 200 without relying on data collected by a depth sensor. In some implementations, a depth sensor included in the computing device 200 may provide the collection of three-dimensional measurements related to the detected physical characteristics and / or traits of the sample frame and / or the user's face / cranial features.

[0024] Figure 3 is a block diagram of an exemplary system for determining display fit measurements and ophthalmic fit measurements for a wearable computing device, according to the implementations described herein. The system may include one or more computing devices 300. The computing device 300 can selectively communicate via a network 306 to access external resources 302 such as, for example, a server computer system, a processor, a database, a memory storage, and the like. The computing device 300 can operate under the control of a control system 370. The computing device 300 can communicate directly (via wired and / or wireless communication) or via the network 306 with one or more external computing devices 304 (such as another wearable computing device, another mobile computing device, etc.). In some implementations, the computing device 300 includes a communication module 380 to facilitate external communication. In some implementations, the computing device 300 includes a sensing system 320 that includes various sensing system components such as, for example, one or more image sensors 322, one or more position / orientation sensors 324 (including, for example, an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer, etc.), one or more audio sensors 326 capable of detecting an audio input, and other such sensors. The computing device 300 can include more or fewer sensing devices and / or combinations of sensing devices.

[0025] In some implementations, computing device 300 may include one or more image sensors or cameras 360. Camera 360 may include, for example, an outward-facing camera, a world-facing camera, etc. that can capture still images and / or videos of the environment outside computing device 300. In some implementations, one or more cameras 360 may include a depth sensor. The still images and / or videos may be displayed by a display device of output system 340 and / or transmitted externally via communication module 380 and network 306, and / or stored in memory 330 of computing device 300. Computing device 300 may include one or more processors 350. Processor 350 may include various modules or engines configured to perform various functions. In some examples, processor 350 includes an object recognition module, a feature recognition module, a pattern recognition module, a configuration identification module, and other such processors. Processor 350 may be formed on a substrate configured to execute one or more machine-executable instructions or pieces of software, firmware, or combinations thereof. Processor 350 can be semiconductor-based, including semiconductor material capable of performing digital logic. Memory 330 may include any type of storage device that stores information in a format readable and / or executable by processor 350. Memory 330 may store applications and modules that perform an operation when executed by processor 350. In some examples, the applications and modules may be stored on an external storage device and loaded into memory 330.

[0026] Figures 4A and 4B show an exemplary computing device, such as computing device 300 described above with respect to FIG. 3, that is operated by a user to capture display fit measurements and / or ophthalmic fit measurements for a wearable computing device customized for use by the user. In FIG. 4A, exemplary computing device 300A is in the form of a handheld computing device, such as a smartphone. In FIG. 4B, exemplary computing device 300 is in the form of a laptop computing device 300B. Figures 4A and 4B provide two examples of computing device 300 that may be used to capture image data for processing by a sizing simulator to implement the systems and methods described herein. The principles described herein may be implemented by other types of computing devices, particularly those on which applications can be executed that utilize image capture functionality, display functionality, and external communication and processing to collect and process display fit measurements and / or ophthalmic measurements for customizing a wearable computing device for a particular user.

[0027] In the example shown in FIG. 4A, the user holds the computing device 300A, whereby the user's head and face are within the field of view of the camera 360A of the computing device 300A, and the camera 360A can capture an image of the user's head and face. The captured image may be displayed to the user on the display device 342A of the computing device 300A, whereby the user can confirm that their head and face are captured within the field of view of the camera 360A. Similarly, as shown in FIG. 4B, the computing device 300B is positioned relative to the user such that the user's head and face are captured within the field of view of the camera 360B of the computing device 300B, and the camera 360B can capture an image of the user's head and face. The captured image may be displayed to the user on the display device 342B of the computing device 300B, whereby the user can confirm that their head and face are captured within the field of view of the camera 360B. In the exemplary arrangements shown in FIGS. 4A and 4B, the image data captured by the camera 360 may be processed (e.g., by the recognition engine and simulation engine of an external processor as shown in FIG. 3) to detect wearable fit and / or display fit and / or ophthalmic fit measurements and to determine, for example, frame and / or lens sizing and contouring, display device configuration, etc. for the customization of wearable computing devices such as the head-mounted wearable computing device 100 shown in FIGS. 1B and 1C for the user.

[0028] In the examples shown in FIGS. 4A and 4B, the user is wearing the exemplary fitting frame 500. The fitting frame 500 is included in the image of the user's head and face captured by the camera 360 of the computing device 300.

[0029] As described above, the detection of wearable fit and / or display fit and / or ophthalmic fit measurements from captured image data can be facilitated by the detection of one or more physical features and / or characteristics associated with a fitting frame worn by the user during the capture of the image data. In some examples, the user may try on or sample a number of different fitting frames from a collection or kit of sample fitting frames. A fitting kit including sample fitting frames may be provided to the user prior to a fitting session. Thereby, the user may evaluate at least some wearable fit parameters. For example, by trying on or sampling a number of different fitting frames from a fitting kit including sample fitting frames, the user may evaluate factors such as physical sizing, comfort, aesthetics, etc. The user may select a fitting frame from a fitting kit including sample fitting frames for use during a fitting session. The sample fitting frames included in the fitting kit may mimic the size, shape, weight, etc. of the actual frame to be incorporated into the head-mounted wearable computing device 100 based on data captured during the fitting session to provide the user with a relatively accurate fit and feel. The image data captured via an application operating on the computing device 200 during the fitting session may be used to determine the display fit and / or ophthalmic fit. The measurement data collected in this manner may be used to adjust the image display area generated by the display device 104, particularly for the user and the selected fitting frame, based on the measurements detected from the image data. The measurement data collected in this manner may be used to incorporate corrective or prescription lenses into the head-mounted wearable computing device 100.The measurement data collected in this format may be used to configure the display device 104 for the user and the selected frame and to incorporate corrective / prescription lenses into the head-mounted wearable computing device 100.

[0030] Figures 5A - 5C show exemplary fitting frames 500, such as a first exemplary fitting frame 500A, a second exemplary fitting frame 500B, and a third exemplary fitting frame 500C, for fitting a head - mounted wearable computing device according to implementations described herein. The exemplary fitting frame 500 may physically or geometrically represent the actual frame of the head - mounted wearable computing device for being worn by a user, but is for sizing / fitting purposes only and is thus non - functional. The exemplary fitting frames 500 shown in Figures 5A - 5C are examples of sample fitting frames 500 that may be included in a fitting kit provided to a user for consideration prior to a fitting session. The principles described herein are applicable to fitting frames having other shapes and / or sizes and / or configurations. The first exemplary fitting frame 500A shown in Figure 5A has a first size and / or shape and / or contour defined by a first set of physical characteristics. The first exemplary fitting frame 500A may be considered an intermediate - sized frame with D - shaped edge portions 503A, 503B. The second exemplary fitting frame 500B shown in Figure 5B has a second size and / or shape and / or contour defined by a second set of physical characteristics. The second exemplary fitting frame 500B may be considered an intermediate - sized frame with rounded edge portions 503A, 503B. The third exemplary fitting frame 500C shown in Figure 5C has a third size and / or shape and / or contour defined by a third set of physical characteristics. The third exemplary fitting frame 500C may be considered a large - sized frame with rectangular edge portions 503A, 503B. The exemplary fitting frames 500 (500A, 500B, 500C) shown in Figures 5A - 5C are presented for purposes of discussion and illustration.The principles described herein may be applied to other fitting frames having other sizes and / or shapes and / or contours defined by other physical characteristics.

[0031] Each of the exemplary fitting frames 500 (500A, 500B, 500C) shown in FIGS. 5A - 5C includes a first edge portion 503A surrounding a first glass portion or first lens 507A, and a second edge portion 503B surrounding a second glass portion or second lens 507B. When an exemplary fitting frame 500 (500A, 500B, 500C) is worn by a user, the first edge portion 503A / first lens 507A may be configured to be positioned to correspond to the user's first eye, and the second edge portion 503B / second lens 507B may be configured to be positioned to correspond to the user's second eye. The bridge portion 509 may extend between the inner ends of the edge portions 503A, 503B to connect the edge portions 503A, 503B of the exemplary fitting frame 500 (500A, 500B, 500C). The bridge portion 509 may be configured to be placed on the user's nose to position the first edge portion 503A / first lens 507A to correspond to the first eye and to position the second edge portion 503B / second lens 507B to correspond to the second eye. The first arm portion 505A may be coupled, e.g., pivotably or rotatably coupled, to the outer end of the first edge portion 503A by a first hinge portion 510A. The second arm portion 505B may be coupled, e.g., pivotably or rotatably coupled, to the outer end of the second edge portion 503B by a second hinge portion 510B. Each of the first and second arm portions 505A, 505B may include ear saddle portions 515A, 515B configured to be placed on the user's respective first and second ears when the exemplary fitting frame 500 (500A, 500B, 500C) is worn by the user.

[0032] As described above, the fitting frame according to the implementation described in this specification may include physical features or characteristics that can be detected, or identified, or recognized in the image data including the fitting frame when worn by the user. FIG. 6A shows an exemplary two-dimensional image plane 600 or image 600 of a user wearing an exemplary fitting frame 500. The exemplary image 600 may be captured by a camera 360 of a computing device 300 operated by the user via an application executed by the computing device 300. In the example shown in FIG. 6A, the image 600 is displayed on a display device 342 of the computing device 300. Thereby, the user can visually confirm that the captured image data includes the capture of the fitting frame 500, the user's face and / or skull and / or optical features, etc. In an example where the computing device 300 is a handheld mobile computing device such as a smartphone, a self-portrait mode or selfie mode may be activated for the capture of image data by the camera 360 of the computing device 300.

[0033] The image 600 may include a two-dimensional array of pixels captured by the camera 360. In some examples, the detected features within the image 600 may correspond to the positions of the detected features within the image 600, e.g., two-dimensional coordinate positions. FIG. 6B shows some examples of detectable keypoints or detectable features associated with the fitting frame 500 worn by the user while the image data is being captured. The detection of these types of keypoints or features in the image data may enable the determination of the three-dimensional pose of the fitting frame 500 on the user's face, and this three-dimensional pose itself may be used to determine display fit information and / or ophthalmic fit information associated with the fitting frame 500 when worn by the user. The detection of these types of keypoints associated with the fitting frame 500 worn by the user in the image data is important in the determination of display fit and / or ophthalmic fit information that enables the head-mounted wearable computing device 100 to be configured for the user.

[0034] The exemplary array shown in FIG. 6B includes an exemplary key point 610A at a position in the image 600 corresponding to the hinge joint location between the first edge portion 503A and the first arm portion 505A of the fitting frame 500, for example, at a position corresponding to the first hinge portion 510A that pivotally couples the edge portion 503A and the first arm portion 505A. The exemplary array shown in FIG. 6B includes an exemplary key point 610B at a position in the image 600 corresponding to the hinge joint location between the second edge portion 503B and the second arm portion 505B of the fitting frame 500. The exemplary array shown in FIG. 6B includes exemplary key points 620A at a position corresponding to the bottom peripheral portion of the first lens 507A of the fitting frame 500, and exemplary key points 620B at a position corresponding to the bottom peripheral portion of the second lens 507B of the fitting frame 500. The exemplary key points 620A, 620B are merely exemplary key points corresponding to specific peripheral portions of the lens 507. The key points may be defined further and / or alternatively at other peripheral portions of the lens 507, such as, for example, the upper peripheral portion, the left peripheral portion, and / or the right peripheral portion of the lens 507. In some examples, the key points defined along the peripheral portion of the lens 507 may correspond to a designated section of the lens 507. In some examples, the key points defined along the peripheral portion of the lens 507 may correspond to a specific location along the periphery of the lens 507. The exemplary array shown in FIG. 6B includes an exemplary key point 630 at a position in the image 600 corresponding to the bridge portion 509 of the fitting frame 500.

[0035] The exemplary image 600 shown in FIGS. 6A and 6B is a substantially frontal image, and the exemplary key points 610, 620, 630 or features 610, 620, 630 are simply detectable key points or features related to the fitting frame 500 worn by the user that can be detected in the image 600 to detect the display fit and / or ophthalmic fit for the user by way of the fitting frame 500. Some examples of other key points or features related to the fitting frame 500 may be detected in the image 600 to facilitate the determination of the display fit and / or ophthalmic fit. Examples of other detectable key points or detectable features may include, for example, the upper portion of the lens 507, the inner peripheral portion of the lens 507 (i.e., the portion of the lens 507 adjacent to the bridge portion 509 of the fitting frame 500), the outer peripheral portion of the lens 507 (i.e., the portion of the lens 507 adjacent to the hinge portion 510 of the fitting frame 500), and the like. The distribution of detectable key points or features across the fitting frame 500, and the separation of the key points or features, may improve the detection accuracy and the correlation with the three-dimensional model. In addition, other key points or features related to the fitting frame 500 may be detected in a side face image of the user, or in an image captured at a predetermined angle with respect to the user, which may include the position of the portion of the arm portion 505 of the fitting frame 500, the position of the ear saddle location 515 with respect to the user's ear, the position of the edge portion 503 with respect to the user's face, and the like. To determine the display fit and / or ophthalmic fit of the fitting frame 500 for the user, more or fewer key points and / or combinations of key points related to the fitting frame 500 may be detected in the image 600. As described above, the detection of specific key points related to the fitting frame 500 worn by the user in the image data is important in determining the display fit and / or ophthalmic fit information that enables the head-mounted wearable computing device 100 to be configured for the user.

[0036] FIG. 6C shows a plurality of face landmarks (along with the exemplary detectable keypoints 610, 620, 630 or features 610, 620, 630 described above with respect to FIG. 6B), and these face landmarks may be detected in the two-dimensional image 600 captured by the camera 360 of the computing device 300. In some examples, the pupils, particularly the pupil centers 640 of each of the user's eyes, may be detected in the image 600. In some examples, other face landmarks 650 may be detected in the image 600. In the example shown in FIG. 6C, some exemplary face landmarks 650 include, for purposes of discussion and illustration only, the bridge of the nose, the forehead, and the quadrants surrounding the bridge of the nose on the cheeks. Other face landmarks may be detected to facilitate identification of features related to the positioning of the fitting frame 500 on the user's head / face. The face landmarks 650 may provide, for example, indexing for positioning the fitting frame 500 relative to the user's head / face and the like.

[0037] In some implementations, as shown in FIG. 6D, a side or profile image 660 may be captured. The exemplary image 660 shown in FIG. 6D is a substantially complete profile image 660 for discussion and illustration purposes only. Information detected from the profile image 660 may be detected from a side image taken at an angle of less than 90 degrees from the front image 600 shown in FIGS. 6A - 6C. As shown in FIG. 6D, additional keypoints or features associated with the fitting frame, and additional face / skull landmarks may be detected in the side or profile image 660. In the example shown in FIG. 6D, the keypoint 670 may be detected at a position in the image 660 corresponding to the ear saddle portion 515A of the arm portion 505A of the fitting frame 500. The face / skull landmark 650 may be detected, for example, at a portion of the user's ear, such as at the wrinkle portion on the front side of the ear, the upper part of the ear, etc. Other keypoints and / or face / skull features may also be detected from the side / profile image 660. Similarly, for example, the tilt or angle of the fitting frame 500 with respect to the user's face (e.g., the forward tilt angle PA) may be detected from the side or profile image 660. That is, the angle between the edge portion 503 and the arm portion 505 of the fitting frame 500 may be known, but the tilt or angle of the fitting frame 500 varies based on many factors. These factors may include, for example, how the user wears the fitting frame 500 (closer to / farther from the eyes, higher / lower on the nose, etc.), the positioning of the ears relative to the user's nose, and other such factors.

[0038] In some implementations, the detected keypoints associated with the fitting frame 500 (such as the exemplary detected keypoints 610, 620, 630, 670 or features 610, 620, 630, 670 shown in FIGS. 6B - 6D), alone and / or together with the detected face / skull features of the user (such as the detected pupil center 640 and face landmarks 650 shown in FIGS. 6C and 6D), may be used to determine the pose of the fitting frame 500 on the user's face, e.g., a three - dimensional pose. This three - dimensional pose of the fitting frame 500 on the user's face may be used, for example, to determine the display fit and / or ophthalmic fit for the head - mounted wearable computing device 100 in the form of smart glasses. In some examples, the configuration of the frame 102 of the head - mounted wearable computing device 100 being configured corresponds to the configuration of the fitting frame 500 worn by the user in the image 600. In some examples, the configuration of the frame 102 of the head - mounted wearable computing device 100 being configured does not necessarily correspond to the configuration of the fitting frame 500.

[0039] In some examples, for example, the two - dimensional front - facing image 600 of the fitting frame 500 on the user's face and the data extracted from the corresponding detected three - dimensional pose may be synthesized to evaluate the gazeability of the fitting frame 500 for the user. That is, this data may be synthesized to determine that the configuration of the display device 104 included in the head - mounted wearable computing device 100 enables the display device 104 to be configured such that the content output by the display device 104 is within the field of view of the user wearing the head - mounted wearable computing device 100. For example, the display device 104 may be configured such that the content output by the display device 104 and displayed at the output coupler 144 is captured within the user's field of view and visible to the user.

[0040] As described above, the three-dimensional pose information related to the pupil (e.g., pupil center 640) and the user's face (e.g., detected face landmarks 650) can, together with the three-dimensional pose information of the fitting frame 500, enable the determination of lens-related measurements (e.g., pupil height, vertex distance, tilt angle, etc.). As described above, the three-dimensional position and orientation, or pose, of the fitting frame 500 may be determined based on the detection and / or identification of one or more keypoints 610, 620, 630 or features 610, 620, 630 in the image data captured by the camera 360 of the computing device 300 operated by the user. In some examples, additional information may be obtained from the side or profile image 660, as described above with respect to FIG. 6D. The fitting frame 500 may have a slight bend, but in some examples, the fitting frame 500 may be considered a substantially rigid body with three degrees of freedom. Thus, relatively accurate detection of at least one known location on the fitting frame 500 may be used to determine the three-dimensional position of the fitting frame 500. One or more of the keypoints 610, 620, 630 or features 610, 620, 630 may be reliably detectable in the image 600, and the positions of the detected one or more keypoints 610, 620, 630 or features 610, 620, 630 are known in the three-dimensional space related to the fitting frame 500.

[0041] As described above, the systems and methods according to the implementations described herein can facilitate the fitting of a head-mounted wearable computing device based on image data captured via an application operating on a computing device operated by a user. The image data may be captured by the user, and the wearable computing device may be fitted and configured for the user in an autonomous, unmonitored, or unsupervised manner without the need for access to a retail store and / or an in-person or virtual appointment with a sales agent. The capture and processing of the image data are described in more detail below.

[0042] A user desiring to select, fit, and configure a head-mounted wearable computing device 100 as described above may have access to, or be provided with, one or more fitting frames or a sample set of fitting frames for consideration prior to the fitting process. In some examples, the user may try on each fitting frame included in the sample set of fitting frames to evaluate factors such as the size, comfort, appearance, compatibility with facial features, overall wearability, etc. associated with each. The user may select a fitting frame 500 from the sample set of fitting frames based on these and / or other factors and use it in the fitting and configuration process.

[0043] In some examples, a sample set of fitting frames to be tried on and / or evaluated may be selected by a user from a plurality of fitting frames available for trying on. In some examples, a system (e.g., from within a module of an application executed on a computing device 300 operated by a user) may propose a sample set of fitting frames to the user based on an analysis of the user's physical characteristics and the shape and / or size and / or configuration of frames that may fit the detected physical characteristics. In some examples, the sample set of fitting frames may be provided to the user before initiating a process for fitting and configuring the head-mounted wearable computing device 100.

[0044] The ability to physically try on one or more pairs of fitting frames 500 provides the user with an opportunity to physically evaluate the fit in order to make determinations such as which fitting frame 500 is physically the most comfortable and whether it properly / comfortably aligns with facial and / or cranial features (eyes, bridge of the nose, ear saddle areas, cheek contact areas, etc.). This evaluation of the wearable fit by the user, utilizing a physical fitting frame 500 prior to the fitting and configuration of the head-mounted wearable computing device 100, may result in a head-mounted wearable computing device 100 that is more suitable for a particular user. The selection of the fitting frame 500 in this manner, based on the physical wearable fit by the user, may provide a more accurate fitting and configuration of the head-mounted wearable computing device 100 for the user.

[0045] In some situations, a fitting frame 500 may be available to the user for fitting and configuring the head-mounted wearable computing device 100. In this situation, the fitting and configuration process may be performed as described below using additional means for determining the wearable fit of a particular frame 102 for incorporation into the head-mounted wearable computing device 100.

[0046] In some examples, the user may operate a computing device 300 (such as in FIGS. 4A and / or 4B) to launch an application and initiate fitting and configuring the head-mounted wearable computing device 100 using a selected fitting frame 500 (such as selected from a sample set of fitting frames as described above). In some examples, the application executed on the computing device 300 operated by the user may prompt the user to initiate image capture while wearing the selected fitting frame 500. Thereafter, this process will be described based on the capture of the front image 600, as described above with respect to FIGS. 6A-6C, merely to facilitate discussion and illustration. However, in some implementations, data detected in a profile image 660 and / or a semi-profile image as shown in FIG. 6D may be used in fitting and configuring the head-mounted display device 100 for the user.

[0047] In some examples, the application may prompt the user to confirm the selection of a particular fitting frame 500 worn by the user during capture of the image data. Thus, in some situations, configuration information regarding the selected fitting frame 500 worn during image capture may be known. This may include, for example, linear and / or angular measurements and / or contours regarding the selected fitting frame 500, two-dimensional and / or three-dimensional models of the selected fitting frame 500, and other such information. This known information regarding the selected fitting frame 500 may be accessible to an application executing on the computing device 300. For example, this known configuration information may be stored in a database accessible to an application executing on the computing device 300. The known configuration information regarding the selected fitting frame may further enhance the accuracy and efficiency of the fitting and configuration of the head-mounted wearable computing device 100 for the user.

[0048] As shown in FIG. 7 and as described above, various key points or features related to the fitting frame 500 can be detected within the front image 600 captured by the camera 360 of the computing device 300 operated by the user. In the exemplary arrangement shown in FIG. 7, the first key point 610A is disposed at a first known position 710A on the fitting frame 500 corresponding to the hinge joint location between the first edge portion 503A and the first arm portion 505A of the fitting frame 500. Similarly, the second key point 610B is disposed at a second known position 710B on the fitting frame 500 corresponding to the hinge joint location between the second edge portion 503B and the second arm portion 505B of the fitting frame 500. The third key point 620A is disposed at a third known position 720A on the fitting frame 500 corresponding to the bottom peripheral portion of the first lens 507A, and the fourth key point 620 is disposed at a fourth known position 720B on the fitting frame 500 at a position corresponding to the bottom peripheral portion of the second lens 507B. The fifth key point 630 is disposed at a fifth known position 730 on the fitting frame 500 at a position corresponding to the central portion of the bridge portion 509 of the fitting frame 500. Detection of the key points 610, 620, 630 and identification of the corresponding known positions 710, 720, 730 on the fitting frame 500 may provide fixed reference points on the fitting frame 500 that may be used to detect and / or refine the three-dimensional pose of the fitting frame 500 on the user's face. The detected features or traits associated with each of the detected key points 610, 620, 630 can be compared to known features or traits (known positions, geometries, contours, distances, etc.) to, for example, determine the depth with respect to the camera 360. This can facilitate determination of the three-dimensional pose of the fitting frame 500 on the user's face.

[0049] For example, the detected keypoints 610, 620, 630 on the fitting frame 500 and the corresponding known positions 710, 720, 730 may be matched to corresponding sets of locations / places in the three-dimensional model of the fitting frame 500. As described above, the three-dimensional model of the fitting frame 500 may be pre-stored and accessible via an application executed on the computing device 300. Thus, the known set of correspondences between the three-dimensional model of the fitting frame 500 and the corresponding places detected in the image 600 may be used to determine the three-dimensional pose of the fitting frame 500.

[0050] In some implementations, detected face / skull landmarks such as the exemplary landmarks 650 shown in FIGS. 6A-6D may facilitate the detection and identification of keypoints or features of the fitting frame 500. For example, the detection and identification of the bottom peripheral portion of the first lens 507 that defines the exemplary keypoint 620A described above may be compacted or improved, and / or the accuracy may be increased based on the detection below the keypoint 620A of the landmark 650 that defines the user's cheek, the landmark 650 to the right of the keypoint 620A that defines the bridge of the user's nose, etc. Other keypoints defined on other peripheral portions of the lens 507 (i.e., the upper and / or left and / or right peripheral portions of the lens 507) may be compacted and / or improved based on the detection of other landmarks 650 in a similar fashion.

[0051] In some implementations, the detection of key points or features associated with the fitting frame 500 worn by the user in the image 600 may rely on a comparison against a database to which synthetic data, actual data, and data augmented with a combination of synthetic and actual data have been added. The synthetic data may include, for example, three-dimensional renderings of glasses and three-dimensional renderings of the head in different combinations of poses, lighting conditions, etc., which may be combined to generate an exceptionally large number of different rendered combinations of glasses worn on the head under various conditions. The actual data may include, for example, image data capturing actual people wearing glasses. The synthetic data and the actual data may be combined to generate additional combinations of glasses on the head in different poses, conditions, etc. This data (synthetic data, actual data, and combined data) may be accumulated to train a neural network, which can then continuously output improved results in the matching of key points on the fitting frame to locations in the three-dimensional model.

[0052] In some situations, a well-developed / densely augmented neural network may be available for a particular fitting frame 500, including training data representing a relatively large number of poses, positions, and angles of the fitting frame 500. In this situation, the front image 600, described above with respect to FIGS. 6A - 6C, for example, may be fed into the neural network established for that fitting frame 500, and the correspondence between the image 600 of the user wearing the fitting frame 500 and the data added to the neural network can be detected without necessarily requiring the detection of key points in the image 600 prior to the processing of the image 600 by the neural network.

[0053] In some implementations, a process similar to the fitting and configuration process may be applied to the calibration or recalibration of the head-mounted wearable computing device 100 after the head-mounted wearable computing device 100 has been delivered to the user. For example, a new product setup process for the head-mounted wearable computing device 100 configured for the user may include fine-tuning of the display characteristics after the head-mounted wearable computing device 100 has been delivered. For example, an application executed on the computing device 300 operated by the user may prompt the user to capture image data including the user wearing the head-mounted wearable computing device 100. Detection of key points on the head-mounted wearable computing device 100 may be detected to determine the three-dimensional pose of the head-mounted wearable computing device 100 relative to the user's face and to the user's eyes / eyeboxes. The control software that controls the operation of the display device 104 may adjust the output of the content, for example, to optimize where the content is displayed. This may optimize gazeability and improve the user viewing experience.

[0054] The measurement values or parameters described above may be collected to determine display fit characteristics for a user wearing the selected fitting frame 500. The measurement values or parameters described above may enable fitting of a display device (such as the display device 104 of the head-mounted wearable computing device 100 shown in FIGS. 1B and 1C) of a head-mounted wearable computing device such that the user's pupils are aligned as centrally as possible within the eye box 140, whereby the user's line of sight passes through the output coupler 144 and the content displayed therein is visible to the user on the lens 107. The measurement values or parameters described above may enable the configuration of the display device 104 of the head-mounted wearable computing device 100 such that the display of content is visible to the user within a relatively large portion of the user's field of view.

[0055] The systems and methods according to the implementations described herein may provide for the collection of measurement values for determining ophthalmic fit characteristics for a user wearing the selected fitting frame 500. In some implementations, the systems and methods may provide for the collection of measurement values for determining both display fit characteristics and ophthalmic fit characteristics. As described above with respect to FIGS. 2A-2D, the ophthalmic measurement values may be detected within image data captured by the camera 360 of the computing device 300 operated by the user. For example, these measurement values, including pupil height, interpupillary distance, monocular pupillary distance, tilt angle, vertex distance, and other such parameters or measurement values, may be detected within the image data and processed for correction to the head-mounted wearable computing device 100 or incorporation of prescription lenses. In some examples, these ophthalmic measurement values, together with the user's correction / prescription lens needs, are used in cooperation with the display fit measurement values described above to provide both an appropriate display fit for a particular user.

[0056] FIG. 8 is a flowchart of an exemplary method 800 of operating a computing system to configure a head-mounted wearable computing device that includes a display for a user. A user operating a computing device (e.g., the computing device 300 described above) may cause an application to execute on the computing device. The application may provide for detection of display fit measurements and / or ophthalmic fit measurements from image data captured by the computing device (block 810). A camera (e.g., the camera 360 of the computing device 300 described above) is operated to capture image data of a user wearing a fitting frame (block 820). The fitting frame may be selected from one or more sample fitting frames available to the user (e.g., the fitting frame 500 described above with respect to FIGS. 5A-7). The system may detect one or more keypoints or features (e.g., the keypoints 610, 620, 630 and / or 670, face / skull landmarks 640, 650, etc. described above) on the fitting frame (block 830).

[0057] In some examples, the analysis of the image capture data for detection of keypoints and / or features and / or landmarks may be performed by an object recognition module and / or a pattern recognition module of a processor 350 of a computing device, e.g., the computing device described above. In some examples, the detection of keypoints and / or features and / or landmarks may be performed by an external device, such as an object recognition module and / or a pattern recognition module of a server included in an external resource 302 that communicates with a computing device 300 as described above.

[0058] Display fit measurements including the three-dimensional pose of a fitting frame may be determined based on the positions of detected keypoints and / or features and / or landmarks relative to the configuration of the fitting frame (block 840). Ophthalmic fit measurements may be detected from the image data and from the display fit measurements, if necessary (block 850), for a user who requires that a corrective / prescription lens be incorporated into a head-mounted wearable computing device (blocks 860, 870, and 880). In some examples, the analysis for determining the display fit measurements and / or the ophthalmic fit measurements may be performed by a computing device (such as a configuration identification module of a processor 350 of the computing device 300 described above). In some examples, the analysis for determining the display fit measurements and / or the ophthalmic fit measurements may be performed by an external computing device (such as a configuration identification module of a server included in an external resource 302 that communicates with the computing device 300 described above). The ophthalmic fit measurements and the display fit measurements may be incorporated into a head-mounted wearable computing device for the user to provide a viewing experience adjusted for the user's optical needs.

[0059] Figure 9 shows examples of a computing device 900 and a mobile computing device 950 that may be used with the techniques described herein. Computing device 900 is intended to represent various forms of digital computers, such as a laptop, desktop, tablet, workstation, personal digital assistant, smart device, appliance, electronic sensor-based device, television, server, blade server, mainframe, and other appropriate computing devices. Mobile computing device 950 is intended to represent various forms of mobile devices, such as a personal digital assistant, cellular phone, smartphone, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described in this document and / or claimed.

[0060] Computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and the storage device 906. The processor 902 can be a semiconductor-based processor. The memory 904 can be a semiconductor-based memory. Each of the components 902, 904, 906, 908, 910, and 912 is interconnected using various buses and may be mounted on a common motherboard or in other forms as required. The processor 902 can process instructions for execution within the computing device 900, including instructions stored in the memory 904 or the storage device 906 for displaying graphical information for a GUI on an external input / output device such as a display 916 coupled to the high-speed interface 908. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory as required. Also, multiple computing devices 900 may be connected to each device that provides a portion of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0061] The memory 904 stores information within the computing device 900. In one implementation, the memory 904 is one or more volatile memory units. In another implementation, the memory 904 is one or more non-volatile memory units. The memory 904 may be another form of computer-readable medium, such as a magnetic or optical disk. Generally, the computer-readable medium may be a non-transitory computer-readable medium.

[0062] Storage device 906 can provide mass storage for computing device 900. In one implementation, storage device 906 can be or include a computer-readable medium such as a floppy disk device, hard disk device, optical disk device, or tape device, flash memory or other similar solid state memory device, or an array of devices, including devices or other configurations in a storage area network. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods and / or ways of performing operations by a computer, such as those described above. The information carrier is a computer-readable medium or machine-readable medium such as memory 904, storage device 906, or memory on processor 902.

[0063] High-speed controller 908 manages bandwidth-intensive operations for computing device 900, while low-speed controller 912 manages lower-bandwidth-intensive operations. Such a functional assignment is merely exemplary. In one implementation, high-speed controller 908 is coupled to high-speed expansion port 910, which can receive memory 904, display 916 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In an implementation, low-speed controller 912 is coupled to storage device 906 and low-speed expansion port 914. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, or networking devices such as a switch or router, e.g., via a network adapter.

[0064] Computing device 900 may be implemented in many different forms, as shown. For example, computing device 900 may be implemented as a standard server 920 or multiple times in a group of such servers. Computing device 900 may be implemented as part of a rack server system 924. Additionally, computing device 900 may be implemented in a computer such as a laptop computer 922. Alternatively, components from computing device 900 may be combined with other components in a mobile device (not shown) such as device 950. Each of such devices may include one or more of computing devices 900, 950, and the overall system may be composed of a number of computing devices 900, 950 that communicate with each other.

[0065] Computing device 950 includes, among other components, a processor 952, a memory 964, input / output devices such as a display 954, a communication interface 966, and a transceiver 968. Device 950 may be provided with a storage device such as a microdrive or other device to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 are interconnected using various buses, and a plurality of the components may be mounted on a common motherboard or in other forms as required.

[0066] Processor 952 can execute instructions within computing device 950, including instructions stored in memory 964. The processor may be implemented as a chipset of separate and multiple analog and digital processors. The processor may provide coordination of other components of device 950, such as, for example, control of the user interface, applications operated by device 950, and wireless communication by device 950.

[0067] Processor 952 may communicate with a user via a control interface 958 and a display interface 956 coupled to a display 954. The display 954 may be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other suitable display technology. The display interface 956 may include appropriate circuitry for driving the display 954 to provide graphical and other information to the user. The control interface 958 may receive commands from the user and convert them for submission to the processor 952. Additionally, an external interface 962 may be provided to communicate with the processor 952 to enable near area communication of the device 950 with other devices. The external interface 962 may provide, for example, wired communication in some implementations, or wireless communication in other implementations, and multiple interfaces may be used.

[0068] Memory 964 stores information within computing device 950. Memory 964 can be implemented as one or more of one or more computer-readable media, one or more volatile memory units, and one or more non-volatile memory units. Extended memory 984 may be provided and connected to device 950 via an expansion interface 982 that may include, for example, a SIMM (Single In-line Memory Module) card interface. Such extended memory 984 may provide additional storage space for device 950 or store applications or other information for device 950. In particular, extended memory 984 may include instructions for executing or supplementing the processes described above and may also include secure information. Thus, for example, extended memory 984 may be provided as a security module for device 950 and may be programmed with instructions that enable secure use of device 950. Additionally, secure applications may be provided with additional information, such as by placing identification information on the SIMM card in a non-hackable format via the SIMM card.

[0069] The memory may include, for example, flash memory and / or NVRAM memory as described below. In one implementation, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer-readable or machine-readable medium such as, for example, memory 964, extended memory 984, or memory on processor 952, which may be received on transceiver 968 or external interface 962.

[0070] Device 950 may communicate wirelessly via a communication interface 966 that may include a digital signal processing circuit if necessary. The communication interface 966 may provide communication under various modes or protocols, such as, among others, GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA (registered trademark), CDMA2000, or GPRS. Such communication may be performed, for example, via a radio frequency transceiver 968. In addition, short-range communication may be performed using Bluetooth, low-power Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 980 may provide additional navigation-related and location-related wireless data to device 950 as needed by an application operating on device 950.

[0071] Device 950 may also communicate audibly using a voice codec 960 that may receive voice information from a user and convert it into usable digital information. The voice codec 960 may similarly generate audible voice for the user, for example, through a speaker in the handset of device 950. Such voice may include voice from a voice telephone call, may include recorded voice (such as a voice message, a music file, etc.), and may also include voice generated by an application operating on device 950.

[0072] Computing device 950 may be implemented in a number of different forms as shown. For example, computing device 950 may be implemented as a mobile phone 983. Computing device 950 may also be implemented as part of a smartphone 981, a personal digital assistant, or other similar mobile device.

[0073] The various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that are executable and / or interpretable in a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device, which may be either special purpose or general purpose.

[0074] These computer programs (also known as modules, programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0075] To provide for interaction with a user, the systems and techniques described herein can be implemented in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or an LED (light emitting diode)) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and the input received from the user can be in any form including acoustic, voice, or tactile input.

[0076] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or includes middleware components (e.g., an application server), or includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), and the Internet.

[0077] A computing system can include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server is created by computer programs that operate on respective computers and have a client-server relationship with each other.

[0078] In some embodiments, the computing device shown in FIG. 9 can include sensors that interface with an AR / VR headset (AR glasses / glasses / VR headset / AR headset / HMD device 990). For example, one or more sensors included in the computing device 950 shown in FIG. 9 or other computing devices can provide input to the AR / VR headset 990 or generally provide input into the AR / VR space. The sensors can include, but are not limited to, touchscreens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 950 can use the sensors to determine the absolute position and / or detected rotation of the computing device in the AR / VR space for use as input into the AR / VR space. For example, the computing device 950 can be incorporated into the AR / VR space as a virtual object such as a controller, laser pointer, keyboard, weapon, etc. Positioning of the computing device / virtual object by the user when incorporated into the AR / VR space can enable the user to position the computing device in order to view the virtual object in a certain form in the AR / VR space.

[0079] In some embodiments, one or more input devices included in or connected to computing device 950 can be used as input into the AR / VR space. The input devices can include, but are not limited to, a touch screen, keyboard, one or more buttons, track pad, touch pad, pointing device, mouse, trackball, joystick, camera, microphone, earphone or earbud with input capabilities, gaming controller, or other connectable input devices. A user who interacts with an input device included in computing device 950 when the computing device is incorporated into the AR / VR space can cause a specific action to occur in the AR / VR space.

[0080] In some embodiments, one or more output devices included in computing device 950 can provide output and / or feedback to a user of AR / VR headset 990 in the AR / VR space. The output and feedback can be visual, tactile, or auditory. The output and / or feedback can include, but are not limited to, rendering the AR / VR space or virtual environment, vibration, turning one or more lights or strobes on and off or blinking and / or flashing, sounding an alarm, chiming, playing a song, playing an audio file. The output devices can include, but are not limited to, a vibration motor, vibration coil, piezoelectric device, electrostatic device, light emitting diode (LED), strobe, and speaker.

[0081] In some embodiments, computing device 950 can be disposed within AR / VR headset 990 to generate an AR / VR system. AR / VR headset 990 can include one or more positioning elements that enable the placement of computing device 950, such as smartphone 981, at an appropriate location within AR / VR headset 990. In such embodiments, the display of smartphone 981 can render a stereoscopic image representing an AR / VR space or virtual environment.

[0082] In some embodiments, computing device 950 may appear as another object in a computer-generated 3D environment. Interaction by a user with computing device 950 (e.g., rotating, vibrating, touching the touch screen, swiping a finger along the touch screen) can be interpreted as interaction with an object in the AR / VR space. As just one example, the computing device can be a laser pointer. In such an example, computing device 950 appears as a virtual laser pointer in a computer-generated 3D environment. When the user operates computing device 950, the user in the AR / VR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 950 in the AR / VR environment in computing device 950 or AR / VR headset 990.

[0083] In some embodiments, computing device 950 may include a touch screen. For example, a user can interact with the touch screen in a specific manner that mimics what occurs on the touch screen by what occurs in the AR / VR space. For example, the user may use a pinching motion to zoom in on content displayed on the touch screen. This pinching motion on the touch screen can zoom in on the information provided in the AR / VR space. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment. In the AR / VR space, the pages of the book can be displayed in the AR / VR space, and a user's finger swipe along the touch screen can be interpreted as turning / flipping the pages of the virtual book. As each page is turned / flipped, in addition to seeing the page content change, the user may be provided with audio feedback such as the sound of the page change in the book.

[0084] In some embodiments, in addition to the computing device, one or more input devices (e.g., a mouse, a keyboard) can be rendered in a computer-generated 3D environment. The rendered input devices (e.g., a rendered mouse, a rendered keyboard) can be used as rendered in the AR / VR space to control objects in the AR / VR space.

[0085] Numerous embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of the invention.

[0086] In addition, the logic flows depicted in the figures do not require the specific order, or sequence, shown to achieve the desired result. In addition, other steps may be provided, or steps may be eliminated from the described flow, and other components may be added to, or removed from, the described system. Accordingly, other embodiments are included within the scope of the appended claims.

[0087] In addition to the above description, the user is provided with controls that enable the user to make choices regarding whether and when the systems, programs, devices, networks, or features described herein may enable the collection of user information (e.g., the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location), and whether the user receives content or communications from the server. In addition, certain data may be processed in one or more ways before being stored or used so that user information is removed. For example, the user's identity may be processed so that the user information cannot be determined for the user, or the user's geographical location may be generalized (to the city, zip code, or state level) where location information is obtained, such that the specific location of the user cannot be determined. Accordingly, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

[0088] A computer system (e.g., a computing device) may be configured to wirelessly communicate with a network server over a network via an established communication link with the network server using any known wireless communication technologies and protocols, including radio frequency (RF), microwave frequency (MWF), and / or infrared frequency (IRF) wireless communication technologies and protocols adapted for communication over a network.

[0089] According to the disclosed embodiments, the implementation of the various techniques described herein may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or combinations thereof. The implementation may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium) for processing by a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers) or for controlling the operation thereof. In some implementations, the tangible computer-readable storage medium may be configured to store instructions that, when executed, cause the processor to execute a process. Computer programs, such as the computer programs described above, may be written in any form of programming language, including compiled languages or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program may be processed on one computer or multiple computers at one site or may be distributed and interconnected by a communication network across multiple sites so as to be processed.

[0090] The specific structural and functional details disclosed herein are merely representative for purposes of describing exemplary embodiments. However, the exemplary embodiments may be embodied in many alternative forms and should not be construed as limited to only the embodiments shown herein.

[0091] The terms used in this specification are for the purpose of describing particular embodiments only and are not intended to be limiting of the embodiments. As used in this specification, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes" and / or "including", when used in this specification, specify the presence of the stated feature, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0092] When an element is referred to as being "coupled to", "connected to", or "responsive to" another element, or as being "on" another element, the element can be directly coupled to, connected to, or responsive to, or on, the other element, or intervening elements may be present. In contrast, when an element is referred to as being "directly coupled to", "directly connected to", "directly responsive to", or "directly on" another element, intervening elements are absent. As used in this specification, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0093] Spatially relative terms such as "beneath", "below", "lower", "above", "upper", etc. may be used herein to facilitate description of one element or feature in relation to another element or feature as shown in the drawings. It will be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the drawings. For example, if a device in a drawing is turned over, an element described as "below" or "beneath" another element or feature will then be oriented "above" the other element or feature. Thus, the term "below" can include both upward and downward orientations. The device may be in other orientations (rotated 70 degrees or otherwise), and the spatially relative descriptors used herein may be interpreted accordingly.

[0094] Exemplary embodiments of the concepts are described herein with reference to cross-sectional views that are schematic illustrations of idealized embodiments (and intermediate structures) of the exemplary embodiments. As such, for example, variations from the shapes of the illustrations are to be expected as a result of manufacturing techniques and / or tolerances. Accordingly, the exemplary embodiments of the concepts described are not to be construed as limited to the particular shapes of the regions illustrated herein and include, for example, departures from the shapes resulting from manufacturing. Thus, the regions illustrated in the figures are essentially schematic and their shapes are not intended to illustrate the actual shape of a region of the device nor are they intended to limit the scope of the exemplary embodiments.

[0095] The terms "first", "second", etc. may be used herein to describe various elements, but it will be understood that these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a "first" element could be termed a "second" element without departing from the teachings of this embodiment.

[0096] Unless otherwise defined, all terms (including technical and scientific terms) used herein shall have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. Terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or this specification, and it will be further understood that they shall not be interpreted in an idealized or overly formal sense unless explicitly so defined herein.

[0097] Certain features of the described implementations are illustrated as described herein, but many modifications, alternatives, variations, and equivalents will now occur to those of ordinary skill in the art. Accordingly, the appended claims are intended to cover such modifications and variations as falling within the scope of the implementations. They are provided by way of example only and not of limitation, and it should be understood that various changes in form and detail may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described.

Claims

1. A method, comprising: detecting, by an application executed on a computing device, at least one feature on a fitting frame worn by a user from an image captured by the computing device; accessing configuration information related to the fitting frame detected in the image, wherein the configuration information includes physical features of the fitting frame, and the method further comprises: determining a three-dimensional pose of the fitting frame captured in the image based on the at least one feature and the detected configuration information related to the fitting frame, wherein the three-dimensional pose is determined with respect to features on the user's face, and the method further comprises: determining a configuration of a display device of a wearable computing device based on the three-dimensional pose of the fitting frame captured in the image; A method comprising the above.

2. Detecting the at least one feature comprises: detecting at least one of the bridge portion of the fitting frame, the hinge location between the edge portion and the arm portion of the fitting frame, the peripheral portion of the lens of the fitting frame, or the saddle portion of the arm portion of the fitting frame. The method according to claim 1.

3. Determining the three-dimensional pose of the fitting frame comprises: accessing a three-dimensional model of the fitting frame captured in the image; Comparing a known position and a known orientation of the at least one feature detected in the image with a corresponding position and a corresponding orientation of the at least one feature in the three-dimensional model of the fitting frame, and executing the comparison; The method according to claim 1 or claim 2, comprising: **Claim 4** Detecting at least one feature includes detecting a plurality of features on the fitting frame in the image captured by the computing device; Executing the comparison includes, for each of the plurality of features, comparing a known position and a known orientation of the feature detected in the image with a corresponding position and a corresponding orientation of the feature in the three-dimensional model of the fitting frame; The method according to claim 3. **Claim 5** Determining the three-dimensional pose of the fitting frame includes performing an association between a two-dimensional position of each feature detected in the image and a corresponding three-dimensional position of each feature in the three-dimensional model of the fitting frame. The method according to claim 4. **Claim 6** Detecting a plurality of face landmarks in the captured image; Determining ophthalmic fit measurements for the wearable computing device based on the detected plurality of face landmarks; The method according to claim 1, further comprising: **Claim 7** Detecting the plurality of face landmarks includes: Detecting a pupil height in the image of the fitting frame worn by the user; Detecting at least one of an interpupillary distance or a monocular pupil distance in the image of the fitting frame worn by the user; Determining a forward tilt angle of the fitting frame worn by the user based on the determined three-dimensional pose of the fitting frame and the detected plurality of facial landmarks; The method according to claim 6, comprising: **Claim 8** Determining the configuration of the display device of the wearable computing device based on the three-dimensional pose of the fitting frame captured in the image includes adapting the configuration of the display device of the wearable computing device to correspond to the ophthalmic fit measurement values. The method according to claim 6 or claim 7. **Claim 9** Determining the configuration of the display device of the wearable computing device includes: Determining a user's visual field based on the ophthalmic fit measurement values and the three-dimensional pose of the fitting frame; Configuring an output coupler of the display device such that content output by the display device is displayed within the determined user's visual field; The method according to claim 8, comprising: **Claim 10** Accessing the configuration information associated with the fitting frame includes accessing a three-dimensional model of the fitting frame. The method according to claim 1. **Claim 11** The three-dimensional model of the fitting frame includes at least one of known linear measurement values, known angular measurement values, or known contours associated with the fitting frame. The method according to claim 10. **Claim 12** Determining the three-dimensional pose of the fitting frame includes: Detecting at least one corresponding feature in the three-dimensional model of the fitting frame; Detecting the three-dimensional pose of the fitting frame based on a comparison between the at least one feature detected in the image and the corresponding at least one feature detected in the three-dimensional model of the fitting frame; The method according to claim 10 or claim 11, comprising . **Claim 13** When executed by at least one processor, causing the at least one processor to capture an image of a fitting frame worn by a user; detect at least one feature on the fitting frame from the captured image; be configured to access configuration information related to the fitting frame, the configuration information including physical characteristics of the fitting frame; When executed by the at least one processor, causing the at least one processor to determine a three-dimensional pose of the fitting frame captured in the image based on the detection of the at least one feature and the configuration information related to the fitting frame; wherein the three-dimensional pose is determined relative to features on the user's face; When executed by the at least one processor, causing the at least one processor to determine a configuration of a display device of a wearable computing device based on the three-dimensional pose of the fitting frame captured in the image; A program comprising executable instructions configured to perform the above. **Claim 14** The instructions cause the at least one processor to detect the at least one feature on the fitting frame in the captured image, and detecting the at least one feature comprises detecting a first feature including a bridge portion of the fitting frame; Detecting a second feature including a hinge portion between an edge portion and an arm portion of the fitting frame; Detecting a third feature including a peripheral edge portion of a lens of the fitting frame, or Detecting a fourth feature including a saddle portion of the arm portion of the fitting frame, The program according to claim 13, comprising at least one of the above.

15. The instructions cause the at least one processor to determine the three-dimensional pose of the fitting frame, and determining the three-dimensional pose of the fitting frame includes: Accessing a three-dimensional model of the fitting frame captured in the image, and Performing a comparison between a known position and a known orientation of the at least one feature detected in the image and a corresponding position and a corresponding orientation of the at least one feature in the three-dimensional model of the fitting frame. The program according to claim 13 or claim 14, comprising the above.

16. The instructions cause the at least one processor to detect a plurality of features and perform the comparison, and performing the comparison includes, for each of the plurality of features: Comparing a known position and a known orientation of the feature detected in the image with a corresponding position and a corresponding orientation of the feature in the three-dimensional model of the fitting frame The program according to claim 15, comprising the above.

17. The instructions cause the at least one processor to determine the three-dimensional pose, and determining the three-dimensional pose includes: Performing an association between a two-dimensional position of each feature detected in the image and a corresponding three-dimensional position of each of the features in the three-dimensional model of the fitting frame The program according to claim 16, comprising **Claim 18** The instructions also cause the at least one processor to detect a plurality of facial landmarks in the captured image, and determine ophthalmic fitting measurements for the wearable computing device based on the detected plurality of facial landmarks. The program according to claim 13. **Claim 19** The instructions cause the at least one processor to detect the plurality of facial landmarks, and detecting the plurality of facial landmarks comprises detecting the pupil height in the image of the fitting frame worn by the user, detecting at least one of the interpupillary distance or the monocular pupillary distance in the image of the fitting frame worn by the user, and determining the forward tilt angle of the fitting frame worn by the user based on the determined three-dimensional pose of the fitting frame and the detected plurality of facial landmarks. The program according to claim 18, comprising **Claim 20** The instructions cause the at least one processor to adapt the configuration of the display device of the wearable computing device to correspond to the ophthalmic fitting measurements, and adapting the configuration comprises determining a user's field of view based on the ophthalmic fitting measurements and the three-dimensional pose of the fitting frame, and configuring the output coupler of the display device such that content output by the display device is displayed within the determined field of view. The program according to claim 18 or claim 19, comprising

Images