Image-based fit detection for head-mounted wearable computing devices

Abstract

A system and method for detecting display fit measurements and / or ophthalmic measurements for a head-mounted wearable computing device including a display device are provided. An image of a fitting frame worn by a user of the computing device is captured by the user through an application running on the computing device. One or more keypoints and / or features and / or landmarks are detected in the image including the fitting frame. A three-dimensional pose of the fitting frame is determined based on configuration information associated with the detected keypoints and / or features and / or landmarks and the fitting frame. The display device of the head-mounted wearable computing device may be configured based on the three-dimensional pose of the fitting frame captured in the image.

Description

Technology Field

[0001] Cross-reference regarding related applications

[0002] This application is a continuation of U.S. application no. 17 / 444,963 filed on August 12, 2021, and claims priority to the entire contents of which are incorporated herein by reference.

[0003] Technology field

[0004] The present specification generally relates to the prediction of wear fit and / or display fit and / or ophthalmic fit of wearable devices, and in particular to the prediction of display fit and ophthalmic fit of head-worn computing devices including display performance. Background Technology

[0005] Wearable devices may include head-worn devices, such as smart glasses, headsets, and earbuds; wrist and / or hand-worn devices, such as smart watches, smart bracelets, and smart rings; smart pendants; fitness trackers; cameras; body sensors; and other similar devices. In some cases, users may want to select and / or customize wearable devices to fit and / or function. For example, users may want to select and / or customize smart glasses to include frame selection, prescription lens integration, and other features. Existing systems for procuring these types of wearable devices do not provide accurate fit and customization, particularly without access to retail stores.

[0006] In one general embodiment, a method for detecting a display fit measurement for a head-mounted wearable computing device including a display device comprises: detecting at least one key point of a fitting frame worn by a user from an image captured by the computing device by an application running on the computing device; accessing configuration information associated with the fitting frame detected in the image; determining a three-dimensional pose of the fitting frame captured in the image based on the at least one key point detection and the configuration information associated with the fitting frame; and determining the configuration of the display device of the head-mounted wearable computing device based on the three-dimensional pose of the fitting frame captured in the image.

[0007] In some embodiments, detecting at least one key point includes detecting a bridge portion of the fitting frame; a hinge point between a rim portion and an arm portion of the fitting frame; a lens edge portion of the fitting frame; or at least one of a saddle portion of an arm portion of the fitting frame.

[0008] In some implementations, the step of determining the three-dimensional pose of the fitting frame includes: accessing a three-dimensional model of the fitting frame captured in the image; and performing a comparison, comprising comparing a known position and orientation of at least one key point detected in the image with a corresponding position and orientation of the at least one key point in the three-dimensional model of the fitting frame. The step of detecting at least one key point includes detecting a plurality of key points for the fitting frame in an image captured by the computing device; and the step of performing the comparison may include, for each of the plurality of key points, comparing a known position and orientation of the key point detected in the image with a corresponding position and orientation of the key point in the three-dimensional model of the fitting frame. The step of determining the three-dimensional pose of the fitting frame may include performing a correspondence between the two-dimensional position of each key point detected in the image and the corresponding three-dimensional position of each key point in the three-dimensional model of the fitting frame.

[0009] In some embodiments, the method may further include the step of detecting a plurality of face landmarks in the captured image; and the step of determining an ophthalmic fit measurement for a head-mounted wearable computing device based on the detected plurality of face landmarks. The step of detecting a plurality of face landmarks may include: the step of detecting pupil height in an image of a fitting frame worn by the user; the step of detecting at least one of a distance between pupils or a monocular pupil distance in an image of a fitting frame worn by the user; and the step of determining a pantoscopic angle of a fitting frame worn by the user based on the three-dimensional pose of the determined fitting frame and the detected plurality of face landmarks. The step of determining the configuration of a display device of a head-mounted wearable computing device based on the three-dimensional pose of the fitting frame captured in the image may include the step of adjusting the configuration of the display device of the head-mounted wearable computing device to accommodate the ophthalmic fit measurement. The step of determining the configuration of a display device of the head-mounted wearable computing device may include: determining a user's field of view based on the ophthalmic fit measurements and the three-dimensional pose of the fitting frame; and configuring an output coupler of the display device so that content output by the display device is displayed within the determined field of view.

[0010] In another general embodiment, a non-transient computer-readable medium may store an executable instruction configured to cause at least one processor to perform the following operations when executed by at least one processor, the following operations include: capturing an image of a fitting frame worn by a user; detecting at least one keypoint for the fitting frame from the captured image; accessing configuration information associated with the fitting frame; determining a three-dimensional pose of the fitting frame captured in the image based on the detection of the at least one keypoint and the configuration information associated with the fitting frame; and determining the configuration of a display device of a head-mounted wearable computing device based on the three-dimensional pose of the fitting frame captured in the image.

[0011] In some embodiments, detecting at least one key point for the fitting frame from the captured image comprises: detecting a first key point including a bridge portion of the fitting frame; detecting a second key point including a hinge point between a rim portion and an arm portion of the fitting frame; detecting a third key point including a lens periphery edge portion of the fitting frame; or detecting at least one of a fourth key point including a saddle portion of the arm portion of the fitting frame. In some embodiments, determining a three-dimensional pose for the fitting frame comprises: accessing a three-dimensional model of the fitting frame captured in the image; and performing a comparison of the known position and orientation of at least one key point detected in the image with the corresponding position and orientation of the at least one key point in the three-dimensional model of the fitting frame.

[0012] In some embodiments, the instruction causes the at least one processor to detect a plurality of key points and perform the comparison, which may include, for each of the plurality of key points, comparing the known position and orientation of the key point detected in the image with the corresponding position and orientation of the key point in the 3D model of the fitting frame. In some embodiments, the instruction causes the at least one processor to determine a 3D pose, which includes performing a correspondence between the 2D position of each key point detected in the image and the corresponding 3D position of each key point in the 3D model of the fitting frame.

[0013] In some embodiments, the instruction causes the at least one processor to detect a plurality of face landmarks in a captured image and to determine an ophthalmic fit measurement for a head-mounted wearable computing device based on the detected plurality of face landmarks. The instruction, when executed, causes the at least one processor to detect a plurality of face landmarks, which includes detecting pupil height in an image of a fitting frame worn by the user; detecting at least one of a distance between pupils or a monocular pupil distance in an image of a fitting frame worn by the user; and determining a pantoscopic angle of the fitting frame worn by the user based on the three-dimensional pose of the determined fitting frame and the detected plurality of face landmarks. In some embodiments, the instruction causes the at least one processor to adjust the configuration of a display device of a head-mounted wearable computing device to accommodate the ophthalmic fit measurement, which includes determining the user's field of view based on the ophthalmic fit measurement and the three-dimensional pose of the fitting frame; and includes configuring an output coupler of the display device so that content output by the display device is displayed within the determined field of view.

[0014] Details of one or more embodiments are described in the accompanying drawings and the description below. Other configurations will become apparent from the description, drawings, and claims below. Brief explanation of the drawing

[0015] FIG. 1a illustrates an exemplary system according to an embodiment described in this specification. FIG. 1b is a front view of an exemplary wearable computing device illustrated in FIG. 1a according to an embodiment described in this specification, and FIG. 1c is a rear view. Figures 2a-2d illustrate exemplary ophthalmic measurements. FIG. 3 is a block diagram of a system according to an embodiment described in the present specification. FIGS. 4a and 4b illustrate an image capture mode for determining display fit measurements and ophthalmic fit measurements according to an embodiment described herein. FIGS. 5a-5c are perspective views of an exemplary fitting frame according to an embodiment described herein. FIGS. 6a through 6c illustrate an exemplary front image of an exemplary fitting frame worn by a user and exemplary features and / or key points and / or landmarks detected in the front image, according to an embodiment described herein. FIG. 6d illustrates an exemplary profile image of an exemplary fitting frame worn by a user and exemplary features and / or key points and / or landmarks detected in the profile image, according to an embodiment described herein. FIG. 7 illustrates a front view of an exemplary fitting frame according to an embodiment described in this specification. FIG. 8 is a flowchart of an exemplary method according to an embodiment described in this specification. FIG. 9 illustrates an exemplary computing device of the computing system discussed in this specification. Specific details for implementing the invention

[0016] The present disclosure relates to a system and method for selecting and fitting a wearable computing device, including determining wear fit and / or display fit and / or ophthalmic fit parameters associated with the wearable computing device. A system and method according to an embodiment described herein provides a determination of wear fit and / or display fit and / or ophthalmic fit for a wearable computing device based on measurements detected within image data. In some embodiments, the system and method provide a detection of wear fit and / or display fit and / or ophthalmic fit based on known parameters associated with a sample frame of the wearable computing device detected within image data according to an embodiment described herein. In some examples, the system and method described herein provide a detection of wear fit and / or display fit and / or ophthalmic fit from image data for a wearable computing device in the form of smart glasses, comprising a display and / or a corrective / prescription lens device customized to the specific physical characteristics, needs, and preferences of a specific user. The system and method according to the embodiments described in this document can facilitate image data capture for detecting measurement data by a user in a self-directed, unsupervised, or unsupervised manner without direct or virtual access to a retail store and / or salesperson.

[0017] Below, a system and method according to an embodiment described herein are described for fitting a wearable computing device in the form of a head-mounted display device, such as smart glasses including a display device, for example, so that content displayed by the display device can or will be seen by a user wearing the wearable computing device. In situations where a user may benefit from a head-mounted wearable computing device including a corrective lens, the system and method described herein provide ophthalmic fit measurements to enable the integration of the corrective / prescription lens into the head-mounted wearable device and to take into account the features of the corrective / prescription lens in the configuration of the display device of the head-mounted wearable computing device. That is, the system and method according to an embodiment described herein may use an image-based capture system and method to provide fitting of the display device and the corrective / prescription lens so that content displayed by the display device is visible to a user wearing the head-mounted wearable computing device.

[0018] FIG. 1a illustrates a user operating a mobile computing device in a system where the mobile computing device can access one or more external resources (1100) through a network (1200). FIG. 1a provides a third-person view of a user in an environment (1000) that includes various examples of mobile computing devices (100, 180, 190, 200). An exemplary mobile computing device illustrated in FIG. 1a includes a first head-worn wearable computing device (100), a second head-worn wearable computing device (180), a wrist-worn computing device (190), and a portable computing device (200). In some examples, the first head-worn computing device (100) may include various components such as a head-mounted display device, a display in the form of smart glasses or a headset, a camera capable of capturing images of the surrounding environment, audio input / output functions, user input functions, etc. In some examples, the second head-worn computing device (180) may be an ear-worn computing device, such as headphones or earbuds, and may include audio input / output functions, a camera capable of capturing images of the surrounding environment, user input functions, etc. In some examples, the wrist-worn computing device (190) may include a display, a camera capable of capturing images of the surrounding environment, audio input / output functions, user input functions, etc. (e.g., a smart watch or a wristband) In some examples, the portable computing device (200) may include a display, a camera capable of capturing images of the surrounding environment, audio input / output functions, user input functions, etc., as in a smartphone. FIG. 1a illustrates only some examples of mobile computing devices. The principles described herein may be applied to other types of mobile computing devices not specifically illustrated in FIG. 1a.In some examples, mobile computing devices including the exemplary computing device illustrated in FIG. 1a can communicate with each other and / or external resources (1100) to exchange information and receive and transmit inputs and / or outputs, etc.

[0019] FIG. 1b is a front view of the exemplary head-mounted wearable computing device (100) illustrated in FIG. 1a, and FIG. 1b is a rear view. In some embodiments, the exemplary head-mounted wearable computing device (100) may take the form of a pair of smart glasses or augmented reality glasses, an augmented reality and / or virtual reality headset or goggles, as in the example illustrated in FIG. 1b and FIG. 1c. Below, a system and method according to an embodiment described herein will be described with respect to a wearable computing device (100) in the form of smart glasses merely for convenience of discussion and explanation. The principles described herein may be applied to other types of wearable computing devices and / or combinations of mobile / wearable computing devices working together.

[0020] As illustrated in FIG. 1b, an exemplary head-mounted wearable computing device (100) comprises a frame (102). As illustrated in FIG. 1b and FIG. 1c, the frame (102) comprises a rim portion (103) surrounding a glass portion(s) (107) or a lens (107). An arm portion (105) is coupled, for example, pivotably or rotatably to each rim portion (103) of the frame (102) by a respective hinge portion (110). In some examples, the lens (107) may be a corrective / prescription lens. In some examples, the lens (107) may be a glass portion that does not necessarily include corrective / prescription parameters. In some examples, a bridge portion (109) may connect the rim portion (103) of the frame (102). A display device (104) may be coupled to a portion of the frame (102). In the example illustrated in FIG. 1b and FIG. 1c, the display device (104) is coupled to the arm portion (105) of the frame (102). With the display device (104) coupled to the arm portion (105), the eye box (140) extends toward the lens(s) (107) for content output at the output coupler (144) so ​​that the content output by the display device (104) can be seen by the user. In some examples, the output coupler (144) may substantially match the lens(s) (107). The head-mounted wearable computing device (100) may also include an audio output device (106) (e.g., one or more speakers), a lighting device (108), a sensing system (111), a control system (112), at least one processor (114), and an outward-facing image sensor (116) or camera (116). In some embodiments, the display device (104) may include a see-through near-eye display. For example, the display device (104) may be configured to project light from a display source onto a portion of teleprompter glass that functions as a beam splitter positioned at a certain angle (e.g., 30-45 degrees).The beam splitter may allow for reflection and transmission values ​​such that the remaining light is transmitted while light from the display source is partially reflected. This optical design enables the user to see both physical items of the world next to the content (e.g., digital images, user interface elements, virtual content, etc.) generated by the display device (104), for example, through a lens (107). In some embodiments, waveguide optics may be used to depict content on the display device (304).

[0021] In some embodiments, the wearable computing device (100) may include an eye-tracking device (120) comprising one or more sensors (125) for detecting and tracking the direction and movement of the eyes. Data captured by the sensor(s) (125) may be processed to detect and track the direction and movement of the eyes as user input. In some embodiments, the sensing system (111) may include various sensing devices, and the control system (112) may include various control system devices, for example, comprising one or more processors (114) operably connected to a component of the control system (112). In some embodiments, the control system (112) may include a communication module that provides communication and information exchange between the wearable computing device (100) and another external device.

[0022] When determining the size and fitting of a wearable computing device (100), such as the exemplary smart glasses illustrated in FIG. 1a through 1c, for a specific user, numerous different size and fitting measurements and / or parameters may be considered. These may include, for example, wear fit parameters or wear fit measurements. Wear fit parameters / measurements may consider how a specific frame (102) fits and / or looks and / or feels to a specific user. Wear fit parameters / measurements may consider numerous factors, for example, whether the frame (102) is wide enough to be comfortable against the temples and whether it is too wide to remain in a relatively fixed position when the user wears it (i.e., too large or too wide). Wear fit parameters / measurements may consider other factors such as whether the rim portion (103) and the bridge portion (109) are sized so that the bridge portion (109) rests comfortably on the user's nose bridge, whether the arm portion (105) is sized so that it rests comfortably on the user's ears, and other comfort-related considerations. Wear fit parameters / measurements may consider how the user naturally wears the frame (102), such as head posture / how the user naturally holds their head, and how the user positions the frame (102) relative to their face. Wear fit parameters / measurements may consider whether the size and / or shape and / or contour of the frame (102) is aesthetically pleasing to the user and compatible with the user's facial features. In some examples, wear fit parameters / measurements may consider whether the fit associated with a specific frame configuration can accommodate the integration of a display device so that the display area can be adjusted to the user's natural field of vision.In some examples, wear fit parameters may consider whether the eye-tracking camera configuration associated with a specific frame configuration can effectively track the user's gaze and movements.

[0023] Display fit parameters or display fit measurements may be considered when sizing and fitting a wearable computing device (100) for a specific user. Display fit parameters / measurements are used to configure a display device (104) for a set of frames (102) for a specific user so that content displayed by the display device (104) is visible to the user. For example, display fit parameters / measurements may provide a placement indication of the display device (104) so ​​that content displayed by the display device (104) is captured within at least a set portion of the user's field of vision. For example, display fit parameters / measurements may be used to configure the display device (104) to provide at least one set of visibility levels corresponding to the amount, portion, or percentage of the display of content shown to the user at a set brightness level and a set pupil size at the periphery of the user's field of vision (e.g., the least visible corner). Display fit parameters / measurements may be used to configure the display device (104) to provide the best possible placement of the display of content to the user for a specific set of frames (102).

[0024] In some examples, ophthalmic fit parameters or ophthalmic fit measurements may be considered when determining the size and fitting of a wearable computing device (100) including a prescription or corrective lens (107). Some examples of ophthalmic fit measurements are illustrated in FIGS. 2a-2d. Ophthalmic fit measurements may include pupil height (PH) (distance from the center of the pupil to the bottom of the lens (107)), for example, the left pupil height and the right pupil height. Ophthalmic fit measurements may include interpupillary distance (IPD) (distance between pupils). IPD may be characterized by monocular pupillary distances, for example, left pupil distance LPD (distance from the center of the bridge of the nose to the left pupil) and right pupil distance RPD (distance from the center of the bridge of the nose to the right pupil). Ophthalmic fit measurements may include pantoscope angle (PA) (an angle defined by the tilt of the lens (107) relative to the vertical). Ophthalmic fit measurements may include, for example, a vertex distance V (distance from the cornea to the lens (107)) including a left vertex distance and a right vertex distance. Ophthalmic fit measurements may include other parameters or measurements that provide sizing and / or fitting of a head-mounted wearable computing device (100) including a display device (104) as described above. FIG. 2c illustrates a vertex distance (V) associated with a relatively lower diopter lens (107). FIG. 2d illustrates a vertex distance (V) associated with a relatively higher diopter lens (107). Ophthalmic fit measurements may be taken into account when fitting a wearable computing device (100) including a display device (104) to a user.For example, an ophthalmic fit measurement (along with a display fit measurement) can provide a placement of a display of content by a display device (104) within an eye box defined by a three-dimensional volume extending between the lens (107) and the user's eye, and the display of content is within the user's field of vision and within the user's modified field of vision so that the user can see it.

[0025] In a system and method according to an embodiment described herein, image data may be captured through an application running on a computing device operated by a user, such as the computing device (200) described above in relation to FIG. 1a, or on other computing devices operated by the user. Wear fit measurements and / or display fit measurements and / or ophthalmic fit measurements may be detected from the image data acquired in this manner to sizing and fitting a head-mounted wearable computing device, such as the wearable computing device (100) described above, which includes a display and / or corrective lens for a specific user. The image data may be captured through an application running on a computing device (200) operated by the user in an unsupervised or unguided manner, without the need to access a retail store or make an appointment with a fitting / sales expert. In some embodiments, the detection of wear fit and / or display fit and / or ophthalmic fit measurements may be based on the detection of physical features or attributes of a sample frame worn by the user contained in the image data. A three-dimensional model of the user's head may be generated, produced, or defined based on measurements detected within the captured image data (e.g., measurements of the user's physical attributes). In some embodiments, the detection of physical features and / or attributes of the image data may be provided using an image sensor or camera of the computing device (200) without relying on data collected by a depth sensor. In some embodiments, the depth sensor included in the computing device (200) may provide the collection of three-dimensional measurements associated with the detected physical features and / or attributes of the sample frame and / or the user's facial / skull attributes.

[0026] FIG. 3 is a block diagram of an exemplary system for determining display fit measurements and ophthalmic fit measurements for a wearable computing device according to an embodiment described herein. The system includes one or more computing devices (300). The computing device (300) may optionally communicate via a network (306) to access external resources (302), such as, for example, a server computer system, a processor, a database, a memory storage, etc. The computing device (300) may operate under the control of a control system (370). The computing device (300) may communicate with one or more external computing devices (304) (other wearable computing devices, other mobile computing devices, etc.) directly (via wired and / or wireless communication) or via a network (306). In some embodiments, the computing device (300) includes a communication module (380) to facilitate external communication. In some embodiments, the computing device (300) includes a sensing system (320) comprising various sensing system components, such as one or more image sensors (322), one or more position / orientation sensor(s) (324) (e.g., inertial measurement unit, accelerometer, gyroscope, magnetometer, etc.), one or more audio sensors (326) capable of detecting audio input, and other sensors. The computing device (300) may include more or fewer sensing devices and / or combinations of sensing devices.

[0027] In some embodiments, the computing device (300) may include one or more image sensor(s) or camera(s) (360). The camera(s) (360) may include, for example, an outward-facing camera, an outward-facing camera, etc., capable of capturing still images and / or moving images of the environment outside the computing device (300). In some embodiments, one or more camera(s) (360) may include a depth sensor. Still and / or moving images may be displayed by a display device of the output system (340) and / or transmitted externally via a communication module (380) and a network (306) and / or stored in the memory (330) of the computing device (300). The computing device (300) may include one or more processor(s) (350). The processor (350) may include various modules or engines configured to perform various functions. In some examples, the processor(s) (350) includes object recognition module(s), feature recognition module(s), pattern recognition module(s), configuration identification module(s), and other such processors. The processor(s) (350) may be formed on a substrate configured to execute one or more machine-executable instructions or pieces of software, firmware, or a combination thereof. The processor(s) (350) may be semiconductor-based, comprising a semiconductor material capable of performing digital logic. The memory (330) may include any type of storage device that stores information in a format that can be read and / or executed by the processor(s) (350). The memory (330) may store applications and modules that cause specific actions to be performed when executed by the processor(s) (350). In some examples, the applications and modules may be stored in an external storage device and loaded into the memory (330).

[0028] FIGS. 4A and 4B illustrate an exemplary computing device, such as the computing device (300) described in relation to FIG. 3, operated by a user to capture display fit measurements and / or ophthalmic fit measurements for a wearable computing device that is customized for use by a user. In FIG. 4A, the exemplary computing device (300A) is in the form of a portable computing device such as a smartphone. In FIG. 4B, the exemplary computing device (300) is in the form of a laptop computing device (300B). FIGS. 4A and 4B provide two examples of a computing device (300) that can be used to capture image data to be processed by a sizing simulator to execute the system and method described herein. The principles described in this specification may be performed by other types of computing devices, and in particular by computing devices capable of running applications to utilize image capture functions, display functions, external communication, and processing to collect and process display fit measurements and / or ophthalmic fit measurements to customize a wearable computing device for a specific user.

[0029] In the example illustrated in FIG. 6a, the user holds the computing device (300A) so that the user's head and face are within the field of view of the camera (360A) of the computing device (300A), and the camera (360A) can capture an image of the user's head and face. Since the captured image can be displayed to the user on the display device (342A) of the computing device (300A), the user can verify that their head and face have been captured within the field of view of the camera (360A). Similarly, as illustrated in FIG. 4b, the computing device (300B) is positioned relative to the user so that the user's head and face are captured within the field of view of the camera (360B) of the computing device (300B), and the camera (360B) can capture an image of the user's head and face. Since the captured image can be displayed to the user on the display device (342B) of the computing device (300B), the user can verify that their head and face have been captured within the field of view of the camera (360B). In the exemplary configuration shown in FIGS. 4a and 4b, image data captured by the camera (360) is processed (e.g., by the recognition engine(s) and simulation engine(s) of the external processor(s) as shown in FIG. 3) to detect wear fit and / or display fit and / or ophthalmic fit measurements, and to determine the frame and / or lens size and contour, display device configuration, etc., for customizing a wearable computing device, such as the head-mounted wearable computing device shown in FIGS. 1b and 1c, for a user.

[0030] In FIGS. 4a and 4b, the user is wearing an exemplary fitting frame (500). The fitting frame (500) is included in an image of the user's head and face captured by a camera (360) of a computing device (300).

[0031] As described above, the detection of wear fit and / or display fit and / or ophthalmic fit measurements from captured image data may be facilitated by the detection of one or more physical features and / or attributes associated with the fitting frame worn by the user while capturing the image data. In some examples, the user may try on or sample various fitting frames or sample fitting frame kits from a collection. A fitting kit containing sample fitting frames may be provided to the user prior to a fitting session. This allows the user to evaluate at least some wear fit parameters. For example, by trying on or sampling multiple different fitting frames from a fitting kit containing sample fitting frames, the user may evaluate factors such as physical size, comfort, and aesthetics. The user may select a fitting frame from a fitting kit containing sample fitting frames to use during the fitting session. The sample fitting frames included in the fitting kit may provide the user with a relatively accurate fit and feel by mimicking the size, shape, weight, etc. of the actual frame to be integrated into the head-mounted wearable computing device (100) based on data captured during the fitting session. Image data captured through an application running on a computing device (200) during a fitting session can be used to determine a display fit and / or an ophthalmic fit. Measurement data collected in this way can be used to adjust the display device (104), specifically the image display device area generated by the display device (104), for the user and the selected fitting frame based on measurements detected from the image data. Measurement data collected in this way can be used to integrate a corrective or prescription lens into a head-mounted wearable computing device (100).Measurement data collected in this way can be used to configure a display device (104) for the user and selected frame and to integrate a corrective / prescription lens into a head-mounted wearable computing device (100).

[0032] FIGS. 5a through 5c illustrate exemplary fitting frames (500) for fitting a head-mounted wearable computing device according to an implementation described herein, such as a first exemplary fitting frame (500A), a second exemplary fitting frame (500B), and a third exemplary fitting frame (500C). The exemplary fitting frames (500) may physically or geometrically represent the actual frame of the head-mounted wearable computing device to be worn by the user, but are not functional as they are merely for the purpose of sizing / fitting. The exemplary fitting frames (500) illustrated in FIGS. 5a-5c are examples of sample fitting frames (500) that may be included in a fitting kit provided to the user for consideration before a fitting session. The principles described herein are applicable to fitting frames having different shapes and / or sizes and / or configurations. The first exemplary fitting frame (500A) illustrated in FIG. 5a has a first size and / or shape and / or contour defined by a first set of physical features. The first exemplary fitting frame (500A) can be considered as a medium-sized frame having D-shaped rim portions (503A, 503B). The second exemplary fitting frame (500B) illustrated in FIG. 5b has a second size and / or shape and / or contour defined by a second set of physical features. The second exemplary fitting frame (500B) can be considered as a medium-sized frame having rounded rim portions (503A, 503B). The third exemplary fitting frame (500C) illustrated in FIG. 5c has a third size and / or shape and / or contour defined by a third set of physical features. The third exemplary fitting frame (500C) can be considered as a large frame having rectangular rim portions (503A, 503B). The exemplary fitting frames (500) (500A, 500B, 500C) illustrated in FIGS. 5a-5c are presented for the purpose of discussion and example.The principles described in this specification may be applied to other fitting frames having different sizes and / or shapes and / or contours defined by other physical features.

[0033] Each exemplary fitting frame (500) (500A, 500B, 500C) illustrated in FIG. 5a-5c includes a first rim portion (503A) surrounding a first glass portion or a first lens (507A), and a second rim portion (503B) surrounding a second glass portion or a second lens (507B). When the exemplary fitting frame (500) (500A, 500B, 500C) is worn by a user, the first rim portion (503A) / first lens (507A) may be configured to be positioned corresponding to the user's first eye, and the second rim portion (503B) / first lens (507B) may be configured to be positioned corresponding to the user's second eye. The bridge portion (509) may extend between the inner end portions of the rim portions (503A, 503B) to connect the rim portions (503A, 503B) of the exemplary fitting frame (500) (500A, 500B, 500C). The bridge portion (509) may be configured to be seated on the user's nose, positioned so that the first rim portion (503A) / first lens (507A) corresponds to the first eye, and the second rim portion (503B) / second lens (507B) corresponds to the second eye. The first arm portion (505A) may be pivotally or rotatably coupled to the outer end of the first rim portion (503A) by the first hinge portion (510A). The second arm portion (505B) may be pivotally or rotatably coupled to the outer end of the second rim portion (503B) by the second hinge portion (510B). The first and second arm portions (505A, 505B) may each include an ear saddle portion (515A, 515B) configured to rest on the user's respective first and second ears when the exemplary fitting frame (500) (500A, 500B, 500C) is worn by the user.

[0034] As described above, a fitting frame according to an embodiment described herein may include physical features or attributes that can be detected, identified, or recognized in image data including a fitting frame worn by a user. FIG. 6a illustrates an exemplary two-dimensional image plane (600) or image (600) of a user wearing an exemplary fitting frame (500). The exemplary image (600) may be captured by a camera (360) of a computing device (300) operated by the user through an application executed by the computing device (300). As illustrated in FIG. 6a, the image (600) is displayed on a display device (342) of the computing device (300). This allows the user to visually confirm whether the captured image data includes a capture of the fitting frame (500), the user's face and / or skull and / or optical features, etc. In the example where the computing device (300) is a portable mobile computing device such as a smartphone, the self-portrait mode or selfie mode may be called for capturing image data by the camera (360) of the computing device (300).

[0035] The image (600) may include a two-dimensional array of pixels captured by the camera (360). In some examples, features detected within the image (600) may correspond to the location of the features detected within the image (600), e.g., a two-dimensional coordinate position. FIG. 6b illustrates some examples of detectable keypoints or detectable features associated with the fitting frame (500) worn by the user while the image data is being captured. The three-dimensional pose of the fitting frame (500) on the user's face can be determined through the keypoints or features detected in the image data, which can ultimately be used to determine display fit information and / or ophthalmic fit information associated with the fitting frame (500) worn by the user. The detection of these types of keypoints associated with the fitting frame (500) worn by the user in the image data is important for determining display fit and / or ophthalmic fit information that enables the head-mounted wearable computing device (100) to be configured for the user.

[0036] The exemplary configuration illustrated in FIG. 6b includes an exemplary keypoint (610A) at a position in the image (600) corresponding to a hinge joint point between the first rim portion (503A) and the first arm portion (505A) of the fitting frame (500), for example, at a position corresponding to a first hinge portion (510A) that rotatably connects the rim portion (503A) and the first arm portion (505A). For example, the exemplary configuration illustrated in FIG. 6b includes an exemplary keypoint (610B) at a position in the image (600) corresponding to a hinge joint point between the second rim portion (503B) and the second arm portion (505B) of the fitting frame (500). For example, the exemplary configuration illustrated in FIG. 6b includes an exemplary keypoint (620A) in a position corresponding to the lower peripheral edge portion of the first lens (507A) of the fitting frame (500) and an exemplary keypoint (620B) in a position corresponding to the lower peripheral edge portion of the second lens (507B) of the fitting frame (500). The exemplary keypoints (620A, 620B) are simply exemplary keypoints corresponding to specific peripheral edge portions of the lens (507). Keypoints may also be defined at other peripheral edge portion(s) of the lens (507), such as the upper peripheral edge portion, the left peripheral edge portion, and / or the right peripheral edge portion of the lens (507), for example. In some examples, keypoint(s) defined along the peripheral edge portion of the lens (507) may correspond to designated section(s) of the lens (507). In some examples, keypoint(s) defined along the peripheral edge portion of the lens (507) may correspond to specific points along the peripheral edge of the lens (507). For example, the exemplary configuration illustrated in FIG. 6b includes an exemplary keypoint (630) at a position in the image (600) corresponding to the bridge portion (509) of the fitting frame (500).

[0037] The exemplary image (600) illustrated in FIGS. 6a and 6b is substantially a frontal image, and the exemplary keypoints (610, 620, 630) or features (610, 620, 630) are merely some examples of detectable keypoints or features associated with the fitting frame (500) worn by the user, which are detected in the image (600) to detect the display fit and / or ophthalmic fit associated with the fitting frame (500) for the user. To facilitate the determination of the display fit and / or ophthalmic fit, other keypoints or features associated with the fitting frame (500) may be detected in the image (600). Examples of other detectable keypoints or detectable features include the upper portion of the lens (507), the inner peripheral portion of the lens (507) (i.e., the portion of the lens (507) adjacent to the bridge portion (509) of the fitting frame (500), the outer periphery portion of the lens (507) (i.e., the portion of the lens (507) adjacent to the hinge portion (510) of the fitting frame (500), etc. The distribution of detectable keypoints or features across the fitting frame (500) and the separation of keypoints or features can improve detection accuracy and correlation with a three-dimensional model. Additionally, other keypoints or features associated with the fitting frame (500) may be detected in the user's profile image or an image taken at a certain angle to the user, which may include the position of the arm portion(s) of the fitting frame (500), the position of the ear saddle point(s) (515) relative to the user's ear, the position of the rim portion(s) (503) relative to the user's face, etc. To determine the display fit and / or ophthalmic fit of the fitting frame (500) for the user, more or fewer key points and / or combinations of key points associated with the fitting frame (500) may be detected in the image (600).As mentioned above, detection of specific key points associated with the fitting frame (500) worn by the user in the image data is important for determining display fit and / or ophthalmic fit information that enables the head-mounted wearable computing device (100) to be configured for the user.

[0038] FIG. 6c illustrates a plurality of face landmarks (along with the exemplary detectable keypoints (610, 620, 630) or features (610, 620, 630) described above in relation to FIG. 6b), which can be detected in a two-dimensional image (600) captured by a camera (360) of a computing device (300). In some examples, the pupil, specifically the pupil center (640) for each eye of the user, can be detected in the image (600). In some examples, other face landmarks (650) can be detected in the image (600). In turn, as illustrated in FIG. 6c, some exemplary face landmarks (650) include the bridge of the nose, the temples, and the quadrant surrounding the bridge of the nose on the cheeks, solely for the purpose of discussion and illustration. Other face landmarks may be detected to facilitate the identification of features associated with the positioning of the fitting frame (500) on the user's head / face. The face landmark (650) can provide indexing for positioning the fitting frame (500) for example, the user's head / face, etc.

[0039] In some embodiments, a side or profile image (660) may be captured as illustrated in FIG. 6d. The exemplary image (660) illustrated in FIG. 6d is a substantially full profile image (660) for the purpose of discussion and explanation only. Information detected in the profile image (660) may be detected from a side image taken at less than 90 degrees from the front image (600) illustrated in FIG. 6a-6c. As illustrated in FIG. 6d, additional keypoints or features associated with the fitting frame, and additional face / skull landmarks may be detected in the side or profile image (660). In turn, as illustrated in FIG. 6, a keypoint (670) may be detected at a position within the image (660) corresponding to the ear saddle portion (515A) of the arm portion (505A) of the fitting frame (500). Face / skull landmarks (650) may be detected, for example, at parts of the user's ears, such as the anterior crease of the ear, the upper part of the ear, etc. Other key points and / or face / skull features may also be detected in the side / profile image (660). Likewise, the tilt and angle (e.g., pantoscopic angle, PA) of the fitting frame (500) relative to the user's face may be detected from the side or profile image (660). That is, while the angle between the rim part (503) and the arm part (505) of the fitting frame (500) is known, the tilt or angle of the fitting frame (500) may vary depending on various factors. These factors may include, for example, how the user wears the fitting frame (500) (closer to / further from the eyes, higher or lower nose, etc.), the positioning of the ears relative to the user's nose, and other such factors.

[0040] In some embodiments, detected keypoints associated with the fitting frame (500) (such as the detected exemplary keypoints (610, 620, 630, 670) or features (610, 620, 630, 670) shown in FIGS. 6b-6d) may be used alone and / or together with the user's detected face / skull features (such as the detected pupil center (640) and face landmark (650) shown in FIGS. 6c and 6d) to determine the pose of the fitting frame (500) on the user's face, e.g., a three-dimensional pose. This three-dimensional pose of the fitting frame (500) on the user's face may be used to determine the display fit and / or ophthalmic fit for a head-mounted wearable computing device (100) in the form of smart glasses, e.g. In some examples, the configuration of the frame (102) of the head-mounted wearable computing device (100) to be configured corresponds to the configuration of the fitting frame (500) worn by the user in the image (600). In some examples, the configuration of the frame (102) of the head-mounted wearable computing device (100) to be configured does not necessarily correspond to the configuration of the fitting frame (500).

[0041] In some examples, for instance, data extracted from a two-dimensional frontal image (600) and a corresponding detected three-dimensional pose of the fitting frame (500) on the user's face can be synthesized to evaluate the gazing possibility of the fitting frame (500) for the user. That is, this data can be synthesized to determine the configuration of a display device (104) included in a head-mounted wearable computing device (100), and the display device (104) can be configured so that the content output by the display device (104) is within the field of vision of the user wearing the head-mounted wearable computing device (100). For example, the display device (104) can be configured so that the content output by the display device (104) and displayed on the output coupler (144) is captured within the user's field of vision and visible to the user.

[0042] As described above, three-dimensional pose information associated with the pupil (e.g., pupil center(s) (640)) and the user's face (e.g., detected face landmark(s) (650)) can be used to determine lens-related measurements (e.g., pupil height, vertex distance, foreground angle, etc.) together with the three-dimensional pose information of the fitting frame (500). As mentioned above, the three-dimensional position and orientation, or pose, of the fitting frame (500) may be determined based on the detection and / or identification of one or more keypoints (610, 620, 630) or features (610, 620, 630) of image data captured by the camera (360) of the computing device (300) operated by the user. In some examples, additional information may be obtained from a side or profile image (660) as described above in relation to FIG. 6d. Although there may be some curvature in the fitting frame (500), in some examples, the fitting frame (500) may be considered as a substantially rigid body having a movement of 3 degrees. Thus, relatively accurate detection of at least one known point on the fitting frame (500) can be used to determine the three-dimensional position of the fitting frame (500). One or more of the key points (610, 620, 630) or features (610, 620, 630) can be reliably detected in the image (600), and the position of the one or more detected key points (610, 620, 630) or features (610, 620) can be known in the three-dimensional space associated with the fitting frame (500).

[0043] As described above, the system and method according to the embodiments described herein can facilitate the fitting of a head-mounted wearable computing device based on image data captured through an application running on a computing device operated by a user. The image data can be captured by the user, and the wearable computing device can be fitted and configured to the user in a self-directed, unsupervised, or non-supervised manner without the need to visit a retail store and / or in person or make a virtual appointment with a sales agent. The capture and processing of image data are described in more detail below.

[0044] As described above, a user who wishes to select, fit, and configure a head-mounted wearable computing device (100) may have access to or be provided with one or more fitting frames or sample sets of fitting frames to consider prior to the fitting process. In some examples, the user may try on each fitting frame included in the sample set of fitting frames and evaluate factors such as size, comfort, appearance, compatibility with facial features, and overall wearability associated with each. The user may select a fitting frame (500) from the sample set of fitting frames based on these and / or other factors for future use in the fitting and configuration process.

[0045] In some examples, a sample set of fitting frames to be tried on and / or evaluated may be selected by the user from among a plurality of fitting frames available for try-on. In some examples, the system (e.g., within a module of an application running on a computing device (300) operated by the user) may suggest a sample set of fitting frames to the user based on an analysis of the shape and / or size and / or configuration of the frames that may be compatible with the user's physical features and the detected physical features. In some examples, a sample set of fitting frames may be provided to the user before initiating the process for fitting and configuring the head-mounted wearable computing device (100).

[0046] The ability to physically try on one or more pairs of fitting frames (500) provides the user with the opportunity to physically evaluate the fit to determine which of the fitting frames (500) is physically most comfortable and properly / comfortably aligned with facial / or skull features (eyes, bridge of the nose, ear saddle points, cheek contact points, etc.). Evaluation of the wear fit by the user using the physical fitting frames (500) prior to fitting and configuring the head-mounted wearable computing device (100) can yield a head-mounted wearable computing device (100) that is more suitable for a specific user. This method of selecting the fitting frames (500) based on the physical wear fit by the user can yield a more accurate fitting and configuration of the head-mounted wearable computing device (100) for the user.

[0047] In some situations, a single fitting frame (500) may be available to the user for fitting and configuring a head-mounted wearable computing device (100). In such situations, the fitting and configuration process may be performed as described below, with additional measures taken to determine the wear fit of a specific frame (102) for integration into the head-mounted wearable computing device (100).

[0048] In some examples, the user operates the computing device (300) (e.g., as in FIG. 4a and / or 4b) to run an application and initiates the fitting and configuration of the head-mounted wearable computing device (100) using a selected fitting frame (500) (e.g., selected from a sample set of fitting frames described above). In some examples, the application running on the computing device (300) operated by the user may prompt the user to initiate image capture while wearing the selected fitting frame (500). For ease of discussion and explanation, this process will be described below based on the capture of a front image (600) as described in FIG. 6a-6c. However, in some embodiments, data detected from a profile image (660) and / or a semi-profile image, as shown in FIG. 6d, may be used for the fitting and configuration of the head-mounted display device (100) for the user.

[0049] In some examples, the application may prompt the user to confirm the selection of a specific fitting frame (500) to be worn while capturing image data. Thus, in some situations, configuration information associated with the selected fitting frame (500) to be worn at the time of image capture may be known. This may include, for example, linear and / or angular measurements and / or contours associated with the selected fitting frame (500), a two-dimensional model and / or a three-dimensional model of the selected fitting frame (500), and other such information. This known information associated with the selected fitting frame (500) may be accessed by an application running on the computing device (300). For example, this known configuration information may be stored in a database accessible to the application running on the computing device (300). The known configuration information associated with the selected fitting frame may further improve the accuracy and efficiency of fitting and configuration of the head-mounted wearable computing device (100) for the user.

[0050] As illustrated in FIG. 7, various keypoints or features associated with the fitting frame (500) can be detected within a front image (600) captured by a camera (360) of a computing device (300) operated by a user. In the exemplary arrangement illustrated in FIG. 7, a first keypoint (610A) is located at a known first position (710A) on the fitting frame (500) corresponding to a hinge joint point between the first rim portion (503A) and the first arm portion (505A) of the fitting frame (500). Similarly, a second keypoint (610B) is located at a known second position (710B) on the fitting frame (500) corresponding to a hinge joint point between the second rim portion (503B) and the second arm portion (505B) of the fitting frame (500). The third key point (620A) is located at the third known position (720A) on the fitting frame (500) corresponding to the lower peripheral edge portion of the first lens (507A), and the fourth key point (620) is located at the fourth known position (720B) on the fitting frame (500) at the position corresponding to the lower peripheral edge portion of the second lens (507B). The fifth key point (630) is located at the known fifth position (730) of the fitting frame (500), that is, at the position corresponding to the central portion of the bridge portion (509) of the fitting frame (500). Detection of the key points (610, 620, 630) of the fitting frame (500) and identification of the corresponding known positions (710, 720, 730) can provide fixed reference points on the fitting frame (500) that can be used to detect and refine the three-dimensional pose of the fitting frame (500) for the user's face. The detected features or attributes associated with each of the detected key points (610, 620, 630) can be compared with known features or attributes (known position, geometry, contour, distance, etc.) to determine the depth for, for example, the camera (360). This can facilitate the determination of the three-dimensional pose of the fitting frame (500) for the user's face.

[0051] For example, the detected keypoints (610, 620, 630) and the corresponding known positions (710, 720, 730) of the fitting frame (500) can be matched with the corresponding set of points / positions of the 3D model of the fitting frame (500). As described above, the 3D model of the fitting frame (500) can be stored in advance and accessed through an application running on the computing device (300). Thus, the known set of correspondences between the 3D model of the fitting frame (500) and the corresponding points detected in the image (600) can be used to determine the 3D pose of the fitting frame (500).

[0052] In some embodiments, detected face / skull landmarks, such as the exemplary landmark (650) illustrated in FIGS. 6a-6d, can facilitate the detection and identification of keypoints or features of the fitting frame (500). For example, the detection and identification of the lower peripheral edge portion of the first lens (507) defining the exemplary keypoint (620A) described above can be compressed or enhanced and / or the accuracy increased based on the detection of the landmark (650) below the keypoint (620A) defining the user's cheek, the landmark (650) to the right of the keypoint (620A) defining the user's nose bridge, etc. Other keypoints defined on other peripheral edge portions of the lens (507) (i.e., the upper and / or left and / or right peripheral edge portions of the lens (507)) can be compressed and / or enhanced in a similar manner based on the detection of other landmarks (650).

[0053] In some embodiments, detection of keypoints or features associated with the fitting frame (500) worn by the user in an image (600) may rely on comparison with synthetic data, real data, and a database filled with a combination of synthetic data and real data. The synthetic data may include 3D renderings of glasses and 3D renderings of the head in various combinations of, for example, poses and lighting conditions, which can be combined to generate a very large number of different rendering combinations of glasses worn on the head under various conditions. The real data may include, for example, image data capturing a real person wearing glasses. By combining the synthetic data and the real data, additional combinations of glasses worn on the head in various poses and conditions can be generated. This data (synthetic data, real data, and combined data) may be accumulated to train a neural network capable of outputting continuously improving results in matching keypoints of the fitting frame with points of a 3D model.

[0054] In some situations, a well-developed / densely filled neural network containing training data representing a relatively large number of poses, positions, and angles of the fitting frame (500) may be available for a specific fitting frame (500). In such situations, the image, for example, the frontal image (600) described above in relation to FIGS. 6a through 6c, may be input into the neural network set up for the fitting frame (500), and the correspondence between the image (600) of the user wearing the fitting frame (500) and the data filling the neural network may be detected without necessarily requiring the detection of key points of the image (600) before the neural network processes the image (600).

[0055] In some embodiments, a process similar to the fitting and configuration process may be applied to the calibration or recalibration of the head-mounted wearable computing device (100) after the head-mounted wearable computing device (100) is delivered to the user. For example, a new product setup process for the head-mounted wearable computing device (100) configured for the user may include fine-tuning of display characteristics when the head-mounted wearable computing device (100) is delivered. For example, an application running on a computing device (300) operated by the user may prompt the user to capture image data including the user wearing the head-mounted wearable computing device (100). Detection of keypoints on the head-mounted wearable computing device (100) may be detected to determine the three-dimensional pose of the head-mounted wearable computing device (100) on the user's face relative to the user's eyes / eye boxes. Control software controlling the operation of the display device (104) may adjust the output of the content, for example, to optimize the position where the content is displayed. This can optimize gaze possibilities and improve the user viewing experience.

[0056] The measurements or parameters described above may be collected to determine display fit characteristics for a user wearing a selected fitting frame (500). The measurements or parameters described above allow for the fitting of a display device of a head-mounted wearable computing device (such as the display device (104) of the head-mounted wearable computing device (100) shown in FIG. 1b and FIG. 1c) so that the user's pupil is aligned as much as possible in the center within the eye box (140), allowing the user's gaze to pass through the output coupler (144), whereby the displayed content is seen by the user through the lens (107). The measurements or parameters described above enable the configuration of the display device (104) of the head-mounted wearable computing device (100) so that the display of content is seen by the user within a relatively large portion of the user's field of vision.

[0057] A system and method according to an embodiment described herein may provide for the collection of measurements to determine ophthalmic fitting characteristics for a user wearing a selected fitting frame (500). In some embodiments, the system and method may provide for the collection of measurements to determine both display fit characteristics and ophthalmic fit characteristics. As described above in relation to FIGS. 2a through 2d, ophthalmic measurements may be detected within image data captured by a camera (360) of a computing device (300) operated by a user. For example, such measurements, including pupil height, interpupillary distance, monocular pupillary distance, foreground angle, vertex distance, and other parameters or measurements, may be detected within the image data and processed for the integration of corrective or prescription lenses into a head-mounted wearable computing device (100). In some examples, such ophthalmic measurements are used in conjunction with the display fit measurements described above to provide both the user's corrective / prescription lens requirements and an appropriate display fit for a specific user.

[0058] FIG. 8 is a flowchart of an exemplary method (800) for operating a computing system to configure a head-mounted wearable computing device including a display for a user. A user operating the computing device (e.g., the computing device (300) described above) may cause the computing device to run an application. The application may provide a display fit measurement and / or ophthalmic fit measurement detection from image data captured by the computing device (block 810). A camera (e.g., the camera (360) of the computing device (300) described above) is operated to capture image data of a user wearing a fitting frame (block 820). The fitting frame may be selected from one or more sample fitting frames available to the user (e.g., the fitting frame (500) described above in relation to FIG. 5a-7). The system can detect one or more keypoints or features on the fitting frame (e.g., keypoints (610, 620, 630 and / or 670), the face / skull landmarks described above (640, 650), etc.) (block 830).

[0059] In some examples, the analysis of image capture data for the detection of keypoints and / or features and / or landmarks may be performed by a computing device, for example, an object recognition module and / or pattern recognition module of the processor (350) of the computing device described above. In some examples, the detection of keypoints and / or features and / or landmarks may be performed by an external device, such as an object recognition module and / or pattern recognition module of a server included in an external resource (302) that communicates with the computing device (300) as described above.

[0060] Display fit measurements, including the three-dimensional pose of the fitting frame, may be determined based on the positions of keypoints and / or features and / or landmarks detected for the configuration of the fitting frame (Block 840). If necessary (Block 850), ophthalmic fit measurements may be detected from image data and display fit measurements for a user requiring a corrective / prescription lens to be integrated into a head-mounted wearable computing device (Blocks 860, 870, and 880). In some examples, the analysis for determining the display fit measurements and / or ophthalmic fit measurements may be performed by a computing device (e.g., a configuration identification module of the processor (350) of the computing device (300) described above). In some examples, the analysis for determining the display fit measurements and / or ophthalmic fit measurements may be performed by an external computing device (e.g., a configuration identification module of a server included in an external resource (302) communicating with the computing device (300) described above). Ophthalmic fit measurements and display fit measurements can be integrated into a head-mounted wearable computing device for a user to provide a viewing experience tailored to the user's optical needs.

[0061] FIG. 9 illustrates examples of computing devices (900) and mobile computer devices (950) that may be used with the techniques described herein. The computing device (900) is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital terminals, smart devices, home appliances, electronic sensor-based devices, televisions, servers, blade servers, mainframes, and other suitable computing devices. The computing device (950) is intended to represent various forms of mobile devices, such as personal digital terminals, cellular phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely exemplary and are not intended to limit the implementation of the invention described and / or claimed herein.

[0062] The computing device (900) includes a processor (902), memory (904), a storage device (906), a high-speed interface (908) connected to the memory (904) and a high-speed expansion port (910), and a low-speed interface (912) connected to the low-speed bus (914) and the storage device (906). The processor (902) may be a semiconductor-based processor. The memory (904) may be a semiconductor-based memory. Each of the components (902, 904, 906, 908, 910 and 912) may be interconnected using various buses and mounted on a common motherboard or in a suitable other way. The processor (902) may process instructions for execution within the computing device (900), including instructions stored in the memory (904) or the storage device (906), and display graphic information for a GUI on an external input / output device, such as a display (916) connected to the high-speed interface (908). In other embodiments, multiple processors and / or multiple buses may be appropriately used together with multiple memories and multiple types of memory. Additionally, multiple computing devices (900) may be connected to each device that provides parts of the necessary operation (e.g., server bank, blade server group, or multiprocessor system).

[0063] Memory (904) stores information within a computing device (900). In one embodiment, memory (904) is a volatile memory unit or units. In another embodiment, memory (904) is a non-volatile memory unit or units. Additionally, memory (904) may be another form of computer-readable medium, such as a magnetic or optical disk. Generally, the computer-readable medium may be a non-transient computer-readable medium.

[0064] A storage device (906) may provide large storage for a computing device (900). In one embodiment, the storage device (906) may be a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other configurations, or may include a computer-readable medium. A computer program product may be tangibly stored in an information carrier. A computer program product may also include instructions that perform one or more methods as described above and / or computer-implemented methods when executed. The information carrier is a computer or machine-readable medium, such as memory (904), a storage device (906), or memory on a processor (902).

[0065] The high-speed controller (908) manages bandwidth-intensive operations for the computing device (900), whereas the low-speed controller (912) manages low-bandwidth-intensive operations. The assignment of these functions is merely exemplary. In one embodiment, the high-speed controller (908) is connected to a high-speed expansion port (910) capable of accommodating memory (904), a display (916) (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In an embodiment, the low-speed controller (912) is coupled to a storage device (906) and a low-speed expansion port (914). The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router via a network adapter, for example.

[0066] The computing device (900) may be implemented in a number of different forms as illustrated in the drawing. For example, it may be implemented as a standard server (920) or as a group of such servers. It may also be implemented as part of a rack server system (924). It may also be implemented in a computer such as a laptop computer (922). Alternatively, components from the computing device (900) may be combined with other components within a mobile device (not shown), such as a device (950). Each of the devices may include one or more of the computing devices (900, 950), and the entire system may be composed of a number of computing devices (900, 950) communicating with each other.

[0067] The computing device (950) includes, in particular among several other components, an input / output device such as a processor (952), memory (964), and a display (954), a communication interface (966), and a transceiver (968). The device (950) may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components (950, 952, 964, 954, 966, and 968) is interconnected using various buses, and some components may be mounted on a common motherboard or in other suitable ways.

[0068] The processor (952) can execute instructions within the computing device (950) including instructions stored in memory (964). The processor may be implemented as a chipset of chips including separate and multiple analog and digital processors. The processor may provide coordination of other components of the device (950), such as user interfaces, applications executed by the device (950), and wireless communication by the device (950).

[0069] The processor (952) can communicate with the user through a control interface (958) and a display interface (956) connected to the display (954). The display (954) may include, for example, a TFT LCD (Thin-Film Transistor Liquid Crystal Display) or an OLED (Organic Light-Emitting Diode) display or other suitable display technology. The display interface (956) may include suitable circuitry for driving the display (954) to provide graphics and other information to the user. The control interface (958) may receive commands from the user and convert them to submit to the processor (952). Additionally, an external interface (962) may be provided for communication with the processor (952) to enable short-range communication between the device (950) and other devices. The external interface (962) may be provided for wired communication in some embodiments or for wireless communication in other embodiments, for example, and multiple interfaces may be used.

[0070] Memory (964) stores information within the computing device (950). Memory (964) may be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Additionally, extended memory (984) may be provided to and connected to the device (950) through an extended interface (982) which may include, for example, a Single In Line Memory Module (SIMM) card interface. The extended memory (984) may provide additional storage space for the device (950) or may store applications or other information about the device (950). In particular, the extended memory (984) may include instructions that perform or supplement the aforementioned processes and may also include security information. Thus, for example, the extended memory (984) may be provided as a security module for the device (950) and may be programmed with instructions that allow the secure use of the device (950). In addition, security applications may be provided with additional information through SIMM cards, such as by deploying identification information on the SIMM card in a way that makes it impossible to hack.

[0071] The memory may include, for example, flash memory and / or NVRAM memory as described below. In one embodiment, a computer program product is tangibly stored in an information carrier. The computer program product also includes instructions that perform one or more methods as described above when executed. The information carrier is a computer or machine-readable medium, such as memory (964), extended memory (984), or memory on a processor (952), which can be received, for example, through a transceiver (968) or an external interface (962).

[0072] The device (950) may communicate wirelessly through a communication interface (966) that may include a digital signal processing circuit as needed. The communication interface (966) may be provided for communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may occur, for example, through a radio frequency transceiver (968). Additionally, short-range communication may occur, such as using Bluetooth, low-power Bluetooth, Wi-Fi, or other transceivers (not shown). Additionally, a Global Positioning System (GPS) receiver module (980) may provide additional navigation and location-related wireless data to the device (950) that can be appropriately used by applications running on the device (950).

[0073] The device (950) may also communicate audibly using an audio codec (960) capable of receiving information spoken by a user and converting it into available digital information. The audio codec (960) may likewise generate audible sound for the user, such as through a speaker within the handset of the device (950), for example. Such sound may include sounds from voice phone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on the device (950).

[0074] The computing device (950) may be implemented in a number of different forms as illustrated in the drawing. For example, it may be implemented as a cellular phone (983). It may also be implemented as part of a smartphone (981), a personal digital assistant (PDA), or other similar mobile device.

[0075] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include embodiments in one or more computer programs executable and / or interpretable on a programmable system comprising at least one programmable processor, which may be dedicated or general-purpose, and may be connected to receive data and instructions from a storage system, at least one input device, and at least one output device and to transmit data and instructions thereto.

[0076] These computer programs (also known as modules, programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural languages ​​and / or object-oriented programming languages ​​and / or assembly / machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program, device, and / or device, e.g., magnetic disk, optical disk, memory, programmable logic device (PLD), used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0077] To provide interaction with a user, the systems and techniques described herein may be implemented in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or an LED (light emitting diode)) to display information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) to which a user can provide input to the computer. Other types of devices may be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form including acoustic, voice, or tactile input.

[0078] The systems and techniques described herein may be implemented in a computing system comprising, for example, a backend component such as a data server, a middleware component such as an application server, a user computer having a graphical user interface, a frontend component such as a web browser through which a user can interact with an embodiment of the systems and techniques described herein, or any combination of one or more of said backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication, for example, by a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0079] A computing system may include users and servers. Users and servers are typically located far apart from each other and generally interact through a communication network. The relationship between users and servers is generated by computer programs running on their respective computers that have a user-server relationship with each other.

[0080] In some embodiments, the computing device illustrated in FIG. 9 may include a sensor that interfaces with an AR / VR headset (VR glasses / glasses / VR headset / AR headset / HMD device (990)). For example, one or more sensors included in the computing device (950) or other computing devices illustrated in FIG. 9 may provide input to the AR / VR headset (990) or generally provide input to the AR / VR space. The sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biosensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device (950) may use the sensors to determine the absolute position and / or detected rotation of the computing device within the AR / VR space, which can be used as input to the AR / VR space. For example, the computing device (950) may be integrated into the AR / VR space as a virtual object such as a controller, laser pointer, keyboard, weapon, etc. When integrated into an AR / VR space, the positioning of computing devices / virtual objects by the user causes the user to position the computing device and view virtual objects in a specific way within the AR / VR space.

[0081] In some embodiments, one or more input devices included in or connected to the computing device (950) may be used as inputs to the AR / VR space. The input devices may include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or earbuds with input functions, a game controller, or other connectable input devices. When the computing device is integrated into the AR / VR space, a user interacting with the input device included in the computing device (950) may cause a specific action to occur in the AR / VR space.

[0082] In some embodiments, one or more output devices included in the computing device (950) may provide output and / or feedback to a user of the AR / VR headset (990) in an AR / VR space. The output and feedback may be visual, tactile, or auditory. The output and / or feedback may include, but are not limited to, rendering of the AR / VR space or virtual environment, vibration, turning on and off or blinking and / or flashing of one or more lights or strobes, sounding an alarm, chiming, playing a song, and playing an audio file. The output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), strobes, and speakers.

[0083] In some embodiments, a computing device (950) may be placed within an AR / VR headset (990) to create an AR / VR system. The AR / VR headset (990) may include one or more positioning elements that allow the placement of a computing device (950), such as a smartphone (981), at an appropriate location within the AR / VR headset (990). In these embodiments, the display of the smartphone (981) may render a stereoscopic image representing an AR / VR space or virtual environment.

[0084] In some embodiments, the computing device (950) may be displayed as another object in a computer-generated 3D environment. Interactions with the computing device (950) by a user (e.g., rotation, shaking, touchscreen touch, finger swiping across the touchscreen) may be interpreted as interactions with the object in the AR / VR space. As just one example, the computing device may be a laser pointer. In this example, the computing device (950) is displayed as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device (950), the user in the AR / VR space sees the movement of the laser pointer. The user receives feedback from interactions with the computing device (950) in the AR / VR environment on the computing device (950) or on the AR / VR headset (990).

[0085] In some embodiments, the computing device (950) may include a touchscreen. For example, a user may interact with the touchscreen in a specific way that makes what is happening on the touchscreen appear as if it is happening in an AR / VR space. For example, a user may use a pinching-type motion to zoom in on content displayed on the touchscreen. The pinching-type motion on the touchscreen may cause the information provided in the AR / VR space to be magnified. In some embodiments, the computing device may be rendered as a virtual book in a computer-generated 3D environment. Pages of the book may be displayed in the AR / VR space, and swiping the user's finger through the touchscreen may be interpreted as turning or flipping the pages of the virtual book. As each page is turned / flipped, in addition to the ability to see changes in the page content, audio feedback, such as the sound of turning pages in a book, may be provided to the user.

[0086] In some embodiments, in addition to computing devices (e.g., mouse, keyboard), one or more input devices may be rendered in a computer-generated 3D environment. The rendered input devices (e.g., rendered mouse, rendered keyboard) can be used as rendered in the AR / VR space to control objects in the AR / VR space.

[0087] Many embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention.

[0088] Additionally, the logical flows depicted in the drawings do not necessarily require a specific depicted order or chronological order to achieve the desired results. Furthermore, other steps may be provided, steps may be omitted from the described flow, and other components may be added to or removed from the described system. Accordingly, other embodiments are also within the scope of the following claims.

[0089] In addition to the above description, controls are provided to the user to make choices regarding when the systems, programs, devices, networks, or configurations described herein collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or user's current location) and when the user is transmitted to a communication from content or a server. Additionally, certain data is handled in one or more different ways before it is stored or used, so that user information is removed. For example, the user's identity is handled so that user information regarding the user cannot be determined, or the user's geographic location is generalized from where the location information was obtained (to the city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user may have control over what information regarding the user is collected, how the information is used, and what information is provided to the user.

[0090] A computer system (e.g., a computing device) may be configured to communicate wirelessly with a network server over a network through a communication link established with the network server, using any known wireless communication technology and protocol, including radio frequency (RF), microwave frequency (MWF), and / or infrared frequency (IRF) wireless communication technology and a protocol suitable for communication over a network.

[0091] According to aspects of this specification, embodiments of the various techniques described herein may be implemented in digital electronic circuits or in computer hardware, firmware, software, or a combination thereof. Embodiments may be implemented as computer programs (e.g., computer programs tangibly stored in information carriers, machine-readable storage devices, computer-readable media, tangible computer-readable media) to be processed by a data processing device (e.g., a programmable processor, a computer, or a number of computers) or to control the operation of a data processing device. In some embodiments, a tangible computer-readable storage medium may be configured to store instructions that cause a processor to perform a process when executed. Computer programs such as the computer programs described above may be written in any form of programming language, including compiled or interpreted languages, and may be distributed as standalone programs or as modules, components, subroutines, or any other units suitable for use in a computing environment. Computer programs may be distributed to be processed on a single computer or at a single site, or on a number of computers distributed across multiple sites and interconnected by a communication network.

[0092] The specific structural and functional details disclosed herein are represented for the purpose of illustrating exemplary embodiments. However, exemplary embodiments may be implemented in many alternative forms and should not be construed as being limited only to the embodiments described herein.

[0093] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. As used herein, the terms “comprises,” “comprising,” “includes,” and / or “including” include the specified configurations, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other configurations, steps, actions, elements, components, and / or groups thereof.

[0094] Additionally, where an element is referred to as being "combined," "connected," "responding" to, or existing "on" another element, it will be understood that the element is directly combined, connected to, responding to, or existing upon, or that intermediate elements may exist. In contrast, where an element is referred to as being "directly combined," "directly connected," "directly responded," or existing "immediately above" another element, intermediate elements may not exist. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0095] Spatially relative terms such as "beneath," "below," "lower," "above," and "upper" may be used herein for convenience of description to explain one element or configuration in relation to other element(s) or configuration(s) depicted in the drawings. Spatially relative terms are to be understood as intended to include other orientations of the device in use or operation in addition to the orientation depicted in the drawings. For example, if the device in the drawings is inverted, an element described as "below" or "beneath" another element or configuration or configuration orients "above" another element or configuration. Thus, the term "below" may include both upper and lower orientations. The device may be oriented differently (rotated by 70 degrees or other orientations), and spatially relative terms used herein may be interpreted accordingly.

[0096] Exemplary embodiments of the concept are described herein with reference to cross-sectional drawings, which are schematic diagrams of ideal embodiments (and intermediate structures) of the exemplary embodiments. As such, variations from the shapes of the examples are expected, for example, as a result of manufacturing techniques and / or tolerances. Accordingly, the exemplary embodiments of the described concept should not be interpreted as being limited to the specific shapes of the regions illustrated herein, and should include, for example, variations in shape resulting from manufacturing. Accordingly, the regions illustrated in the drawings are essentially schematic, and their shapes are not intended to illustrate the actual shapes of the regions of the device, nor are they intended to limit the scope of the exemplary embodiments.

[0097] It will be understood that while terms such as "first," "second," etc., may be used in this specification to describe various elements, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. Accordingly, without departing from the teachings of the present embodiment, the "first" element may be named the "second" element.

[0098] Unless otherwise defined, terms used herein (including technical and scientific terms) have the same meaning as generally understood by those skilled in the art to which the concept belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with that meaning in the context of the relevant technology and / or this specification, and should be understood that they should not be interpreted in an idealized or overly formal sense unless explicitly defined in this specification.

[0099] Although specific configurations of the described embodiments have been described as set forth in this specification, many modifications, substitutions, changes, and equivalents will arise for those skilled in the art. Therefore, it should be understood that the appended claims are intended to include such modifications and changes within the scope of the implementation. These embodiments are merely illustrative and not limiting, and it should be understood that various changes in form and detail may be made. Any part of the apparatus and / or method described in this specification may be combined in any combination except mutually exclusive combinations. The embodiments described in this specification may include various combinations and / or sub-combinations of functions, components, and / or configurations of the other embodiments described.

Claims

Claim 1 A method for detecting fit measurements for a wearable computing device, comprising: receiving a selection of a frame by an application running on the computing device; detecting at least one feature of a frame worn by a user in a two-dimensional (2D) image captured by a computing device operated by a user; accessing configuration information associated with the frame; determining a three-dimensional pose of the frame captured in the image based on the detection of the at least one feature and the configuration information associated with the frame; and determining the configuration of a display device of the wearable computing device based on the three-dimensional pose of the frame captured in the image. Claim 2 The method of claim 1, wherein the step of detecting the at least one feature comprises the step of detecting at least one of the following: a bridge portion of the frame; a hinge point between the rim portion and the arm portion of the frame; a lens periphery edge portion of the frame; or a saddle portion of the arm portion of the frame. Claim 3 A method according to claim 1, wherein the step of determining the three-dimensional pose of the frame comprises: accessing a three-dimensional model of the frame captured in the image; and performing a comparison comprising comparing a known position and a known orientation of at least one feature detected in the image with a corresponding position and a corresponding orientation of the at least one feature in the three-dimensional model of the frame. Claim 4 The method of claim 3, wherein the step of detecting at least one feature comprises detecting a plurality of features for the frame in an image captured by the computing device; and the step of performing the comparison comprises, for each of the plurality of features, comparing a known position and a known orientation of the feature detected in the image with a corresponding position and a corresponding orientation of the feature in a three-dimensional model of the frame. Claim 5 A method according to claim 4, wherein the step of determining the three-dimensional pose of the frame comprises the step of performing a correspondence between the two-dimensional position of each feature detected in the image and the corresponding three-dimensional position of each feature in the three-dimensional model of the frame. Claim 6 A method according to claim 1, further comprising: a step of detecting a plurality of face landmarks in an image captured by the computing device; and a step of determining an ophthalmic fit measurement for a wearable computing device based on the plurality of face landmarks detected in the image. Claim 7 The method of claim 6, wherein the step of detecting the plurality of face landmarks comprises: detecting the pupil height in an image including a frame worn by the user; detecting at least one of the distance between pupils or the monocular pupil distance in an image including a frame worn by the user; and determining the pantoscopic angle of the frame worn by the user based on the three-dimensional pose of the frame and the plurality of face landmarks. Claim 8 The method of claim 6, wherein the step of determining the configuration of a display device of a wearable computing device based on a three-dimensional pose of a frame captured in the image comprises the step of adjusting the configuration of the display device of the wearable computing device to accommodate the ophthalmic fit measurement. Claim 9 The method of claim 8, wherein the step of determining the configuration of a display device of the wearable computing device comprises: determining the user's field of view based on the ophthalmic fit measurement and the three-dimensional pose of the frame; and configuring an output coupler of the display device so that content output by the display device is displayed within the field of view. Claim 10 A method according to claim 1, wherein the step of accessing configuration information associated with the frame includes the step of accessing a three-dimensional model of the frame. Claim 11 The method of claim 10, wherein the three-dimensional model of the frame comprises at least one of a known linear measurement, a known angular measurement, or a known contour associated with the frame. Claim 12 A method according to claim 10, wherein the step of determining the three-dimensional pose of the frame comprises: detecting at least one corresponding feature in a three-dimensional model of the frame; and detecting the three-dimensional pose of the frame based on a comparison of at least one feature detected in the image and at least one corresponding feature detected in a three-dimensional model of the frame. Claim 13 A non-transient computer-readable medium for storing executable instructions, wherein the instructions are configured such that when executed by at least one processor, the at least one processor: receives a selection of a frame; captures an image of a frame worn by a user; detects at least one feature of the frame from the image; accesses configuration information associated with the frame; determines a three-dimensional pose of the frame captured in the image based on the detection of the at least one feature and the configuration information associated with the frame; and determines the configuration of a display device of a wearable computing device based on the three-dimensional pose of the frame captured in the image. Claim 14 A non-transient computer-readable medium according to claim 13, wherein detecting at least one feature of the frame from the image comprises: detecting a first feature including a bridge portion of the frame; detecting a second feature including a hinge point between a rim portion and an arm portion of the frame; detecting a third feature including a lens periphery edge portion of the frame; or detecting at least one of a fourth feature including a saddle portion of an arm portion of the frame. Claim 15 A non-transient computer-readable medium according to claim 13, wherein determining the three-dimensional pose of the frame comprises: accessing a three-dimensional model of the frame captured in the image; and performing a comparison of a known position and a known orientation of at least one feature detected in the image with a corresponding position and a corresponding orientation of the at least one feature in the three-dimensional model of the frame. Claim 16 A non-transient computer-readable medium according to claim 15, wherein detecting a plurality of features and performing the comparison comprises, for each of the plurality of features: comparing a known position and a known orientation of a feature detected in an image with a corresponding position and a corresponding orientation of a feature in a three-dimensional model of a frame. Claim 17 A non-transient computer-readable medium according to claim 16, wherein determining the three-dimensional pose comprises: performing a correspondence between the two-dimensional position of each feature detected in the image and the corresponding three-dimensional position of each feature in the three-dimensional model of the frame. Claim 18 A non-transient computer-readable medium according to claim 13, wherein the instruction enables the at least one processor to further: detect a plurality of face landmarks in the image; and determine an ophthalmic fit measurement for a wearable computing device based on the plurality of face landmarks detected in the image. Claim 19 A non-transient computer-readable medium according to claim 18, wherein detecting the plurality of face landmarks comprises: detecting a pupil height in an image including a frame worn by the user; detecting at least one of a distance between pupils or a monocular pupil distance in an image including a frame worn by the user; and determining a pantoscopic angle of the frame worn by the user based on a three-dimensional pose of the frame and the plurality of face landmarks. Claim 20 A non-transient computer-readable medium according to claim 18, wherein the instruction causes the at least one processor to adjust the configuration of a display device of the wearable computing device to accommodate the ophthalmic fit measurement, the adjustment comprising: determining the user's field of view based on the ophthalmic fit measurement and the three-dimensional pose of the frame; and configuring an output coupler of the display device so that content output by the display device is displayed within the field of view.

Images