Image-based fit detection for head-mounted wearable computing devices

Abstract

A system and method of detecting display fit measurements and / or ophthalmic measurements of a head-mounted wearable computing device including a display device is provided. An image of a fit frame worn by a user of the computing device is captured by the user through an application running on the computing device. One or more keypoints and / or features and / or landmarks are detected in the image including the fit frame. A three-dimensional pose of the fit frame is determined based on the detected keypoints and / or features and / or landmarks and configuration information associated with the fit frame. The display device of the head-mounted wearable computing device can then be configured based on the three-dimensional pose of the fit frame as captured in the image.

Description

[0001] Cross-reference to related applications

[0002] This application is a continuation-to-claim claim to U.S. Application No. 17 / 444,963, filed August 12, 2021, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This specification generally relates to the prediction of wearable adaptation and / or display adaptation and / or ophthalmic adaptation for wearable devices, and specifically to the prediction of display adaptation and ophthalmic adaptation for head-mounted computing devices including display capabilities. Background Technology

[0004] Wearable devices can include head-mounted devices such as smart glasses, headphones, earbuds, etc.; wrist and / or hand-worn devices such as smartwatches, smart bracelets, smart rings, etc.; smart bracelets; fitness trackers; cameras; body sensors; and other such devices. In some examples, users may want to select and / or customize wearable devices for fit and / or functionality. For example, a user might want to select and / or customize a pair of smart glasses to include a selectable frame, incorporating prescription lenses, and other such features. Existing systems for procuring these types of wearable devices do not provide accurate fit and customization, especially without visiting retail locations. Summary of the Invention

[0005] In one general aspect, a method for detecting display adaptation measurements of a head-mounted wearable computing device including a display device includes: by an application executing on the computing device, detecting at least one key point on an adapter frame worn by a user from an image captured by the computing device; accessing configuration information associated with the adapter frame detected in the image; determining a three-dimensional pose of the adapter frame captured in the image based on the detection of the at least one key point and the configuration information associated with the adapter frame; and determining a configuration of the display device of the head-mounted wearable computing device based on the three-dimensional pose of the adapter frame captured in the image.

[0006] In some implementations, detecting the at least one key point includes detecting at least one of the following: the bridging portion of the adapter frame; the hinge point between the edge portion and the arm portion of the adapter frame; the outer peripheral edge portion of the lens of the adapter frame; or the saddle portion of the arm portion of the adapter frame.

[0007] In some implementations, determining the 3D pose of the adapter frame includes: accessing a 3D model of the adapter frame captured in the image; and performing a comparison, including comparing the known position and known orientation of the at least one keypoint detected in the image with the corresponding position and corresponding orientation of the at least one keypoint in the 3D model of the adapter frame. Detecting at least one keypoint may include detecting a plurality of keypoints on the adapter frame in the image captured by the computing device; and performing the comparison may include, for each of the plurality of keypoints, comparing the known position and known orientation of the keypoint detected in the image with the corresponding position and corresponding orientation of the keypoint in the 3D model of the adapter frame. Determining the 3D pose of the adapter frame may include mapping the 2D position of each keypoint detected in the image to the corresponding 3D position of the keypoint in the 3D model of the adapter frame.

[0008] In some implementations, the method may further include: detecting multiple facial landmarks in the captured image; and determining ophthalmic adaptation measurements of the head-mounted wearable computing device based on the detected multiple facial landmarks. Detecting the multiple facial landmarks may include: detecting pupil height in an image of the adaptation frame worn by the user; detecting at least one of interpupillary distance and / or monocular pupillary distance in an image of the adaptation frame worn by the user; and determining a panoramic view of the adaptation frame worn by the user based on the determined three-dimensional pose of the adaptation frame and the detected multiple facial landmarks. Determining the configuration of the display device of the head-mounted wearable computing device based on the three-dimensional pose of the adaptation frame captured in the image may include: adapting the configuration of the display device of the head-mounted wearable computing device to accommodate the ophthalmic adaptation measurements. Determining the configuration of the display device of the head-mounted wearable computing device may include: determining the user's field of view based on the ophthalmic adaptation measurements of the adaptation frame and the three-dimensional pose; and configuring the output connector of the display device such that the content output by the display device is displayed within the determined field of view.

[0009] In another general aspect, a non-transitory computer-readable medium for storing executable instructions, which, when executed by at least one processor, are configured to cause the at least one processor to: capture an image of an adapter frame worn by a user; detect at least one key point on the adapter frame from the captured image; access configuration information associated with the adapter frame; determine a three-dimensional pose of the adapter frame captured in the image based on the detection of the at least one key point and the configuration information associated with the adapter frame; and determine a configuration of a display device for a head-mounted wearable computing device based on the three-dimensional pose of the adapter frame captured in the image.

[0010] In some implementations, the instructions cause the at least one processor to detect at least one keypoint on the adapter frame in the captured image, including at least one of the following: detecting a first keypoint, the first keypoint including a bridging portion of the adapter frame; detecting a second keypoint, the second keypoint including a hinge portion between an edge portion and an arm portion of the adapter frame; detecting a third keypoint, the third keypoint including an outer peripheral edge portion of a lens of the adapter frame; or detecting a fourth keypoint, the fourth keypoint including a saddle portion of the arm portion of the adapter frame. In some implementations, the instructions cause the at least one processor to determine the three-dimensional pose of the adapter frame, including: accessing a three-dimensional model of the adapter frame captured in the image; and comparing the known position and known orientation of the at least one keypoint detected in the image with the corresponding position and corresponding orientation of the at least one keypoint in the three-dimensional model of the adapter frame.

[0011] In some implementations, the instructions cause the at least one processor to detect a plurality of keypoints and perform the comparison, including, for each of the plurality of keypoints, comparing the known position and known orientation of the keypoint detected in the image with the corresponding position and corresponding orientation of the keypoint in the 3D model of the adapter frame. In some implementations, the instructions cause the at least one processor to determine the 3D pose, including mapping the 2D positions of each keypoint detected in the image with the corresponding 3D positions of the keypoints in the 3D model of the adapter frame.

[0012] In some implementations, the instructions further instruct the at least one processor to: detect multiple facial landmarks in a captured image; and determine ophthalmic adaptation measurements of the head-mounted wearable computing device based on the detected multiple facial landmarks. The instructions may cause the at least one processor to detect the multiple facial landmarks, including: detecting pupil height in an image of the adaptation frame worn by the user; detecting at least one of interpupillary distance or monocular pupillary distance in the image of the adaptation frame worn by the user; and determining a panoramic view of the adaptation frame worn by the user based on the determined three-dimensional pose of the adaptation frame and the detected multiple facial landmarks. In some implementations, the instructions instruct the at least one processor to adapt the configuration of the display device of the head-mounted wearable computing device to accommodate the ophthalmic adaptation measurements, including: determining a user field of view based on the ophthalmic adaptation measurements of the adaptation frame and the three-dimensional pose; and configuring the output connector of the display device such that content output by the display device is displayed within the determined field of view.

[0013] Details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the specification, drawings, and claims. Attached Figure Description

[0014] Figure 1A An example system based on the implementation described in this article is shown.

[0015] Figure 1B It is based on the implementation method described in this article. Figure 1A The example wearable computing device shown is a front view, and Figure 1C It is based on the implementation method described in this article. Figure 1A The example wearable computing device shown is a rear view.

[0016] Figures 2A to 2D Example ophthalmic measurements are shown.

[0017] Figure 3 This is a block diagram of the system based on the implementation method described in this article.

[0018] Figures 4A to 4B An image capture mode for determining display adaptation measurements and ophthalmic adaptation measurements, according to the implementation described herein, is shown.

[0019] Figures 5A to 5C This is a perspective view of an example adaptation framework based on the implementation described in this article.

[0020] Figures 6A to 6CAn example frontal image of an example adapter frame worn by a user, according to the implementation described herein, is shown, along with example features and / or key points and / or landmarks detected in the frontal image.

[0021] Figure 6D An example side image of an example adapter frame worn by a user, according to the implementation described herein, is shown, along with example features and / or key points and / or landmarks detected in the side image.

[0022] Figure 7 This is the front view of an example adaptation framework based on the implementation described in this article.

[0023] Figure 8 This is a flowchart of an example method based on the implementation described in this article.

[0024] Figure 9 An example computing device of the computing system discussed in this paper is shown. Detailed Implementation

[0025] This disclosure relates to systems and methods for selecting and fitting wearable computing devices, including determining wearable fitting parameters and / or display fitting parameters and / or ophthalmic fitting parameters associated with the wearable computing device. Systems and methods according to implementations described herein provide for determining wearable fitting and / or display fitting and / or ophthalmic fitting of the wearable computing device based on measurements detected within image data. In some implementations, systems and methods according to implementations described herein provide for the detection of wearable fitting and / or display fitting and / or ophthalmic fitting based on known parameters associated with a sample frame of the wearable computing device detected within image data. In some examples, for wearable computing devices in the form of smart glasses, the systems and methods described herein provide for the detection of wearable fitting and / or display fitting and / or ophthalmic fitting from image data, the smart glasses including a display and / or corrective / prescription lenses customized for the specific physical attributes, needs, and preferences of a particular user. The systems and methods described herein can facilitate the capture of image data for use by users in a self-guided, unsupervised, or unmonitored manner to detect measurement data without visiting retail facilities and / or making in-person or virtual appointments with sales agents.

[0026] The following describes systems and methods according to implementations of wearable computing devices in the form of head-mounted display devices (such as, for example, smart glasses including a display device), such that content displayed by the display device is viewable by / visible to a user wearing the wearable computing device. Where the user would benefit from a head-mounted wearable computing device including corrective lenses, the systems and methods described herein can provide determination of ophthalmic fitting measurements, thereby allowing the incorporation of corrective / prescription lenses into the head-mounted wearable computing device, while taking into account the characteristics of the corrective / prescription lenses in the configuration of the display device used for the head-mounted wearable computing device. That is, the systems and methods according to implementations described herein can employ image-based capture systems and methods to provide fitting between the display device and the corrective / prescription lenses, such that content displayed by the display device is visible to a user wearing the head-mounted wearable computing device.

[0027] Figure 1A A user is shown operating a mobile computing device in the system, wherein the mobile computing device can access one or more external resources 1100 via network 1200. Figure 1A A third-person view of the user in the surrounding environment 1000 is provided, which includes various examples of mobile computing devices 100, 180, 190, and 200. Figure 1A The illustrated example mobile computing devices include a first head-mounted wearable computing device 100, a second head-mounted wearable computing device 180, a wrist-worn computing device 190, and a handheld computing device 200. In some examples, the first head-mounted computing device 100 may include various components, such as a display, a camera capable of capturing images of the surrounding environment, audio input / output capabilities, user input capabilities, etc., such as in a head-mounted display device, in the form of smart glasses, or in the form of headphones. In some examples, the second head-mounted computing device 180 may be an ear-worn computing device such as headphones or earbuds, and may include audio input / output capabilities, a camera capable of capturing images of the surrounding environment, user input capabilities, etc. In some examples, the wrist-worn computing device 190 may include a display, a camera capable of capturing images of the surrounding environment, audio input / output capabilities, user input capabilities, etc., such as in a smartwatch or wristband. In some examples, the handheld computing device 200 may include a display, a camera capable of capturing images of the surrounding environment, audio input / output capabilities, user input capabilities, etc., such as in a smartphone. Figure 1A Only a few examples of mobile computing devices are shown. The principles described in this article can be applied to... Figure 1A Other types of mobile computing devices not specifically shown. In some examples, including... Figure 1AThe mobile computing devices shown in the example computing devices can communicate with each other and / or with external resources 1100 to exchange information, receive and send inputs and / or outputs, etc.

[0028] Figure 1B yes Figure 1A The example head-mounted wearable computing device 100 shown is a front view, and Figure 1C yes Figure 1A The image shows a rear view of an example head-mounted wearable computing device 100. In some implementations, the example head-mounted wearable computing device 100 may take the form of smart glasses or augmented reality glasses (such as those used in...). Figure 1B and Figure 1C (As shown in the examples) or in the form of augmented reality and / or virtual reality headsets or goggles, etc. In the following, for ease of discussion and illustration only, the system and method of implementing a wearable computing device 100 in the form of smart glasses according to the methods described herein will be described. The principles described herein can be applied to other types of wearable computing devices and / or combinations of mobile / wearable computing devices working together.

[0029] like Figure 1B As shown, the example head-mounted wearable computing device 100 includes a frame 102. Figure 1B and Figure 1C In the example shown, frame 102 includes an edge portion 103 surrounding glass portion 107 or lens 107. For example, arm portion 105 is pivotally or rotatably coupled to a corresponding edge portion 103 of frame 102 via a corresponding hinge portion 110. In some examples, lens 107 may be a corrective / prescription lens. In some examples, lens 107 may be a glass portion that does not necessarily incorporate corrective / prescription parameters. In some examples, bridging portion 109 may connect to the edge portion 103 of frame 102. Display device 104 may be coupled into a portion of frame 102. Figure 1B and Figure 1CIn the example shown, display device 104 is attached to arm portion 105 of frame 102. With display device 104 attached to arm portion 105, eyebox 140 extends toward lens 107 for outputting content at output connector 144, where the content output by display device 104 is visible to the user. In some examples, output connector 144 may substantially coincide with lens 107. Head-mounted wearable computing device 100 may also include audio output device 106 (e.g., one or more speakers), illumination device 108, sensing system 111, control system 112, at least one processor 114, and outward-facing image sensor 116 or camera 116. In some implementations, display device 104 may include a see-through near-eye display. For example, display device 104 may be configured to project light from a display source onto a portion of teleprompter glass, which acts as a beam splitter positioned at an angle (e.g., 30-45 degrees). This beam splitter allows for both reflection and transmission values, meaning that light from the display source is partially reflected while the remaining light is transmitted through. Such an optical design allows a user, for example, to simultaneously see physical objects in the world through lens 107, adjacent to content generated by display device 104 (e.g., digital images, user interface elements, virtual content, etc.). In some implementations, waveguide optics can be used to depict the content on display device 104.

[0030] In some implementations, the wearable computing device 100 may include a gaze tracking device 120, which includes, for example, one or more sensors 125, to detect and track eye gaze direction and movement. Data captured by the sensors 125 may be processed to detect and track gaze direction and movement as user input. In some implementations, the sensing system 111 may include various sensing devices, and the control system 112 may include various control system devices, including, for example, one or more processors 114 operatively coupled to components of the control system 112. In some implementations, the control system 112 may include a communication module that provides communication and exchange of information between the wearable computing device 100 and other external devices.

[0031] When a wearable computing device 100 (such as...) is identified for a specific user... Figures 1A to 1CWhen considering the size and fit of the example smart glasses shown, many different size and fit measurements and / or parameters can be taken into account. This can include, for example, wearable fit parameters or wearable fit measurements. Wearable fit parameters / measurements can take into account how well a particular frame 102 fits on a particular user and / or how it looks and / or feels. Wearable fit parameters / measurements can take into account many factors, such as whether, for example, the frame 102 is wide enough to be comfortable relative to the temples, but not so wide that the frame 102 cannot remain relatively stationary when worn by the user (i.e., too large or too wide). Wearable fit parameters / measurements can take into account other factors, such as whether the dimensions of the edge portion 103 and the bridging portion 109 are sized to allow the bridging portion 109 to rest comfortably on the user's nose, whether the dimensions of the arm portion 105 are sized to allow it to rest comfortably on the user's ears, and other such comfort-related considerations. Wearable adaptation parameters / measurements may take into account considerations during wear, including how the user wears the frame 102 naturally, such as head posture / how the user naturally holds his / her head, how the user positions the frame relative to his / her face, etc. Wearable adaptation parameters / measurements may consider whether the size and / or shape and / or contour of the frame 102 is aesthetically pleasing to the user and whether it is compatible with the user's facial features. In some examples, wearable adaptation parameters / measurements may consider whether the adaptation associated with a particular frame configuration can accommodate the incorporation of a display device so that the display area can be coordinated with the user's natural field of view. In some examples, wearable adaptation parameters may consider whether the eye-tracking camera arrangement associated with a particular frame configuration can effectively track the user's eye gaze and movement.

[0032] When determining the size and fit of the wearable computing device 100 for a specific user, display fit parameters or display fit measurements may be considered. Display fit parameters / measurements can be used to configure the display device 104 for a specific set of frames 102 for a particular user, such that the content displayed by the display device 104 is visible to the user. For example, display fit parameters / measurements can provide an indication of the placement of the display device 104, such that the content displayed by the display device 104 is captured within at least a predetermined portion of the user's field of view. For example, display fit parameters / measurements can be used to configure the display device 104 to provide at least a predetermined level of gazeability, which corresponds to the amount, portion, or percentage of content displayed to the user at a predetermined brightness level and a predetermined pupil size at the periphery of the user's field of view (e.g., the minimum visible corner). These display fit parameters / measurements can be used to configure the display device 104 to provide the best possible placement for content display to the user for a specific set of frames 102.

[0033] In some examples, when sizing and fitting a wearable computing device 100 that includes a prescription or corrective lens 107, ophthalmic fitting parameters or ophthalmic fitting measurements may be considered. Figures 2A to 2D Some example ophthalmic fitting measurements are shown. Ophthalmic fitting measurements may include, for example, pupillary height PH (distance from the center of the pupil to the bottom of lens 107), including, for example, left pupillary height and right pupillary height. Ophthalmic fitting measurements may include interpupillary distance IPD (distance between pupils). IPD may be characterized by monocular pupillary distances, such as left pupillary distance LPD (distance from the center of the bridge of the nose to the left pupil) and right pupillary distance RPD (distance from the center of the bridge of the nose to the right pupil). Ophthalmic fitting measurements may include panoramic angle PA (angle defined by the tilt of lens 107 relative to the vertical direction). Ophthalmic fitting measurements may include vertex distance V (distance from the cornea to lens 107), including, for example, left vertex distance and right vertex distance. Ophthalmic fitting measurements may include other such parameters, or provide sizing and / or fitting measurements of the head-mounted wearable computing device 100 including display device 104 as described above. Figure 2C The vertex distance V associated with the relatively low diopter lens 107 is shown. Figure 2D The vertex distance V associated with the relatively high diopter lens 107 is shown. When adapting a wearable computing device 100 including a display device 104 to a user, ophthalmic adaptation measurements can be considered. For example, ophthalmic adaptation measurements (along with display adaptation measurements) can provide that the display of content is placed within an eyebox by the display device 104, defined by a three-dimensional volume extending between the lens 107 and the user's eye, where the display of content will be within the user's field of vision and, moreover, within the user's corrected field of vision, and thus visible to the user.

[0034] In the systems and methods implemented according to the methods described herein, it is possible to use a user-operated computing device (such as those mentioned above) Figure 1AImage data is captured by an application running on a computing device (200) or other computing device operated by a user. Wearable adaptation measurements and / or display adaptation measurements and / or ophthalmic adaptation measurements can be detected from the image data acquired in this manner to sizing and fitting head-mounted wearable computing devices, such as the wearable computing device 100 described above, which includes a display and / or corrective lenses for a specific user. Image data can be captured in an unsupervised or unmonitored manner via an application running on the user-operated computing device 200, without the need for visits to retail outlets or appointments with fitting / sales professionals. In some implementations, the detection of wearable adaptation measurements and / or display adaptation measurements and / or ophthalmic adaptation measurements can be based on the detection of physical features or attributes of a sample frame worn by the user included in the image data. A three-dimensional model of the user's head can be generated or defined based on the measurements detected within the captured image data (e.g., measurements of the user's physical attributes). In some implementations, the detection of physical features and / or attributes in the image data can be provided using an image sensor or camera of the computing device 200, without relying on data collected by a depth sensor. In some implementations, the depth sensor included in the computing device 200 can provide a set of three-dimensional measurements associated with the detected physical features and / or attributes of the sample frame and / or the user's facial / cranial features.

[0035] Figure 3 This is a block diagram of an example system for determining display adaptation measurements and ophthalmic adaptation measurements of a wearable computing device according to the implementation described herein. The system may include one or more computing devices 300. The computing device 300 may selectively communicate via a network 306 to access external resources 302, such as, for example, server computer systems, processors, databases, memory storage, etc. The computing device 300 may operate under the control of a control system 370. The computing device 300 may communicate directly (via wired and / or wireless communication) or via the network 306 with one or more external computing devices 304 (another wearable computing device, another mobile computing device, etc.). In some implementations, the computing device 300 includes a communication module 380 to facilitate external communication. In some implementations, computing device 300 includes a sensing system 320 comprising various sensing system components, such as one or more image sensors 322, one or more position / orientation sensors 324 (including, for example, inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.), one or more audio sensors 326 capable of detecting audio input, and other such sensors. Computing device 300 may include more or fewer sensing devices and / or combinations of sensing devices.

[0036] In some implementations, computing device 300 may include one or more image sensors or cameras 360. Cameras 360 may include, for example, outward-facing cameras, world-facing cameras, etc., capable of capturing still and / or moving images of the environment outside computing device 300. In some implementations, one or more cameras 360 may include depth sensors. Still and / or moving images may be displayed by a display device of output system 340 and / or transmitted externally via communication module 380 and network 306, and / or stored in memory 330 of computing device 300. Computing device 300 may include one or more processors 350. Processors 350 may include various modules or engines configured to perform various functions. In some examples, processor 350 includes object recognition modules, feature recognition modules, pattern recognition modules, configuration identification modules, and other such processors. Processor 350 may be formed in a substrate configured to execute one or more machine-executable instructions or software, firmware, or combinations thereof. Processor 350 may be semiconductor-based, comprising semiconductor materials capable of executing digital logic. Memory 330 may include any type of storage device that stores information in a format that can be read and / or executed by processor 350. Memory 330 may store applications and modules that perform certain operations when executed by processor 350. In some examples, applications and modules may be stored on external storage devices and loaded into memory 330.

[0037] Figure 4A and Figure 4B Example computing devices are shown that are operated by the user to capture display adaptation measurements and / or ophthalmic adaptation measurements for wearable computing devices to be customized for user use, such as those described above. Figure 3 The computing device 300 is described. Figure 4A In the example, computing device 300A is in the form of a handheld computing device, such as a smartphone. Figure 4B In the example, computing device 300 is in the form of laptop computing device 300B. Figure 4A and Figure 4B Two examples of computing devices 300 are provided, which can be used to capture image data for processing by a sizing simulator to perform the systems and methods described herein. The principles described herein can also be performed by other types of computing devices, and in particular by computing devices capable of executing applications thereon to utilize image capture capabilities, display capabilities, and external communication, as well as processing to collect and process display-adaptive measurements and / or ophthalmic measurements for customizing wearable computing devices for specific users.

[0038] exist Figure 4AIn the example shown, the user is holding the computing device 300A such that the user's head and face are within the field of view of the camera 360A on the computing device 300A, and the camera 360A can capture images of the user's head and face. The captured images can be displayed to the user on the display device 342A of the computing device 300A, allowing the user to verify that his / her head and face have been captured within the field of view of the camera 360A. Similarly, as Figure 4B As shown, the computing device 300B is positioned relative to the user, such that the user's head and face are captured within the field of view of the camera 360B of the computing device 300B, and the camera 360B can capture images of the user's head and face. The captured images can be displayed to the user on the display device 342B of the computing device 300B, allowing the user to verify that his / her head and face have been captured within the field of view of the camera 360B. Figure 4A and Figure 4B In the example arrangement shown, image data captured by the camera 360 can be processed (e.g., via... Figure 3 The external processor (including its recognition and simulation engines) detects wearable fit measurements and / or display fit measurements and / or ophthalmic fit measurements, and determines, for example, frame and / or lens size and profile, display device configuration, etc., to customize the wearable computing device, such as... Figure 1B and Figure 1C The head-mounted wearable computing device 100 shown is for the user.

[0039] exist Figure 4A and Figure 4B In the example shown, the user wears an example adapter frame 500. The adapter frame 500 is included in an image of the user's head and face captured by the camera 360 of the computing device 300.

[0040] As described above, detecting wearable fit measurements and / or displaying fit measurements and / or ophthalmic fit measurements from captured image data can be facilitated by detecting one or more physical features and / or attributes associated with the fit frame worn by the user during image data capture. In some examples, the user can try or sample multiple different fit frames from a collection or kit of sample fit frames. Fitting kits including sample fit frames can be provided to the user prior to the fitting session. This allows the user to evaluate at least some wearable fit parameters. For example, by trying or sampling multiple different fit frames including fitting kits containing sample fit frames, the user can evaluate factors such as, for example, physical sizing, comfort, aesthetics, etc. The user can select a fit frame including sample fit frames for use from the fitting kits during the fitting session. The sample fit frames are included in the fitting kits and, based on data captured during the fitting session, can simulate the size, shape, weight, etc., of the actual frame to be incorporated into the head-mounted wearable computing device 100 in order to provide the user with a relatively accurate fit and feel. Image data is captured during the fitting session via an application running on computing device 200. This image data can be used to determine display fitting and / or ophthalmic fitting. Based on measurements detected from the image data, the measurement data collected in this way can be used to customize display device 104 for the user and the selected fitting frame, and particularly the image display area generated by display device 104. The measurement data collected in this way can be used to incorporate corrective or prescription lenses into head-mounted wearable computing device 100. The measurement data collected in this way can be used to configure display device 104 for the user and the selected frame, and also to incorporate corrective / prescription lenses into head-mounted wearable computing device 100.

[0041] Figures 5A to 5C Example adapter frames 500, such as a first example adapter frame 500A, a second example adapter frame 500B, and a third example adapter frame 500C, are shown according to the implementation described herein for adapting to a head-mounted wearable computing device. The example adapter frames 500 may physically or geometrically represent the actual frame of the head-mounted wearable computing device to be worn by a user, but are only for sizing / adaptation purposes and therefore do not function. Figures 5A to 5C The example adapter frame 500 shown is an example of a sample adapter frame 500, which can be included in adapter components provided to the user for consideration prior to the adaptation session. The principles described herein apply to adapter frames with other shapes and / or sizes and / or configurations. Figure 5AThe first example adapter frame 500A shown has a first size and / or shape and / or profile defined by a first set of physical features. The first example adapter frame 500A can be considered as a medium-sized frame with D-shaped edge portions 503A, 503B. Figure 5B The second example adapter frame 500B shown has a second size and / or shape and / or profile defined by a second set of physical features. The second example adapter frame 500B can be considered as a medium-sized frame with rounded edge portions 503A, 503B. Figure 5C The third example adapter frame 500C shown has a third size and / or shape and / or profile defined by a third set of physical features. The third example adapter frame 500C can be considered as a large-sized frame with rectangular edge portions 503A, 503B. For discussion and illustration purposes, [the following is presented]. Figures 5A to 5C The example adapter frames shown are 500 (500A, 500B, 500C). The principles described herein can be applied to other adapter frames with other dimensions and / or shapes and / or profiles defined by other physical characteristics.

[0042] Figures 5A to 5CEach example adapter frame 500 (500A, 500B, 500C) shown includes a first edge portion 503A surrounding a first glass portion or a first lens 507A and a second edge portion 503B surrounding a second glass portion or a second lens 507B. When a user wears the example adapter frame 500 (500A, 500B, 500C), the first edge portion 503A / first lens 507A can be configured to be positioned corresponding to the user's first eye, and the second edge portion 503B / first lens 507B can be configured to be positioned corresponding to the user's second eye. A bridging portion 509 may extend between the inner end portions of the edge portions 503A, 503B to connect the edge portions 503A, 503B of the example adapter frame 500 (500A, 500B, 500C). The bridging portion 509 can be configured to rest on the user's nose to position the first edge portion 503A / first lens 507A corresponding to the first eye and the second edge portion 503B / second lens 507B corresponding to the second eye. The first arm portion 505A can be connected via a first hinge portion 510A, for example, pivotally or rotatably connected to the outer end portion of the first edge portion 503A. The second arm portion 505B can be connected via a second hinge portion 510B, for example, pivotally or rotatably connected to the outer end portion of the second edge portion 503B. The first arm portion 505A and the second arm portion 505B may each include ear saddle portions 515A, 515B, configured to rest on the user's respective first and second ears when the example adapter frame 500 (500A, 500B, 500C) is worn by the user.

[0043] As described above, the adaptation framework according to the implementation described herein may include the following physical features or attributes that can be detected, identified, or recognized in image data including the adaptation framework worn by the user. Figure 6A An example two-dimensional image plane 600 or image 600 of a user wearing an example adapter frame 500 is shown. The example image 600 can be captured by a camera 360 of a user-operated computing device 300 through an application executed by the computing device 300. Figure 6A In the example shown, image 600 is displayed on display device 342 of computing device 300. This allows the user to visually verify that the captured image data includes the capture adapter frame 500, the user's face and / or cranial and / or optical features, etc. In an example where computing device 300 is a handheld mobile computing device (such as a smartphone), a self-portrait mode or selfie mode can be invoked for image data captured by the camera 360 of computing device 300.

[0044] Image 600 may include a two-dimensional pixel array captured by camera 360. In some examples, features detected within image 600 may correspond to the locations of features detected within image 600, such as two-dimensional coordinate locations. Figure 6B Examples of detectable keypoints or detectable features associated with the adapter frame 500 worn by the user are shown when capturing image data. Keypoints or features detected in the image data can allow determination of the three-dimensional pose of the adapter frame 500 on the user's face, which can then be used to determine display adaptation information and / or ophthalmic adaptation information associated with the adapter frame 500 worn by the user. Detecting these types of keypoints associated with the adapter frame 500 worn by the user in the image data is crucial in determining the display adaptation information and / or ophthalmic adaptation information that allows the head-mounted wearable computing device 100 to be configured for the user.

[0045] Figure 6B The example arrangement shown includes an example key point 610A, which is located in image 600 at a position corresponding to the hinge connection point between the first edge portion 503A and the first arm portion 505A of the adapter frame 500, for example, at a position corresponding to the first hinge portion 510A that pivotally connects the edge portion 503A and the first arm portion 505A. Figure 6B The example arrangement shown includes example key point 610B, which is located in image 600 at the hinge connection point between the second edge portion 503B and the second arm portion 505B of the adapter frame 500. Figure 6B The illustrated example arrangement includes example keypoints 620A and 620B. Example keypoint 620A is located at the bottom outer peripheral edge portion of the first lens 507A corresponding to the adapter frame 500, and example keypoint 620B is located at the bottom outer peripheral edge portion of the second lens 507B corresponding to the adapter frame 500. Example keypoints 620A and 620B are merely example keypoints corresponding to specific outer peripheral edge portions of lens 507. Keypoints may also and / or alternatively be defined at other outer peripheral edge portions of lens 507, such as, for example, the top outer peripheral edge portion, left outer peripheral edge portion, and / or right outer peripheral edge portion of lens 507. In some examples, keypoints defined along the outer peripheral edge portion of lens 507 may correspond to designated sections of lens 507. In some examples, keypoints defined along the outer peripheral edge portion of lens 507 may correspond to specific points along the outer periphery edge of lens 507. Figure 6B The example arrangement shown includes example keypoint 630, which is located in image 600 at a position corresponding to the bridging portion 509 of the adapter frame 500.

[0046] Figure 6A and Figure 6B The example image 600 shown is a substantially frontal image, and the example key points 610, 620, 630 or features 610, 620, 630 are merely examples of detectable key points or features associated with the adapter frame 500 worn by the user, which can be detected in image 600 to detect display adaptation and / or ophthalmic adaptation associated with the user's adapter frame 500. Other key points or features associated with the adapter frame 500 can be detected in image 600 to facilitate determination of display adaptation and / or ophthalmic adaptation. Examples of other detectable key points or detectable features may include, for example, the top portion of lens 507, the inner peripheral portion of lens 507 (i.e., the portion of lens 507 adjacent to the bridging portion 509 of adapter frame 500), the outer peripheral portion of lens 507 (i.e., the portion of lens 507 adjacent to the hinge portion 510 of adapter frame 500), etc. The detection of the distribution of key points or features across the adapter frame 500 and the separation of key points or features can improve detection accuracy and relevance to the 3D model. Additionally, other key points or features associated with the adapter frame 500 can be detected in a side view of the user or an image captured at an angle relative to the user. These other key points or features may include portions of the arm portion 505 of the adapter frame 500, the position of the ear saddle point 515 relative to the user's ear, the position of the edge portion 503 relative to the user's face, etc. More or fewer key points and / or combinations of key points associated with the adapter frame 500 can be detected in the image 600, thereby determining the display adaptation and / or ophthalmic adaptation of the adapter frame 500 for the user. As described above, detecting specific key points associated with the adapter frame 500 worn by the user in the image data is crucial when determining the display adaptation information and / or ophthalmic adaptation information that allows the configuration of the head-mounted wearable computing device 100 for the user.

[0047] Figure 6C This illustrates multiple facial landmarks that can be detected in a two-dimensional image 600 captured by a camera 360 of a computing device 300 (along with those mentioned above). Figure 6B The described examples may detect keypoints 610, 620, 630 or features 610, 620, 630. In some examples, the pupils of each of the user's eyes can be detected in image 600, and specifically the pupil center 640. In some examples, other facial landmarks 650 can be detected in image 600. Figure 6C In the examples shown, for discussion and illustration purposes only, some example facial landmarks 650 include the bridge of the nose, quadrants surrounding the bridge of the nose at the temples and cheeks. Other facial landmarks can be detected to identify features related to the positioning of the adapter frame 500 on the user's head / face. Facial landmarks 650 can provide indexing, for example, for positioning the adapter frame 500 relative to the user's head / face.

[0048] In some implementations, a side or lateral image of 660 can be captured, such as... Figure 6D As shown. For discussion and illustrative purposes only. Figure 6D The example image 660 shown is a substantially complete side view image 660. Information can be detected from the side view image 660, which is related to... Figures 6A to 6C The frontal image shown was taken at an angle of less than 90 degrees. (Example) Figure 6D As shown, additional keypoints or features associated with the fitting frame, as well as additional facial / cranial landmarks, can be detected in the side or lateral image 660. Figure 6D In the example shown, a key point 670 can be detected in image 660 at the location corresponding to the ear saddle portion 515A of the arm portion 505A of the adapter frame 500. For example, facial / cranial landmarks 650 can be detected at portions of the user's ear (e.g., the anterior crease of the ear, the top portion of the ear, etc.). Other key points and / or facial / cranial features can also be detected from the side / lateral image 660. Similarly, the tilt or angle of the adapter frame 500 relative to the user's face (e.g., panoramic view PA) can be detected from the side or lateral image 660. That is, while the angle between the edge portion 503 and the arm portion 505 of the adapter frame 500 may be known, the tilt or angle of the adapter frame 500 will vary based on many factors. These factors may include, for example, how the user wears the adapter frame 500 (closer / farther from the eyes, higher / lower on the nose, etc.), the positioning of the ear relative to the user's nose, and other such factors.

[0049] In some implementations, the detected key points (such as...) are associated with the adapter framework 500. Figures 6B to 6D The example key points 610, 620, 630, 670 or features 610, 620, 630, 670 shown are detected individually and / or in conjunction with detected facial / cranial features of the user (such as...). Figure 6C and Figure 6D The detected pupil center 640 and facial landmark 650 shown can be used together to determine pose, for example, the three-dimensional pose of the adapter frame 500 on the user's face. This three-dimensional pose of the adapter frame 500 on the user's face can be used to determine display adaptation and / or ophthalmic adaptation of the head-mounted wearable computing device 100 (e.g., in the form of smart glasses). In some examples, the configuration of the frame 102 of the head-mounted wearable computing device 100 to be configured corresponds to the configuration of the adapter frame 500 worn by the user in image 600. In some examples, the configuration of the frame 102 of the head-mounted wearable computing device 100 to be configured does not necessarily correspond to the configuration of the adapter frame 500.

[0050] In some examples, data extracted from, for example, a two-dimensional frontal image 600 and the corresponding detected three-dimensional pose of the fitting frame 500 on the user's face can be synthesized to assess the gazeability of the user's fitting frame 500. That is, this data can be synthesized to determine if the configuration of the display device 104 included in the head-mounted wearable computing device 100 allows the display device 104 to be configured such that the content output by the display device 104 is within the field of vision of the user wearing the head-mounted wearable computing device 100. For example, the display device 104 can be configured such that the content output by the display device 104 and displayed at the output connector 144 is captured within the user's field of vision and is visible to the user.

[0051] As described above, the three-dimensional pose information related to the user's pupil (e.g., pupil center 640) and face (e.g., detected facial landmark 650), along with the three-dimensional pose information of the adapter frame 500, allows for the determination of lens-related measurements (e.g., pupil height, vertex distance, panoramic view, etc.). As described above, the three-dimensional position and orientation or pose of the adapter frame 500 can be determined based on the detection and / or identification of one or more keypoints 610, 620, 630 or features 610, 620, 630 in image data captured by the camera 360 of the user-operated computing device 300. In some examples, additional information can be obtained from the side or lateral image 660, as described above regarding... Figure 6D As described. Although some buckling may exist in the adapter frame 500, in some examples, the adapter frame 500 can be considered a substantially rigid body with three degrees of mobility. Therefore, the three-dimensional position of the adapter frame 500 can be determined using relatively accurate detection of at least one known point on the adapter frame 500. One or more key points or features of key points 610, 620, 630 or features 610, 620, 630 can be reliably detected in image 600, and the position of the detected one or more key points 610, 620, 630 or features 610, 620, 630 is known in the three-dimensional space associated with the adapter frame 500.

[0052] As described above, the systems and methods implemented herein can facilitate the adaptation of head-mounted wearable computing devices based on image data captured via an application running on a user-operated computing device. The image data can be captured by the user, and the wearable computing device can be adapted and configured for the user in a self-orienting, unsupervised, or unmonitored manner, without requiring in-person or virtual appointments with retail outlets and / or sales agents. The capture and processing of image data will be described in more detail below.

[0053] Users who wish to select, adapt, and configure the head-mounted wearable computing device 100 as described above may be provided with one or more adapter frames or a sample set of adapter frames for consideration prior to the adaptation process. In some examples, users may try each adapter frame included in the sample set of adapter frames to evaluate factors associated with each adapter frame, such as size, comfort, appearance, compatibility with facial features, overall wearability, etc. Users may select an adapter frame 500 from the sample set of adapter frames based on these and / or other factors for use during the adaptation and configuration process.

[0054] In some examples, a sample set of adaptable frames to be tried and / or evaluated can be selected by the user from a plurality of adaptable frames that can be used for trying. In some examples, the system (e.g., from a module of an application running on a user-operated computing device 300) can suggest a sample set of adaptable frames to the user based on analysis of the user's physical characteristics and the shape and / or size and / or configuration of the frames, the shape and / or size and / or configuration of which can be compatible with the detected physical characteristics. In some examples, the sample set of adaptable frames can be provided to the user before initiating the adaptation and configuration process for the head-mounted wearable computing device 100.

[0055] The ability to physically try out one or more pairs of adapter frames 500 provides users with the opportunity to physically evaluate the fit to determine which adapter frame 500 is physically most comfortable and properly / comfortably aligned with facial and / or cranial features (eyes, bridge of the nose, saddle point of the ear, cheek contact point, etc.). This evaluation of wearable fit by the user using the physical adapter frames 500 before fitting and configuring the head-mounted wearable computing device 100 can result in a head-mounted wearable computing device 100 that is more suitable for the specific user. Selecting adapter frames 500 based on the user's physical wearable fit in this way allows for a more accurate fit and configuration of the head-mounted wearable computing device 100 for the user.

[0056] In some cases, a single adapter frame 500 may be available to the user for adaptation and configuration to the head-mounted wearable computing device 100. In this case, the adaptation and configuration process may be performed as described below, wherein additional means are taken to determine the wearability of a particular frame 102 for integration into the head-mounted wearable computing device 100.

[0057] In some examples, the user can operate the computing device 300 (e.g., as in...). Figure 4A and / or Figure 4BThe application is launched and the adaptation and configuration of the head-mounted wearable computing device 100 is initiated using the selected adapter frame 500 (e.g., selected from a sample set of adapter frames as described above). In some examples, the application running on the user-operated computing device 300 may prompt the user to initiate image capture when wearing the selected adapter frame 500. In the following description, for ease of discussion and illustration only, the process will be based on the capture of a frontal image 600, as described above regarding... Figures 6A to 6C As described. However, in some implementations, in situations such as Figure 6D The data detected in the side image 660 and / or the half-side image shown can be adapted and configured for the user's head-mounted display device 100.

[0058] In some examples, the application may prompt the user to confirm the selection of a specific adapter frame 500 to be worn by the user during image data capture. Therefore, in some cases, configuration information associated with the selected adapter frame 500 to be worn during image capture may be known. This may include, for example, linear and / or angular measurements and / or contours associated with the selected adapter frame 500, a two-dimensional and / or three-dimensional model of the selected adapter frame 500, and other such information. The application running on the computing device 300 may access this known information associated with the selected adapter frame 500. For example, this known configuration information may be stored in a database accessible to the application running on the computing device 300. The known configuration information associated with the selected adapter frame can further enhance the accuracy and efficiency of fitting and configuring the head-mounted wearable computing device 100 for the user.

[0059] like Figure 7 As shown and as described above, various key points or features associated with the adapter frame 500 can be detected within a frontal image 600 captured by the camera 360 of the user-operated computing device 300. Figure 7In the illustrated arrangement, the first key point 610A is located at a first known position 710A on the adapter frame 500, corresponding to the hinge connection point between the first edge portion 503A and the first arm portion 505A of the adapter frame 500. Similarly, the second key point 610B is located at a second known position 710B on the adapter frame 500, corresponding to the hinge connection point between the second edge portion 503B and the second arm portion 505B of the adapter frame 500. The third key point 620A is located at a third known position 720A on the adapter frame 500, corresponding to the bottom outer peripheral edge portion of the first lens 507A, and the fourth key point 620 is located at a fourth known position 720B on the adapter frame 500, corresponding to the bottom outer peripheral edge portion of the second lens 507B. The fifth key point 630 is located at a fifth known position 730 on the adapter frame 500, corresponding to the center portion of the bridging portion 509 of the adapter frame 500. The detection of keypoints 610, 620, and 630, and the identification of corresponding known locations 710, 720, and 730 on the adaptation frame 500, can provide reference anchor points on the adaptation frame 500. These reference anchor points can be used to detect and / or refine the 3D pose of the adaptation frame 500 on the user's face. Detected features or attributes associated with each detected keypoint 610, 620, and 630 can be compared with known features or attributes (known location, geometry, contour, distance, etc.) to determine, for example, depth relative to camera 360. This can facilitate the determination of the 3D pose of the adaptation frame 500 on the user's face.

[0060] For example, the detected keypoints 610, 620, 630 and their corresponding known locations 710, 720, 730 on the adapter frame 500 can be matched with corresponding sets of points / locations on the 3D model of the adapter frame 500. As mentioned above, the 3D model of the adapter frame 500 can be previously stored and accessed via an application running on the computing device 300. Therefore, the known sets of correspondences between the 3D model of the adapter frame 500 and the corresponding points detected in the image 600 can be used to determine the 3D pose of the adapter frame 500.

[0061] In some implementations, detected facial / cranial landmarks (such as...) Figures 6A to 6DThe example landmark 650 shown can facilitate the detection and identification of key points or features of the adapter frame 500. For example, based on the detection below the key point 620A of the landmark 650 defining the user's cheek, the landmark 650 to the right of the key point 620A defining the user's bridge of the nose, etc., the detection and identification of the bottom outer peripheral edge portion of the first lens 507 defining the aforementioned example key point 620A can be compressed or refined and / or the accuracy improved. Other key points defined at other outer peripheral edge portions of the lens 507 (i.e., the top outer peripheral edge portion and / or the left outer peripheral edge portion and / or the right outer peripheral edge portion of the lens 507) can be compressed and / or refined based on the detection of other landmarks 650 in a similar manner.

[0062] In some implementations, the detection of key points or features associated with the fitting frame 500 worn by the user in image 600 may rely on comparison with a database populated with synthetic data, real data, and combinations of synthetic and real data. Synthetic data may include, for example, 3D renderings of glasses and the head in different combinations of pose, lighting conditions, etc., which can be combined to produce a vast number of different rendering combinations of glasses worn on the head under various conditions. Real data may include, for example, image data capturing actual people wearing glasses. Synthetic and real data can be combined to produce additional combinations of glasses worn on the head under different poses, conditions, etc. This data (synthetic data, real data, and combined data) can be used to train a neural network, which can then continuously output improved results of matching key points on the fitting frame with points in the 3D model.

[0063] In some cases, a well-developed / densely padded neural network can be used for a specific adapter frame 500, including a relatively large amount of training data representing the poses, positions, and angles of the adapter frame 500. In this case, images (e.g., those mentioned above) can be used. Figures 6A to 6C The described frontal image 600 is fed into a neural network built for the adapter 500, and the correspondence between the image 600 of the user wearing the adapter 500 and the data filling the neural network can be detected without detecting key points in the image 600 before the neural network processes the image 600.

[0064] In some implementations, after the head-mounted wearable computing device 100 has been delivered to the user, a process similar to adaptation and configuration can be applied to the calibration or recalibration of the head-mounted wearable computing device 100. For example, a new product setup process for the head-mounted wearable computing device 100, already configured for the user, may include fine-tuning of display characteristics once the head-mounted wearable computing device 100 has been delivered. For example, an application running on the user-operated computing device 300 may prompt the user to capture image data, including the user wearing the head-mounted wearable computing device 100. Detection of key points on the head-mounted wearable computing device 100 can be used to determine the three-dimensional pose of the head-mounted wearable computing device 100 on the user's face and relative to the user's eyes / eyebox. Control software controlling the operation of the display device 104 can adjust the output of content to, for example, optimize where the content is displayed. This can optimize gazeability and improve the user's viewing experience.

[0065] The aforementioned measurements or parameters can be collected to determine the display adaptation characteristics for a user wearing the selected adapter frame 500. These measurements or parameters can allow the display device of the head-mounted wearable computing device (such as...) Figure 1B and Figure 1C The adaptation of the display device 104 of the head-mounted wearable computing device 100 shown ensures that the user's pupil is aligned as centrally as possible within the eye box 140, allowing the user's eyes to focus on the content displayed through the output connector 144, which is visible to the user on the lens 107. The aforementioned measurements or parameters allow for the configuration of the display device 104 of the head-mounted wearable computing device 100 such that the content is visible to the user within a relatively large portion of their field of vision.

[0066] The systems and methods described herein can provide the collection of measurements to determine the ophthalmic fit characteristics of a user wearing a selected adapter frame 500. In some implementations, the systems and methods can provide the collection of measurements to determine both display fit characteristics and ophthalmic fit characteristics. (See above regarding...) Figures 2A to 2D As described, ophthalmic measurements can be detected within image data captured by a camera 360 of a user-operated computing device 300. These measurements (including, for example, pupillary height, interpupillary distance, monocular pupillary distance, panoramic view, vertex distance, and other such parameters or metrics) can be detected and processed within the image data for the purpose of incorporating corrective or prescription lenses into the head-mounted wearable computing device 100. In some examples, these ophthalmic measurements are used in conjunction with the aforementioned display adaptation measurements to provide an appropriate display adaptation for a particular user along with the user's corrective / prescription lens needs.

[0067] Figure 8This is a flowchart of an example method 800 for operating a computing system to configure a head-mounted wearable computing device, which includes a display for a user. A user operating the computing device (e.g., computing device 300 described above) can cause the computing device to execute an application. The application can provide the display of adaptation measurements and / or ophthalmic adaptation measurements from image data captured by the computing device (box 810). A camera (such as, for example, camera 360 of computing device 300 described above) is operated to capture image data of the user wearing an adaptation frame (box 820). This adaptation frame can be selected from one or more sample adaptation frames available to the user (such as, for example, those mentioned above). Figures 5A to 7 The described adapter frame 500). The system can detect one or more key points or features on the adapter frame (e.g., key points 610, 620, 630 and / or 670, the aforementioned facial / cranial landmarks 640, 650, etc.) (frame 830).

[0068] In some examples, the analysis of image capture data for detecting key points and / or features and / or landmarks can be performed by a computing device (e.g., the object recognition module and / or pattern recognition module of the processor 350 of the aforementioned computing device). In some examples, the detection of key points and / or features and / or landmarks can be performed by an external device, such as, for example, the object recognition module and / or pattern recognition module of a server, which is included in an external resource 302 communicating with the computing device 300, as described above.

[0069] Display adaptation measurements, including the 3D pose of the adaptation frame, can be determined based on the position of detected keypoints and / or features and / or landmarks relative to the configuration of the adaptation frame (box 840). If needed, ophthalmic adaptation measurements can be detected from image data and from the display adaptation measurements (box 850) for users who need to incorporate corrective / prescription lenses into a head-mounted wearable computing device (boxes 860, 870, and 880). In some examples, the analysis used to determine the display adaptation measurements and / or ophthalmic adaptation measurements can be performed by a computing device (e.g., the configuration identification module of the processor 350 of the aforementioned computing device 300). In some examples, the analysis used to determine the display adaptation measurements and / or ophthalmic adaptation measurements can be performed by an external computing device (such as the configuration identification module of a server included in an external resource 302 communicating with the aforementioned computing device 300). The ophthalmic adaptation measurements and display adaptation measurements can be incorporated into the user's head-mounted wearable computing device to provide a viewing experience tailored to the user's optical needs.

[0070] Figure 9Examples of computing devices 900 and 950 that can be used with the techniques described herein are shown. Computing device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, tablet computers, workstations, personal digital assistants, smart devices, appliances, electronic sensor-based devices, televisions, servers, blade servers, mainframes, and other suitable computing devices. Computing device 950 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementations of the invention described and / or claimed in this document.

[0071] The computing device 900 includes a processor 902, a memory 904, a storage device 906, a high-speed interface 908 connected to the memory 904 and a high-speed expansion port 910, and a low-speed interface 912 connected to a low-speed bus 914 and the storage device 906. The processor 902 may be a semiconductor-based processor. The memory 904 may be a semiconductor-based memory. Each of components 902, 904, 906, 908, 910, and 912 is interconnected using various buses and may be mounted on a common motherboard or otherwise suitably mounted. The processor 902 can process instructions for execution within the computing device 900, including instructions stored in the memory 904 or on the storage device 906, to display graphical information of a GUI on an external input / output device, such as a display 916 coupled to the high-speed interface 908. In other implementations, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. Furthermore, multiple computing devices 900 can be connected, with each device providing a portion of the necessary operation (e.g., as a server library, a set of blade servers, or a multiprocessor system).

[0072] Memory 904 stores information within computing device 900. In one implementation, memory 904 is one or more volatile memory cells. In another implementation, memory 904 is one or more non-volatile memory cells. Memory 904 may also be another form of computer-readable medium, such as a magnetic disk or optical disk. Typically, computer-readable media can be non-transitory computer-readable media.

[0073] Storage device 906 provides large-capacity storage for computing device 900. In one implementation, storage device 906 may be or include computer-readable media such as floppy disk devices, hard disk devices, optical disk devices, magnetic tape devices, flash memory or other similar solid-state storage devices, or device arrays, including devices in storage area networks or other configurations. The computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods and / or computer-implemented methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 904, storage device 906, or memory on processor 902.

[0074] High-speed controller 908 manages bandwidth-intensive operations of computing device 900, while low-speed controller 912 manages lower bandwidth-intensive operations. This allocation of functions is merely exemplary. In one implementation, high-speed controller 908 is coupled to memory 904, display 916 (e.g., via a graphics processor or accelerator), and to high-speed expansion port 910 which can accept various expansion cards (not shown). In this implementation, low-speed controller 912 is coupled to storage device 906 and low-speed expansion port 914. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), and may be coupled, for example, via a network adapter to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers.

[0075] As shown in the figure, computing device 900 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 920, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 924. Furthermore, it can be implemented in a computer such as a laptop computer 922. Alternatively, components from computing device 900 can be combined with other components (such as device 950) in a mobile device (not shown). Each such device may include one or more computing devices 900, 950, and the entire system may consist of multiple computing devices 900, 950 communicating with each other.

[0076] Computing device 950 includes a processor 952, a memory 964, input / output devices such as a display 954, a communication interface 966 and a transceiver 968, and other components. Device 950 may also be provided with storage devices, such as microdrives or other devices, to provide additional storage. Each of components 950, 952, 964, 954, 966, and 968 is interconnected using various buses, and several of the components may be mounted on a common motherboard or otherwise suitably mounted.

[0077] Processor 952 can execute instructions within computing device 950, including instructions stored in memory 964. The processor can be implemented as a chipset comprising individual and multiple analog and digital processor chips. The processor can provide, for example, coordination with other components of device 950, such as control of the user interface, applications running on device 950, and wireless communication of device 950.

[0078] Processor 952 can communicate with the user via control interface 958 and display interface 956 connected to display 954. Display 954 can be, for example, a TFT LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display or other suitable display technology. Display interface 956 may include appropriate circuitry for driving display 954 to present graphics and other information to the user. Control interface 958 can receive commands from the user and translate them for submission to processor 952. Furthermore, an external interface 962 can be provided to communicate with processor 952 to enable near-field communication between device 950 and other devices. External interface 962 may provide wired communication in some implementations, or wireless communication in others, and multiple interfaces may be used.

[0079] Memory 964 stores information within computing device 950. Memory 964 can be implemented as one or more computer-readable media, one or more volatile memory cells, or one or more non-volatile memory cells. Extended memory 984 may also be provided and connected to device 950 via an extended interface 982, which may include, for example, a SIMM (Single In-line Memory Module) card interface. Such extended memory 984 can provide additional storage space for device 950, or it may also store applications or other information for device 950. Specifically, extended memory 984 may include instructions for performing or supplementing the above processes, and may also include security information. Therefore, for example, extended memory 984 may be provided as a security module of device 950 and can be programmed with instructions that allow secure use of device 950. Furthermore, secure applications and additional information, such as placing identification information on the SIMM card in an unbreakable manner, may be provided via a SIMM card.

[0080] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 964, extended memory 984, or memory on processor 952, which may be received, for example, via transceiver 968 or external interface 962.

[0081] Device 950 can conduct wireless communication via communication interface 966, which may include digital signal processing circuitry if necessary. Communication interface 966 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS message sending and receiving, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. This communication can occur, for example, via radio frequency transceiver 968. Furthermore, short-range communication can occur, such as using Bluetooth, Bluetooth Low Energy, Wi-Fi, or other transceivers (not shown). Additionally, GPS (Global Positioning System) receiver module 980 can provide device 950 with additional navigation and location-related wireless data, which can be appropriately used by applications running on device 950.

[0082] Device 950 can also use audio codec 960 for audible communication, which can receive voice information from a user and convert it into usable digital information. Audio codec 960 can also generate audible sound for the user, such as through a speaker (e.g., in the handheld device of device 950). Such sound can include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications operating on device 950.

[0083] As shown in the figure, the computing device 950 can be implemented in many different forms. For example, it can be implemented as a cellular phone 983. It can also be implemented as part of a smartphone 981, a personal digital assistant, or other similar mobile devices.

[0084] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0085] These computer programs (also referred to as modules, programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0086] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor or LED (light-emitting diode)) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0087] The systems and technologies described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with the implementation of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

[0088] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship is established through computer programs running on the respective computers and having a client-server relationship with each other.

[0089] In some embodiments, Figure 9 The computing device depicted may include sensors that interface with AR / VR headsets (AR glasses / eyeglasses / VR headsets / AR headsets / HMD device 990). For example, sensors may be included in computing device 950 or... Figure 9 One or more sensors on other computing devices depicted may provide input to the AR / VR headset 990, or generally to the AR / VR space. Sensors may include, but are not limited to, touchscreens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 950 may use sensors to determine its absolute position and / or detected rotation in the AR / VR space, which can then be used as input to the AR / VR space. For example, the computing device 950 may be incorporated into the AR / VR space as a virtual object, such as a controller, laser pointer, keyboard, weapon, etc. When a computing device / virtual object is incorporated into the AR / VR space, the user's positioning of the computing device / virtual object may allow the user to position the computing device to view the virtual object in some way within the AR / VR space.

[0090] In some embodiments, one or more input devices included on or connected to the computing device 950 can be used as input to the AR / VR space. Input devices may include, but are not limited to, touchscreens, keyboards, one or more buttons, touchpads, pointing devices, mice, trackballs, joysticks, cameras, microphones, headphones or earphones with input capabilities, game controllers, or other connectable input devices. When the computing device is incorporated into the AR / VR space, a user interacting with the input devices included on the computing device 950 can cause specific actions to occur within the AR / VR space.

[0091] In some embodiments, one or more output devices included on the computing device 950 may provide output and / or feedback to a user of the AR / VR headset 990 in an AR / VR space. The output and feedback may be visual, tactile, or audio. The output and / or feedback may include, but is not limited to, presenting an AR / VR space or virtual environment, vibration, turning one or more lights or flashes on and off, playing alarms, ringing, playing songs, and playing audio files. Output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light-emitting diodes (LEDs), flashlights, and speakers.

[0092] In some embodiments, a computing device 950 may be placed within an AR / VR headset 990 to create an AR / VR system. The AR / VR headset 990 may include one or more positioning elements that allow the computing device 950 (such as a smartphone 981) to be placed in the appropriate location within the AR / VR headset 990. In such an embodiment, the display of the smartphone 981 may render stereoscopic images representing an AR / VR space or virtual environment.

[0093] In some embodiments, the computing device 950 may appear as another object in a computer-generated 3D environment. User interactions with the computing device 950 (e.g., rotating, shaking, touching a touchscreen, swiping a finger on the touchscreen) can be interpreted as interactions with an object in the AR / VR space. As an example only, the computing device could be a laser pointer. In such an example, the computing device 950 appears as a virtual laser pointer in the computer-generated 3D environment. When the user manipulates the computing device 950, the user in the AR / VR space perceives the laser pointer as moving. The user receives feedback from interactions with the computing device 950 in the AR / VR environment on the computing device 950 or the AR / VR headset 990.

[0094] In some embodiments, computing device 950 may include a touchscreen. For example, a user may interact with the touchscreen in a manner that simulates content occurring on the touchscreen and content occurring in an AR / VR space. For instance, a user may use pinch gestures to zoom in on content displayed on the touchscreen. Such pinch gestures on the touchscreen can cause information provided in the AR / VR space to be zoomed in or out. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment. In the AR / VR space, pages of the book may be displayed, and the user's finger swiping on the touchscreen may be interpreted as turning / flipping the pages of the virtual book. When each page is turned / flipped, in addition to seeing the page content change, audio feedback, such as the sound of pages turning in a book, may be provided to the user.

[0095] In some embodiments, one or more input devices other than a computing device (e.g., a mouse, a keyboard) can be rendered in a computer-generated 3D environment. The rendered input devices (e.g., a rendered mouse, a rendered keyboard) can be used to control objects in AR / VR space.

[0096] Several embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the invention.

[0097] Furthermore, the logical flow depicted in the accompanying drawings does not require the specific order or sequence shown to achieve the desired result. Additionally, other steps may be provided, or steps may be eliminated from the described flow, and other components may be added to or removed from the described system. Therefore, other embodiments are within the scope of the appended claims.

[0098] In addition to the above description, users are provided with controls that allow them to choose whether and when user information (e.g., information about a user's social networks, social actions or activities, occupation, user preferences, or current location) can be collected by the system, program, device, network, or feature described herein, and whether the server sends content or communications to the user. Furthermore, some data may be processed in one or more ways before it is stored or used, resulting in the removal of user information. For example, a user's identity may be processed so that user information cannot be determined, or the user's geographic location may be generalized (e.g., to the city, zip code, or state level) if location information is available, making it impossible to determine the user's specific location. Therefore, users can control what information about themselves is collected, how that information is used, and what information is provided to them.

[0099] Computer systems (e.g., computing devices) can be configured to communicate wirelessly with a network server via a communication link established with the network server, using any known wireless communication technologies and protocols suitable for communication over a network (including radio frequency (RF), microwave frequency (MWF), and / or infrared frequency (IRF) wireless communication technologies and protocols).

[0100] According to aspects of this disclosure, implementations of the various techniques described herein can be implemented as digital electronic circuits or computer hardware, firmware, software, or combinations thereof. Implementations can be implemented as computer program products (e.g., computer programs tangibly embodied in an information carrier, machine-readable storage device, computer-readable medium, or tangible computer-readable medium) for processing by or controlling the operation of a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, the tangible computer-readable storage medium may be configured to store instructions that, upon execution, cause the processor to perform a process. Computer programs (such as those described above) can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for a computing environment. Computer programs can be deployed for processing on one or more computers at a single site, or distributed across multiple sites and interconnected via a communication network.

[0101] The specific structural and functional details disclosed herein are merely for the purpose of describing representative examples of the embodiments. However, these examples may be embodied in many alternative forms and should not be construed as being limited to the embodiments set forth herein.

[0102] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the embodiments. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that, when used in this specification, the terms “comprising,” “including,” “containing,” and / or “having” specify the presence of the said features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0103] It should be understood that when an element is referred to as “connected,” “linked,” or “in response” to or “on another element,” it may be directly connected, linked, or in response to or on another element, or there may be intermediate elements present. Conversely, when an element is referred to as “directly connected,” “directly linked,” or “directly in response” to or “directly on another element,” there are no intermediate elements present. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0104] For ease of description, spatial relative terms such as “below,” “under,” “lower,” “above,” “upper,” etc., may be used herein to describe an element or feature relative to another element or feature shown in the figures. It should be understood that, in addition to the orientation depicted in the figures, spatial relative terms are intended to cover different orientations of the device in use or operation. For example, if the device in the figures is flipped, an element described as “below” or “under” other elements or features would be oriented “above” other elements or features. Thus, the term “below” can encompass both above and below orientations. The device may be oriented in other ways (rotated 70 degrees or otherwise), and the spatial relative descriptors used herein may be interpreted accordingly.

[0105] This document describes exemplary embodiments of the concept with reference to cross-sectional illustrations, which are schematic diagrams of idealized embodiments (and intermediate structures) of the exemplary embodiments. Therefore, variations in the illustrated shapes are expected due to, for example, manufacturing techniques and / or tolerances. Consequently, the described exemplary embodiments of the concept should not be construed as limited to the specific shapes of the areas shown herein, but rather include, for example, shape deviations caused by manufacturing processes. Therefore, the areas shown in the figures are schematic in nature, and their shapes are not intended to represent the actual shapes of areas of the device, nor are they intended to limit the scope of the exemplary embodiments.

[0106] It should be understood that although the terms "first," "second," etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Therefore, without departing from the teachings of this embodiment, a "first" element may be referred to as a "second" element.

[0107] Unless otherwise defined, the terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which these concepts belong. It will be further understood that terms such as those defined in common dictionaries should be interpreted as having the same meaning as they have in the relevant field and / or the context of this specification, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.

[0108] While certain features of the described implementations have been shown as described herein, many modifications, substitutions, alterations, and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover such modifications and alterations that fall within the scope of the implementations. It should be understood that they are presented by way of example only and not limitation, and various changes in form and detail are possible. Any part of the apparatus and / or method described herein can be combined in any combination, except for mutually exclusive combinations. The implementations described herein may include various combinations and / or sub-combinations of the functions, components, and / or features of the different implementations described.

Claims

1. A method comprising: An application running on a computing device detects at least one feature on a frame worn by a user from an image captured by the computing device; Access configuration information associated with the frame detected in the image; Detect multiple facial landmarks in the captured image; Based on the detection of the at least one feature and the plurality of facial landmarks and based on the configuration information associated with the frame, the three-dimensional pose of the frame captured in the image relative to the user's face is determined; as well as The configuration of the display device of the wearable computing device is determined based on the three-dimensional pose of the frame relative to the user's face captured in the image, wherein determining the configuration of the display device of the wearable computing device includes determining the configuration of the display device such that the content output by the display device is within the user's field of view.

2. The method according to claim 1, wherein, Detecting the at least one feature includes detecting at least one of the following: The bridging portion of the frame; The hinge point between the edge portion and the arm portion of the frame; The outer peripheral edge portion of the lens of the frame; or The saddle portion of the arm portion of the frame.

3. The method according to claim 1, wherein, Determining the three-dimensional pose of the frame includes: Access the 3D model of the frame captured in the image; and Performing a comparison includes comparing the known location and known orientation of the at least one feature detected in the image with the corresponding location and corresponding orientation of the at least one feature in the three-dimensional model of the frame.

4. The method according to claim 3, wherein, Detecting at least one feature includes detecting multiple features on the frame in the image captured by the computing device; as well as Performing the comparison includes, for each of the plurality of features, comparing the known location and known orientation of the at least one feature detected in the image with the corresponding location and corresponding orientation of the at least one feature in the three-dimensional model of the frame.

5. The method according to claim 4, wherein, Determining the three-dimensional pose of the frame includes mapping the two-dimensional position of each feature detected in the image to the corresponding three-dimensional position of the feature in the three-dimensional model of the frame.

6. The method according to claim 1, wherein, Detecting the multiple facial landmarks includes: Detect pupil height in the image of the frame worn by the user; Detecting at least one of interpupillary distance or monocular pupillary distance in the image of the frame worn by the user; and The panoramic view of the frame worn by the user is determined based on the determined three-dimensional pose of the frame and the detected multiple facial landmarks.

7. The method according to claim 6, wherein, Determining the configuration of the display device of the wearable computing device based on the three-dimensional pose of the frame captured in the image includes adapting the configuration of the display device of the wearable computing device to accommodate ophthalmic fitting measurements.

8. The method according to claim 7, wherein, Determining the configuration of the display device of the wearable computing device includes: The user's field of view is determined based on the ophthalmic adaptation measurements and the three-dimensional pose of the framework; and Configure the output connector of the display device such that the content output by the display device is displayed within the determined user field of view.

9. The method according to claim 1, wherein, Accessing the configuration information associated with the framework includes accessing the framework's three-dimensional model.

10. The method according to claim 9, wherein, The three-dimensional model of the frame includes at least one of known linear measurements, known angular measurements, or known contours associated with the frame.

11. The method according to claim 9 or 10, wherein, Determining the three-dimensional pose of the frame includes: Detect at least one corresponding feature in the three-dimensional model of the framework; and The three-dimensional pose of the frame is detected by comparing at least one feature detected in the image with at least one corresponding feature detected in the three-dimensional model of the frame.

12. A non-transitory computer-readable medium storing executable instructions, said executable instructions being configured, when executed by at least one processor, to cause said at least one processor to: Capture images of the frame worn by the user; Detect at least one feature on the frame from the captured image; Access configuration information associated with the framework; Detect multiple facial landmarks in the captured image; Based on the detection of the at least one feature and the plurality of facial landmarks and based on the configuration information associated with the frame, the three-dimensional pose of the frame captured in the image relative to the user's face is determined; as well as The configuration of the display device of the wearable computing device is determined based on the three-dimensional pose of the frame relative to the user's face captured in the image, wherein determining the configuration of the display device of the wearable computing device includes determining the configuration of the display device such that the content output by the display device is within the user's field of view.

13. The non-transitory computer-readable medium according to claim 12, wherein, The instructions cause the at least one processor to detect at least one feature on the frame in the captured image, including at least one of the following: Detect a first feature, the first feature including the bridging portion of the frame; Detect a second feature, the second feature including a hinge portion between the edge portion and the arm portion of the frame; Detect a third feature, said third feature including the outer peripheral edge portion of the lens of the frame; or A fourth feature is detected, which includes the saddle portion of the arm portion of the frame.

14. The non-transitory computer-readable medium according to claim 12, wherein, The instructions cause the at least one processor to determine the three-dimensional pose of the frame, including: Access the 3D model of the frame captured in the image; and The known position and known orientation of the at least one feature detected in the image are compared with the corresponding position and corresponding orientation of the at least one feature in the three-dimensional model of the frame.

15. The non-transitory computer-readable medium according to claim 14, wherein, The instructions cause the at least one processor to detect a plurality of features and perform the comparison, including for each of the plurality of features: The known location and known orientation of the feature detected in the image are compared with the corresponding location and corresponding orientation of the feature in the three-dimensional model of the frame.

16. The non-transitory computer-readable medium according to claim 15, wherein, The instructions cause the at least one processor to determine the three-dimensional pose, including: The two-dimensional position of each feature detected in the image is mapped to the corresponding three-dimensional position of the feature in the three-dimensional model of the frame.

17. The non-transitory computer-readable medium according to claim 12, wherein, Detecting the multiple facial landmarks includes: Detect pupil height in the image of the frame worn by the user; Detect at least one of interpupillary distance or monocular pupillary distance in the image of the frame worn by the user; The panoramic view of the frame worn by the user is determined based on the determined three-dimensional pose of the frame and the multiple facial landmarks examined.

18. The non-transitory computer-readable medium according to claim 12, wherein, The instructions cause the at least one processor to adapt the configuration of the display device of the wearable computing device to ophthalmic fitting measurements, including: The user's field of view is determined based on the ophthalmic adaptation measurements and the three-dimensional pose of the framework; and Configure the output connector of the display device so that the content output by the display device is displayed within the user's field of view.

Images