Personalized calibration of user interfaces
By detecting the user's physiological characteristics and adjusting the position and scaling ratio of user interface elements, the interface inadequacy problem caused by differences in arm length and hand size between different users is solved, and the interactive experience of wearable devices is improved.
Patent Information
- Application Number
- CN202380030708.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-04-01
- Filing Date
- 2023-03-29
- Publication Date
- 2024-11-12
AI Technical Summary
The existing wearable electronic device user interfaces are not adapted to the position and size of the interactive elements due to differences in arm length and hand size between different users, which may cause the user interface elements to be too far or too close to the user, affecting the interactive experience.
By detecting the user's physiological characteristics, such as arm length and hand size, the calibration process is initiated, calibration parameters are collected and stored, and the position and scaling of user interface elements are adjusted to meet personalized user needs.
Improves the accessibility and interactive comfort of user interface elements, ensures that users can interact with electronic devices in the optimal way, and enhances the interactive experience of augmented reality (AR) elements.
Smart Images

Figure CN118946872A_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. application serial number 17 / 711,447, filed on April 1, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field
[0003] The present disclosure relates to user interfaces for electronic devices, including augmented reality (AR) enabled wearable electronic devices, such as smart glasses. More particularly, but not by way of limitation, the present disclosure describes techniques for detecting physiological characteristics of a user (such as arm length and hand size), adjusting parameters of user interface (UI) controls based on the detected physiological characteristics, and then rendering a customized UI to the electronic device. Background Art
[0004] A user of a wearable electronic device can select display features by interacting with the wearable electronic device. A wearable electronic device (such as an electronic eyewear device) may have a user interface that accepts gesture input. When interacting with such an electronic eyewear device, the user's arms and hands may move in front of the AR camera of the electronic eyewear device to make gestures, which are recognized by the electronic eyewear device as different input commands to control the operation of the electronic eyewear device. However, not all users have appendages (such as arms and hands) of the same size. Some users may have hands of different sizes, different arm lengths, etc. compared to other users. For example, an adult with long arms and large hands may interact with the user interface of an electronic device differently than a child with short arms and small hands. A user interface (UI) control may potentially be too far or too close to a user if the UI control is not rendered at the correct size or distance for the corresponding user. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] The features of the various embodiments disclosed will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements.
[0006] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be construed as limiting. The drawings include the following figures:
[0007] Figure 1A is a side view of an example eyewear device including an optical assembly having an image display;
[0008] Figure 1Byes Figure 1A A top cross-sectional view of the optical components and electronics in a portion of the eyewear device is shown;
[0009] Figure 2A is a rear view of an example eyewear device;
[0010] Figure 2B is a rear view of an example eyewear device;
[0011] Figure 2C is a rear view of an example eye-mounted device depicting an image display;
[0012] Figure 2D is a rear view of an example eye-mounted device depicting an image display;
[0013] Figure 3 is a block diagram illustrating an example of using the example eye-mounted device shown in any of the preceding figures to capture visible light;
[0014] Figure 4 is a system block diagram of an example eye-wearing device;
[0015] Figure 5 is configured for use with Figure 4 A block diagram of electronic components of a mobile device for use with the system;
[0016] Figure 6 yes Figure 4 A block diagram of a sample backend server system of the type shown;
[0017] Fig. 7A , Figure 7B and Figure 7C are graphs depicting hand parameters of finger length, inter-knuckle distance, and palm size, respectively;
[0018] Fig. 8A is a diagram depicting a summonable element at a distance from a user's hand that exceeds a maximum reachable distance of the user;
[0019] Figure 8B is depicted being displaced from an out-of-reach position to a position adjacent to the position of the user's hand when summoned to the user Fig. 8A A diagram of the summonable element;
[0020] Fig. 9A is a diagram depicting summonable elements at locations on the world grid that are obscured by world items;
[0021] Fig. 9B is a graph that depicts a position that has been moved from a position obscured by world objects to the nearest visible position that is not obscured by world objects. Fig. 9A A diagram of the summonable element;
[0022] Fig. 10A is a graph depicting the size of the user's hand relative to the hand tracking UI elements;
[0023] Fig. 10B is a diagram depicting scaling of the positioning of a hand tracking UI element to the size of a user's hand;
[0024] Fig.11A is a diagram that depicts interactive elements (buttons) that are too close for comfortable user selection;
[0025] Fig. 11B is a graph where the spacing between interactive elements has been adjusted to a certain scale based on the knuckle spacing parameter Fig.11A Diagram of interactive elements;
[0026] Fig.12 is a diagram depicting application of capturing gesture calibration parameters in a sample configuration; and
[0027] Fig.13 is a diagram showing the flow of the calibration process in the sample configuration. DETAILED DESCRIPTION
[0028] A calibration process is described for customizing a user interface (UI) of an eye-worn device to improve interaction with augmented reality (AR) elements. Physiological characteristics of the user, such as arm length and hand size, are determined for images captured by the eye-worn device and used to adjust parameters of UI controls before rendering the UI.
[0029] Examples in the present disclosure are directed to techniques for calibrating a user interface (UI) for an eye-wearing device. When a user's calibration gesture is detected, a calibration timer for the calibration process is started. As long as the calibration gesture remains active and the calibration timer has not expired, calibration parameters (e.g., the user's palm size, maximum reachable distance, knuckle spacing, and finger length) are collected. When the calibration timer expires, the collected calibration parameters are stored in a parameter memory and are used to adjust the user interface to reflect the collected calibration parameters. The calibration gesture is detected and the calibration parameters are collected by placing all user interface elements locked to the user's position in a user interface view of the eye-wearing device and receiving input indicating a predetermined calibration gesture made by the user. The system then determines the maximum distance that the user can reach and the size of the user's hand based on the distance between the hand tracking landmarks of the user's hand. The collected parameters are used to adjust the user interface parameters to adjust the position and scaling of the user interface elements to customize the user interface to individual users.
[0030] In one example, the system modifies an interactive element of the user interface based on the measured maximum reachable distance to adjust the position of the interactive element to be within the reachable distance of the user. The system can also adjust the scaling of hand tracking user interface elements or the spacing of groups of user interface elements to scale to the size of the user's hand.
[0031] In another example, the system determines whether a summonable user interface element is located at a distance from the user's hand that exceeds the measured maximum reachable distance, or is located at a location in the display that is obscured from the user's view by a physical object. When the summonable user interface element is located at a location that exceeds the measured maximum reachable distance from the user's hand or is obscured from the user's view in the display by a physical object, the summonable user interface element is shifted from the location that exceeds the measured maximum reachable distance or is obscured from the user's view in the display by the physical object to a location that is within the reachable distance of the user's hand and is not obscured from the user's view in the display by the physical object. Upon completion of the interaction with the summonable user interface element, the summonable user interface element returns to the location that exceeds the measured maximum reachable distance, or returns to the location that is obscured from the user's view in the display by the physical object.
[0032] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. Many details and examples are included for the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects, and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.
[0033] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is conducted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.
[0034] Additional objects, advantages and novel features of each example will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized and attained by means of the methods, tools and combinations particularly pointed out in the appended claims.
[0035] For purposes of illustration and discussion, the orientation of the eye-mounted device, associated components, and any complete device incorporating an eye scanner and camera (such as those shown in any of the accompanying drawings) is given by way of example only. In operation for a particular variable optical processing application, the eye-mounted device may be oriented in any other direction (e.g., upward, downward, sideways, or any other orientation) suitable for the particular application of the eye-mounted device. Additionally, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used by way of example only and are not limiting with respect to the direction or orientation of any optical device or component of an optical device constructed as otherwise described herein.
[0036] Reference will now be made in detail to the examples illustrated in the accompanying drawings and discussed below. Fig.13 A sample eye-mounted device and associated system for providing social connections between users of the eye-mounted devices are described.
[0037] The system described in this article includes three types of hardware components: eye-wearing devices, mobile devices, and servers. The eye-wearing devices will be related to Figures 1 to Figure 3 Describe, the mobile device will be about Figure 5 Describes the server and Figure 6 The corresponding system will be about Figure 4 The operation of the software components (including the eye-wearing device and the application software on the mobile device) and examples of system operation will be described with respect to FIGS. Fig.13 Such software components include: system software for calibrating the user interface of the eye-wearing device. However, it will be understood that the mobile device, the server, or both can be removed from the system as long as the eye-wearing device is configured to include sufficient processing and storage capabilities to perform the described functions of the mobile device, the server, or both.
[0038] In a sample configuration, an eye-mounted device with augmented reality (AR) capabilities is used in the system described herein. Eye-mounted devices are expected to be used in the system described herein because such devices are scalable and customizable to achieve a personalized experience, achieve the effect of being applied anytime, anywhere, and ensure user privacy by allowing the user to see only the information transmitted. In a sample configuration, the eye-mounted device can be used without any dedicated hardware, such as the SPECTACLES available from Snap, Inc. of Santa Monica, California. TM .
[0039] Figure 1A The invention is a diagram including an image display 180C ( Figure 2A ) is an illustration of a side view of an example hardware configuration of an eye-mounted device 100 having an optical assembly 180A. The eye-mounted device 100 includes a plurality of visible light cameras 114A and 114B ( Figure 3 ), wherein the first visible light camera 114A is located on the right temple 110A, and the second visible light camera 114B is located on the left temple 110B ( Figure 2A ). In the illustrated example, the optical component 180A may be located on the right side of the eye-mounted device 100. The optical component 180A may be located on the left side of the eye-mounted device 100 or at other locations.
[0040] Visible light cameras 114A and 114B may include image sensors that are sensitive to wavelengths in the visible light range. Each of visible light cameras 114A and 114B has a different forward-facing angle of coverage, for example, visible light camera 114A has a depicted angle of coverage 111A ( Figure 3 ). The coverage angle is the angular range over which the respective image sensors of visible light cameras 114A and 114B detect incident light and generate image data. Examples of such visible light cameras 114A and 114B include high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and video graphic array (VGA) cameras, such as 640p (e.g., 640×480 pixels totaling 300,000 pixels), 720p, 1080p, 4K, or 8K. Image sensor data from visible light cameras 114A and 114B may be captured along with geolocation data, digitized by an image processor, and stored in memory.
[0041] To provide stereoscopic vision, visible light cameras 114A and 114B may be coupled to an image processor ( Figure 4Image processor 412 may include circuitry to receive signals from visible light cameras 114A and 114B and process those signals from visible light cameras 114A and 114B into a format suitable for storage in memory ( Figure 4 The timestamp may be added by image processor 412 or other processor controlling the operation of visible light cameras 114A and 114B. Visible light cameras 114A and 114B allow the stereo camera to simulate human binocular vision. The stereo camera also provides a method for simulating two captured images (i.e., two images captured by visible light cameras 114A and 114B) with the same timestamp. Figure 3 The image pairs 358A and 358B in the figure are used to reproduce the three-dimensional scene ( Figure 3 4. The present invention also provides the ability to generate a three-dimensional image of a scene 315 in the image. Such three-dimensional images allow for an immersive virtual experience that feels realistic, for example, for virtual reality or video games. For stereoscopic vision, a pair of images 358A and 358B can be generated at a given instant - one image for each of visible light cameras 114A and 114B. Depth perception is provided by optical components 180A and 180B when the generated pair of images 358A and 358B from the forward-facing fields of view (FOV) 111A and 111B of visible light cameras 114A and 114B are stitched together (e.g., by image processor 412).
[0042] In an example, the eyewear device 100 includes a frame 105, a right rim 107A, a right temple 110A extending from a right lateral side 170A of the frame 105, and a see-through image display 180C (including an optical assembly 180A for presenting a GUI or other image to a user). FIG. 2A to FIG. 2B ). The eyewear device 100 includes a first visible light camera 114A connected to the frame 105 or the right temple 110A to capture a first image of a scene. The eyewear device 100 also includes a second visible light camera 114B connected to the frame 105 or the left temple 110B to capture a second image of the scene that at least partially overlaps the first image (e.g., simultaneously with the first visible light camera 114A). Although Figure 1A and Figure 1B , but the processor 432 ( Figure 4 ) is coupled to the eye-mounted device 100 and is connected to the visible light cameras 114A and 114B and the memory 434 (accessible to the processor 432) Figure 4 ), and the programming in the memory 434 can be provided in the eye-wearing device 100 itself.
[0043] although Figure 1A , but the eye-mounted device 100 may also include a head motion tracker ( Figure 1B 109) or an eye tracker ( Figure 2A Element 113 in, or Figure 2B and Figure 2C 213 in the eyewear). The eyewear device 100 may also include perspective image displays 180C and 180D of the optical components 180A and 180B, respectively, for presenting a display image sequence. The eyewear device 100 may also include an image display driver ( Figure 4 The eye-mounted device 100 may further include a memory 434 and a processor 432 ( Figure 4 ), which processor is capable of accessing the image display driver 442 and the memory 434, and the programming in the memory 434. Execution of the programming by the processor 432 configures the eye-mounted device 100 to perform functions, including the function of presenting an initial display image of a sequence of display images via the see-through image displays 180C and 180D, the initial display image having an initial field of view corresponding to the initial head orientation or the initial eye gaze direction as determined by the eye tracker 113 or 213.
[0044] Programming executed by the processor 432 may also configure the eye-mounted device 100 to detect movement of the user of the eye-mounted device 100 by: (i) detecting movement of the user of the eye-mounted device 100 via a head motion tracker ( Figure 1B 109 in the figure) to track the head movement of the user's head, or (ii) via an eye tracker ( Figure 2A Element 113 in, or Figure 2B and Figure 2C Element 213 in ) tracks eye movement of the user's eye of the eye-mounted device 100. Programming executed by processor 432 may also configure the eye-mounted device 100 to determine a field of view adjustment to an initial field of view of an initial display image based on the detected movement of the user. The field of view adjustment may include a continuous field of view corresponding to continuous head directions or continuous eye directions. Programming executed by processor 432 may also configure the eye-mounted device 100 to generate continuous display images in a display image sequence based on the field of view adjustment. Programming executed by processor 432 may also configure the eye-mounted device 100 to present continuous display images via the perspective image displays 180C and 180D of optical components 180A and 180B.
[0045] Figure 1B It is a depiction Figure 1A1 is an illustration of a top cross-sectional view of the optical components and electronics in a portion of the eye-mounted device 100 depicted, depicting the first visible light camera 114A, the head motion tracker 109, and the circuit board 140A. The second visible light camera 114B is constructed and positioned substantially similarly to the first visible light camera 114A, except that the connection and coupling is on the other lateral side 170B ( Figure 2A ). As shown, the eye-mounted device 100 includes a first visible light camera 114A and a circuit board, which may be a flexible printed circuit board (PCB) 140A. A first hinge 126A connects the right temple 110A to an articulated arm 125A of the eye-mounted device 100. In some examples, components of the first visible light camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the right temple 110A or the first hinge 126A.
[0046] As shown, the eye-wearing device 100 may include a head motion tracker 109, which includes, for example, an inertial measurement unit (IMU). An inertial measurement unit is an electronic device that uses a combination of accelerometers and gyroscopes (sometimes also magnetometers) to measure and report the body's specific force, angular rate, and sometimes the magnetic field around the body. The inertial measurement unit works by detecting linear acceleration using one or more accelerometers and detecting rotation rate using one or more gyroscopes. A typical configuration of an inertial measurement unit includes an accelerometer, gyroscope, and magnetometer for each axis (for each of the three axes): a horizontal axis (X) for left and right movement, a vertical axis (Y) for top and bottom movement, and a depth or distance axis (Z) for up and down movement. The accelerometer detects the gravity vector. The magnetometer defines the rotation in the magnetic field (e.g., facing south, north, etc.), similar to a compass that generates a heading reference. The three accelerometers detect acceleration along the horizontal, vertical, and depth axes defined above, which may be defined relative to the ground, the eye-mounted device 100 , or a user wearing the eye-mounted device 100 .
[0047] The eye-mounted device 100 can detect movement of the user of the eye-mounted device 100 by tracking head movement of the user's head via the head movement tracker 109. The head movement includes a change in the head direction from the initial head direction on the horizontal axis, the vertical axis, or a combination thereof during the presentation of the initial display image on the image display. In one example, tracking the head movement of the user's head via the head movement tracker 109 includes measuring the initial head direction on the horizontal axis (e.g., X-axis), the vertical axis (e.g., Y-axis), or a combination thereof (e.g., lateral or diagonal movement) via an inertial measurement unit. Tracking the head movement of the user's head via the head movement tracker 109 also includes measuring continuous head directions on the horizontal axis, the vertical axis, or a combination thereof via the inertial measurement unit during the presentation of the initial display image.
[0048] Tracking head movement of the user's head via the head motion tracker 109 may include determining a change in head orientation based on both the initial head orientation and the continuous head orientation. Detecting movement of the user of the eyewear device 100 may also include determining that a change in head orientation exceeds a deviation angle threshold on a horizontal axis, a vertical axis, or a combination thereof in response to tracking head movement of the user's head via the head motion tracker 109. In a sample configuration, the deviation angle threshold is between approximately 3° and 10°. As used herein, the term "approximately" when referring to an angle means a difference of ±10% from a specified amount.
[0049] Changes along the horizontal axis slide three-dimensional objects (such as characters, Bitmoji, app icons, etc.) into or out of the field of view by, for example, hiding, unhiding, or otherwise adjusting the visibility of the three-dimensional objects. Changes along the vertical axis, for example, in one example, when the user looks up, weather information, time of day, date, calendar appointments, etc. are displayed. In another example, when the user looks down on the vertical axis, the eye-wearing device 100 can be powered off.
[0050] like Figure 1B As shown, the right temple 110A includes a temple body 211, which is configured to accommodate a temple cap. Figure 1B The temple cap is omitted in the cross-section of FIG. Disposed within the right temple 110A are various interconnected circuit boards, such as a PCB or flexible PCB 140A, which include circuit boards for a first visible light camera 114A, (one or more) microphones 130, (one or more) speakers 132, low power wireless circuitry (e.g., for communicating via Wireless short-range network communications) and high-speed wireless circuit systems (e.g., for A controller circuit for wireless LAN communication).
[0051] The first visible light camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a visible light camera cover lens that is aimed through (one or more) openings formed in the right temple 110A. In some examples, the frame 105 connected to the right temple 110A includes (one or more) openings for the visible light camera cover lens. The frame 105 may include a front side that is configured to face outward away from the user's eyes. The opening for the visible light camera cover lens may be formed on the front side and through the front side. In this example, the first visible light camera 114A has an outwardly facing coverage angle 111A under the line of sight or viewing angle of the right eye of the user of the eye-wearing device 100. The visible light camera cover lens may also be adhered to the outwardly facing surface of the right temple 110A, in which the opening is formed with an outwardly facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intermediary component.
[0052] First visible light camera 114A may be connected to first perspective image display 180C of first optical assembly 180A to generate a first background scene of a first continuously displayed image. Second visible light camera 114B may be connected to second perspective image display 180D of second optical assembly 180B to generate a second background scene of a second continuously displayed image. The first background scene and the second background scene may partially overlap to present a three-dimensional observable area of the continuously displayed image.
[0053] Flexible PCB 140A may be disposed within right temple 110A and coupled to one or more other components housed in right temple 110A. Although shown as being formed on circuit board 140A of right temple 110A, first visible light camera 114A may be formed on another circuit board (not shown) of one of left temple 110B, articulated arm 125A, articulated arm 125B, or frame 105.
[0054] Figure 2A is a diagram depicting a rear view of an example hardware configuration of the eyewear device 100. Figure 2A As shown, the eyewear device 100 is configured to be worn by a user. Figure 2A The example of the eyewear device 100 is glasses. The eyewear device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or a helmet.
[0055] In the glasses example, the eyewear device 100 includes a frame 105 including a right rim 107A connected to a left rim 107B via a nose bridge 106, which is configured to accommodate the user's nose. The right rim 107A and the left rim 107B include respective rings 175A and 175B, which hold respective optical elements 180A and 180B, such as lenses and see-through displays 180C and 180D. As used herein, the term lens is meant to encompass a transparent or translucent piece of glass or plastic having curved and flat surfaces that cause light to converge / diverge or cause little or no convergence / divergence.
[0056] Although shown as having two optical elements 180A and 180B, the eye-mounted device 100 may include other arrangements, such as a single optical element, depending on the application or intended user of the eye-mounted device 100. As further shown, the eye-mounted device 100 includes a right temple 110A adjacent to the right lateral side 170A of the frame 105 and a left temple 110B adjacent to the left lateral side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective sides 170A and 170B (as shown), or may be implemented as separate components attached to the frame 105 on the respective sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.
[0057] exist Figure 2A In an example, an eye scanner 113 including an infrared transmitter 115 and an infrared camera 120 is provided. The visible light camera typically includes a blue light filter to block infrared light detection. In an example, the infrared camera 120 is a visible light camera with the blue light filter removed, such as a low-resolution video graphics array (VGA) camera (e.g., 640×480 pixels, a total of 300,000 pixels). The infrared transmitter 115 and the infrared camera 120 can be co-located on the frame 105. For example, both are shown as being connected to the upper portion of the left mirror edge 107B. The frame 105 or one or more of the temples 110A and 110B may include a circuit board (not shown) that includes the infrared transmitter 115 and the infrared camera 120. For example, the infrared transmitter 115 and the infrared camera 120 can be connected to the circuit board by soldering.
[0058] Other arrangements of the infrared transmitter 115 and the infrared camera 120 may be implemented, including arrangements in which both the infrared transmitter 115 and the infrared camera 120 are on the right rim 107A or at different locations on the frame 105. For example, the infrared transmitter 115 may be on the left rim 107B and the infrared camera 120 may be on the right rim 107A. In another example, the infrared transmitter 115 may be on the frame 105 and the infrared camera 120 may be on one of the temples 110A or 110B, or vice versa. The infrared transmitter 115 may be connected essentially anywhere on the frame 105, the right temple 110A, or the left temple 110B to emit the infrared light pattern. Similarly, the infrared camera 120 may be connected essentially anywhere on the frame 105, the right temple 110A, or the left temple 110B to capture at least one reflection change in the emitted infrared light pattern.
[0059] Infrared transmitter 115 and infrared camera 120 can be arranged to face inward toward the user's eyes, with a partial or full field of view of the eyes to identify the corresponding eye position and gaze direction. For example, infrared transmitter 115 and infrared camera 120 can be positioned directly in front of the eyes, in the upper half of frame 105, or in temple 110A or temple 110B at both ends of frame 105.
[0060] Figure 2B is an illustration depicting a rear view of another example hardware configuration of an eye-mounted device 200. In this example configuration, the eye-mounted device 200 is depicted as including an eye scanner 213 on a right temple 210A. As shown, an infrared transmitter 215 and an infrared camera 220 are co-located on the right temple 210A. The eye scanner 213 or one or more components of the eye scanner 213 can be located on the left temple 210B and other locations of the eye-mounted device 200 (e.g., the frame 105). The infrared transmitter 215 and the infrared camera 220 are co-located on the right temple 210A. Figure 2A , but the eye scanner 213 can be varied to be sensitive to different wavelengths of light, such as Figure 2A previously described. Figure 2A Similarly, the eyewear device 200 includes a frame 105 including a right rim 107A connected to a left rim 107B via a nosepiece 106. The rims 107A-107B may include respective rings that hold respective optical elements 180A and 180B, including see-through displays 180C and 180D.
[0061] Figure 2C and Figure 2D1 is a diagram depicting a rear view of an example hardware configuration of an eye-mounted device 100 including two different types of see-through image displays 180C and 180D. In one example, these see-through image displays 180C and 180D of optical assemblies 180A and 180B include integrated image displays. Figure 2C As shown, the optical components 180A and 180B include any suitable type of display matrices 180C and 180D, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a waveguide display, or any other such display.
[0062] The optical components 180A and 180B also include one or more optical layers 176A-N, which may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components in any combination. The optical layer 176 may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eyes. The prisms of the optical layer 176 may extend over all or at least a portion of the corresponding mirror rings 175A and 175B formed in the mirror edges 107A and 107B to allow the user to see the second surface of the prism when the user's eyes look through the corresponding mirror edges 107A and 107B. The first surface of the prism of the optical layer 176 faces upward from the frame 105, and the display matrix covers the prism so that photons and light emitted by the display matrix strike the first surface. The prism may be designed in a size and shape so that light is refracted within the prism and directed toward the user's eyes through the second surface of the prism of the optical layer 176. In this regard, the second surface of the prisms of the optical layer 176 can be convex to direct light toward the center of the eye. The prisms can be sized and shaped to magnify the image projected by the see-through image displays 180C and 180D, and the light travels through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the see-through image displays 180C and 180D.
[0063] In another example, the see-through image displays 180C and 180D of the optical assemblies 180A and 180B may include Figure 2DThe projected image display shown. The optical assemblies 180A and 180B include a projector 150, which can be a three-color projector using a scanning mirror, a galvanometer, a laser projector, or other types of projectors. During operation, a light source (such as the projector 150) is disposed in or on one of the temples 110A or 110B of the eye-mounted device 100. The optical assemblies 180A and 180B can include one or more optical strips 155A-N that are spaced across the width of the lens of the optical assemblies 180A and 180B or across the depth of the lens between the front and rear surfaces of the lens.
[0064] As photons projected by the projector 150 travel through the lenses of the optical assemblies 180A and 180B, the photons encounter the optical strips 155. When a particular photon encounters a particular optical strip, the photon is turned toward the user's eye, or it passes to the next optical strip. A combination of modulation of the projector 150 and modulation of the optical strips can control a specific photon or light beam. In an example, the processor controls the optical strips 155 by issuing a mechanical signal, an acoustic signal, or an electromagnetic signal. Although shown as having two optical assemblies 180A and 180B, the eye-mounted device 100 may include other arrangements, such as a single or three optical assemblies, or the optical assemblies 180A and 180B may have different arrangements depending on the application or intended user of the eye-mounted device 100.
[0065] like Figure 2C and Figure 2D As further shown, the eyewear device 100 includes a right temple 110A adjacent to the right lateral side 170A of the frame 105 and a left temple 110B adjacent to the left lateral side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective lateral sides 170A and 170B (as shown), or may be implemented as separate components attached to the frame 105 on the respective sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.
[0066] In one example, the see-through image display includes a first see-through image display 180C and a second see-through image display 180D. The eye-wearing device 100 may include a first lens ring 175A and a second lens ring 175B, which hold the corresponding first optical component 180A and the second optical component 180B. The first optical component 180A may include a first see-through image display 180C (e.g., a display matrix or optical strip, and a projector in the right temple 110A). The second optical component 180B may include a second see-through image display 180D (e.g., a display matrix or optical strip and a projector 150B (shown as projector 150) in the right temple 110A). The continuous field of view of the continuous display image may include a viewing angle between approximately 15° and 30° measured horizontally, vertically, or diagonally, and more specifically a viewing angle of 24°. The continuous display image with a continuous field of view represents a combined three-dimensional observable area visible by stitching together the two display images presented on the first image display and the second image display.
[0067] As used herein, "angle of view" describes the angular range of the field of view associated with the displayed image presented on each of the image displays 180C and 180D of the optical assemblies 180A and 180B. "Angle of coverage" describes the angular range of the field of view associated with the displayed image presented on each of the image displays 180C and 180D of the optical assemblies 180A and 180B. "Angle of coverage" describes the angular range of the lens of the visible light camera 114A or 114B or the infrared camera 220 that can image. Typically, the image circle produced by the lens is large enough to completely cover the film or sensor, possibly including some vignetting (i.e., the brightness or saturation of the image decreases toward the periphery compared to the center of the image). If the coverage angle of the lens does not fill the sensor, the image circle will be visible, typically with significant vignetting toward the edges, and the effective viewing angle will be limited to the coverage angle. "Field of view" is intended to describe the observable area field that a user of the eyewear device 100 can see through his or her eyes via the displayed image presented on the image displays 180C and 180D of the optical assemblies 180A and 180B. Image display 180C of optical assemblies 180A and 180B may have a field of view covering an angle between 15° and 30° (eg, 24°) and a resolution of 480×480 pixels (or greater, eg, 720p, 1080p, 4K, or 8K).
[0068] Figure 3 The block diagram in FIG. 4 shows an example of capturing visible light using cameras 114A and 114B. Visible light is captured by first visible light camera 114A having a circular field of view (FOV) 111A. The selected rectangular first raw image 358A is used by image processor 412 ( Figure 4) for image processing. Visible light is also captured by a second visible light camera 114B having a circular FOV 111B. A rectangular second raw image 358B selected by an image processor 412 is used for image processing by the processor 412. The raw images 358A and 358B have overlapping fields of view 313. The processor 412 processes the raw images 358A and 358B and generates a three-dimensional image 315 for display by displays 180C and 180D. The three-dimensional image 315 is also referred to as an immersive image hereinafter.
[0069] Figure 4 The system block diagram in shows a high-level functional block diagram including example electronic components arranged in a sample configuration in the eye-wearing device 100 or 200. The illustrated electronic components include a processor 432, a memory 434, and the see-through image displays 180C and 180D.
[0070] The memory 434 includes instructions for execution by the processor 432 to implement the functionality of the eye-mounted devices 100 and 200 (including instructions for the high-speed processor 432 to control the image 315). Such functionality can be implemented by processing instructions of the eye tracking programming 445, calibration software 460, and object tracking software 470 stored in the memory 434 and executed by the high-speed processor 432. The high-speed processor 432 receives power from the battery 450 and executes the instructions stored in the memory 434. The memory 434 can be a separate component, or the memory 434 can be integrated "on-chip" with the processor 432 to perform the functionality of the eye-mounted devices 100 and 200 and communicate with external devices via a wireless connection.
[0071] The eye-mounted devices 100 and 200 may be combined with eye tracking programming 445 (e.g., using Figure 2B ) and can be implemented by the infrared transmitter 215 and the infrared camera 220 in the mobile device 500 ( Figure 5 ) and a server system 498 connected via various networks to provide user interface adjustments. The mobile device 500 can be a smartphone, a tablet, a laptop, an access point, or any other such device capable of connecting to the eye-wearing device 100 or 200 using both a low-power wireless connection 425 and a high-speed wireless connection 437. The mobile device 500 is further connected to the server system 498 via a network 495. The network 495 can include any combination of wired connections and wireless connections.
[0072] The eye-mounted devices 100 and 200 may include an image display driver 442 , an image processor 412 , a low-power circuit system 420 , and a high-speed circuit system 430 . Figure 4The components for the eye-mounted devices 100 and 200 are shown to be located on one or more circuit boards (e.g., PCB or flexible PCB 140A and 140B) in the temples. Alternatively or additionally, the depicted components may be located in the temples, frames, hinges, articulated arms, or nose bridges of the eye-mounted devices 100 and 200. The visible light cameras 114A and 114B may include digital camera elements such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, lenses, or any other corresponding visible light or light capturing elements that may be used to capture data, including images of scenes with unknown objects.
[0073] The eye tracking programming 445 implements user interface field of view adjustment instructions, including instructions for causing the eye-mounted device 100 or 200 to track eye movements of the user's eyes of the eye-mounted device 100 or 200 via the eye tracker 213. Other implemented instructions (functions) cause the eye-mounted device 100 and 200 to determine a FOV adjustment to the initial FOV 111A-111B based on the detected eye movements of the user corresponding to the continuous eye directions. Further implemented instructions generate continuous display images of the display image sequence based on the field of view adjustment. The continuous display images are generated to the user as visible output via the user interface. Such visible output appears on the perspective image displays 180C and 180D of the optical assemblies 180A and 180B, which are driven by the image display driver 442 to present a display image sequence, including an initial display image with an initial field of view and a continuous display image with a continuous field of view.
[0074] Calibration software 460 and object tracking software 470 will be described below in conjunction with FIGS. 7 to 8. Fig.13 Described in further detail.
[0075] like Figure 4As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In an example, an image display driver 442 is coupled to the high-speed circuit system 430 and is operated by the high-speed processor 432 to drive the image displays 180C and 180D of the optical components 180A and 180B. The high-speed processor 432 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required for the eye-mounted device 100 or 200. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436. In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the eye-mounted device 100 or 200, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432, which executes the software architecture for the eye-mounted device 100 or 200, is also used to manage data transmission with the high-speed wireless circuit system 436. In some examples, high-speed wireless circuitry 436 is configured to implement a wireless communication protocol, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communications standard, also referred to herein as In other examples, other high-speed communication standards may be implemented by high-speed wireless circuitry 436 .
[0076] The low-power wireless circuitry 424 and the high-speed wireless circuitry 436 of the eyewear devices 100 and 200 may include a short-range transceiver. and wireless WAN, LAN, or WAN transceivers (e.g., cellular or ). Mobile device 500 (including a transceiver that communicates via low-power wireless connection 425 and high-speed wireless connection 437) can be implemented using details of the architecture of eye-mounted devices 100 and 200, as can other elements of network 495.
[0077] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, color mapping tables, camera data generated by the visible light cameras 114A-114B and the image processor 412, and images generated by the image display driver 442 for display on the see-through image displays 180C and 180D of the optical assemblies 180A and 180B. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate, individual element of the eye-mounted device 100 or 200. In some such examples, electrical routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a system-on-chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage addressing of the memory 434 so that the low-power processor 422 will initiate the high-speed processor 432 whenever a read or write operation involving the memory 434 is required.
[0078] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the mobile device 500 and the eye-mounted devices 100 and 200 via the network 495. The eye-mounted devices 100 and 200 may be connected to a host computer. For example, the eye-mounted device 100 or 200 may be paired with the mobile device 500 via a high-speed wireless connection 437, or may be connected to the server system 498 via the network 495. In addition, as explained in more detail below, a gallery 490 of snapshots and AR objects may be maintained by the server system 498 for each user and invoked by communications that provide links to the snapshots and AR objects stored in the gallery 490.
[0079] The output components of the eyewear devices 100 and 200 include visual components, such as Figure 2C and Figure 2DThe image displays 180C and 180D of the described optical components 180A and 180B (e.g., displays such as liquid crystal displays (LCDs), plasma display panels (PDPs), light emitting diode (LED) displays, projectors, or waveguides). The image displays 180C and 180D of the optical components 180A and 180B are driven by an image display driver 442. The output components of the eye-mounted devices 100 and 200 also include auditory components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, and the like. The input components of the eye-mounted devices 100 and 200, the mobile device 500, and the server system 498 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), pointer-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and force of touch or touch gestures), audio input components (e.g., microphones), and the like.
[0080] The eye-mounted devices 100 and 200 may include additional peripheral device elements, such as ambient light and spectral sensors, biometric sensors, thermal sensors 440, or other display elements integrated with the eye-mounted devices 100 or 200. For example, the peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such components described herein. The eye-mounted devices 100 and 200 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.
[0081] For example, the biometric components of the eye-mounted devices 100 and 200 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring bio-signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying individuals (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and the like. The motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), and the like. The position components include position sensor components for generating position coordinates (e.g., Global Positioning System (GPS) receiver components), and positioning system coordinates. or Transceiver, altitude sensor components (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), orientation sensor components (e.g., a magnetometer), and the like. Such positioning system coordinates may also be received from mobile device 500 via low power wireless circuitry 424 or high speed wireless circuitry 436 over wireless connections 425 and 437.
[0082] Figure 5 Is to describe the Figure 4 A block diagram of a sample configuration of a mobile device 500 for use in conjunction with the system. Figure 5 5 is a high-level functional block diagram of an example mobile device 500 that a user can use with the eye-wearing device 100 to calibrate a user interface as described herein. The mobile device 500 may include a flash memory 505 that stores programming to be executed by the CPU 510 to perform all or a subset of the functions described herein. The mobile device 500 may also include a camera 525 that includes one or more visible light cameras (a first visible light camera and a second visible light camera with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. The flash memory 505 may also include a plurality of images or videos generated via the camera 525.
[0083] The mobile device 500 may further include an image display 530, a mobile display driver 535 for controlling the image display 530, and a display controller 540. Figure 5 In the example of , the image display 530 may include a user input layer 545 (e.g., a touch screen) that is layered over or otherwise integrated into the screen used by the image display 530. Examples of touch screen-type mobile devices that may be used include, but are not limited to, smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen-type devices are provided by way of example; the subject technology as described herein is not intended to be limited thereto. For purposes of this discussion, therefore, Figure 5 A block diagram illustration of an example mobile device 500 is provided having a user interface including a touch screen input layer 545 for receiving input (through touch, multi-touch, gestures, etc., by hand, stylus, or other tool) and an image display 530 for displaying content.
[0084] like Figure 5As shown, the mobile device 500 includes at least one digital transceiver (XCVR) 550, shown as a WWAN (Wireless Wide Area Network) XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 500 also includes additional digital or analog transceivers, such as for short-range network communication (such as via NFC, VLC, DECT, ZigBee, or ) short-range transceiver (XCVR) 555. For example, the short-range XCVR 555 can take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as IEEE 802.11. One of the standards.
[0085] In order to generate location coordinates for locating the mobile device 500, the mobile device 500 may also include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 500 may utilize either or both of the short-range XCVR 555 and the WWAN XCVR 550 to generate location coordinates for positioning. For example, based on a cellular network, or The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the mobile device 500 via the XCVR 550, 555 through one or more network connections.
[0086] The transceivers 550, 555 (i.e., network communication interfaces) may conform to one or more of the various digital wireless communication standards utilized by modern mobile networks. Examples of WWAN transceivers 550 include, but are not limited to, transceivers configured to operate according to Code Division Multiple Access (CDMA) and 3rd Generation Partnership Project (3GPP) network technologies (including, for example, but not limited to, 3GPP Type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). The transceiver may also incorporate broadband cellular network technology known as "5G". For example, transceivers 550, 555 provide two-way wireless communications of information (including digitized audio signals, static images and video signals, web page information for display, and web-related inputs), as well as various types of mobile message communications to / from the mobile device 500.
[0087] The mobile device 500 may also include a microprocessor that acts as a central processing unit (CPU) 510. A processor is a circuit having the following elements that are structured and arranged to perform one or more processing functions, typically performing various data processing functions. Although discrete logic components can be used, the example utilizes components that constitute a programmable CPU. For example, a microprocessor includes one or more integrated circuit (IC) chips that combine electronic components for performing the functions of the CPU 510. For example, the CPU 510 can be based on any known or available microprocessor architecture, such as Reduced Instruction Set Computing (RISC) using an ARM architecture, as commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of the processor circuit system can be used to constitute the CPU 510 or processor hardware in smart phones, laptop computers, and tablet computers.
[0088] The CPU 510 acts as a programmable main controller for the mobile device 500, by configuring the mobile device 500 to perform various operations, for example, according to instructions or programming executable by the CPU 510. For example, such operations may include various general operations of the mobile device 500, as well as operations related to programming for a messaging application and an AR camera application on the mobile device 500. Although the processor may be configured by using hard-wired logic, a typical processor in the mobile device is a general-purpose processing circuit that is configured by executing programming.
[0089] Mobile device 500 also includes a memory or storage system for storing programming and data. Figure 5 In the example shown, the memory system may include flash memory 505, random-access memory (RAM) 560, and other memory components 565 as needed. RAM 560 may be used as short-term storage for instructions and data processed by CPU 510, for example, as working data processing memory. Flash memory 505 typically provides long-term storage.
[0090] Thus, in the example of mobile device 500, flash memory 505 may be used to store programming or instructions for execution by CPU 510. Depending on the type of device, mobile device 500 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows Mobile, Amazon Fire OS (operating system), RIM BlackBerry OS, or similar systems.
[0091] Finally, the mobile device 500 may include an audio transceiver 570 that can receive audio signals from the environment via a microphone (not shown) and provide audio output via a speaker (not shown). The audio signal can be coupled with the video signal and other messages through a messaging application or social media application implemented on the mobile device 500.
[0092] The techniques described herein may also be used with one or more computing systems described herein, or with one or more other systems. For example, the various processes described herein may be implemented using hardware or software or a combination of both. For example, at least one of the processors, memories, storage devices, output devices, input devices, or communication connections discussed below may each be at least a portion of one or more hardware components. A dedicated hardware logic component may be constructed to implement at least a portion of one or more techniques described herein. For example, but not limited to, such hardware logic components may include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc. The applications of the devices and systems that may include various aspects may include a wide variety of electronic and computer systems. The techniques may be implemented using two or more specific interconnected hardware modules or devices with associated control and data signals that may communicate between and through the modules, or may be part of an ASIC. In addition, the techniques described herein may be implemented by a software program executable by a computer system. As examples, implementations may include distributed processing, component / object distributed processing, and parallel processing. Additionally, virtual computer system processing may be constructed to implement one or more techniques or functions as described herein.
[0093] Figure 6 The block diagram in FIG. 1 shows a computer system for implementing a processing element such as Figure 4 The backend server system shown. Figure 6 is a block diagram of a sample machine 600 on which Figure 4600. In some embodiments, the machine 600 may be connected (e.g., using a network) to other machines. In a networked deployment, the machine 600 may operate in the capacity of a server or a client machine in a server-client network environment. In some embodiments, the machine 600 may function as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment.
[0094] In a sample configuration, the machine 600 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a server, a network router, a switch or a bridge, or any machine capable of executing instructions (sequentially or otherwise) specifying actions to be taken by the machine. For example, the machine 600 may be used as a workstation, a front-end server, or a back-end server for a communication system. The machine 600 may implement the methods described herein by running software for implementing features for calibrating a user interface as described herein. Further, although only a single machine 600 is shown, the term "machine" should also be considered to include any collection of machines that individually or collectively execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.
[0095] Examples as described herein may include, or may operate on, a processor, logic or multiple components, modules or mechanisms (referred to herein as "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in some manner. In an example, a circuit may be arranged as a module in a specified manner (e.g., internally or relative to an external entity (such as other circuits)). In an example, all or part of one or more computer systems (e.g., stand-alone systems, client or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application parts or applications) as a module that operates to perform a specified operation. In an example, the software may reside on a machine-readable medium. When executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.
[0096] Thus, the term "module" is understood to encompass at least one of a tangible hardware entity or a software entity, i.e., a physically constructed, specifically configured (e.g., hardwired) or temporarily (e.g., transiently) configured (e.g., programmed) entity that operates in a specified manner or performs part or all of any operations described herein. Considering examples in which modules are temporarily configured, each module need not be instantiated at any time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. Thus, software can configure a hardware processor, for example, to constitute a particular module at one time and to constitute a different module at a different time.
[0097] The machine (e.g., computer system) 600 may include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 604, and a static memory 606, some or all of which may communicate with each other via an interconnect (e.g., a bus) 608. The machine 600 may also include a display unit 610 (shown as a video display), an alphanumeric input device 612 (e.g., a keyboard), and a user interface (UI) navigation device 614 (e.g., a mouse). In an example, the display unit 610, the input device 612, and the UI navigation device 614 may be a touch screen display. The machine 600 may additionally include a mass storage device (e.g., a drive unit) 616, a signal generating device 618 (e.g., a speaker), a network interface device 620, and one or more sensors 622. Example sensors 622 include one or more of a global positioning system (GPS) sensor, a compass, an accelerometer, temperature, light, a camera, a video camera, a sensor of physical state or position, a pressure sensor, a fingerprint sensor, a retinal scanner, or other sensors. The machine 600 may also include an output controller 624, such as a serial (e.g., universal serial bus (USB)), parallel or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate with or control one or more peripheral devices (e.g., printer, card reader, etc.).
[0098] The mass storage device 616 may include a machine-readable medium 626 on which is stored one or more sets of data structures or instructions 628 (e.g., software) that embody or are utilized by any one or more of the techniques or functionality described herein. The instructions 628 may also reside, completely or at least partially, within the main memory 604, within the static memory 606, or within the hardware processor 602 during execution thereof by the machine 600. In an example, one or any combination of the hardware processor 602, the main memory 604, the static memory 606, or the mass storage device 616 may constitute a machine-readable medium.
[0099] Although the machine-readable medium 626 is shown as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, or at least one of an associated cache and server) configured to store one or more instructions 628. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 600 and causing the machine 600 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state hard disks (SSD); and CD-ROM and digital video disk (DVD)-ROM disks. In some examples, the machine-readable medium may include a non-transitory machine-readable medium. In some examples, the machine-readable medium may include a machine-readable medium that is not a transitory propagating signal.
[0100] The instructions 628 may also be transmitted or received over the communication network 632 using a transmission medium via the network interface device 620. The machine 600 may communicate with one or more other machines using any of a variety of transmission protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a Plain Old Telephone Service (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineer (IEEE) 802.11 family of standards, known as ), IEEE 802.15.4 family of standards, Long Term Evolution (LTE) family of standards, Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, etc. In an example, the network interface device 620 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas 630 to connect to a communication network 632. In an example, the network interface device 620 may include multiple antennas 630 to perform wireless communications using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. In some examples, the network interface device 620 may perform wireless communications using multi-user MIMO technology.
[0101] The features and flow charts described herein may be embodied in one or more methods as method steps, or in one or more applications as previously described. According to some configurations, an "application" or "applications" is a program(s) that performs the functions defined in a program. Various programming languages may be used to generate one or more applications, which are structured in a variety of ways, such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems such as IOS TM ANDROID TM , The invention relates to a mobile software running on a mobile phone or other mobile operating system. In this example, the third-party application can call API (application programming interface) calls provided by the operating system to facilitate the functions described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, alternatively, the methods and processes disclosed herein can be embodied in dedicated computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).
[0102] The program aspects of the technology can be considered as "products" or "manufactures", typically in the form of at least one of executable code or associated data, which are carried or embodied in a type of machine-readable medium. For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media includes any or all tangible memories of computers, processors, etc. or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage devices at any time for software programming. All or part of the software can sometimes communicate through the Internet or various other telecommunications networks. For example, such communications can enable software to be loaded from one computer or processor to another. Therefore, another type of medium that can carry programming, media content or metadata files includes optical, electrical and electromagnetic waves, such as through wired and optical landline networks and through various air links across the physical interface between local devices. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be regarded as media that carry software. As used herein, unless restricted to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.
[0103] Thus, machine-readable media can take the form of tangible storage media in a variety of forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer or the like, such as can be used to implement the client device, media gateway, transcoder, etc. shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables; copper wires and optical fibers, including wires that constitute a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals or sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common computer-readable media forms include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other medium from which a computer can read at least one of programming code and / or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.
[0104] When using an eye-mounted device of the type described above, it is desirable that the user can position user interface (UI) elements and interact with them comfortably. However, not all UIs enable the user to interact comfortably because the UI is not calibrated to the user's physiological characteristics. The user distance calibration process implemented by the calibration software 460 will now be described, which solves this problem by having the user indicate a comfortable distance for UI interaction, and then placing UI elements at reachable distances within the UI accordingly.
[0105] Responsive UI generally refers to adapting two-dimensional (2D) elements to different screen sizes in 2D by scaling or moving the 2D elements. The process becomes more complicated when the third dimension is added. It can be argued that the third dimension (3D) equivalent to the 2D screen size is the field of view of the headset, as the field of view is the digital space that the user can see at any given time. However, there are situations where the field of view size is exceeded that AR elements need to be adapted to.
[0106] Augmented reality (AR) is unique because it merges digital objects with the physical world in real time. What a user sees in the user's environment cannot be fully controlled in an AR system, while in a virtual reality (VR) system, the user's environment can be fully controlled. This makes it difficult to integrate digital elements with the physical space in an AR system. In addition, users need to be able to easily find UI elements without tactile feedback to identify objects in the AR environment. AR devices also need to understand items in the physical world in order to seamlessly integrate digital and physical spaces. For example, it would be undesirable for a user to be unable to interact with objects because they are blocked by walls in the physical or digital space.
[0107] Gesture detection has been used with AR devices to interact with the AR environment and make selections that change the displayed image. Gesture detection must be very accurate because the user can see their hands, which makes any errors more obvious. In order to improve the accuracy of gesture detection, adjustments to gesture detection and interaction have been made. However, because users have different physical characteristics (e.g., different arm lengths and hand sizes), a one-size-fits-all UI interface is not very accurate and needs to be adjusted to the user's physical characteristics to improve accuracy.
[0108] Augmented reality also offers the option to adapt the UI size to suit the user as an individual, rather than just adjusting the physical screen size. With physical products, a limited variety of sizes can be manufactured, which is why items are usually designed to fit most people. While this works for most (but not all) people, the experience is not optimized for the individual, as in most cases it would be cost-prohibitive to make a custom-sized product for each user. This can cause ergonomic issues for those who fall outside of the average that the product was designed for. However, this is not necessarily the case with digital elements, which can be easily and quickly resized.
[0109] The proposed UI adapts to the user's ergonomics and their surroundings by using digital parameters calibrated through real-time hand and device tracking. These parameters can be collected when capturing the user's hand position for AR actions.
[0110] For example, a sample calibration operation of the UI may include the following steps:
[0111] 1. All elements locked to the user's position are placed in the UI view of the eye-wearing device 100.
[0112] 2. The user fully extends her arms.
[0113] 3. The user performs a predetermined calibration gesture.
[0114] 4. Using hand tracking landmarks (e.g., knuckles, fingertips, etc.), determine the maximum z distance the user can reach and the size of the user's hand based on the distance between the hand tracking landmarks.
[0115] 5. Then, the position and scale of the UI elements are adjusted using the transformation on the UI view of the eye-wearing device 100. For example, the relative position of the UI view to the camera is changed using the set position command and placed within the reach of the user. In addition, the size of the UI elements and the interaction radius of the gestures are changed using the set scale command to obtain maximum comfort.
[0116] 6. The system calibration adjustment mechanism allows the calculated distance to be adjusted for different users who may use the eye-wearing device 100 at different times.
[0117] To implement such calibration steps, the calibration software 460 uses real-time hand and device tracking capabilities to determine parameters that can be used to characterize the user's hands and arms used when providing gesture input. In a sample configuration, these parameters include the following user-based parameters:
[0118] PalmSize
[0119] MaximumReach
[0120] Knuckle Spacing
[0121] FingerLength: thumb, index finger, middle finger, ring finger, little finger
[0122] In describing the calibration software 460 that captures these parameters, the following terminology will be used:
[0123] Interactive AR elements: UI elements, such as buttons or sliders, that the user’s hands must directly interact with to perform the associated action.
[0124] User Lock / Head Lock: A head-locked or user-locked view follows the user everywhere at a fixed distance. Interactive AR elements can be placed in the head-locked view.
[0125] World Mesh: A real-time 3D reconstruction of the world based on what the device sees.
[0126] Once calibration is complete, interactive AR elements and groups of these elements respond by indicating their behavior. Individual elements can include responsive resizing based on Parameter and Scale values. The size of UI elements can be fixed to a ratio of a certain parameter. For example, by indicating Parameter: PalmSize and Scale: 0.5, a button can be indicated to be half the size of a palm. The default scale value is 1.
[0127] The "Shift on World Occlude" boolean (default is FALSE) can also be used to dictate behavior if an element is occluded by the world grid and cannot be interacted with. The default value hides the element, but enabling this on UI elements the "Shift on World Occlude" boolean can be used to shift occluded UI elements to the nearest non-occluded position.
[0128] The Positioning variable can also be used to indicate whether the element is summonable or user locked (values are Summonable, UserLocked, or None), which tells the system how to adjust the positioning once calibration is complete.
[0129] The Groups of Elements variable can also be used to identify the group size, including Parameter, Scale, and Overflow values. The Groups of Elements variable provides responsive resizing, but for the entire group. The Groups of Elements variable indicates the size of the entire group, and where to place additional elements if there is overflow. For example, the Parameter value can be indicated as Parameter: HandSize, Scale: 1, Overflow: Right. In addition, Item Spacing: Parameter, Scale (default: 1) can be used to indicate the spacing between items.
[0130] In the following, we will discuss Figures 7 to Fig.13 The calibration process implemented by the calibration software 460 is described in greater detail. Note that world tracking of positions in world coordinates and a world grid may be enabled within the calibration software 460 to implement the calibration process in a sample configuration.
[0131] When the calibration software 460 is first opened, the user is prompted to fully extend their arm and make a calibration gesture. In a sample configuration, the calibration gesture is a specific hand gesture to be held for a specific period of time, with all hand joints visible. The hand joints must be visible and unobstructed because their positions are used to collect parameters during the calibration period. The object tracking model applied by the object tracking software 470 detects the calibration gesture by looking at the positions of landmarks (e.g., knuckles, fingers) relative to each other.
[0132] When the calibration gesture is detected, the calibration timer starts, beginning the calibration period. If the user stops making the calibration gesture before the time is up, the timer is canceled and no parameters are stored. This prevents accidental calibration adjustments.
[0133] While the calibration timer is active, user specific parameters are collected. For example, hand parameters may be collected including:
[0134] Hand size: the distance between the knuckles and the base of the hand;
[0135] Finger length: the distance between the knuckle and the tip of each finger; and
[0136] Inter-knuckle distance: The average distance between the knuckles from the little finger to the index finger. The hand parameters of finger length 700, inter-knuckle distance 710 and palm size 720 are respectively Fig. 7A , Figure 7B and Figure 7C Depicted in.
[0137] Other user-specific parameters collected may include Maximum Reach, which is the maximum distance between the hand and the camera while the calibration period is active. The bottom landmark of the hand is used because it is least likely to be occluded, providing higher accuracy. At the end of the calibration period, the average of the distances collected for each parameter may be calculated and stored.
[0138] Once the calibration timer expires, the collected parameters will be stored for future sessions. This allows the collected calibration parameters to be retrieved between calibration sessions so that the user does not have to calibrate each time they open an application that includes gesture input functionality. The calibration parameters can be stored in a persistent storage device on the eye-mounted device 100, in an associated mobile device 500, or at a system server 498. The calibration parameters can also be stored at the system level by storing them as system parameters.
[0139] Before the UI element is used by the eye-wearing device 100, the UI element can be modified based on the stored calibration parameters. For example, the maximum reachable distance can be used to adjust the position of an out-of-reach interactive UI element. In addition, the UI element locked by the user can be placed at a fixed distance from the user. If this fixed distance exceeds the maximum reachable distance, the user cannot interact with the UI element locked by the user. In this case, the UI element locked by the user can be shifted to a position within the reachable distance.
[0140] like Fig. 8A As shown, if a summonable element 800 at position 810 is visible and the user chooses to summon the summonable element 800, the summonable elements 800 are checked to see if their position 810 is at a distance 820 from the user's hand 830 that exceeds a maximum reachable distance 840; that is, to check if the summonable element 800 is out of reach. Figure 8B As shown, when summoned to the user, the summonable elements 800 can be displaced from an out-of-reach position 810 to a position 850 adjacent to the position of the user's hand 830. Once the summoning is canceled, the summonable elements 800 return to their original position 810.
[0141] Fig. 9A A summonable element 800 is shown at a position 810 in the world grid that is obscured by a world item 900. In order to be able to interact with the summonable element 800, the summonable element 800 is displaced from the position 810 to the nearest visible position 910 that is not obscured by the world item 900, such as Fig. 9B As shown. This shifting can be performed on both the summonable view and the user-locked view. This adjustment can be repeated during the use of the gesture feature so that the summonable element can remain within the user's reach.
[0142] like Fig. 10A As shown, the measured hand size parameters can be used to adjust the scale of hand tracking UI elements and element groups. The scale of responsive UI elements is adjusted by fixing them to a certain parameter. For example, hand size 1000 (which can be a combination of palm size 720 and maximum measured finger length 700) can be used to scale interactive UI element 1010 from Fig. 10A Adjust the position shown to Fig. 10B 1000. Thus, interactive UI element 1010 is adjusted to scale to the user's hand size.
[0143] like Fig.11A and Fig. 11B As shown, the spacing of UI element groups can also be adjusted to accommodate the user's measured hand size. For example, if Fig.11A As shown, if elements 1100 are grouped together (such as in a menu), the spacing between elements 1100 can be adjusted to the desired position using measured hand parameters (such as knuckle spacing 710). Fig. 11B The spacing shown in the figure limits accidental movement and increases comfort. Fig.11A As shown, when the element (button) 1100 is too close, it is easy to accidentally click the wrong button 1100, whether due to user action (e.g., the user clicks too close to the edge of the button 1100) or model jitter. Fig. 11B As shown, in the responsive menu, the group spacing can be adjusted to a scale of the 1x finger spacing parameter 710, which can be adjusted after each calibration.
[0144] Additionally, the interaction and gesture radius can also be adjusted after calibration. For example, a pinch gesture or button interaction can be adjusted by using a slider to detect a pinch action by determining whether two of the user's fingertip positions are within a specific radius. The size of the radius can be adjusted based on the values of the finger length parameter 700 and the knuckle spacing parameter 710. The finger length parameter 700 can also be used to adjust for occlusion in the event of jitter in the z-axis value.
[0145] Once the UI parameters are adjusted for the user as described above, the customized UI is ready for use.
[0146] To recalibrate the UI, the user can make the calibration gesture again. The determined calibration parameters can be stored in the application that implements the gesture feature so that the user does not need to recalibrate every time the application is opened. However, the object tracking model of the object tracking software 470 can continue to monitor the calibration gesture so that the user can choose to recalibrate when needed. For example, when the eye-mounted device 100 is to be used by a friend, the calibration process can be automatically or manually initiated to calibrate the UI to the friend's arm length and hand size.
[0147] Fig.12 1 is a diagram depicting an application 1200 for capturing gesture calibration parameters in a sample configuration. As shown, the application 1200 is implemented by the eye-mounted device 100. The eye-mounted device 100 includes an object tracking model in the object tracking software 470, which interacts with the hand tracker routine 1230 when the status indicator 1232 indicates that the hand tracker routine 1230 is active to track landmarks 1234 (e.g., knuckles, fingers, etc.) of the user's hand. When the hand tracker routine 1230 is active, the calibration parameters are updated with the captured landmark data 1234. In particular, when the hand tracker routine 1230 is active and a calibration gesture is detected by the gesture detector 1240, a calibration process 1250 is initiated and runs for a length of time specified by a timer 1252 (determined based on the device time 1220 provided by the eye-mounted device 100) to capture landmark data 1234 to update parameters of finger length 1254, palm size 1256, knuckle spacing 1258, and maximum reachable distance 1260. Once the timer 1252 has timed out and the calibration process has ended, the updated parameter data from the calibration session is sent to the storage device 434 and responsive user interface (UI) 1270 of the eye-mounted device 100 for adjusting UI parameters to customize for the user. For example, the UI may be adjusted as noted above with respect to FIGS. 7 to 11. As indicated, the device tracking software 1210 of the eye-mounted device 100 may also be used during runtime to provide a world grid 1280 for adjusting the UI on the world grid occlusion (see Fig. 9A and Fig. 9B ).
[0148] Fig.131 is a diagram illustrating the flow of the calibration process 1300 in a sample configuration. As shown, when the application 1200 is opened, the gesture detector 1240 begins hand tracking. When a calibration gesture is detected by the gesture detector 1240, the calibration process 1300 is initiated and a calibration timer 1252 is initiated. As long as the calibration is determined to remain active at 1310, the calibration process 1300 checks whether the calibration gesture remains active at 1320. If not, the calibration process 1300 is canceled. However, if the calibration gesture remains active, the session calibration parameters are updated at 1330, as described above with respect to Figures 7 to 11. The calibration process continues until the timer is determined to have expired at 1310, indicating that the calibration process 1300 has been completed at 1340. Once the calibration process 1300 is completed, the calibration parameters stored in the memory 434 of the eye-wearing device 100 are updated. In addition, the UI is adjusted at 1350 to reflect the updated calibration parameter measurements. As long as the application 1200 is active, the gesture detector 1240 continues to monitor for calibration gestures in order to re-initiate the calibration process.
[0149] The scope of protection is limited only by the claims that follow. The scope is intended and should be interpreted to be as broad as possible consistent with the ordinary meaning of the language used in the claims when interpreted in light of this specification and its subsequent prosecution history, and to encompass all structural and functional equivalents. Notwithstanding the foregoing, no claim is intended to cover subject matter that does not comply with the requirements of Sections 101, 102, or 103 of the Patent Act, nor should it be so interpreted. Any unintended inclusion of such subject matter is hereby disclaimed.
[0150] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as conferring any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not recited in the claims.
[0151] It will be understood that the terms and expressions used herein have the ordinary meanings given by such terms and expressions relative to their corresponding corresponding research and exploration fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second, etc.) can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only includes those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.
[0152] Unless otherwise indicated, all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range consistent with the functions to which they are related and with the customary practices in the art to which they belong. For example, unless otherwise expressly stated, a parameter value or similar value may vary from the stated amount by as much as ±10%.
[0153] In addition, in the foregoing detailed description, for the purpose of streamlining the present disclosure, individual features are grouped together in various examples. This method of disclosure should not be interpreted as reflecting such an intention that the examples claimed for protection require more features than those explicitly stated in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, wherein each claim exists independently as a separate claimed subject matter.
[0154] Although the foregoing has described the content of the best mode and other examples, it should be understood that various modifications can be made therein, and the subject matter disclosed herein can be implemented in various forms and examples, and they can be applied to many applications, only some of which are described herein. The following claims are intended to claim all modifications and variations that fall within the true scope of the present concept.
Claims
1. An eyewear device, comprising: a display that presents a user interface; a camera, which captures images; a memory storing instructions; and a processor coupled to the display, the camera, and the memory, wherein the processor executes the instructions to configure the eye-wearing device to: capturing the image with the camera; detecting within the captured image a calibration gesture formed by at least one appendage of the user; performing a calibration process in response to the calibration gesture, the calibration process comprising identifying calibration parameters from the at least one appendage; and The user interface is adjusted for presentation on the display in response to the calibration parameters.
2. The eyewear device according to claim 1, further comprising: parameter memory, wherein the processor performs the calibration process by executing instructions to configure the eye-wearing device to: starting a calibration timer in response to detecting the calibration gesture; performing the calibration process when the calibration gesture is detected in the image and the calibration timer has not expired; and Upon expiration of the calibration timer, the identified calibration parameters are stored in the parameter memory.
3. The eyewear device according to claim 1, wherein: The at least one appendage includes at least one of a hand or an arm, and wherein the identified calibration parameter includes at least one of a hand size, a maximum reach, a knuckle spacing, or a finger length of the user.
4. The eyewear device according to claim 1, wherein: Execution of the instructions further causes the processor to re-initiate the calibration process when another calibration gesture is detected in the captured image.
5. The eyewear device according to claim 1, wherein: The calibration gesture comprises a predetermined hand gesture of the user, whereby all hand joints on at least one hand of the user are visible.
6. The eyewear device according to claim 5, wherein: Execution of the instructions further causes the processor to identify landmark locations of at least one of a hand or an arm of the user and to measure relative distances between respective landmark locations.
7. The eyewear device according to claim 1, further comprising: A parameter memory, wherein the calibration parameters are stored as system parameters in the parameter memory.
8. The eyewear device according to claim 1, wherein: Adjusting the user interface includes modifying at least one interactable element of the user interface based on the identified calibration parameter.
9. The eyewear device according to claim 8, wherein: Modifying the at least one interactable element of the user interface includes using the measured maximum reachable distance to adjust a position of the at least one interactable element of the user interface to be within a reachable distance of the user.
10. The eyewear device according to claim 9, wherein: Execution of the instructions also causes the processor to determine whether a summonable user interface element is located at a distance from a user's hand that exceeds the measured maximum reachable distance, and when the summonable user interface element is located at a distance exceeding the measured maximum reachable distance, shift the summonable user interface element from the position exceeding the measured maximum reachable distance to a position within the reachable distance of the user's hand.
11. The eyewear device according to claim 10, wherein: Execution of the instructions further causes the processor to return the summonable user interface element to a position beyond the measured maximum reachable distance when interaction with the summonable user interface element is completed.
12. The eyewear device according to claim 6, wherein: Execution of the instructions also causes the processor to shift a summonable user interface element or a user-locked view from a position in the display that is obscured from the user's view by a physical object to a position in the display that is not obscured by the physical object, thereby enabling the user to interact with the summonable user interface element or view the user-locked view.
13. The eyewear device according to claim 1, wherein: Execution of the instructions further causes the processor to use the identified calibration parameters to adjust a scaling of user interface elements or to adjust spacing of groups of user interface elements to be scaled to the size of the user's hand.
14. The eyewear device according to claim 1, wherein: Execution of the instructions further causes the processor to use the identified calibration parameters to adjust an interaction and gesture radius for capturing images with the camera.
15. The eyewear device according to claim 1, wherein: Execution of the instructions further causes the processor to detect a new user of the eye-mounted device, and when the new user is detected, initiate the calibration process.
16. A method for calibrating a user interface (UI) of an eyewear device, comprising: detecting within a captured image from the camera a calibration gesture formed by at least one appendage of the user; performing a calibration process in response to the calibration gesture, the calibration process comprising identifying calibration parameters from the at least one appendage; and The UI is adjusted for presentation on the display in response to the calibration parameters.
17. The method according to claim 16, wherein: Executing the calibration process includes: initiating a calibration timer in response to detecting the calibration gesture; performing the calibration process when the calibration gesture is detected in an image and the calibration timer has not expired; and Upon expiration of the calibration timer, the identified calibration parameters are stored in a parameter memory.
18. The method according to claim 16, wherein: The at least one appendage includes at least one of a hand or an arm, and wherein the identified calibration parameter includes at least one of a hand size, a maximum reach, a knuckle spacing, or a finger length of the user.
19. The method according to claim 16, wherein: Detecting the calibration gesture and performing the calibration process includes: placing all user interface elements locked to the user's position in a user interface view of the eyewear device; receiving input indicative of a predetermined calibration gesture performed by the user; and The maximum distance that the user can reach and the size of the user's hands are determined based on the distance between the hand tracking landmarks of the user's hands.
20. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to calibrate a user interface (UI) of an eye-wearing device by performing operations comprising: detecting within a captured image from the camera a calibration gesture formed by at least one appendage of the user; performing a calibration process in response to the calibration gesture, the calibration process comprising identifying calibration parameters from the at least one appendage; and The UI is adjusted for presentation on the display in response to the calibration parameters.
Citation Information
Cited By
Quick-calibration AR equipment automatic debugging platform
CN223955119U
