AR glasses as IOT remote control

Abstract

An AR-enabled wearable electronic device, such as smart glasses, is suitable for use as a (Internet of Things) IoT remote control device in which a user can control pointers on a television screen, a computer screen, or other IoT-enabled device to select items by looking towards the items and selecting using gestures. A built-in six-degree-of-freedom (6DoF) tracking function is used for moving a pointer on a screen to facilitate navigation. The display screen is tracked in the real-world coordinates using ray casting techniques to determine the intersection of the user's gaze with the screen. Hand and head gesture detection is used to allow a user to complete various control actions by performing different gestures. These techniques are particularly useful for smart displays that provide AR enhanced content that can be viewed on a display of a wearable electronic device that supports AR.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 947,607, filed on September 19, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to remote control devices for devices supporting the Internet of Things (IoT). More particularly, but not by way of limitation, the present disclosure describes the use of wearable electronic devices (such as smart glasses) supporting augmented reality (AR) as remote control devices for devices supporting IoT. Background Art

[0004] The so-called "Internet of Things" or "IoT" is a network of physical objects embedded with sensors, software, and other technologies for enabling connectivity and data exchange with other devices via the Internet. For example, IoT devices are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices have been provided to act as smart home hubs to connect different smart home products. IoT devices have also been used in many other applications. Application layer protocols and supporting frameworks have been provided to implement such IoT applications. Artificial intelligence has also been combined with IoT infrastructure to enable more efficient IoT operations, improve human-computer interaction, and enhance data management and analysis.

[0005] In recent years, so-called "smart" TVs have incorporated IoT functionality (such as internet connectivity) to facilitate streaming services. However, navigation on a smart TV can be quite cumbersome. Typically, a user must navigate through menus and screens by using a physical 4-way arrow key on a remote control device. The number of actions a user can perform on a selected item is limited by the physical space available for buttons on the remote control device. The requirement to use a relatively complex remote control device has been a source of increasing frustration for customers. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The features of the various embodiments disclosed will be readily understood from the following detailed description with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar element.

[0007] Unless otherwise indicated, the various elements shown in the figures are not drawn to scale. For clarity, the size of the various elements may be enlarged or reduced. These figures depict one or more embodiments and are presented only as examples and should not be interpreted as limiting. The drawings include the following figures:

[0008] Figure 1A is a side view of an example eyewear device including an optical assembly having an image display;

[0009] Figure 1B yes Figure 1A A top cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0010] Figure 2A-2D is a rear view of an example eye-mounted device depicting an image display;

[0011] Figure 3 is a schematic diagram showing an example of using the example eye-mounted device shown in any of the preceding figures to capture visible light;

[0012] Figure 4 is a system block diagram of an example eye-wearing device;

[0013] Figure 5A is a schematic diagram of a smart television display having a cursor displayed thereon;

[0014] Figure 5B yes Figure 5A Schematic diagram of a smart TV display in which a gesture is provided to trigger a contextual information tooltip that displays an IMDB rating and recent user ratings / reviews for an item pointed to by a cursor;

[0015] Figure 6 is a schematic diagram of a sample calibration screen on a smart TV display for use when calibrating a user's AR-enabled electronic eyewear device to the smart TV display in a sample configuration;

[0016] Figure 7 is a schematic diagram showing raycasting for identifying a location on a screen being viewed by a user's AR-enabled electronic eyewear device for use when placing a cursor on a smart TV display; and

[0017] Figure 8 is a flow chart of a method for controlling a cursor on a smart TV screen or a computer monitor screen in a sample configuration. DETAILED DESCRIPTION

[0018] AR-enabled wearable electronic devices (such as smart glasses) are suitable for controlling (Internet of Things) IoT devices. The AR-enabled device acts as a remote control where a user can control a pointer on a television screen, computer screen, or other IoT-enabled device to select items by looking at the items and using gestures to select them. In a sample configuration, an AR-enabled wearable electronic device (such as the SPECTACLES available from Snap Inc. of Santa Monica, California) TM ) is used as an IoT remote control device for controlling network-connected devices (such as smart TVs). Built-in six degrees of freedom (6DoF) tracking (e.g., an inertial measurement unit (IMU) combined with a camera frame) functionality is used to move a pointer on the screen to facilitate navigation. To position the cursor, the display screen is tracked in real-world coordinates to determine the intersection of the user's line of sight with the screen using ray casting techniques. Hand and head gesture detection is used to allow the user to complete various control actions by performing different gestures. The described technology is particularly useful for smart displays that provide AR-enhanced content that can be viewed on the display of an AR-enabled wearable electronic device.

[0019] In a sample configuration, an AR-enabled eyewear device is suitable for remotely controlling an Internet of Things (IoT)-enabled device (e.g., a smart TV) having an IoT display. The AR-enabled eyewear device may include a camera, a display, a memory storing instructions, and a processor coupled to the camera, the display, and the memory. The processor executes instructions to configure the eyewear device to implement a method, the method comprising pairing the eyewear device with an IoT-enabled device for communication through a communication interface therebetween, calibrating the eyewear device to a real-world coordinate position of the IoT display, determining an intersection of a field of view (FOV) of the eyewear device with the IoT display, and sending a cursor position update to the IoT-enabled device through the communication interface based on the intersection. The AR-enabled eyewear device may also be configured to detect at least one of a gesture or a head pose, and send a gesture event to the IoT-enabled device through the communication interface. In the sample configuration, the gesture event may include at least one gesture identification (ID) of at least one gesture or head pose detected, which is used by the IoT-enabled device to perform an action corresponding to the at least one gesture ID. In an alternative configuration, the AR-enabled eyewear device may also receive augmented data corresponding to the gesture ID on its display.

[0020] The AR enabled eye-mounted device may be recalibrated to adjust for sensor offset by processing a camera frame from a camera, detecting an IoT device in the camera frame, returning a bounding box of the IoT display in the camera frame, obtaining a depth map of the bounding box in the camera frame, combining the bounding box and the depth map to determine a current position of the IoT display relative to the eye-mounted device in real-world coordinates, and using the determined current position of the IoT display to adjust for sensor offset.

[0021] In other configurations, the AR-enabled eye-worn device may include an inertial measurement unit (IMU) that provides IMU data. The AR-enabled eye-worn device may be calibrated to the real-world coordinate position of the IoT display by detecting a quick response (QR) code displayed on at least three corners of the IoT display, obtaining the three-dimensional (3D) coordinate position of the detected QR code from camera frame data of a camera and IMU data of the IMU, and determining the position of the IoT display in real-world coordinates based on the 3D coordinate position. In an alternative configuration, the QR code may be communicated to the IoT-enabled device via a communication interface.

[0022] In yet another configuration, the intersection of the FOV of the eye-worn device and the IoT display may be determined by tracking the position of the IoT display relative to the eye-worn device in real-world coordinates and using ray casting to determine the intersection of the FOV of the eye-worn device and the tracked position of the IoT display. The AR-enabled eye-worn device may also send cursor position updates to the IoT device as the user of the eye-worn device moves his or her head or moves around in a room containing the IoT-enabled device. When the intersection of the FOV of the eye-worn device and the IoT display cannot be determined, the AR-enabled eye-worn device may send at least one of a default cursor position or an error message to the IoT-enabled device.

[0023] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. For the purpose of providing a thorough understanding of the disclosed subject matter and its related teachings, many details and examples are included. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing specific aspects, and are not intended to be restrictive. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0024] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including a link or the like, through which an electrical or magnetic signal generated or supplied by one system element is conducted to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" means supported directly by an element, or indirectly supported by an element through another element that is integrated into or supported by the element.

[0025] Additional objects, advantages and novel features of the examples will be set forth in part in the following description, and in part will become apparent to those skilled in the art upon examination of the following and the accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized and attained by means of the methods, tools and combinations particularly pointed out in the appended claims.

[0026] For purposes of illustration and discussion, the orientation of the eye-mounted device, associated components, and any complete device incorporating an eye scanner and camera (such as those shown in any of the accompanying drawings) is given by way of example only. In operation for a particular variable optical processing application, the eye-mounted device may be oriented in any other direction (e.g., upward, downward, sideways, or any other orientation) suitable for the particular application of the eye-mounted device. Additionally, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used by way of example only and are not limiting with respect to the direction or orientation of any optical device or component of an optical device constructed as otherwise described herein.

[0027] Reference will now be made in detail to the examples shown in the accompanying drawings and discussed below. Figure 8 Sample eye-worn devices and associated systems and methods for controlling a pointer on a smart television's display or a computer display device are described.

[0028] The system described herein includes two types of hardware components: an AR-enabled eyewear device and an IoT-enabled display device, such as a smart TV or computer monitor. However, it should be understood that other IoT-enabled devices can be remotely controlled using the techniques described herein. Figure 4 Describes an eye-mounted device that supports AR, and will refer to FIG. Figure 8 A system for controlling a pointer on a smart television display or a computer display device is described.

[0029] In a sample configuration, an eye-mounted device with augmented reality (AR) capabilities is used in the system described herein. It is desirable that an AR-enabled eye-mounted device be used in the system described herein because such devices are scalable and customizable to enable a personalized experience, enable the effect of being applied anytime, anywhere, and ensure user privacy by enabling only the user to see the transmitted information. In a sample configuration, an AR-enabled eye-mounted device, such as the SPECTACLES available from Snap Inc. of Santa Monica, California, can be used without any dedicated hardware. TM .

[0030] Figure 1A The invention is a diagram including an image display 180C ( Figure 2A ) is an illustration of a side view of an example hardware configuration of an AR-enabled eyewear device 100 including an optical assembly 180A. The AR-enabled eyewear device 100 includes a plurality of visible light cameras 114A and 114B ( Figure 3 ), wherein the first visible light camera 114A is located on the right temple 110A, and the second visible light camera 114B is located on the left temple 110B ( Figure 2A ). In the illustrated example, the optical component 180A is located on the right side of the AR-enabled eyewear device 100. The optical component 180A may be located on the left side of the AR-enabled eyewear device 100 or at other locations.

[0031] Visible light cameras 114A and 114B may include image sensors that are sensitive to wavelengths in the visible light range. Each of visible light cameras 114A and 114B has a different forward-facing angle of coverage, for example, visible light camera 114A has a depicted field of view (FOV) 111A ( Figure 3 ). The coverage angle is the angular range over which the respective image sensors of visible light cameras 114A and 114B detect incident light and generate image data. Examples of such visible light cameras 114A and 114B include high-resolution complementary metal-oxide-semiconductor (CMOS) image sensors and video graphic array (VGA) cameras, such as 640p (e.g., 640×480 pixels totaling 300,000 pixels), 720p, 1080p, 4K, or 8K. Image sensor data from visible light cameras 114A and 114B may be captured along with geolocation data, digitized by an image processor, and stored in memory.

[0032] To provide stereoscopic vision, visible light cameras 114A and 114B may be coupled to an image processor ( Figure 4 Image processor 412 may include circuitry that receives signals from visible light cameras 114A and 114B and processes those signals from visible light cameras 114A and 114B into a format suitable for storage in memory ( Figure 4 The timestamp may be added by image processor 412 or other processor controlling the operation of visible light cameras 114A and 114B. Visible light cameras 114A and 114B allow the stereo camera to simulate human binocular vision. The stereo camera also provides a method for simulating two captured images (i.e., two images captured by visible light cameras 114A and 114B) with the same timestamp. Figure 3 The image pairs 358A and 358B in the figure are used to reproduce the three-dimensional scene ( Figure 3 4. The present invention also provides the ability to generate a three-dimensional image of a scene 315 in the image processing unit 412. Such three-dimensional images allow for an immersive virtual experience that feels realistic, for example, for virtual reality or video games. For stereoscopic vision, a pair of images 358A and 358B may be generated at a given moment - one image for each of visible light cameras 114A and 114B. Depth perception is provided by optical components 180A and 180B when the generated pair of images 358A and 358B from forward-facing fields of view 111A and 111B of visible light cameras 114A and 114B are stitched together (e.g., by image processor 412).

[0033] In an example, the AR-enabled eyewear device 100 includes a frame 105, a right rim 107A, a right temple 110A extending from a right lateral side 170A of the frame 105, and a see-through image display 180C ( FIG. 2A to FIG. 2B ), the perspective image display 180C includes an optical component 180A that presents a graphical user interface (GUI) or other image to a user. The AR-enabled eyewear device 100 includes a first visible light camera 114A connected to the frame 105 or the right temple 110A to capture a first image of a scene. The AR-enabled eyewear device 100 also includes a second visible light camera 114B connected to the frame 105 or the left temple 110B to capture a second image of the scene that at least partially overlaps with the first image (e.g., simultaneously with the first visible light camera 114A). Although Figure 1A and Figure 1B , but the processor 432 ( Figure 4) is coupled to the AR-enabled eye-mounted device 100 and is connected to the visible light cameras 114A and 114B and the memory 434 accessible to the processor 432 ( Figure 4 ), and the programming in the memory 434 can be provided in the AR-enabled eye-mounted device 100 itself.

[0034] although Figure 1A , but the AR-enabled eyewear device 100 may also include a head motion tracker (e.g., Figure 1B Inertial Measurement Unit (IMU) 109) or eye tracker ( Figure 2A Element 113 in, or Figure 2B and Figure 2C The AR-enabled eyewear device 100 may also include respective perspective image displays 180C and 180D of the optical components 180A and 180B for presenting a display image sequence. The AR-enabled eyewear device 100 may also include an image display driver ( Figure 4 442 in the image display driver), which is coupled to the perspective image displays 180C and 180D to drive the image displays 180C and 180D. The perspective image displays 180C and 180D and the image display driver are described in further detail below. The AR-enabled eyewear device 100 may also include a memory 434 and a processor 432 ( Figure 4 ), the processor being able to access the image display driver 442 and the memory 434, and the programming in the memory 434. Execution of the programming by the processor 432 configures the AR-enabled eye-mounted device 100 to perform functions, including the function of presenting an initial display image of a sequence of display images via the see-through image displays 180C and 180D, the initial display image having an initial field of view corresponding to an initial head orientation or an initial eye gaze direction determined by the eye tracker 113 or 213.

[0035] Programming executed by the processor 432 may also configure the AR enabled eye-mounted device 100 to detect movement of a user of the AR enabled eye-mounted device 100 by: (i) detecting movement of a user of the AR enabled eye-mounted device 100 via a head motion tracker (e.g., Figure 1B ) tracking the head movement of the user's head via an IMU 109 in the image sensor, or (ii) via an eye tracker ( Figure 2A Element 113 in, or Figure 2B and Figure 2CElement 213 in ) tracks eye movements of the user's eyes of the AR-enabled eye-mounted device 100. Programming executed by the processor 432 may also configure the AR-enabled eye-mounted device 100 to determine a field of view adjustment for an initial field of view of an initial display image based on the detected movement of the user. The field of view adjustment may include a continuous field of view corresponding to a continuous head direction or a continuous eye direction. Programming executed by the processor 432 may also configure the AR-enabled eye-mounted device 100 to generate continuous display images in a display image sequence based on the field of view adjustment. Programming executed by the processor 432 may also configure the AR-enabled eye-mounted device 100 to present continuous display images via the perspective image displays 180C and 180D of the optical components 180A and 180B.

[0036] Figure 1B It is a depiction Figure 1A 1 is an illustration of a top cross-sectional view of optical components and electronics in a portion of an AR-enabled eye-mounted device 100, depicting a first visible light camera 114A, an IMU 109, and a circuit board 140A. A second visible light camera 114B is substantially similar in construction and placement to the first visible light camera 114A, except that the connection and coupling is on the other lateral side 170B ( Figure 2A ). As shown, the AR-enabled eyewear device 100 includes a first visible light camera 114A and a circuit board, which may be a flexible printed circuit board (PCB) 140A. The first hinge 126A connects the right temple 110A to the articulated arm 125A of the AR-enabled eyewear device 100. In some examples, the first visible light camera 114A, the flexible PCB 140A, or other electrical connectors or contacts may be located on the right temple 110A or the first hinge 126A.

[0037] As shown, the AR-enabled eye-wearing device 100 may include a head tracker 109, which includes, for example, an inertial measurement unit (IMU). An IMU is an electronic device that uses a combination of an accelerometer and a gyroscope (sometimes also a magnetometer) to measure and report the body's specific force, angular velocity, and sometimes the magnetic field around the body. The IMU works by detecting linear acceleration using one or more accelerometers and detecting rotation rate using one or more gyroscopes. A typical configuration of an IMU includes an accelerometer, a gyroscope, and a magnetometer for each axis (for each of the three axes): a horizontal axis (X) for left and right movement, a vertical axis (Y) for top and bottom movement, and a depth or distance axis (Z) for up and down movement. The accelerometer detects the gravity vector. The magnetometer defines rotation in the magnetic field (e.g., facing south, north, etc.), similar to a compass that generates a heading reference. The three accelerometers detect acceleration along the horizontal, vertical, and depth axes defined above, which may be defined relative to the ground, the AR-enabled eye-mounted device 100 , or a user wearing the AR-enabled eye-mounted device 100 .

[0038] The AR-enabled eye-mounted device 100 can detect movement of a user of the AR-enabled eye-mounted device 100 by tracking head movement of the user's head via the head motion tracker 109. The head movement includes a change in the head direction from the initial head direction on a horizontal axis, a vertical axis, or a combination thereof during presentation of an initial display image on the image display. In one example, tracking the head movement of the user's head via the head motion tracker 109 includes measuring the initial head direction on a horizontal axis (e.g., an X-axis), a vertical axis (e.g., a Y-axis), or a combination thereof (e.g., a lateral or diagonal movement) via an IMU of the head motion tracker 109. Tracking the head movement of the user's head via the head motion tracker 109 also includes measuring continuous head directions on a horizontal axis, a vertical axis, or a combination thereof via the IMU during presentation of the initial display image.

[0039] Tracking head movement of the user's head via the head motion tracker 109 may include determining a change in head orientation based on both the initial head orientation and the continuous head orientation. Detecting movement of the user of the AR-enabled eye-mounted device 100 may also include determining that a change in head orientation exceeds a deviation angle threshold on a horizontal axis, a vertical axis, or a combination thereof in response to tracking head movement of the user's head via the head motion tracker 109. In a sample construction, the deviation angle threshold is between approximately 3° and 10°. As used herein, the term "approximately" when referring to an angle means a difference of ±10% from the stated amount.

[0040] Changes along the horizontal axis slide three-dimensional objects (such as characters, Bitmoji, app icons, etc.) into or out of the field of view by, for example, hiding, unhiding, or otherwise adjusting the visibility of the three-dimensional objects. Changes along the vertical axis (e.g., in one example, when the user looks up) display weather information, time of day, date, calendar schedule, etc. In another example, when the user looks down on the vertical axis, the AR-enabled eyewear device 100 can be powered off.

[0041] like Figure 1B As shown, the right temple 110A includes a temple body 211, which is configured to accommodate a temple cap. Figure 1B Disposed within the right temple 110A are various interconnected circuit boards, such as a PCB or flexible PCB 140A, including circuit boards for a first visible light camera 114A, (one or more) microphones 130, (one or more) speakers 132, low power wireless circuitry (e.g., for communicating with a user via a wireless interface). Wireless short-range network communications) and high-speed wireless circuit systems (e.g., for A controller circuit for wireless LAN communication).

[0042] The first visible light camera 114A is coupled to or disposed on the flexible PCB 140A and is covered by a visible light camera cover lens that is aimed through (one or more) openings formed in the right temple 110A. In some examples, the frame 105 connected to the right temple 110A includes (one or more) openings for the visible light camera cover lens. The frame 105 may include a front facing side that is configured to face outward away from the user's eyes. The opening for the visible light camera cover lens may be formed on the front facing side and through the front facing side. In this example, the first visible light camera 114A has an outward-facing field of view 111A of the line of sight or perspective of the right eye of the user of the eye-wearing device 100 supporting AR. The visible light camera cover lens may also be adhered to the outward-facing surface of the right temple 110A, in which the opening is formed with an outward-facing coverage angle, but in a different outward direction. The coupling may also be indirect via an intermediary component.

[0043] First visible light camera 114A may be connected to first perspective image display 180C of first optical assembly 180A to generate a first background scene of a first continuously displayed image. Second visible light camera 114B may be connected to second perspective image display 180D of second optical assembly 180B to generate a second background scene of a second continuously displayed image. The first background scene and the second background scene may partially overlap to present a three-dimensional observable area of ​​the continuously displayed image.

[0044] Flexible PCB 140A may be disposed within right temple 110A and coupled to one or more other components housed in right temple 110A. Although shown as being formed on circuit board 140A of right temple 110A, first visible light camera 114A may be formed on another circuit board (not shown) of one of left temple 110B, articulated arm 125A, articulated arm 125B, or frame 105.

[0045] Figure 2A is a diagram depicting a rear view of an example hardware configuration of an AR-enabled eye-mounted device 100. Figure 2A As shown, the AR-enabled eyewear device 100 is in a form configured for being worn by a user. Figure 2A The example of the AR-enabled eyewear device 100 is glasses. The AR-enabled eyewear device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0046] In the glasses example, the AR-enabled eyewear device 100 includes a frame 105 including a right rim 107A connected to a left rim 107B via a nose bridge 106, the nose bridge 106 being configured to accommodate the user's nose. The right rim 107A and the left rim 107B include respective rings 175A and 175B that hold respective optical elements 180A and 180B, such as lenses and see-through displays 180C and 180D. As used herein, the term lens is meant to encompass a transparent or translucent sheet of glass or plastic having a curved surface and a flat surface that causes light to converge / diverge or causes little or no convergence / divergence.

[0047] Although shown as having two optical elements 180A and 180B, the AR-enabled eyewear device 100 may include other arrangements, such as a single optical element, depending on the application or intended user of the AR-enabled eyewear device 100. As further shown, the AR-enabled eyewear device 100 includes a right temple 110A adjacent to the right lateral side 170A of the frame 105 and a left temple 110B adjacent to the left lateral side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective lateral sides 170A and 170B (as shown), or may be implemented as separate components attached to the frame 105 on the respective lateral sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0048] exist Figure 2AIn an example, an eye scanner 113 including an infrared transmitter 115 and an infrared camera 120 is provided. The visible light camera typically includes a blue light filter to block infrared light detection. In an example, the infrared camera 120 is a visible light camera with the blue light filter removed, such as a low-resolution video graphic array (VGA) camera (e.g., 640×480 pixels, a total of 300,000 pixels). The infrared transmitter 115 and the infrared camera 120 can be co-located on the frame 105. For example, both are shown as being connected to the upper portion of the left mirror edge 107B. The frame 105 or one or more of the temples 110A and 110B may include a circuit board (not shown) that includes the infrared transmitter 115 and the infrared camera 120. For example, the infrared transmitter 115 and the infrared camera 120 can be connected to the circuit board by soldering.

[0049] Other arrangements of the infrared transmitter 115 and the infrared camera 120 may be implemented, including arrangements in which both the infrared transmitter 115 and the infrared camera 120 are on the right rim 107A or at different locations on the frame 105. For example, the infrared transmitter 115 may be on the left rim 107B and the infrared camera 120 may be on the right rim 107A. In another example, the infrared transmitter 115 may be on the frame 105 and the infrared camera 120 may be on one of the temples 110A or 110B, or vice versa. The infrared transmitter 115 may be connected essentially anywhere on the frame 105, the right temple 110A, or the left temple 110B to emit the infrared light pattern. Similarly, the infrared camera 120 may be connected essentially anywhere on the frame 105, the right temple 110A, or the left temple 110B to capture at least one reflection change in the emitted infrared light pattern.

[0050] Infrared transmitter 115 and infrared camera 120 can be arranged to face inward toward the user's eyes, with a partial or full field of view of the eyes to identify the corresponding eye position and gaze direction. For example, infrared transmitter 115 and infrared camera 120 can be positioned directly in front of the eyes, at the top of frame 105, or in temple 110A or temple 110B at both ends of frame 105.

[0051] Figure 2Bis an illustration depicting a rear view of another example hardware configuration of an AR-enabled eyewear device 200. In this example configuration, the AR-enabled eyewear device 200 is depicted as including an eye scanner 213 on a right temple 210A. As shown, an infrared transmitter 215 and an infrared camera 220 are co-located on the right temple 210A. The eye scanner 213 or one or more components of the eye scanner 213 can be located on the left temple 210B and other locations of the AR-enabled eyewear device 200 (e.g., the frame 105). The infrared transmitter 215 and the infrared camera 220 are co-located on the right temple 210A. Figure 2A , but the eye scanner 213 can be changed to be sensitive to different wavelengths of light, such as Figure 2A previously described. Figure 2A resemblance, Figure 2B The AR-enabled eyewear device 200 includes a frame 105 including a right rim 107A connected to a left rim 107B via a nose bridge 106. The rims 107A-107B may include corresponding rims that hold corresponding optical elements 180A and 180B, including see-through displays 180C and 180D.

[0052] Figure 2C and Figure 2D 1 is a diagram depicting a rear view of an example hardware configuration of an AR-enabled eyewear-mounted device 100 including two different types of see-through image displays 180C and 180D. In one example, these see-through image displays 180C and 180D of optical assemblies 180A and 180B include integrated image displays. Figure 2C As shown, the optical components 180A and 180B include any suitable type of display matrices 180C and 180D, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a waveguide display, or any other such display.

[0053] The optical components 180A and 180B also include one or more optical layers 176A-N, which may include lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components in any combination. The optical layer 176 may include a prism having a suitable size and configuration and including a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eyes. The prisms of the optical layer 176 may extend over all or at least a portion of the corresponding mirror rings 175A and 175B formed in the mirror edges 107A and 107B to allow the user to see the second surface of the prism when the user's eyes look through the corresponding mirror edges 107A and 107B. The first surface of the prism of the optical layer 176 faces upward from the frame 105, and the display matrix covers the prism so that the photons and light emitted by the display matrix hit the first surface. The size and shape of the prism can be designed so that the light is refracted within the prism and directed toward the user's eyes through the second surface of the prism of the optical layer 176. In this regard, the second surface of the prisms of the optical layer 176 can be convex to direct light toward the center of the eye. The size and shape of the prisms can be designed to magnify the image projected by the see-through image displays 180C and 180D, and the light travels through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the see-through image displays 180C and 180D.

[0054] In another example, the see-through image displays 180C and 180D of the optical assemblies 180A and 180B may include Figure 2D The projected image display shown. Optical assemblies 180A and 180B include a projector 150, which can be a three-color projector using a scanning mirror, a galvanometer, a laser projector, or other type of projector. During operation, a light source (such as projector 150) is disposed in or on one of the temples 110A or 110B of the AR-enabled eye-mounted device 100. Optical assemblies 180A and 180B may include one or more optical strips 155A-N that are spaced across the width of the lens of the optical assemblies 180A and 180B or across the lens depth between the front and rear surfaces of the lens.

[0055] As photons projected by the projector 150 travel through the lenses of the optical assemblies 180A and 180B, the photons encounter the optical strip 155. When a particular photon encounters a particular optical strip, the photon is redirected toward the user's eye, or it passes to the next optical strip. A combination of modulation of the projector 150 and modulation of the optical strip can control a specific photon or light beam. In an example, the processor controls the optical strip 155 by emitting a mechanical signal, an acoustic signal, or an electromagnetic signal. Although shown as having two optical assemblies 180A and 180B, the AR-enabled eye-mounted device 100 may include other arrangements, such as a single or three optical assemblies, or the optical assemblies 180A and 180B may have different arrangements depending on the application or intended user of the AR-enabled eye-mounted device 100.

[0056] like Figure 2C and Figure 2D As further shown, the AR-enabled eyewear device 100 includes a right temple 110A adjacent to the right lateral side 170A of the frame 105 and a left temple 110B adjacent to the left lateral side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective lateral sides 170A and 170B (as shown), or may be implemented as separate components attached to the frame 105 on the respective lateral sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0057] In one example, the perspective image display includes a first perspective image display 180C and a second perspective image display 180D. The AR-enabled eyewear device 100 may include a first lens ring 175A and a second lens ring 175B, which hold corresponding first optical components 180A and second optical components 180B. The first optical component 180A may include a first perspective image display 180C (e.g., a display matrix or optical strip and a projector in the right temple 110A). The second optical component 180B may include a second perspective image display 180D (e.g., a display matrix or optical strip and a projector 150 in the right temple 110A). The continuous field of view of the continuous display image may include a viewing angle between approximately 15° and 30° measured horizontally, vertically, or diagonally, and more specifically a viewing angle of 24°. The continuous display image with a continuous field of view represents a combined three-dimensional observable area visible by stitching together the two display images presented on the first image display and the second image display.

[0058] As used herein, "angle of view" describes the angular range of the field of view (FOV) associated with the displayed image presented on each of the image displays 180C and 180D of the optical components 180A and 180B. "Coverage angle" describes the angular range or FOV that the lens of the visible light camera 114A or 114B or the infrared camera 220 is able to image. Typically, the image circle produced by the lens is large enough to completely cover the film or sensor, possibly including some vignetting (i.e., the brightness or saturation of the image decreases toward the periphery compared to the center of the image). If the coverage angle of the lens does not fill the sensor, the image circle will be visible, typically with significant vignetting toward the edges, and the effective viewing angle will be limited to the coverage angle. The FOV is intended to describe the observable area field that a user of the AR-enabled eye-wearing device 100 can see through his or her eyes via the displayed images presented on the image displays 180C and 180D of the optical components 180A and 180B. Image display 180C of optical assemblies 180A and 180B may have a FOV with an angle of coverage between 15° and 30° (eg, 24°) and a resolution of 480×480 pixels (or greater, eg, 720p, 1080p, 4K, or 8K).

[0059] Figure 3 The block diagram in FIG. 4 shows an example of capturing visible light using cameras 114A and 114B. Visible light is captured by first visible light camera 114A having circular FOV 111A. The selected rectangular first raw image 358A is used by image processor 412 ( Figure 4 ) for image processing. Visible light is also captured by a second visible light camera 114B having a circular FOV 111B. A rectangular second raw image 358B selected by an image processor 412 is used for image processing by the processor 412. The raw images 358A and 358B have overlapping FOVs 313. The processor 412 processes the raw images 358A and 358B and generates a three-dimensional image 315 for display by displays 180C and 180D. The three-dimensional image 315 is also referred to as an immersive image hereinafter.

[0060] Figure 4 The system block diagram in shows a high-level functional block diagram including example electronic components arranged in a sample configuration in the AR-enabled eye-mounted device 100 or 200. The illustrated electronic components include a processor 432, a memory 434, and the see-through image displays 180C and 180D.

[0061] The memory 434 includes instructions for execution by the processor 432 to implement the functions of the AR-enabled eye-mounted devices 100 and 200 (including instructions for the high-speed processor 432 to control the image 315). Such functions can be implemented by processing instructions of the eye tracking programming 445 and gesture detection / object tracking software 470 stored in the memory 434 and executed by the high-speed processor 432. As described below with reference to Figure 8 As described, the gesture detection / object tracking software 470 may include depth determination software 472, object tracking software 474, object detection software 476, and hand / gesture detection software 478 for calibration and intersection determination of the AR-enabled eye-mounted device 100, as well as head tracking and hand / gesture detection through the AR-enabled eye-mounted device 100.

[0062] The high-speed processor 432 receives power from the battery 450 and executes instructions stored in the memory 434. The memory 434 can be a separate component, or the memory 434 can be integrated "on-chip" with the processor 432 to perform the functions of the AR-enabled eye-mounted devices 100 and 200 and communicate with external devices via wireless connections.

[0063] The AR-enabled eyewear devices 100 and 200 may be combined with eye tracking programming 445 (e.g., using Figure 2B 495) and can provide user interface adjustments via a mobile device 480 and a server system 498 connected via various networks. The mobile device 480 can be a smartphone, a tablet, a laptop, an access point, or any other such device capable of connecting to the AR-enabled eye-mounted device 100 or 200 using both a low-power wireless connection 425 and a high-speed wireless connection 437. The mobile device 480 is further connected to the server system 498 via a network 495. The network 495 can include any combination of wired and wireless connections.

[0064] The AR-enabled eye-mounted devices 100 and 200 may include an image display driver 442 , an image processor 412 , a low-power circuit system 420 , and a high-speed circuit system 430 . Figure 4The components of the eye-mounted devices 100 and 200 for supporting AR are shown to be located on one or more circuit boards (e.g., PCBs or flexible PCBs 140A and 140B) in the corresponding temples 110A and 110B. Alternatively or in addition, the depicted components may be located in the temples, frames, hinges, articulated arms, or nose bridges of the AR-enabled eye-mounted devices 100 and 200. The visible light cameras 114A and 114B may include digital camera elements such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, lenses, or any other corresponding visible light or light capturing elements that can be used to capture data (including images of scenes with unknown objects).

[0065] The eye tracking programming 445 implements user interface FOV adjustment instructions, including instructions for causing the AR-enabled eye-mounted device 100 or 200 to track eye movements of a user's eyes of the AR-enabled eye-mounted device 100 or 200 via the eye tracker 213. Other implemented instructions (functions) cause the AR-enabled eye-mounted device 100 and 200 to determine a FOV adjustment to the initial FOV 111A-111B based on detected eye movements of the user corresponding to continuous eye directions. Further implemented instructions generate continuous display images of a display image sequence based on the FOV adjustment. The continuous display images are generated to the user as visible output via the user interface. Such visible output appears on the perspective image displays 180C and 180D of the optical assemblies 180A and 180B, which are driven by the image display driver 442 to present a display image sequence, including an initial display image with an initial FOV and a continuous display image with a continuous FOV.

[0066] The object tracking model applied by the gesture detection / object tracking software 470 can, for example, detect the user's gestures as well as objects in the environment, which will be recognized by device or server-based object recognition software associated with the AR-enabled eye-mounted device 100 or 200 in the sample configuration.

[0067] like Figure 4As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In an example, an image display driver 442 is coupled to the high-speed circuit system 430 and is operated by the high-speed processor 432 to drive the image displays 180C and 180D of the optical components 180A and 180B. The high-speed processor 432 can be any processor required by the AR-enabled eye-mounted device 100 or 200 that can manage high-speed communications and operations of any general-purpose computing system. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436. In some examples, the high-speed processor 432 executes an operating system, such as a LINUX operating system or other such operating system of the AR-enabled eye-mounted device 100 or 200, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432 that executes the software architecture for the AR-enabled eye-mounted device 100 or 200 is also used to manage data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuitry 436 is configured to implement communications standards such as those of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 (also referred to herein as ) wireless communication protocol. In other examples, other high-speed communication standards can be implemented by the high-speed wireless circuit system 436.

[0068] The low-power wireless circuitry 424 and high-speed wireless circuitry 436 of the AR-enabled eyewear devices 100 and 200 may include a short-range transceiver. and wireless WAN, LAN, or WAN transceivers (e.g., cellular or ). Mobile device 480 (including a transceiver that communicates via low-power wireless connection 425 and high-speed wireless connection 437) can be implemented using details of the architecture of AR-enabled eye-mounted devices 100 and 200, as can other elements of network 495.

[0069] The memory 434 includes any storage device capable of storing various data and applications, including, among other things, color maps, camera data generated by the visible light cameras 114A-B, the image processor 412, and images generated by the image display driver 442 for display on the see-through image displays 180C and 180D of the optical assemblies 180A and 180B. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate, separate element of the AR-enabled eye-mounted device 100 or 200. In some such examples, electrical routing lines can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a system-on-chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage addressing of the memory 434 so that the low-power processor 422 will start the high-speed processor 432 whenever a read or write operation involving the memory 434 is required.

[0070] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the mobile device 480 and the AR-enabled eye-mounted devices 100 and 200 via a network 495. The AR-enabled eye-mounted device 100 or 200 may be connected to a host computer. For example, the AR0-enabled eye-mounted device 100 or 200 may be paired with the mobile device 480 via a high-speed wireless connection 437, or may be connected to the server system 498 via a network 495. In addition, a gallery 490 of snapshots and AR objects may be maintained by the server system 498 for each user and invoked by communications that provide links to the snapshots and AR objects stored in the gallery 490.

[0071] The output components of the AR-enabled eyewear devices 100 and 200 include visual components such as Figure 2C and Figure 2DThe image displays 180C and 180D of the described optical components 180A and 180B (e.g., displays such as liquid crystal displays (LCDs), plasma display panels (PDPs), light emitting diode (LED) displays, projectors, or waveguides). The image displays 180C and 180D of the optical components 180A and 180B are driven by an image display driver 442. The output components of the AR-enabled eye-mounted devices 100 and 200 may also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), other signal generators, and the like. The input components of the AR-enabled eye-mounted devices 100 and 200, the mobile device 480, and the server system 498 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touch pad, a trackball, a joystick, a motion sensor, or other directional instruments), tactile input components (e.g., a physical button, a touch screen or other tactile input component that provides the location and force of a touch or touch gesture), audio input components (e.g., a microphone), and the like.

[0072] The AR-enabled eye-mounted devices 100 and 200 may include additional peripheral elements, such as ambient light and spectral sensors, biometric sensors, thermal sensors 440, or other display elements integrated with the AR-enabled eye-mounted devices 100 or 200. For example, the peripheral elements may include any I / O components, including output components, motion components, position components, or any other such components described herein. The AR-enabled eye-mounted devices 100 and 200 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0073] For example, the biometric recognition components of the AR-enabled eye-mounted devices 100 and 200 may include components for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring bio-signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identifying individuals (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and the like. The motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), and the like. The position components include position sensor components for generating position coordinates (e.g., Global Positioning System (GPS) receiver components), and positioning system coordinates. or Transceiver, altitude sensor components (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), orientation sensor components (e.g., a magnetometer), and the like. Such positioning system coordinates may also be received from mobile device 480 via low power wireless circuitry 424 or high speed wireless circuitry 436 over wireless connections 425 and 437.

[0074] The techniques described herein may also be used with one or more of the computer systems described herein, or with one or more other systems. For example, the various processes described herein may be implemented using hardware or software or a combination of the two. For example, at least one of the processors, memories, storage devices, one or more output devices, one or more input devices, or communication connections discussed below may each be at least a portion of one or more hardware components. A dedicated hardware logic component may be constructed to implement at least a portion of one or more techniques described herein. For example, but not limited to, such hardware logic components may include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chipsystems (SOCs), complex programmable logic devices (CPLDs), etc. The applications of the devices and systems that may include various aspects may include a wide variety of electronic and computer systems. The techniques may be implemented using two or more specific interconnected hardware modules or devices with associated control and data signals, which may communicate between and through the modules, or may be part of an ASIC. In addition, the techniques described herein can be implemented by software programs executable by a computer system. As an example, implementations can include distributed processing, component / object distributed processing, and parallel processing. In addition, as described herein, virtual computer system processing can be constructed to implement one or more techniques or functions.

[0075] Examples as described herein may include, or may operate on, a processor, logic or multiple components, modules or mechanisms (referred to herein as "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in some manner. In an example, a circuit may be arranged as a module in a specified manner (e.g., internally or relative to an external entity (such as other circuits)). In an example, all or part of one or more computer systems (e.g., stand-alone systems, client systems, or server computer systems) or one or more hardware processors may be configured as a module by firmware or software (e.g., instructions, application parts, or applications), which operates to perform the specified operation. In an example, the software may reside on a machine-readable medium. When executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.

[0076] Thus, the term "module" is understood to encompass at least one of a tangible hardware entity or a software entity, i.e., a physically constructed, specifically configured (e.g., hardwired) or temporarily (e.g., transiently) configured (e.g., programmed) entity that operates in a specified manner or performs part or all of any operations described herein. Considering examples in which modules are temporarily configured, each module need not be instantiated at any time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. Thus, software can configure a hardware processor, for example, to constitute a particular module at one time and to constitute a different module at a different time.

[0077] In a sample configuration, the processes described herein may be implemented by instructions stored in the memory 434 of the AR-enabled eye-mounted device 100 or 200. The memory 434 may include a machine-readable medium on which one or more sets of data structures or instructions (e.g., software) are stored that embody or are used by any one or more techniques or functions described herein. During execution of the instructions by the AR-enabled eye-mounted device 100, the instructions may also reside completely or at least partially within the high-speed processor 432 or the low-power processor 422. In an example, one or any combination of the hardware processors 432 and 422 and the memory 434 constitute a machine-readable medium.

[0078] The term "machine-readable medium" as used herein may include a single medium or multiple media (e.g., a centralized or distributed database, or at least one of an associated cache and server) configured to store instructions for implementing the processes described herein. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by processors 432 and 422 and causing the AR-enabled eye-wearing device 100 or 200 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting examples of machine-readable media may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include non-volatile memory, such as semiconductor memory devices (e.g., Electrically Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM)), and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; Random Access Memory (RAM); Solid State Drive (SSD); and CD-ROM and Digital Video Disc (DVD)-ROM disks. In some examples, the machine-readable medium may include non-transitory machine-readable media. In some examples, the machine-readable medium may include machine-readable media that is not a transitory propagating signal.

[0079] Instructions may also be transmitted or received via wireless connection 425 or 437 or directly via the Internet 495. The AR-enabled eye-mounted devices 100 and 200 may communicate with one or more other AR-enabled eye-mounted devices 100 or 200 or mobile device 480 using any of a variety of transmission protocols (e.g., frame relay, internet protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a Plain Old Telephone Service (POTS) network, and a wireless data network (e.g., the Institute of Electrical and Electronics Engineer (IEEE) 802.11 family of standards, known as ), IEEE 802.15.4 family of standards, Long Term Evolution (LTE) family of standards, Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, etc. In an example, high-speed wireless circuit system 436 and / or low-power wireless circuit system 424 may include multiple antennas to use at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology for wireless communication. In some examples, wireless circuit systems 424 and 436 may use multi-user MIMO technology for wireless communication.

[0080] The features and flow charts described herein may be embodied in one or more methods as method steps, or in one or more applications as previously described. According to some configurations, an "application" or "applications" is a program(s) that performs the functions defined in a program. Various programming languages ​​may be used to generate one or more applications, which are structured in a variety of ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language). In a specific example, a third-party application (e.g., an application created by an entity other than the vendor of a particular platform using ANDROID TM or IOS TM Software development kit (SDK) applications can be developed on mobile operating systems such as IOS TM ANDROID TM , The invention relates to a mobile software running on a mobile phone or other mobile operating system. In this example, the third-party application can call an API (application programming interface) provided by the operating system to facilitate the functions described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in dedicated computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0081] The program aspects of the technology can be considered as "products" or "manufactures", typically in the form of at least one of executable code or associated data, which are carried or embodied in a type of machine-readable medium. For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memories of computers, processors, etc. or their associated modules, such as various semiconductor memories, tape drives, disk drives, etc., which can provide non-transitory storage for software programming at any time. All or part of the software can sometimes be transmitted over the Internet or various other telecommunications networks. For example, such communications can enable software to be loaded from one computer or processor to another computer or processor. Therefore, another type of medium that can carry programming, media content or metadata files includes optical, electrical and electromagnetic waves, such as through wired and optical landline networks and through various air links across the physical interface between local devices. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be regarded as media that carry software. As used herein, unless restricted to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0082] Therefore, machine-readable media can take the form of tangible storage media in various forms. Non-volatile storage media include, for example, optical disks or disks, such as any storage device in any one or more computers or the like, such as client devices, media gateways, transcoders, etc., which can be used to implement the drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute a bus within a computer system. Carrier transmission media can take the form of electrical signals or electromagnetic signals or sound waves or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common computer-readable media forms include, for example: floppy disks, floppy disks, hard disks, tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassettes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other media from which a computer can read at least one of a programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0083] Reference will be made to Figure 5- Figure 8 A software application is described for using an AR-enabled wearable electronic device (such as smart glasses) as an IoT remote control, where a user can control a pointer on a display screen of the IoT-enabled device to select items by looking at the items and selecting using gestures.

[0084] The techniques described herein with respect to certain examples use the built-in six-degrees-of-freedom (6DoF) tracking provided by the IMU 109, combined with the camera frame capabilities of the AR-enabled eyewear device 100 (such as the SPECTACLES available from Snap Inc. of Santa Monica, California). TM ) to control a cursor on a TV or computer screen by performing head movements or gestures. A user of the AR enabled eye-mounted device 100 can look at an item she wants to select to move the cursor over that item. The hand tracking functionality of the AR enabled eye-mounted device 100 also enables the user to use different gestures to complete a variety of actions. The gestures are converted into actions using a pre-existing gesture recognition framework and ML model of the AR enabled eye-mounted device 100. The IMU 109 can also be used to collect head movement data to detect certain head gestures (such as tilting the head sideways or left / right) to, for example, indicate a desire to switch channels or watch the next / previous video, etc.

[0085] Hand / head gestures can be divided into two categories:

[0086] 1. Global gestures for actions such as "Go to Home", "Turn off TV", "Launch selected service", "Go to next", "Go to previous", etc.

[0087] 2. Gestures for actions on selected items (such as "Preview", "Select", "Show Rating", "More Information", etc.).

[0088] The AR-enabled eyewear device 100 described herein may therefore provide a manufacturer-independent remote control device and application programming interface (API) that may be adapted by television manufacturers and third-party smart TV and computer monitor application developers.

[0089] Figure 5A is a schematic diagram of a smart TV display 500 with a cursor 502 displayed thereon as a registered overlay object for operation by an AR-enabled eyewear device 100 in a sample configuration. Figure 5BAs shown, the AR-enabled eye-mounted device 100 can recognize a gesture 504 to, for example, trigger a contextual information tooltip 506 that displays the IMDB rating and recent user ratings / reviews of the item pointed to by the cursor 502. Similarly, the contextual information tooltip 506 can be a menu of related items for navigating to the target. Depending on the situation, the contextual item tooltip 506 can be displayed on the smart TV display 500, or can be superimposed on the display 180C of the AR-enabled eye-mounted device 100 as enhanced content provided by the smart TV or by the server 498.

[0090] So-called smart TVs include Internet capabilities and are usually connected via cable or Connect to a secure, low-latency local area network (LAN). The systems and methods described herein provide an application programming interface (API) exposed by the AR-enabled eye-wearing device 100 for devices on the same private LAN subnet, which can be used to exchange information such as cursor position, detected hand / head gestures (=actions), etc. A software development kit (SDK) is provided, which can be used by clients (e.g., AR-enabled eye-wearing devices 100 and smart TV displays 500) to simplify communication with the API. In a sample configuration, the protocol used for the API can be a hypertext transfer protocol (HTTP), a representational state transfer (REST) ​​API, a network socket (websocket), etc., or a lighter-weight protocol for IoT applications, such as message queue telemetry transport (MQTT), which also allows streaming to efficiently synchronize cursor positions. The pairing flow (part of the API) enables the smart TV communication application 510 to pair with the AR-enabled eye-wearing device 100 to establish a connection. In a sample configuration, the smart TV communication application 510 can be an application or library suitable for pairing with one or more AR-enabled eye-mounted devices 100 to communicate events therebetween. Actions and events can be registered to simplify integration. The smart TV communication application 510 can initiate pairing using the SDK. The SDK first internally sends out a broadcast to which the AR-enabled eye-mounted device 100 responds, so that the smart TV communication application 510 can obtain the private Internet Protocol (IP) address of the AR-enabled eye-mounted device 100 and then send a connection request to the AR-enabled eye-mounted device 100.

[0091] Once the smart TV communication application 510 and the AR-enabled eye-wearing device 100 are paired, the respective devices can use the API (via the SDK) to register callbacks and subscribe to detected actions (hand / head gestures) and changes in cursor position. In the sample configuration, a set of predefined hand / head gestures is supported. Once a known gesture is detected, the detected gesture and / or the corresponding action is passed to the smart TV communication application 510 that has registered a callback (HTTP REST endpoint or subscribes to the action in the case of MQTT). In the sample configuration, each gesture / action has a recorded identifier (gesture ID), and the respective device can decide for itself how to use certain gestures and / or actions. For example, a gesture showing a "thumbs up" can be assigned a gesture ID that is recognized by the smart TV display 500 as indicating that the volume on the smart TV is to be turned up. Similarly, a head tilt can be recognized by the AR-enabled eye-wearing device 100 and assigned a gesture ID that is recognized by the smart TV display 500 as a request to change the channel up or down. Thus, a volume control signal (increase volume) or a channel change signal may be sent to the smart TV communication application 510 for use by the internal processor of the smart TV display 500 to increase the volume or change the channel through an action recognized by the AR-enabled eye-mounted device 100.

[0092] During pairing of the Smart TV communication application 510 and the AR-enabled eye-mounted device 100, or in the device settings of the Smart TV display 500, the user can initiate a calibration process (part of the API) to calibrate the gyroscope sensor and IMU data from the AR-enabled eye-mounted device 100 to a predetermined portion (e.g., the center) of the Smart TV display 500. To perform such calibration, the user is required to look at the Smart TV display 500, which includes three detectable codes in its corners for registering the AR-enabled eye-mounted device 100 with the Smart TV display 500 in real-world coordinates.

[0093] For example, Figure 6is a schematic diagram of a sample calibration screen 600 on a smart TV display 500 for use when calibrating a user's AR-enabled eye-mounted device 100 to the smart TV display 500 in a sample configuration. As shown, a quick response (QR) code 610 is displayed in three corners of the smart TV display 500. The QR code 610 is detected, and the depth data (depth map) is used by the existing depth service of the AR-enabled eye-mounted device 100 to obtain the 3D coordinate position of the detected QR code 610 from the camera frame data of the AR-enabled eye-mounted device 100. Using these 3D coordinate positions, the TV screen rectangular (plane) real-world coordinates can be determined. Using a six-degree-of-freedom (6DOF) tracker that uses IMU data combined with the camera frame of the AR-enabled eye-mounted device 100, the screen rectangle of the smart TV display 500 can be tracked by the AR-enabled eye-mounted device 100 relative to the user's position. For example, the IMU data can be used to detect the rotation and / or tilt of the head of the user of the AR-enabled eye-mounted device 100 relative to the real-world coordinates of the rectangle of the smart TV display 500.

[0094] Alternatively, the QR code 610 can be delivered by the AR enabled eye-mounted device 100 (e.g., using Chromecast or otherwise delivered to the smart TV communication application 510 via an API between the smart TV display 500 and the AR enabled eye-mounted device 100) to the smart TV display 500 to be displayed in a corner of the smart TV display 500 to facilitate the calibration process.

[0095] Object detection can also be used to recalibrate the position of the smart TV display 500. For example, an object tracking service (e.g., a simultaneous localization and mapping (SLAM) service) can produce an offset (sensor offset) in the tracked position (tracked object position and real physical object position) over time. In order to correct such sensor offset, the tracked object position can be calibrated at certain intervals to minimize the sensor offset by visually detecting the smart TV display 500 using the existing object detection framework / infrastructure of the gesture detection / object tracking software 470. The object detection framework can process the camera input stream (frame), detect the smart TV display 500, and return the bounding box of the smart TV display 500 in the camera frame. The depth service can be used to obtain a depth map (at time x) of the bounding box detected in the camera frame. Combining this information allows the current correction screen rectangle of the smart TV display 500 to be calculated in real-world coordinates. Any offset in the position of the cursor 502 generated over time due to the use of IMU data and SLAM service data can be adjusted by such recalibration. It will be appreciated that the adjustment data may be transmitted to the TV display via Chromecast, or otherwise passed to the smart TV communication application 510 via an API between the smart TV display 500 and the AR-enabled eye-mounted device 100 .

[0096] In an example configuration, once the AR-enabled eye-mounted device 100 is calibrated relative to the smart TV display 500, the position of the cursor 502 on the smart TV display 500 can be determined by finding the intersection between the screen rectangle of the smart TV display 500 (the tracked 3D position) and an orthogonal line originating from the AR-enabled eye-mounted device 100 directed toward the center of the FOV of the smart TV display 500 using a ray casting technique. For example, Figure 7 A raycast from an orthogonal line 700 of an AR-enabled eye-mounted device 100 to a tracked 3D position of a screen rectangle 710 is shown for identifying an intersection position 720 on a smart TV display 500 that the user's AR-enabled eye-mounted device 100 is viewing at any given time for use when placing a cursor 502 on the smart TV display 500. The intersection position 720 can be solved by a regular linear algebraic equation well known to those skilled in the art. In a sample configuration, the infrastructure provided by LensStudio (such as available from Snap Inc. of Santa Monica, California) can be reused to calculate the intersection position 720 of the cursor 502.

[0097] For those cases where there is no solution to the intersection position equation (e.g., the user is not looking at the smart TV display 500 or the user's line of sight is passing over the smart TV display 500, or the user is next to or behind the smart TV display 500), a default position can be sent to the smart TV's smart communication application 510, and / or an appropriate message can be sent to each of the clients connected to the communication API.

[0098] Figure 8 is a flow chart of a method for controlling a cursor 502 on a smart TV display 500, a computer display, or other IoT device display in a sample configuration. As shown, the smart TV display 500 can be adapted by the SDK to include a communication application 510 to facilitate communication with the AR-enabled eye-wearing device 100 via an API to exchange information such as cursor position, detected hand / head gestures, etc. Similarly, the AR-enabled eye-wearing device 100 can be adapted to include an AR remote control application 800 developed using the SDK.

[0099] During operation, when the AR remote control application 800 has been launched at 810, the communication application 510 can initiate pairing with the AR-enabled eye-mounted device 100. The communication application 510 initiates pairing and subscribes to dynamic events by sending a request using, for example, low-latency, low-payload overhead MQTT at 812. At 820, the AR remote control application 800 is paired with the smart TV display 500 and is registered for events such as detected gestures.

[0100] At 822, a calibration process 830 is initiated. For example, a reference Figure 6 As described above for Figure 6 As described above, the calibration process 830 may require depth determination and / or object detection and tracking. In a sample configuration, the depth determination software 472, object tracking software 474, and object detection software 476 of the gesture detection / object tracking software 470 of the AR-enabled eye-mounted device 100 may be called at 832 to support calibration calculations. For example, the object detection software 476 and the object tracking software 474 may implement a SLAM process for automatic recalibration instead of the QR code 610.

[0101] Once the AR-enabled eyewear device 100 is calibrated at 830, the AR-enabled eyewear device 100 is calibrated at 840 using, for example, Figure 7The ray casting technique described above is used to calculate the intersection 720 of the FOV of the AR-enabled eye-mounted device 100 and the smart TV display 500. In a sample configuration, the depth determination software 472, the object tracking software 474, and the object detection software 476 of the gesture detection / object tracking software 470 of the AR-enabled eye-mounted device 100 can be called at 842 to support the intersection calculation. The intersection determined at 840 is identified as the desired cursor position, and a cursor position update 852 is sent to all registered clients (e.g., the smart TV communication application 510) at 850. The SLAM service of the gesture detection / object tracking software 470 can be used to track the position of the rectangle of the smart TV display 500 in real-world coordinates to determine such an intersection.

[0102] If a hand or head gesture is detected by hand / gesture detection software 478 at 860, a gesture event 862 is sent to all registered clients (e.g., smart TV communication application 510). In a sample configuration, gesture event 862 includes a gesture ID (e.g., thumb up, head tilt, head rotation, etc.) recognized by hand / gesture detection software 478. Registered clients can use the gesture ID to perform an input action that has been mapped to the gesture ID by the corresponding registered client (e.g., change station, volume control, select rating / review, etc.).

[0103] Steps 840-860 are continuously repeated as long as the AR remote control application 810 is operating. As described above, the AR enabled eyewear device 100 may also be recalibrated from time to time to adjust the sensor offset by visually detecting the smart TV display 500 using the existing object detection framework / infrastructure of the gesture detection / object tracking software 470 and adjusting the sensor offset.

[0104] At 870 , the AR remote control application 800 is exited.

[0105] Thus, the AR remote control application 800 enables a user wearing an AR-enabled eye-mounted device 100 to present a cursor 502 on the smart TV display 500 at the intersection 720 of the orthogonal rays projected from the AR-enabled eye-mounted device 100 and the smart TV display 500, and updates the position of the cursor 502 as the user moves her head and moves around the room. As long as the user's FOV intersects the smart TV display 500, the intersection 720 is continuously tracked and updated to reflect the updated cursor position. The user can also perform gestures to make selections on the smart TV display 500. In a sample configuration, once the desired selection is made, the AR remote control application 800 can be exited so that the cursor 502 does not interfere with the viewing of information on the smart TV display 500 during use. A switch can be provided on the AR-enabled eye-mounted device 100 for turning the AR remote control application 800 on / off as needed. Alternatively, the AR remote control application 800 can remain active, but the cursor can be programmed to disappear for a predetermined amount of time after the user's selection.

[0106] In other configurations, the user may perform a gesture that is recognized by the AR-enabled eye-mounted device 100 and provided to the smart TV display 500, as described above. However, instead of making a selection that is displayed on the smart TV display 500, the gesture ID may be mapped to augmented data that is provided to the display of the AR-enabled eye-mounted device 100 from the smart TV communication application 510 or from a third-party server as an overlay. For example, a user may select a streaming application from the smart TV display 500 using the above techniques. The server of the streaming application may send augmented data in the form of a menu or other display that may be navigated directly on the user's AR-enabled eye-mounted device 100. Any selection of augmented data presented on the AR-enabled eye-mounted device 100 may be communicated to the smart TV communication application 510 so that the desired selection may be made by the smart TV display 500.

[0107] Those skilled in the art will appreciate that the remote control operations described herein are not limited to smart TVs. Any device that can be connected to a local area network and accepts remote control input (e.g., an IoT device) can be controlled by the AR-enabled eye-wearing device 100 using the techniques described herein.

[0108] The scope of protection is defined solely by the claims that now follow. When interpreted in light of this specification and the subsequent prosecution history, the scope is intended and should be interpreted to be consistent with the ordinary meaning of the language used in the claims and to include all structural and functional equivalents. Notwithstanding the foregoing, no claim is intended to include subject matter that does not satisfy the requirements of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0109] Except as stated immediately above, nothing that has been stated or shown is intended or should be construed as conferring any component, step, feature, object, benefit, advantage, or equivalent to the public, whether or not recited in the claims.

[0110] It will be understood that the terms and expressions used herein have the ordinary meanings given by such terms and expressions relative to their corresponding corresponding research and exploration fields, unless the specific meanings thereof have been set forth in addition herein. Relational terms (such as first and second, etc.) can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "include", "comprise", "contain", "cover" or any other variation thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including or including a series of elements or steps not only includes those elements or steps, but also may include other elements or steps that are not explicitly listed or inherent to such processes, methods, articles or devices. In the absence of further constraints, an element beginning with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0111] Unless otherwise indicated, all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the claims that follow) are approximate and not exact. Such quantities are intended to have a reasonable range that is consistent with the functions to which they are related and with the customary practices in the art to which they belong. For example, unless otherwise expressly stated, a parameter value or similar value may vary from the stated amount or range by as much as ±10%.

[0112] In addition, in the foregoing detailed description, it can be seen that, for the purpose of streamlining the present disclosure, individual features are grouped together in various examples. This method of disclosure should not be interpreted as reflecting an intention that the examples claimed for protection require more features than those explicitly recited in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected lies in less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim existing independently as a separate claimed subject matter.

[0113] While the foregoing has described what are considered to be the best modes and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied in many applications, only some of which are described herein. The following claims are intended to claim any and all modifications and variations that fall within the true scope of the present concept.

Claims

1. An eye-worn device adapted to remotely control an IoT-enabled device having an IoT display, the eye-worn device comprising: camera; a memory for storing instructions; and a processor coupled to the camera and the memory, wherein the processor executes the instructions to configure the eye-wearing device to: pairing the eye-worn device with an IoT-enabled device to communicate via a communication interface therebetween; Calibrate the eyewear device to the real-world coordinate position of the IoT display; Determining an intersection point of a field of view (FOV) of the eyewear device and the IoT display; as well as A cursor position update is sent to the IoT enabled device via the communication interface based on the intersection.

2. The eyewear device according to claim 1, wherein: The processor also executes instructions to configure the eye-wearing device to implement the following operations: detecting at least one of a hand gesture or a head pose; and sending gesture events to the IoT-enabled device via the communication interface, The gesture event includes at least one gesture identifier (ID) of at least one detected hand gesture or head gesture, and the gesture identifier is used by the IoT-enabled device to perform an action corresponding to the at least one gesture ID.

3. The eyewear device according to claim 2, further comprising a display, wherein: The processor also executes instructions to configure the eye-mounted device to receive augmented data corresponding to the gesture ID on the display.

4. The eyewear device according to claim 1, wherein: The processor also executes instructions to configure the eye-mounted device to recalibrate the eye-mounted device by processing a camera frame from the camera, detecting an IoT device in the camera frame, returning a bounding box of the IoT display in the camera frame, obtaining a depth map of the bounding box in the camera frame, combining the bounding box and the depth map to determine a current position of the IoT display relative to the eye-mounted device in real-world coordinates, and adjusting a sensor offset using the determined current position of the IoT display.

5. The eyewear device according to claim 1, wherein: The processor also executes instructions for pairing the eye-mounted device with the IoT-enabled device by configuring the eye-mounted device to subscribe to dynamic events, the dynamic events comprising messages related to at least one of a hand gesture or a head pose on the communication interface.

6. The eyewear device according to claim 1, wherein: The processor also executes instructions to pair the eye-worn device with the IoT-enabled device by sending a request through the communication interface in a protocol including at least one of Hypertext Transfer Protocol (HTTP), Representational State Transfer (REST) ​​API, web sockets, or Message Queue Telemetry Transport (MQTT).

7. The eyewear device according to claim 1, further comprising an inertial measurement unit (IMU), wherein: The processor also executes instructions for calibrating the eye-mounted device to the real-world coordinate position of the IoT display by detecting a quick response (QR) code displayed in at least three corners of the IoT display, obtaining a three-dimensional (3D) coordinate position of the detected QR code from camera frame data of the camera and IMU data of the IMU, and determining the position of the IoT display in real-world coordinates from the 3D coordinate position.

8. The eyewear device according to claim 7, wherein: The processor also executes instructions to communicate the QR code to the IoT enabled device via the communication interface.

9. The eyewear device according to claim 1, wherein: The processor also executes instructions for determining the intersection of the FOV of the eye-mounted device and the IoT display by tracking the position of the IoT display relative to the eye-mounted device in real-world coordinates and using ray casting to determine the intersection of the FOV of the eye-mounted device and the tracked position of the IoT display.

10. The eyewear device according to claim 1, wherein: The processor also executes instructions to configure the eye-worn device to send cursor position updates to the IoT device when a user of the eye-worn device moves her head or moves around in a room containing the IoT-enabled device.

11. The eyewear device according to claim 10, wherein: The processor also executes instructions to configure the eye-mounted device to send at least one of (a) a default cursor position or (b) an error message to the IoT-enabled device when the intersection of the FOV of the eye-mounted device and the IoT display cannot be determined.

12. The eyewear device according to claim 1, wherein: The IoT-enabled devices include smart TVs.

13. A method of remotely controlling an IoT-enabled device having an augmented reality (AR) enabled IoT display using an eye-worn device, comprising: Pairing the AR-enabled eye-worn device with the IoT-enabled device to communicate via a communication interface between them; Calibrate the AR-enabled eyewear device to the real-world coordinate position of the IoT display; Determining an intersection point of a field of view (FOV) of the AR enabled eyewear device and the IoT display; and A cursor position update is sent to the IoT enabled device via the communication interface based on the intersection.

14. The method according to claim 13, further comprising: detecting at least one of a hand gesture or a head pose; as well as sending gesture events to the IoT-enabled device via the communication interface, The gesture event includes at least one gesture identifier (ID) of at least one detected hand gesture or head gesture, and the gesture identifier is used by the IoT-enabled device to perform an action corresponding to the at least one gesture ID.

15. The method of claim 13, further comprising recalibrating the AR enabled eye-mounted device by processing a camera frame from the camera, detecting an IoT device in the camera frame, returning a bounding box of the IoT display in the camera frame, obtaining a depth map of the bounding box in the camera frame, combining the bounding box and the depth map to determine a current position of the IoT display relative to the AR enabled eye-mounted device in real-world coordinates, and adjusting a sensor offset of the AR enabled eye-mounted device using the determined current position of the IoT display.

16. The method according to claim 13, wherein: Calibrating the AR-enabled eye-mounted device to the real-world coordinate position of the IoT display includes detecting a quick response (QR) code displayed in at least three corners of the IoT display, obtaining a three-dimensional (3D) coordinate position of the detected QR code from camera frame data of a camera of the AR-enabled eye-mounted device and inertial measurement unit (IMU) data from an IMU of the AR-enabled eye-mounted device, and determining a position of the IoT display in real-world coordinates from the 3D coordinate position.

17. The method according to claim 13, wherein: Determining the intersection of the FOV of the AR enabled eye-mounted device and the IoT display includes tracking a position of the IoT display relative to the AR enabled eye-mounted device in real-world coordinates, and using ray casting to determine the intersection of the FOV of the AR enabled eye-mounted device and the tracked position of the IoT display.

18. The method of claim 13, further comprising sending cursor position updates to the IoT device when a user of the AR enabled eye-worn device moves her head or moves around in a room containing the IoT enabled device.

19. The method of claim 18, further comprising sending at least one of (a) a default cursor position or (b) an error message to the IoT enabled device when the intersection of the FOV of the eye-mounted device and the IoT display cannot be determined.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to remotely control an augmented reality (AR) enabled device having an IoT display using an eye-worn device that supports IoT by performing operations including: Pairing the AR-enabled eye-worn device with the IoT-enabled device to communicate via a communication interface between them; Calibrate the AR-enabled eyewear device to the real-world coordinate position of the IoT display; Determining an intersection point of a field of view (FOV) of the AR enabled eyewear device and the IoT display; as well as A cursor position update is sent to the IoT enabled device via the communication interface based on the intersection.

Images