AR glasses as IOT device for enhanced screen experience

Abstract

An AR-enabled wearable electronic device, such as smart glasses, is adapted for use as an (Internet of Things) IoT device in which a user can select items to control pointers by viewing the items and selecting using gestures and present augmented reality content relative to a television screen, computer screen or other IoT-enabled device. A built-in six-degree-of-freedom (6DoF) tracking function is used for tracking a television display screen, so that cursor navigation and AR content display are facilitated. The tracking information is used to trigger an AR effect as needed in response to the gesture to display augmented reality content relative to a physical television display screen (e.g., on a television display screen or in an environment around). These techniques are particularly useful for smart displays that provide enhanced AR content that can be viewed on a display that supports an AR wearable electronic device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 18 / 097,911, filed January 17, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to devices for enhancing the screen experience of Internet of Things (IoT)-enabled devices. More specifically, but not by way of limitation, the present disclosure describes using augmented reality (AR)-enabled wearable electronic devices (such as smart glasses) to extend the screen of an IoT-enabled television with augmented reality content. Background Art

[0004] The "Internet of Things" or "IoT" is a network of physical objects embedded with sensors, software, and other technologies that are capable of connecting and exchanging data with other devices via the Internet. For example, IoT devices are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Many IoT-enabled devices are available that act as smart home hubs to connect different smart home products. IoT devices have also been used in many other applications. Application layer protocols and supporting frameworks have been provided for implementing such IoT applications. Artificial intelligence has also been integrated with IoT infrastructure to enable more efficient IoT operation, improve human-computer interaction, and enhance data management and analysis.

[0005] "Smart" TVs already include IoT features, such as internet connectivity, to facilitate streaming services. However, navigation on smart TVs can be quite cumbersome. Typically, users must navigate menus and screens using the physical four-way arrow keys on the remote control. The number of actions a user can perform on a selected item is limited by the physical space available for buttons on the remote control. The requirement to use relatively complex remote controls has become a growing source of frustration for customers. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The features of the various embodiments disclosed will be readily understood from the following detailed description, in which reference is made to the accompanying drawings. In the description and throughout the several views of the drawings, reference numerals are used with each element. When multiple similar elements are present, a single reference numeral may be assigned to the similar elements.

[0007] Unless otherwise noted, the various elements shown in the figures are not drawn to scale. For clarity, the sizes of various elements may be exaggerated or reduced. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following:

[0008] Figure 1A is a side view of an example eyewear device including an optical assembly having an image display;

[0009] Figure 1B yes Figure 1A a top cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0010] Figures 2A-2D is a rear view of an example eyewear device depicting an image display;

[0011] Figure 3 is a schematic diagram illustrating an example of capturing visible light using the example eyewear device shown in any of the preceding figures;

[0012] Figure 4 is a system block diagram of an example eyewear device;

[0013] Figure 5A is a schematic diagram of a smart television display with a cursor displayed thereon;

[0014] Figure 5B yes Figure 5A Schematic diagram of a smart TV display in which a hand gesture has been provided to trigger a contextual information tooltip that displays the IMDB rating and recent user ratings / reviews of the item pointed to by the cursor;

[0015] Figure 6 is a diagram of an example calibration screen on a smart TV display for calibrating a user's AR-enabled electronic eyewear device to the smart TV display in an example configuration;

[0016] Figure 7 is a schematic diagram illustrating ray casting for identifying a location on a screen being viewed by a user's AR-enabled electronic eyewear device for use in placing a cursor on a smart TV display;

[0017] Figure 8 is a flow chart of a method for controlling a cursor on a smart TV screen or computer display screen in an example configuration;

[0018] Figure 9A is a schematic diagram of a smart television display including a preview screen that automatically (or in response to a gesture) triggers augmented reality content (e.g., a filter including one or more 2D or 3D objects, in this case in the form of a Ferris wheel) to be displayed on an AR-enabled wearable electronic device at one of several different fixed positions relative to the television display screen;

[0019] Figure 9Bis a schematic diagram of a display on an AR-enabled wearable electronic device that is a smart television display, having augmented reality content in the form of a Ferris wheel anchored at a specific location relative to the television display screen; and

[0020] Figure 10 is a flow chart of a method for providing augmented reality content with respect to a smart television display or a computer display in an example configuration. DETAILED DESCRIPTION

[0021] Television screens typically do not support a three-dimensional (3D) experience. Even if they do, they cannot provide an immersive experience that extends beyond the boundaries of the TV screen. The technology described herein provides a method for displaying augmented reality content relative to a TV screen, seamlessly integrated into the real-world environment. The six degrees of freedom (6DoF) tracking capability of augmented reality (AR) glasses (e.g., an inertial measurement unit (IMU) combined with a camera frame) allows users of AR glasses to easily navigate using a pointer on the TV screen. Hand and head gesture detection can be used to perform various actions through different gestures. The tracking information and gesture detection used to present a cursor on the TV screen can also be used to trigger AR effects as needed to display augmented reality content relative to the physical TV display (e.g., on or in the environment surrounding the TV display).

[0022] AR-enabled wearable electronic devices, such as smart glasses, are suitable for controlling (Internet of Things) IoT devices. The AR-enabled device can be used as a remote control, where a user can control a pointer on a television screen, computer screen, or other IoT-enabled device to select items by looking at the items and selecting them using gestures. In an example configuration, an AR-enabled wearable electronic device (such as SPECTACLES available from Snap Inc. of Santa Monica, California) TM) is used as an IoT remote control device to control network-connected devices such as smart TVs. Built-in six degrees of freedom (6DoF) tracking (e.g., an inertial measurement unit (IMU) combined with a camera frame) capability is used to move a pointer on the screen to facilitate navigation. To position the cursor, the display is tracked in real-world coordinates to determine the intersection of the user's view and the screen using a ray casting technique. Hand and head pose detection is used to allow the user to perform various control actions by performing different poses. The tracking information and pose detection used to render the cursor to the TV screen are also used to trigger AR effects for the display of augmented reality content to enhance the user interface of an AR-enabled device, thereby extending the TV display to show mixed reality content positioned relative to the coordinates of the TV display. The described technology is particularly useful for smart displays that provide augmented AR content that can be viewed on the display of an AR-enabled wearable electronic device.

[0023] In an example configuration, an AR-enabled eyewear device is adapted to present augmented reality content relative to an Internet of Things (IoT)-enabled device (e.g., a smart TV) having an IoT display. The AR-enabled eyewear device may include a camera, a display, a memory storing instructions, and a processor coupled to the camera, the display, and the memory. The processor executes the instructions to configure the eyewear device to implement a method comprising pairing the AR-enabled eyewear device with the IoT-enabled device to communicate via a communication interface therebetween, calibrating the AR-enabled eyewear device to a real-world coordinate location of the IoT display, selecting augmented reality content to display on the display of the AR-enabled eyewear device relative to the real-world coordinate location of the IoT display, and rendering the selected augmented reality content to the display of the AR-enabled eyewear device at a location relative to the real-world coordinate location of the IoT display.

[0024] In an example configuration, an AR-enabled eyewear device is configured to render augmented reality content (e.g., filters available from Snap Inc. of Santa Monica, California) received from an IoT-enabled device to a display of the AR-enabled eyewear device, trigger animations, hide specific objects, show specific objects, detect at least one of a hand gesture or a head gesture, and send gesture events to the IoT-enabled device via a communication interface. For example, a user can navigate to a certain screen on a TV or press a button on the side of the TV to trigger an AR event to be displayed on the AR-enabled eyewear device. The TV can use an application programming interface provided by the AR-enabled eyewear device to send and request rendering of AR content at any time. The gesture event can include a gesture identification (ID) of at least one detected hand gesture or head gesture, which is used by the IoT-enabled device to perform an action corresponding to the gesture ID. The AR-enabled eyewear device can also be configured to receive augmentation data, including selected AR content corresponding to the gesture ID, for rendering to the display.

[0025] In an example configuration, augmented reality content may include at least one animatable two-dimensional or three-dimensional object that is sent to a device display for display at a location relative to a pre-specified anchor location of an IoT display identified in a Cartesian coordinate system. A gesture event may be recognized that initiates movement of the at least one animatable two-dimensional or three-dimensional object. The rendered augmented reality content may be displayed on a display of an AR eyewear-enabled device at a location relative to the pre-specified anchor location that is outside the dimensions of the IoT display in a world coordinate system.

[0026] The AR eyewear-enabled device may further process instructions from the IoT-enabled device to render augmented reality content (e.g., a filter from Snap Inc. of Santa Monica, California) to the device display, trigger animations, update the augmented reality content with new data, hide the augmented reality content or specific elements within the augmented reality content, display specific objects, or send any payload to the AR eyewear-enabled device that can be received and interpreted to dynamically determine what to display on the display of the AR eyewear-enabled device. The AR eyewear-enabled device may also select augmented reality content to display relative to the IoT display by selecting a streaming application from the IoT display, wherein the streaming application includes augmented data that is navigated directly on the display of the AR eyewear-enabled device.

[0027] In an example configuration, the augmented reality content is scaled relative to the height and width dimensions of the IoT display so that the augmented reality content is displayed proportionally on the IoT display. For example, the augmented reality content can be scaled to fit the IoT display by fitting the augmented reality content along the (x, y) axis of the IoT display, fitting the augmented reality content on the x-axis or y-axis while the other axis is proportionally distorted or scaled, or fitting the augmented reality content to a specific percentage of the screen plane size of the IoT display.

[0028] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. In order to provide a thorough understanding of the disclosed subject matter and its related teachings, many details and examples are included. However, those skilled in the relevant art will understand how to apply the related teachings without these details. Aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, as the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only for the purpose of describing particular aspects and are not intended to be limiting. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0029] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including links that transmit electrical or magnetic signals generated or supplied by one system element to another coupled or connected system element. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communication media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" refers to being directly supported by an element or indirectly supported by an element through another element that is integrated into or supported by the element.

[0030] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized and obtained by means of the methods, instrumentalities and combinations particularly pointed out in the appended claims.

[0031] The orientations of the eye-mounted devices, related components, and any complete devices including eye scanners and cameras, such as shown in any of the accompanying figures, are provided as examples only for purposes of illustration and discussion. In operation of a particular variable optical processing application, the eye-mounted devices can be oriented in any other direction suitable for the particular application of the eye-mounted devices, such as upward, downward, sideways, or any other orientation. Furthermore, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used only as examples and do not limit the direction or orientation of any optical device or optical device component constructed as otherwise described herein.

[0032] Reference will now be made in detail to examples shown in the accompanying drawings and discussed below.Example eye-mounted devices and associated systems and methods for controlling a pointer on a display of a smart TV or computer display device will be described with reference to Figures 1-8.

[0033] The system described herein includes two types of hardware components: an AR eyewear-enabled device and an IoT-enabled display device, such as a smart TV or computer monitor. However, it should be understood that other IoT-enabled devices can be remotely controlled using the techniques described herein. The AR eyewear-enabled device will be described with reference to Figures 1-4, and a system for controlling a pointer on a display of a smart TV or computer display device will be described with reference to Figures 5-8.

[0034] In an example configuration, an augmented reality (AR) enabled eyewear device is used in the system described herein. It is desirable to use an AR enabled eyewear device in the system described herein because such a device is scalable and customizable to enable a personalized experience, enable effects to be applied anytime, anywhere, and ensure user privacy by only allowing the user to view the transmitted information. In an example configuration, an AR enabled eyewear device, such as the SPECTACLES available from Snap Inc. of Santa Monica, California, can be used without any specialized hardware. TM .

[0035] Figure 1A is a diagram depicting a side view of an example hardware configuration of an AR-enabled eyewear device 100 including an optical assembly 180A ( Figure 2A ). The AR eyewear device 100 includes a plurality of visible light cameras 114A and 114B forming a stereo camera ( Figure 3 ), wherein the first visible light camera 114A is located on the right temple 110A, and the second visible light camera 114B is located on the left temple 110B ( Figure 2AIn the example shown, the optical component 180A is located on the right side of the AR-enabled eyewear device 100. The optical component 180A may be located on the left side of the AR-enabled eyewear device 100 or elsewhere.

[0036] Visible light cameras 114A and 114B may include image sensors that are sensitive to wavelengths in the visible light range. Each of visible light cameras 114A and 114B has a different front-facing overlap angle, for example, visible light camera 114A has a depicted field of view (FOV) 111A ( Figure 3 ). The overlap angle is the angular range over which the respective image sensors of visible light cameras 114A and 114B detect incident light and generate image data. Examples of such visible light cameras 114A and 114B include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640 x 480 pixels, for a total of 0.3 megapixels), 720p, 1080p, 4K, or 8K. Image sensor data from visible light cameras 114A and 114B can be captured along with geolocation data, digitized by an image processor, and stored in memory.

[0037] To provide stereoscopic vision, visible light cameras 114A and 114B may be coupled to an image processor ( Figure 4 Image processor 412 may include circuitry for receiving signals from visible light cameras 114A and 114B and processing those signals from visible light cameras 114A and 114B into a format suitable for storage in memory ( Figure 4 The timestamp may be added by image processor 412 or other processor controlling the operation of visible light cameras 114A and 114B. Visible light cameras 114A and 114B allow the stereo camera to simulate human binocular vision. The stereo camera also provides a time stamp based on two captured images ( Figure 3 The image pair 302A and 302B) is used to reproduce the three-dimensional scene ( Figure 3The ability to produce a three-dimensional image of a scene 306 in the image. This three-dimensional image allows for realistic and immersive virtual experiences, such as for use in virtual reality or video games. For stereoscopic vision, a pair of images 302A and 302B can be generated at a given moment—one for each of visible light cameras 114A and 114B. Depth perception is provided by optical assemblies 180A and 180B when the pair of generated images 302A and 302B from forward-facing fields of view 111A and 111B of visible light cameras 114A and 114B are stitched together (e.g., by image processor 412).

[0038] In one example, the AR-enabled eyewear device 100 includes a frame 105, a right rim 107A, a right temple 110A extending from a right lateral side 170A of the frame 105, and a see-through image display 180C (including an optical assembly 180A). Figure 2A -B) to present a graphical user interface (GUI) or other images to the user. The AR-enabled eyewear device 100 includes a first visible light camera 114A connected to the frame 105 or the right temple 110A to capture a first image of a scene. The AR-enabled eyewear device 100 also includes a second visible light camera 114B connected to the frame 105 or the left temple 110B to capture (e.g., simultaneously with the first visible light camera 114A) a second image of the scene that at least partially overlaps the first image. Although Figure 1A and Figure 1B Not shown, but the processor 432 ( Figure 4 ) is coupled to the AR eyewear-enabled device 100 and is connected to the visible light cameras 114A and 114B and the memory 432 accessible to the processor 432 ( Figure 4 ), and the programming in the memory 434 can be set in the AR eye-wearing device 100 itself.

[0039] Although Figure 1A Although not shown, the AR eyewear device 100 may also include a head motion tracker (e.g., Figure 1B Inertial Measurement Unit (IMU) 109) or eye movement tracker ( Figure 2A Element 113 or Figure 2B and Figure 2C The AR eyewear device 100 may further include perspective image displays 180C and 180D of the optical components 180A and 180B, respectively, for presenting a display image sequence. The AR eyewear device 100 may further include an image display driver ( Figure 44 and 4. The AR eyewear device 100 may further include a memory 434 and a processor 432 ( Figure 4 ), the processor 432 can access the image display driver 442 and the memory 434, as well as program in the memory 434. The execution of the program by the processor 432 configures the AR eyewear device 100 to perform functions, including the function of presenting an initial display image of a sequence of display images via the see-through image displays 180C and 180D, wherein the initial display image has an initial field of view corresponding to the initial head orientation or the initial eye gaze direction as determined by the eye movement tracker 113 or 213.

[0040] Execution of the programming by the processor 432 may further configure the AR eyewear-enabled device 100 to detect motion of the user of the AR eyewear-enabled device 100 by: (i) detecting motion of the user of the AR eyewear-enabled device 100 via a head motion tracker (e.g., Figure 1B 109) to track the head movement of the user's head, or (ii) via an eye movement tracker ( Figure 2A Element 113 or Figure 2B and 2C Element 213 of the AR eyewear-enabled device 100 tracks eye movements of a user's eye. Execution of the programming by the processor 432 may further configure the AR eyewear-enabled device 100 to determine a field of view adjustment to an initial field of view of an initial display image based on the detected user movement. The field of view adjustment may include a continuous field of view corresponding to continuous head orientations or continuous eye orientations. Execution of the programming by the processor 432 may further configure the AR eyewear-enabled device 100 to generate a continuous display image of a display image sequence based on the field of view adjustment. Execution of the programming by the processor 432 may also configure the AR eyewear-enabled device 100 to present the continuously displayed images via the see-through image displays 180C and 180D of the optical assemblies 180A and 180B.

[0041] Figure 1B It is a depiction Figure 1A 1. An illustration of a top cross-sectional view of optical components and electronics in a portion of an AR-enabled eyewear device 100 is shown, depicting a first visible light camera 114A, a head motion tracker (IMU) 109, and a circuit board 140A. A second visible light camera 114B is constructed and positioned substantially similarly to the first visible light camera 114A, except that it is connected and coupled on another lateral side 170B. Figure 2AAs shown, the AR eyewear-supporting device 100 includes a first visible light camera 114A and a circuit board, which may be a flexible printed circuit board (PCB) 140A. A first hinge 126A connects the right temple 110A to an articulated arm 125A that supports the AR eyewear-supporting device 100. In some examples, the first visible light camera 114A, the flexible PCB 140, or other electrical connectors or contacts may be located on the right temple 110A or the first hinge 126A.

[0042] As shown, the AR eyewear-enabled device 100 may include a head motion tracker 109, which includes, for example, an inertial measurement unit (IMU). An IMU is an electronic device that uses a combination of accelerometers and gyroscopes, and sometimes a magnetometer, to measure and report the body's specific force, angular velocity, and sometimes the magnetic field around the body. The IMU works by detecting linear acceleration using one or more accelerometers and detecting rotation rate using one or more gyroscopes. A typical configuration of an IMU includes an accelerometer, gyroscope, and magnetometer for each of three axes: a horizontal axis (X) for left and right movement, a vertical axis (Y) for top and bottom movement, and a depth or distance axis (Z) for up and down movement. The accelerometer detects the gravity vector. The magnetometer defines rotation in the magnetic field (e.g., facing south, facing north, etc.), just like a compass that generates a heading reference. The three accelerometers detect acceleration along the horizontal, vertical, and depth axes defined above, which can be defined relative to the ground, the AR eyewear-enabled device 100, or the user wearing the AR eyewear-enabled device 100.

[0043] The AR eyewear-enabled device 100 can detect movement of a user of the AR eyewear-enabled device 100 by tracking head movement of the user's head via the head movement tracker 109. The head movement includes a change in head orientation from an initial head orientation on a horizontal axis, a vertical axis, or a combination thereof during presentation of an initial display image on the image display. In one example, tracking head movement of the user's head via the head movement tracker 109 includes measuring the initial head orientation on a horizontal axis (e.g., an X-axis), a vertical axis (e.g., a Y-axis), or a combination thereof (e.g., a lateral or diagonal movement) via an IMU of the head movement tracker 109. Tracking head movement of the user's head via the head movement tracker 109 also includes measuring continuous head orientations on a horizontal axis, a vertical axis, or a combination thereof via the IMU during presentation of the initial display image.

[0044] Tracking the head movement of the user's head via the head movement tracker 109 may include determining a change in head orientation based on both the initial head orientation and the continuous head orientation. Detecting the movement of the user of the AR eyewear-enabled device 100 may also include determining that the change in head orientation exceeds a deviation angle threshold on the horizontal axis, the vertical axis, or a combination thereof in response to tracking the head movement of the user's head via the head movement tracker 109. In an example configuration, the deviation angle threshold is between approximately 3° and 10°. As used herein, the term "approximately" when referring to an angle means ±10% from the stated amount.

[0045] Changes along the horizontal axis slide three-dimensional objects (such as characters, Bitmojis, app icons, etc.) into and out of view by, for example, hiding, unhiding, or otherwise adjusting the visibility of the three-dimensional objects. Changes along the vertical axis, for example, when the user looks up, in one example, display weather information, time of day, date, calendar appointments, etc. In another example, when the user looks down on the vertical axis, the AR-enabled eyewear device 100 can be powered off.

[0046] like Figure 1B As shown, the right temple 110A includes a temple body 211 configured to accommodate a temple cap. Figure 1B Disposed within the right temple 110A are various interconnected circuit boards, such as a PCB or flexible PCB 140A, including circuitry for a first visible light camera 114A, a microphone 130, a speaker 132, a low power wireless circuit system (e.g., for communicating via a Wireless short-range network communications) and high-speed wireless circuit systems (e.g., for A controller circuit for wireless LAN communication).

[0047] A first visible light camera 114A is coupled to or disposed on the flexible PCB 140A and is overlapped by a visible light camera cover lens, which is aimed through an opening formed in the right temple 110A. In some examples, the frame 105 connected to the right temple 110A includes an opening for the visible light camera cover lens. The frame 105 may include a front-facing side configured to face outward, away from the user's eye. The opening for the visible light camera cover lens may be formed in and through the front-facing side. In this example, the first visible light camera 114A has an outward-facing field of view 111A in the line of sight or perspective of the user's right eye supporting the AR eyewear device 100. The visible light camera cover lens may also be adhered to the outward-facing surface of the right temple 110A, with the opening formed with an outward-facing overlap angle, but in a different outward direction. The coupling may also be indirect via an intermediate component.

[0048] First visible light camera 114A may be coupled to first see-through image display 180C of first optical assembly 180A to generate a first background scene for a first continuously displayed image. Second visible light camera 114B may be coupled to second see-through image display 180D of second optical assembly 180B to generate a second background scene for a second continuously displayed image. The first and second background scenes may partially overlap to present a three-dimensional observable area for the continuously displayed image.

[0049] Flexible PCB 140A may be disposed within right temple 110A and coupled to one or more other components housed within right temple 110A. Although shown as being formed on circuit board 140A of right temple 110A, first visible light camera 114A may be formed on another circuit board (not shown) within left temple 110B, articulated arm 125A, articulated arm 125B, or frame 105.

[0050] Figure 2A is a diagram depicting a rear view of an example hardware configuration supporting the AR eyewear device 100. Figure 2A As shown, the AR eyewear device 100 is in a form configured to be worn by a user. Figure 2A The example of the AR eyewear device 100 is glasses. The AR eyewear device 100 may take other forms and may be combined with other types of frames, such as a headgear, headphones, or a helmet.

[0051] In the glasses example, the AR eyewear-enabled device 100 includes a frame 105 including a right rim 107A connected to a left rim 107B via a nose bridge 106, the nose bridge 106 being configured to accommodate the user's nose. The right and left rims 107A and 107B include respective rings 175A and 175B that hold respective optical elements 180A and 180B, such as lenses and see-through displays 180C and 180D. As used herein, the term lens is intended to encompass a transparent or translucent piece of glass or plastic with curved and flat surfaces that cause light to converge / diverge or cause little or no convergence / divergence.

[0052] Although shown as having two optical elements 180A and 180B, the AR-enabled eyewear device 100 may include other arrangements, such as a single optical element, depending on the application or intended user of the AR-enabled eyewear device 100. As further shown, the AR-enabled eyewear device 100 includes a right temple 110A adjacent to the right lateral side 170A of the frame 105 and a left temple 110B adjacent to the left lateral side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective lateral sides 170A and 170B (as shown), or implemented as separate components attached to the frame 105 on the respective lateral sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0053] exist Figure 2A In an example, an eye scanner 113 is provided that includes an infrared emitter 115 and an infrared camera 120. Visible light cameras typically include a blue light filter to block infrared light detection. In one example, the infrared camera 120 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640x 480 pixels, a total of 0.3 megapixels), with the blue filter removed. The infrared emitter 115 and the infrared camera 120 can be co-located on the frame 105. For example, both are shown as being connected to the upper portion of the left rim 107B. The frame 105 or one or more of the temples 110A and 110B can include a circuit board (not shown) that includes the infrared emitter 115 and the infrared camera 120. For example, the infrared emitter 115 and the infrared camera 120 can be connected to the circuit board by soldering.

[0054] Other arrangements of the infrared emitter 115 and the infrared camera 120 may be implemented, including arrangements where both the infrared emitter 115 and the infrared camera 120 are located on the right rim 107A or at different locations on the frame 105. For example, the infrared emitter 115 may be located on the left rim 107B, and the infrared camera 120 may be located on the right rim 107A. In another example, the infrared emitter 115 may be located on the frame 105, and the infrared camera 120 may be located on one of the temples 110A or 110B, or vice versa. The infrared emitter 115 may be connected to substantially any location on the frame 105, the right temple 110A, or the left temple 110B to emit the infrared light pattern. Similarly, the infrared camera 120 may be connected to substantially any location on the frame 105, the right temple 110A, or the left temple 110B to capture at least one reflection change in the infrared light emission pattern.

[0055] The infrared emitter 115 and the infrared camera 120 can be arranged to face inwardly toward the user's eyes, with a partial or full field of view of the eyes, to identify the corresponding eye position and gaze direction. For example, the infrared emitter 115 and the infrared camera 120 can be positioned directly in front of the eyes, on the upper portion of the frame 105, or in the temples 110A or 110B at both ends of the frame 105.

[0056] Figure 2B is a diagram depicting a rear view of another example hardware configuration of an AR-enabled eyewear device 200. In this example configuration, the AR-enabled eyewear device 200 is depicted as including an eye scanner 213 on the right temple 210A. As shown, an infrared emitter 215 and an infrared camera 220 are co-located on the right temple 210A. The eye scanner 213 or one or more components of the eye scanner 213 can be located on the left temple 210B and other locations of the AR-enabled eyewear device 200, such as the frame 105. The infrared emitter 215 and the infrared camera 220 are similar to Figure 2A The infrared emitter and infrared camera in FIG, but the eye scanner 213 can be changed to be sensitive to different wavelengths of light, as previously described in Figure 2A Similar to Figure 2A , Figure 2B The AR-enabled eyewear device 200 includes a frame 105 including a right rim 107A connected to a left rim 107B via a nosepiece 106. The rims 107A-B may include respective rings that hold respective optical elements 180A and 180B including see-through displays 180C and 180D.

[0057] Figure 2C and Figure 2D is a diagram depicting a rear view of an example hardware configuration of an AR-enabled eyewear device 100 including two different types of see-through image displays 180C and 180D. In one example, these see-through image displays 180C and 180D of optical assemblies 180A and 180B include integrated image displays. Figure 2C As shown, optical assemblies 180A and 180B include display matrices 180C and 180D of any suitable type, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a waveguide display, or any other such display.

[0058] Optical assemblies 180A and 180B also include one or more optical layers 176A-N, which may include any combination of lenses, optical coatings, prisms, reflectors, waveguides, optical strips, and other optical components. Optical layer 176 may include prisms of suitable size and configuration, including a first surface for receiving light from the display matrix and a second surface for emitting light toward the user's eye. The prisms of optical layer 176 may extend over all or at least a portion of the corresponding bezels 175A and 175B formed in rims 107A and 107B, allowing the user to see the second surface of the prisms when the user's eye is looking through the corresponding rims 107A and 107B. The first surface of the prisms of optical layer 176 faces upward from frame 105, and the display matrix overlays the prisms so that photons and light emitted by the display matrix strike the first surface. The prisms are sized and shaped so that light is refracted within the prisms and directed toward the user's eye through the second surface of the prisms of optical layer 176. In this regard, the second surface of the prisms of optical layer 176 can be convex to direct light toward the center of the eye. The prisms are sized and shaped to magnify the images projected by see-through image displays 180C and 180D, and the light travels through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from see-through image displays 180C and 180D.

[0059] In another example, the see-through image displays 180C and 180D of the optical assemblies 180A and 180B may include: Figure 2D The projected image display shown. The optical assemblies 180A and 180B include a projector 150, which can be a three-color projector using a scanning mirror, a galvanometer, a laser projector, or other type of projector. During operation, a light source such as the projector 150 is disposed in or on one of the temples 110A or 110B that supports the AR eyewear device 100. The optical assemblies 180A and 180B may include one or more light strips 155A-N that are spaced across the width of the lens of the optical assemblies 180A and 180B or across the depth of the lens between the front and back surfaces of the lens.

[0060] As photons projected by projector 150 travel through the lenses of optical assemblies 180A and 180B, the photons encounter light strips 155. When a particular photon encounters a particular light strip, the photon is either redirected to the user's eye or passed to the next light strip. A combination of modulation of projector 150 and modulation of the light strips can control specific photons or light beams. In one example, the processor controls the light strips 155 by initiating mechanical, acoustic, or electromagnetic signals. Although shown as having two optical assemblies 180A and 180B, the AR eye-mounted device 100 can include other arrangements, such as a single or three optical assemblies, or the optical assemblies 180A and 180B can have different arrangements depending on the application or intended user of the AR eye-mounted device 100.

[0061] like Figure 2C and Figure 2D As further shown, the AR-enabled eyewear device 100 includes a right temple 110A adjacent to the right lateral side 170A of the frame 105 and a left temple 110B adjacent to the left lateral side 170B of the frame 105. The temples 110A and 110B can be integrated into the frame 105 on the respective lateral sides 170A and 170B (as shown), or implemented as separate components attached to the frame 105 on the respective lateral sides 170A and 170B. Alternatively, the temples 110A and 110B can be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0062] In one example, the see-through image display includes a first see-through image display 180C and a second see-through image display 180D. The AR eyewear-supporting device 100 may include first and second lens rings 175A and 175B, which hold corresponding first and second optical components 180A and 180B. The first optical component 180A may include a first see-through image display 180C (e.g., a display matrix or a light strip and projector in the right temple 110A). The second optical component 180B may include a second see-through image display 180D (e.g., a display matrix or a light strip and projector 150 in the right temple 110A). The continuous field of view of the continuous display image may include a viewing angle of between approximately 15° and 30° measured horizontally, vertically, or diagonally, and more specifically 24°. The continuous display image with a continuous field of view represents a combined three-dimensional observable area visible by stitching together the two display images presented on the first and second image displays.

[0063] As used herein, “angle of view” describes the angular range of the field of view (FOV) associated with the displayed image presented on each of the image displays 180C and 180D of the optical assemblies 180A and 180B. The “angle of overlap” describes the angular range or FOV that the lens of the visible light camera 114A or 114B or the infrared camera 220 can image. Typically, the image circle produced by the lens is large enough to completely cover the film or sensor, possibly including some vignetting (i.e., the brightness or saturation of the image decreases towards the periphery compared to the center of the image). If the overlap angle of the lens does not fill the sensor, the image circle will be visible, typically with strong vignetting towards the edges, and the effective angle of view will be limited by the overlap angle. The FOV is intended to describe the field of view of the observable area that a user of the AR eyewear-enabled device 100 can see through his or her eyes via the displayed images presented on the image displays 180C and 180D of the optical assemblies 180A and 180B. Image display 180C of optical assemblies 180A and 180B may have a FOV with an overlap angle between 15° and 30° (eg, 24°) and a resolution of 480 x 480 pixels (or greater; eg, 720p, 1080p, 4K, or 8K).

[0064] Figure 3 The block diagram in FIG. 1 shows an example of capturing visible light with cameras 114A and 114B. Visible light is captured by first visible light camera 114A having a circular FOV 111A. The selected rectangular first raw image 302A is used by image processor 412 ( Figure 4 ) for image processing. Visible light is also captured by second visible light camera 114B having circular FOV 111B. Rectangular second raw image 302B selected by image processor 412 is used for image processing by processor 412. Raw images 302A and 302B have overlapping FOV 304. Processor 412 processes raw images 302A and 302B and generates a three-dimensional image 306 for display on displays 180C and 180D. Three-dimensional image 306 is also referred to as an immersive image hereinafter.

[0065] Figure 4 The system block diagram in FIG. 4 shows a high-level functional block diagram including example electronic components provided in an example configuration in an AR-enabled eyewear device 100 or 200. The electronic components shown include a processor 432, a memory 434, and see-through image displays 180C and 180D.

[0066] The memory 434 includes instructions for execution by the processor 432 to implement functions that support the AR eyewear devices 100 and 200, including instructions for the high-speed processor 432 to control the image 306. Such functions can be implemented by processing instructions of the eye movement tracking program 445 and gesture detection / object tracking software 470 stored in the memory 434 and executed by the high-speed processor 432. Figure 8 As described, the gesture detection / object tracking software 470 may include depth determination software 472, object tracking software 474, object detection software 476, and hand / gesture detection software 478 to support calibration and intersection determination of the AR eye-wear device 100, as well as head tracking and hand / gesture detection performed by the AR eye-wear device 100.

[0067] The high-speed processor 432 receives power from the battery 450 and executes instructions stored in the memory 434. The memory 434 can be a separate component, or the memory 434 can be integrated "on-chip" with the processor 432 to perform functions supporting the AR eyewear devices 100 and 200 and communicate with external devices via wireless connections.

[0068] AR eyewear devices 100 and 200 may be coupled with an eye movement tracking program 445 (e.g., using Figure 2B 4 and 5. The user interface of the device 480 may be implemented as an infrared emitter 215 and an infrared camera 220 in the AR eyewear device 100 or 200, and may provide user interface adjustments via a mobile device 480 and a server system 498 connected via various networks. The mobile device 480 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the AR eyewear-enabled device 100 or 200 using both a low-power wireless connection 425 and a high-speed wireless connection 437. The mobile device 480 is also connected to the server system 498 via a network 495. The network 495 may include any combination of wired and wireless connections.

[0069] The AR eyewear-supporting devices 100 and 200 may include an image display driver 442 , an image processor 412 , a low-power circuit system 420 , and a high-speed circuit system 430 . Figure 4The components supporting the AR eyewear devices 100 and 200 shown in FIG are located on one or more circuit boards, such as PCBs or flexible PCBs 140A and 140B, in the respective temples 110A and 110B. Alternatively or additionally, the depicted components may be located in the temples, frames, hinges, articulated arms, or nose bridges supporting the AR eyewear devices 100 and 200. The visible light cameras 114A and 114B may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, lenses, or any other corresponding visible light or light capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0070] The eye movement tracking program 445 implements user interface FOV adjustment instructions, including instructions for causing the AR eyewear-enabled device 100 or 200 to track eye movements of a user's eyes of the AR eyewear-enabled device 100 or 200 via the eye movement tracker 213. Other implemented instructions (functions) cause the AR eyewear-enabled device 100 or 200 to determine FOV adjustments to the initial FOV 111A-B based on detected eye movements of the user corresponding to successive eye directions. Further implemented instructions generate successive display images of a display image sequence based on the FOV adjustments. The successive display images are generated as visible output to the user via the user interface. The visible output appears on the see-through image displays 180C and 180D of the optical assemblies 180A and 180B, which are driven by the image display driver 442 to present a display image sequence, including an initial display image having an initial FOV and successive display images having successive FOVs.

[0071] For example, the object tracking model applied by the gesture detection / object tracking software 470 can detect the user's hand gestures as well as objects in the environment, which will be recognized by the device or server-based object recognition software associated with the AR eyewear-enabled device 100 or 200 in the example configuration.

[0072] like Figure 4As shown, the high-speed circuitry 430 includes a high-speed processor 432, memory 434, and high-speed wireless circuitry 436. In this example, an image display driver 442 is coupled to the high-speed circuitry 430 and operated by the high-speed processor 432 to drive the image displays 180C and 180D of the optical assemblies 180A and 180B. The high-speed processor 432 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required to support the AR eyewear device 100 or 200. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission over a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuitry 436. In some examples, the high-speed processor 432 executes an operating system such as the Linux operating system or other such operating system that supports the AR eyewear device 100 or 200, and the operating system is stored in the memory 434 for execution. In addition to any other responsibilities, the high-speed processor 432, which executes the software architecture that supports the AR eyewear device 100 or 200, is used to manage data transmission with the high-speed wireless circuitry 436. In some examples, the high-speed wireless circuitry 436 is configured to implement a wireless communication protocol, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as In other examples, other high-speed communication standards may be implemented by high-speed wireless circuitry 436 .

[0073] The low-power wireless circuitry 424 and high-speed wireless circuitry 436 that support the AR eyewear devices 100 and 200 may include short-range transceivers. and wireless WAN, LAN, or WAN transceivers (e.g., cellular or ). Mobile device 480, including a transceiver that communicates via low-power wireless connection 425 and high-speed wireless connection 437, can be implemented using details of the architecture supporting AR eyewear devices 100 and 200, as can other elements of network 495.

[0074] Memory 434 comprises any storage device capable of storing various data and applications, including, among other things, color maps, camera data generated by visible light cameras 114A-B and image processor 412, and images generated for display by image display driver 442 on see-through image displays 180C and 180D of optical assemblies 180A and 180B. While memory 434 is shown as being integrated with high-speed circuitry 430, in other examples, memory 434 may be a separate, standalone component supporting AR eyewear device 100 or 200. In some such examples, circuitry may provide a connection from image processor 412 or low-power processor 422 to memory 434 via a system-on-chip (SoC) including high-speed processor 432. In other examples, high-speed processor 432 may manage addressing of memory 434 so that low-power processor 422 will initiate high-speed processor 432 whenever a read or write operation involving memory 434 is required.

[0075] The server system 498 can be one or more computing devices as part of a service or network computing system, for example, including a processor, memory, and a network communication interface to communicate with the mobile device 480 and the AR eyewear-enabled devices 100 and 200 via the network 495. The AR eyewear-enabled devices 100 and 200 can be connected to a host computer. For example, the AR eyewear-enabled device 100 or 200 can be paired with the mobile device 480 via a high-speed wireless connection 437, or connected to the server system 498 via the network 495. In addition, a gallery 490 of snapshots and AR objects can be maintained by the server system 498 for each user and called by communications that provide links to the snapshots and AR objects stored in the gallery 490.

[0076] The output components supporting the AR eyewear devices 100 and 200 include visual components such as Figure 2C and Figure 2DThe image displays 180C and 180D of the optical components 180A and 180B are driven by an image display driver 442 (e.g., a liquid crystal display (LCD), a plasma display panel (PDP), a light emitting diode (LED) display, a projector, or a waveguide). The image displays 180C and 180D of the optical components 180A and 180B are driven by an image display driver 442. The output components supporting the AR eyewear devices 100 and 200 may also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), and other signal generators. The input components supporting the AR eyewear devices 100 and 200, the mobile device 480, and the server system 498 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical keyboard, or other alphanumeric input components), a point-based input component (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing tool), a tactile input component (e.g., a physical button, a touch screen or other tactile input component that provides the position and force of a touch or a touch hand gesture), and an audio input component (e.g., a microphone).

[0077] The AR-enabled eyewear devices 100 and 200 may include additional peripheral elements, such as ambient light and spectral sensors, biometric sensors, thermal sensors 440, or other display elements integrated with the AR-enabled eyewear devices 100 or 200. For example, the peripheral elements may include any I / O components, including output components, motion components, position components, or any other such components described herein. The AR-enabled eyewear devices 100 and 200 may take other forms and may be combined with other types of frameworks, such as headgear, headphones, or helmets.

[0078] For example, the biometric components supporting the AR eyewear devices 100 and 200 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition). The motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, and rotation sensor components (e.g., gyroscopes). The position components include position sensor components (e.g., global positioning system (GPS) receiver components) for generating position coordinates, and positioning system coordinates for generating a position sensor component. or transceiver, altitude sensor components (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), and orientation sensor components (e.g., a magnetometer), etc. Such positioning system coordinates can also be received from mobile device 480 via low-power wireless circuitry 424 or high-speed wireless circuitry 436 over wireless connections 425 and 437.

[0079] The techniques described herein may also be used with one or more of the computer systems described herein, or with one or more other systems. For example, the various processes described herein may be implemented using hardware or software, or a combination of both. For example, at least one of the processors, memories, storage devices, output devices, input devices, or communication connections discussed herein may each be at least a portion of one or more hardware components. Dedicated hardware logic components may be constructed to implement at least a portion of one or more of the techniques described herein. For example, and without limitation, such hardware logic components may include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), and the like. Applications of the various devices and systems may include a wide range of electronic and computer systems. The techniques may be implemented using two or more specific interconnected hardware modules or devices, wherein relevant control and data signals may be communicated between and through the modules, or as part of an ASIC. Furthermore, the techniques described herein may be implemented using a software program executable by a computer system. For example, implementations may include distributed processing, component / object distributed processing, and parallel processing. Furthermore, virtual computer system processing may be constructed to implement one or more of the techniques or functions described herein.

[0080] As described herein, examples may include or may run on a processor, logic, or multiple components, modules, or mechanisms (referred to herein as "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in some manner. In one example, circuits may be arranged as modules in a specified manner (e.g., internally or relative to external entities, such as other circuits). In one example, all or part of one or more computer systems (e.g., stand-alone, client, or server computer systems) or one or more hardware processors may be configured as a module via firmware or software (e.g., instructions, application components, or applications) that operates to perform the specified operation. In one example, the software may reside on a machine-readable medium. When executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.

[0081] Thus, the term "module" is understood to include at least one of a tangible hardware or software entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a particular manner or perform part or all of any of the operations described herein. Considering examples where modules are temporarily configured, each of the modules need not be instantiated at any one time. For example, where the module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured as corresponding different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one time and to constitute different modules at different times.

[0082] In an example configuration, the processes described herein may be implemented by instructions stored in the memory 434 of the AR eyewear-enabled device 100 or 200. The memory 434 may include a machine-readable medium having stored thereon one or more sets of data structures or instructions (e.g., software) embodying or used by any one or more of the techniques or functions described herein. During execution of the instructions by the AR eyewear-enabled device 100, the instructions may also reside completely or at least partially within the high-speed processor 432 or the low-power processor 422. In an example, one or any combination of the hardware processors 432 and 422 and the memory 434 constitute a machine-readable medium.

[0083] As used herein, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database or associated cache and server) configured to store instructions for implementing the processes described herein. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions executed by the processors 432 and 422 and enabling the AR eyewear device 100 or 200 to perform any one or more of the techniques disclosed herein, or any medium capable of storing, encoding, or carrying data structures used by or associated with these instructions. Non-limiting examples of machine-readable media may include solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media may include non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSD); and CD-ROM and digital video disc (DVD)-ROM disks. In some examples, the machine-readable medium may include non-transitory machine-readable media. In some examples, machine-readable media may include machine-readable media that is not a transitory propagating signal.

[0084] Instructions may also be sent or received via wireless connection 425 or 437 or directly via the Internet 495. The AR eyewear-enabled devices 100 and 200 may communicate with one or more other AR eyewear-enabled devices 100 or 200 or mobile device 480 using any of a variety of transport protocols (e.g., frame relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Example communication networks may include, among others, a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile phone network (e.g., a cellular network), a plain old telephone service (POTS) network, and a wireless data network (e.g., known as a cellular network). The high-speed wireless circuitry 436 and / or the low-power wireless circuitry 424 may include multiple antennas to communicate wirelessly using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. In some examples, wireless circuitry 424 and 436 may communicate wirelessly using multi-user MIMO technology.

[0085] The features and flow charts described herein may be embodied as method steps in one or more methods, or in one or more applications as described above. According to some configurations, one or more "applications" are programs that perform the functions defined in the program. Various programming languages ​​may be used to generate one or more of the applications structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In one particular example, a third-party application (e.g., an application created by an entity other than a particular platform vendor using an ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM 、 The present invention relates to a mobile software running on a mobile operating system such as an iPhone or other mobile operating system. In this example, the third-party application can call an API (application programming interface) provided by the operating system to facilitate the functions described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in dedicated computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0086] The programmatic aspects of the technology may be considered a "product" or "article of manufacture," typically in the form of at least one of executable code or associated data, carried or embodied in a machine-readable medium. For example, the programming code may include code for a touch sensor or other functionality described herein. "Storage" type media includes any or all tangible memory of a computer, processor, etc., or its associated modules, such as various semiconductor memories, tape drives, and disk drives, which can provide non-transitory storage for software programming at any time. All or part of the software may sometimes be communicated over the Internet or various other telecommunications networks. For example, such communication can enable the software to be loaded from one computer or processor to another. Thus, another type of media that can carry programming, media content, or metadata files includes optical, electrical, and electromagnetic waves, such as those used at physical interfaces between local devices over wired and fiber optic landline networks and over various airlinks. The physical elements that carry such waves, such as wired or wireless links, or optical links, etc., can also be considered media that carry software. As used herein, unless restricted to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions or data to a processor for execution.

[0087] Thus, machine-readable media can take the form of various forms of tangible storage media. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer or the like, such as can be used to implement the client devices, media gateways, transcoders, and the like shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wire, and optical fiber, including the wires that make up a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer-readable media include, for example, floppy disks, diskettes, hard disks, magnetic tape, any other magnetic medium, CD-ROMs, DVDs or DVD-ROMs, any other optical medium, punched card tape, any other physical storage medium with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chip or cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read at least one of programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0088] A software application for using an AR-enabled wearable electronic device (such as smart glasses) as an IoT remote control will be described with reference to Figures 5-8, where a user can control a pointer on a display screen of the IoT-enabled device to select items by viewing the items and selecting using gestures.

[0089] The techniques described herein with respect to certain examples utilize a camera frame function (such as SPECTACLES available from Snap Inc. of Santa Monica, California) that is compatible with the AR eyewear device 100. TM ) provided by the IMU 109 in combination with the AR eyewear 100 to control a cursor on a TV or computer screen by performing head movements or gestures. A user of the AR eyewear 100 can look at the item she wants to select to move the cursor to it. The hand tracking functionality of the AR eyewear 100 also enables the user to perform a wide variety of actions using different hand gestures. The gestures are converted into actions using a pre-existing gesture recognition framework and ML model that supports the AR eyewear 100. The IMU 109 can also be used to collect head motion data to detect certain head gestures (such as tilting the head sideways or left / right) to, for example, indicate a desire to switch channels or go to the next / previous video.

[0090] Hand / head gestures can be divided into two categories:

[0091] 1. Global gestures for actions such as "Go to home", "Turn off TV", "Launch selected service", "Go to next", "Go to previous", etc.

[0092] 2. Gestures for performing actions on selected items (such as "Preview", "Select", "Show Rating", "More Information", etc.).

[0093] The AR-enabled eyewear device 100 described herein may therefore provide a manufacturer-independent remote control device and application programming interface (API) that can be adapted by television manufacturers and third-party smart TV and computer monitor application developers.

[0094] Figure 5A is a schematic diagram of a smart TV display 500 in an example configuration having a cursor 502 displayed thereon as an aligned overlay object to be operated by an AR-enabled eyewear device 100. Figure 5B As shown, a hand gesture 504 can be recognized by the AR eyewear-enabled device 100 to, for example, trigger a contextual information tooltip 506 that displays the IMDB rating and recent user ratings / reviews of the item pointed to by the cursor 502. Similarly, the contextual information tooltip 506 can be a menu for navigating to related items of interest. The contextual item tooltip 506 can be displayed on the smart TV display 500 or can be overlaid on the display 180C of the AR eyewear-enabled device 100 as augmented content provided by the smart TV or server 498, as the case may be.

[0095] So-called smart TVs include Internet capabilities and are typically connected via cable or Connected to a secure, low-latency local area network (LAN). The systems and methods described herein provide an application programming interface (API) exposed by the AR eyewear-enabled device 100 to devices on the same private LAN subnet, which can be used to exchange information such as cursor position, detected hand / head gestures (=actions), etc. A software development kit (SDK) is provided that can be used by clients (e.g., the AR eyewear-enabled device 100 and the smart TV display 500) to simplify communication with the API. In an example configuration, the protocol used for the API can be Hypertext Transfer Protocol (HTTP), Representational State Transfer (REST) ​​API, and websockets, or a more lightweight protocol for IoT applications (such as Message Queuing Telemetry Transport (MQTT)), which also allows streaming media to efficiently synchronize cursor position. A pairing flow (part of the API) enables the smart TV communication application 510 to pair with the AR eyewear-enabled device 100 to establish a connection. In an example configuration, the smart TV communication application 510 can be an application or library suitable for pairing with one or more AR eyewear-enabled devices 100 to communicate events between them. Actions and events can be registered to simplify integration. The smart TV communication application 510 can use the SDK to initiate pairing. The SDK first sends a broadcast internally for the AR eyewear-supported device 100 to respond to, so that the smart TV communication application 510 can obtain the private Internet Protocol (IP) address of the AR eyewear-supported device 100, and then sends a connection request to the AR eyewear-supported device 100.

[0096] Once the Smart TV communication application 510 and the AR eyewear-enabled device 100 are paired, the respective devices can use an API (via an SDK) to register callbacks and subscribe to detected actions (hand / head gestures) and changes in cursor position. In the example configuration, a predefined set of hand / head gestures is supported. Once a known gesture is detected, the detected gesture and / or corresponding action is communicated to the Smart TV communication application 510, which has registered a callback (HTTP REST endpoint or, in the case of MQTT, subscribed to the action). In the example configuration, each gesture / action has a documented identifier (gesture ID), and the respective device can independently determine how to use certain gestures and / or actions. For example, a hand gesture displaying a "thumbs up" can be assigned a gesture ID that is recognized by the Smart TV display 500 as indicating that the volume on the Smart TV should be turned up. Similarly, a head tilt can be recognized by the AR eyewear-enabled device 100 and assigned a gesture ID that is recognized by the Smart TV display 500 as a request to change the channel up or down. Thus, a volume control signal (increase volume) or a channel change signal can be sent to the smart TV communication application 510 for use by the internal processor of the smart TV display 500, thereby increasing the volume or changing the channel through an action recognized by the AR eye-wear-enabled device 100.

[0097] During pairing of the Smart TV communication application 510 and the AR eyewear-enabled device 100 or in the device settings of the Smart TV display 500, the user can initiate a calibration process (part of the API) to calibrate the gyroscope sensor and IMU data from the AR eyewear-enabled device 100 to a predetermined portion (e.g., the center) of the Smart TV display 500. To perform such calibration, the user is required to look at the Smart TV display 500, which includes three detectable codes in its corners for aligning the AR eyewear-enabled device 100 with the Smart TV display 500 in real-world coordinates.

[0098] For example, Figure 61 is a diagram illustrating an example calibration screen 600 on a smart TV display 500 in an example configuration for calibrating a user's AR eyewear-enabled device 100 to the smart TV display 500. As shown, Quick Response (QR) codes 610 are displayed at three corners of the smart TV display 500. The QR codes 610 are detected, and the depth data (depth map) is used by the existing depth service of the AR eyewear-enabled device 100 to obtain the 3D coordinate position of the detected QR code 610 from the camera frame data of the AR eyewear-enabled device 100. Using these 3D coordinate positions, the real-world coordinates of the TV screen rectangle (plane) can be determined. By using a six-degree-of-freedom (6DOF) tracker (which uses IMU data combined with the camera frame of the AR eyewear-enabled device 100), the screen rectangle of the smart TV display 500 can be tracked by the AR eyewear-enabled device 100 relative to the user's position. For example, the IMU data can be used to detect the rotation and / or tilt of the head of the user of the AR eyewear-enabled device 100 relative to the real-world coordinates of the rectangle of the smart TV display 500.

[0099] Alternatively, the QR code 610 can be communicated by the AR eyewear-enabled device 100 (e.g., by using Chromecast or otherwise communicating to the smart TV communication application 510 via an API between the smart TV display 500 and the AR eyewear-enabled device 100) to the smart TV display 500 for display in a corner of the smart TV display 500, thereby facilitating the calibration process.

[0100] Object detection can also be used to recalibrate the position of the Smart TV display 500. For example, object tracking services (e.g., simultaneous localization and mapping (SLAM) services) may cause drift (sensor drift) in the tracked position (the tracked object position relative to the real-world physical object position) over time. To correct for such sensor drift, the tracked object position can be calibrated at regular intervals to minimize sensor drift by visually detecting the Smart TV display 500 using the existing object detection framework / infrastructure of the gesture detection / object tracking software 470. The object detection framework can process the camera input stream (frames), detect the Smart TV display 500, and return a bounding box of the Smart TV display 500 in the camera frame. A depth service can be used to obtain a depth map (at time x) of the detected bounding box in the camera frame. Combining this information allows the current calibrated screen rectangle of the Smart TV display 500 to be calculated in real-world coordinates. Any drift in the position of the cursor 502 over time due to the use of IMU data and SLAM service data can be adjusted through such recalibration. It should be understood that the adjustment data can be Chromecast to the TV display or otherwise communicated to the smart TV communication application 510 via an API between the smart TV display 500 and the AR eyewear-enabled device 100.

[0101] In an example configuration, once the AR enabled eyewear device 100 is calibrated relative to the smart TV display 500, the position of the cursor 502 on the smart TV display 500 can be determined by finding the intersection between the screen rectangle of the smart TV display 500 (the tracked 3D position) and an orthogonal line originating from the center of the FOV of the AR enabled eyewear device 100 pointing toward the smart TV display 500 using a ray casting technique. For example, Figure 7 A ray cast of an orthogonal line 700 from the AR-enabled eyewear device 100 to the tracked 3D position of the screen rectangle 710 is shown to identify an intersection position 720 on the smart TV display 500 that is being viewed by the user's AR-enabled eyewear device 100 at any given time for placing the cursor 502 on the smart TV display 500. The intersection position 720 can be solved by a regular linear algebraic equation well known to those skilled in the art. In an example configuration, the infrastructure provided by LensStudio (available from Snap, Inc. of Santa Monica, California) can be reused to calculate the intersection position 720 of the cursor 502.

[0102] For those cases where the intersection position equation has no solution (e.g., the user is not looking at the smart TV display 500 or is looking across the smart TV display 500, or the user is next to or behind the smart TV display 500), a default position can be sent to the smart TV's smart communication application 510, and / or an appropriate message can be sent to each of the clients connected to the communication API.

[0103] Figure 8 is a flow chart of a method for controlling a cursor 502 on a smart TV display 500, a computer display, or other IoT device display in an example configuration. As shown, the smart TV display 500 can be adapted by the SDK to include a communication application 510 to facilitate communication with the AR eyewear-enabled device 100 via an API to exchange information such as cursor position, detected hand / head gestures, etc. Similarly, the AR eyewear-enabled device 100 can be adapted to include an AR remote control application 800 developed using the SDK.

[0104] During operation, when the AR remote control application 800 is launched at 810, pairing with the AR eyewear-enabled device 100 can be initiated by the communication application 510. The communication application 510 initiates pairing and subscribes to dynamic events by sending a request using, for example, low-latency, low-payload MQTT at 812. At 820, the AR remote control application 800 pairs with the smart TV display 500 and registers for events such as detecting a hand gesture.

[0105] The calibration process 830 is initiated at 822. For example, a reference Figure 6 As described above, refer to Figure 6 As described, the calibration process 830 may require depth determination and / or object detection and tracking. In an example configuration, depth determination software 472, object tracking software 474, and object detection software 476 of the gesture detection / object tracking software 470 supporting the AR eyewear device 100 may be called at 832 to support calibration calculations. For example, the object detection software 476 and the object tracking software 474 may implement a SLAM process for automatic recalibration instead of the QR code 610.

[0106] Once the AR eyewear device 100 has been calibrated at 830, the AR eyewear device 100 is calibrated at 840 using, for example, the method described above with reference to FIG. Figure 7The described ray casting technique calculates the intersection (collision) 720 of the FOV of the AR eyewear-supporting device 100 and the smart TV display 500. In an example configuration, the depth determination software 472, object tracking software 474, and object detection software 476 of the gesture detection / object tracking software 470 supporting the AR eyewear-supporting device 100 can be called at 842 to support the intersection calculation. The intersection determined at 840 is identified as the desired cursor position, and a cursor position update 852 is sent to all registered clients (e.g., the smart TV communication application 510) at 850. The position of the rectangle of the smart TV display 500 can be tracked in real-world coordinates using the SLAM service of the gesture detection / object tracking software 470 to determine such an intersection.

[0107] If a hand or head gesture is detected by the hand / gesture detection software 478 at 860, a gesture event 862 is sent to all registered clients (e.g., the smart TV communication application 510). In an example configuration, the gesture event 862 includes a gesture ID (e.g., thumbs up, head tilt, head rotation, etc.) recognized by the hand / gesture detection software 478. The gesture ID can be used by the registered clients to perform the input action that has been mapped to the gesture ID by the corresponding registered client (e.g., changing the station, volume control, selecting a rating / review, etc.).

[0108] Steps 840-860 are continuously repeated as long as the AR remote control application 810 is in operation. As described above, the AR eyewear-enabled device 100 can also be recalibrated from time to time to adjust for sensor offset by visually detecting the smart TV display 500 using the existing object detection framework / infrastructure of the gesture detection / object tracking software 470 and adjusting for sensor offset.

[0109] At 870 , the AR remote control application 800 is exited.

[0110] Thus, the AR remote control application 800 enables a user wearing an AR-enabled eyewear device 100 to present a cursor 502 on the Smart TV display 500 at the intersection 720 of orthogonal rays projected from the AR-enabled eyewear device 100 and the Smart TV display 500, and updates the cursor 502's position as the user moves their head and around the room. As long as the user's field of view intersects the Smart TV display 500, the intersection 720 is continuously tracked and updated to reflect the updated cursor position. The user can also perform gestures to make selections on the Smart TV display 500. In an example configuration, once a desired selection is made, the AR remote control application 800 can be exited, so that the cursor 502 does not interfere with viewing information on the Smart TV display 500 during use. A toggle key can be provided on the AR-enabled eyewear device 100 for turning the AR remote control application 800 on and off as needed. Alternatively, the AR remote control application 800 can remain active, but the cursor can be programmed to disappear a predetermined amount of time after a user selection.

[0111] The software applications described herein may also be adapted to enable AR-enabled wearable electronic devices (such as smart glasses) to extend a television screen with augmented reality content. Such software applications are described with reference to Figures 9-10.

[0112] As described above, the presence of a TV screen can be detected and the rectangular TV screen plane can be tracked in real-world coordinates. To provide augmented reality content to the TV screen, tracking information of the TV plane can be used to display the augmented reality content relative to the TV display screen.

[0113] As described above, applications on the smart TV can be adapted by the SDK to interact with the AR eyewear-enabled device 100 via the communication application 510 to facilitate communication with the AR eyewear-enabled device 100 through the API to exchange information such as cursor position, detected hand / head gestures, etc. The operating system of the smart TV can also be modified to integrate the SDK to utilize the API to provide users with an enhanced experience in the TV settings screen, home screen, etc. In addition to including an AR remote control application, the AR eyewear-enabled device 100 can also be adapted to display contextual user interfaces and augmented reality content based on user input, which is stitched onto the TV screen using the SDK. In such a configuration, the SDK empowers the TV and TV applications to trigger and display augmented reality content based on user input (such as hand gestures). For example, the SDK specifies when and where the selected augmented reality content is displayed.

[0114] As described above with reference to Figure 5, a contextual (augmented) menu next to the TV can be triggered by a hand gesture. Since the TV screen plane is being tracked, the menu can be seamlessly positioned and "stitched" to the TV display, thereby extending the TV display to include the menu. Similarly, the SDK can empower the TV and TV applications to trigger contextual AR visual effects. For example, Figure 9A As shown, the display screen 900 may include a preview screen that automatically (or in response to a gesture) triggers augmented reality content (e.g., in the form of a Ferris wheel) 910 to be displayed on the AR-enabled wearable electronic device 100 at one of several fixed positions relative to the television display screen 900. Figure 9B As shown, the augmented reality content in the form of a Ferris wheel 910 can be a two-dimensional or three-dimensional object anchored relative to an anchor position of the television display screen 900 for display on the AR-enabled wearable electronic device 100. The anchor position can be defined in a Cartesian coordinate system of the television display screen 900. For example, Figure 9B , augmented reality content in the form of Ferris wheel 910 is anchored to the lower left corner of television display screen 900 ((x, y) = (0, 0)), but can be positioned elsewhere on or relative to television display screen 900, even including locations outside of television display screen 900 (e.g., at positions where x>1 or x<0 and / or y>1 or y<0). Augmented reality content 910 can also be scaled relative to the height and width of television display screen 900 so that augmented reality content 910 is displayed proportionally. For example, the height of Ferris wheel 910 can be specified to be 30% of the height of television display screen 900 and at a specified proportional width of television display screen 900.

[0115] Figure 10 is a flow chart of a method for providing augmented reality content 910 to a smart TV display 500, a computer display, or other IoT device display in an example configuration. As shown, the smart TV display 500 can be adapted by the SDK to include a communication application 510 to facilitate communication with the AR-enabled eyewear device 100 via an API to exchange information such as detected hand / head gestures, augmented reality content, etc. Similarly, the AR-enabled eyewear device 100 can be adapted to include a TV augmented reality content application 1000 developed using the SDK.

[0116] During operation, when the TV augmented reality content application 1000 is launched at 1010, pairing with the AR eyewear-enabled device 100 can be initiated by the communication application 510. The communication application 510 initiates pairing and subscribes to dynamic events by sending a request using, for example, MQTT with low latency and low payload overhead at 1012. At 1020, the TV augmented reality content application 1000 is paired with the smart TV display 500 and registered for events such as detected hand gestures.

[0117] The calibration process 1030 is initiated at 1022. For example, a reference Figure 6 Alternatively, the calibration process 1030 may use a custom machine learning (ML) model that detects the television display screen. Figure 6 As described above, the calibration process 1030 may require depth determination and / or object detection and tracking. In an example configuration, depth determination software 472, object tracking software 474, and object detection software 476 of the gesture detection / object tracking software 470 supporting the AR eyewear device 100 may be called at 1032 to support calibration calculations. For example, the object detection software 476 and the object tracking software 474 may implement a SLAM process for automatic recalibration instead of the QR code 610.

[0118] Once the AR eyewear-enabled device 100 has been calibrated at 1030, the user can make a specific hand or head gesture detected by the AR glasses 100 at 1040. At 1042, the hand or head gesture is recognized by the hand / gesture detection module 478, and the gesture recognition data is provided to the AR eyewear-enabled device 100. A gesture event 1044 is sent to the connected communication application 510 of the smart TV display 500. If a hand or head gesture is detected by the hand / gesture detection software 478 at 1040, the gesture event 1044 is sent to all registered clients (e.g., the smart TV communication application 510). In an example configuration, the gesture event 1044 may include a gesture ID recognized by the hand / gesture detection software 478 (e.g., thumb up, head tilt, head rotation, etc.). The gesture ID can be used by the registered client to perform the input action that has been mapped to the gesture ID by the corresponding registered client (e.g., selecting a specified or pre-selected augmented reality (AR) or virtual reality (VR) overlay, navigating to another / next screen, hiding a single element or object that is currently augmented as part of the currently rendered augmented reality content, animating a character in the currently rendered augmented reality content, etc.). In general, the client (smart TV or TV application) can send any payload to the AR eyewear-enabled device 100, which can be received and interpreted to dynamically determine what to display on the display of the AR eyewear-enabled device 100.

[0119] In response, the communication application 510 sends augmented reality content at 1052, which has been pre-specified by the smart TV display 500 or selected by the recognized gesture. In an example configuration, the augmented reality content can be selected from Lens Studio or an augmented reality (AR) / virtual reality (VR) overlay (or "lens," not to be confused with a camera lens) selected from a pre-selected set of filters available to the smart TV display 500 and provided to the AR glasses 100. The application programming interface (API) of Lens Studio can provide the following capabilities: it positions an object (part of a filter sent from an IoT device (e.g., a smart TV) to the AR eyewear-enabled device 100) relative to the physical location of the smart TV display 500 in the field of view of the AR eyewear-enabled device 100. For example, when developing a filter in Lens Studio, a TV positioning component can be attached to each object within the filter to configure the position of the filter relative to the TV display 500, etc. For example, objects A, B, and C of a filter may be positioned whereby object A is positioned in the upper left corner of television display 500, object B is positioned in the center of television display 500, one meter in front of the television display screen on the z-axis, and object C is positioned in the center of the left edge of television display 500. However, not all objects that are part of such a filter need to use the API - only objects that are to be in a fixed position relative to television display 500 are required.

[0120] The augmented reality content provided at 1052 may optionally also include additional data in a custom payload. At 1050, the augmented reality content received from the communication application 510 is rendered to the television display device 500 at a location relative to the tracked television display data 1054. The location may be specified in the filter itself or may be the current location of the cursor.

[0121] In an example configuration, the filter may include a user interface (UI) menu that is pinned to a specific coordinate location (e.g., the left edge of the TV display 500). For example, a third party such as Netflix, when presenting a program such as Stranger Things, may use LensStudio to create a filter, send the filter to the AR eyewear-enabled device 100, and interact with the filter by sending messages back and forth. Figure 9A and 9BIn the example configuration, the filter is an animated 2D or 3D object (such as a 3D Ferris wheel 910) that can be positioned and anchored to the lower left corner of the TV screen for display on the display of the AR eyewear device 100. The augmented reality content 910 can be any 2D or 3D animated or non-animated element associated with (or independent of) the displayed content on the TV display 900, thereby enhancing the user's visual experience. The user can use gestures (e.g., head tilt) to animate the displayed content. For example, the movement of the 3D Ferris wheel 910 can be controlled by the user's head tilt. In the example configuration, the filter uses the tracked TV screen plane to render its content relative to the physical TV display 500. Because the filter is fixed in real-world coordinates relative to the TV display 500, the augmented reality content appears to be in a fixed position to the user even if the user walks around the room.

[0122] The communication application 510 of the smart TV display 500 can send data to the AR eyewear-enabled device 100 at any time to update the filter content at 1062, and the AR eyewear-enabled device 100 can update the filter with the newly received data at 1060. For example, a user can use hand gestures to scroll through augmented reality content in an augmented reality menu presented to the smart TV display 500. Similarly, the communication application 510 of the smart TV display 500 can send an instruction to the AR eyewear-enabled device to hide AR content at any time at 1072, and the AR eyewear-enabled device 100 can hide the AR content or specific elements of the rendered AR content by, for example, stopping filter rendering at 1070. For example, when the communication application 510 receives another hand gesture event, the communication application 510 can send a request to the AR eyewear-enabled device 100 to hide the rendered filter. The data update 1062 and the hide content instruction 1072 can include, for example, text and instructions in the MQTT format.

[0123] At 1080 , the TV augmented reality content application 1000 is exited.

[0124] In an example configuration, a filter collection (such as LensStudio) may include pre-built components or application programming interfaces (APIs) that allow creators of television content to select and position augmented reality objects and elements relative to the tracked television screen plane. The API may be used to position augmented reality objects and elements to the center of an edge, a corner, a specific preset (x, y) coordinate on the television screen plane (between 0-1, where on the x-axis, 0 is the left edge and 1 is the right edge, and where on the y-axis, 0 is the bottom edge and 1 is the top edge), a preset x, y offset (in defined world units, e.g., millimeters) relative to a point on the television display 500 (e.g., positioning an element one meter to the left of the top left corner of the television display 500 or some other location where x < 0 or x > 1 and / or y < 0 or y > 1), or at a coordinate z relative to the vertical vector of the television display (in defined world units, e.g., millimeters). Additionally, an anchor point on the (x, y, z) axis of an element between 0-1 may be specified. The API may also allow creators of television content to trigger the timing of the appearance of augmented reality objects based on the timing of the presentation of predetermined objects or frames to the television display.

[0125] In an example configuration, the size of the rendered augmented reality element can be adjusted to different display screens by configuring scaling options. For example, the rendered augmented reality element can be scaled to fit the screen size along the (x, y) axis, scaled to fit one dimension (x or y axis) with the other axis proportionally distorted or scaled along the (x, y, z) axis, or the size of the rendered augmented reality element can be set to a specific percentage of the screen plane dimensions (x, y, z).

[0126] like Figure 8 As shown in the example of , the AR eyewear-enabled device 100 can also be recalibrated from time to time to adjust for sensor offset by visually detecting the smart TV display 500 using the existing object detection framework / infrastructure of the gesture detection / object tracking software 470 and adjusting for sensor offset.

[0127] Thus, the TV augmented reality content application 1000 enables a user wearing an AR-enabled eyewear device 100 to overlay augmented reality content at a position relative to the smart TV display 500 that remains fixed as the user moves their head and as they move around the room. The user can also perform gestures to select filters for display on the smart TV display 500, or filters can be automatically triggered by interacting with certain display elements (e.g., cursor selection or interaction with a portion of the TV display or with an element displayed on the TV display). As with the cursor, a toggle key can be provided on the AR-enabled eyewear device 100 to turn the TV augmented reality content application 1000 on / off as needed. Alternatively, the TV augmented reality content application 1000 can remain active, but the augmented reality content can be programmed to disappear a predetermined amount of time after activation.

[0128] In other configurations, the user may perform a gesture that is recognized by the AR eyewear-enabled device 100 and provided to the smart TV display 500, as described above. However, rather than presenting the augmented data as an overlay from the smart TV communication application 510 or from a third-party server to the display of the AR eyewear-enabled device 100, the user may select a streaming application from the smart TV display 500, and the streaming application's server may send augmented data in the form of a menu or other display that can be navigated directly on the user's AR eyewear-enabled device 100. Any selection of augmented data presented on the AR eyewear-enabled device 100 may be communicated to the smart TV communication application 510 so that the desired selection can be made by the smart TV display 500.

[0129] Those skilled in the art will appreciate that the operations described herein are not limited to smart TVs. Any device that can be connected to a local area network and accepts control input (e.g., an IoT device) can be controlled by the AR eyewear device 100 using the techniques described herein.

[0130] The scope of protection is limited only by the claims that now follow. When interpreted in light of this specification and subsequent prosecution history, the scope is intended and should be interpreted to be broad consistent with the ordinary meaning of the language used in the claims and to encompass all structural and functional equivalents. Notwithstanding the foregoing, none of the claims are intended to encompass subject matter that does not satisfy the requirements of sections 101, 102, or 103 of the Patent Act, nor should they be construed in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0131] Except as stated immediately above, nothing stated or described is intended or should be construed as conferring upon the public any element, step, feature, object, benefit, advantage, or equivalent, whether or not recited in the claims.

[0132] It will be understood that the terms and expressions used herein have the ordinary meaning consistent with these terms and expressions relative to their corresponding respective investigations and research fields, unless otherwise specified herein with specific meanings. Relational terms such as first and second can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The term "comprises," "comprising," "includes," "including," or any other variant thereof is intended to encompass non-exclusive inclusion, so that the process, method, article, or device comprising or including a series of elements or steps not only includes these elements or steps, but can also include other elements or steps that are not explicitly listed or inherent to such process, method, article, or device. Without further limitation, an element starting with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0133] Unless otherwise indicated, all measurements, values, ratings, positions, quantities, dimensions, and other specifications set forth in this specification (including the following claims) are approximate and not exact. These quantities are intended to have a reasonable range consistent with the functions to which they relate and with customary practice in the art to which they pertain. For example, unless expressly stated otherwise, parameter values ​​or similar values ​​may vary from the stated quantity by as much as ±10%.

[0134] Furthermore, in the foregoing detailed description, various features are grouped together in various examples for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than expressly recited in each claim. Rather, as reflected in the following claims, protected subject matter lies in less than all features of any single disclosed example. Accordingly, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as separately claimed subject matter.

[0135] While the foregoing describes what is the best mode and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in a variety of forms and examples, and may be used in many applications, only some of which are described herein. The following claims are intended to claim all modifications and variations that fall within the true scope of the present concept.

Claims

1. An eyewear device adapted to present content to an Internet of Things (IoT)-enabled device, the IoT-enabled device having an IoT display, the eyewear device comprising: camera; a memory that stores instructions; Device display; as well as a processor coupled to the camera, the device display, and the memory, wherein the processor executes instructions to configure the eye-mounted device to: pairing the eyewear device with the IoT-enabled device to communicate via a communication interface therebetween; Calibrate the eyewear device to the real-world coordinate position of the IoT display; selecting augmented reality content to display on the device display relative to the real-world coordinate location of the IoT display; as well as The selected augmented reality content is rendered to the device display at a location relative to the real-world coordinate location of the IoT display.

2. The eyewear device of claim 1 , wherein the processor further executes instructions to configure the eyewear device to: detecting at least one of a hand gesture or a head gesture; and Sending gesture events to the supporting IoT device via the communication interface, in, The gesture event includes at least one gesture identification (ID) of at least one detected hand gesture or head gesture, which is used by the IoT-enabled device to perform an action corresponding to the at least one gesture ID.

3. The eye-mounted device of claim 2 , wherein the processor further executes instructions to configure the eye-mounted device to receive augmented data, the augmented data comprising the selected augmented reality content corresponding to the gesture ID for rendering to the device display.

4. An eye-mounted device according to claim 3, wherein the augmented reality content includes at least one animatable two-dimensional or three-dimensional object, which is sent to the device display to be displayed at a position relative to a pre-specified anchor position of the IoT display identified in Cartesian coordinates.

5. The eyewear device of claim 4, wherein the processor further executes instructions to recognize a gesture event that initiates movement of at least one animatable two-dimensional or three-dimensional object.

6. The eyewear device of claim 4 , wherein the processor further executes instructions to display the rendered augmented reality content on the device display at a position relative to the pre-specified anchor position, wherein The position is outside the size of the IoT display in real-world coordinates.

7. The eyewear device of claim 1 , wherein the processor further executes instructions to process IoT instructions from the IoT-enabled device to: rendering the augmented reality content on the device display; Trigger animation; updating the augmented reality content with new data; hiding the augmented reality content or specific elements within the augmented reality content; Displaying a specific object in the augmented reality content; or Send any payload, which can be received and interpreted to dynamically determine what to show on the device display.

8. The eyewear device of claim 1 , wherein the processor further executes instructions to scale the augmented reality content relative to a height and a width dimension of the IoT display so that the augmented reality content is displayed proportionally on the IoT display.

9. The eye-mounted device of claim 8, wherein the processor executes instructions to scale the augmented reality content to fit the IoT display by fitting the augmented reality content along an (x, y) axis of the IoT display, fitting the augmented reality content on either the x or y axis while the other axis is proportionally distorted or scaled, or fitting the augmented reality content to a specific percentage of a screen plane size of the IoT display.

10. The eyewear device of claim 1 , wherein the processor further executes instructions to select augmented reality content for display relative to the IoT display by selecting a streaming application from the IoT display, wherein The streaming application includes augmentation data that is navigated directly on the device display.

11. The eyewear device of claim 1 , wherein the IoT-enabled device comprises a smart TV.

12. A method of presenting content to an Internet of Things (IoT) enabled device, the IoT enabled device having an IoT display, the method comprising: Pairing an augmented reality (AR) enabled eyewear device with the IoT enabled device to communicate via a communication interface therebetween; Calibrate the AR eyewear-enabled device to the real-world coordinate position of the IoT display; selecting augmented reality content to display on a device display of the AR eyewear-enabled device at the real-world coordinate location relative to the IoT display; as well as Rendering the selected augmented reality content to the device display of the AR eyewear-enabled device at a location relative to the real-world coordinate location of the IoT display.

13. The method according to claim 12, further comprising: detecting at least one of a hand gesture or a head gesture; as well as Sending gesture events to the supporting IoT device via the communication interface, The gesture event includes at least one gesture identification (ID) of at least one detected hand gesture or head gesture, which is used by the IoT-enabled device to perform an action corresponding to the at least one gesture ID.

14. The method according to claim 13, further comprising: The AR-enabled eyewear device is configured to receive augmentation data including the selected augmented reality content corresponding to the gesture ID for rendering to the device display.

15. The method according to claim 14, wherein The augmented reality content includes at least one animatable two-dimensional or three-dimensional object, which is sent to the AR-enabled eyewear device to be displayed at a position relative to a pre-designated anchor position of the IoT display identified in a Cartesian coordinate system.

16. The method according to claim 15, further comprising: A gestural event is recognized, and movement of the at least one animatable two-dimensional or three-dimensional object is initiated in response to the recognized gestural event.

17. The method according to claim 15, further comprising: The rendered augmented reality content is displayed on the device display at a location relative to the pre-specified anchor location, wherein the location is outside the size of the IoT display in real-world coordinates.

18. The method according to claim 12, further comprising: The augmented reality content is scaled relative to the height and width dimensions of the IoT display so that the augmented reality content is displayed proportionally on the IoT display.

19. The method of claim 18, wherein scaling the augmented reality content comprises: Adapting the augmented reality content along the (x, y) axis of the IoT display, adapting the augmented reality content on the x or y axis while the other axis is proportionally distorted or scaled, or adapting the augmented reality content to a specific percentage of the screen plane size of the IoT display.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to present content to an Internet of Things (IoT)-enabled device having an IoT display by performing the following operations, the operations comprising: Pairing an augmented reality (AR) enabled eyewear device with the IoT enabled device to communicate via a communication interface therebetween; Calibrate the AR eyewear-enabled device to the real-world coordinate position of the IoT display; selecting augmented reality content to display relative to the real-world coordinate location of the IoT display; as well as The selected augmented reality content is rendered to the device display at a location relative to the real-world coordinate location of the IoT display.

Images