Extending user interface of mobile application to AR glasses

Abstract

An architecture is provided for packaging a visual overlay-based user interface (UI) into a mobile device application as a user interface extension that allows certain flows and logic to be displayed on an eyewear device when connected to the mobile device application. Extending the UI of the mobile device application to the display of the eyewear device allows inexpensive experimenting of the augmented reality (AR) UI of the eyewear device and allows reuse of traffic logic across the mobile device and the associated eyewear device. For example, once a destination is selected in a navigation application on a mobile device, a mobile device application for a map or navigation may be extended to display a direction on an associated eyewear device. In this example, traffic logic still exists in a navigation application on the mobile device, but the user sees the AR direction overlaid on the display of the eyewear device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application serial number 17 / 897,395, filed on August 29, 2022, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to user interfaces for electronic devices, including wearable electronic devices that support augmented reality (AR), such as smart glasses. More specifically, but not limited to, the present disclosure describes techniques for extending the user interface of an application developed for a mobile device such as a smartphone to enable rendering of a portion of the user interface on smart glasses. Background Art

[0004] AR devices (such as smart glasses) are similar to mobile phones, but are typically more limited in available resources (battery life, thermal stress, memory, quality of Internet access, location data) and user input (no keyboard, trackpad, mouse, or touchscreen). In addition, developing experiences for such AR devices requires a significant investment, which can be a barrier for existing mobile application developers. This is further complicated by the fact that the user interface (UI) of an application developed for a mobile phone application is usually not transferable to a glasses device. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The features of the various embodiments disclosed will be readily understood from the following detailed description, with reference to the accompanying drawings. In the description and throughout the several views of the accompanying drawings, each element uses a single reference numeral. When multiple similar elements are present, a single reference numeral may be assigned to the same element.

[0006] Unless otherwise noted, the various elements shown in the figures are not drawn to scale. For clarity, the sizes of various elements may be enlarged or reduced. Several figures depict one or more embodiments and are presented by way of example only and should not be construed as limiting. The drawings include the following:

[0007] Figure 1A is a side view of an example eyewear device including an optical assembly having an image display;

[0008] Figure 1B yes Figure 1A a top cross-sectional view of the optical components and electronics in a portion of the eyewear device shown;

[0009] Figures 2A-2D is a rear view of an example eyewear device depicting an image display;

[0010] Figure 3is a block diagram illustrating an example of capturing visible light using the example eyewear device shown in any of the figures;

[0011] Figure 4 is a system block diagram of an example eyewear device;

[0012] Figure 5 is configured with Figure 4 A block diagram of electronic components of a mobile device for use with the system;

[0013] Figure 6 yes Figure 4 a block diagram of an example backend server system of the type shown;

[0014] Fig. 7A is a schematic diagram of a mobile phone application that communicates with an eyewear device to provide a user interface in an example configuration;

[0015] Figure 7B It is shown by Fig. 7A provided by navigation applications running on mobile devices Fig. 7A A schematic diagram of a user interface displayed on a glasses device;

[0016] Figure 8 is a communication diagram of a tethered mode between a mobile device and an eyewear device provided by a third party application in an example configuration;

[0017] Fig. 9 is a communication diagram of a tethered mode between a mobile device and an eyewear device in an example configuration, wherein the tethered mode is provided by an operating system of the mobile device; and

[0018] Fig.10 is a flow chart of a method for providing a UI of a mobile device application to an associated eyewear device in an example configuration. DETAILED DESCRIPTION

[0019] The systems and methods described herein relate to an architecture for seamlessly packaging a visual overlay user interface (UI) into a mobile device application when connected to the mobile device application to serve as a user interface extension that allows specific processes and logic to be displayed on a glasses device without creating a new glasses device-specific visual overlay or experience. Extending the UI of a mobile device application to the display of a glasses device allows for inexpensive experimentation with glasses device augmented reality UIs for a broad community of application developers. Extending the UI of a mobile device application to the display of a glasses device also allows for reuse of business logic across mobile devices and associated glasses devices. The techniques described herein also enable incremental additions to glasses device application processes wherever needed.

[0020] In an example configuration, the systems and methods described herein provide a simple way to extend an existing mobile device application so that a portion of its UI can be rendered on a glasses device using a visual overlay UI. For example, a visual overlay can be implemented as a filter implemented in Snapchat and in SPECTACLES TM and overlay, both available from Snap Inc. of Santa Monica, California. For example, once a destination is selected in a navigation application on a mobile device, an existing mobile device application for maps or navigation can be extended to display directions on an associated eyewear device. In this example, the business logic will still exist (e.g., execute) in the navigation application on the mobile device, but the user will be able to see augmented reality (AR) directions on the associated eyewear device.

[0021] The systems and methods described herein enable a mobile device to provide a user interface of a mobile device application executed thereon to an eyeglass device. The mobile device establishes a wireless connection with the eyeglass device and loads a visual overlay of at least one user interface for the mobile device application suitable for the eyeglass device to the eyeglass device via the wireless connection. The mobile device application is adapted to enable a user to select an eyeglass display mode to display one or more of the loaded visual overlays suitable for the eyeglass device during execution of the mobile device application, and a selection of at least one of the loaded visual overlays on the display of the eyeglass device as an output of the mobile device application is provided to the eyeglass device.

[0022] Communication of the UI visual overlay with the eyeglass device can be established in one of at least two different configurations. In the first configuration, the eyeglass device includes eyeglass UI extension software, which includes a background service that is executed to establish a wireless connection with the eyeglass device. The background service exposes services for communicating with the eyeglass device via a wireless connection to a mobile device application. The mobile device application further uses the exposed services of the eyeglass UI extension software to register itself as an eyeglass extension. The exposed services of the eyeglass UI extension software can further maintain a list of active registered mobile device applications. In the second configuration, the operating system of the mobile device includes an eyeglass connectivity service that is executed to establish a wireless connection with the eyeglass device. The eyeglass connectivity service is exposed to the mobile device application to communicate with the eyeglass device via a wireless connection. The mobile device application indicates that it contains an eyeglass UI extension by declaring a eyeglass category in its operating system manifest. The eyeglass connectivity service of the operating system maintains a list of active mobile device applications in the declared eyeglass category.

[0023] In other configurations, the mobile device may also include a glasses software development kit (SDK) that provides an interface for managing connectivity between a mobile device application and a glasses device via a wireless connection. The glasses SDK provides a layout for a visual overlay that can define the layout of the loaded visual overlay in a descriptive manner and use data binding to set and update data provided to the glasses device via a wireless connection. In an example configuration, the layout includes an XML file of resources added to the mobile device application. The mobile device application may also present an icon on the display to indicate the connectivity status of the mobile device and the glasses.

[0024] The following detailed description includes systems, methods, techniques, instruction sequences, and computer program products that illustrate the examples set forth in this disclosure. In order to provide a thorough understanding of the disclosed subject matter and its related teachings, many details and examples are included. However, those skilled in the relevant art can understand how to apply the related teachings without these details. The various aspects of the disclosed subject matter are not limited to the specific devices, systems, and methods described, because the related teachings can be applied or practiced in various ways. The terms and nomenclature used herein are only used for the purpose of describing specific aspects and are not intended to be limiting. In general, well-known instruction instances, protocols, structures, and techniques are not necessarily shown in detail.

[0025] As used herein, the terms "connect," "connected," "coupled," and "coupled" refer to any logical, optical, physical, or electrical connection, including links that transmit electrical or magnetic signals generated or supplied by one system element to another coupled or connected system element, etc. Unless otherwise specified, coupled or connected elements or devices are not necessarily directly connected to each other and may be separated by intermediate components, elements, or communications media, one or more of which can modify, manipulate, or carry electrical signals. The term "on" refers to being directly supported by an element or indirectly supported by an element through another element that is integrated into or supported by the element.

[0026] Additional objects, advantages and novel features of the examples will be set forth in part in the following description and in part will become apparent to those skilled in the art upon examination of the following and accompanying drawings, or may be learned by production or operation of the examples. The objects and advantages of the subject matter may be realized and obtained by means of the methods, tools and combinations particularly pointed out in the appended claims.

[0027] The orientations of the eyeglass device, related components, and any complete device including an eye scanner and camera shown in any of the figures are given as examples only for illustration and discussion purposes. In operation of a particular variable optical processing application, the eyeglass device may be oriented in any other orientation suitable for the particular application of the eyeglass device, such as upward, downward, sideways, or any other orientation. Furthermore, as used herein, any directional terms, such as front, back, inward, outward, toward, left, right, lateral, longitudinal, up, down, upper, lower, top, bottom, and side, are used only as examples and do not limit the direction or orientation of any optical device or optical device component constructed as described herein.

[0028] Reference will now be made in detail to the examples illustrated in the drawings and discussed below. An example eyeglass device and related systems and methods for extending a user interface of a mobile device application to an eyeglass device will be described with reference to FIGS. 1-10.

[0029] The system described in this article includes three types of hardware components: eyeglasses, mobile devices, and servers. The eyeglasses will be described with reference to FIGS. 1-3 , and the mobile devices will be described with reference to Figure 5 Describes the server and will refer to Figure 6 The corresponding system 400 will refer to Figure 4 The operation of the software components, including the application software on the eyewear device and the mobile device, and examples of system operation will be described with reference to Figures 7-10. Such software components include system software for tethering the eyewear device to the mobile device to receive a UI from an application on the mobile device, for example, to be displayed as an AR overlay on the display of the eyewear device.

[0030] In an example configuration, an eyewear device with augmented reality (AR) capabilities is used in the system described herein. It is desirable to use an eyewear device in the system described herein because such a device is scalable and customizable to enable a personalized experience, enable effects to be applied anytime, anywhere, and ensure user privacy by only allowing the user to view the transmitted information. In an example configuration, the eyewear device can be used without any dedicated hardware, such as the SPECTACLES available from Snap Inc. of Santa Monica, California. TM .

[0031] Figure 1A 1 is a side view depicting an example hardware configuration of the eyeglass device 100, which includes an optical assembly 180A ( Figure 2A ). The eyewear device 100 includes a plurality of visible light cameras 114A and 114B forming a stereo camera ( Figure 3), wherein the first visible light camera 114A is located on the right temple 110A, and the second visible light camera 114B is located on the left temple 110B ( Figure 2A ). In the example shown, the optical component 180A is located on the right side of the eyeglass device 100. The optical component 180A may be located on the left side of the eyeglass device 100 or at other locations.

[0032] Visible light cameras 114A and 114B may include image sensors that are sensitive to wavelengths in the visible light range. Each of visible light cameras 114A and 114B has a different front-facing overlap angle, for example, visible light camera 114A has an overlap angle 111A ( Figure 3 ). The overlap angle is the angular range over which the respective image sensors of visible light cameras 114A and 114B detect incident light and generate image data. Examples of such visible light cameras 114A and 114B include high-resolution complementary metal oxide semiconductor (CMOS) image sensors and video graphics array (VGA) cameras, such as 640p (e.g., 640 x 480 pixels, 0.3 megapixels total), 720p, 1080p, 4K, or 8K. Image sensor data from visible light cameras 114A and 114B may be captured along with geolocation data, digitized by an image processor, and stored in memory.

[0033] To provide stereoscopic vision, visible light cameras 114A and 114B may be coupled to an image processor ( Figure 4 Image processor 412 may include circuitry for receiving signals from visible light cameras 114A and 114B and processing those signals from visible light cameras 114A and 114B into a format suitable for storage in a memory ( Figure 4 The timestamp may be added by image processor 412 or other processor controlling the operation of visible light cameras 114A and 114B. Visible light cameras 114A and 114B allow the stereo camera to simulate human binocular vision. The stereo camera also provides a time stamp based on two captured images (i.e., images from visible light cameras 114A and 114B, respectively) with the same timestamp. Figure 3 A pair of images 358A and 358B) are used to reproduce a three-dimensional scene ( Figure 34. The ability to produce a three-dimensional image of a scene 315 in the image (see FIG. 4). Such three-dimensional images allow for immersive virtual experiences that feel realistic, such as for use in virtual reality or video games. For stereoscopic vision, a pair of images 358A and 358B may be generated at a given moment—one for each of visible light cameras 114A and 114B. Depth perception is provided by optical assemblies 180A and 180B when a pair of generated images 358A and 358B from forward-facing fields of view (FOVs) 111A and 111B of visible light cameras 114A and 114B are stitched together (e.g., by image processor 412).

[0034] In one example, the eyewear device 100 includes a frame 105, a right frame 107A, a right temple 110A extending from a right side surface 170A of the frame 105, and a see-through image display 180C ( Figure 2A -B) to present a GUI or other image to the user. The eyeglass device 100 includes a first visible light camera 114A connected to the frame 105 or the right temple 110A to capture a first image of a scene. The eyeglass device 100 also includes a second visible light camera 114B connected to the frame 105 or the left temple 110B to capture (e.g., simultaneously with the first visible light camera 114I) a second image of the scene, the second image at least partially overlapping the first image. Although Figure 1A and Figure 1B , but the processor 432 ( Figure 4 ) is coupled to the eyewear device 100 and is connected to the visible light cameras 114A and 114B and the processor 432 ( Figure 4 ) accessible memory 434, and the programming in the memory 434 can be set in the eyeglass device 100 itself.

[0035] Although Figure 1A , but the eyeglass device 100 may also include a head motion tracker ( Figure 1B Element 109) or eye movement tracker ( Figure 2A Element 113 or Figure 2B and Figure 2C The eyeglass device 100 may further include perspective image displays 180C and 180D of the optical components 180A and 180B, respectively, for presenting a display image sequence. The eyeglass device 100 may further include an image display driver ( Figure 4 The transparent image displays 180C and 180D and the image display driver are described in further detail below. The eyewear device 100 may also include a memory 434 and a processor 432 ( Figure 4), the processor 432 can access the image display driver 442 and the memory 434, and program in the memory 434. The execution of the programming by the processor 432 configures the eyewear device 100 to perform functions, including the function of presenting an initial display image of a display image sequence via the perspective image displays 180C and 180D, wherein the initial display image has an initial field of view corresponding to the initial head direction or the initial eye gaze direction determined by the eye movement tracker 113 or 213.

[0036] Execution of the programming by processor 432 may further configure eyewear device 100 to detect movement of a user of eyewear device 100 by: (i) detecting movement of a user of eyewear device 100 via a head motion tracker ( Figure 1B 109) to track the head movement of the user's head, or (ii) via an eye movement tracker ( Figure 2A Element 113 or Figure 2B and 2C 213) tracks eye movement of the user's eye of the eyeglass device 100. Execution of the programming by the processor 432 may further configure the eyeglass device 100 to determine a field of view adjustment of an initial field of view of an initial display image based on the detected user movement. The field of view adjustment may include a continuous field of view corresponding to a continuous head direction or a continuous eye direction. Execution of the programming by the processor 432 may further configure the eyeglass device 100 to generate a continuous display image of a display image sequence based on the field of view adjustment. Execution of the programming by the processor 432 may further configure the eyeglass device 100 to present the continuously displayed images via the perspective image displays 180C and 180D of the optical components 180A and 180B.

[0037] Figure 1B It is a depiction Figure 1A 1 is an illustration of a top cross-sectional view of the optical components and electronics in a portion of eyewear device 100 depicting first visible light camera 114A, head motion tracker 109, and circuit board 140A. Second visible light camera 114B is constructed and positioned substantially similar to first visible light camera 114A, except that it is connected and coupled on another side 170B ( Figure 2A ). As shown, the eyeglass device 100 includes a first visible light camera 114A and a circuit board, which may be a flexible printed circuit board (PCB) 140A. A first hinge 126A connects the right temple 110A to an articulated arm 125A of the eyeglass device 100. In some examples, the first visible light camera 114A, the flexible PCB 140A, or other components of electrical connectors or contacts may be located on the right temple 110A or the first hinge 126A.

[0038] As shown, the eyeglass device 100 may include a head motion tracker 109, which includes, for example, an inertial measurement unit (IMU). An inertial measurement unit is an electronic device that uses a combination of an accelerometer and a gyroscope, and sometimes a magnetometer, to measure and report the specific force, angular velocity, and sometimes the magnetic field around the object. The inertial measurement unit works by detecting linear acceleration using one or more accelerometers and detecting rotation rate using one or more gyroscopes. A typical configuration of an inertial measurement unit is that each of the three axes contains an accelerometer, a gyroscope, and a magnetometer: the horizontal axis is used for left and right movement (X), the vertical axis (Y) is used for top and bottom movement, and the depth or distance axis is used for up and down movement (Z). The accelerometer detects the gravity vector. The magnetometer defines the rotation in the magnetic field (e.g., facing south, facing north, etc.), just like a compass that generates a heading reference. The three accelerometers detect acceleration along the horizontal, vertical, and depth axes defined above, which can be defined relative to the ground, the eyeglass device 100, or the user wearing the eyeglass device 100.

[0039] The eyeglass device 100 can detect the movement of the user of the eyeglass device 100 by tracking the head movement of the user's head via the head movement tracker 109. The head movement includes a change in the head direction relative to the initial head direction on the horizontal axis, the vertical axis, or a combination thereof during the presentation of the initial display image on the image display. In one example, tracking the head movement of the user's head via the head movement tracker 109 includes measuring the initial head direction on the horizontal axis (e.g., X-axis), the vertical axis (e.g., Y-axis), or a combination thereof (e.g., lateral or diagonal movement) via an inertial measurement unit. Tracking the head movement of the user's head via the head movement tracker 109 also includes: measuring the continuous head direction on the horizontal axis, the vertical axis, or a combination thereof via the inertial measurement unit during the presentation of the initial display image.

[0040] Tracking the head movement of the user's head via the head movement tracker 109 may include: determining a change in the head direction based on the initial head direction and the continuous head direction. Detecting the movement of the user of the eyewear device 100 may also include: in response to tracking the head movement of the user's head via the head movement tracker 109, determining that the change in the head direction exceeds a deviation angle threshold on the horizontal axis, the vertical axis, or a combination thereof. In the example configuration, the deviation angle threshold is between approximately 3° and 10°. As used herein, the term "approximately" means ±10% of the amount when referring to an angle.

[0041] Changes along the horizontal axis slide three-dimensional objects (such as characters, Bitmojis, application icons, etc.) into and out of the field of view by, for example, hiding, unhiding, or otherwise adjusting the visibility of the three-dimensional objects. Changes along the vertical axis, for example, when the user looks up, in one example, display weather information, time, date, calendar appointments, etc. In another example, when the user looks down on the vertical axis, the eyewear device 100 can be powered off.

[0042] like Figure 1B As shown, the right temple 110A includes a temple body 211 configured to accommodate a temple cap. Figure 1B The temple cap is omitted in the cross section of FIG. Disposed inside the right temple 110A are various interconnected circuit boards, such as a PCB or flexible PCB 140A, which include a first visible light camera 114A, a microphone 130, a speaker 132, a low-power wireless circuit system (e.g., for communicating via Wireless short-range network communications) and high-speed wireless circuit systems (e.g., for A controller circuit for wireless LAN communication).

[0043] The first visible light camera 114A is coupled to or disposed on the flexible PCB 140A and is overlapped by a visible light camera cover lens, which is aimed through an opening formed in the right temple 110A. In some examples, the frame 105 connected to the right temple 110A includes an opening for the visible light camera cover lens. The frame 105 may include a forward-facing side that is configured to face outward away from the user's eyes. The opening for the visible light camera cover lens may be formed on the forward-facing side and through the forward-facing side. In this example, the first visible light camera 114A has an outward-facing overlap angle 111A with the line of sight or viewing angle of the user's right eye of the eyeglass device 100. The visible light camera cover lens may also be adhered to the outward-facing surface of the right temple 110A, where the opening is formed with an outward-facing overlap angle, but in a different outward direction. Coupling may also be achieved indirectly via an intermediate component.

[0044] First visible light camera 114A may be connected to first perspective image display 180C of first optical assembly 180A to generate a first background scene of a first continuously displayed image. Second visible light camera 114B may be connected to second perspective image display 180D of second optical assembly 180B to generate a second background scene of a second continuously displayed image. The first background scene and the second background scene may partially overlap to present a three-dimensional observable area of ​​the continuously displayed image.

[0045] Flexible PCB 140A may be disposed within right temple 110A and coupled to one or more other components housed in right temple 110A. Although shown as being formed on circuit board 140A of right temple 110A, first visible light camera 114A may be formed on another circuit board (not shown) in left temple 110B, articulated arm 125A, articulated arm 125, or one of frame 105.

[0046] Figure 2A is a diagram depicting a rear view of an example hardware configuration of the eyewear device 100. Figure 2A As shown, the eyewear device 100 is in a form configured to be worn by a user. Figure 2A The example of the eyewear device 100 is glasses. The eyewear device 100 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0047] In the eyeglass example, the eyeglass device 100 includes a frame 105 including a right frame 107A connected to a left frame 107B via a bridge 106, and the bridge 106 is configured to accommodate the user's nose. The right frame 107A and the left frame 107B include corresponding holes 175A and 175B, which hold corresponding optical elements 180A and 180B, such as lenses and perspective displays 180C and 180D. As used herein, the term lens refers to an overlay transparent or translucent glass or plastic sheet having curved and flat surfaces that cause light to converge / diverge or little or no convergence / divergence.

[0048] Although shown as having two optical elements 180A and 180B, the eyeglass device 100 may include other arrangements, such as a single optical element, depending on the application or intended user of the eyeglass device 100. As further shown, the eyeglass device 100 includes a right temple 110A adjacent to the right side 170A of the frame 105 and a left temple 110B adjacent to the left side 170B of the frame 105. The temples 110A and 110B may be integrated into the frame 105 on the respective sides 170A and 170B (as shown), or implemented as separate components attached to the frame 105 on the respective sides 170A and 170B. Alternatively, the temples 110A and 110B may be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0049] exist Figure 2AIn an example, an eye scanner 113 is provided, which includes an infrared emitter 115 and an infrared camera 120. Visible light cameras typically include a blue light filter to block infrared light detection. In one example, the infrared camera 120 is a visible light camera, such as a low-resolution video graphics array (VGA) camera (e.g., 640x 480 pixels, a total of 0.3 megapixels), in which the blue filter is removed. The infrared emitter 115 and the infrared camera 120 can be co-located on the frame 105. For example, both are shown as being connected to the upper portion of the left mirror frame 107B. The frame 105 or one or more of the temples 110A and 110B may include a circuit board (not shown) that includes the infrared emitter 115 and the infrared camera 120. For example, the infrared emitter 115 and the infrared camera 120 can be connected to the circuit board by welding.

[0050] Other arrangements of the infrared emitter 115 and the infrared camera 120 can be implemented, including arrangements where both the infrared emitter 115 and the infrared camera 120 are on the right frame 107A or at different locations on the frame 105. For example, the infrared emitter 115 can be on the left frame 107B, and the infrared camera 120 can be on the right frame 107A. In another example, the infrared emitter 115 can be on the frame 105, and the infrared camera 120 can be on one of the temples 110A or 110B, or vice versa. The infrared emitter 115 can be substantially connected to any location on the frame 105, the right temple 110A, or the left temple 110B to emit an infrared light pattern. Similarly, the infrared camera 120 can be substantially connected to any location on the frame 105, the right temple 110A, or the left temple 110B to capture at least one reflection change in the infrared light emission pattern.

[0051] Infrared emitter 115 and infrared camera 120 can be arranged to face inwardly toward the user's eyes, with a partial or full field of view of the eyes, to identify the corresponding eye position and gaze direction. For example, infrared emitter 115 and infrared camera 120 can be positioned directly in front of the eyes, at the top of frame 105, or in temples 110A or 110B at both ends of frame 105.

[0052] Figure 2B 1 is a diagram depicting a rear view of another example hardware configuration of an eyeglass device 200. In this example configuration, the eyeglass device 200 is depicted as including an eye scanner 213 on the right temple 210A. As shown, an infrared emitter 215 and an infrared camera 220 are co-located on the right temple 210A. The eye scanner 213 or one or more components of the eye scanner 213 can be located on the left temple 210B and other locations of the eyeglass device 200, such as the frame 105. The infrared emitter 215 and the infrared camera 220 are similar to Figure 2A, but the eye scanner 213 can be varied to be sensitive to different wavelengths of light, as previously described in Figure 2A Similar to Figure 2A , the eyewear device 200 includes a frame 105 including a right frame 107A connected to a left frame 107B via a bridge 106. The frames 107A-B may include respective apertures that hold respective optical elements 180A and 180B including see-through displays 180C and 180D.

[0053] Figure 2C and Figure 2D 1 is a diagram depicting a rear view of an example hardware configuration of the eyewear device 100 including two different types of see-through image displays 180C and 180D. In one example, these see-through image displays 180C and 180D of the optical assemblies 180A and 180B include integrated image displays. Figure 2C As shown, optical assemblies 180A and 180B include any suitable type of display matrices 180C and 180D, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a waveguide display, or any other such display.

[0054] The optical components 180A and 180B also include one or more optical layers 176A-N, which may include lenses, optical coatings, prisms, reflectors, waveguides, light strips, and any other combination of optical components. The optical layer 176 may include a prism of suitable size and configuration, and include a first surface for receiving light from the display matrix and a second surface for emitting light to the user's eyes. The prisms of the optical layer 176 may extend over all or at least a portion of the corresponding holes 175A and 175B formed in the frames 107A and 107B to allow the user to see the second surface of the prism when the user's eyes are viewed through the corresponding frames 107A and 107B. The first surface of the prism of the optical layer 176 faces upward from the frame 105, and the display matrix is ​​superimposed on the prism so that the photons and light emitted by the display matrix hit the first surface. The size and shape of the prism can be designed so that the light is refracted within the prism and directed to the user's eyes through the second surface of the prism of the optical layer 176. In this regard, the second surface of the prisms of the optical layer 176 can be convex to direct light toward the center of the eye. The prisms can be sized and shaped to magnify the image projected by the see-through image displays 180C and 180D, and the light passes through the prisms so that the image viewed from the second surface is larger in one or more dimensions than the image emitted from the see-through image displays 180C and 180D.

[0055] In another example, the see-through image displays 180C and 180D of the optical assemblies 180A and 180B may include Figure 2D The projected image display shown. Optical assemblies 180A and 180B include a projector 150, which may be a three-color projector using a scanning mirror, a galvanometer, a laser projector, or other types of projectors. During operation, a light source such as projector 150 is disposed in or on one of the temples 110A or 110B of the eyeglass device 100. Optical assemblies 180A and 180B may include one or more light strips 155A-N spaced across the width of the lens of the optical assemblies 180B and 180A or across the depth of the lens between the front and rear surfaces of the lens.

[0056] When the photons projected by the projector 150 pass through the lenses of the optical components 180A and 180B, the photons encounter the light strip 155. When a particular photon encounters a particular light strip, the photon is either redirected to the user's eye or passed to the next light strip. A combination of the modulation of the projector 150 and the modulation of the light strip can control a particular photon or light beam. In one example, the processor controls the light strip 155 by initiating a mechanical, acoustic, or electromagnetic signal. Although shown as having two optical components 180A and 180B, the eyeglass device 100 may include other arrangements, such as a single or three optical components, or the optical components 180A and 180B may have different arrangements depending on the application or intended user of the eyeglass device 100.

[0057] like Figure 2C and Figure 2D As further shown, the eyewear device 100 includes a right temple 110A adjacent to the right side 170A of the frame 105 and a left temple 110B adjacent to the left side 170B of the frame 105. The temples 110A and 110B can be integrated into the frame 105 on the respective sides 170A and 170B (as shown), or implemented as separate components attached to the frame 105 on the respective sides 170A and 170B. Alternatively, the temples 110A and 110B can be integrated into the articulated arms 125A and 125B attached to the frame 105.

[0058] In one example, the perspective image display includes a first perspective image display 180C and a second perspective image display 180D. The eyeglass device 100 may include a first hole 175A and a second hole 175B, which hold the corresponding first optical component 180A and the second optical component 180B. The first optical component 180A may include a first perspective image display 180C (e.g., a light strip and a projector in the right temple 110A, or a display matrix). The second optical component 180B may include a second transparent image display 180D (e.g., a light strip and a projector 150 in the right temple 110A, or a display matrix). The continuous field of view of the continuous display image may include a viewing angle between about 15° and 30° measured horizontally, vertically or diagonally, and more specifically 24°. The continuous display image with a continuous field of view represents a combined three-dimensional observable area visible by splicing together the two display images presented on the first image display and the second image display.

[0059] As used herein, "angle of view" describes the angular range of the field of view associated with the displayed image presented on each of the image displays 180C and 180D of the optical assemblies 180A and 180B. "Overlap angle" describes the angular range of the lens of the visible light camera 114A or 114B or the infrared camera 220 that can image. Typically, the image circle produced by the lens is large enough to completely cover the film or sensor, possibly including some vignetting (i.e., the brightness or saturation of the image is reduced toward the periphery compared to the center of the image). If the overlap angle of the lens does not fill the sensor, the image circle will be visible, typically with strong vignetting toward the edges, and the effective angle of view will be limited by the overlap angle. "Field of view" is intended to describe the field of view of the observable area that can be seen by the user of the eyewear device 100 through his or her eyes via the images presented on the image displays 180C and 180D of the optical assemblies 180A and 180B. Image display 180C of optical assemblies 180A and 180B may have a field of view with an overlap angle between 15° and 30° (eg, 24°) and a resolution of 480 x 480 pixels (or greater; eg, 720p, 1080p, 4K, or 8K).

[0060] Figure 3 The block diagram in FIG. 4 shows an example of capturing visible light with cameras 114A and 114B. Visible light is captured by first visible light camera 114A having a circular field of view (FOV) 111A. Selected rectangular first raw image 358A is used to generate a first raw image 358A for processing by image processor 412 ( Figure 4) for image processing. Visible light is also captured by a second visible light camera 114B having a circular FOV 111B. A rectangular second raw image 358B selected by an image processor 412 is used for image processing by the processor 412. The raw images 358A and 358B have overlapping fields of view 313. The processor 412 processes the raw images 358A and 358B and generates a three-dimensional image 315 for display by displays 180C and 180D. The three-dimensional image 315 is also referred to as an immersive image hereinafter.

[0061] Figure 4 The system block diagram in shows a high-level functional block diagram including example electronic components arranged in an example configuration in the eyewear device 100 or 200. The electronic components shown include a processor 432, a memory 434, and the see-through image displays 180C and 180D.

[0062] The memory 434 includes instructions for execution by the processor 432 to implement the functions of the eyeglass devices 100 and 200, including instructions for the high-speed processor 432 to control the image 315. Such functions can be implemented by processing instructions of the eye movement tracking program 445, the tethered mode application software 460, and the object tracking software 470 stored in the memory 434 and executed by the high-speed processor 432. The high-speed processor 432 receives power from the battery 450 and executes the instructions stored in the memory 434. The memory 434 can be a separate component, or the memory 434 can be integrated "on-chip" with the processor 432 to perform the functions of the eyeglass devices 100 and 200 and communicate with external devices via a wireless connection.

[0063] Eyewear devices 100 and 200 may be combined with eye movement tracking program 445 (e.g., using Figure 2B ) and can be implemented via the mobile device 500 ( Figure 5 ) and a server system 498 connected via various networks to provide user interface adjustments. The mobile device 500 may be a smartphone, tablet, laptop, access point, or any other such device capable of connecting to the eyewear device 100 or 200 using a low-power wireless connection 425 and a high-speed wireless connection 437. The mobile device 500 is also connected to the server system 498 via a network 495. The network 495 may include any combination of wired and wireless connections.

[0064] The glasses apparatuses 100 and 200 may include an image display driver 442 , an image processor 412 , a low power circuit system 420 , and a high speed circuit system 430 . Figure 4The components of the eyeglass devices 100 and 200 shown in the drawings are located on one or more circuit boards, such as PCBs or flexible PCBs 140A and 140B, in the temples. Alternatively or additionally, the depicted components may be located in the temples, frames, hinges, articulated arms, or bridges of the eyeglass devices 100 and 200. The visible light cameras 114A and 114B may include digital camera elements, such as complementary metal oxide semiconductor (CMOS) image sensors, charge coupled devices, lenses, or any other corresponding visible light or light capturing elements that can be used to capture data, including images of scenes with unknown objects.

[0065] The eye movement tracking program 445 implements user interface field of view adjustment instructions, including instructions for causing the eyeglass device 100 or 200 to track eye movements of the user's eyes of the eyeglass device 100 and 200 via the eye movement tracker 213. Other implemented instructions (functions) cause the eyeglass device 100 and 200 to determine a FOV adjustment to the initial FOV 111A-B based on the detected user eye movements corresponding to the continuous eye directions. Further implemented instructions generate a continuous display image of the display image sequence based on the field of view adjustment. The continuous display image is generated to the user as a visible output via the user interface. The visible output appears on the perspective image displays 180C and 180D of the optical components 180A and 180B, which are driven by the image display driver 442 to present a display image sequence, including an initial display image with an initial field of view and a continuous display image with a continuous field of view.

[0066] For example, the object tracking model applied by the object tracking software 470 can detect gestures of the user as well as objects in the environment that would be recognized by object recognition software associated with the eyewear device 100 or 200 in the example configuration.

[0067] The tethered mode application software 460 will be described in more detail below in conjunction with Figures 7-10.

[0068] like Figure 4As shown, the high-speed circuit system 430 includes a high-speed processor 432, a memory 434, and a high-speed wireless circuit system 436. In this example, an image display driver 442 is coupled to the high-speed circuit system 430 and operated by the high-speed processor 432 to drive the image displays 180C and 180D of the optical components 180A and 180B. The high-speed processor 432 can be any processor capable of managing high-speed communications and operations of any general-purpose computing system required for the eyeglass device 100 or 200. The high-speed processor 432 includes the processing resources required to manage high-speed data transmission on a high-speed wireless connection 437 to a wireless local area network (WLAN) using the high-speed wireless circuit system 436. In some examples, the high-speed processor 432 executes an operating system such as a LINUX operating system or other such operating system of the eyeglass device 100 or 200, and the operating system is stored in the memory 434 for execution. In addition to any other duties, the high-speed processor 432, which executes the software architecture of the eyeglass device 100 or 200, is used to manage data transmission with the high-speed wireless circuit system 436. In some examples, the high-speed wireless circuitry 436 is configured to implement a wireless communication protocol, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, also referred to herein as In other examples, other high-speed communication standards may be implemented by high-speed wireless circuitry 436 .

[0069] The low-power wireless circuit system 424 and the high-speed wireless circuit system 436 of the eyeglass devices 100 and 200 may include a short-range transceiver. and wireless WAN, LAN, or WAN transceivers (e.g., cellular or ). Mobile device 500, including a transceiver that communicates via low power wireless connection 425 and high speed wireless connection 437, can be implemented using details of the architecture of eyewear devices 100 and 200, as can other elements of network 495.

[0070] The memory 434 includes any storage device capable of storing various data and applications, including color maps, camera data generated by the visible light cameras 114A-B and the image processor 412, and images generated for display by the image display driver 442 on the see-through image displays 180C and 180D of the optical assemblies 180A and 180B. Although the memory 434 is shown as being integrated with the high-speed circuit system 430, in other examples, the memory 434 can be a separate independent element of the eyewear device 100 or 200. In some such examples, the circuitry can provide a connection from the image processor 412 or the low-power processor 422 to the memory 434 through a system on a chip including the high-speed processor 432. In other examples, the high-speed processor 432 can manage the addressing of the memory 434 so that the low-power processor 422 will start the high-speed processor 432 whenever a read or write operation involving the memory 434 is required.

[0071] The server system 498 may be one or more computing devices as part of a service or network computing system, for example, including a processor, a memory, and a network communication interface to communicate with the mobile device 500 and the eyeglass devices 100 and 200 via the network 495. The eyeglass devices 100 and 200 may be connected to a host computer. For example, the eyeglass device 100 or 200 may be paired with the mobile device 500 via a high-speed wireless connection 437, or connected to the server system 498 via the network 495. In addition, a gallery 490 of snapshots and AR objects may be maintained by the server system 498 for each user and invoked by providing communications that link to the snapshots and AR objects stored in the gallery 490.

[0072] The output components of the eyeglass devices 100 and 200 include visual components such as Figure 2C and Figure 2DThe image displays 180C and 180D of the optical components 180A and 180B described above (e.g., displays such as liquid crystal displays (LCDs), plasma display panels (PDPs), light emitting diode (LED) displays, projectors, or waveguides). The image displays 180C and 180D of the optical components 180A and 180B are driven by an image display driver 442. The output components of the eyeglass devices 100 and 200 also include acoustic components (e.g., speakers), tactile components (e.g., vibration motors), and other signal generators. The input components of the eyeglass devices 100 and 200, the mobile device 500, and the server system 498 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric inputs, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing tools), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and force of touch or touch gestures), and audio input components (e.g., microphones), etc.

[0073] The eyeglass devices 100 and 200 may include additional peripheral device elements, such as ambient light and spectral sensors, biometric sensors, thermal sensors 440, or other display elements integrated with the eyeglass devices 100 or 200. For example, the peripheral device elements may include any I / O components, including output components, motion components, position components, or any other such components described herein. The eyeglass devices 100 and 200 may take other forms and may be combined with other types of frames, such as headgear, headphones, or helmets.

[0074] For example, the biometric components of the eyeglass devices 100 and 200 may include components for detecting expressions (e.g., hand expressions, facial expressions, vocal expressions, body postures, or eye tracking), measuring biological signals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), etc. The motion components include acceleration sensor components (e.g., accelerometers), gravity sensor components, and rotation sensor components (e.g., gyroscopes), etc. The position components include position sensor components for generating position coordinates (e.g., global positioning system (GPS) receiver components), and positioning system coordinates. or transceiver, altitude sensor components (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), and orientation sensor components (e.g., a magnetometer), etc. Such positioning system coordinates may also be received from mobile device 500 via low power wireless circuitry 424 or high speed wireless circuitry 436 over wireless connections 425 and 437.

[0075] Figure 5 It is depicted with Figure 4 A block diagram of an example configuration of a mobile device 500 for use in conjunction with the system. Figure 5 5 is a high-level functional block diagram of an example mobile device 500, which a user can use with the eyeglass device 100 to extend the user interface to the eyeglass device 100 as described herein. The mobile device 500 may include a flash memory 505 that stores programming to be executed by the CPU 510 to perform all or a subset of the functions described herein. The mobile device 500 may also include a camera 525, which includes one or more visible light cameras (a first visible light camera and a second visible light camera with overlapping fields of view) or at least one visible light camera and a depth sensor with substantially overlapping fields of view. The flash memory 505 may also include multiple images or videos generated via the camera 525.

[0076] The mobile device 500 may further include an image display 530, a mobile display driver 535 for controlling the image display 530, and a display controller 540. Figure 5 In the example of , image display 530 may include a user input layer 545 (e.g., a touch screen) that is layered over or otherwise integrated into the screen used by image display 530. Examples of touch screen-type mobile devices that may be used include (but are not limited to) smartphones, personal digital assistants (PDAs), tablet computers, laptop computers, or other portable devices. However, the structure and operation of touch screen-type devices are provided by way of example; the subject technology described herein is not intended to be limited thereto. For purposes of this discussion, Figure 5 Thus, a block diagram illustration of an example mobile device 500 is provided having a user interface including a touch screen input layer 545 for receiving input (via touch, multi-touch or gestures by a hand, stylus or other tool, etc.) and an image display 530 for displaying content.

[0077] like Figure 5 As shown, the mobile device 500 includes at least one digital transceiver (XCVR) 550, shown as a WWAN (wireless wide area network) XCVR, for digital wireless communication via a wide area wireless mobile communication network. The mobile device 500 may also include additional digital or analog transceivers, such as a short-range transceiver (XCVR) 555 for short-range network communication, for example via NFC, VLC, DECT, ZigBee, or For example, the short-range XCVR 555 may take the form of any available two-way wireless local area network (WLAN) transceiver of a type compatible with one or more standard communication protocols implemented in wireless local area networks, such as IEEE 802.11. One of the standards.

[0078] In order to generate location coordinates for locating the mobile device 500, the mobile device 500 may also include a global positioning system (GPS) receiver. Alternatively or additionally, the mobile device 500 may utilize either or both of the short-range XCVR 555 and the WWAN XCVR 550 to generate location coordinates for positioning. For example, based on a cellular network, or The positioning system can generate very accurate position coordinates, especially when used in combination. Such position coordinates can be transmitted to the mobile device 500 via the XCVR 550, 555 through one or more network connections.

[0079] The transceivers 550, 555 (i.e., network communication interfaces) may conform to one or more of the various digital wireless communication standards used by modern mobile networks. Examples of WWAN transceivers 550 include, but are not limited to, transceivers configured to operate according to code division multiple access (CDMA) and third generation partnership project (3GPP) network technologies (including, but not limited to, for example, 3GPP type 2 (or 3GPP2) and LTE, sometimes referred to as "4G"). The transceiver may also incorporate broadband cellular network technology known as "5G". For example, transceivers 550, 555 provide two-way wireless communication of information including digitized audio signals, still images and video signals, web page information for display and network-related input, and various types of mobile message communications to / from the mobile device 500.

[0080] The mobile device 500 may also include a microprocessor that serves as a central processing unit (CPU) 510. A processor is a circuit having elements that are constructed and arranged to perform one or more processing functions (typically various data processing functions). Although discrete logic components may be used, these examples utilize components that form a programmable CPU. The microprocessor, for example, includes one or more integrated circuit (IC) chips that contain electronic components to perform the functions of the CPU 510. For example, the CPU 510 may be based on any known or available microprocessor architecture, such as a reduced instruction set computing (RISC) using an ARM architecture, as is commonly used today in mobile devices and other portable electronic devices. Of course, other arrangements of processor circuitry may be used to form the CPU 510 or processor hardware in smartphones, laptops, and tablet computers.

[0081] The CPU 510 functions as a programmable main controller of the mobile device 500 by configuring the mobile device 500 to perform various operations, e.g., according to instructions or programming executable by the CPU 510. For example, such operations may include various general operations of the mobile device 500, as well as operations related to programming of a messaging application and an AR camera application on the mobile device 500. Although a processor may be configured through the use of hardwired logic, a typical processor in a mobile device is a general-purpose processing circuit that is configured by executing programming.

[0082] Mobile device 500 also includes a memory or storage system for storing programming and data. Figure 5 In the example shown, the memory system may include flash memory 505, random access memory (RAM) 560, and other memory components 565, as needed. RAM 560 may be used as short-term storage for instructions and data being processed by CPU 510, such as working data processing memory. Flash memory 505 typically provides long-term storage.

[0083] Thus, in the example of mobile device 500, flash memory 505 may be used to store programming or instructions for execution by CPU 510. Depending on the type of device, mobile device 500 stores and runs a mobile operating system through which specific applications are executed. Examples of mobile operating systems include Google Android, Apple iOS (for iPhone or iPad devices), Windows mobile, Amazon Fire OS (operating system), or RIM BlackBerry OS, among others.

[0084] The mobile device 500 may include an audio transceiver 570 that can receive audio signals from the environment via a microphone (not shown) and provide audio output via a speaker (not shown). The audio signals may be coupled with video signals and other messages through a messaging application or social media application implemented on the mobile device 500.

[0085] Finally, the mobile device 500 can execute the tethered mode application software 460 loaded into the flash memory 505. In one example, the tethered mode application software 460 can include eyeglass-enhanced mobile device application software 462 (e.g., navigation software) that has been adapted to incorporate visual overlays into a UI for display on an associated eyeglass device 100 as described herein. The adaptation can include eyeglass UI extension software 464, which is a background service responsible for connectivity with the eyeglass device 100. The eyeglass software development kit (SDK) 466 allows the development of eyeglass UI extension software 464 to manage connectivity with the eyeglass device 100 in an example configuration. The tethered mode application software 460 will be described in more detail below with reference to Figures 7-10.

[0086] The techniques described herein may also be used in one or more of the computer systems described herein, or in one or more other systems. For example, the various processes described herein may be implemented in hardware or software or a combination of both. For example, at least one of the processors, memories, storage devices, output devices, input devices, or communication connections discussed below may each be at least a portion of one or more hardware components. A dedicated hardware logic component may be constructed to implement at least a portion of one or more of the techniques described herein. For example and without limitation, such a hardware logic component may include a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip (SOC), a complex programmable logic device (CPLD), etc. The applications of the devices and systems that may include various aspects may widely include various electronic and computer systems. The techniques may be implemented using two or more specific interconnected hardware modules or devices, wherein related control and data signals may be communicated between and through the modules, or as part of an ASIC. In addition, the techniques described herein may be implemented by a software program executable by a computer system. For example, implementations may include distributed processing, component / object distributed processing, and parallel processing. In addition, virtual computer system processing may be constructed to implement one or more of the techniques or functions described herein.

[0087] Figure 6 The block diagram in FIG. 1 shows a computer system for implementing a processing element such as Figure 4 The backend server system 498 shown in . Figure 6 is a block diagram of an example machine 600 on which the Figure 4 One or more configurations of an example backend server system 498 of the type shown. In alternative configurations, the machine 600 may operate as a standalone device or may be connected (e.g., using a network) to other machines. In a networked deployment, the machine 600 may operate as a server machine, a client machine, or both in server-client network environment. In one example, the machine 600 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment.

[0088] In an example configuration, the machine 600 may be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a network device, a server, a network router, a switch or a bridge, or any machine capable of executing an instruction (sequential or otherwise) that specifies the action to be taken by the machine. For example, the machine 600 may be used as a workstation, a front-end server, or a back-end server for a communication system. The machine 600 may implement the methods described herein by running software for providing the visual overlays and other UI features described herein. In addition, although only a single machine 600 is shown, the term "machine" should also be deemed to include any machine collection that executes a set (or multiple sets) of instructions individually or jointly to perform any one or more of the methods discussed herein, such as cloud computing, software as a service (SaaS), and other computer cluster configurations.

[0089] As described herein, examples may include or may run on a processor, logic or multiple components, modules or mechanisms (referred to herein as "modules"). A module is a tangible entity (e.g., hardware) that is capable of performing a specified operation and may be configured or arranged in a certain manner. In one example, a circuit may be arranged as a module in a specified manner (e.g., internally or relative to an external entity, such as other circuits). In one example, all or part of one or more computer systems (e.g., independent, client or server computer systems) or one or more hardware processors may be configured as a module by firmware or software (e.g., instructions, application portions or applications), which operates to perform specified operations. In one example, the software may reside on a machine-readable medium. When executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.

[0090] Thus, the term "module" is understood to include at least one of a tangible hardware or software entity, an entity that is physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., temporarily) configured (e.g., programmed) to operate in a particular manner or to perform part or all of any operations described herein. Considering examples where modules are temporarily configured, each of the modules does not need to be instantiated at any time. For example, where a module includes a general-purpose hardware processor configured using software, the general-purpose hardware processor can be configured into corresponding different modules at different times. The software can configure the hardware processor accordingly, for example, to constitute a particular module at one time and to constitute different modules at different times.

[0091] The machine (e.g., computer system) 600 may include a hardware processor 602 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 604, and a static memory 606, some or all of which may communicate with each other via an interconnect (e.g., a bus) 608. The machine 600 may also include a display unit 610 (shown as a video display), an alphanumeric input device 612 (e.g., a keyboard), and a user interface (UI) navigation device 614 (e.g., a mouse). In one example, the display unit 610, the input device 612, and the UI navigation device 614 may be a touch screen display. The machine 600 may additionally include a mass storage device (e.g., a drive unit) 616, a signal generating device 618 (e.g., a speaker), a network interface device 620, and one or more sensors 622. Example sensors 622 include one or more of a global positioning system (GPS) sensor, a compass, an accelerometer, a temperature, a light, a camera, a video camera, a physical state or position sensor, a pressure sensor, a fingerprint sensor, a retinal scanner, or other sensors. The machine 600 may also include an output controller 624, such as a serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate with or control one or more peripheral devices (e.g., a printer, a card reader, etc.).

[0092] The mass storage device 616 may include a machine-readable medium 626 having stored thereon one or more sets of data structures or instructions 628 (e.g., software) that embody or are utilized by any one or more of the techniques or functionality described herein. The instructions 628 may also reside, completely or at least partially, within the main memory 604, within the static memory 606, or within the hardware processor 602 during execution of the instructions 628 by the machine 600. In one example, one or any combination of the hardware processor 602, the main memory 604, the static memory 606, or the mass storage device 616 may constitute a machine-readable medium.

[0093] Although the machine-readable medium 626 is shown as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database or at least one of an associated cache and server) configured to store one or more instructions 628. The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions executed by the machine 600 and causing the machine 600 to perform any one or more of the techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with these instructions. Non-limiting machine-readable medium examples may include solid-state memory and optical and magnetic media. Specific examples of machine-readable media may include non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state hard disks (SSD); and CD-ROM and digital video disc (DVD)-ROM disks. In some examples, the machine-readable medium may include a non-transitory machine-readable medium. In some examples, machine-readable media may include machine-readable media that is not a transitory propagating signal.

[0094] The instructions 628 may also be sent or received over the communication network 632 via the network interface device 620 using a transmission medium. The machine 600 may communicate with one or more other machines using any of a variety of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone service (POTS) network, and a wireless data network (e.g., a wireless network known as a cellular network). The network interface device 620 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas 630 to connect to the communication network 632. In one example, the network interface device 620 may include multiple antennas 630 to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-input (MIMO), or multiple-input single-output (MISO) technology. In some examples, the network interface device 620 may use multi-user MIMO technology for wireless communication.

[0095] The features and flow charts described herein may be embodied as method steps in one or more methods, or in one or more applications as described above. According to some configurations, one or more "applications" are programs that perform the functions defined in the program. Various programming languages ​​may be used to generate one or more of the applications structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In one specific example, a third-party application (e.g., an application created by an entity other than a particular platform vendor using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on platforms such as IOS TM ANDROID TM , The invention relates to mobile software running on a mobile operating system of a mobile phone or other mobile operating system. In this example, a third-party application can call an API (application programming interface) provided by the operating system to facilitate the functions described herein. The application can be stored in any type of computer-readable medium or computer storage device and executed by one or more general-purpose computers. In addition, the methods and processes disclosed herein can alternatively be embodied in dedicated computer hardware or an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0096] The program aspects of the technology may be considered a "product" or "article of manufacture," typically in the form of at least one of executable code or associated data, which is carried or embodied in a machine-readable medium. For example, the programming code may include code for a touch sensor or other functions described herein. "Storage" type media include any or all tangible memories of a computer, processor, etc., or related modules thereof, such as various semiconductor memories, tape drives, and disk drives, which can provide non-transitory storage for software programming at any time. All or part of the software can sometimes be communicated over the Internet or various other telecommunications networks. For example, such communication can load software from one computer or processor to another computer or processor. Therefore, another type of medium that can carry programs, media content, or metadata files includes optical, electrical, and electromagnetic waves, such as those used over physical interfaces between local devices through wired and fiber optic landline networks and through various air links. Physical elements that carry such waves, such as wired links or wireless links, or optical links, etc., can also be considered as media that carry software. As used herein, unless limited to "non-transitory," "tangible," or "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions or data to a processor for execution.

[0097] Therefore, machine-readable media can take the form of tangible storage media in various forms. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer or the like, such as client devices, media gateways, code converters, etc., as shown in the drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute a bus within a computer system. Carrier transmission media can take the form of electrical or electromagnetic signals, or the form of sound or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Therefore, common forms of computer-readable media include, for example: floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punched card tapes, any other physical storage media with hole patterns, RAMs, PROMs and EPROMs, FLASH-EPROMs, any other storage chips or boxes, carriers that transmit data or instructions, cables or links that transmit such carriers, or any other medium from which a computer can read at least one of a programming code or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.

[0098] A software application for enabling a mobile device application to communicate with an eyeglass device to provide a user interface will be described with reference to FIGS. 7-10 .

[0099] Fig. 7A is a schematic diagram of tethered mode application software 460 implemented by a mobile device (e.g., a smartphone) 500 for communicating with the eyeglass device 100 to provide a user interface (UI) in an example configuration. As shown, the tethered mode application software 460 includes an eyeglass-enhanced mobile device application 462, which can be any mobile application that enhances AR functionality that can be displayed on the eyeglass device 100. In the example configuration, the extension can be in the form of a visual overlay UI plus data binding for displaying the eyeglass device 100 (e.g., SPECTACLES available from Snap Inc. of Santa Monica, California). TM ). The tethered mode application software 460 also includes glasses UI extension software (LNS) 464, which includes a background service running on the mobile device 500, which is responsible for connectivity with the eyeglass device 100 via connectivity provided by the mobile device application 462 or incorporated into the operating system of the mobile device (e.g., Google Android, Apple iOS (for iPhone or iPad devices), Windows mobile, Amazon Fire OS (operating system), or RIM BlackBerry OS, etc.). The glasses UI extension software 464 provides the user of the mobile device application 462 with the option to render all or part of the UI onto the eyeglass device 100. The glasses software development kit (SDK) 466 provides an interface for software application developers to manage connectivity that interacts with the glasses-enhanced mobile device application 462 and the eyeglass device 100 to provide a UI to the eyeglass device 100, as described herein.

[0100] Figure 7B It is shown that by Fig. 7A The AR enhanced navigation application 462 running on the mobile device 500 is Fig. 7A Schematic diagram of the display of the user interface embedded in the display on the right side of the eyewear device 100. Figure 8 and Fig. 9 Described in more detail, the mobile device 500 communicates with the eyeglass device 100 to enter a tethered mode, whereby the UI of the AR enhanced navigation application 462 is provided as an AR overlay to one or more image displays of the eyeglass device 100. In an example configuration, the AR overlay may consist only of XML or other files that define the UI in a descriptive manner. However, a custom application UI image may also be streamed to the display of the eyeglass device 100.

[0101] For example, in the case of an AR enhanced navigation application 462, the user may choose to enter the tethered mode by, for example, selecting a corresponding icon on the display of the mobile device 500. The display 710 may indicate to the user whether the AR enhanced navigation application 462 is in the tethered mode. When the AR enhanced navigation application 462 is in the tethered mode, the navigation UI features may be overlaid on one or more displays 720 of the eyewear device 100, such as Figure 7B For example, a destination 730, directions 740 (with directional arrows), and the remaining time to the destination 750 may be overlaid on the display 720 in the same manner as conventional AR filters available (e.g., in Snapchat, available from Snap Inc. of Santa Monica, California). The appearance of UI elements 720, 730, and 740 is developed by a developer using the glasses SDK 466, and the connectivity ( Figure 8 and Fig. 9 ) is managed by the glasses UI extension software 464. When the AR enhanced navigation application 462 updates its output, the UI on the display 720 is updated.

[0102] As described above, the glasses UI extension software (LNS) 464 can provide connectivity via the mobile device application 462, or the connectivity can be incorporated into the operating system of the mobile device 500. Therefore, there are two use cases for tethered mode:

[0103] 1. Tethered mode is provided as a third-party application that runs on all major mobile device operating systems without the need for native operating system support; and

[0104] 2. Tethered mode is integrated into the mobile device operating system.

[0105] In either tethered mode, the UI (e.g., visual overlay) and binding data are sent from the mobile device 500 to the eyeglass device 100. In the case of "tethered mode provided as a third-party application", the eyeglass SDK 466 binds to the exposed services of the eyeglass UI extension SDK service 464. On the other hand, in the case of "tethered mode integrated into the mobile device operating system", the eyeglass SDK 466 binds to the eyeglass connectivity service 900 of the operating system. Once bound, the eyeglass SDK 466 communicates with the corresponding service via the OS binder (e.g., Android binder) using a well-defined API / contract (e.g., AIDL file on Android). The corresponding service communicates with the corresponding service via The Bluetooth low energy (BLE) communicates with the paired eyeglass device 100 .

[0106] These use cases are Figure 8 and Fig. 9 shown.

[0107] Tethered mode provided as a third-party application

[0108] Figure 8 8 is a communication diagram of a tethered mode between a mobile device 500 and a glasses device 100 provided by a third-party application 462 in an example configuration. As shown, the glasses UI extension SDK service 464 is used to establish communication between the communication software (tethered mode application 800) of the glasses device 100 and the third-party application 462. In the example configuration, the tethered mode application 800 can use a filter rendering application or software (e.g., LensCore available from Snap, Inc. of Santa Monica, California) to render a custom filter generated by a filter generation application or software (e.g., Lens Studio available from Snap, Inc. of Santa Monica, California) as a visual overlay, which is responsible for parsing and rendering the tethered visual overlay sent from the mobile device 500.

[0109] exist Figure 8 In the configuration of , the third-party application 462 registers itself as a Glass extension using the Glass UI extension SDK service 464 at 810. The Glass UI extension SDK service 464 internally uses the third-party application 462 (e.g., available from Snap Inc. of Santa Monica, California) to ) to register itself with the exposed service of the glasses UI extension SDK service 464 (the third-party application must be installed on the mobile device 500). The exposed service of the glasses UI extension SDK service 464 can maintain a list of active registered applications by using the API of the mobile device operating system (e.g., PackageManager on an Android device). At 820, the exposed service of the glasses UI extension SDK service 464 also pairs with the glasses device 100 using a pre-established wireless communication service such as BLE.

[0110] The exposed service of the glasses UI extension SDK service 464 notifies the third-party application 462 of the status of the glasses device 100 and the status change of the connectivity by sending a notification 830. If the third-party application 462 including the glasses extension detects (based on receiving the notification 830) that the glasses 100 are connected and ready for use, the third-party application 462 can suggest to the user that compatible content be displayed on the glasses 100 instead of on the display of the mobile device 500 or on both devices. For example, the message window 710 ( Figure 7B) may be displayed on the display of the mobile device 500 to indicate to the user the connectivity status with the eyeglass device 100. Once the glasses display mode is selected, the button and / or display on the mobile device 500 may optionally be grayed out to indicate that the display has been transferred to the eyeglass device 100. The user who selects the glasses display mode may also send a request at 840 to access the glasses device 100 via BLE or Direct or other wireless communication configurations to initiate the transfer of the visual overlay to render the accompanying data to the eyewear device 100.

[0111] Once the exposed service of the glasses UI extension SDK service 464 indicates at 840 that the connection with the glasses 100 is complete and the tethered mode is active, the third-party application 462 requests to load a UI visual overlay for the third-party application 462 at 850. The exposed service of the glasses UI extension SDK service 464 also sends a visual overlay (e.g., filter) load request 852 to the glasses device 100. Upon receiving the visual overlay load request 852, the tethered mode application 800 of the glasses device 100 reads and parses the received visual overlay at 860, and renders an appropriate UI layout for display on the glasses device 100. The tethered mode application 800 may also provide a response 870 to the exposed service of the glasses UI extension SDK service 464, which in turn notifies the third-party application 462 at 872 that one or more visual overlays have been loaded.

[0112] Once the visual overlay has been loaded, the eyeglass device 100 is ready for use with the third-party application 462. At 880, the third-party application 462 provides UI data for display on the eyeglass device 100. The UI data is provided to the tethered mode application 800 by the exposed service of the eyeglass UI extension SDK service 464 as a settings data request 882 for display on the eyeglass device 100. At 890, the tethered mode application 800 updates the binding and refreshes the UI of the eyeglass device. A response 884 indicating the status of the settings data request 882 is provided. Steps 880, 882, 884, and 890 are repeated as the third-party application 462 provides updated UI data during execution of the third-party application 462. At any time during this process, the user can put on the eyeglass device 100 and view the UI from the third-party application 462 as an overlay on the display of the eyeglass device 100 (see Figure 7B ). The glasses display mode may continue until the glasses display mode is terminated on the glasses device 100 (eg, by a voice command) or by a suitable button or voice selection on the mobile device 500.

[0113] Tethered mode integrated into the mobile device's OS

[0114] Fig. 9 1 is a communication diagram of a tethered mode between a mobile device 500 and an eyeglass device 100, wherein the tethered mode is provided by an operating system of the mobile device 500 in an example configuration. It should be understood that the communication in this configuration is similar to that of FIG. 1 except that connectivity between the mobile device 500 and the eyeglass device 100 is provided by an integrated eyeglass connectivity service 900 of the operating system of the eyeglass device 500 instead of by the services of the eyeglass UI extension SDK service 464. Figure 8 The communication in is the same.

[0115] exist Fig. 9 In the configuration of , the third-party application 462 indicates that it contains the glasses UI extension by declaring the appropriate glasses category in its operating system manifest (e.g., AndroidManifest.xml for Android OS). The package manager (e.g., Android PackageManager) can maintain a list of active applications in the declared glasses category. The package manager can notify the active applications in the declared glasses category of any state changes of glasses status and connectivity. The glasses connectivity service 900 of the operating system can maintain a list of active applications by using the package manager API of the mobile device operating system. At 820, the glasses connectivity service 900 of the operating system can also use a pre-established wireless communication service (such as Low Energy (BLE)) is paired with the eyeglass device 100.

[0116] The glasses connectivity service 900 of the operating system notifies the third-party application 462 of the status of the glasses device 100 and the change of the status of the connectivity by sending a notification 830. If the third-party application 462 including the glasses extension detects (based on receiving the notification 830) that the glasses 100 are connected and ready for use, the third-party application 462 can suggest to the user that compatible content be displayed on the glasses 100 instead of on the display of the mobile device 500 or on both devices. For example, the message window 710 ( Figure 7B ) may be displayed on the display of the mobile device 500 to indicate to the user the connectivity status with the eyeglass device 100. Once the glasses display mode is selected, the button and / or display on the mobile device 500 may optionally be grayed out to indicate that the display has been transferred to the eyeglass device 100. The user who selects the glasses display mode may send a request at 840 to access the glasses device 100 via BLE or Direct initiates the transfer of the visual overlay to render the accompanying data to the eyewear device 100 .

[0117] Once the glasses connectivity service 900 of the operating system indicates at 840 that the connection with the glasses 100 is complete and the tethered mode is active, the third-party application 462 can request to load the UI visual overlay of the third-party application 462 at 850. The glasses connectivity service 900 of the operating system also sends a visual overlay loading request 852 to the glasses device 100. After receiving the visual overlay loading request 852, the tethered mode application 800 of the glasses device 100 reads and parses the received visual overlay at 860, and renders an appropriate UI layout for display on the glasses device 100. The tethered mode application 800 can also provide a response 870 to the glasses connectivity service 900 of the operating system, which in turn notifies the third-party application 462 at 872 that one or more visual overlays have been loaded.

[0118] Once the visual overlay has been loaded, the eyeglass device 100 is ready for use with the third-party application 462. At 880, the third-party application 462 provides UI data for display on the eyeglass device 100. The UI data is provided by the eyeglass connectivity service 900 of the operating system to the tethered mode application 800 as a settings data request 882 for display on the eyeglass device 100. At 890, the tethered mode application 800 updates the binding and refreshes the UI of the eyeglass device. A response 884 indicating the status of the settings data request 882 is provided. Steps 880, 882, 884, and 890 are repeated as the third-party application 462 provides updated UI data during use. At any time during this process, the user can put on the eyeglass device 100 and view the UI from the third-party application 462 as an overlay on the display of the eyeglass device 100 (see Figure 7B ). The glasses display mode may continue until the glasses display mode is terminated on the glasses device 100 (eg, by a voice command) or by a suitable button or voice selection on the mobile device 500.

[0119] Fig.10 is a flow chart of a method 1000 for providing a UI of a mobile device application 462 to an associated eyewear device 100 in an example configuration.

[0120] At 1010, by using Figure 8 and Fig. 9 Any of the described corresponding technologies establishes a tethered mode between the mobile device 500 and the eyeglass device 100 , while a wireless connection (eg, a BLE connection) is established between the mobile device 500 and the eyeglass device 100 .

[0121] At 1020, once the wireless connection is established, the mobile device 500 can load a visual overlay suitable for the eyeglass device 100 to the eyeglass device 100 via the wireless connection for use in at least one user interface of the mobile device application 462. The visual overlay can be a filter downloaded from the application, enabling the developer to select filters to build an augmented reality experience (e.g., Lens Studio), or a filter independently created by the application developer.

[0122] At 1030, the mobile device application 462 enables the user to select an eyeglass display mode for displaying one or more of the loaded visual overlays appropriate for the eyeglass device 100 during execution of the mobile device application 462. The mobile device application 462 may present a button or other selection functionality to the display of the mobile device application 462 for selection of the eyeglass device display mode.

[0123] At 1040, in the eyeglass device display mode, a selection of at least one of the loaded visual overlays is provided to the eyeglass device 100 via a wireless connection to be overlaid on the display of the eyeglass device 100 as an output of the mobile device application 462. The eyeglass device 100 can present the selected visual overlay as an overlay on the display of the eyeglass device in response to the output of the mobile device application 462, thereby extending the UI of the mobile device application 462 to the eyeglass device 100.

[0124] In such Figure 7B The third-party application 462 shown is an example of a navigation application with a glasses UI extension, where the user can type in a destination address on the mobile device 500, confirm the destination location, select "Navigate", and then select to present directions on the glasses 100 instead of the mobile device 500. This will prompt the sending of a navigation application visual overlay UI to the glasses device 100, which includes the destination location (e.g., a street name rendering box) 730, directions with a directional arrow 740, and an estimated time of arrival 750. Subsequently, based on the GPS location of the mobile device 500, the navigation application 462 can instruct the glasses device 100 what to display at a certain moment, such as "Turn right onto Eighth Street", and render the instruction with a right arrow at 740.

[0125] The glasses SDK 466 can be used to build a UI for display on the glasses device 100. Many simple layout capabilities can be supported, which allow application developers to define the layout of the tethered visual overlay in a descriptive manner and use data binding to set and update data. In the example configuration, the layout can be an XML file added to the application resources in the same way as a conventional OS layout file. In the example configuration, the user can package the filter created in LensStudio into a mobile device application. The filter will then be sent to the glasses device 100 and rendered there. Alternatively, a simple method for defining a non-complex UI layout can be provided via an XML file, which can, for example, define how to render an image vertically centered on the left side of the field of view, and render text horizontally centered at the top of the field of view, and use data binding to update the text (for example, using a key as a variable for the text content, which can be updated via an update message sent from a mobile device). The mobile application 462 can send these XML files (along with the reference image) to the glasses device 100. The general filter can then parse the XML and render the project based on the definition.

[0126] The XML for the above example might look like this:

[0127] <ui>

[0128] <img position＝"left,center"src＝"leftArrow.png">

[0129] <text position="center,top"text="$text1"fontSize="10">

[0130] < / ui>

[0131] The update message may contain a dictionary containing data binding updates, including ("text1"=>"HelloWorld").

[0132] For Android OS, an example UI layout in XML format for sending navigation text from the mobile device 500 to the glasses device 100 may be provided as follows:

[0133] <? xml version="1.0" encoding="utf-8"? >

[0134] <LinearLayout xmlns:snap="http: / / schemas.snap.com / tetheredmode / res / android"

[0135] snap:layout_width="match_parent"

[0136] snap:layout_height="match_parent"

[0137] snap:orientation="horizontal">

[0138] <ImageView

[0139] snap:layout_width="wrap_content"

[0140] snap:layout_height="wrap_content"

[0141] snap:src="@drawable / navigate_left_icon"

[0142] snap:visible=”#navigate_left” / >

[0143] <TextView

[0144] snap:layout_width="wrap_content"

[0145] snap:layout_height="wrap_content"

[0146] snap:text="#navigation_text" / >

[0147] <ImageView

[0148] snap:layout_width="wrap_content"

[0149] snap:layout_height="wrap_content"

[0150] snap:src="@drawable / navigate_right_icon"

[0151] snap:visible=”#navigate_right” / >

[0152]

[0153] In an example configuration, the eyeglass device 100 may operate as a "dumb" receiver that receives only XML data for display. However, in other configurations, the eyeglass device 100 may process voice, touch, and / or gesture commands, whereby the wearer of the eyeglass device 100 may provide instructions that are interpreted as interactions with the third-party application 462. For example, if the third-party application 462 is a cooking application, the eyeglass device 100 may interpret voice commands such as "previous" or "next" to move between steps in a recipe displayed on her eyeglass device 100 without having to use her hands to move between steps in the recipe. Other keywords may also be provided and interpreted as appropriate for the third-party application 462.

[0154] When developing a UI for a third-party application 462, the developer can use the glasses SDK 466 to register the third-party application 462, load the UI layout, and update the data. For example, the glasses SDK 466 is used to register the application (itself) and load the Spectacles SDK 466 available from Snap Inc. of Santa Monica, California. TM The code for using the above layout together with the navigation application can be provided as:

[0155]

[0156]

[0157] To make it easier for third-party application developers and use familiar patterns, the Spectacle SDK 466 can follow standard OS conventions. For example, for Android OS, a new resource category is provided for spectacles ("spectacles" in the above example) under "res" (for resources), which contains the UI visual overlay of the application spectacles extension. This can be represented as follows:

[0158]

[0159] In this example, filters 1-3 are transferred from Lens Studio to the third-party application 462 and used as the UI of the eyewear device 100. Alternatively, the developer can add filters to Lens Studio or create a new UI unique to the third-party application 462 for display on the eyewear device 100. For example, the developer can download filters for up, down, left, and right direction arrows. On the other hand, the developer can create new graphics corresponding to the respective direction arrows.

[0160] In the case of a filter-based UI, the filter can be downloaded to the third-party application 462 as needed, or a developer can create a UI unique to the third-party application 462. The filter can be obtained from, for example, LensStudio implemented by the backend server 498. Once the UI is created and loaded into the eyewear device 100, and the eyewear display mode is established, the eyewear UI responsive to the mobile device application 462 can be automatically displayed.

[0161] The scope of protection is limited only by the following claims. When interpreted in light of this specification and subsequent prosecution history, the scope is intended and should be interpreted to be as broad as consistent with the ordinary meaning of the language used in the claims and to encompass all structural and functional equivalents. Nonetheless, none of the claims is intended to encompass subject matter that does not satisfy the requirements of Sections 101, 102, or 103 of the Patent Act, nor should it be interpreted in such a manner. Any unintended inclusion of such subject matter is hereby disclaimed.

[0162] Except as stated above, nothing stated or described is intended or should be construed as conferring any element, step, feature, object, benefit, advantage, or equivalent to the public, whether or not recited in the claims.

[0163] It should be understood that the terms and expressions used herein have the common meaning consistent with these terms and expressions in their corresponding respective investigation and research fields, unless otherwise specified herein. Relational terms such as first and second can be used only to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The term "comprises", "comprising", "includes", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device that includes or includes a series of elements or steps not only includes these elements or steps, but also can include other elements or steps that are not explicitly listed or inherent to such process, method, article or device. In the absence of further restrictions, an element starting with "one" or "an" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0164] Unless otherwise indicated, all measurements, values, ratings, positions, quantities, dimensions, and other specifications set forth in this specification (including the following claims) are approximate and not exact. These quantities are intended to have a reasonable range consistent with the functions to which they are related and the customary practices in the art to which they belong. For example, unless otherwise expressly stated, parameter values ​​or similar values ​​may vary from the stated quantity by ±10%.

[0165] In addition, in the foregoing detailed description, various features are grouped together in various examples for the purpose of simplifying the present disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed examples require more features than are explicitly recited in each claim. On the contrary, as reflected in the following claims, the subject matter to be protected is less than all the features of any single disclosed example. Therefore, the following claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately claimed subject matter.

[0166] Although the foregoing has described what is the best mode and other examples, it should be understood that various modifications may be made therein, and the subject matter disclosed herein may be implemented in various forms and examples, and they may be applied to many applications, only some of which are described herein. The following claims are intended to claim all modifications and variations that fall within the true scope of the present concept.

Claims

1. A mobile device adapted to provide a user interface (UI) of a mobile device application executed thereon to an eyewear device, comprising: monitor; a memory for storing instructions; as well as a processor coupled to the display and the memory, wherein the processor executes the instructions to configure the mobile device to: Providing a wireless connection with the eyewear device; loading, via the wireless connection, to the eyeglass device a visual overlay of at least one user interface for the mobile device application suitable for the eyeglass device; enabling a user to select an eyewear display mode for displaying one or more of the loaded visual overlays appropriate for the eyewear device during execution of the mobile device application; and A selection of at least one of the loaded visual overlays is provided via the wireless connection to be overlaid on a display of the eyewear device as an output of the mobile device application. 2 . The mobile device according to claim 1 , further comprising glasses UI extension software, the glasses UI extension software comprising a background service executed by the processor to establish the wireless connection with the glasses device.

3. The mobile device according to claim 2, wherein: The background service exposes services to the mobile device application for communicating with the eyeglass device via the wireless connection.

4. The mobile device according to claim 2, wherein: The mobile device application exposes a service of the Glasses UI extension software for registering itself as a Glasses extension.

5. The mobile device according to claim 4, wherein: The exposed service of the Glasses UI extension software maintains a list of active registered mobile device applications. 6 . The mobile device of claim 1 , further comprising an operating system having an eyeglass connectivity service, the eyeglass connectivity service executed by the processor to establish the wireless connection with the eyeglass device.

7. The mobile device according to claim 6, wherein: The eyeglass connectivity service is exposed to the mobile device application to communicate with the eyeglass device via the wireless connection.

8. The mobile device according to claim 6, wherein: The mobile device application indicates that it contains a Glasses UI extension by declaring a Glasses category in its operating system manifest.

9. The mobile device according to claim 8, wherein: The operating system's glasses connectivity service maintains a list of active mobile device applications in the declared glasses category.

10. The mobile device of claim 1, further comprising a glasses software development kit (SDK) providing an interface for managing connectivity between the mobile device application and the glasses device via the wireless connection.

11. The mobile device according to claim 10, wherein: The glasses SDK provides a layout for the visual overlay, which enables the layout of the loaded visual overlay to be defined in a descriptive manner, and uses data binding to set and update data provided to the glasses device via the wireless connection.

12. The mobile device according to claim 11, wherein: The layout includes an XML file that adds resources to the mobile device application.

13. The mobile device of claim 1, further comprising instructions that, when executed by the processor, configure the mobile device application to present an icon on the display to indicate a connectivity status of the mobile device and the glasses.

14. A method of providing a user interface (UI) for a mobile device application executed on a mobile device to an eyewear device that communicates with the mobile device via a wireless connection, comprising: Establishing the wireless connection between the mobile device and the eyeglass device; loading, from the mobile device to the eyeglass device via the wireless connection, a visual overlay of at least one user interface for the mobile device application suitable for the eyeglass device; enabling a user to select an eyewear display mode for displaying one or more of the loaded visual overlays appropriate for the eyewear device during execution of the mobile device application; as well as A selection of at least one of the loaded visual overlays is provided via the wireless connection to be overlaid on a display of the eyewear device as an output of the mobile device application.

15. The method according to claim 14, wherein: Establishing the wireless connection between the mobile device and the eyeglass device includes: a background service of eyeglass UI extension software of the mobile device exposing a service for communicating with the eyeglass device via the wireless connection to the mobile device application.

16. The method of claim 15, further comprising the mobile device application registering itself as a Glasses extension using the exposed service of the Glasses UI extension software, and maintaining a list of active registered mobile device applications by the exposed service of the Glasses UI extension software.

17. The method according to claim 14, wherein: Establishing the wireless connection between the mobile device and the eyeglass device includes: the operating system of the mobile device establishing the wireless connection with the eyeglass device by exposing the eyeglass connectivity service of the operating system to the mobile device application, so that the mobile device application can communicate with the eyeglass device via the wireless connection.

18. The method according to claim 17, further comprising: The mobile device application indicates that it contains a glasses UI extension by declaring a glasses category in its operating system manifest, and the glasses connectivity service of the operating system maintains a list of active mobile device applications in the declared glasses category.

19. The method according to claim 14, further comprising: An interface is provided for managing connectivity between the mobile device application and the eyeglass device via the wireless connection, and an XML file for the visual overlay is added to the resources of the mobile device application, the XML file defining the layout of the loaded visual overlay in a descriptive manner and using data binding to set and update data provided to the eyeglass device via the wireless connection.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to provide a user interface for a mobile device application executed on a mobile device to an eyewear device that communicates with the mobile device via a wireless connection by performing the following operations: Establishing a wireless connection between the mobile device and the eyewear device; loading, from the mobile device to the eyeglass device via the wireless connection, a visual overlay of at least one user interface for the mobile device application suitable for the eyeglass device; enabling a user to select an eyewear display mode for displaying one or more of the loaded visual overlays appropriate for the eyewear device during execution of the mobile device application; as well as A selection of at least one of the loaded visual overlays is provided via the wireless connection to be overlaid on a display of the eyewear device as an output of the mobile device application.

Images