Interactive Reticle

Abstract

1. A head mounted near-eye display system for controlling a network connected device, the system including an image light guide operable to transmit an image-bearing light beam from an image source to an eyebox where a virtual image may be viewed, the image source operable to form a reticle within the virtual image; and a user input mechanism operable to control a reticle such that the reticle may be visually superimposed on a network connected device, the network connected device operable to receive a command over the network to change a state, and when the reticle is visually superimposed on the network connected device, the user input mechanism operable to select the network connected device, thereby causing a state of the network connected device to be changed.

Description

[Technical Field]

[0001] The present disclosure relates generally to electronic AR / VR displays, and more particularly to displays that utilize image light guides and are operable to display interactions between virtual and real worlds. [Background technology]

[0002] Head-mounted near-eye display systems are being developed for a wide range of applications, including military, commercial, industrial, firefighting, and entertainment applications. In many of these applications, there is value in creating a virtual image that can be visually superimposed on a real-world image in the user's field of view. Optical image light guides can deliver image-bearing light to the viewer in a small space, directing the virtual image toward the viewer's pupil and enabling this superimposition function.

[0003] A head-mounted near-eye display system provides the operator with a variety of virtual images and icons, and it would also be beneficial to provide the operator with the ability to interact with real-world objects, including, but not limited to, smart home networked devices. Summary of the Invention

[0004] By way of example, and not by way of limitation, the present disclosure provides systems and methods for displaying and controlling a virtual reticle or alignment guide. Additionally, the present disclosure describes an optical system having a virtual reticle that allows an operator to recognize, select, and control an external system.

[0005] The reticle system may include a wearable near-eye display system with a camera or other visual input device and image source, and may include various user input controls and a processing unit connected thereto. The reticle system displays a virtual image reticle, such as a crosshair or bounding box, whose color, shape, position, and input elements may be configured to suit the user's preferences. The system further provides various means for the operator to use and control the reticle and execute commands. Furthermore, the reticle controls may be operable for a remote user.

[0006] When networked with a system, device, object, or program operable to receive commands from the near-eye display system, the reticle system may enable an operator to initiate commands and receive information from such external systems, objects, or programs.

[0007] The novel reticle systems and methods also include specifications for interacting with, sending commands to, and receiving data from other devices and systems, such as Internet of Things (IOT) devices.

[0008] In one aspect, a head-mounted display system for controlling a network-connected device having a reticle is provided. The system includes the network-connected device operable to receive a command to change a state via a communications network, a transparent waveguide for forming a virtual image representation, the transparent waveguide operable to generate the reticle, and a user input mechanism for controlling the reticle so that it can be visually superimposed on the network-connected device. In another aspect, the user input mechanism is configured to select the network-connected device when the reticle is visually superimposed thereon, thereby enabling the head-mounted display system to change the state of the network-connected device. The system may include a virtual menu on the virtual image representation, the virtual menu including a plurality of selectable options associated with the network-connected device, and a reticle visually superimposed on one or more of the options, the reticle operable to select or deselect one or more of the options to change the state of the network-connected device.

[0009] In other aspects, the network-connected device changes state when a reticle is visually superimposed over the network-connected device for a desired period of time, the reticle is displayed within the virtual image when the preview is disabled, the reticle may be assigned different characteristics including color, shape, orientation, form, etc., the head-mounted display system is operable to select, deselect, control, move, or otherwise affect a virtual menu or other selectable virtual object by positioning the reticle within the virtual image, and / or the head-mounted display system is operable to activate, deactivate, control, or otherwise change the state of a networked IOT object by visually superimposing the reticle over the networked IOT object.

[0010] The reticle may be in the form of a crosshair or bounding box representing the field of view of an associated camera or other visual input device. The reticle may be assigned to a real-world object, such as a finger or stylus, such that movement of the reticle corresponds to movement of the finger or stylus. The system may also include a second reticle that operates independently of or interacts with the first reticle. The user input mechanism may be a user input button, a touch sensor, a voice command, or a facial gesture or movement captured by an operator-facing camera. [Brief explanation of the drawings]

[0011] The accompanying drawings are incorporated herein as part of this specification. The drawings described herein illustrate embodiments of the disclosed subject matter and illustrate selected principles and teachings of the present disclosure. However, the drawings do not illustrate every possible implementation of the disclosed subject matter and are not intended to limit the scope of the present disclosure in any way.

[0012] [Figure 1a] 1 is a schematic top view of an image light guide having an incoupling diffractive optical element and an outcoupling diffractive optical element that provides an expanded eyebox for a viewer, according to an exemplary embodiment of the disclosed subject matter. FIG. [Figure 1b] 1 is a schematic diagram of a person wearing a head-mounted near-eye display system, according to an exemplary embodiment of the disclosed subject matter. [Figure 2a] 1 is a schematic diagram of a camera-generated image of a real-world object conveyed through a head-mounted near-eye display system, according to an exemplary embodiment of the disclosed subject matter. FIG. [Figure 2b] 1 is a schematic diagram of a real-world object viewed through a head-mounted near-eye display system, according to an exemplary embodiment of the disclosed subject matter. [Figure 3a]1 is a schematic diagram of a real-world object as viewed through a head-mounted near-eye display system and a reticle appearing at optical infinity visually superimposed on the real-world object, in accordance with an exemplary embodiment of the disclosed subject matter. [Figure 3b] FIG. 1 is a schematic diagram of a head-mounted near-eye display system having a remote device connected via a communications network, according to an exemplary embodiment of the disclosed subject matter. [Figure 3c] FIG. 1 is a schematic diagram of a virtual menu transmitted through waveguides in the left and right eye assemblies of a head-mounted near-eye display system, and a reticle visually superimposed on the menu options, in accordance with an exemplary embodiment of the subject matter of the present disclosure. [Figure 3d] 1 is a schematic diagram of one or more networked devices and a virtual image including a reticle visually superimposed on the networked devices, according to an exemplary embodiment of the disclosed subject matter. [Figure 3e] 3d is a schematic diagram of a virtual image according to FIG. 3d transmitted through waveguides in the left and right eye assemblies of a head-mounted near-eye display system, and a reticle visually superimposed on a network-connected device, according to an exemplary embodiment of the subject matter of the present disclosure. [Figure 4a] FIG. 1 is a schematic diagram of a real-world object as viewed through a head-mounted near-eye display system and a virtual bounding box visually superimposed on and / or around the real-world object, according to an exemplary embodiment of the subject matter of this disclosure. [Figure 4b] FIG. 1 is a schematic diagram of a real-world object as viewed through a head-mounted near-eye display system, and a virtual bounding box visually superimposed on and / or around the real-world object indicating the camera's narrowed field-of-view boundary, according to an exemplary embodiment of the subject matter of this disclosure. [Figure 5] 1 is a flow diagram and method for reticle interaction with real-world objects via a head-mounted near-eye display system, according to an exemplary embodiment of the disclosed subject matter. [Figure 6] FIG. 1 is a schematic diagram of multiple real-world objects viewed through a head-mounted near-eye display system in which a reticle appearing at optical infinity is visually superimposed on the multiple real-world objects, in accordance with an exemplary embodiment of the subject matter of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram of a head-mounted near-eye display system driven by microdisplay technology according to an exemplary embodiment of the disclosed subject matter. DETAILED DESCRIPTION OF THE INVENTION

[0013] It should be understood that the present invention may assume various alternative orientations and step arrangements unless expressly specified to the contrary. It should also be understood that the specific assemblies and systems illustrated in the accompanying drawings and described in the following specification are merely exemplary embodiments of the inventive concepts defined herein. Accordingly, specific dimensions, orientations, or other physical characteristics relating to the disclosed embodiments are not to be considered limiting unless expressly stated otherwise. Also, although not applicable, like elements in the various embodiments described herein may be generally referred to within this section of the specification using like reference numerals.

[0014] As used herein, terms such as "first," "second," etc. do not necessarily imply any order, sequence, or priority relationship, but are merely used to more clearly distinguish one element or set of elements from another, unless otherwise specified.

[0015] As used herein, the term "exemplary" is meant to convey "an example of" and is not intended to suggest any preferred or ideal embodiment.

[0016] As used herein, the terms "viewer," "operator," "observer," and "user" are considered equivalent and refer to a person viewing a virtual image transmitted by one of the considered image light guides, particularly those located within an optical viewing device.

[0017] As used herein, the term "operable" has its conventional meaning of a device or component that is capable of producing an action in response to a stimulus, such as, for example, in response to an electrical signal.

[0018] As used herein, the term "set" refers to a non-empty set, as the concept of a collection of elements or members of a set is commonly understood in elementary mathematics. Unless explicitly stated otherwise, the term "subset" is used herein to refer to a non-empty proper subset, i.e., a subset of a larger set that has one or more members. For a set S, a subset may include the complete set S. However, a "proper subset" of set S is strictly contained in set S and excludes at least one member of set S.

[0019] As used herein, the term "reticle" refers to a virtual image generated by a projector / image source of a near-eye display system. The reticle may take the form of, but is not limited to, a crosshair, a pointer, a bounding box, or other visually identifiable form that may be visually superimposed on a real object or virtual image that may be used to execute commands issued from the head-mounted near-eye display system. The bounding box may indicate the outer edge of the virtual image, or simply a center alignment point. The size and shape of the bounding box may be adjusted within the virtual image as the zoom level of the camera is adjusted.

[0020] Optical systems, such as head-mounted near-eye display systems, can generate virtual images via an image source. Unlike methods for forming real images, virtual images are not formed on a display surface. That is, when a display surface is positioned at the perceived location of the virtual image, no image is formed on the surface. Virtual images have many advantages unique to augmented reality displays. For example, the apparent size of the virtual image is not limited by the size or location of the display surface. Furthermore, the source object of the virtual image can be small; for example, a magnifying glass provides a virtual image of the object. Compared to systems that project real images, a more realistic viewing experience can be provided by forming a virtual image that appears to be at a distance. Providing a virtual image also eliminates the need to correct for screen artifacts, which may be required when projecting a real image.

[0021] Referring now to the drawings, head-mounted near-eye display systems have a variety of applications, including military, commercial, industrial, firefighting, and entertainment applications. As described herein, head-mounted near-eye display systems are operable to form virtual color images that can be visually superimposed on the real world in the field of view of a head-mounted near-eye display system user. Referring now to FIG. 1a, an optically transparent parallel-plate waveguide 100, also known as a planar waveguide, transmits image-bearing light WI generated by a polychromatic or monochromatic projector system 110 to an HMD user. The planar waveguide 100 can transmit the image-bearing light WI in a narrow space to direct the image toward the HMD user's pupil, enabling the superimposition of a virtual image 46 on a real object in the HMD user's field of view.

[0022] A collimated, relative angle-encoded light beam from a color image projector source 110 can be coupled into the optically transparent planar waveguide 100 by an input coupling optical element, such as an incoupling diffractive optical element IDO, which can be attached to or formed on a surface of the parallel-plate planar waveguide 100 or disposed within the waveguide 100. Such a diffractive optical element can be formed as, but not limited to, a diffraction grating or a holographic optical element. For example, the diffraction grating can be formed as a surface-relief grating. After propagating along the planar waveguide 100, the diffracted color image-bearing light WG can be redirected out of the planar waveguide 100 by a similar output coupling optical element, such as an outcoupling diffractive optical element ODO, which can be configured to provide pupil dilation along one or more directions. Additionally, one or more diffractive redirecting gratings can be optically positioned along the waveguide 100 between the input grating IDO and the output grating ODO to provide pupil dilation in one or more directions. Image-bearing light WO output from the parallel-plate planar waveguide 100 provides an extended eyebox E for the viewer. Although the waveguide 100 is shown in the right eye assembly 25 of a head-mounted near-eye display system, it may also be located in the left eye assembly 24 or in both eye assemblies 24, 25.

[0023] FIG. 1b illustrates a head-mounted near-eye display system 20 in the form of glasses (i.e., smart glasses) in one embodiment of a reticle system. The head-mounted near-eye display system 20 may include at least a right temple arm 12 and a processing unit 18 having memory for storing data, storing computer programs, storing computer apps, and executing computer programs and apps. Furthermore, the processing unit 18 may be connected to an input device 16 and / or user input buttons 14. The input device 16 is operable to communicate user input to the processing unit 18. In one embodiment, the input device 16 is a touchpad or touch sensor. The touch sensor 16 is operable to receive input from one or more fingers of a user 10 and / or from a stylus. Those skilled in the relevant art will recognize that when input gestures are described as being performed by the fingers of the operator 10, the input gestures may also be performed by a stylus. Gestures received by the touch sensor 16 may include, but are not limited to, tapping the touch sensor 16, swiping / dragging the touch sensor 16 in a front-to-back direction, swiping / dragging the touch sensor 16 in a back-to-front direction, swiping / dragging the touch sensor 16 in a top-to-bottom direction, swiping / dragging the touch sensor 16 in a bottom-to-top direction, swiping / dragging the touch sensor 16 in a front-to-back and back-to-front direction simultaneously (e.g., a pinch motion), and vice versa, swiping / dragging the touch sensor 16 in a bottom-to-top and top-to-bottom direction simultaneously, and vice versa, swiping / dragging the touch sensor 16 in a front-to-back-and-front direction, swiping / dragging the touch sensor 16 in a back-to-front-and-back direction, swiping / dragging the touch sensor 16 in a bottom-to-top-and-down direction, and swiping / dragging the touch sensor 16 in a top-to-bottom-and-top direction. Touch sensor 16 may further detect whether one finger, two fingers, or three fingers are utilized during a gesture.In one embodiment, the user input buttons 14 are operable to communicate commands to the processing unit 18. In another embodiment, the operator 10 can send commands via a handheld device or voice input. The processing unit 18 can convert the operator's 10 interactions with the user input buttons 14, or sequences of gestures, into sequences of symbols suitable for comparison with stored sequences of symbols used to control specific operations on the device, including turning a camera on or off, focusing a camera, showing and hiding virtual images, enabling or disabling a computer program, controlling a computer program, enabling or disabling a visual guide, controlling a visual guide, selecting and controlling menu items, selecting and controlling networked real-world objects and devices 42, etc.

[0024] Head-mounted near-eye display system 20 may include augmented reality (AR) monocular or binocular smart glasses capable of displaying a virtual image 46 to the wearer's eyes, and may be connected to an external processing unit (e.g., a smartphone) capable of executing software to at least partially control the display of said virtual image 46 and / or the display of a reticle within virtual image 46. Additionally or alternatively, head-mounted near-eye display system 20 may include a processing unit suitable for controlling the display of virtual image 46.

[0025] The head-mounted near-eye display system 20 may show a preview of a virtual image 46 of what the camera 22 sees. The camera 22 may be operable via user input buttons 14, touch sensors 16, gestures, or verbal commands, allowing the operator 10 to initiate multiple tasks including taking pictures, recording video, two-way communication, barcode scanning, selecting and controlling networked real-world objects and devices 42, and the like.

[0026] In one embodiment, the head-mounted near-eye display system 20 may further include at least one camera 22 positioned to view hand gestures of the user 10 and real-world objects within its FOV. In one embodiment, the camera 22 is located adjacent to the periphery of the left eye assembly 24 or the right eye assembly 25 of the head-mounted near-eye display system. The FOV of the camera 22 is located generally in front of the operator 10. In another embodiment, the camera 22 is located on the left or right temple arm 12 of the head-mounted near-eye display system 20. The FOV of the camera 22 is positioned generally toward the front of the operator 10. In another embodiment, the camera 22 or additional cameras may extend the FOV of the head-mounted near-eye display system 20 in additional orientations, such as to areas above, behind, or to the sides of the operator 10.

[0027] In one embodiment, the head-mounted near-eye display system 20 may further comprise at least one operator-facing camera 28 positioned to view facial movements of the operator 10, including blinks, winks, eye movements, lip movements, facial expressions, etc., which may be used to initiate control of certain operations on the device, including turning the camera on and off, focusing the camera, displaying and hiding the virtual image display, annotating the virtual image display, enabling or disabling a computer program, controlling the computer program, enabling or disabling a visual guide, controlling the visual guide, selecting and controlling menu items on a virtual menu, selecting and controlling real-world networked items and devices 42 such as IoT devices, etc. In one embodiment, as illustrated in FIGS. 2a-3a and 3c, the operator-facing camera 28 is located on the eyeglass frame of the left eye assembly 24 or the right eye assembly 25. In FIGS. 2a-3a and 3c, a first operator-facing camera 28 is shown located adjacent to the left temple arm 12, and a second operator-facing camera 28 is shown located on a lower portion of the eyeglass frame of the left eye assembly 24.

[0028] 2a, in one embodiment, a real-world object 42 is captured by a camera 22 and a virtual image 46 of the real-world object 42 is displayed via a head-mounted near-eye display system 20. The virtual image 46 is located at optical infinity through a right-eye eyebox, in this example, generated by a waveguide 100 in a right-eye assembly 25 in conjunction with other optical components of the head-mounted near-eye display system 20. In another embodiment, the virtual image 46 is located at optical infinity through a left-eye eyebox, generated by a waveguide 100 in a left-eye assembly 24 in conjunction with other optical components of the head-mounted near-eye display system 20. In yet another embodiment, the virtual image 46 is located at optical infinity through right-eye and left-eye waveguides 100 in conjunction with other optical components of the binocular head-mounted near-eye display system 20. The camera 22 is operable to focus on objects, scan documents and items, read barcodes and other digitally encoded machine-readable optical labels, digitally record photographs, record videos of the real-world object 42, and the like. The image captured by the camera may be displayed in a virtual image 46 .

[0029] 2b, the virtual image 46 is disabled, allowing the observer 10 to have an unobstructed view of the real-world object 42. In another embodiment, the head-mounted near-eye display system 20 further comprises one or more corrective lenses that allow the operator 10 to see the real world in a manner similar to normal corrective glasses.

[0030] As shown in the embodiment of Figure 3a, reticle 60 is enabled. Virtual image 46 includes reticle 60, which appears to be superimposed on real-world object 42. In this example, reticle 60 appears in the center of virtual image 46, indicating the center of focus of camera 22 without obscuring operator 10's view.

[0031] In one embodiment, reticle 60 can be moved to different orientations in the FOV by operator 10 via user input buttons 14, touch sensor 16, voice commands, facial gestures or movements captured by operator-facing camera 28, or other input options. In another embodiment, reticle 60 may be paired with one or more eyes of operator 10, and reticle 60 may be moved according to an eye or gaze tracker in operator-facing camera 28 that measures the movement and / or rotation of the user's eyes. In one embodiment, reticle 60 remains in its assigned position until moved again by operator 10.

[0032] In one embodiment, multiple reticles may be generated and manipulated independently of the first reticle 60. For example, the second reticle 62 may be visible to the same eye of the observer 10 as the first reticle 60, or to the other eye of the observer or operator 10, as shown in Figure 3c.

[0033] 3b, in one embodiment, head-mounted near-eye display system 20 may be connected to a device 202 operated by remote party 200 such that remote party 200 can view data received by camera 22 and transmitted by head-mounted near-eye display system 20. Head-mounted near-eye display system 20 and remote device 202 may be connected via a communications network 104, such as, but not limited to, a LAN, a WAN (e.g., the Internet), a public network, and / or a private network. For example, remote party 200, such as a technical expert, can connect a computer 202 having a display to head-mounted near-eye display system 20 to view image and / or video data transmitted by head-mounted near-eye display system 20 for use by operator 10. Operator 10 may enable remote party 200 to assume control of the shape or form of reticle 60, such as, for example, changing the shape of bounding box 80 and other characteristics of reticle 60 and moving reticle 60 to different positions within virtual image 46. In one embodiment, the remote party 200 can also execute commands with the reticle 60, including, but not limited to, capturing images (i.e., photographs), recording video, scanning barcodes, interacting with networked real-world objects 42, and interacting with items within the virtual image 46. In one embodiment, the remote operator 200 can utilize the reticle 60 to annotate the virtual image 46. For example, the remote operator 200 can utilize the reticle 60 to draw lines and / or arrows to identify one or more features within the virtual image 46. In one embodiment, the head-mounted near-eye display system 20 is operable to align / synchronize the x and y coordinates of the remote operator's input device (e.g., a computer mouse) with the x and y coordinates of the virtual image 46, allowing the remote operator 200 to control the reticle 60 within the virtual image 46.In one embodiment, two or more remote parties 200 can view the image and / or video data transmitted by the head-mounted near-eye display system 20 in real time.

[0034] The reticle 60 may be visually superimposed on the real-world object 42. In one embodiment, the operator 10 is operable to select or interact with the real-world object 42 (also referred to herein as a networked device) by dwelling the operator's 10 gaze and the reticle 60 on the real-world object 42 for a period of time. In another embodiment, the operator 10 is operable to select or interact with the real-world object 42 by positioning the reticle 60 over the real-world object 42, pressing the user input button 14, manipulating the touch sensor 16, inputting one or more voice commands, facial gestures or movements captured by the operator-facing camera 28, or other input options. The nature of the interaction with the real-world object 42 is defined by integration with an associated computer program or application (i.e., app). For example, the head-mounted near-eye display system 20 and the reticle 60 may be used to control smart devices such as personal home assistants, door locks, lighting, ambient temperature controls, refrigerators, entertainment devices, vehicle entrances, vehicle controls, printers, elevators, etc.

[0035] In another embodiment, the interaction may include designating a real object to be actuated in place of the reticle 60, such as a stylus, one or more fingers, one or more hands, a baton, a ring, a glove, and the like. In one embodiment, connecting a real object, such as a stylus, a baton, a ring, or a glove, is achieved by entering the real-world object's serial number or code into a software application of the head-mounted near-eye display system 20. In one embodiment, the software application of the head-mounted near-eye display system 20 is the Vuzix Companion application, which is incorporated herein by reference in its entirety. In another embodiment, connecting a real object to the head-mounted near-eye display system 20 is achieved via a Bluetooth connection. In yet another embodiment, assigning the reticle 60 to a real object, such as a finger or hand, includes identifying / selecting the real object with the camera 22 and user input, and the real object is then tracked by the camera 22 via computer vision techniques, such as object recognition. For example, reticle 60 may be assigned to an object, such as a finger or stylus, such that movement of the object corresponds to (i.e., causes movement of) reticle 60. In an exemplary embodiment in which reticle 60 is assigned to an object, virtual reticle 60 may be hidden or de-emphasized (e.g., reduced in size or changed in color).

[0036] In one embodiment, the head-mounted near-eye display system 20 is operable to map the coordinates of the interactable real-world object 42 for later and / or real-time identification of the real-world object 42. By mapping the coordinates of the interactable real-world object 42, the head-mounted near-eye display system 20 can identify the real-world object 42 on which the reticle 60 is positioned such that the real-world object 42 is manipulated upon receiving observer input. For example, the head-mounted near-eye display system 20 can be utilized to map the coordinates of a wirelessly controllable lighting device. When the reticle 60 is positioned over the mapped wireless lighting device, the observer 10 can activate / deactivate the wireless lighting device via the input methods described herein.

[0037] In another embodiment, the head-mounted near-eye display system 20 is operable to connect with and identify the interactable real-world object 42 via an RFID tag and / or QR code having a unique identifier (e.g., a URL) located on / in the real-world object 42. For example, the head-mounted near-eye display system 20 can identify the wireless illumination device over which the observer 10 has positioned the reticle 60 via an RFID tag signal and activate / deactivate the wireless illumination device via the input methods described above. In embodiments utilizing RFID-tagged interactable real-world objects 42, the head-mounted near-eye display system 20 includes an RFID reader.

[0038] In another embodiment, the head-mounted near-eye display system 20 is operable to recognize the interactable real-world objects 42 via computer vision techniques such as object recognition. For example, the camera 22 of the head-mounted near-eye display system 20 may transmit video sequences to the processing unit 18 for real-time processing to identify the interactable real-world objects 42. In one embodiment, the head-mounted near-eye display system 20 may utilize edge detection in processing the sequences of images and / or videos captured by the camera 22.

[0039] 2a, in an embodiment, a real-world object 42 captured by camera 22 is displayed as a virtual image 46 visually superimposed on the real-world object 42. Upon receiving an input signal from operator 10 to interact with real-world object 42, head-mounted near-eye display system 20 may display an animation of virtual image 46 to visually indicate to operator 10 that an interaction is being performed or has been performed. For example, if real-world object 42 includes a desk lamp, the animation visually indicating the interaction may include the activation of a pull-string switch, as indicated by the double-headed arrow in FIG. 2a.

[0040] In one embodiment, interacting with a real-world object 42 can pin the reticle 60 to the object, and the reticle 60 will remain fixed to the object for a period of time. In another embodiment, the reticle 60 will remain pinned to the real-world object 42, even if it falls outside the FOV of the operator 10, for example, until contact with the real-world object 42 is terminated by the operator 10.

[0041] 3c, the reticle 60 can be positioned over a virtual object 64 in a virtual menu to select or interact with the virtual object 64, for example, by dwelling the operator's 10 gaze on the virtual object 64 for a period of time while the reticle 60 is positioned over the virtual object 64, by manipulating the user input buttons 14, the touch sensor 16, one or more voice commands, one or more facial gestures or movements captured by the operator-facing camera 28, or other input options. The nature of the interaction with the virtual object 64 is defined by integration with an associated computer program or software application. For example, the virtual object 64 may include a menu option, an augmented reality element, a virtual reality element, or any other selectable and / or interactive virtual item.

[0042] As shown in FIG. 3d, in one embodiment, the head-mounted near-eye display system 20 is operable to remotely connect to networked real-world objects and devices 42 to enable remote control of one or more states of the networked real-world objects and devices 42 via a reticle 60. For example, the head-mounted near-eye display system 20 is operable to display a virtual image 46 of a room in the operator's 10's home. The operator 10 can navigate the reticle 60 within the virtual image 46 to select a networked device 42 via the input device 16. In this manner, the operator 10 can turn networked lights on and off, adjust the settings of a networked thermostat, and / or control the state of a networked television from locations inside or outside the home. For example, the head-mounted near-eye display system 20 is operable to communicate with the networked real-world objects and devices 42 via a local area network (LAN), a wireless local area network (WLAN), a personal area network (PAN), and / or a wide area network (WAN). In one embodiment, the virtual image 46 of the room in the operator's 10's home displayed by the head-mounted near-eye display system 20 is a still image. For example, the virtual image 46 may be a still image captured by the camera 22 during the process of connecting the head-mounted near-eye display system 20 to the networked device 42. In another embodiment, the virtual image 46 of the room in the operator's 10's home displayed by the head-mounted near-eye display system 20 is a real-time image and / or video. For example, the virtual image 46 may be a real-time video captured by a camera located at a remote location.

[0043] Referring now to FIG. 4a, the head-mounted near-eye display system 20 software enables the reticle 60 to take the form of a bounding box 80. In this embodiment, when enabled, the bounding box 80 displays a shape, such as a rectangle, that defines a zoom boundary 82 of the FOV of the camera 22. In other embodiments, the bounding box 80 may be displayed as a square, circle, oval, or other shape. In one embodiment, the bounding box 80 may be one or more of multiple colors, shades, border weights, and gradient shading, and may include supporting elements, such as the second reticle 62, or various camera FOV configuration indicators, such as lines of sight, a rule of thirds grid, a centerline, etc. The bounding box 80 may also be used to scan barcodes or QR codes. Scanning of such barcodes and QR codes may be improved if the bounding box 80 is configured to reflect the precise location of the camera 22.

[0044] As shown in FIG. 4b, the bounding box 80 may be resized proportionally to the zoom boundary 82.

[0045] FIG. 5 is a flow diagram 300 illustrating one method for using the reticles 60, 62 disclosed herein to interact with real-world objects 42. In step 310, the reticle software is initiated. In step 312, the reticle software searches for network connections accessible to the head-mounted near-eye display system 20. In a next step 314, the reticle software obtains a list of all network-connected devices in the operator's 10's local environment whose state may be changed. Examples of such network-connected devices include personal home assistants, door locks, lights, temperature controls, refrigerators, entertainment devices, vehicle entrances and controls, printers, elevators, etc. In one embodiment, only devices with which the operator 10 is authorized to interact are obtained. In one embodiment, the list of network-connected devices includes devices for which the operator 10 has authority to change the state and that are not local to the operator's 10's current location. In one embodiment, the operator 10 can choose to list local and / or non-local devices. In decision step 316, the reticle software determines whether the reticle highlight mode is on or off. If the reticle highlight mode is off (or inactive), the reticle software enters a sleep mode. Sleep mode refers to a state of the reticle software that allows the operator 10 to continue operating the head-mounted near-eye display system 20 without further significant interaction with the operator 10 until the operator 10 initiates a reticle software command. In decision step 316, if the reticle highlight mode is on (or active), the method 300 proceeds to step 320. As shown in FIG. 6 , step 320 displays to the operator 10 a reticle 60 on each of the real-world devices 42 connected to the network and capable of changing its state. In one embodiment, the reticle 60 is a pointing-type reticle.In another embodiment, the reticle 60 is a bounding box-type reticle 80 that covers and / or surrounds at least a portion of the real-world object 42 as viewed by the operator 10. Decision step 322 is then entered, where the operator 10 can select one of the devices 42 by selecting one of the reticles 60 displayed to the operator. Next, step 324 is entered, where the reticle software displays a list of states that the selected device 42 can have. From this list, the operator 10 can select the state in which the operator 10 wants the device 42 to be. Decision step 326 is entered, where it is determined whether the operator 10 has chosen to change the current state of the device 42. If the operator 10 has chosen not to change the state of the device 42, decision step 328 is entered, where it is determined whether the operator 10 wants to clear the state option list. If the menu is not to be cleared, flow passes to step 324. If the list of states for the device 42 is to be cleared, flow passes to step 320. At decision step 326, if the operator 10 selects to change the current state of the device 42, flow passes to step 330. At step 330, the reticle software forms and issues a command to change the state of the selected device 42. At step 332, the menu of possible device 42 states is cleared and no longer displayed to the operator 10. Flow passes to step 320. At any time, the operator 10 can switch the reticle software into highlight mode or turn highlight mode off. This toggles whether the reticle 60 is displayed over real-world objects 42 in the operator's 10 environment. Step 340 switches highlight mode on. Step 350 switches highlight mode off.

[0046] Head-mounted near-eye display system 20 is described herein as utilizing one or more optically transparent parallel-plate waveguides 100 with one or more diffractive optical elements. However, embodiments of the presently disclosed subject matter may also be utilized in head-mounted near-eye display systems that are not optically transparent, such as head-mounted near-eye display system 400 illustrated in FIG.

[0047] One or more features of the embodiments described herein may be combined to create additional embodiments not shown. While various embodiments have been described in detail above, it should be understood that they are presented for illustrative purposes, and not for limiting purposes. It will be apparent to those skilled in the relevant art that the subject matter of the present disclosure may be embodied in other specific forms, variations, and modifications without departing from the scope, spirit, or essential characteristics thereof. The above-described embodiments are therefore to be considered in all respects as illustrative and not restrictive. The scope of the present invention is indicated by the appended claims, and all changes that come within the meaning and range of equivalents thereof are intended to be embraced therein.

Claims

1. 1. A head-mounted near-eye display system for controlling network-connected devices, comprising: an image source operable to form a reticle in a virtual image; a user input mechanism operable to control the reticle such that the reticle can be visually superimposed on the network-connected device; the network-connected device is operable to receive a command to change state over the network; The system, wherein the user input mechanism is operable to select the network-connected device when the reticle is visually superimposed over the network-connected device, thereby changing the state of the network-connected device.

2. 10. The system of claim 1, wherein the networked device changes state when the reticle is visually superimposed on the networked device for a predetermined period of time.

3. 2. The system of claim 1, further comprising a menu formed within the virtual image, the menu having a plurality of selectable options associated with the network-connected device, the reticle operable to be visually superimposed on one or more of the options, and the reticle operable to select or deselect one or more of the options to change the state of the network-connected device.

4. a camera positioned to capture an image of the network-connected device; a processing unit in communication with the camera and the image source; a waveguide arranged to transmit a virtual image of the image of the network-connected device to an eyebox; 10. The system of claim 1, wherein the waveguide is positioned to transmit an animated virtual image of the image of the network-connected device to the eyebox to visually indicate a change in state of the network-connected device.

5. The system of claim 1 , further comprising a camera or other visual input device, and wherein the reticle is a bounding box representing the field of view of the camera or other visual input device.

6. The system of claim 5 , wherein the bounding box is aligned with a barcode or QR code to scan an associated item or device.

7. The system of claim 1 , wherein the reticle is a crosshair.

8. The system of claim 1 , wherein the reticle is assigned to a real-world object, and movement of the reticle corresponds to movement of the real-world object.

9. The system of claim 1 , wherein the reticle is operable to select, deselect, control, or otherwise affect a virtual menu or other selectable virtual object.

10. The system of claim 1, wherein the reticle is a first reticle and further comprising a second reticle arranged to operate together with the first reticle or independently of the first reticle.

11. 10. The system of claim 1, wherein the reticle is operable to activate, deactivate, control, or otherwise change the state of a plurality of networked devices.

12. The system of claim 1 , wherein the reticle color, shape, orientation, or form is configured to be changed via the user input mechanism.

13. The system of claim 1 , wherein the user input mechanism includes at least one of a user input button, a touch sensor, or an operator-facing camera operable to capture facial gestures or movements.

14. 10. The system of claim 1, further comprising a processing unit operable to map coordinates of network-connected devices, whereby the network-connected devices are identifiable when the reticle is visually superimposed on the network-connected devices.

15. 10. The system of claim 1, further comprising an RFID reader operable to identify a network-connected device, whereby the network-connected device is identifiable when the reticle is visually superimposed over the network-connected device.

16. 10. The system of claim 1, further comprising a processing unit operable to identify network-connected devices via computer vision techniques, whereby the network-connected devices are identifiable when the reticle is visually superimposed on the network-connected devices.

17. 10. The system of claim 1, further comprising a camera and a processing unit in signal communication with the camera and the user input mechanism, the processing unit operable to interface with a remote device such that a remote party can view data received by the camera.

18. 10. The system of claim 1, further comprising a processing unit operable to remotely connect to one or more network-connected devices, wherein the waveguide is operable to transmit a virtual image comprising the one or more network-connected devices, and wherein the user input mechanism is operable to navigate the reticle within the virtual image such that the reticle is visually superimposed on the network-connected devices, thereby changing the state of the network-connected devices.

19. 20. The system of claim 18, wherein the virtual image is a static image of a room in which one or more networked devices are located.

20. 20. The system of claim 18, wherein the virtual image is a real-time video of a room in which one or more networked devices are located.

21. The system of claim 1, wherein the user input mechanism includes a voice command.

Images