Using representations of user interface devices to train neural networks

Abstract

Disclosed herein is an example of a wearable display system capable of determining user interface (UI) events regarding a virtual UI device (e.g., a button) and a pointer (e.g., a finger or stylus) using a neural network. When a virtual UI device is shown to a user and the user interacts with the virtual UI device using a pointer, the wearable display system can render a representation of the UI device on an image of the captured pointer. The representation of the UI device can include high-contrast concentric shapes (or shapes with similar or identical centers of gravity). The neural network can be trained using training images with representations of virtual UI devices and pointers.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 62 / 537,311, filed on July 26, 2017, entitled “TRAINING A NEURAL NETWORK WITHREPRESENTATION OF USER INTERFACE DEVICES,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to virtual reality and augmented reality imaging and visualization systems, and in particular to representations of user interface devices for training and using machine learning models (e.g., neural networks) to determine user interface events. Background Art

[0004] A deep neural network (DNN) is a computational machine learning model. DNN belongs to a class of artificial neural networks (NN). Using NN, a computational graph that simulates the characteristics of biological neural networks is constructed. Biological neural networks include features that are prominent for computing and responsible for many capabilities of biological systems, which otherwise may be difficult to capture by other methods. In some implementations, such networks are arranged into a sequential hierarchical structure with unidirectional connections. For example, the output of an artificial neuron in a particular layer can be connected to the input of an artificial neuron in a subsequent layer. A DNN can be a NN with a large number of layers (e.g., 10, 100 or more layers).

[0005] Different NNs differ from each other in different perspectives. For example, the topology or architecture (e.g., the number of layers and how the layers are interconnected) and weights of different NNs may be different. The weights of a NN may be approximately analogous to the synaptic strength of neural connections in biological systems. The weights affect the strength of the effect propagated from one layer to another. The output of an artificial neuron (or a node of a NN) may be a nonlinear function of the weighted sum of its inputs. The weights of a NN may be the weights that appear in these sums. Summary of the invention

[0006] In one aspect, a wearable display system is disclosed. The wearable display system includes: an image capture device configured to capture an image including a pointer; a non-transitory computer-readable storage medium configured to store: an image, a virtual user interface (UI) device associated with the image at an image position on the image, and a neural network for determining UI events trained using: a training image associated with a training virtual UI device, the training image including a representation of the training virtual UI device and a training pointer, and training UI events about the training virtual UI device and the training pointer in the training image; a display configured to display the virtual UI device at a display position when the image capture device captures the image, wherein the image position is related to the display position; and a hardware processor in communication with the image capture device, the display, and the non-transitory computer-readable storage medium, the processor being programmed by executable instructions to: receive an image from the image capture device; render a representation of the virtual UI device onto the image at the image position; and determine UI events about the pointer in the image and the virtual UI device associated with the image using a neural network.

[0007] In another aspect, a system for training a neural network to determine user interface events is disclosed. The system includes: a computer-readable memory storing executable instructions; and one or more processors programmed by the executable instructions to at least: receive a plurality of images, wherein one of the plurality of images includes one of a plurality of pointers, wherein the image is associated with a virtual user interface (UI) device in a plurality of virtual UI devices at an image location on the image, and wherein the image is associated with a UI event in a plurality of UI events about the virtual UI device and the pointer in the image; render a representation of the virtual UI device onto the image at the image location to generate a training image; generate a training set including input data and corresponding target output data, wherein the input data includes the training image, and wherein the corresponding target output data includes the UI event; and train the neural network using the training set to determine UI events associated with the virtual UI device and the pointer.

[0008] In another aspect, a method for training a neural network to determine user interface events is disclosed. The method is under the control of a hardware processor and includes: receiving a plurality of images, wherein a first image of the plurality of images includes a first representation of a pointer of a plurality of pointers, wherein the first image is associated with a first representation of a virtual user interface (UI) device of a plurality of virtual UI devices at a first image position in the first image, and wherein the first image is associated with a UI event of a plurality of UI events about the virtual UI device and the pointer in the first image; rendering the first representation of the virtual UI device onto the first image at the first image position to generate a first training image; generating a training set including input data and corresponding target output data, wherein the input data includes the first training image, and wherein the corresponding target output data includes the UI event; and using the training set to train the neural network to determine UI events associated with the virtual UI device and the pointer.

[0009] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. Neither this summary nor the following detailed description is intended to define or limit the scope of the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 An example of a physical environment perceived by a user of an augmented reality device is shown.

[0011] Figure 2 An example of an augmented environment is shown, which includes a physical environment perceived by a user of an augmented reality device and a virtual remote control.

[0012] Figure 3 An example of a representation of buttons of a virtual remote control rendered onto an image captured by an imaging device of an augmented reality device, the image corresponding to the physical environment perceived by the user is shown.

[0013] Figure 4A and Figure 4B An example stereoscopic image pair captured by two imaging devices of an augmented reality device with a representation of a button rendered on the image is shown.

[0014] Figures 5A-5D An example representation of a user interface device is shown.

[0015] Figure 6 A flow chart of an illustrative method of training a neural network to determine user interface events using a representation of a user interface device is shown.

[0016] Figure 7A flow chart of an illustrative method of using a neural network to determine a user interface event using a representation of a user interface device is shown.

[0017] Figure 8 Depicted is an illustration of an augmented reality scene with certain virtual reality objects and certain actual reality objects viewed by a person, according to one embodiment.

[0018] Fig. 9 An example of a wearable display system according to one embodiment is shown.

[0019] Fig.10 Aspects of a method for simulating a three-dimensional image using multiple depth planes according to one embodiment are shown.

[0020] Fig.11 An example of a waveguide stack for outputting image information to a user is shown according to one embodiment.

[0021] Fig.12 An example output beam that may be output by a waveguide is shown according to one embodiment.

[0022] Fig.13 is a schematic diagram showing a display system according to an embodiment.

[0023] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure.

[0024] Specific implementation method

[0025] Overview

[0026] The virtual user interface (UI) device can be based on the style or implementation of the window, icon, menu, pointer (WIMP) UI device. Such a virtual UI device style or implementation is referred to as the simple virtual UI device. Implementation. In some embodiments, the process of detecting a WIMP UI event is divided into two processes, the first process is the calculation of the position of a pointer (eg, a finger, fingertip, or stylus), and the second process is the determination of the interaction of the pointer with the virtual UI device.

[0027] One of the challenges is the need to locate two different objects (the pointer and the virtual UI device). On a traditional 2D graphical user interface (GUI), the position of the UI device is known because it is generated in the same coordinates used by the GUI pointer device (e.g., a mouse pointer). With an augmented reality device (ARD, such as, for example, a Fig. 9In the wearable display system 900 described in the embodiment of the present invention, the UI device itself can be generated to appear at a specific position in the world coordinate system. Errors caused by the posture of the ARD and the calibration of the ARD can be introduced. In some implementations, the virtual UI device can be rendered in the ARD coordinate system (e.g., with respect to the coordinate frame of the ARD). Using the ARD coordinate system, the calibration of the ARD display can be different from the calibration of one or more outward-facing cameras used to capture the pointer image to determine the position of the pointer. Using any coordinate system, two numbers (e.g., the position of the virtual UI device and the position of the pointer) may need to be subtracted, and zero or zero crossing must be detected. Noise in the process may make such analysis very difficult. Even if there is no more deep neural network (DNN) for locating the pointer, the challenge still exists. Disclosed herein is a system and method for directly using DNN to determine such interactions or intersections (referred to as UI events herein). In some embodiments, the position of the pointer and the virtual UI device can also be considered. In some embodiments, the focus can be used to determine the interaction or intersection between the pointer and the virtual UI device. The pointer tip and the UI device may need to be in the same focus state for the interaction and the UI event to occur.

[0028] The process of training a NN involves presenting both input data and corresponding target output data to the network. This data, including both example inputs and target outputs, may be referred to as a training set. Through the training process, the weights of the network may be adjusted incrementally or iteratively so that, given specific input data in the training set, the output of the network begins to match (e.g., as closely as possible) the target output corresponding to the specific input data.

[0029] Constructing a training set for training a NN can present challenges. Construction of a training set is important for training the NN and therefore for the successful operation of the NN. In some embodiments, the amount of data required can be very large, such as tens, hundreds, thousands, millions, or more examples of the correct behavior of the network. The network can learn using the training set to correctly generalize its learning results to predict the correct output for an input (e.g., a novel input that may not be present in the original training set). Disclosed herein are systems and methods for generating training data for training a NN to determine user interface (UI) events associated with a virtual UI device and pointer (e.g., activation of a virtual button by a stylus). An example of such a system may be a gesture recognition system.

[0030] A display such as a head mounted augmented reality display (ARD), a mixed reality display (MRD), or a virtual reality display (VRD) may implement such a trained NN to determine UI events regarding virtual UI devices and pointers. Certain examples described herein refer to the ARD, but this is for illustration only and not limitation. In other examples, an MRD or VRD may be used in place of the ARD. A user may interact with the ARD or devices of the user environment using a pointer (such as a finger, fingertip, or stylus), resulting in UI events such as actuation or activation of a virtual UI device (e.g., a button). The ARD may use the NN to determine such activations of virtual UI devices or UI events regarding virtual UI devices and pointers. The NN may be trained using images with representations of UI devices rendered on the images.

[0031] The representation of the UI device used to train the NN and the representation of the UI device displayed to the user by the ARD may differ in style. For example, the representation of the UI device displayed to the user by the ARD may be a stylized UI device, such as a stylized button. The representation of the UI device rendered on the image for training the NN may include high-contrast concentric shapes (or shapes with similar or identical centers of gravity). In some implementations, such representations of UI devices may be advantageously standardized so that similar UI devices have similar representations when rendered onto images for training the NN. The standardized representation of the UI device used to train the NN may be referred to as the standard representation of the UI device. For example, when different types of buttons are rendered onto images for training the NN, they may have similar representations. The representation of the UI device may be rendered onto a captured image (e.g., a single-view image, a stereoscopic image pair, or a multi-view image set). When the user interacts with the virtual UI device using the NN, the ARD may determine that a UI event has occurred by processing a pointer image captured using the outward-facing camera of the ARD. Before the NN processes the image to determine the UI event, the standard representation of the UI device may be rendered onto the captured image perceived by the user. In some implementations, the normalized representations may be normalized to train multiple NNs for the same, similar, or different tasks (eg, identifying different types of UI events, such as touching or pointing with a finger).

[0032] Example User Environment

[0033] Figure 1An example of a physical environment perceived by a user of the ARD is shown. The example environment 100a includes a living room of a user's home. The environment 100a has physical objects such as a television (TV) 104, a physical remote control 108 (sometimes simply referred to as a remote control), a TV cabinet 112, and a window 116. While the user is wearing the ARD, the user can perceive and interact with the physical objects. For example, the user can watch the TV 104 while wearing the ARD. The user can control the TV 104 using the physical remote control 108. For example, the user can control the physical remote control 108 to turn the TV 104 on / off or change the channel or volume of the TV 104. The user can also interact with the TV 104 using a virtual remote control. The virtual remote control can be generated based on the functions of the physical remote control 108. For example, the virtual remote control can simulate some or all of the functions of the physical remote control 108 (and can also provide additional or alternative functions).

[0034] In one implementation, the specification of the virtual remote control may be stored in a data storage device, such as Fig. 9 The specification may include instructions on how to render the virtual remote control (e.g., specifying the layout, buttons and other controls, etc.), the communication channel required to emulate the remote control (e.g., according to the IRDA specification), the actual code to perform the emulation on that channel (e.g., the exact IR pulse sequence associated with the selection of "channel 2", etc.), etc.

[0035] The user can activate the virtual remote control. Upon activation, ARD can render the virtual remote control in the user's field of view (FOV). The virtual remote control can simulate the functions of a target object (such as a physical remote control). The user can activate the virtual remote control by actuating a user input device, such as, for example, clicking on a mouse, tapping on a touchpad, sliding on a touch screen, hovering or touching a capacitive button, pressing a key on a keyboard or game controller (such as a 5-way d-pad), pointing a joystick, wand or totem at a target, pressing a button on a remote control, performing other interactions with a user input device, etc. The user can also activate the virtual remote control using head, eye or body gestures (such as staring at or pointing at a target object for a period of time).

[0036] In some implementations, to activate the virtual remote control, the user may indicate a selection of a target device associated with the virtual remote control. For example, the user may indicate a selection of a physical remote control to activate the corresponding virtual remote control. Figure 1As shown in , if a user wants to interact with a virtual remote control that is based on a function of a physical remote control 108, the user may indicate the physical remote control 108 through gestures (such as touching, pointing with a finger, visually enclosing an object by, for example, pinching or using other gestures). As an example, the user may point in the direction of the physical remote control 108 for an extended period of time. As another example, the user may select a virtual remote control associated with the physical remote control 108 by making a gesture for grabbing the physical remote control 108. The user may also use a user input device (e.g., Fig.11 ARD can recognize the pinch gesture as a command to render and present the virtual remote control associated with the television 104 to the user. As another example, if the user wants to select a virtual remote control for controlling the television 104, the user can use a body gesture (such as grabbing the television 104 or pointing at the television 104) to indicate the selection of the television 104.

[0037] Example Enhanced Environment

[0038] In addition to being a display, an ARD (or MRD or VRD) can be an input device. Non-limiting exemplary modes of input for such a device include gestures (e.g., hand gestures) or actions with a pointer, stylus, or other physical object. Gestures may involve movements of a user's hand, such as a hand pointing in a direction. Actions may involve touching, pressing, releasing, sliding up or down or left or right, moving along a track, or other types of motion in 3D space. In some implementations, virtual user interface (UI) devices, such as virtual buttons or sliders, may appear in the virtual environment perceived by the user. These UI devices may be similar to two-dimensional (2D) or three-dimensional (3D) windows, icons, menus, pointer (WIMP) UI devices (e.g., appearing in iOS TM or Android TM UI devices in the operating system). Examples of these UI devices include virtual buttons, up and down buttons, spinners, selectors, radio buttons, radio button lists, check boxes, picture boxes, check box lists, drop-down lists, drop-down menus, selection lists, list boxes, combo boxes, text boxes, sliders, links, keyboard keys, switches, sliders, touch surfaces, or combinations thereof.

[0039] Figure 2An example of an augmented environment 200 is shown, including a physical environment perceived by a user of an augmented reality device and a virtual remote control. The virtual remote control may mirror at least a portion of some or all of the functionality or layout of a physical remote control. This may make it easier for a user who is familiar with the functionality or layout of a physical remote control to operate the virtual remote control. After the user selects a virtual remote control, the ARD may render the virtual remote control so that it is visually visible near the user. For example, Figure 2 As shown in , ARD can render a virtual remote control 124 including a control panel 128, which appears within the reach of the user (e.g., within arm's length) at a specific location in three dimensions (3D), so that the user can conveniently interact with the virtual remote control 124. In some embodiments, as the user moves around his environment, the virtual remote control can move with the user accordingly. For example, ARD can present the virtual remote control 124 at a certain distance from the user, regardless of the user's current location in the user's environment.

[0040] ARD may render virtual remote control 124 as superimposed on the user's physical environment. For example, ARD may render virtual remote control 124 as if it were in front of a wall. Virtual remote control 124 may have a non-transparent rendering so that the user may perceive that the virtual remote control obscures a portion of the user's physical environment, such that the virtual remote control appears to be in front of a portion of the environment. In some implementations, virtual remote control 124 may be rendered at least partially transparent so that the user may see through the virtual remote control. For example, Figure 2 As shown in , the user can see the windowsill and the wall even though the user can perceive the virtual remote control 124 as being in front of the window and the wall. Parts of the virtual remote control (e.g., virtual UI devices such as buttons that can be activated or actuated by the user) can be rendered less transparently than other parts (e.g., the body or frame), so that the virtual remote control 124 blocks less of the background environment.

[0041] The user may also move the rendering position, size, or orientation of the virtual remote control 124. For example, the user may move the virtual remote control 124 up / down, left / right, etc., closer to (or farther away from) the user. The user may also fix the rendering position of the virtual remote control 124 to be a certain distance from the user in the user environment or to be located at a specific position in the user environment (e.g., appearing to the user in three dimensions).

[0042] The user may cause a UI event, such as actuation or activation of a virtual UI device (e.g., a button) of the virtual remote control 124, by using a pointer (e.g., a finger, fingertip, or stylus) that interacts with the ARD or a device in the user's environment (e.g., the television 104). The ARD may use the NN to determine such activations of virtual UI devices or UI events with respect to the virtual UI devices and the pointer. The NN may be trained using images with representations of the UI devices, which will be described in further detail below, which may be different from the representations of the UI devices shown to the user by the ARD. Representations of the UI devices used to render onto one or more images to train the NN may include high-contrast concentric shapes (or shapes with similar or identical centers of gravity). Such representations of the UI devices may be rendered onto a captured image (e.g., a monoscopic image, a stereoscopic image pair, or a multi-view image set). While the user is interacting with the virtual UI device using the NN, the ARD may determine that a UI event has occurred by processing the pointer image captured using the ARD's outward-facing camera. Before the NN processes the image to determine the UI event, a standard representation of the UI device may be rendered onto the captured image perceived by the user.

[0043] In some implementations, the representations of UI devices used to train the NN can be advantageously standardized so that similar UI devices have similar representations when rendered onto images used to train the NN. For example, in some implementations, the standard representation of a UI device is a drawable renderable representation visualization that is used for any type of UI device of a particular type. The standard representation used by the NN can be, but is not necessarily, the same as the representation made visible to the user by the ARD. Devices of a particular type can be arranged according to industry standards or other logical groupings or classifications of device types (e.g., TV remote controls, or TV remote controls classified by manufacturer, or TV remote controls classified by manufacturer and TV category (e.g., LCD display, LED display, diagonal size, price, etc.)). In other cases, the standard representation can refer to functionality such as a standard representation of a push button, a standard representation of a slider, a standard representation of a touch screen, and the like. The standardized representation of a UI device used to train the NN can be referred to as a standard representation of a UI device. For example, when buttons of different types are rendered onto images used to train the NN, they can have similar representations. In some implementations, the normalized representations can be normalized to train multiple NNs for the same task, similar tasks (e.g., pressing a button or releasing a pressed button), or different tasks (e.g., recognizing different types of UI events such as using a finger touch or pointing).

[0044] exist Figure 2In the embodiment of the present invention, the user can use the ARD to perceive the environment 200. The environment 200 may include physical objects such as the TV 104, the physical remote control 108 for controlling the TV 104, the TV stand 112, and the window 116. The environment 200 may also include a virtual remote control 124. The virtual remote control 124 may simulate the functions of the physical remote control 108 to control the TV 104. For example, the layout and functions of the virtual UI devices such as the virtual buttons 132a of the virtual remote control 124 may be substantially the same as the physical buttons of the physical remote control 104.

[0045] The virtual remote controller 124 may include virtual UI devices, such as a virtual keyboard, virtual buttons, virtual switches, toggle switches or sliders, a virtual touch surface, or any components thereof (e.g., keyboard keys). These virtual UI devices may be part of a control panel 128 of the virtual remote controller 124. To interact with the virtual remote controller 124, a user may initiate a UI event (e.g., activate or deactivate) with respect to the virtual UI device. For example, a user may interact with the virtual UI device by touching, pressing, releasing, sliding up / down or left / right, moving along a track, or other types of motion in 3D space.

[0046] Upon actuation or activation of a virtual UI device, such as virtual button 132a of virtual remote control 124, ARD can communicate with television 104 as if it were a physical remote control 108. As an example, in Figure 2 In , the user can use a pointer (e.g., a finger of the right hand 120b or a stylus) to activate the virtual UI device. The ARD can use an outward-facing imaging system to image the position of the pointer. As further described below, based on the position of the pointer, the ARD can calculate which UI event (such as activation) is related to the virtual UI device and the pointer. Figure 2 In the example shown in , the ARD can determine that the user's right index finger is activating virtual button 132a. In addition to using gestures to activate virtual buttons, users can also use user input devices (such as a stylus, pointer, wand, or totem) to activate virtual buttons.

[0047] Once the ARD detects that the user has activated the virtual UI device of the virtual remote control 124, the ARD can accordingly send a signal to the corresponding device (e.g., the TV 104) via a signal generator such as an IR transmitter to instruct the device to perform an operation based on the virtual UI device by the user. For example, the user can touch the virtual button 132a on the virtual remote control 124. If the virtual button 132a is associated with increasing the volume of the TV 104, the ARD can accordingly generate a signal (such as an IR signal generated by an IR transmitter on the ARD) and transmit the signal to the TV 104 (which may have an IR detector), thereby causing the TV 104 to increase its volume. The signal generated by the ARD can be the same as the signal generated by the corresponding physical remote control.

[0048] In the case where the signal is a line of sight signal (such as an IR signal that must be pointed at the IR detector of the television), the transmitter on the ARD may need to be pointed at the device (just as a physical remote control must be pointed at its associated device). Advantageously, the ARD can be configured to determine whether a requested command (e.g., increasing the volume of the television 104 or changing the channel) has occurred (e.g., by using a microphone on the ARD to determine an increase in sound intensity or using an outward-facing camera to determine that the display of the television 104 has changed, respectively). If the effect of the command is not produced by the device being controlled, the ARD can instruct the user to change the user's posture so that the transmitter of the ARD is pointed at the device being controlled. For example, the ARD can generate a visual graphic (or a voice instruction) to point the user's head at the device being controlled. In some implementations, such as when using wireless RF signals or ultrasonic acoustic signals, communication between the ARD and the device may not require clear line of sight, and the foregoing functionality may be optional.

[0049] When the user is done using the virtual remote control 104, the user can use a gesture to make the display of the virtual remote control 124 disappear. As an example, when the user is watching a TV show, the user may determine that he no longer needs the virtual remote control. As a result, the user can wave his hand to indicate that he has finished using the virtual remote control 124. The user can also press a virtual UI device (e.g., power button 136a) on the virtual remote control 124 to turn off the virtual remote control 124. In response, the ARD can stop displaying the virtual remote control 124 or display the virtual remote control 124 so that it is barely visible visually (e.g., with increased transparency), which can help the user select the virtual remote control 124 later.

[0050] In some implementations, the ARD may automatically or in response to a user command temporarily hide the virtual remote control 124 from the user's FOV, or move the virtual remote control 124 outside the user's FOV or to the edge of the user's FOV. For example, the ARD may also automatically hide the virtual remote control 124 if a threshold condition is met. The ARD may detect that no virtual UI device (such as a button of the virtual remote control 124) has been inactive for a threshold period of time (such as 10 seconds, 1 minute, 2 minutes, etc.). The ARD may gradually fade out the virtual remote control 124, such as by increasing the transparency of the virtual remote control 124. For example, as part of the fade-out process, the virtual remote control 124 may change from opaque to transparent. The ARD may also fade out the virtual remote control 124 by reducing the visibility of the virtual remote control 124. For example, the ARD may gradually reduce the size of the virtual remote control 124 or change the color of the virtual remote control 124 from a dark color to a light color.

[0051] Although some examples herein are described in the context of using a virtual remote control (e.g., activating one or more virtual UI devices, such as buttons of a virtual remote control) to control a physical device (e.g., a physical television), this is for illustration only and is not intended to be limiting. Embodiments of virtual remote controls or virtual UI devices may additionally or alternatively be used to control virtual devices. For example, a user of an ARD may use a virtual remote control to control a virtual television rendered by the ARD and displayed to the user.

[0052] Example rendering of a UI device representation

[0053] Disclosed herein are systems and methods for generating a training set (including example inputs and target outputs), training a neural network (NN) using the training set, and using the trained NN. The topology of the NN can be any functional topology, such as Alex-Net or its derivatives. The topology of the NN can include a recurrent network that can be used to provide temporal context for class classification.

[0054] The NN may be trained using a classified input data set. For example, different UI events (e.g., a virtual UI device is activated or not activated) may correspond to different category values ​​in the input data set. In some embodiments, the input data set may include some quantitative values. The NN may be trained to recognize two or more states corresponding to different classification values, such as a state of a virtual UI device being activated and a state of an inactivated virtual UI device. For example, if the virtual UI device is a button, the NN may be trained to recognize states: "pressed" and "not pressed". Other states (e.g., "touched") are also possible.

[0055] Example inputs and target outputs may include images with representations of virtual UI devices rendered on them. The representations of UI devices rendered on the images may not be user-visible representations, but rather drawable renderable visualizations that may be used for specific types of UI devices. For example, in some implementations, all button-type UI devices may be represented as pure white disks.

[0056] In some embodiments, the trained NN can use camera images to determine whether a UI event has occurred. The camera can be positioned so as to see a pointer (e.g., a finger or stylus), and the captured image can be provided as input to the NN. The output of the NN can include a determination as to whether a UI event has occurred, and if so, the specific UI event that has occurred.

[0057] In some embodiments, the UI device, in its representation used to train the NN, may be rendered onto each image input to the NN, both in training and in operation. "Rendered onto" may mean that the UI device, in its representation, is rendered to appear exactly as it would at the virtual location if viewed from the position of the camera used to capture the image (and utilizing, alone or in combination, a viewing cone appropriate to the measurements of the camera and its associated lens; the intrinsic parameters may include, for example, the camera's focal length, principal point offset, and axis skew).

[0058] In this way, the image is an augmented image, containing both a pointer and a representation of a virtual UI device. When the NN is trained, all images presented may have such representations of one or more virtual UI devices rendered on each image. Each such image may be associated with its state (e.g., "button pressed"). Negative examples may similarly be provided to the DNN during the training process, where a UI device may be rendered on an image, but the virtual UI device may not be activated (e.g., "button not pressed").

[0059] Figure 2 An example of an augmented environment 200 is shown, which includes a physical environment perceived by a user of an augmented reality device and a virtual remote control. A user interface including the remote control 124 and virtual UI events (e.g., virtual button 132a) may be visually presented to the user in any suitable graphical form. The DNN may be provided with images perceived by the user and by a camera (such as Fig.11 The modified image may include an annotated virtual UI device, such as a standardized representation of the UI device, rather than the version actually shown to the user ( Figure 3 ). Figure 3An example of a representation of buttons of a virtual remote control rendered onto an image captured by an imaging device of an augmented reality device, the image corresponding to the physical environment perceived by the user is shown.

[0060] In some embodiments, a representation of a button or UI device may be rendered without occlusion so that the representation may appear “on top” of the pointer (e.g., from the perspective of the image) ( Figure 3 ). For example, in Figure 3 132b of the virtual button 132a is shown in FIG. 132b, occluding the user's fingertip. The representation 132b of the virtual button 132a is displayed as "on top" of the user's fingertip. Regardless of whether the NN is trained using images of representations rendered with or without occlusion, the NN can advantageously determine UI events in images of representations rendered with or without occlusion. Examples of representations of UI devices are provided below with reference to FIG. Figures 5A-5D Give a description.

[0061] In some embodiments, the position of the pointer and the virtual UI device can also be considered. In some embodiments, the focus can be used to determine the interaction or intersection between the pointer and the virtual UI device. For the interaction and UI events to occur, the pointer tip and the UI device may need to be in the same focus state.

[0062] In some embodiments, multiple virtual UI devices may be presented to the user simultaneously. Figure 2 As shown in , the user can perceive multiple virtual UI devices of the remote control 124, such as virtual buttons 132a, 136a. The image with the representation rendered thereon may include representations of multiple buttons (e.g., Figure 3 132b, 136b of the virtual buttons 132a, 136a shown in FIG. 132b). The specific virtual UI device involved in the UI event can be determined. For example, when generating a training set, the virtual UI device involved in the UI can be determined ex post facto even after the image is captured. Since the required accuracy may be small, a method based on pointer positioning with respect to the virtual UI device can be used for this ex post facto determination.

[0063] In some embodiments, the training set may include pairs of stereo images.For example, a forward-facing imaging system of an ARD may include two or more imaging devices (eg, cameras) for capturing stereo images. Figure 4A and Figure 4BAn example stereoscopic image pair 400a, 400b captured by two imaging devices of an augmented reality device with representations of buttons rendered on the images is shown. For example, representations 132b1, 136b1 of virtual buttons 132a, 136a are shown in image 400a, and corresponding representations 132b2, 136b2 of virtual buttons 132a, 136a are shown in image 400b. The representations 132b1, 132b2 of virtual button 132a in the pair of images 400a, 400b appear to be "on top" of a user's fingertip. Figures 4A-4B , as seen by the visible difference between the positions of the left hand 120a and the television 104 in images 400a, 400b, the left hand 120a is closer to the user than the right hand 120b, and the television 104 and the background (e.g., a wall) are farther away from the right hand 120b. Because the right hand 120b and the UI device and its representation (e.g., virtual buttons 132a, 136a and their corresponding representations 132b1, 132b2, 136b1, 136b2) are at the same depth, there is no relative difference between the right hand 120b and the UI device, and their representations exist. In some implementations, a single field of view image set can be used in training the NN and using the NN to determine UI events. For example, the forward-facing imaging system of the ARD may include multiple imaging devices (e.g., cameras) for capturing multi-field of view images. Such a NN may advantageously have better results expected in this case.

[0064] Example representation of UI devices

[0065] Figures 5A-5D An example representation of a user interface (UI) device is shown. To promote a unique image of the UI device, the representation of the UI device used to train the NN can include high-contrast concentric shapes (or shapes with similar or identical centers of gravity). Figure 5A A representation 500a of a button is shown having three concentric rings 504a-512a rendered on a fingertip 516a without being obscured by the fingertip 516a. The representation 500a may include a black ring 508a with a white ring 504a outside the black ring 508a and a white ring 512a inside the black ring 508a. Figure 5A (and Figures 5B-5D ), the dashed lines shown adjacent to white rings 504a and 512a are not part of the representation of button 500a. Figure 5A , depicting a white ring surrounding representation 500a being rendered from the portion of the image captured by the outward-facing imaging system.

[0066] Figure 5BA representation 500b of a button is shown having two concentric rings 504b-508b rendered on a fingertip 516b without being obscured by the fingertip 516b. The representation 500b may include a black ring 508b with a white ring 504b outside the black ring 508b and no white ring inside the black ring 508b. Figure 5C A representation 500c of a button is shown having two rings 504c-508c and a circle 512c rendered on a fingertip 516c without being obscured by the fingertip 516c. The representation 500c may include a black ring 508c with a white ring 504c outside the black ring 508c and a white ring inside the black ring 508c.

[0067] Figure 5D A representation 500d of a button is shown having three concentric rings 504d-512d rendered with alpha blending on a fingertip 516d without being obstructed by the fingertip 516d. The representation 500d may include a black ring 508d with a white ring 504d on the outside of the black ring 508d, and a white ring 512d on the inside of the black ring 508a, with the white ring 512d rendered with alpha blending such as a ring that becomes increasingly transparent (e.g., an alpha transparency value approaches zero) as the distance from the black ring increases. The four dashed lines shown on top of the white ring 504d are not part of the representation of the button 500d. These dashed lines are not part of the representation of the button 500d. Figure 5D 512d, with different transparency values ​​depicting the regions of the white ring 512d. In some embodiments, the four regions of the white ring 512d can be considered as four concentric rings 512d1-512d4 with different transparency values.

[0068] In some implementations, rendering representations on images used to train the NN may or may not take into account occlusion of the pointer. For example, images of the pointer may be captured stereoscopically with structured light projection, a time-of-flight camera, or a combination thereof. Based on these images, a depth field may be associated with any image. This depth field may be used to provide occlusion for the representation of the UI device when rendering both training data and input data.

[0069] Example Machine Learning Models and NNs

[0070] In some embodiments, the machine learning model includes a classification model. The classification model may include a supervised classification model, a semi-supervised classification model, an unsupervised classification model, or a combination thereof. The machine learning model may include a neural network, a linear regression model, a logistic regression model, a decision tree, a support vector machine, a naive Bayesian network, a k-nearest neighbor (KNN) model, a k-means model, a random forest model, or any combination thereof. The machine learning model may include an association rule learning model, an inductive logic programming model, an enhanced learning model, a feature learning model, a similarity learning model, a sparse dictionary learning model, a genetic algorithm model, a rule-based machine learning model, a learning classifier system model, or any combination thereof.

[0071] A layer of a neural network (NN) such as a deep neural network (DNN) may apply a linear or nonlinear transformation to its input to generate its output. A deep neural network layer may be a normalization layer, a convolutional layer, a softsign layer, a rectified linear layer, a cascade layer, a pooling layer, a recursive layer, a class initial layer, or any combination thereof. A normalization layer may normalize the brightness of its input to generate its output by, for example, L2 normalization. For example, a normalization layer may normalize the brightness of multiple images relative to each other at once to generate multiple normalized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization may nonlinearly normalize the contrast of an image by normalizing a local region of the image on a per-pixel basis to have a mean of zero and a variance of one (or a mean and variance of other values). Local response normalization may normalize an image on a local input region to have a mean of zero and a variance of one (or a mean and variance of other values). Normalization layers can speed up the training process.

[0072] A convolutional layer may apply a set of kernels that convolve its input to generate its output. A softsign layer may apply a softsign function to its input. The softsign function (softsign(x)) may be, for example, (x / (1+|x|)). The softsign layer may ignore the effects of outliers for each element. A rectified linear layer may be a rectified linear layer unit (ReLU) or a parameterized rectified linear layer unit (PReLU). A ReLU layer may apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) may be, for example, max(0, x). A PReLU layer may apply a PReLU function to its input to generate its output. For example, if x ≥ 0, the PReLU function PReLU(x) may be x, and if x < 0, it may be ax, where a is a positive number. A cascade layer may cascade its input to generate its output. For example, a cascade layer may cascade four 5×5 images to generate a 20×20 image. A pooling layer may apply a pooling function that downsamples its input to generate its output. For example, a pooling layer may downsample a 20×20 image to a 10×10 image. Non-limiting examples of pooling functions include max pooling, average pooling, or min pooling.

[0073] At time point t, the recursive layer can calculate a hidden state s(t), and the recursive connection can provide the hidden state s(t) to the recursive layer at time t as an input at a subsequent time point t+1. The recursive layer can calculate its output at time t+1 based on the hidden state s(t) at time t. For example, the recursive layer can apply a softsign function to the hidden state s(t) at time t to calculate its output at time t+1. The hidden state of the recursive layer at time t+1 has the hidden state s(t) of the recursive layer at time t as its input. The recursive layer can calculate the hidden state s(t+1) by applying, for example, a ReLU function to its input. The class initiation layer may include one or more of a normalization layer, a convolutional layer, a softsign layer, a rectified linear layer (such as a ReLU layer and a PReLU layer), a cascade layer, a pooling layer, or any combination thereof.

[0074] In different implementations, the number of layers in the NN can be different. For example, the number of layers in the DNN can be 50, 100, 200, or more. The input types of the deep neural network layers can be different in different implementations. For example, a layer can receive the outputs of multiple layers as its input. The input of a layer can include the outputs of five layers. As another example, the input of a layer can include 1% of the layers of the NN. The output of a layer can be the input of multiple layers. For example, the output of a layer can be used as the input of five layers. As another example, the output of a layer can be used as the input of 1% of the layers of the NN.

[0075] The input size or output size of a layer can be very large. The input size or output size of a layer can be n×m, where n represents the width of the input or output, and m represents the height of the input or output. For example, n or m can be 11, 21, 31, or more. In different implementations, the channel size of the input or output of a layer can be different. For example, the channel size of the input or output of a layer can be 4, 16, 32, 64, 128, or more. In different implementations, the kernel size of a layer can be different. For example, the kernel size can be n×m, where n represents the width of the kernel, and m represents the height of the kernel. For example, n or m can be 5, 7, 9, or more. In different implementations, the stride size of a layer can be different. For example, the stride size of a deep neural network layer can be 3, 5, 7, or more.

[0076] In some embodiments, a NN may refer to multiple NNs that together compute the output of the NN. Different ones of the multiple NNs may be trained for different tasks. For example, different ones of the multiple NNs may be trained to determine the occurrence of different UI events with respect to similar types of virtual UI devices and pointers (e.g., different types of activated virtual UI devices, such as touch or pointing). As another example, different ones of the multiple NNs may be trained to determine the occurrence of similar UI events with respect to similar types of virtual UI devices and different pointers (e.g., a stylus or a fingertip). As a further example, different ones of the multiple NNs may be trained to determine the occurrence of similar UI events with respect to different types of virtual UI devices (e.g., buttons or sliders) and pointers. A processor (e.g., Fig. 9 The processor of the local processing and data module 924 in the multiple NNs can calculate the output of the NN in the multiple NNs to determine the output of the NN. For example, the output of the NN in the multiple NNs can include a likelihood score. The processor can determine the output of the NN including the multiple NNs based on the likelihood scores of the outputs of different NNs in the multiple NNs.

[0077] Example Neural Network Training Method

[0078] Figure 6 A flow chart of an illustrative method 600 for training a machine learning model (e.g., a neural network) to determine a user interface event using a representation of a user interface device is shown. At block 604, an image of a pointer may be received. The image may be associated with a virtual user interface (UI) device (e.g., a virtual button) at an image location in the image. The image may be associated with a UI event with respect to the UI device and the pointer (e.g., a button has been pressed).

[0079] The UI event may correspond to a state in a plurality of states of a virtual UI device. A plurality of states include activation or inactivation of a virtual UI device, such as touching, pressing, releasing, sliding up / down or left / right, moving along a track or other types of motion. The UI device may be a button, up / down button, fine-tuner, selector, radio button, radio button list, check box, picture frame, check box list, drop-down list, drop-down menu, selection list, list box, combo box, text box, slider, link, keyboard key, switch, slider, touch surface or a combination thereof. A UI pointer may be an object associated with a user or a part of a user, such as a pointer, pen, pencil, marker, fluorescent pen, a user's finger or a combination thereof.

[0080] At box 608, a representation of a virtual UI device can be rendered on the image at the image location to generate a training image. The representation of the virtual UI device can be different from the representation of the UI device shown to the user (e.g., a stylized button). In some cases, the representation is a standard representation as described herein. The representation of the virtual UI device rendered to the image can include a plurality of high-contrast shapes. In some embodiments, a plurality of high-contrast shapes include a plurality of high-contrast concentric shapes. Alternatively or additionally, the center of gravity of the shapes in the plurality of shapes can be within a threshold distance from each other. The threshold distance can be based on 0, 1, 2, 5, 10 or more pixels or percentages (e.g., 0%, 0.1%, 1%, 2% or more) of the size of the shapes in the plurality of shapes.

[0081] In some embodiments, a plurality of shapes may include a first shape and a second shape that are adjacent to each other. For example, the first shape may be inside or outside the second shape. A first shape among the plurality of shapes may be associated with a first color (e.g., black or dark). A second shape among the plurality of shapes may be associated with a second color (e.g., white or light). The second shape may be partially transparent. For example, the second shape may include a first region and a second region. The second region of the second shape may be further away from the first region of the second shape. The first region of the second shape may be associated with a first transparency value (e.g., alpha transparency value). The second region of the second shape is associated with a second transparency value. The second transparency value may be less than the first transparency value. For example, a second region that is further away than the first region may have a greater transparency value and be more transparent. A plurality of high-contrast shapes may include a third shape having a third color (e.g., white or light). The third shape may be adjacent to the first shape. For example, the third shape may be inside or outside the first shape.

[0082] At box 612, a training set including input data and corresponding target output data can be generated. The input data can include training images. The target output data can include UI events. In some embodiments, the training set can include a stereo image pair or a multi-field image set. For example, a first representation of a virtual UI device can be rendered on a first image of a stereo image pair. A second representation of a virtual UI device can be rendered on a second image of the pair of stereo images. The two representations can be different, the same, or have different sizes.

[0083] At box 616, a neural network (NN) can be trained using a training set to determine UI events. The training set may include a monoscopic image, a stereoscopic image pair, or a multi-view image set having a representation of a UI device for training the NN. The process of training the NN involves presenting input data of the training set and corresponding target output data to the network. Through the training process, the weights of the network can be incrementally or iteratively adjusted so that given specific input data in the training set, the output of the network becomes matched (e.g., as close as possible) to the target output corresponding to the specific input data.

[0084] Example method for UI event determination

[0085] Figure 7 A flow chart of an illustrative method for determining a user interface event using a representation of a user interface device using a machine learning model (e.g., a neural network) is shown. A user device such as a head-mounted or wearable ARD or display system can implement method 700. At box 704, the ARD can receive a neural network (NN) trained using a training set including training images. The training images can include representations of pointers and virtual UI devices. The training images can be associated with UI events about the virtual UI device and the pointer. For example, the NN can be a NN trained using method 600.

[0086] At box 708, the ARD may receive an image of the pointer. The image is associated with a virtual UI device at the image location. For example, when the virtual device is displayed to the user by the ARD, the image may be captured by a forward-facing imaging system of the ARD. The UI device may be a button, up and down button, spinner, selector, radio button, radio button list, check box, picture box, check box list, drop-down list, drop-down menu, selection list, list box, combo box, text box, slider, link, keyboard key, switch, slider, touch surface, or a combination thereof. The UI pointer may be an object associated with a user or a part of a user, such as a pointer, pen, pencil, marker, highlighter, a user's finger, or a combination thereof.

[0087] At box 712, ARD can render the representation of the virtual UI device to the image at the image position associated with the virtual UI device. As described in detail with reference to method 600, the representation of the virtual UI device can be rendered to the image at the image position to generate a training image. The representation of the virtual UI device may include a plurality of high-contrast shapes. In some embodiments, a plurality of high-contrast shapes include a plurality of high-contrast concentric shapes. Alternatively or additionally, the center of gravity of the shapes of the plurality of shapes may be within a threshold distance from each other. In some embodiments, the plurality of shapes may include adjacent shapes of different colors (e.g., black, dark, white, or light). The representation of the virtual UI device rendered by ARD at box 712 may be similar or identical to the representation of the UI device rendered at box 608 for generating a training image.

[0088] At box 716, the ARD can use the NN to determine UI events about the pointer in the image and the virtual UI device associated with the image. The UI event can correspond to a state in multiple states of the virtual UI device. The multiple states include activation or inactivation of the virtual UI device, such as touching, pressing, releasing, sliding up / down or left / right, moving along a track, or other types of motion. Optionally, the ARD can generate virtual content, virtual image information, or a modified version thereof based on the determined UI event, and cause the virtual content to be provided to the wearer of the ARD via its display. Additionally or alternatively, the ARD can perform an action based on the UI event. For example, the ARD can make a reference to the UI event. Figure 1 The depicted television 104 is turned on. A description of generating virtual content or performing actions based on UI events is provided in U.S. Patent Application No. 15 / 829,249, filed on December 1, 2017, entitled “VIRTUAL USER INPUT CONTROLS IN A MIXED REALITY ENVIRONMENT,” the contents of which are incorporated herein by reference in their entirety.

[0089] Sample augmented reality scene

[0090] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears to be, or can be perceived to be, real. Virtual reality "VR" scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual input; augmented reality "AR" scenarios typically involve the presentation of digital or virtual image information as an enhancement to the visualization of the real world around the user; or mixed reality "MR" scenarios, which typically involve the fusion of the real world with the virtual world to produce a new environment in which physical or virtual objects coexist and interact in real time. It turns out that the human visual perception system is very complex, and it is challenging to produce a VR, AR or MR technology that promotes a comfortable, natural feeling, rich presentation of virtual image elements and other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR and MR technologies.

[0091] Figure 8 An illustration of an augmented reality scene with certain virtual reality objects and certain actual reality objects viewed by a person is depicted. Figure 8 An augmented reality scene 800 is depicted in which a user of AR technology sees a real-world park-like setting 810 featuring people, trees, buildings in the background, and a concrete platform 820. In addition to these items, the user of the AR technology also perceives that he "sees" a robotic statue 830 standing on the real-world platform 820, as well as a cartoon-like avatar character 840 (e.g., Bumblebee) flying by that appears to be an anthropomorphic representation of Bumblebee, even though these elements do not exist in the real world.

[0092] In order for a three-dimensional (3D) display to produce a realistic sense of depth, and more specifically a simulated sense of surface depth, it is desirable for each point in the display's field of view to generate an accommodation response corresponding to its virtual depth. If the accommodation response to a displayed point does not correspond to the virtual depth of that point (as determined by the binocular depth cues of convergence and stereopsis), the human eye may experience accommodation conflict, resulting in imaging instability, detrimental eye strain, headaches, and, in the absence of accommodation information, an almost complete lack of surface depth.

[0093] VR, AR, and MR experiences may be provided by a display system having a display in which images corresponding to multiple depth planes are provided to a viewer. The images may be different for each depth plane (e.g., providing a slightly different representation of a scene or object) and may be focused on separately by the viewer's eyes, thereby providing depth cues to the user based on the eye accommodation required to bring different image features of the scene located at different depth planes into focus and / or based on observing different image features at different depth planes that are out of focus. As discussed elsewhere herein, such depth cues provide a credible perception of depth. To produce or enhance VR, AR, and MR experiences, the display system may use biometric information to enhance these experiences.

[0094] Example Wearable Display System

[0095] Fig. 9 An example of a wearable display system 900 is shown, which can be used to present a VR, AR, or MR experience to a display system wearer or viewer 904. The wearable display system 900 can be programmed to perform any application or embodiment described herein. The display system 900 includes a display 908, and various mechanical and electronic modules and systems that support the functions of the display 908. The display 908 can be coupled to a frame 912, which can be worn by a display system user, wearer, or viewer 904, and is configured to position the display 908 in front of the eyes of the wearer 904. The display 908 can be a light field display. In some embodiments, a speaker 916 is coupled to the frame 912 and positioned adjacent to the user's ear canal. In some embodiments, another speaker (not shown) is positioned adjacent to another ear canal of the user to provide stereo / formable sound control. The display 908 is operably coupled 920 to a local processing and data module 924, such as by a wired lead or a wireless connection, which can be mounted in various configurations, such as fixedly attached to the frame 912, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 904 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0096] The frame 912 may have one or more cameras attached or mounted on the frame 912 to obtain images of the wearer's (two) eyes. In one embodiment, the camera(s) may be mounted to the frame 912 in front of the wearer's eyes so that the eyes can be imaged directly. In other embodiments, the camera may be mounted along the handle of the frame 912 (e.g., near the wearer's ears). In such embodiments, the display 908 may be coated with a material that reflects light from the wearer's eyes back to the camera. The light may be infrared light, as iris features are prominent in infrared images.

[0097] The local processing and data module 924 may include a hardware processor, as well as non-transitory digital memory, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing, caching, and storage of data. The data may include data such as (a) captured from a sensor (which may, for example, be operably coupled to the frame 912 or otherwise attached to the wearer 904), such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope; and / or (b) data acquired and / or processed using a remote processing module 928 and / or a remote data repository 932, possibly for display 908 after such processing or acquisition. The local processing and data module 924 may be operably coupled to the remote processing module 928 and the remote data repository 932 by communication links 936 and / or 940, such as via a wired or wireless communication link, so that these remote processing modules 928, remote data repository 932 can be used as resources for the local processing and data module 924. The image capture device(s) may be used to capture eye images used in the eye image processing process.In addition, the remote processing module 928 and the remote data repository 932 may be operably coupled to each other.

[0098] In some embodiments, the remote processing module 928 may include one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device. Video data may be stored locally in the local processing and data module 924 and / or in a remote data repository 932. In some embodiments, the remote data repository 932 may include a digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 924, allowing for full autonomous use from the remote module.

[0099] In some implementations, the local processing and data module 924 and / or the remote processing module 928 are programmed to perform embodiments of the systems and methods described herein. An image capture device may capture video for a particular application (e.g., video of a wearer's eyes for an eye tracking application, or video of a wearer's hands or fingers for a gesture recognition application). The video may be analyzed by one or both of the local processing and data module 924, the remote processing module 928. In some cases, offloading at least some of the iris code generation to a remote processing module (e.g., in the "cloud") may improve efficiency or computational speed. Parameters of the systems and methods disclosed herein may be stored in the local processing and data module 924 and / or the remote processing module 928.

[0100] The results of the analysis may be used by one or both of the local processing and data module 924, the remote processing module 928 for further operations or processing. For example, in various applications, the wearable display system 900 may use biometric recognition, eye tracking, recognition or classification of gestures, objects, postures, etc. For example, the wearable display system 900 may analyze the captured video of the wearer's 904 hand and recognize the wearer's hand and the gesture of the wearable display system (e.g., picking up a real or virtual object, signaling approval or disapproval (e.g., thumbs up or "thumbs down"), etc.).

[0101] In some embodiments, the local processing and data module 924, the remote processing module 928, and the cloud system can perform some or all of the methods disclosed herein. For example, the remote processing module 928 or the cloud system can perform the above reference Figure 6 Method 600 is described for rendering a representation of a virtual user interface (UI) device (e.g., a button) onto an image of a pointer (e.g., a stylus) and using the image with the representation of the virtual UI device to train a neural network (NN) to determine UI events. As another example, the local processing and data module 924 may perform the above reference Figure 7 The method 700 described herein. The local processing and data module 924 may receive the NN from the remote processing module 928 or the cloud system. The local processing and data module 924 may use the NN to determine UI events related to the pointer in the image and the virtual UI device associated with the image.

[0102] The human visual system is complex, and providing a realistic sense of depth is challenging. Without being limited by theory, it is believed that a viewer of an object may perceive the object as three-dimensional due to a combination of vergence and accommodation. The vergence movement of the two eyes relative to each other (e.g., the rolling movement of the converging eyes toward or away from each other with the pupils fixed on an object) is closely related to the focusing (or "accommodation") of the eye lenses. Under normal circumstances, changing the focus of the eye lenses, or accommodating the eyes, to change the focus from one object to another object at another distance, will automatically result in a matching change in vergence for the same distance, in a relationship called the "accommodation-vergence reflex." Likewise, under normal circumstances, a change in vergence will trigger a matching change in adaptation. A display system that provides a better match between accommodation and vergence can form a more realistic or more comfortable three-dimensional image simulation.

[0103] Fig.10 Aspects of a method for simulating a three-dimensional image using multiple depth planes are shown. Fig.10, eyes 1002 and 1004 adapt to objects at various distances from eyes 1002 and 1004 on the z-axis so that these objects are in focus. Eyes 1002 and 1004 present a specific state of adaptation to enter the focused object at different distances along the z-axis. Therefore, it can be said that a specific state of adaptation is associated with a specific one of the depth planes 1006, associated with an associated focal length, so that when the eye is in an adapted state for that depth plane, the object or portion of the object in the specific depth plane is in a focused state. In some embodiments, a three-dimensional image can be simulated by providing a different representation of the image for each of the eyes 1002 and 1004, and also by providing a different representation of the image corresponding to each depth plane. Although shown as separate for clarity of illustration, it will be understood that, for example, the fields of view of eyes 1002 and 1004 can overlap as the distance along the z-axis increases. In addition, although shown as flat for ease of illustration, it should be understood that the contours of the depth planes can be curved in physical space so that all features in the depth planes are focused by the eyes in a specific state of adaptation. Without being limited by theory, it is believed that the human eye can generally interpret a limited number of depth planes to provide depth perception. Therefore, by providing the eye with a different representation of the image corresponding to each of these limited number of depth planes, a highly credible simulation of perceived depth can be achieved.

[0104] Example Waveguide Stack Assembly

[0105] Fig.11 An example of a waveguide stack for outputting image information to a user is shown. Display system 1100 includes a stack of waveguides, or a stacked waveguide assembly 1105, which can be used to provide a three-dimensional perception to an eye 1110 or a brain using a plurality of waveguides 1120, 1122, 1124, 1126, 1128. In some embodiments, display system 1100 may correspond to Fig. 9 System 900, Fig.11 Some portions of the system 900 are schematically shown in greater detail. For example, in some embodiments, the waveguide assembly 1105 may be integrated into Fig. 9 in display 908.

[0106] Continue to refer Fig.11 , the waveguide assembly 1105 may also include a plurality of features 1130, 1132, 1134, 1136 between the waveguides. In some embodiments, the features 1130, 1132, 1134, 1136 may be lenses. In some embodiments, the features 1130, 1132, 1134, 1136 may not be lenses. Instead, they may be spacers (e.g., cladding and / or structures for forming air gaps).

[0107] The waveguides 1120, 1122, 1124, 1126, 1128 and / or the plurality of lenses 1130, 1132, 1134, 1136 may be configured to send image information to the eye at various levels of wavefront curvature or light divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. An image injection device 1140, 1142, 1144, 1146, 1148 may be used to inject image information into the waveguides 1120, 1122, 1124, 1126, 1128, each of which may be configured to distribute incident light across each respective waveguide for output toward the eye 1110. Light exits the output surfaces of the image injection devices 1140, 1142, 1144, 1146, 1148 and is injected into the corresponding input edges of the waveguides 1120, 1122, 1124, 1126, 1128. In some embodiments, a single beam (e.g., a collimated beam) can be injected into each waveguide to output an entire field of cloned collimated beams directed toward the eye 1110 at a specific angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0108] In some embodiments, image injection devices 1140, 1142, 1144, 1146, 1148 are separate displays, each of which generates image information for injection into corresponding waveguides 1120, 1122, 1124, 1126, 1128. In some other embodiments, image injection devices 1140, 1142, 1144, 1146, 1148 are outputs of a single multiplexed display that may deliver image information to each of image injection devices 1140, 1142, 1144, 1146, 1148, for example, via one or more light conduits, such as fiber optic cables.

[0109] Controller 1150 controls the operation of stacked waveguide assembly 1105 and image injection devices 1140, 1142, 1144, 1146, 1148. In some embodiments, controller 1150 includes programming (e.g., instructions in a non-transitory computer readable medium) that regulates the timing and provision of image information to waveguides 1120, 1122, 1124, 1126, 1128. In some embodiments, controller 1150 can be a single integral device, or a distributed system connected by wired or wireless communication channels. In some embodiments, controller 1150 can be a local processing and data module 924 or a remote processing module 928 (in Fig. 91158). In some embodiments, the controller can communicate with an inward-facing imaging system 1152 (e.g., a digital camera), an outward-facing imaging system 1154 (e.g., a digital camera), and / or a user input device 1156. The inward-facing imaging system 1152 (e.g., a digital camera) can be used to capture an image of the eye 1110, for example to determine the size and / or orientation of the pupil of the eye 1110. The outward-facing imaging system 1154 can be used to image a portion of the world 1158. A user can input commands to the controller 1150 via the user input device 1156 to interact with the display system 1100.

[0110] The waveguides 1120, 1122, 1124, 1126, 1128 may be configured to propagate light within each respective waveguide by total internal reflection (TIR). The waveguides 1120, 1122, 1124, 1126, 1128 may each be planar or have another shape (e.g., curved) with major top and bottom surfaces and edges extending between those major top and bottom surfaces. In the configuration shown, the waveguides 1120, 1122, 1124, 1126, 1128 may each include a light extraction optical element 1160, 1162, 1164, 1166, 1168 configured to extract light from the waveguides by redirecting light propagating in each respective waveguide out of the waveguide to output image information to the eye 1110. The extracted light may also be referred to as outcoupled light, and the light extraction optical element may also be referred to as an outcoupling optical element. The extracted light beam is output by the waveguide at the location where the light propagating in the waveguide strikes the light redirecting element. The light extraction optical elements 1160, 1162, 1164, 1166, 1168 can be, for example, reflective and / or diffractive optical features. Although shown as being disposed on the bottom major surface of the waveguides 1120, 1122, 1124, 1126, 1128 for ease of description and clarity of the drawings, in some embodiments, the light extraction optical elements 1160, 1162, 1164, 1166, 1168 can be disposed on the top and / or bottom major surfaces, and / or can be disposed directly in the volume of the waveguides 1120, 1122, 1124, 1126, 1128. In some embodiments, the light extraction optical elements 1160, 1162, 1164, 1166, 1168 may be formed as a layer of material that is attached to a transparent substrate to form the waveguides 1120, 1122, 1124, 1126, 1128. In some other embodiments, the waveguides 1120, 1122, 1124, 1126, 1128 may be a single piece of material and the light extraction optical elements 1160, 1162, 1164, 1166, 1168 may be formed on a surface and / or in the interior of the piece of material.

[0111] Continue to refer Fig.11 As described herein, each waveguide 1120, 1122, 1124, 1126, 1128 is configured to output light to form an image corresponding to a specific depth plane. For example, the waveguide 1120 closest to the eye can be configured to transmit collimated light injected into such waveguide 1120 to the eye 1110. The collimated light can represent an optical infinity focal plane. The next upward waveguide 1122 can be configured to emit collimated light through a first lens 1130 (e.g., a negative lens) before the collimated light can reach the eye 1110. The first lens 1130 can be configured to produce a slightly convex wavefront curvature so that the eye / brain interprets the light from the next upward waveguide 1122 as coming from the first focal plane closer to the eye 1110 from optical infinity. Similarly, the third upward waveguide 1124 outputs light that passes through both the first lens 1130 and the second lens 1132 before reaching the eye 1110. The combined optical power of the first lens 1130 and the second lens 1132 can be configured to produce another incremental wavefront curvature so that the eye / brain interprets light from the third waveguide 1124 as coming from a second focal plane that is further inward toward the person from optical infinity than light from the next upward waveguide 1122.

[0112] The other waveguide layers (e.g., waveguides 1126, 1128) and lenses (e.g., lenses 1134, 1136) are configured in a similar manner, with the highest waveguide 1128 in the stack sending its output through all lenses between it and the eye for a combined focus representing the focal plane closest to the person. To compensate for the stack of lenses 1130, 1132, 1134, 1136 when viewing / interpreting light from the world 1158 on the other side of the stacked waveguide assembly 1105, a compensating lens layer 1138 may be provided at the top of the stack to compensate for the combined focus of the lens stack 1130, 1132, 1134, 1136 below. Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light extraction optics 1160, 1162, 1164, 1166, 1168 of the waveguides 1120, 1122, 1124, 1126, 1128 and the focusing aspects of the lenses 1130, 1132, 1134, 1136 can be static (e.g., not dynamic or electro-active). In some alternative embodiments, one or both of the above can be dynamic using electro-active features.

[0113] Continue to refer Fig.11, the light extraction optical elements 1160, 1162, 1164, 1166, 1168 can be configured to both redirect light out of their respective waveguides and output the light with an appropriate amount of divergence or collimation for a particular depth plane associated with the waveguide. As a result, waveguides with different associated depth planes can have different configurations of light extraction optical elements that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light extraction optical elements 1160, 1162, 1164, 1166, 1168 can be volume or surface features that can be configured to output light at specific angles. For example, the light extraction optical elements 1160, 1162, 1164, 1166, 1168 can be volume holograms, surface holograms, and / or diffraction gratings. Light extraction optical elements such as diffraction gratings are described in U.S. Patent Publication No. 2015 / 0178939, published on June 25, 2015, which is incorporated herein by reference in its entirety. In some embodiments, features 1130, 1132, 1134, 1136, 1138 may not be lenses. Instead, they may simply be spacers (e.g., cladding and / or structures used to form an air gap).

[0114] In some embodiments, the light extraction optical elements 1160, 1162, 1164, 1166, 1168 are diffractive features that form a diffraction pattern, or "diffractive optical elements" (also referred to herein as "DOEs"). Preferably, the DOEs have a relatively low diffraction efficiency, so that only a portion of the light of the beam is deflected toward the eye 1110 at each intersection of the DOE, while the rest of the light continues to move through the waveguide via total internal reflection. The light carrying the image information is thus split into multiple related exit beams that exit the waveguide at multiple locations, and the result is a fairly uniform exit emission pattern toward the eye 1110 for that particular collimated beam that bounces within the waveguide.

[0115] In some embodiments, one or more DOEs can be switched between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE can include a polymer dispersed liquid crystal layer in which droplets include a diffraction pattern in a host medium, and the refractive index of the droplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the droplets can be switched to a refractive index that does not match the refractive index of the host medium (in which case the pattern actively diffracts incident light).

[0116] In some embodiments, the number and distribution of depth planes and / or depth of field can dynamically change based on the pupil size and / or orientation of the viewer's eye. In some embodiments, an inward-facing imaging system 1152 (e.g., a digital camera) can be used to capture an image of eye 1110 to determine the size and / or orientation of the pupil of eye 1110. In some embodiments, inward-facing imaging system 1152 can be attached to frame 912 (e.g., Fig. 9 ), and can be in electrical communication with a local processing and data module 924 and / or a remote processing module 928, which can process image information from the inward-facing imaging system 1152 to determine, for example, a pupil diameter or orientation of the eye of the user 904.

[0117] In some embodiments, an inward-facing imaging system 1152 (e.g., a digital camera) can observe the user's movements, such as eye movements and facial movements. The inward-facing imaging system 1152 can be used to capture an image of the eye 1110 to determine the size and / or orientation of the pupil of the eye 1110. The inward-facing imaging system 1152 can be used to obtain images for determining the direction in which the user is looking (e.g., eye gestures) or biometric identification of the user (e.g., via iris recognition). The images obtained by the inward-facing imaging system 1152 can be analyzed to determine the user's eye gestures and / or emotions, which the display system 1100 can use to determine which audio or visual content should be presented to the user. The display system 1100 can also use sensors such as an inertial measurement unit (IMU), an accelerometer, a gyroscope, etc. to determine head gestures (e.g., head position or head orientation). Head gestures can be used alone or in combination with eye gestures to interact with a stem track and / or present audio content.

[0118] In some embodiments, one camera may be used for each eye to determine pupil size and / or orientation of each eye, respectively, thereby allowing image information to be presented to each eye to be dynamically customized for that eye. In some embodiments, at least one camera may be used for each eye to determine pupil size and / or eye pose of each eye, respectively, thereby allowing image information to be presented to each eye to be dynamically customized for that eye. In some other embodiments, pupil diameter and / or orientation of only a single eye 1110 (e.g., only a single camera is used per pair of eyes) is determined and assumed to be similar for both eyes of the viewer 904.

[0119] For example, the depth of field can vary inversely with the pupil size of the viewer. As a result, as the pupil size of the viewer's eye decreases, the depth of field increases, so that a plane that was not discernible because the plane was located beyond the focal depth of the eye may become discernible and become more focused as the pupil size decreases and the depth of field increases accordingly. Similarly, the number of spaced-apart depth planes used to present different images to the viewer can be reduced as the pupil size decreases. For example, a viewer may not be able to clearly perceive details of both a first depth plane and a second depth plane at one pupil size without adjusting the eye's accommodation from one depth plane to another depth plane. However, both depth planes may be simultaneously in full focus for the user at another pupil size without changing the accommodation.

[0120] In some embodiments, the display system can change the number of waveguides that receive image information based on a determination of pupil size and / or orientation, or based on receiving an electrical signal indicative of a particular pupil size and / or orientation. For example, if the user's eye cannot distinguish between two depth planes associated with two waveguides, the controller 1150 can be configured or programmed to stop providing image information to one of the waveguides. Advantageously, this can reduce the processing burden on the system, thereby increasing the responsiveness of the system. In embodiments where the DOE of a waveguide is switchable between on and off states, the DOE can be switched to the off state when the waveguide does receive image information.

[0121] In some embodiments, it may be desirable to have the exit beam satisfy the condition of having a diameter less than the diameter of the viewer's eye. However, satisfying this condition may be a challenge given the variability of the viewer's pupil size. In some embodiments, this condition is satisfied over a wide range of pupil sizes by varying the size of the exit beam in response to a determination of the viewer's pupil size. For example, as the pupil size decreases, the size of the exit beam may also decrease. In some embodiments, a variable aperture may be used to vary the size of the exit beam.

[0122] The display system 1100 may include an outward-facing imaging system 1154 (e.g., a digital camera) that images a portion of the world 1158. This portion of the world 1158 may be referred to as a field of view (FOV) and the imaging system 1154 is sometimes referred to as a FOV camera. The entire area available for viewing or imaging by the viewer 904 may be referred to as a field of view (FOR). The FOR may include 4π steradians of solid angle around the display system 1100. In some implementations of the display system 1100, the FOR may include substantially all of the solid angle around the user 904 of the display system 1100, because the user 904 may move their head and eyes to see objects around the user (in front of, behind, above, below, or to the side of the user). The images obtained from the outward-facing imaging system 1154 may be used to track gestures made by the user (e.g., gestures of a hand or finger), detect objects in the world 1158 in front of the user, etc.

[0123] A variety of computer vision techniques may be used to perform object recognition or detection. For example, the wearable system may analyze the imagery provided by the outward-facing imaging system 1154 (reference Fig.11 Description) to perform scene reconstruction, event detection, video tracking, object recognition (e.g., people or documents), gesture detection or recognition, object pose estimation, facial recognition (e.g., images of people or documents from the environment), learning, indexing, motion estimation or image analysis (e.g., identifying marks in documents such as photos, signatures, identification information, travel information, etc.), etc. One or more computer vision algorithms can be used to perform these tasks. The local processing and data module 924 and / or the remote processing module 928 and the remote data repository 932 can be programmed with an object recognizer that crawls the images and executes computer vision algorithms on the images. Non-limiting examples of computer vision algorithms include: Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Binary Robust Invariant Scalable Keypoints (BRISK), Fast Retinal Keypoints (FREAK), Viola-Jones algorithm, Eigenfaces method, Lucas-Kanade algorithm, Horn-Schunk algorithm, mean shift algorithm, visual simultaneous localization and mapping (vSLAM) technology, sequential Bayesian estimator (e.g., Kalman filter, extended Kalman filter, etc.), bundle adjustment, adaptive threshold (and other threshold techniques), iterative closest point (ICP), semi-global matching (SGM), semi-global block matching (SGBM), feature point histogram, various machine learning algorithms (such as, for example, support vector machine, k-nearest neighbor algorithm, naive Bayes, neural network (including convolutional or deep neural network) or other supervised / unsupervised models, etc.), etc.

[0124] Object recognition or detection may additionally or alternatively be performed by various machine learning algorithms. Once trained, the machine learning algorithm may be stored by the ARD (e.g., the local processing and data module 924 and / or the remote processing module 928 and the remote data repository 932). Some examples of machine learning algorithms may include supervised or unsupervised machine learning algorithms, including regression algorithms (such as, for example, ordinary least squares regression), instance-based algorithms (such as, for example, learning vector quantization), decision tree algorithms (such as, for example, classification and regression trees), Bayesian algorithms (such as, for example, naive Bayes), clustering algorithms (such as, for example, k-means clustering), association rule learning algorithms (such as, for example, a priori algorithms), artificial neural network algorithms (such as, for example, Perceptron), deep learning algorithms (such as, for example, Deep Boltzmann Machine or deep neural network), dimensionality reduction algorithms (such as, for example, principal component analysis), integration algorithms (e.g., stacked generalization), and / or other machine learning algorithms. In some embodiments, individual models may be customized for individual data sets. For example, a wearable device may generate or store a basic model. The base model can be used as a starting point for generating additional models that are specific to a data type (e.g., a specific user in a telepresence session), a data set (e.g., an additional set of images obtained by a user in a telepresence session), conditional situations, or other variations. In some embodiments, the wearable HMD can be configured to utilize a variety of techniques to generate models for aggregate data analysis. Other techniques may include using predefined thresholds or data values.

[0125] The display system 1100 may include a user input device 1156 through which a user can input commands to the controller 1150 to interact with the display system 400. For example, the user input device 1156 may include a touch pad, a touch screen, a joystick, a multi-degree-of-freedom (DOF) controller, a capacitive sensing device, a game controller, a keyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, a totem (e.g., acting as a virtual user input device), etc. In some cases, a user may use a finger (e.g., a thumb) to press or slide on a touch-sensitive input device to provide input to the display system 1100 (e.g., to provide user input to a user interface provided by the display system 1100). During use of the display system 1100, the user input device 1156 may be held by a user's hand. The user input device 1156 may communicate with the display system 1100 in wired or wireless communication.

[0126] Fig.12An example of an exit beam output by a waveguide is shown. One waveguide is shown, but it should be understood that other waveguides in the waveguide assembly 1105 can function similarly, where the waveguide assembly 1105 includes multiple waveguides. Light 1205 is injected into the waveguide 1120 at an input edge 1210 of the waveguide 1120 and propagates within the waveguide 1120 by total internal reflection (TIR). At the point where the light 1205 is incident on the diffractive optical element (DOE) 1160, a portion of the light exits the waveguide as an exit beam 1215. The exit beams 1215 are shown as being substantially parallel, but depending on the depth plane associated with the waveguide 1120, they can also be redirected to propagate to the eye 1110 at a certain angle (e.g., forming a divergent exit beam). It will be appreciated that substantially parallel exit beams may be indicative of a waveguide having light extraction optics that outcouple light to form an image that appears to be disposed at a depth plane at a greater distance (e.g., optical infinity) from the eye 1110. Other waveguides or other sets of light extraction optics may output an exit beam pattern with a greater divergence, which would require the eye 1110 to adapt to a closer distance to focus it on the retina and would be interpreted by the brain as light from a distance closer to the eye 1110 than optical infinity.

[0127] Fig.13 Another example of a display system 1100 is shown, which includes a waveguide device, an optical coupler subsystem that optically couples light to or from the waveguide device, and a control subsystem. The display system 1100 can be used to generate a multi-focal volume, image, or light field. The display system 1100 may include one or more primary planar waveguides 1304 (in Fig.13 The planar waveguides 1304 may be similar to those of FIG. Fig.11 The optical system may employ a distributed waveguide arrangement to provide a plurality of waveguides along a first axis (in Fig.13 The distributed waveguide device may, for example, include a distributed planar waveguide 1312 and at least one DOE 1316 (shown by a double-dashed line) associated with the distributed planar waveguide 1312. The distributed planar waveguide 1312 may be similar or identical to the primary planar waveguide 1304 in at least some aspects, with a different orientation therefrom. Likewise, the at least one DOE 1316 may be similar or identical to the DOE 1308 in at least some aspects. For example, the distributed planar waveguide 1312 and / or the DOE 1316 may be composed of the same material as the primary planar waveguide 1304 and / or the DOE 1308, respectively. Fig.13The optical system shown in can be integrated into Fig. 9 In the wearable display system 900 shown in .

[0128] The relayed and exit pupil expanded light is optically coupled from the distributed waveguide device into one or more primary planar waveguides 1304. The primary planar waveguides 1304 are arranged along a second axis (e.g., from Fig.13 The primary planar waveguide 1304 can extend the effective exit path of the light along the second axis (e.g., the X-axis). For example, the distributed planar waveguide 1312 can relay and extend the light along the vertical or Y-axis and pass the light to the primary planar waveguide 1304 that relays and extends the light along the horizontal or X-axis.

[0129] The display system 1100 may include one or more color light sources (e.g., red, green, and blue lasers) 1320, which may be optically coupled to the proximal end of a single-mode optical fiber 1324. The distal end of the optical fiber 1324 may be threaded or received by a hollow tube 1328 of piezoelectric material. The distal end protrudes from the tube 1328 as a freely fixed flexible cantilever 1332. The piezoelectric tube 1328 may be associated with four quadrant electrodes (not shown). The electrodes may be plated, for example, on the outside, outer surface, or outer periphery or outer diameter of the tube 1328. A core electrode (not shown) is also located in the core, center, inner periphery, or inner diameter of the tube 1328.

[0130] The driving electronics 1336, electrically coupled, for example, via wires 1340, drives the opposing electrode pairs to independently bend the piezoelectric tube 1328 in two axes. The protruding distal end of the optical fiber 1324 has a resonant mechanical mode. The frequency of the resonance can depend on the diameter, length, and material properties of the optical fiber 1324. By vibrating the piezoelectric tube 1328 near the first mechanical resonant mode of the optical fiber cantilever 1332, the optical fiber cantilever 1332 is vibrated and can meander through large deflections.

[0131] The tip of the fiber cantilever 1332 is biaxially scanned in an area that fills a two-dimensional (2-D) scan by stimulating resonant vibrations in two axes. The light emitted from the fiber cantilever 1332 forms an image by modulating the intensity of the (multiple) light sources 1320 synchronously with the scanning of the fiber cantilever 1332. A description of such an arrangement is provided in U.S. Patent Publication No. 2014 / 0003762, the entire contents of which are incorporated herein by reference.

[0132] Component 1344 of the optical coupler subsystem collimates the light emitted from the scanning fiber cantilever 1332. The collimated light is reflected by mirror 1348 into the narrow distribution planar waveguide 1312, which contains at least one diffractive optical element (DOE) 1316. The collimated light propagates vertically (relative to the optical fiber) along the distribution planar waveguide 1312 by total internal reflection. Fig.13 1304) and in doing so repeatedly intersects the DOE 1316. The DOE 1316 preferably has a low diffraction efficiency. This causes a portion of the light (e.g., 10%) to diffract toward the edge of the larger primary planar waveguide 1304 at each point of intersection with the DOE 1316, and a small portion of the light to continue down the length of the distributed planar waveguide 1312 on its original trajectory via TIR.

[0133] At each point of intersection with the DOE 1316, additional light is diffracted toward the entrance of the primary waveguide 1312. By splitting the incident light into multiple out-coupled groups, the exit pupil of the light is vertically expanded in the distributed planar waveguide 1312 by the DOE 1316. This vertically expanded light coupled out of the distributed planar waveguide 1312 enters the edge of the primary planar waveguide 1304.

[0134] Light entering the primary waveguide 1304 propagates horizontally (relative to the Fig.13 1304). As the light intersects the DOE 1308 at multiple points, it propagates horizontally along at least a portion of the length of the primary waveguide 1304 via TIR. The DOE 1308 may advantageously be designed or configured to have a phase distribution that is the sum of a linear diffraction pattern and a radially symmetric diffraction pattern to produce both deflection and focusing of the light. The DOE 1308 may advantageously have a low diffraction efficiency (e.g., 10%) such that at each intersection point of the DOE 1308, only a portion of the light of the beam is deflected toward the eye of the view, while the remainder of the light continues to propagate through the waveguide 1304 via TIR.

[0135] At each intersection point between the propagating light and the DOE 1308, a small portion of the light is diffracted toward an adjacent facet of the primary waveguide 1304, allowing the light to escape TIR and exit from a facet of the primary waveguide 1304. In some embodiments, the radially symmetric diffraction pattern of the DOE 1308 additionally imparts a focus level to the diffracted light, both shaping the optical wavefronts of the individual beams (e.g., imparting curvature) and steering the beams at angles that match the designed focus level.

[0136] Thus, these different paths can cause light to be coupled out of the main planar waveguide 1304 through various DOEs 1308 at different angles, focus levels, and / or produce different fill patterns at the exit pupil. The different fill patterns at the exit pupil can be advantageously used to create a light field display with multiple depth planes. Each layer in the waveguide assembly or a group of layers in the stack (e.g., 3 layers) can be used to generate a corresponding color (e.g., red, blue, green). Thus, for example, a first group of three adjacent layers can be used to produce red, blue, and green light at a first focal depth, respectively. A second group of three adjacent layers can be used to produce red, blue, and green light at a second focal depth, respectively. Multiple groups can be used to generate a full 3D or 4D color image light field with various focal depths.

[0137] Additional aspects

[0138] Example of a wearable display system using a trained neural network to determine UI events

[0139] In the first aspect, a wearable display system is disclosed. The wearable display system includes: an image capture device configured to capture an image including a pointer; a non-transitory computer-readable storage medium configured to store: an image, a virtual user interface (UI) device associated with the image at an image position on the image, and a neural network for determining UI events trained using: a training image associated with a training virtual UI device, the training image including a representation of the training virtual UI device and a training pointer, and training UI events about the training virtual UI device and the training pointer in the training image; a display configured to display the virtual UI device at a display position when the image capture device captures the image, wherein the image position is related to the display position; and a hardware processor in communication with the image capture device, the display, and the non-transitory computer-readable storage medium, the processor being programmed by executable instructions to: receive an image from the image capture device; render a representation of the virtual UI device onto the image at the image position; and determine UI events about the pointer in the image and the virtual UI device associated with the image using a neural network. The processor may generate virtual content based on the UI events, and cause the display to present the virtual content to a wearer of the wearable display system.

[0140] In a second aspect, according to the wearable display system of aspect 1, the processor is further programmed to generate virtual content (or virtual image information) based on the UI event; and enable the display to provide the virtual content to the wearer of the wearable display system.

[0141] In a third aspect, a wearable display system according to any one of aspects 1-2, wherein the processor is further programmed to perform an action based on a UI event (e.g., activating another device such as a TV, a car, etc., or connecting to another device such as a phone, a mobile device, an ARD, etc.).

[0142] In a 4th aspect, according to any one of aspects 1-3, the wearable display system, wherein the representation of the virtual UI device comprises a plurality of high contrast shapes.

[0143] In a fifth aspect, a wearable display system according to aspect 4, wherein the plurality of high contrast shapes comprises a plurality of high contrast concentric shapes.

[0144] In a 6th aspect, according to any one of aspects 4-5, the wearable display system, wherein the centers of gravity of the shapes of the plurality of shapes are within a threshold distance of each other.

[0145] In a 7th aspect, according to the wearable display system of aspect 6, the threshold distance is 0.

[0146] In aspect 8, a wearable display system according to any one of aspects 4-7, wherein a first shape among the multiple shapes is associated with a first color, wherein a second shape among the multiple shapes is associated with a second color, and wherein the first shape is adjacent to the second shape.

[0147] In a 9th aspect, according to the wearable display system of aspect 8, wherein the first color is black, and wherein the second color is white.

[0148] In a 10th aspect, according to any one of aspects 8-9, the wearable display system, wherein the second shape is partially transparent.

[0149] In an 11th aspect, according to the wearable display system of aspect 10, the first area of ​​the second shape is associated with a first transparency value, and the second area of ​​the second shape is associated with a second transparency value.

[0150] In a 12th aspect, according to the wearable display system of aspect 11, the second area of ​​the second shape is further away from the first area of ​​the second shape, and the second transparency value is less than the first transparency value.

[0151] In a 13th aspect, according to any one of aspects 8-12, the wearable display system, wherein the second shape is within the first shape.

[0152] In a 14th aspect, according to any one of aspects 8-13, a wearable display system, wherein a third shape of the plurality of shapes is associated with a third color, and wherein the first shape is adjacent to the first shape.

[0153] In aspect 15, according to the wearable display system of aspect 14, the third color is white.

[0154] In a 16th aspect, according to any one of aspects 14-15, the wearable display system, wherein the first shape is within the third shape.

[0155] In aspect 17, according to any one of aspects 1-16, the wearable display system, wherein the UI event corresponds to one of multiple states of the virtual UI device.

[0156] In an 18th aspect, according to the wearable display system of aspect 17, the multiple states include activation or inactivation of the virtual UI device.

[0157] In aspect 19, the wearable display system according to any one of aspects 1-18, wherein the virtual UI device is selected from the group consisting of: buttons, up and down buttons, spinners, selectors, radio buttons, radio button lists, check boxes, picture boxes, check box lists, drop-down lists, drop-down menus, selection lists, list boxes, combo boxes, text boxes, sliders, links, keyboard keys, switches, sliders, touch surfaces, or combinations thereof.

[0158] In a 20th aspect, according to any one of aspects 1-19, the wearable display system, wherein the UI pointer comprises an object associated with the user or a part of the user.

[0159] In a 21st aspect, according to the wearable display system of aspect 20, the object associated with the user includes a pointer, a pen, a pencil, a marker, a highlighter, or a combination thereof, and wherein the part of the user includes a finger of the user.

[0160] 1. An example of a computer system for generating training data for training a machine learning model (e.g., a neural network) example

[0161] In aspect 22, a system for training a neural network to determine user interface events is disclosed. The system includes: a computer readable memory storing executable instructions; and one or more processors programmed by the executable instructions to at least: receive a plurality of images, wherein one of the plurality of images includes a pointer in a plurality of pointers, wherein the image is associated with a virtual UI device in a plurality of virtual user interface (UI) devices at an image location on the image, and wherein the image is associated with a UI event in a plurality of UI events about the virtual UI device and the pointer in the image; render a representation of the virtual UI device onto the image at the image location to generate a training image; and generate a training set including input data and corresponding target output data, wherein the input data includes the training image, and wherein the corresponding target output data includes the UI event.

[0162] In aspect 23, the system of aspect 22, wherein the one or more processors are further programmed to train a machine learning model (e.g., a neural network) using a training set to determine UI events associated with a virtual UI device and a pointer.

[0163] In aspect 24, the system of any of aspects 22-23, wherein the representation of the virtual UI device comprises a plurality of high contrast shapes.

[0164] In a 25th aspect, the system of aspect 24, wherein the plurality of high contrast shapes comprises a plurality of high contrast concentric shapes.

[0165] In aspect 26, the system of any of aspects 24-25, wherein the centers of gravity of the shapes of the plurality of shapes are within a threshold distance of each other.

[0166] In aspect 27, the system of aspect 26, wherein the threshold distance is 0.

[0167] In aspect 28, the system of any of aspects 24-27, wherein a first shape in the plurality of shapes is associated with a first color, wherein a second shape in the plurality of shapes is associated with a second color, and wherein the first shape is adjacent to the second shape.

[0168] In a 29th aspect, the system of aspect 28, wherein the first color is black, and wherein the second color is white.

[0169] In a 30th aspect, the system of any of aspects 28-29, wherein the second shape is partially transparent.

[0170] In a 31st aspect, the system of aspect 30, wherein the first region of the second shape is associated with a first transparency value, and wherein the second region of the second shape is associated with a second transparency value.

[0171] In a 32nd aspect, the system of aspect 31, wherein the second region of the second shape is further away from the first region of the second shape, and wherein the second transparency value is less than the first transparency value.

[0172] In a 33rd aspect, the system of any of aspects 28-32, wherein the second shape is within the first shape.

[0173] In a 34th aspect, the system of any of aspects 28-33, wherein a third shape in the plurality of shapes has a third color, and wherein the first shape is adjacent to the first shape.

[0174] In a 35th aspect, the system of aspect 34, wherein the third color is white.

[0175] In a 36th aspect, the system of any of aspects 34-35, wherein the first shape is within the third shape.

[0176] In aspect 37, the system of any of aspects 22-36, wherein the UI event corresponds to one of a plurality of states of the virtual UI device.

[0177] In a 38th aspect, the system of aspect 37, wherein the plurality of states comprises activation or inactivation of the virtual UI device.

[0178] In aspect 39, the system of any one of aspects 22-38, wherein the plurality of virtual UI devices comprises buttons, up and down buttons, spinners, selectors, radio buttons, radio button lists, check boxes, picture boxes, check box lists, drop-down lists, drop-down menus, selection lists, list boxes, combo boxes, text boxes, sliders, links, keyboard keys, switches, sliders, touch surfaces, or combinations thereof.

[0179] In aspect 40, the system of any of aspects 22-39, wherein the plurality of UI pointers comprises objects associated with the user or a portion of the user.

[0180] In a 41st aspect, the system of aspect 40, wherein the object associated with the user comprises a pointer, a pen, a pencil, a marker, a highlighter, or a combination thereof, and wherein the part of the user comprises a finger of the user.

[0181] Example of a method for determining UI events using a trained neural network

[0182] In aspect 42, a method for determining UI events using a neural network is disclosed. The method is under the control of a hardware processor and includes: accessing a neural network for determining UI events trained using: a training image associated with a training virtual UI device, the training image including a representation of the training virtual UI device and a training pointer, and training UI events about the training virtual UI device and the training pointer in the training image; receiving an image including a pointer, wherein a virtual user interface (UI) device is associated with an image at an image location on the image, wherein when the image is captured (e.g., by an image capture device of an ARD), the virtual UI device is displayed at a display location (e.g., a display location on an ARD display), and wherein the image location is related to the display location; receiving an image from an image capture device; rendering a representation of the virtual UI device onto the image at the image location; and determining UI events about the pointer in the image and the virtual UI device associated with the image using the neural network.

[0183] In aspect 43, the method according to aspect 42 further comprises: generating virtual content (or virtual image information) based on the UI event; and optionally causing the virtual content to be displayed.

[0184] In aspect 44, the method according to any one of aspects 42-43 further includes: performing an action based on a UI event (for example, activating another device such as a TV, a car, etc., or connecting to another device (such as a phone, a mobile device, an ARD, etc.)

[0185] In aspect 45, the method of any of aspects 42-44, wherein the representation of the virtual UI device comprises a plurality of high contrast shapes.

[0186] In aspect 46, the method of aspect 45, wherein the plurality of high contrast shapes comprises a plurality of high contrast concentric shapes.

[0187] In aspect 47, a method according to any of aspects 45-46, wherein the centers of gravity of the shapes of the plurality of shapes are within a threshold distance of each other.

[0188] In aspect 48, the method of aspect 47, wherein the threshold distance is 0.

[0189] In a 49th aspect, the method of any of aspects 45-48, wherein a first shape in the plurality of shapes is associated with a first color, wherein a second shape in the plurality of shapes is associated with a second color, and wherein the first shape is adjacent to the second shape.

[0190] In a 50th aspect, the method of aspect 49, wherein the first color is black, and wherein the second color is white.

[0191] In a 51st aspect, the method of any of aspects 49-50, wherein the second shape is partially transparent.

[0192] In aspect 52, the method of aspect 51, wherein the first region of the second shape is associated with a first transparency value, and wherein the second region of the second shape is associated with a second transparency value.

[0193] In a 53rd aspect, the method of aspect 52, wherein the second region of the second shape is further away from the first region of the second shape, and wherein the second transparency value is less than the first transparency value.

[0194] In a 54th aspect, the method of any one of aspects 49-53, wherein the second shape is within the first shape.

[0195] In a 55th aspect, the method of any of aspects 49-54, wherein a third shape in the plurality of shapes is associated with a third color, and wherein the first shape is adjacent to the first shape.

[0196] In a 56th aspect, according to the method of aspect 55, wherein the third color is white.

[0197] In aspect 57, the method of any of aspects 55-56, wherein the first shape is within the third shape.

[0198] In aspect 58, the method of any of aspects 42-57, wherein the UI event corresponds to one of a plurality of states of the virtual UI device.

[0199] In aspect 59, the method of aspect 58, wherein the plurality of states comprises activation or inactivation of the virtual UI device.

[0200] In aspect 60, a method according to any one of aspects 42-59, wherein the virtual UI device is selected from the group consisting of: a button, an up and down button, a spinner, a selector, a radio button, a radio button list, a check box, a picture box, a check box list, a drop-down list, a drop-down menu, a selection list, a list box, a combo box, a text box, a slider, a link, a keyboard key, a switch, a slider, a touch surface or a combination thereof.

[0201] In aspect 61, a method according to any of aspects 42-60, wherein the UI pointer comprises an object associated with the user or a part of the user.

[0202] In aspect 62, the method of aspect 61, wherein the object associated with the user comprises a pointer, a pen, a pencil, a marker, a highlighter, or a combination thereof, and wherein the part of the user comprises a finger of the user.

[0203] Example of a method for training a machine learning model (e.g., a neural network)

[0204] In aspect 63, a method for training a neural network to determine user interface events is disclosed. The method is under the control of a hardware processor and includes: receiving a plurality of images, wherein a first image of the plurality of images includes a first representation of a pointer of a plurality of pointers, wherein the first image is associated with a first representation of a virtual user interface (UI) device of a plurality of virtual UI devices at a first image position in the first image, and wherein the first image is associated with a UI event of a plurality of UI events about the virtual UI device and the pointer in the first image; rendering the first representation of the virtual UI device onto the first image at the first image position to generate a first training image; generating a training set including input data and corresponding target output data, wherein the input data includes the first training image, and wherein the corresponding target output data includes the UI event; and using the training set to train a neural network for determining UI events associated with the virtual UI device and the pointer.

[0205] In aspect 64, a method according to aspect 63, wherein a second image in the plurality of images comprises a second representation of the pointer, wherein the second image is associated with a second representation of the virtual UI device at a second image location in the second image, and wherein the second image is associated with a UI event.

[0206] In a 65th aspect, according to the method of aspect 64, wherein the first image and the second image form a stereo pair.

[0207] In aspect 66, the method of aspect 64, wherein the first image and the second image are images of a multi-field-of-view image set.

[0208] In aspect 67, the method according to any one of aspects 64-66 further comprises: rendering a second representation of the virtual UI device onto a second image at a second image position to generate a second training image, wherein the input data comprises the second training image.

[0209] In aspect 68, the method of any of aspects 63-67, wherein the first representation of the virtual UI device comprises a plurality of high contrast shapes.

[0210] In a 69th aspect, the method of aspect 68, wherein the plurality of high contrast shapes comprises a plurality of high contrast concentric shapes.

[0211] In aspect 70, the method of any of aspects 68-69, wherein the centers of gravity of the shapes of the plurality of shapes are within a threshold distance of each other.

[0212] In aspect 71, the method of aspect 70, wherein the threshold distance is 0.

[0213] In a 72nd aspect, the method of any of aspects 68-71, wherein a first shape in the plurality of shapes is associated with a first color, wherein a second shape in the plurality of shapes is associated with a second color, and wherein the first shape is adjacent to the second shape.

[0214] In a 73rd aspect, the method of aspect 72, wherein the first color is black, and wherein the second color is white.

[0215] In a 74th aspect, the method of any of aspects 72-73, wherein the second shape is partially transparent.

[0216] In aspect 75, the method of aspect 74, wherein the first area of ​​the second shape is associated with a first transparency value, and wherein the second area of ​​the second shape is associated with a second transparency value.

[0217] In a 76th aspect, according to aspect 75, the method of aspect 75, wherein the second area of ​​the second shape is further away from the first area of ​​the second shape, and wherein the second transparency value is less than the first transparency value.

[0218] In aspect 77, the method of any one of aspects 75-76, wherein the second shape is within the first shape.

[0219] In a 78th aspect, the method of any one of aspects 75-77, wherein a third shape of the concentric shapes has a third color, and wherein the first shape is adjacent to the first shape.

[0220] In a 79th aspect, the method of aspect 78, wherein the third color is white.

[0221] In aspect 80, the method of any one of aspects 78-79, wherein the first shape is within the third shape.

[0222] In aspect 81, the method of any of aspects 63-80, wherein the UI event corresponds to one of a plurality of states of the virtual UI device.

[0223] In aspect 82, the method of aspect 81, wherein the plurality of states comprises activation or inactivation of the virtual UI device.

[0224] In aspect 83, a method according to any one of aspects 63-82, wherein the plurality of virtual UI devices include buttons, up and down buttons, spinners, selectors, radio buttons, radio button lists, check boxes, picture boxes, check box lists, drop-down lists, drop-down menus, selection lists, list boxes, combo boxes, text boxes, sliders, links, keyboard keys, switches, sliders, touch surfaces, or combinations thereof.

[0225] In aspect 84, the method of any of aspects 63-83, wherein the plurality of UI pointers comprises objects associated with the user or a portion of the user.

[0226] In aspect 85, the method of aspect 84, wherein the object associated with the user comprises a pointer, a pen, a pencil, a marker, a highlighter, or a combination thereof, and wherein the part of the user comprises a finger of the user.

[0227] Other considerations

[0228] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings can be embodied in a code module executed by one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute dedicated and specific computer instructions, and fully or partially automated by the code module. For example, a computing system may include a general-purpose computer (e.g., a server) programmed with specific computer instructions or a dedicated computer, dedicated circuit, etc. The code module can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some embodiments, specific operations and methods can be performed by circuits specific to a given function.

[0229] In addition, certain implementations of the functionality of the present disclosure are mathematically, computationally, or technically complex enough that dedicated hardware or one or more physical computing devices (with appropriate dedicated executable instructions) may be necessary to perform the functionality or provide results in substantially real time, for example, due to the amount or complexity of the computations involved. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware is required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time. In addition, training and executing neural networks can be computationally challenging. In some cases, the neural network is executed by one or more graphics processing units (GPUs).

[0230] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, a combination of the same or similar thereof, and the like. Methods and modules (or data) may also be sent as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media), and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

[0231] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood to represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (e.g., logic or arithmetic) or step in a process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith can be performed in other appropriate sequences, for example, in serial, parallel or in some other manner. Tasks or events can be added to or removed from the disclosed example embodiments. In addition, the separation of various system components in the implementation described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0232] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0233] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired attributes disclosed herein. The various features and processes described herein can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications to the implementations described in the present disclosure will be apparent, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are consistent with the widest scope of the present disclosure, principles, and novel features disclosed herein.

[0234] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, although the features above may be described as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a subcombination or variations of the subcombination. Individual features or groups of features are not necessary or essential for each and all embodiments.

[0235] Unless otherwise specified, or understood in the context of use otherwise, conditional language used herein, such as "can", "may", "possibly", "may", "for example", etc., is generally intended to convey that certain embodiments include certain features, elements and / or steps, while other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements and / or steps are required in any way for one or more embodiments, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included or to be performed in any particular embodiment (whether or not there is author input or prompting). The terms "comprise", "include", "have", etc. are synonymous and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so when used, for example, to connect a list of elements, the term "or" means one, some or all of the elements in the list. In addition, unless otherwise specified, the articles "a", "an" and "the" used in this application and the appended claims should be understood to mean "one or more" or "at least one".

[0236] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, and Z" is otherwise understood by context as being generally used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0237] Similarly, although operations may be depicted in the accompanying drawings in a particular order, it should be appreciated that it is not necessary to perform these operations in the particular order shown or in sequence, or to perform all the operations shown, to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be included in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, operations may be rearranged or reordered in other embodiments. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into a variety of software products. In addition, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in different orders and still achieve the desired results.

Claims

1. A wearable display system, comprising: an image capture device configured to capture an image including the pointer; A non-transitory computer-readable storage medium configured to store: The image, a virtual user interface (UI) device associated with the image at an image location on the image, and A neural network for determining UI events trained using: an augmented training image associated with the virtual UI device, the augmented training image comprising a representation of the virtual UI device and a representation of a real-world pointer, and UI events regarding the virtual UI device and the real-world pointer in the augmented training image; a display configured to display the virtual UI device at a display position when the image capture device captures the image; as well as a hardware processor in communication with the image capture device, the display, and the non-transitory computer-readable storage medium, the processor being programmed with executable instructions to: accessing the image captured by the image capture device; rendering a representation of the virtual UI device onto the image at the image location; determining, using the neural network, UI events regarding the pointer in the image and the virtual UI device associated with the image; Generate virtual content based on the UI event; as well as The display is caused to present the virtual content.

2. The wearable display system of claim 1, wherein the representation of the virtual UI device comprises a plurality of high contrast shapes.

3. A wearable display system according to claim 2, wherein the plurality of high-contrast shapes comprises a plurality of high-contrast concentric shapes.

4. The wearable display system of claim 2, wherein the centers of gravity of the shapes of the plurality of shapes are within a threshold distance of each other.

5. The wearable display system of claim 4, wherein the threshold distance is 0.

6. The wearable display system according to claim 2, wherein a first shape in the plurality of shapes is associated with a first color, wherein a second shape in the plurality of shapes is associated with a second color, and The first shape is adjacent to the second shape.

7. A wearable display system according to claim 6, wherein the first color is black, and wherein the second color is white.

8. The wearable display system of claim 6, wherein the second shape is partially transparent.

9. The wearable display system according to claim 8, wherein a first region of the second shape is associated with a first transparency value, and Wherein a second area of ​​the second shape is associated with a second transparency value.

10. The wearable display system according to claim 9, wherein the second region of the second shape is further away from the first region of the second shape, and The second transparency value is smaller than the first transparency value.

11. The wearable display system of claim 6, wherein the second shape is within the first shape.

12. The wearable display system according to claim 6, wherein a third shape in the plurality of shapes is associated with a third color, and Wherein the first shape is adjacent to the first shape.

13. The wearable display system of claim 12, wherein the third color is white.

14. The wearable display system of claim 12, wherein the first shape is within the third shape.

15. The wearable display system of claim 1, wherein the UI event corresponds to one of a plurality of states of the virtual UI device.

16. The wearable display system of claim 15, wherein the plurality of states include activation or inactivation of the virtual UI device.

17. The wearable display system of claim 1, wherein the virtual UI device is selected from the group consisting of a button, an up and down button, a spinner, a selector, a radio button, a radio button list, a check box, a picture box, a check box list, a drop-down list, a drop-down menu, a selection list, a list box, a combo box, a text box, a slider, a link, a keyboard key, a switch, a slider, a touch surface, or a combination thereof.

18. The wearable display system of claim 1, wherein the pointer comprises an object associated with a user or a portion of the user.

19. The wearable display system according to claim 18, wherein the object associated with the user comprises a stylus, a pen, a pencil, a marker, a highlighter, or a combination thereof, and Wherein the part of the user includes a finger of the user.

20. A computerized method implemented by a computing system having one or more hardware computer processors and one or more non-transitory computer-readable storage media storing software instructions executable by the computing system to implement the computerized method, the computerized method comprising: accessing an image captured by an image capture device of the augmented reality helmet, the image including a pointer; Rendering an augmented image including the image and a representation of a virtual UI device on a display of the augmented reality helmet; Applying a neural network to the enhanced image, the neural network is trained using the following method: an augmented training image associated with the virtual UI device, the augmented training image comprising a representation of the virtual UI device and a representation of a real-world pointer, and UI events regarding the virtual UI device and the real-world pointer in the augmented training image; detecting, based on the neural network application, the UI event regarding the pointer in the image and the virtual UI device associated with the image; Generate virtual content based on the UI event; as well as The display is caused to present the virtual content.

21. A system for training a neural network for determining user interface events, the system comprising: a computer readable memory storing executable instructions; as well as One or more processors programmed by the executable instructions to at least: Receive multiple images, wherein one of the plurality of images comprises one of the plurality of pointers, wherein the image is associated with a virtual user interface (UI) device of a plurality of virtual UI devices at an image location on the image, and wherein the image is associated with a UI event in a plurality of UI events regarding the virtual UI device and the pointer in the image; rendering a representation of the virtual UI device onto the image at the image location to generate a training image; Generate a training set including input data and corresponding target output data, wherein the input data includes the training image, and Wherein the corresponding target output data includes the UI event; as well as A neural network is trained using the training set, the neural network being used to determine UI events associated with the virtual UI device and the pointer.

22. The system of claim 21, wherein the representation of the virtual UI device comprises a plurality of high contrast shapes.

23. The system of claim 22, wherein the plurality of high contrast shapes comprises a plurality of high contrast concentric shapes.

24. The system of claim 22, wherein centers of gravity of shapes of the plurality of shapes are within a threshold distance of each other.

25. The system of claim 24, wherein the threshold distance is 0.

26. The system according to claim 22, wherein a first shape in the plurality of shapes is associated with a first color, wherein a second shape in the plurality of shapes is associated with a second color, and The first shape is adjacent to the second shape.

27. The system of claim 26, wherein the first color is black, and wherein the second color is white.

28. The system of claim 26, wherein the second shape is partially transparent.

29. The system according to claim 28, wherein a first region of the second shape is associated with a first transparency value, and Wherein a second area of ​​the second shape is associated with a second transparency value.

30. The system according to claim 29, wherein the second region of the second shape is further away from the first region of the second shape, and The second transparency value is smaller than the first transparency value.

31. The system of claim 26, wherein the second shape is within the first shape.

32. The system of claim 26, wherein a third shape of the plurality of shapes has a third color, and wherein the first shape is adjacent to the third shape.

33. The system of claim 32, wherein the third color is white.

34. The system of claim 32, wherein the first shape is within the third shape.

35. The system of claim 21, wherein the UI event corresponds to one of a plurality of states of the virtual UI device.

36. The system of claim 35, wherein the plurality of states comprises activation or inactivation of the virtual UI device.

37. The system of claim 21, wherein the plurality of virtual UI devices comprises buttons, up and down buttons, spinners, selectors, radio buttons, radio button lists, check boxes, picture boxes, check box lists, drop-down lists, drop-down menus, selection lists, list boxes, combo boxes, text boxes, sliders, links, keyboard keys, switches, sliders, touch surfaces, or combinations thereof.

38. The system of claim 21, wherein the plurality of UI pointers include objects associated with a user or a portion of the user.

39. The system according to claim 38, wherein the object associated with the user comprises a pointer, a pen, a pencil, a marker, a highlighter, or a combination thereof, and Wherein the part of the user includes a finger of the user.

40. A method for training a neural network for determining user interface events, the method comprising: Under the control of the hardware processor: Receive multiple images, wherein a first image of the plurality of images comprises a first representation of a pointer of the plurality of pointers, wherein the first image is associated with a first representation of a virtual user interface (UI) device of a plurality of virtual UI devices at a first image position in the first image, and wherein the first image is associated with a UI event among a plurality of UI events regarding the virtual UI device and the pointer in the first image; rendering a first representation of the virtual UI device onto the first image at the first image location to generate a first training image; Generate a training set including input data and corresponding target output data, wherein the input data includes the first training image, and Wherein the corresponding target output data includes the UI event; as well as A neural network is trained using the training set, the neural network being used to determine UI events associated with the virtual UI device and the pointer.

41. A system comprising: a computer readable memory storing executable instructions; as well as One or more processors programmed by the executable instructions to at least: Receives multiple images, where: An image in the plurality of images includes a pointer in the plurality of pointers, The image is associated with a virtual user interface (UI) device from among a plurality of virtual UI devices at the image location on the image, and The image is associated with a UI event of a plurality of UI events regarding the virtual UI device and the pointer in the image; rendering a representation of the virtual UI device onto the image at the image location to generate a training image; and A training set is generated including input data and corresponding target output data, wherein the input data includes the training images, and wherein the corresponding target output data includes the UI events.

42. The system of claim 41, wherein the one or more processors are further programmed to train a machine learning model using the training set, the machine learning model being used to determine UI events associated with the virtual UI device and the pointer.

43. The system of claim 41, wherein the representation of the virtual UI device comprises a plurality of high contrast shapes.

44. The system of claim 43, wherein centers of gravity of shapes of the plurality of shapes are within a threshold distance of each other.

45. The system of claim 44, wherein the threshold distance is 0.

46. ​​The system of claim 43, wherein a first shape in the plurality of shapes is associated with a first color, wherein a second shape in the plurality of shapes is associated with a second color, and wherein the first shape is adjacent to the second shape.

47. The system of claim 46, wherein the second shape is partially transparent.

48. The system of claim 47, wherein a first region of the second shape is associated with a first transparency value, and wherein a second region of the second shape is associated with a second transparency value.

49. The system of claim 48, wherein the second region of the second shape is further away from the first region of the second shape, and wherein the second transparency value is less than the first transparency value.

50. The system of claim 46, wherein the second shape is within the first shape.

51. The system of claim 46, wherein a third shape of the plurality of shapes has a third color, and wherein the first shape is adjacent to the third shape.

52. The system of claim 41, wherein the UI event corresponds to one of a plurality of states of the virtual UI device.

53. The system of claim 52, wherein the plurality of states comprises activation or inactivation of the virtual UI device.

54. The system of claim 41, wherein the plurality of virtual UI devices comprises buttons, up and down buttons, spinners, selectors, radio buttons, radio button lists, check boxes, picture boxes, check box lists, drop-down lists, drop-down menus, selection lists, list boxes, combo boxes, text boxes, sliders, links, keyboard keys, switches, sliders, touch surfaces, or combinations thereof.

55. The system of claim 41, wherein the plurality of UI pointers include objects associated with a user or a portion of the user.

56. The system of claim 55, wherein the object associated with the user comprises a pointer, a pen, a pencil, a marker, a highlighter, or a combination thereof, and wherein the portion of the user comprises a finger of the user.

57. A method comprising: Receives multiple images, where: An image in the plurality of images includes a pointer in the plurality of pointers, The image is associated with a virtual user interface (UI) device from among a plurality of virtual UI devices at the image location on the image, and The image is associated with a UI event of a plurality of UI events regarding the virtual UI device and the pointer in the image; rendering a representation of the virtual UI device onto the image at the image location to generate a training image; and A training set is generated including input data and corresponding target output data, wherein the input data includes the training images, and wherein the corresponding target output data includes the UI events.

58. The method of claim 57, further comprising: The training set is used to train a machine learning model, wherein the machine learning model is used to determine UI events associated with the virtual UI device and the pointer.

59. The method of claim 57, wherein the UI event corresponds to one of a plurality of states of the virtual UI device.

60. The method of claim 57, wherein the plurality of UI pointers include objects associated with a user or a portion of the user.

Images