Display system and method

Abstract

The present invention relates to display systems and methods. One embodiment relates to a user display device, comprising a housing frame mountable on the head of the user, a lens mountable on the housing frame, and a projection subsystem coupled to the housing frame to determine a position where a display object appears in the user's field of view based at least in part on at least one of detection of the user's head movement and prediction of the user's head movement, and projecting the display object to the user based on the determined position where the display object appears.

Description

[0001] This application is a divisional application of the invention application with application date of March 14, 2014, application number 201810264993.1, and invention name “Display system and method”. Technical Field

[0002] The present invention generally relates to systems and methods configured to facilitate an interactive virtual or augmented reality environment for one or more users. Background Art

[0003] Many display systems can benefit from information regarding the viewer or user's head pose (ie, the user's head orientation and / or position).

[0004] For example, a head mounted display (or helmet display, or smart glasses) is at least loosely coupled to the user's head and therefore moves when the user's head moves. If the user's head movement is detected by the display system, the displayed data can be updated to take into account the change in head posture.

[0005] As an example, if a user wearing a head mounted display views a virtual representation of a 3D object on the display and walks around the area where the 3D object appears, the 3D object may be re-rendered for each viewpoint, giving the user the perception that he or she is walking around the object occupying real space. If the head mounted display is used to display a virtual space (e.g., a rich virtual world) for multiple objects, the measurement of head pose may be used to re-render the scene to match the dynamic changes in the user's head position and orientation and provide an enhanced sense of immersion in the virtual space.

[0006] Especially for display systems that fill a substantial portion of the user's field of view with virtual elements, high head tracking accuracy and low latency throughout the system from first detecting head movement to updating the light sent by the display to the user's visual system are critical. If latency is high, the system can create a mismatch between the user's vestibular and visual sensory systems and produce motion sickness or virtual reality sickness.

[0007] Some head-mounted displays enable viewing of both real and virtual elements simultaneously - often described as an augmented reality or mixed reality approach. In one such configuration, often referred to as a "video see-through" display, a camera captures elements of a real-world scene, a computing system overlays virtual components onto the captured real-world scene, and a non-transparent display displays the composite image to the eye. Another configuration is often referred to as a "light see-through" display, in which a user can look through a transparent (or translucent) element in the display system to directly see light from displayed objects in the environment. The transparent element, often referred to as a "combiner," overlays light from the display onto the user's view of the real world.

[0008] In both video and optical see-through displays, detecting head gestures allows the display system to render virtual objects so that they appear to occupy space in the real world. As the user's head moves in the real world, the virtual objects are re-rendered based on the head gesture so that the virtual objects appear to remain stable relative to the real world. In the case of such optical see-through displays, the user's real-world vision has essentially zero latency, while his or her virtual object vision has a latency that depends on head tracking speed, processing time, rendering time, and display frame rate. If the system latency is high, the apparent position of virtual objects will be unstable during rapid head movements.

[0009] In addition to head-mounted display systems, other display systems can benefit from accurate and low-latency head pose detection. These include head-tracked display systems, where the display is not worn on the user's body, but, for example, mounted on a wall or other surface. The head-tracked display behaves like a window to the scene, and as the user moves their head relative to the "window", the scene is re-rendered to match the user's changing viewpoint. Other systems include a head-mounted projection system, where the head-mounted display projects light into the real world. Summary of the invention

[0010] Embodiments of the present invention are directed to devices, systems, and methods for facilitating one or more users to interact with virtual reality and / or augmented reality.

[0011] One embodiment is directed to a method operating in a virtual imaging system or an augmented reality system, the method comprising, for each of at least some of a plurality of frames presented to an end user, determining a position at which a virtual object in the end user's field of view appears relative to a reference frame of the end user, and adjusting the presentation of at least one subsequent frame based at least in part on the determined position at which the virtual object in the end user's field of view appears. A virtual object may be newly introduced into the end user's field of view in time relative to a frame previously presented to the end user. The newly introduced virtual object may be determined to be likely to attract the end user's attention. The virtual object may be at a new orientation in the frame relative to at least one previous frame. Alternatively, the virtual object may be at a new position for presentation to the end user relative to a previous orientation of the virtual object as previously presented to the end user.

[0012] The method may further include selecting the virtual object based on input indicating the end user's attention to the virtual object. The input indicating the end user's attention to the virtual object may be based at least in part on the appearance of the virtual object in a new orientation presented to the end user relative to the orientation of the virtual object previously presented to the end user. Alternatively, the input indicating the end user's attention to the virtual object may be based at least in part on how quickly the orientation of the virtual object presented to the end user changes relative to the orientation of the virtual object as previously presented to the end user.

[0013] The adjustment of the presentation of at least one subsequent frame may include presenting at least one subsequent frame with the center of the at least one subsequent frame moved toward the location in the field of view of the end user where the virtual object has been determined to appear. Alternatively, the adjustment of the presentation of at least one subsequent frame may include presenting at least one subsequent frame with the center of the at least one subsequent frame moved to the location in the field of view of the end user where the virtual object has been determined to appear.

[0014] The method may further include predicting the occurrence of end-user head movement based at least in part on the determined location of the occurrence of the virtual object in the end-user's field of view. The method may further include estimating at least one value indicating an estimated velocity of the predicted end-user head movement, determining at least one value that at least in part compensates for the estimated velocity of the predicted end-user head movement, and rendering at least one subsequent frame based at least in part on the determined value.

[0015] The method may further include estimating at least one change in velocity in the predicted end-user head movement, wherein the at least one change in velocity occurs between the start of the predicted head movement and the end of the predicted head movement, and wherein estimating at least one value indicative of the estimated velocity of the predicted head movement includes estimating at least one value indicative of the predicted velocity that at least partially accommodates the estimated change in velocity in the predicted end-user head movement.

[0016] Estimating at least one change in velocity in the predicted end-user head movement may include estimating at least one change between a first defined time after the predicted head movement begins and a second defined time before the predicted head movement ends.

[0017] The method may further include estimating at least one value of an estimated acceleration indicative of predicted end-user head movement, determining at least one value of the estimated acceleration that at least partially compensates for the predicted end-user head movement, and rendering at least one subsequent frame based at least in part on the determined value.

[0018] The method may further include receiving information indicating the identity of the end user, and retrieving at least one user-specific historical attribute for the end user based on the received information indicating the identity of the end user, wherein the user-specific historical attribute indicates at least one of a previous head movement velocity for the end user, a previous head movement acceleration for the end user, and a previous eye movement to head movement relationship for the end user.

[0019] The virtual object may be at least one of a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual label object, a virtual field object, a virtual chart object, a virtual map object, a virtual tool object, or a virtual visual representation of a physical object.

[0020] Another embodiment is directed to a method of operation in an augmented reality system, the method comprising: receiving information indicating an identity of an end user, retrieving at least one user-specific historical attribute for the end user based at least in part on the received information indicating the identity of the end user, and providing a frame to the end user based at least in part on the retrieved at least one user-specific historical attribute for the end user. The received information may be image information indicating an image of at least a portion of an eye of the end user.

[0021] The at least one user-specific historical attribute retrieved for the end user may be at least one attribute that provides an indication of at least one head movement attribute for the end user, wherein the head movement attribute indicates at least one previous head movement of the end user. Or the at least one user-specific historical attribute retrieved for the end user may be at least one attribute that provides an indication of at least one previous head movement speed for at least one previous head movement of the end user. Alternatively, the at least one user-specific historical attribute retrieved for the end user may be at least one attribute that provides an indication of a change in the speed of the head movement of the end user across at least a portion of the range of at least one previous head movement. Alternatively, the at least one user-specific historical attribute retrieved for the end user may be at least one attribute that provides an indication of at least one previous head movement acceleration for at least one previous head movement of the end user. Alternatively, the at least one user-specific historical attribute retrieved for the end user may be at least one attribute that provides an indication of a relationship between at least one previous head movement and at least one previous eye movement of the end user. Alternatively, the at least one user-specific historical attribute retrieved for the end user may be at least one attribute that provides an indication of a ratio between at least one previous head movement and at least one previous eye movement of the end user.

[0022] The method may further include predicting at least one endpoint of the end user's head movement and providing frames to the end user based at least in part on at least one user-specific historical attribute retrieved for the end user, including rendering at least one subsequent frame to at least one image buffer, the at least one subsequent frame moving toward the predicted endpoint of the head movement.

[0023] The method may further include rendering a plurality of subsequent frames that are at least partially adapted for at least one head movement attribute of the end user to move toward the predicted end point of the head movement, the head movement attribute indicating at least one previous head movement of the end user.

[0024] The head movement attribute indicating at least one previous head movement of the end user may be a historical head movement velocity for the end user, a historical head movement acceleration for the end user, or a historical ratio between head movement and eye movement for the end user.

[0025] The method may further include predicting the occurrence of head movement of the end user based at least in part on the location where the virtual object appears in the end user's field of view. The location where the virtual object appears may be determined in the same manner as described above.

[0026] Another embodiment is directed to detecting an indication that spacing between some pixels in a frame presented to an end user will be different than spacing between other pixels in the frame, adjusting a first set of pixels based on the detected indication, and providing a portion of at least one subsequent frame with the adjusted first set of pixels to at least partially compensate for the difference between the spacing presented to the end user. Pixel characteristics (e.g., size, brightness, etc.) may be perceptible to the end user.

[0027] The method may further include selecting a first group of pixels of a frame based on a direction of the detected head movement, wherein the direction of the first group of pixels is the same as the direction of the detected head movement, and increasing the size of the first group of pixels of at least one subsequent frame. The method may further include selecting a first group of pixels of a frame based on a direction of the detected head movement, wherein the direction of the first group of pixels is the same as the direction of the detected head movement, and increasing the brightness of the first group of pixels of at least one subsequent frame in response to the detected head movement.

[0028] The method may further include selecting a first group of pixels of a frame based on a direction of detected head movement, wherein the direction of the first group of pixels is opposite to the direction of the detected head movement, and reducing a size of the first group of pixels of at least one subsequent frame in response to the detected head movement.

[0029] The method may further include selecting a first group of pixels of a frame based on a direction of detected head movement, wherein the direction of the first group of pixels is opposite to the direction of the detected head movement, and reducing the brightness of the first group of pixels of at least one subsequent frame in response to the detected head movement.

[0030] Another embodiment id is directed to a method operating in a virtual image rendering system, the method comprising rendering a first complete frame to an image buffer, wherein the first complete frame includes pixel information for sequential rendering of pixels to form an image of a virtual object, starting rendering of the first complete frame, and dynamically interrupting rendering of the first complete frame before rendering of the first complete frame ends by presenting an update to the first complete frame, in which a portion of the pixel information has changed from the first complete frame.

[0031] Another embodiment is directed to a method operating in a virtual image presentation system, the method comprising rendering a first complete frame having a first field and a second field to an image buffer, wherein the first field includes at least a first spiral scan line and the second field includes at least a second spiral scan line, the second spiral scan line being interlaced with at least the first spiral scan line; reading out a frame buffer storing the first complete frame; and dynamically interrupting the reading out of the first complete frame by reading out an update to the first complete frame before the reading out of the first complete frame is completed, wherein a portion of pixel information has changed from the first complete frame. The dynamic interruption of the reading out can be based on a detected end-user head movement, wherein the detected head movement exceeds a nominal head movement value.

[0032] Another embodiment is directed to a method operating in a virtual image rendering system, the method comprising rendering a first complete frame having a first field and a second field to an image buffer, wherein the first field includes at least a first Lissajous scan line and the second field includes at least a second Lissajous scan line, the second Lissajous scan line being interlaced with at least the first Lissajous scan line; reading out a frame buffer storing the first complete frame; and dynamically interrupting the reading out of the first complete frame by reading out an update to the first complete frame before the reading out of the first complete frame is completed, wherein a portion of pixel information has changed from the first complete frame, the update dynamically interrupting based on a detected head movement of an end user exceeding a nominal head movement value. The method may further include phase shifting the Lissajous scan lines to interlace the Lissajous scan lines.

[0033] Another embodiment is directed to a method operating in a virtual image rendering system, the method comprising, for each of a plurality of frames, in response to a detected end-user head movement, determining a corresponding resolution for each of at least two portions in each corresponding frame, and rendering a virtual object based on the determined corresponding resolutions of the at least two portions in the corresponding frames. The portions of the corresponding frames may be at least one of fields of a frame, lines of a frame, pixels of a frame. The method may further include adjusting a characteristic of a drive signal between rendering a first portion of a frame and a second portion of a frame to create a variable resolution in an image of the virtual object. The characteristic of the drive signal may be at least one of an amplitude of the drive signal and a slope of the drive signal.

[0034] The method may further include evaluating an attention point in at least the first image for the end user based on at least one of processed eye tracking data, a determined position where a virtual object in the end user's field of view appears relative to the end user's reference frame, a determined position where the virtual object appears when newly introduced into the end user's field of view, and a determined position where the virtual object appears in a new position in the image relative to the orientation of the virtual image in at least one previous image.

[0035] The method may further include increasing the resolution in the at least one subsequent image in a portion of the at least one subsequent image that is at least adjacent to the assessed point of interest relative to other portions of the at least one subsequent image. The method may further include decreasing the resolution in the at least one subsequent image in a portion of the at least one subsequent image that is at least distal to the assessed point of interest relative to other portions of the at least one subsequent image.

[0036] Another embodiment is directed to a method operating in a virtual image presentation system, the method comprising: displaying at least one virtual object to an end user, and temporarily blanking a portion of the display of the at least one virtual object when a detected head movement exceeds at least one of a nominal head movement value and a predicted head movement is predicted to exceed a head movement value. The method may further include processing tracking data provided via at least one sensor to determine at least one of the detected head movement and the predicted head movement, wherein the head tracking data indicates at least a head orientation of the end user.

[0037] Another embodiment is directed to a projector apparatus for projecting at least a virtual image in an augmented reality system, the projector apparatus comprising a projector assembly, a support supporting the projector assembly so that the projector assembly is movable in at least one free axis, at least one actuator coupled to selectively move the projector assembly, and a control subsystem communicatively coupled to control the actuator so that the projector assembly is moved in response to at least one of a detection of end-user head movement exceeding a nominal head movement value and a prediction of end-user head movement being predicted to exceed the nominal head movement value. The projector assembly may further include at least one first optical fiber having a rear end coupled to receive an image and a front end positioned to transmit the image thereby.

[0038] The support assembly may include a piezoelectric ring receiving at least the first optical fiber proximate to but spaced rearwardly from a leading end of the first optical fiber such that a portion of the first optical fiber proximate its leading end extends from the piezoelectric ring and is free to oscillate at a defined resonant frequency.

[0039] The projector apparatus of claim 87, wherein at least the controlled subsystem is communicatively coupled to receive head tracking data provided via at least one sensor, the head tracking data indicating at least an orientation of a head of an end user. For each of at least some of a plurality of images presented to the end user, the control subsystem determines a position at which a virtual object appears in the end user's field of view relative to the end user's frame of reference, evaluates whether the determined position requires the end user to turn the user's head, and predicts the occurrence of head motion based on the evaluation.

[0040] Another embodiment is directed to an operating method in a virtual image presentation system, the method comprising overrendering a frame for a defined field of view so that pixel information for a set of pixels of the frame exceeds a maximum area displayed at a maximum resolution, determining a portion of the frame to present to an end user based on at least one of a detected head movement and a predicted head movement, and selectively reading out only the determined portion of the frame.

[0041] Another embodiment is directed to a user display device, including a housing frame mountable on a user's head, a lens mountable on the housing frame, and a projection subsystem coupled to the housing frame to determine a location where a display object appears in a user's field of view based at least in part on at least one of detection of user head movement and prediction of user head movement and project the display object to the user based on the determined location where the display object appears. The location where the display object appears can be moved in response to at least one of detection of user head movement or prediction of user head movement exceeding or predicted to exceed a nominal head movement value. The prediction of user head movement can be based on a prediction of the user's movement in focus or a set of historical attributes of the user.

[0042] The user display device may further include a first pair of cameras mountable on the housing frame to track movement of the user's eyes and estimate the focus depth of the user's eyes based on the tracked eye movement. The projection subsystem may project the display object based on the estimated focus depth.

[0043] The user display device may further include a second pair of cameras mountable on the housing frame to capture a field of view image as seen by the user's eyes, wherein the field of view image includes at least one physical object. The projection subsystem may project a display object in a manner such that the display object and the physical object captured by the second pair of cameras are mixed with each other and appear together in the same frame. The appearance position may be based at least in part on the physical object. The display object and the physical object may have a predetermined relationship. The captured field of view image may be used to collect information about the user's head movement, wherein the information about the user's head movement includes the center of the user's attention, the orientation of the user's head, the direction of the user's head, the speed of the user's head movement, the acceleration of the user's head movement, and the distance of the user's head related to the user's local environment.

[0044] The lens may include at least one transparent surface to selectively allow transmission of light so that the user can observe the local environment. The projection subsystem may project the display object in a manner such that the user observes the display object and the local environment as if observing through the transparent surface of the lens.

[0045] The user display device may also include at least one inertial sensor to capture a set of inertial measurements representing movement of the user's head, wherein the set of inertial measurements includes a velocity of the user's head movement, an acceleration of the user's head movement, a direction of the user's head movement, an orientation of the user's head, and an orientation of the user's head.

[0046] The user display device may further include at least one light source to illuminate at least one of the user's head and the user's local environment.

[0047] The projection subsystem may adjust at least one of a perceived size, brightness, and resolution of a set of pixels associated with the display object to compensate for at least one of the detected head movement and the predicted head motion. The display object may be one of a virtual object and an augmented virtual object.

[0048] Additional and other objects, features, and advantages of the present invention are described in the detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An example of using predictive head tracking to render frames for an end user is shown.

[0050] Figure 2An example of a technique for predicting head motion based on features of a virtual object presented to an end user is shown.

[0051] Figure 3 An example of where the frame center is offset is shown.

[0052] Figure 4 An example of a technique for predicting head movement based on a set of historical attributes of an end user is shown.

[0053] Figure 5 Another example of a technique for predicting head motion based on historical attributes is shown.

[0054] Figure 6 An example of retrieving various historical attributes of a user is shown.

[0055] Figure 7 An example of rendering subsequent frames based on the predicted endpoint is shown.

[0056] Figure 8 Another example of rendering subsequent frames is shown.

[0057] Fig. 9 An example of predicting the occurrence of head movement is shown.

[0058] Fig.10 An example of adjusting pixels based on head motion is shown.

[0059] Fig.11 An example of rendering a frame using adjusted pixels is shown.

[0060] Fig.12 Examples of increasing the size and / or brightness of pixels are shown.

[0061] Fig.13 An example of presentation of a dynamic interrupt frame is shown.

[0062] Fig.14 An example of presenting a portion of an updated frame is shown.

[0063] Fig.15 An example of reading an update frame is shown.

[0064] Fig.16 An example of phase shift is shown.

[0065] Fig.17 An example resulting in variable resolution in an image is shown.

[0066] Fig.18 An example of adjusting the amplitude of the drive signal is shown.

[0067] Fig.19An example of adjusting the resolution in subsequent images based on the end user's attention point is shown.

[0068] Fig. 20 Another example of adjusting the resolution is shown.

[0069] Fig.21 An example of determining where a virtual object appears is shown.

[0070] Fig. 22 An example of hiding a portion of a virtual object is shown.

[0071] Fig.23 An example of predicting head motion based on the attractiveness of a virtual object is shown.

[0072] Fig.24 An example of strobing is shown.

[0073] Fig.25 An example of selectively activating an actuator to move a projector assembly is shown.

[0074] Fig.26 An example of selectively reading out a portion of a frame is shown.

[0075] Fig. 27 An example is shown in which a portion is selectively read out based on the determined position of the virtual object.

[0076] Fig.28 Another example of selectively reading out a portion is shown.

[0077] Fig.29 An example of determining a portion of an image to present to an end user is shown.

[0078] Fig.30 An example of dynamically processing a portion of a rendered frame is shown.

[0079] Fig.31 An example of a frame with pixel information is shown.

[0080] Fig.32 An example of a raster scan pattern is shown.

[0081] Fig.33 An example of a spiral scan pattern is shown.

[0082] Fig.34 A Lissajous scan pattern is shown.

[0083] Fig.35 An example of a multi-field spiral scan pattern is shown.

[0084] Fig.36AAn example of raster scan pattern distortion during rapid lateral movement of the end user's head is shown.

[0085] Fig.36B An example of raster scan pattern distortion is shown during vertical upward movement of the end user's head.

[0086] Fig.37A An example of spiral scanline distortion is shown during a rapid lateral motion of the end-user's head to the left.

[0087] Fig.37B An example of spiral scanline distortion is shown during a very rapid lateral motion of the user's head to the left.

[0088] Fig.38 An overview of a virtual image generation system is shown. DETAILED DESCRIPTION

[0089] The following description relates to display systems and methods used in virtual reality and / or augmented reality systems. However, it should be understood that while suitable for use in virtual reality, the invention in its broadest aspects is not limited.

[0090] First refer to Fig.38 , Fig.38 According to one illustrated embodiment, a virtual image generation system 3800 is shown that can operate to provide a virtual image to an end user 3802.

[0091] The virtual image generation system 3800 can be operated as an augmented reality system, which provides virtual object images mixed with physical objects in the field of view of the end user. When the virtual image generation system 3800 is operated as an augmented reality system, there are two basic methods. The first method uses one or more imagers (e.g., cameras) to capture images of the surrounding environment. The virtual image generation system 3800 can mix the image into data representing the image of the surrounding environment. The second method uses one or more at least partially transparent surfaces through which the surrounding environment can be seen, and the virtual image generation system 3800 generates images of virtual objects thereon. As will be apparent to those skilled in the art, at least some of the aspects described herein are particularly suitable for augmented reality systems.

[0092] Virtual image generation system 3800 may be operated as a virtual reality system, providing images of virtual objects in a virtual environment.

[0093] Virtual image generation system 3800 and the various techniques taught herein can be used in applications other than augmented reality and virtual reality systems. For example, the various techniques can be applied to any projection or display system. For example, the various techniques described herein can be applied to a micro-projector, where the motion can be the motion of the end user's hand rather than the head. Therefore, although this article is often described in terms of augmented reality systems, the teachings should not be limited to such systems or such uses.

[0094] At least for augmented reality applications, various virtual objects need to be placed relative to corresponding physical objects in the field of view of the end user 3802. Virtual objects, also referred to herein as virtual labels or tags or annotations, can take any of a variety of forms, primarily any kind of data, information, concept, or logical structure that can be presented as an image. Non-limiting examples of virtual objects may include: virtual text objects, virtual digital objects, virtual alphanumeric objects, virtual label objects, virtual field objects, virtual chart objects, virtual map objects, virtual tool objects, or virtual visual representations of physical objects.

[0095] Head tracking accuracy and latency have always been a problem in virtual reality and augmented reality systems. Tracking inaccuracies and delays create inconsistencies between the end user's visual system and vestibular system. This can cause nausea and discomfort. This is particularly problematic in display systems that fill most of the end user's field of view. Ways to address these may include increasing the frame rate or effective frame rate, such as by flickering or flickering or by other techniques. As described herein, predictive head tracking can be used to address these problems, such as by reducing latency. Predictive head tracking can rely on a large number of different factors or methods, including historical data or attributes for a specific end user. It is also described herein that blanking of a display or presentation can be effectively used, such as blanking during rapid head movements.

[0096] At least for augmented reality applications, the placement of virtual objects that have a spatial relationship with physical objects (e.g., presented to appear spatially close to physical objects in two or three dimensions) is an important issue. For example, considering the surrounding environment, head movement can significantly complicate the placement of virtual objects in the field of view. Whether it is the field of view of the captured image of the surrounding environment and then projected or displayed to the end user 3802, or the field of view of the surrounding environment directly perceived by the end user 3802, it is all true. For example, head movement will likely cause a change in the field of view of the end user 3802, which will likely require the positions of various virtual objects displayed in the field of view of the end user 3802 to be updated. In addition, head movement may occur in a wide variety of ranges and speeds. Not only may the head movement speed change between different head movements, but also the head movement speed may change across a single head movement range or within a single head movement range. For example, the speed of the head movement may initially increase (e.g., linearly or nonlinearly) from the starting point, and may decrease when reaching the end point, obtaining the maximum speed somewhere between the starting point and the end point of the head movement. Rapid head movements may even exceed the ability of a particular display or projection technology to render an image to the end user 3802 that appears to have uniform and / or smooth motion.

[0097] exist Fig.38 In the illustrated embodiment, the virtual image generation system 3800 includes a projection subsystem 3804 operable to project an image on a partially transparent display surface 3806 located in the field of view of the end user 3802, between the eyes 3808 of the end user 3802 and the surrounding environment. The virtual image generation system 3800 may be worn or mounted on the head 3810 of the end user 3802, such as incorporated into a pair of glasses or goggles.

[0098] In the illustrated embodiment, the projection subsystem 3804 includes one or more optical fibers 3812 (e.g., single-mode optical fibers) having a rear end or distal end 3812a into which light is received and a front end or proximal end 3812b from which light is provided to a partially transparent display surface 3806 or projected directly into the eye 3808 of the end user 3802. The projection subsystem 3804 may also include one or more light sources 3815 that generate light (e.g., emit light of different colors in a defined pattern) and communicatively couple the light to the rear end or distal end 3812a of the one or more optical fibers 3812. The light source 3815 may take any of a variety of different forms, such as a set of RGB lasers (e.g., laser diodes capable of outputting red, green, and blue light) that are operable to generate red, green, and blue coherent parallel light, respectively, according to a defined pixel pattern specified in a corresponding frame of pixel information or data. Lasers provide high color saturation and high energy efficiency.

[0099] although Fig.38 A single optical fiber 3812 is shown that decomposes the light into multiple channels, some implementations may employ two or more optical fibers 3812. In such embodiments, the optical fibers 3812 may have staggered tips or beveled and polished tips to bend the light, reducing the optical separation between the channels. The optical fibers 3812 may be conveniently packaged into a ribbon cable. Appropriate optics may produce a combination of the corresponding images produced by each channel.

[0100] One or more optical fibers 3812 may be supported by a yoke 3814 having a portion of a front or proximal end 3812b extending therefrom. The yoke 3814 may be operable to set the front or proximal end 3812b in oscillatory motion. For example, the yoke 3814 may include a tube piezoelectric sensor 3814a ( Fig.38 Only one is shown). A plurality of electrodes 3813 (e.g., four are shown, only one of which is labeled) are radially arranged about the piezoelectric sensor 3814a. Applying a control signal to each electrode 3813 associated with the piezoelectric sensor 3814a, for example via a frame buffer 3828, can cause the front or proximal end 3812b of the optical fiber 3812 to vibrate in a first resonant mode. The magnitude or amount of deviation from the center of the vibration can be controlled via the applied drive signal to obtain any of a variety of at least two-axis patterns. The pattern can, for example, include a raster scan pattern, a spiral or helical scan pattern, or a Lissajous or Figure 8 Scan the pattern.

[0101] Fig.31 According to one illustrated embodiment, a frame 3100 of pixel information or data is shown that specifies the pixel information or data to present an image, such as an image of one or more virtual objects. The frame 3100 schematically illustrates each pixel in cells 3100A, -3100n (only two are labeled, collectively referred to as 3102). A sequence of cells 3104a, 31004b-3100n (three are labeled, collectively referred to as 3104) arranged in rows or lines is shown extending horizontally across Fig.31 Frame 3100 includes multiple lines 3104. Fig.31 Ellipses are used to indicate missing information, such as cells or rows that are omitted for clarity.

[0102] Each cell 3102 of the frame 3100 may specify a value for each of a plurality of colors (collectively 3106) for each pixel and / or brightness corresponding to the cell. For example, the frame 3100 may specify one or more values ​​for red 3106a, one or more values ​​for green 3106b, and one or more values ​​for blue 3106c for each pixel. The values ​​3106 may be specified as a binary representation for each color, such as a 4-bit number for each color. Each cell 3102 of the frame 3100 may additionally include an amplitude or radial value #P06d that specifies an amplitude or radial dimension for each pixel, such as when the frame 3100 is used in conjunction with a system based on a spiral scan line pattern or a system based on a Lissajous scan line pattern.

[0103] Frame 3100 may include one or more fields, collectively referred to as 3110. Frame 3100 may consist of a single field. Alternatively, frame 3100 may include two, or even more fields 3110a-3110b. Fig.31 The frame 3100 shown in FIG. 3100 shows two fields 3110a, 3110b. Pixel information for the complete first field 3110a of the frame 3100 may be specified before pixel information for the complete second field 3110b, for example, before pixel information for the second field 3110b in a sequence, ordered list, or other data structure (e.g., record, linked list). Assuming the rendering system is configured to process more than two fields 3110a-3110b, a third or even fourth field may follow the second field 3110b.

[0104] Fig.32 A raster scan pattern 3200 is schematically represented. In the raster scan pattern 3200, pixels 3202 (only one is marked) are presented in sequence. The raster scan pattern 3200 typically presents pixels from left to right (indicated by arrows 3204a, 3204b) and then from top to bottom (indicated by arrow 3206). Therefore, the presentation can start at the upper right corner and traverse the first row 3208 to the left until the end of the row is reached. The raster scan pattern 3200 then typically starts from the left side of the next row. The presentation can be temporarily stopped or blanked, which returns from the end of a row to the beginning of the next row. The process is repeated row by row until the bottom line 3208n is completed, for example, the bottom rightmost pixel. As the frame 3100 is completed, a new frame begins, once again returning to the right side of the topmost row of the next frame. Once again, the presentation can be stopped when returning from the bottom left to the upper right to present the next frame.

[0105] Many implementations of raster scanning use an interlaced scanning pattern. In an interlaced raster scanning pattern, the rows from the first and second fields 3210a, 3210b are interlaced. For example, when presenting the rows of the first field 3210a, the pixel information for the first field 3210a may be used only for odd rows, while the pixel information for the second field 3210b may be used only for even rows. Thus, the frame 3100 ( Fig.31 ) are typically presented before the lines of the second field 3210b. The first field 3210a may use the pixel information of the first field 3210a to sequentially present line 1, line 3, line 5, etc. Then, frame 3100( Fig.31 ) can present row 2, row 4, row 6, etc. in sequence by using the pixel information of the second field 3210b to follow the first field 3210a.

[0106] Fig.33 Schematically, a spiral scan pattern 3300 is shown according to one illustrated embodiment. The spiral scan pattern 3300 may be composed of a single spiral scan line 3302, which may include one or more complete angular cycles (e.g., 360 degrees), which may be named as a circle or ring. Pixel information is used to specify the color and / or brightness of each sequential pixel as the angle increases. Amplitude or radial value 3208 ( Fig.31 ) specifies the radial dimension #R06 from the starting point 3308 of the spiral scan line 3302.

[0107] Fig.34 A Lissajous scan pattern 3400 is schematically shown according to one illustrated embodiment. The Lissajous scan pattern 3400 may be comprised of a single Lissajous scan line 3402, which may include one or more complete angular periods (e.g., 360 degrees), which may be termed as loops or rings. Alternatively, the Lissajous scan pattern 3400 may include two or more Lissajous scan lines 3402, each of which is phase-shifted relative to one another to nest the Lissajous scan lines 3402. Pixel information is used to specify the color and / or brightness of each sequential pixel with angular increments. Amplitude or radial value 3208 ( Fig.31 ) specifies the radial dimension starting from the starting point of the Lissajous scan line 3402.

[0108] Fig.35 A multi-field spiral scan pattern 3500 is schematically shown according to one illustrated embodiment. The multi-field spiral scan pattern 3500 includes two or more different spiral scan lines, collectively referred to as 3502, Fig.35 Four spiral scan lines 3502a-3502d are shown. The pixel information for each spiral scan line 3502 may be represented by frame 3100 ( Fig.31) are specified by their respective fields (e.g., 3210a, 3210b). Advantageously, multiple spiral scan lines 3502 can be nested simply by applying a phase shift between each successive spiral scan line 3502. The phase difference between the spiral scan lines 3502 should depend on the total number of spiral scan lines 3502 to be employed. For example, four spiral scan lines 3502a-3502d can be separated by a 90 degree phase shift. An exemplary embodiment can operate with 10 different spiral scan lines (i.e., sub-spirals) at a 100 Hz refresh rate. Similar to Fig.33 In one embodiment, one or more amplitude or radial values ​​3208 ( Fig.31 ) specifies the radial dimension 3506 starting from the starting point 3508 of the spiral scan line 3502.

[0109] As from Fig.34 and 35 It is appreciated that the relative spacing between adjacent pixels may vary throughout the image. It may be advantageous to at least partially accommodate or compensate for this non-uniformity. For example, it may be advantageous to adjust the pixel size, such as increasing the perceived pixel size for pixels that are spaced farther apart than other pixels. This may be achieved, for example, via selective blurring (e.g., variable focus lens, variable diffusion, dithering) to increase the Gaussian spot size. Additionally or alternatively, it may be advantageous to adjust the brightness for pixels that are spaced farther apart than other pixels.

[0110] Back to Fig.38 , a piezoelectric sensor 3814a is driven using a sinusoidal wave drive signal with a resonant frequency about a first axis and a resonant frequency about a second axis perpendicular to the first axis to produce a spiral scan pattern. The spiral scan pattern is characterized by a radial dimension that varies with the change of the angular dimension. For example, the radial dimension can vary linearly or nonlinearly when the radial dimension changes from 0 degrees to or through 360 degrees. Phenomenologically, the spiral scan line may appear as a continuous spiral that starts at a starting point and radiates radially outward while rotating in a plane. Each complete angular cycle can be described as constituting a circle or ring. Before the start of the starting point, a spiral scan line with any desired number of circles or rings can be defined. A refresh cycle in which the display or presentation is blanked may occur between the end of the first spiral scan pattern in time and the beginning of the next spiral scan pattern that is continuous in time. The outermost radial dimension of the spiral scan pattern can be set by amplitude modulation of the sinusoidal wave drive signal. Amplitude adjustment of the spiral scan line pattern adjusts the radial dimension without affecting the angular dimension. Thus, the amplitude modulation will not affect the frequency of the cycles (e.g., number of turns or rings) or the number of cycles for a given scan line at a given time. The position of the front or proximal end 3812b in the pattern is synchronized with the output of the light source 3815 to form a two-dimensional or three-dimensional image.

[0111] Although not shown, the projection subsystem 3804 may include one or more optical components (e.g., lenses, filters, gratings, prisms, mirrors, dichroic mirrors, refractors) that, for example, directly or indirectly direct the output from the front end or proximal end 3812b of one or more optical fibers 3812 to the eye 3808 of the end user 3802 via a partially transparent display surface 3806. Although not shown, the projection subsystem 3804 may include one or more optical components that modulate the depth of the Z-axis orientation of the pixel data. This may, for example, take the form of a flexible reflective (e.g., nitride sputter coated with aluminum) film and one or more electrodes operated to cause deflection of the flexible reflective film. The flexible reflective film is positioned to reflect and focus light emitted from the front end or proximal end 3812b of the one or more optical fibers 3812. The flexible reflective film may be selectively operated to focus light in the Z dimension or axis based on a depth map for the pixel data or information. The flexible reflective film may employ Gaussian points to create the appearance of depth, with certain virtual objects in the image appearing in focus while other objects are out of focus. Additionally or alternatively, the system may use one or more Kerr effect lenses.

[0112] Although not necessary for head mounted embodiments, the optical fiber 3812 and optional yoke 3814 can be supported for movement in one or more directions. For example, the optical fiber 3812, and optional yoke 3814 can be supported for 2, 3 or more degrees of freedom of movement via a gimbal 3816. The gimbal 3816 can include a turntable 3816a, a first actuator 3818a (e.g., a motor, solenoid, piezoelectric sensor) that can be operated to rotate or turn around a first axis 3820a. The gimbal 3816 can include a bracket 3816b supported by a frame 3816c on the turntable 3816a, and a second actuator 3818b (e.g., a motor, solenoid, piezoelectric sensor) that can be operated to rotate or turn around a second axis 3820b. The gimbal 3816 may include a rod 3816d rotatably supported by a bracket 3816b, a third actuator 3818c (e.g., a motor, a solenoid, a piezoelectric sensor) operable to rotate or turn around a third axis 3820c. The first, second, and third axes (collectively 3820) may be orthogonal axes.

[0113] exist Fig.38 In the illustrated embodiment, the virtual image generation system 3800 includes a control subsystem 3822. The control subsystem 3822 can take any of a number of different forms, one of which is Fig.38 Shown in.

[0114] The control subsystem 3822 includes a plurality of controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs) 3824, digital signal processors (DSPs), graphics processing units (GPUs) 3826, other integrated circuit controllers such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), such as field PGAs (FPGAs), and / or programmable logic controllers (PLUs). Fig.38 In the illustrated embodiment, the microprocessor 3824 controls the overall operation, while the GPU 3826 renders frames (e.g., groups of pixel data) into one or more frame buffers 3828a-3828n (collectively 3828). Although not shown, one or more additional integrated circuits may control the reading of frames into and / or out of the frame buffers 3828 and the operation of the piezoelectric sensors or electrodes 3814a, synchronizing the two to produce a two-dimensional or three-dimensional image. The reading of frames into and / or out of the frame buffers 3828 may employ dynamic addressing, for example where the frames are over-rendered.

[0115] The control subsystem 3822 includes one or more non-transitory computer or processor readable media to store instructions and data. The non-transitory computer or processor readable media may, for example, include a frame buffer 3828. The non-transitory computer or processor readable media may, for example, include one or more non-volatile memories, such as read-only memory (RAM) 3830 or flash memory. The non-transitory computer or processor readable media may, for example, include one or more volatile memories, such as random access memory (RAM) 3832. The control subsystem 3822 may include other volatile and non-volatile memories, including rotating media storage and solid-state storage devices.

[0116] In implementations employing actuators (collectively 3818 ), the control subsystem 3822 optionally includes one or more dedicated motor controllers 3834 communicatively coupled to drive the actuators 3818 via motor control signals.

[0117] The control subsystem 3822 optionally includes one or more communication ports 3836a, 3836b (collectively 3836) that provide communication with different other systems, components or devices. For example, the control subsystem 3822 may include one or more wired interfaces or ports 3836a that provide wired or optical communication. For another example, the control subsystem 3822 may include one or more wireless interfaces or ports, such as one or more radio devices (i.e., wireless transmitters, receivers, transceivers) 3836b that provide wireless communication.

[0118] As shown, wired interface or port 3836a provides wired or optical communication with environmental imaging system 3838, which includes one or more cameras 3838a positioned and oriented to capture images of the environment in which end user 3802 is located. These can be used to sense, measure or collect information about end user 3802 and / or the environment. For example, these can be used to detect or measure the movement and / or orientation of end user 3802 or a part of the body of end user 3802, such as head 3810. As shown, wired interface or port 3836a optionally provides wired or optical communication with structured lighting system 3840, which includes one or more light sources 3840a positioned and oriented to illuminate end user 3802, a part of end user 3802, such as head 3810, and / or the environment in which end user 3802 is located.

[0119] As shown, a wireless interface or port 3836b provides wireless (e.g., RF, microwave, IR) communication with one or more head-mounted sensor systems 3842, which include one or more inertial sensors 3842a to capture inertial measurements indicating movement of the head 3810 of the end user 3802. These can be used to sense, measure, or collect information about the movement of the head of the end user 3802. For example, these can be used to detect or measure the movement, velocity, acceleration, and / or orientation of the head 3810 of the end user 3802. As shown, a wired interface or port 3836a optionally provides wired or optical communication with an imaging system 3842, which includes, for example, one or more forward-facing imagers or cameras 3842a. These can be used to capture information about the environment in which the end user 3802 is located. These can be used to capture information indicating the distance and orientation of the end user 3802 relative to that environment and specific objects in that environment. When worn on the head, the forward-pointing imager or camera 3842a is particularly suitable for capturing information indicating the distance and orientation of the end user's head 3810 relative to the environment in which the end user 3802 is located and specific objects in that environment. These can be used, for example, to detect head movement, speed and / or acceleration of head movement. These can be used, for example, to detect or infer the center of attention of the end user 3802, for example based at least in part on the orientation of the end user's head 3810. The orientation can be detected in any direction (e.g., up / down, left / right relative to the end user's frame of reference).

[0120] In some implementations, all communications may be wired, while in other implementations, all communications may be wireless. In further implementations, the selection of wired and wireless communications may differ from Fig.38 Thus, the particular choice of wired or wireless communication should not be considered limiting.

[0121] The various components of the control subsystem 3822, such as the microprocessor 3824, GPU 3826, frame buffer 3828, ROM 3830, RAM 3832, and / or optionally a dedicated motor controller 3834, may be communicatively coupled via one or more communication channels, such as one or more buses 3846 (only one shown). The bus 3846 may take different forms, including an instruction bus, a data bus, an address bus, other communication buses, and / or a power bus.

[0122] The ability to predict head motion allows the virtual image generation system 3800 ( Fig.38 )--such as augmented reality systems, quickly update the presentation of images and / or adapt to or compensate for head movements. For example, by the use of the possibility that only head movement is perceived, subsequent frames can be rendered or read out earlier. As will be apparent from the discussion herein, adaptation or compensation can take various forms. For example, subsequent frames can be rendered or read out with an offset field of view or a shift to or to the center of an area of ​​attention or focus of the end user. For another example, subsequent frames can be re-rendered or read out to adapt to or compensate for changes caused by head movement. For example, in certain display or projection technologies (e.g., "flying pixel" technology, in which pixels are displayed sequentially, such as raster scanning, spiral scanning, Lissajous scanning), rapid head movements may cause the spacing between pixels of the frame presented to the end user to change. Adaptation or compensation may include adapting to or compensating for changes in such pixel spacing. For example, the size or predicted size of some pixels can be adjusted relative to other pixels. For another example, the brightness or predicted brightness of some pixels can be adjusted relative to other pixels. As a further example, subsequent frames can be rendered or read out with variable resolutions between different parts of the resulting image. Other adaptation or compensation techniques will be apparent from this discussion. In other aspects, many of these same techniques may be employed for purposes other than adaptation or compensation, and may be used independently for predictive head tracking, perceptual head tracking, and / or in conjunction with non-"flying pixels" based display or projection techniques.

[0123] End-user motion, such as head motion, can have a significant effect on the image. As the augmented reality system attempts to render subsequent frames of a frame in concert with the head motion, the resulting image of the virtual object may be compressed, expanded, or otherwise distorted. This is at least in part a result of the fact that for many display or rendering technologies (i.e., "flying pixel" technologies), the complete image for any given frame is not rendered or displayed simultaneously, but rather is rendered or displayed pixel by pixel. Therefore, there is no true instantaneous field of view for these display or rendering technologies. This may occur in different forms across many different types of image generation techniques, such as raster scanning, spiral scanning, or Lissajous scanning methods. One or more "white" or blank frames or images can mitigate the effects of some rapid head motions.

[0124] For example, Fig.36A An exemplary distortion produced in a raster scan 3600a during rapid lateral movement of the end user's head is shown. The distortion may be non-linear because the head motion may accelerate after initiation and slow down before termination. The distortion depends on the direction, speed and acceleration of the head motion and the direction of raster scan pixel generation (e.g., right to left, top to bottom).

[0125] For example, Fig.36B An exemplary distortion produced in a raster scan 3600 during a rapid vertical movement of the end user's head is shown. The distortion may be non-linear because the head motion may accelerate after initiation and slow down before termination. The distortion depends on the direction, speed and acceleration of the head motion and the direction of raster scan pixel generation (e.g., right to left, top to bottom).

[0126] As another example, Fig.37A An exemplary distortion produced in a spiral scan line 3700a during a rapid lateral motion of the end user's head to the left is shown. The distortion may be nonlinear because the head motion may accelerate after initiation and slow down before termination. The distortion depends on the direction, velocity, and acceleration of the head motion and the direction of the spiral scan pixel generation (e.g., clockwise, increasing radius). The spacing between consecutive loops or circles of the spiral scan line 3700a as shown increases in the direction of the head motion (e.g., to the left of the drawing), and decreases in the opposite direction (e.g., to the right of the drawing).

[0127] As another example, Fig.37B 37 shows an exemplary distortion in spiral scan line 3700b produced during a rapid lateral motion of the end user's head to the left. The distortion may be nonlinear because the head motion may accelerate after initiation and slow down before termination. In fact, if Fig.37B3700b, the distortion may be highly elliptic and decentralized. The distortion depends on the direction, speed, and acceleration of the head motion and the direction of the spiral scan pixel generation (e.g., clockwise, increasing radius). The spacing between successive loops or circles of the spiral scan line 3700b as shown increases in the direction of the head motion (e.g., toward the left of the drawing). For locations where the head motion is too fast for the system, the leftmost portion of each loop or circle may be located in the same direction as the head motion relative to the starting point of the spiral scan line 3700b, as shown. Fig.37B shown.

[0128] One advantage of using a spiral scan pattern is that the conversion of addresses to the image buffer is independent of the direction of motion (eg, head motion, hand motion for a handheld pico projector).

[0129] The above system is used in all embodiments described below. In one embodiment, the system can be used for predictive head tracking based on predicting the movement of the user's focus. Figure 1 Operation of method 100 in an augmented reality system employing predictive head tracking is shown according to one illustrated embodiment.

[0130] At 102, an augmented reality system (e.g., a controller subsystem and / or its processor) presents a plurality of frames as images to an end user of the augmented reality system. The frames typically include pixel information specified for generating one or more virtual objects in a field of view. As previously described, the virtual objects may take any of a variety of different virtual object forms or formats that may visually represent physical objects or information, data, or logical structures that may be represented. Non-limiting examples of virtual objects may include: virtual text objects, virtual digital objects, virtual alphanumeric objects, virtual label objects, virtual field objects, virtual chart objects, virtual map objects, virtual tool objects, or virtual visualization representations of physical objects.

[0131] At 104 , the augmented reality system selects one or more virtual objects based at least on the input indicative of the end user's attention.

[0132] The input may be an actual selection by the end user. The selection may be made by the end user in real time or may have been pre-specified. Thus, the end user may select a certain set of virtual tools as a class of virtual objects that the end user typically focuses on or pays attention to more than other objects.

[0133] Input can be derived from a variety of sources. Input can be related to the virtual object itself. Input can be additionally or alternatively related to physical objects in the field of view of the end user or the field of view of the display or projector. Input can be additionally or alternatively related to the user itself, such as the orientation and / or orientation of the end user and / or a part of the end user (e.g., head, eyes), or historical attributes. The historical attributes can be end-user specific, or more general or universal. Historical attributes can indicate the characteristics of a defined set of end users. End-user characteristics can, for example, include head movement speed, head movement acceleration, and / or the relationship between head movement and eye movement (such as the ratio of one to another). End-user characteristics tracked by historical attributes can even include a tendency to indicate that a given end user pays attention to certain virtual objects. These can be specified by virtual object type (e.g., text, chart), recent virtual objects (e.g., newly appeared objects), virtual object movement (e.g., large changes from image to image, fast or rapid movement, direction of movement) and / or virtual object characteristics (such as color, brightness, size).

[0134] At 106, for each of at least some of the plurality of frames presented to the end user, the augmented reality system (e.g., the controller subsystem and / or its processor) determines a location at which a virtual object appears in the end user's field of view relative to the end user's reference frame. For example, the augmented reality system may determine the location of a newly introduced virtual object, a virtual object of a defined type, a virtual object that moves quickly or over a large distance, or a virtual object that was once the end user's focus.

[0135] At 108, the augmented reality system adjusts the presentation of at least one subsequent frame based at least in part on the determined location of the virtual object in the end user's field of view. Various ways of adjusting the appearance of the virtual object in the field of view are discussed herein, including non-exhaustive adaptation or compensation, adjusting pixel size, adjusting pixel brightness, adjusting resolution, windowing, and / or blanking or flickering.

[0136] Figure 2 Another method 200 of operating in an augmented reality system is shown according to an illustrated embodiment. Figure 1 Method 200 may be employed when performing actions 104 and / or 106 of method 100 .

[0137] Method 200 employs techniques for predicting head motion based on features of virtual objects that are or will be presented to an end user. For example, it is anticipated that a newly introduced virtual object or the motion of a virtual object (e.g., due to surprise, speed, and / or distance) may attract the user's attention to a virtual object, resulting in a head motion to bring a particular virtual object into or closer to the center of the end user's field of view. Additionally or alternatively, the augmented reality system may rely on other features of virtual objects when assessing which ones are most likely to attract attention. For example, highly attractive (e.g., flickering, shimmering), large, fast-moving, or bright virtual objects are more likely to attract attention than other virtual objects.

[0138] For the case of a newly introduced virtual object, at #ab02, the augmented reality system (e.g., a controller subsystem and / or its processor) selects and / or determines the location where the virtual object appears when it is newly introduced into the end user's field of view. A virtual object is considered to be newly introduced when it does not appear in a relevant frame previously (temporally) presented to the end user. In particular, the augmented reality system relies on the fact that a newly introduced virtual object is more likely to attract eventual attention than a virtual object that appears in an immediately preceding frame. Additionally or alternatively, the augmented reality system can rely on other features of the virtual object by evaluating which is most likely to attract attention, for example, to select or prioritize among multiple newly introduced virtual objects. For example, a highly attractive (e.g., flickering, shimmering), large, fast-moving, or bright virtual object may be more able to attract attention than other virtual objects.

[0139] For the case of a moving virtual object, at 204, the augmented reality system (e.g., a controller subsystem and / or its processor) selects and / or determines where the virtual object appears in a new orientation in the frame relative to the orientation of the same virtual object in at least one previous frame. In this way, sudden movements, rapid movements, and / or large spatial orientation shifts of the virtual object from one frame to one or more subsequent frames may tend to attract the attention or focus of the end user. Additionally or alternatively, the augmented reality system may rely on other features of the virtual object by evaluating which is most likely to attract attention, for example, to select or prioritize between multiple newly introduced virtual objects. For example, a highly attractive (e.g., flickering, shimmering), large, or bright virtual object is more likely to attract attention than other virtual objects.

[0140] Figure 3 A method 300 is shown operating in an augmented reality system according to one illustrated embodiment. Figure 1 Method 300 is employed when action 108 in method 100 is performed.

[0141] At 302, the augmented reality system (e.g., a controller subsystem and / or its processor) presents at least one subsequent frame whose center is at least moved toward, if not centered on, the position where the virtual object appears in the determined field of view of the end user. The center of the subsequent frame or image may be moved to co-locate at the position of the selected virtual object that is predicted to attract the end user's attention. Alternatively, the center of the subsequent frame may be moved to approach the position of the selected virtual object that is predicted to attract the end user's attention. This can be performed in two or three dimensions. For example, the two-dimensional or three-dimensional orientation of the virtual object can be used to adjust the field of view of the subsequent image in two or three dimensions, respectively. The moving subsequent frame or image is preferably consistent in time with the predicted end user head movement. Therefore, the moving subsequent frame or image should be presented to the end user as consistent in time with the real head movement as possible. As discussed herein, this may take into account velocity, acceleration, and changes in velocity and acceleration.

[0142] Figure 4 A method 400 of operating in an augmented reality system is shown according to an illustrated embodiment. Figure 1 Method 400 is used when method 100 is used.

[0143] Optionally, at 402, the augmented reality system receives information indicating the identity of the end user. The information can take any of a variety of different forms. For example, the information can be a username or other user identifier (e.g., encrypted) entered by the end user or from a transponder, magnetic stripe, or machine-readable symbol associated with the end user. For example, the information can include biometric information indicating one or more physical characteristics of the end user. In a particularly advantageous implementation, the augmented reality system can receive image data representing a portion of one or both eyes of the end user (e.g., retina). For example, the augmented reality system can project light into one or both eyes of the end user, for example, via one or more optical fibers. The light can be modulated to, for example, increase the signal-to-noise ratio and / or limit heating of the eye. The image sensor can capture an image of a portion of the eye, for example, via one or more optical fibers of the projected light, the optical fiber providing a bidirectional path. Alternatively, a dedicated optical fiber can also be used. As a further alternative, the image sensor can be placed in a position close to the eye, eliminating the use of an optical fiber as a return path to the image sensor. Certain parts of the human eye (e.g., retinal blood vessels) may be considered to have sufficient unique features to serve as a unique end-user identifier.

[0144] Optionally, at 404, the augmented reality system retrieves at least one user-specific historical attribute for the end user based on the received information indicating the identity of the end user. The user-specific historical attribute may indicate at least one of the following: a previous head movement velocity for the end user, a previous head movement acceleration for the end user, a previous eye movement to head movement relationship for the end user, a tendency of the end user to pay attention to virtual objects of certain types or having certain characteristics.

[0145] At 406, the augmented reality system (e.g., the controller subsystem and / or its processor) predicts the occurrence of head movement of the end user based at least in part on the determined location of the virtual object in the end user's field of view. Again, the augmented reality system can rely on the attractiveness of the virtual object to predict head movement, such as on an end-user-by-end-user basis.

[0146] The augmented reality system may employ the estimated velocity and / or estimated change in velocity or estimated acceleration to at least partially synchronize image presentation with the predicted head movement of the end user. The estimated change in velocity in the predicted head movement may be based on an extended range between a first defined time after the predicted head movement begins and a second defined time before the predicted head movement ends.

[0147] At 408, the augmented reality system estimates at least one value indicative of an estimated speed of the predicted end-user head movement. The augmented reality system may estimate the speed based on one or more values, parameters, or features. For example, the augmented reality system may rely on the range of motion required to move the end-user head to a new position to observe a selected or identified virtual object. The augmented reality system may rely on an average speed for a sample of people, or may rely on historical head movement speeds for a particular end-user. The augmented reality system may rely on historical attributes for a particular end-user. The speed span may be expressed in terms of angular velocity.

[0148] At 410, the augmented reality system estimates at least one change in velocity in the predicted end-user head motion that occurs within a range of head motion between the predicted head motion start and the predicted head motion end. The change in velocity may occur in different increments throughout certain portions of the predicted range of motion.

[0149] At 412, the augmented reality system estimates at least one value of an estimated acceleration indicative of a predicted end-user head motion. The estimated acceleration may be performed over the entire range of head motion or only over a portion thereof. The estimated acceleration may be performed at discrete intervals of the range of head motion. The estimate of acceleration may be determined for one or more intervals of some defined duration after the head motion begins. The estimate of acceleration may be determined for one or more intervals of some defined duration before the head motion terminates. Spacing the estimates from the start and / or end points may avoid large variations in the acceleration measurements.

[0150] Optionally, at 414, the augmented reality system determines at least one value that at least partially accommodates or compensates for the estimated speed of the predicted head movement of the end user. For example, the augmented reality system may determine a value related to the total number of frames to be presented in a given time and / or a value specifying where and / or how fast one or more virtual objects should move across a series of images to be rendered and / or presented. These may be used to render subsequent frames.

[0151] Optionally, at 416, the augmented reality system renders at least one subsequent frame based at least in part on at least one value that at least in part compensates for the estimated speed of the predicted head movement of the end user. For example, the augmented reality system can determine a value related to the total number of frames to be presented in a given time and / or a value that specifies where and / or how fast one or more virtual objects should move across a series of images to be rendered and / or presented. These can be used to render subsequent frames.

[0152] In another embodiment, the system can be used to predict head tracking based on historical attributes of the user. Figure 5 A method 500 of operating in an augmented reality system employing predictive head tracking is shown according to one illustrated embodiment.

[0153] The augmented reality system can employ historical attributes when performing predictive head tracking. The historical attributes may be end-user specific or more generic or generalized. The historical attributes may indicate a defined set of end-user characteristics. The end-user characteristics may, for example, include head movement velocity, head movement acceleration, and / or a relationship between head movement and eye movement (e.g., a ratio of one to the other). The end-user characteristics tracked by the historical attributes may even include a tendency to indicate that a given end-user pays attention to certain virtual objects.

[0154] At 502, the augmented reality system receives information indicative of an identity of an end user. The information can take any of a number of different forms, such as information actively provided by the end user, information read from a non-transitory storage medium, information read from the user (e.g., biometric data or characteristics), or information inferred from end user behavior.

[0155] At 504, the augmented reality system retrieves at least one user-specific historical attribute for the end user based at least in part on the received information indicating the identity of the end user. The identity information can be received, generated, or determined in any of a number of different ways.

[0156] At 506, the augmented reality system provides frames to the end user based at least in part on the retrieved at least one user-specific historical attribute for the end user. For example, the augmented reality system can provide frames from a frame buffer to a projector or display device (e.g., a light source paired with one or more optical fibers), or can render frames to a frame buffer. The augmented reality system can provide light via at least one optical fiber that is movable in at least two-axis directions. The augmented reality system can receive image information indicating an image of at least a portion of an eye of the end user via at least one optical fiber, which also provides frames to the end user.

[0157] Figure 6 According to one illustrated embodiment, a method 600 is shown for operating in an augmented reality system using predictive head tracking. Figure 5 Method 600 is used when action 504 of method 500 is performed.

[0158] At 602, the augmented reality system retrieves at least one historical attribute that provides an indication of at least one head movement attribute for an end user. The head movement attribute indicates at least one previous head movement of the end user. The historical attribute can be stored on a non-transitory medium, such as in a database or other logical structure.

[0159] At 604 , the augmented reality system retrieves at least one historical attribute that provides an indication of a head movement speed for at least one previous head movement of the end user.

[0160] At 606 , the augmented reality system retrieves at least one historical attribute of an indication of a change in head movement velocity across at least a portion of at least one previous range of head movement of the end user.

[0161] At 608, the augmented reality system retrieves at least one historical attribute providing an indication of head movement acceleration for at least one previous head movement of the end user.

[0162] At 610, the augmented reality system retrieves at least one historical attribute that provides an indication of a relationship between head movement and eye movement for at least one previous head and eye movement combination for the end user. The relationship can, for example, be expressed as a ratio of a head movement value representing the head movement of at least one previous head movement of the end user and a value representing at least one previous eye movement. The values ​​can represent the amount of head and eye movement, respectively, for example, expressed as an angular change. The ratio may be a ratio of a historical average of the end user's head movement and a historical average of the eye movement. Additionally or alternatively, other relationships between head and eye movements can be employed, such as velocity or acceleration.

[0163] Figure 7 According to one illustrated embodiment, a method 700 is shown for operating in an augmented reality system using predictive head tracking. Figure 5 Method 700 is employed when action 506 of method 500 is performed.

[0164] At 702, the augmented reality system predicts at least an endpoint of the end user's head movement. For example, when the appearance of a virtual object is used for the predicted head movement, the relative position of a specific virtual object can be used as the endpoint.

[0165] At 704, the augmented reality system renders at least one subsequent frame to at least one image buffer. The at least one subsequent frame is shifted at least toward, or even to, the predicted end point of the head movement.

[0166] Figure 8 According to one illustrated embodiment, a method 800 is shown for operating in an augmented reality system using predictive head tracking. Figure 7 Method 800 is employed when action 704 of method 700 is performed.

[0167] At 802, the augmented reality system renders a plurality of subsequent frames in a manner at least partially adapted to at least one head motion attribute for the end user, the plurality of subsequent frames being offset at least toward the predicted end point of the head motion. The head motion attribute may indicate different physical characteristics of the head motion, in particular historical physical characteristics of the end user's head motion. The head motion attribute may, for example, include one or more of: historical head motion velocity for the end user, historical head motion acceleration for the end user, and / or historical relationship (e.g., ratio) between head motion and eye motion for the end user. The offset may be achieved by rendering the subsequent frames by offsetting the corresponding image or the center of the corresponding image relative to the image corresponding to the previous frame.

[0168] Fig. 9According to one illustrated embodiment, a method 900 is shown for operating in an augmented reality system using predictive head tracking. Figure 7 Method 900 is used when action 703 of method 700 is performed.

[0169] At 902, an augmented reality system predicts occurrence of head movement of an end user based at least in part on the appearance of a virtual image in a field of view of the end user.

[0170] The appearance may be the appearance in time of a new virtual object newly introduced into the field of view presented to the end user relative to a previous frame presented as an image to the end user. Alternatively or additionally, the appearance may be the appearance of the virtual object in a new orientation in the field of view presented to the end user relative to the orientation of the virtual object previously presented to the end user. The prediction may, for example, take into account factors. For example, the prediction may be based in part on the size or significance of the virtual object, the amount or percentage of the orientation change, the speed, sudden acceleration, or other changes in the orientation of the virtual object.

[0171] The system can also be used to dynamically control pixel characteristics. Fig.10 A method 1000 of operating in an augmented reality system is shown according to one illustrated embodiment.

[0172] At 1002, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) detects an indication that spacing between some pixels in a frame presented to an end user will be different than spacing between other pixels in the same frame. For example, the augmented reality system may detect an indication that spacing between pixels of a first group of pixels in a frame presented to the end user will be different than spacing between pixels of at least a second group of pixels in the frame presented to the end user. For example, when pixels of a frame are presented sequentially (e.g., readout of a frame buffer) over a period of time (e.g., a "flying pixel" pattern, such as a raster scan pattern, a spiral scan pattern, a Lissajous scan pattern), rapid head movements may result in changes in pixel spacing between different portions of an image or frame.

[0173] At 1004, in response to detecting that spacing between some pixels in a frame presented to an end user will be different from spacing between other pixels in the frame, the augmented reality system provides at least a first set of pixels adjusted to at least partially compensate for at least one pixel characteristic perceptible by the end user to at least a portion of at least one subsequent frame. This may at least partially compensate for spacing between pixels in different portions of an image presented to the end user.

[0174] Fig.11 A method 1100 is shown operating in an augmented reality system according to one illustrated embodiment. Fig.10Method 1100 is used when method 1000 is used.

[0175] Optionally, at 1102, the augmented reality system (e.g., the controller subsystem and / or its processor) receives a signal indicative of an output of at least one head-mounted inertial sensor worn by a user. The inertial sensor can take a variety of forms, such as a gyroscope sensor or an accelerometer. The inertial sensor can be a single-axis or multi-axis device. The inertial sensor can take the form of a MEMS device.

[0176] Optionally, at 1104, the augmented reality system receives a signal indicative of the output of at least one head-mounted imager worn by the user. The imager may, for example, take the form of a digital camera or other image capture device. These may be forward-facing cameras to capture a field of view that is at least close to the end user's field of view.

[0177] Optionally, at 1106, the augmented reality system detects head movement that exceeds a nominal head movement value. For example, the augmented reality system may detect head movement that exceeds a nominal velocity and / or a nominal acceleration. The augmented reality system may employ signals from an inertial sensor to detect movement, and in particular acceleration. The augmented reality system may employ signals from a head-mounted camera to detect changes in the orientation of physical objects in the surrounding environment, in particular fixed physical objects such as walls, floors, and ceilings. The augmented reality system may employ any number of image processing techniques. The detected orientation changes allow the augmented reality system to determine changes in head orientation, velocity of movement, and acceleration. In addition to or in lieu of inertial sensor and head-mounted imaging information, the augmented reality system may employ other information. For example, the augmented reality system may employ signals from a system that monitors the surrounding environment and is not worn by the user but tracks the user. Such a system may employ one or more imagers, such as a digital camera, to monitor the surrounding environment. The imager detects movement of the end user and a portion of the end user such as the head. Again, a variety of image processing techniques may be employed. Such a system may be advantageously paired with a structured light system. Alternatively, Method #CB00 may be performed independent of detected or even predicted head movement.

[0178] At 1108, the augmented reality system selects a first set of pixels of the frame, for example, based on the direction of the detected head movement. The augmented reality system can additionally select a first set of pixels of the frame based on other criteria, such as the speed of the detected head movement.

[0179] At 1110, the augmented reality system adjusts at least one of the size and / or brightness of at least some of the pixels of the first group of pixels of at least one subsequent frame. The adjustment can be designed to at least accommodate or at least partially compensate for undesirable changes in the frames or images caused by head movement.

[0180] Optionally, the augmented reality system renders at least one subsequent frame at 1112. The rendered subsequent frame includes pixel information that has been adjusted to at least partially accommodate or compensate for undesirable changes in the frame or image caused by head movement.

[0181] Optionally, at 1114, the augmented reality system reads at least one subsequent frame from at least one frame buffer storing one or more subsequent frames. For example, the augmented reality system can selectively read at least one subsequent frame from at least one frame buffer. This can take advantage of over-rendering, in which the frame is over-rendered relative to the size of the image area or field of view. The system, especially when the system is worn on the head, will in most cases be dedicated to a fixed display surface with a known area and a known resolution. This is in contrast to computers and other other devices that are intended to provide signals to displays of multiple sizes and resolutions. Therefore, the augmented reality system selectively reads in or out of the frame buffer instead of reading in or out the entire frame from the frame buffer. If the new frame of the subsequent image that is rendered to create displays a portion that is outside the previous image, then the over-running of the GPU that may be required when over-rendering is not performed, over-rendering can prevent this situation. For example, without over-rendering, the augmented reality system will need to render a new frame every time the end user's head is moved. In conjunction with over-rendering, a set of dedicated electronic devices can be used to select or read out the desired portion of the over-rendered frame, which is essentially a window moving in the previously rendered frame.

[0182] Fig.12 A method 1200 is shown operating in an augmented reality system according to one illustrated embodiment. Fig.11 Method 1200 is employed when actions 1108 and 1110 of method 1100 are performed.

[0183] At 1202, an augmented reality system (e.g., a controller subsystem and / or its processor) selects at least a first group of pixels of a frame such that the first group of pixels is in a given direction (e.g., the same direction, opposite direction) relative to a detected head movement direction.

[0184] At 1202 , the augmented reality system adjusts a pixel size of the selected group of pixels as pixels in a first group of pixels of at least one subsequent frame presented to the user.

[0185] For example, the augmented reality system may select a first group of pixels of a frame so that the first group of pixels are located in the same direction as the detected head movement relative to the other pixels. For example, the first group of pixels are relatively oriented toward the left in the image relative to a second group of pixels that are generally oriented toward the right in the image. For example, the first group of pixels are relatively oriented toward the top in the image relative to a second group of pixels that are generally oriented toward the bottom in the image. The augmented reality system may provide one or more subsequent frames or images in which the pixels in the first group have an increased size relative to some other pixels in the subsequent frames. This may at least partially accommodate or at least partially compensate for diffusion between pixels resulting from rapid head movements that the augmented reality system cannot keep up with.

[0186] For example, the augmented reality system may select a first set of pixels of a frame such that the first set of pixels are located in a direction opposite to the detected head motion relative to other pixels. The augmented reality system may provide one or more subsequent frames or images in which the first set of pixels have a reduced size relative to some other pixels in the subsequent frames. This may at least partially accommodate or at least partially compensate for inter-pixel diffusion resulting from rapid head motion that the augmented reality system cannot keep up with.

[0187] Adjusting (e.g., increasing, decreasing) the size of the selected group of pixels may include adjusting a variable focus component. Adjusting (e.g., increasing, decreasing) the size of the selected group of pixels may include adjusting a source of a variable size. Adjusting (e.g., increasing, decreasing) the size of the selected group of pixels may include adjusting a jitter.

[0188] As a further example, the augmented reality system may select a first group of pixels of a frame such that the first group of pixels are located in the same direction as the detected head motion relative to other pixels. The augmented reality system may provide one or more subsequent frames or images in which the pixels in the first group have increased brightness relative to some other pixels in subsequent frames. This may at least partially accommodate or at least partially compensate for inter-pixel diffusion resulting from rapid head motion that the augmented reality system cannot keep up with.

[0189] As yet a further example, the augmented reality system may select a first group of pixels of a frame such that the first group of pixels are located in a direction opposite to the detected head movement relative to other pixels. The augmented reality system may provide one or more subsequent frames or images in which the pixels in the first group have reduced brightness relative to some other pixels in subsequent frames. This may at least partially accommodate or at least partially compensate for inter-pixel diffusion resulting from rapid head movement that the augmented reality system cannot keep up with.

[0190] As above, the augmented reality system can adjust only the size of selected pixels, only the brightness of selected pixels, or both the size and brightness of selected pixels. Further, the augmented reality system can adjust the brightness of some pixels, the size of other pixels, or even the brightness and size of other pixels, and / or not adjust the brightness or size of further pixels.

[0191] The system can also be used to dynamically update less than an entire frame based on the entire frame, as shown below. Fig.13 A method 00 of operating in an augmented reality system is shown according to one illustrated embodiment.

[0192] At 1302, the augmented reality system (e.g., the controller subsystem and / or its processor) renders a first complete frame to an image buffer. The first complete frame includes pixel information for sequential presentation of pixels to form an image of multiple virtual objects. The first complete frame can take a variety of forms suitable for different display technologies. For example, the complete frame may include pixel information suitable for forming a complete raster scan frame, which can be an interlaced raster scan frame with two fields. Each field of the interlaced raster scan includes a plurality of lines, the first field including odd lines and the second field including even lines. The odd and even lines can be interlaced, at least as displayed to the end user. A particularly advantageous technique employs spiral scan lines. The spiral scanning method can employ a single field per frame, for example, consisting of a single spiral trace. Alternatively, the spiral scanning method can employ two or more fields per frame, for example, consisting of two or more spiral traces presented sequentially. The spiral traces can advantageously be simply interlaced or nested by introducing a phase shift between each field of the frame. Another technique employs a Lissajous scanning method. The Lissajous scanning method may employ a single field per frame, e.g. consisting of a single Lissajous trace. Alternatively, the Lissajous scanning method may employ two or more fields per frame, e.g. consisting of two or more Lissajous traces presented sequentially. The Lissajous traces may advantageously be simply interleaved or nested by introducing a phase shift between each field of the frame.

[0193] At 1304, the augmented reality system begins rendering of a first complete frame. This may include reading out a frame buffer, for example to drive a light source and ends of one or more optical fibers. The reading may include dynamically determining which portion of the frame buffer is to be read out.

[0194] Optionally, the augmented reality system detects that the end user's head movement exceeds a nominal head movement value at 1306. This can be done using any of the different methods discussed previously.

[0195] At 1308, the augmented reality system dynamically interrupts presentation of the first complete frame before presentation of the entire first complete frame is completed. In detail, the augmented reality system begins presentation of an update to the first complete frame. At least a portion of the pixel information in the update to the first complete frame has changed from the first complete frame. For example, in an interlaced raster scan-based system, the augmented reality system may present all or a portion of the first field, replacing the second field with an updated second field. For another example, in an interlaced spiral scan-based system, the augmented reality system may present all or a portion of the first field (e.g., a first spiral scan line or trace), replacing the second field with an updated second field (e.g., an updated second spiral scan line or trace that is different from the initial second spiral scan or trace). Similarly, in an interlaced Lissajous scan-based system, the augmented reality system may present the first field (e.g., a first Lissajous scan line or trace, i.e., a complete Figure 8 The presentation may be interpreted during the presentation of a field, such as during the presentation of the first or second or third field. The presentation may be interpreted during the presentation of any given line (e.g., a complete cycle of a raster scan, a spiral or Lissajous scan).

[0196] Fig.14 A method 1400 for operating in an augmented reality system is shown according to one illustrated embodiment. Fig.13 Method 1400 is used when method 1300 is used.

[0197] At 1402, an augmented reality system (eg, a controller subsystem and / or a processor thereof) renders an updated first full frame. The updated first full frame includes pixel information that differs from pixel information of the first full frame in at least one aspect.

[0198] Rendering the updated first complete frame may include rendering the updated complete frame with the first field and at least the second field. The second field may be interlaced with the first field, typically presented sequentially following presentation of the first field. For example, the first field may be composed of even-numbered lines in a raster scan and the second field may be composed of odd-numbered lines. For another example, the first field may be composed of first spiral scan lines or first Lissajous scan lines and the second field may be composed of second spiral scan lines or second Lissajous scan lines. Thus, rendering the updated first complete frame may include rendering the updated complete frame using the first field and at least the second field, the second field being interlaced with at least the first field.

[0199] At 1404, the augmented reality system presents a portion of the updated first complete frame to replace a corresponding portion of the first complete frame. Thus, after an initial non-updated first complete frame is interrupted, a portion of the updated frame replaces all or a portion of the first complete frame.

[0200] For example, the augmented reality system presents the second field of the updated first complete frame to replace the second field of the initial (i.e., not updated) first complete frame. For another example, the augmented reality system may present the second portion of the first field accompanied by the second field of the updated first complete frame to replace the corresponding portion of the first field of the first initial (i.e., not updated) first complete frame and the entire second field.

[0201] For another example, the augmented reality system may present a portion of a field (e.g., a line, a portion of a line, a group of pixels, a pixel) of the updated first complete frame in place of a corresponding portion of a corresponding field of the first complete frame. For example, the augmented reality system may present a portion of a field of an updated first complete frame of an updated raster scan frame in place of a corresponding portion of a corresponding field of an initial (i.e., non-updated) first complete frame of a raster scan frame.

[0202] As another example, the augmented reality system may present lines of the updated first complete frame in place of corresponding lines of the initial (i.e., non-updated) first complete frame. As yet another example, the augmented reality system may present a spiral line of the updated first complete frame in place of a corresponding spiral line of the initial (i.e., non-updated) first complete frame. As a further example, the augmented reality system may present a portion of a line of the updated first complete frame in place of a corresponding portion of a corresponding line of the initial (i.e., non-updated) first complete frame. As yet a further example, the augmented reality system may present at least one pixel of the updated first complete frame in place of a corresponding portion of at least one pixel of the initial (i.e., non-updated) first complete frame. As yet another additional example, the augmented reality system may present a complete cycle of a Lissajous pattern scan of the updated first complete frame in place of a corresponding portion of a complete cycle of a Lissajous pattern scan of the initial (i.e., non-updated) first complete frame.

[0203] Fig.15 A method 1500 is shown operating in an augmented reality system according to one illustrated embodiment. Fig.13 When method 1300 is used, method 1500 is used.

[0204] At 1502, an augmented reality system (e.g., a controller subsystem and / or its processor) renders a first complete frame to a frame buffer. The first complete frame may, for example, include a first field and at least a second field. The first field may, for example, include pixel information for at least a first spiral scan line and the second field may include pixel information for at least a second spiral scan line. The scan lines of the second field may be interlaced with the scan lines of the first field. The first field may, for example, include pixel information for at least a first Lissajous scan line and the second field may include pixel information for at least a second Lissajous scan line. The scan lines of the second field may be interlaced with the scan lines of the first field. The interlacing of scan lines for both spiral and Lissajous scan patterns can be effectively implemented using phase shifting. The number of fields or scan lines may be greater than 2, such as 3, 4, 8, 16 or more.

[0205] At 1504, the augmented reality system begins reading out the frame buffer storing the first complete frame. The augmented reality system can drive the light source and yoke or other devices or structures to generate an image based on the pixel data specified in the frame from the image buffer.

[0206] The augmented reality system renders an updated first full frame to the frame buffer at 1506. The updated first full frame includes pixel information specifying a frame, a portion of which has changed from information specified by the initial (ie, non-updated) first full frame.

[0207] At 1508, the augmented reality system begins reading out the updated first full frame before the reading of the first full frame from the frame buffer is complete, thereby interrupting the presentation of the initial (i.e., non-updated) first full frame. Some implementations may utilize two or more frame buffers, allowing rendering to one frame buffer while reading out frames from other frame buffers. This should not be considered to limit different implementations of the augmented reality system to employ one, two, three, or even more frame buffers.

[0208] Fig.16 A method 1600 of operating in an augmented reality system is shown according to one illustrated embodiment. Fig.13 Method 1600 is used when method 1300 is used.

[0209] At 1602, an augmented reality system (eg, a controller subsystem and / or a processor thereof) generates pixel information for a first scan line (eg, spiral, Lissajous).

[0210] Optionally, the augmented reality system generates pixel information for a second scan line (e.g., spiral, Lissajous) phase shifted relative to the first scan line (e.g., spiral, Lissajous) at 1604. The phase shift advantageously uses the first scan line for spiral and Lissajous scan lines to interact or nest with the second scan line.

[0211] Optionally, the augmented reality system generates pixel information for a third scan line (e.g., spiral, Lissajous) phase shifted relative to the second scan line (e.g., spiral, Lissajous) at 1606. The phase shift advantageously uses the first and second scan lines for the spiral and Lissajous scan lines to interact or nest with the third scan line.

[0212] Optionally, the augmented reality system generates pixel information for a fourth scan line (e.g., spiral, Lissajous) phase shifted relative to the third scan line (e.g., spiral, Lissajous) at 1608. The phase shift advantageously uses the first, second, and third scan lines for the spiral and Lissajous scan lines to interact or nest with the fourth scan line.

[0213] Fig.17 A method 1700 of operating in an augmented reality system is shown according to one illustrated embodiment.

[0214] At 1702, for each of a plurality of frames, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) determines a corresponding resolution for each of at least two portions of the respective frame. The portions may be fields, lines, other subdivisions, or even individual pixels.

[0215] At 1704, the augmented reality system causes presentation of virtual images based on the plurality of frames, at least some of the images presented to the end user having variable resolutions, for example, the spacing between adjacent pixels may vary from one portion to another.

[0216] Fig.18 A method 1800 is shown operating in an augmented reality system according to one illustrated embodiment. Fig.17 Method 1800 is used when method 1700 is used.

[0217] At 1802, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) renders a frame into corresponding pixel data for a spiral scan pattern.

[0218] At 1804, the augmented reality system adjusts the amplitude of the drive signal between the time of presentation of the first part of the first frame in the frame and the time of presentation of the second part of the first frame in the frame. The amplitude change results in a variable resolution in the image corresponding to the first frame in the frame. The augmented reality system can, for example, change the slope or gradient of the drive signal. This is particularly useful when using a spiral scan pattern. For example, the first field of the frame may have a slope or gradient, and its second field may have a different slope or gradient, thereby changing the effective resolution with a single frame. A higher resolution or pixel density may be used at or near the end user's interest or attraction, while a lower resolution or pixel density may be used away from these locations. In the case where the center or image is moved toward the center of the end user's attraction or focus, a high resolution may appear near the center of the image, while the surrounding portions appear at a lower resolution. This essentially achieves what can be named a foviated display with controllable pixels.

[0219] Fig.19 A method 1900 for operating in an augmented reality system is shown according to one illustrated embodiment. Fig.17 Method 1800 is used when method 1700 is used.

[0220] At 1902, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) evaluates an attention point for at least a first image for an end user. The augmented reality system may use any of the aforementioned techniques for evaluating the attention point. For example, determining whether and where a new virtual object will appear, or where a virtual object will move in the end user's field of view. For another example, the augmented reality system may evaluate the relative attractiveness (e.g., speed, color, size, brightness, glint) of a virtual object. This may also employ eye tracking information, which indicates the location in the field of view where the end user's eyes are tracking or focusing.

[0221] Eye tracking information may be provided, for example, via one or more head-mounted sensors, such as head-mounted cameras. Such eye tracking information may be discerned, for example, by projecting a glint of light at the end user's eye and detecting the return or reflection of at least some of the projected light. For example, a projection subsystem that creates or projects an image may project pixels, dots, or other elements of light from at least one optical fiber to create light that strikes the end user's cornea. Eye tracking may employ one, two, three, or even more spots or points of light. The more spots or points of light there are, the more information can be discerned. The light source (e.g., a laser diode) may be pulsed or modulated, such as synchronized with the frame rate of a camera or image sensor. In this case, the spots or points may appear as lines that follow the eye movement. The direction of the line as a trace across the sensor indicates the direction of the eye movement. The orientation of the line (e.g., vertical, horizontal, diagonal) indicates the direction of the eye movement. The length of the line indicates the speed of the eye movement.

[0222] For eye tracking, the light can be modulated (e.g., in time, brightness) to increase the signal-to-noise ratio. Additionally or alternatively, the light can be of a specific wavelength (e.g., near-IR), allowing it to be distinguished from background light or even light forming an image that the end user is viewing. The light can be modulated to reduce the total amount of energy (e.g., heat) provided to the eye by the augmented reality system. The flickering light can be returned to the sensor via the same or another optical fiber. The sensor can, for example, take the form of a two-dimensional image sensor, such as a CCD sensor or a CMOS sensor.

[0223] In this way, the augmented reality system can detect and track corresponding eye movements, providing an indication of the point or location of the end user's attention or focus. The augmented reality system can logically associate a virtual object or virtual event (e.g., the appearance or movement of a virtual object) with an identified point or location of the end user's attention or focus. For example, the augmented reality system can specify that a virtual object appears at or near the point or location of the end user's attention or focus as a virtual object that is attractive to the end user.

[0224] At 1904, the augmented reality system adjusts (e.g., increases, decreases) the resolution in at least one portion of at least one subsequent image. The augmented reality system can use any of the different techniques described herein and other techniques to adjust the resolution of the subsequent page portion relative to other portions of the same subsequent page.

[0225] Fig. 20 A method 2000 is shown operating in an augmented reality system according to one illustrated embodiment. Fig.19 Action 1904 of method 1900 is performed using method 2000.

[0226] At 2002, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) increases resolution in a portion of at least one subsequent image that is at least proximal to the assessed attention point relative to other portions of at least one subsequent image. As previously explained, the resolution of the spiral scan pattern can be adjusted by controlling the magnitude or amplitude (e.g., current, voltage) of the drive signal. The resolution can be adjusted by adjusting the slope of the drive signal. Thus, the resolution can be increased by increasing the amplitude of the drive signal while the phase remains unchanged.

[0227] At 2004, the augmented reality system reduces the resolution in a portion of at least one subsequent image that is distal to the assessed attention point relative to other portions of the at least one subsequent image. The resolution may be reduced by reducing the amplitude of the drive signal while the phase remains constant.

[0228] In some implementations, the resolution is increased only, in some parts, while the resolution is neither increased nor decreased in other parts. In other implementations, the resolution is decreased only, in some parts, while the resolution is neither increased nor decreased in other parts. In still other implementations, the resolution is increased in some parts, while the resolution is decreased in other parts.

[0229] Fig.21 According to one illustrated embodiment, a method 2100 operating in an augmented reality system is shown. Fig.17 Method 1700 adopts method 2100.

[0230] At 2102, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) processes eye tracking data. The eye tracking data indicates at least one orientation of at least one eye of an end user. The eye tracking data is provided via at least one sensor. For example, the eye tracking data may be provided via a head-mounted sensor. In some implementations, the eye tracking data is collected via an optical fiber using a sensor located at or near a distal end of the optical fiber. For example, the optical fiber may collect light reflected from a portion of an end user's eye, which may be flickering light. The optical fiber may be the same optical fiber used to create an image for displaying or projecting an image to the end user.

[0231] At 2104, the augmented reality system processes the head tracking data. The head tracking data at least indicates the orientation of the end user's head. The head tracking data is provided via at least one sensor.

[0232] For example, one or more head-mounted or head-mounted sensors such as inertial sensors (e.g., gyroscope sensors, accelerometers). Head motion tracking can be implemented using one or more head-mounted or head-mounted light sources and at least one sensor. Head tracking can use one, two, three or even more light spots or points. The more light spots or points, the more information that can be distinguished. The light source (e.g., a laser diode) can be pulsed or modulated, for example synchronized with the frame rate of a camera or image sensor (e.g., a forward-facing camera). The laser light source may be modulated at a frequency lower than the frame rate of the camera or image sensor. In this case, the spots or points may appear as lines such as head movement. The direction of the line as a line trace across the sensor can indicate the direction of the head movement. The direction of the line (e.g., vertical, horizontal, diagonal) indicates the direction of the head movement. The length of the line indicates the speed of the head movement. The reflected light can also provide information about objects in the surrounding environment, such as distance and / or geometry (e.g., planar, curved) and / or orientation (e.g., angled or vertical). For example, one laser beam may produce information about direction and velocity (e.g., dash or line length). A second laser beam may add information about depth or distance (e.g., Z axis). A third laser beam may add information about the geometry and / or orientation of surfaces in the surrounding environment. The laser or other light source may be pulsed during head movement or during a portion of head movement.

[0233] Additionally or alternatively, head tracking data may be provided via non-head-mounted sensors. For example, a camera or imaging system images the end user including the end user's head, tracking its movement. This may be, for example, tracking movement relative to some external reference frame, such as a reference frame defined by the tracking system or the room in which the tracking system is located.

[0234] At 2106, the augmented reality system determines where the virtual object appears in the end user's field of view relative to the end user's reference frame. The appearance may be the appearance of a new virtual object when newly introduced into the end user's field of view. The appearance may be the appearance of a virtual object at a new position relative to the position of the virtual object in at least one previous image. The augmented reality system may use any of the various techniques described elsewhere herein to determine the position of the virtual object.

[0235] The system may also use blanking to improve the end-user perceived experience.

[0236] Fig. 22 A method 2200 operating in an augmented reality system is shown according to one illustrated embodiment. The method 2200 can effectively employ culling to improve the end-user perceived experience.

[0237] At 2202, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) displays at least one virtual object to an end user. The augmented reality system may render frames to a frame buffer, read the frames to drive one or more light sources and / or a yoke or other system to produce at least biaxial motion or trajectory of light.

[0238] At 2204, the augmented reality system detects and / or predicts the occurrence of head movement of the end user. The augmented reality system can use any of the multiple techniques described elsewhere herein to detect and / or predict the occurrence of head movement. Without limitation, these techniques include directly sensing head movement, such as via inertial sensors or inductors, or via head-mounted imagers or environmental imagers that image the area in which the end user is presented or visible. These techniques also include indirectly predicting head movement, such as by determining where new virtual objects will appear, where existing virtual objects will move, or the position at which particularly attractive virtual objects will be placed in an image.

[0239] At 2206, the augmented reality system evaluates whether the detected and / or predicted head movement exceeds or is predicted to exceed the nominal head movement value. The augmented reality system can use any of the multiple techniques described elsewhere in this document to evaluate whether the detected and / or predicted head movement exceeds or is predicted to exceed the nominal head movement value. Such an evaluation can include simply comparing the detected or predicted speed with the nominal speed. Such an evaluation can include simply comparing the detected or predicted acceleration with the nominal acceleration. Such an evaluation can include simply comparing the detected or predicted range with the nominal range. Such an evaluation can include more complex comparisons, including averaging or integrating multiple speeds, accelerations, or ranges during the movement. Such an evaluation can even use historical attributes or other information.

[0240] At 2208, the augmented reality system temporarily hides at least a portion of the display of at least one virtual object from the end user. For example, the augmented reality system may stop reading from a frame buffer. Additionally or alternatively, the augmented reality system may turn off illumination or light sources. This may include temporarily turning off a backlight of an LCD display.

[0241] Fig.23 A method 2300 of operating in an augmented reality system is shown according to one illustrated embodiment. Fig. 22 When method 2200 is used, method 2300 is used.

[0242] At 2302, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) processes head tracking data. The head tracking data indicates at least one orientation of an end user's head. The head tracking data may be provided via at least one sensor, which may or may not be worn by the end user. The augmented reality system may process the head tracking data using any of a variety of techniques described elsewhere herein.

[0243] At 2304, for each frame in at least some of the images presented to the end user, the augmented reality system (e.g., a controller subsystem and / or a processor thereof) determines a location at which a virtual object appears in the end user's field of view relative to the user's reference frame. The location at which the virtual object appears is determined when it is newly introduced into the end user's field of view. The location at which the virtual object appears in the new orientation in the image is determined relative to the orientation of the virtual object in at least one previous image. The augmented reality system may employ any of a variety of techniques described elsewhere herein to determine the location at which the virtual object appears.

[0244] At 2306, the augmented reality system evaluates whether the appearance of the determined virtual object is sufficiently attractive. For example, the augmented reality system can evaluate the relative visual attractiveness of the virtual object (e.g., speed, color, size, brightness, flickering light, transparency, special optical effects). For another example, the augmented reality system can evaluate the relative interest appeal (e.g., novelty, recency, previous attention, previous identification by the end user, previous interaction by the end user).

[0245] At 2308, the augmented reality system evaluates whether the determined position requires the end user to turn their head relative to the current position of the end user's head. The augmented reality system may employ the current position and / or orientation of the end user's head and the relative position and / or orientation of the virtual object. The augmented reality system may determine a distance, such as an angular distance between the end user's current focus and the position and / or orientation of the virtual object. The augmented reality system may determine whether the determined distance is within the range of eye movement or whether the end user must also turn their head. If the end user must turn their head, the system may evaluate how far the end user must turn their head. For example, the augmented reality system may employ information that specifies the relationship between the end user's eye movement and head movement. This information may indicate the extent to which the end user can shift their gaze via eye movement alone before turning their head. It is worth noting that the relationship between eye movements and head movements can be specified in a variety of different directions, such as a) from top to bottom, b) from bottom to top, c) from left to right, d) from right to left, e) diagonally from bottom left to top right, f) diagonally from bottom right to top left, g) diagonally from top left to bottom right, h) diagonally from top right to bottom left.

[0246] At 2310, the augmented reality system predicts the occurrence of head movement based on the evaluation. The augmented reality system can use one or more factors to form an evaluation to predict whether head movement will occur, the direction and / or orientation of the head movement, and / or the speed or acceleration of the head movement. The augmented reality system can use historical data that is either specific to the end user or more general to a group of end users. The augmented reality system can implement one or more machine learning algorithms to increase the accuracy of head movement predictions.

[0247] Fig.24 A method 2400 of operating in an augmented reality system is shown according to one illustrated embodiment. Fig. 22 Method 2400 is used during operation 2208 of method 2200.

[0248] At 2402, an augmented reality system (e.g., a controller subsystem and / or its processor) flashes or blinks a display or a backlight of the display. The flashing or blinking occurs on all or a portion of the detected head movement or predicted head movement. This can advantageously be effective in reducing the perception of inconsistencies in the presentation of frames or virtual objects. This can also be effective in increasing the perceived frame rate.

[0249] Fig.25 A method 2500 of operating in an augmented reality system is shown according to one illustrated embodiment.

[0250] At 2502, an augmented reality system (e.g., a controller subsystem and / or its processor) detects and / or predicts the occurrence of end-user head movement. For example, the augmented reality system may process head tracking data indicating at least one orientation of end-user head movement. Additionally or alternatively, the augmented reality system may determine a position where a virtual object appears in the end-user's field of view relative to the end-user reference frame, evaluate whether the determined position requires the end-user to turn the end-user's head, and predict the occurrence of head movement based on the evaluation. The augmented reality system may employ any of a variety of techniques described elsewhere herein to detect and / or predict the occurrence of head movement.

[0251] At 2504, the augmented reality system determines whether the detected and / or predicted head movement exceeds the nominal head movement value. The augmented reality system can employ any of a variety of techniques described elsewhere herein to determine whether the detected and / or predicted head movement exceeds the nominal head movement value.

[0252] At 2506, in response to determining that the detected and / or predicted head motion exceeds the nominal head motion value, the augmented reality system selectively activates an actuator to move the projector in at least one degree of freedom. Moving the projector may include translating the first optical fiber along at least one axis. Moving the projector may include rotating the first optical fiber about at least one axis.

[0253] Fig.26 A method 2600 of operating in an augmented reality system is shown according to one illustrated embodiment.

[0254] The augmented reality system can overrender frames to produce larger frames than required for the maximum area and maximum resolution of a given display technology. For example, in a head-mounted or head-mounted augmented reality system, the area available for display or projection can be set by various parameters of the device. Similarly, although the augmented reality system may be able to operate at multiple different resolutions, the device will set an upper limit or maximum resolution. The overrendered frame includes pixel information for a group of pixels that exceeds the maximum area displayed at the maximum resolution. This can advantageously allow the augmented reality system to read out only a portion of the frame (e.g., a portion of each field of the frame if it is not interrupted). This can allow the augmented reality system to move the image presented to the user.

[0255] At 2602, an augmented reality system (e.g., a controller subsystem and / or its processor) over-renders each of a plurality of frames for a defined field of view. This requires generating more pixel information required for a maximum area of ​​maximum resolution. For example, the area of ​​a frame may be increased by a percentage of the maximum area, such as by increasing pixel information in a horizontal, vertical, or diagonal direction defined by the frame. The larger the size of the frame, the more freedom the augmented reality system has to move the boundaries of an image presented to the user.

[0256] At 2604, the augmented reality system sequentially buffers the rendered frames in at least one frame buffer. The augmented reality system may use a frame buffer that is larger than the size of the frame required for the maximum resolution maximum display size. Some implementations use multiple frame buffers. This may facilitate interruption of the presentation of frames as described elsewhere herein.

[0257] At 2606, the augmented reality system determines a portion of each image to present. The augmented reality system may determine the portion based on a number of different factors. For example, the factor may indicate a position in an image or scene where the end user is paying attention, focusing, or has otherwise attracted the end user's attention. Once again, a variety of techniques may be employed, including but not limited to eye tracking. For another example, the factor may indicate a position in an image or scene where the end user is predicted to be paying attention, focusing, or will otherwise attract the end user's attention. Once again, a variety of techniques may be employed, including but not limited to: identifying newly appearing virtual objects, fast or rapidly moving virtual objects, visually attractive virtual objects, previously designated virtual objects (e.g., designated by the end user or by the interaction of a previously tracked end user) and / or virtual objects that attract attention based on the inherent properties of vertical objects. Virtual objects that attract attention based on the inherent properties of vertical objects may, for example, include virtual objects that visually represent objects or items of concern or worry (e.g., an impending threat) for a broad end user or a specific end user.

[0258] At 2608, the augmented reality system selectively reads out a portion of the rendered frame from the frame buffer. The portion is presented based at least in part on the determined portion of the respective image. For example, the portion read out can have a center moved to be close to, or even matched or co-aligned with, the identified location. The identified location can, for example, be a location in a previous image or frame that has attracted the end user's attention. The identified location can, for example, be a location in a subsequent frame that the augmented reality system has predicted will attract the end user's attention.

[0259] Fig. 27 A method 2700 of operating in an augmented reality system is shown according to one illustrated embodiment. Fig.26 Method 2700 may be used when performing method 2600. For example, method 2700 may be used to predict locations in subsequent frames or images that will attract the end user's attention.

[0260] At 2702, for each of at least some of the frames, the augmented reality system (e.g., a controller subsystem and / or a processor thereof) determines a location at which a virtual object appears within an end user's field of view relative to an end user's reference frame.

[0261] At 2704, the augmented reality system selectively reads out the frame buffer based at least in part on determining the location where the virtual object in the field of view appears. For example, the portion that is read out can have a center that is moved to be close to, or even match or co-aligned with, the identified location. Alternatively, the boundaries of the portion that is read out can be moved to cover the surrounding area immediately adjacent to the determined location in two or even three dimensions. For example, the augmented reality system can select a portion (e.g., 80%) of the entire over-rendered frame that will be read out of the frame buffer for presentation to the end user. The augmented reality system can select the portion so that the boundary is moved relative to the current location of the end user's attention, such as in an image currently being presented to the end user. The augmented reality system can select the boundary based on a combination of the current location and the predicted location, while setting the boundary so that these two locations will be presented to the end user in a subsequently presented image.

[0262] Fig.28 A method 2800 of operating in an augmented reality system is shown according to one illustrated embodiment. Fig.26 Method 2800 may be used when performing method 2600. For example, method 2800 may be used to determine locations in an image that have attracted or are predicted to attract the end user's attention.

[0263] At 2802, the augmented reality system (e.g., the controller subsystem and / or its processor) determines the location where the new virtual object appears when the new virtual object is newly introduced into the field of view of the end user. The augmented reality system can use any of the multiple technologies described herein to identify the introduction of the virtual object, which is new relative to the immediate previous frame or image presented to the end user. In this way, even if the virtual object has been previously presented to the end user in some other part of the presentation, if a sufficient number of intermediate images have been presented, the virtual object can be identified as newly introduced to make the reintroduction of the virtual object attract the attention of the end user.

[0264] At 2804, the augmented reality system determines the location of the appearance of the virtual object in the new orientation of the frame relative to the orientation in at least one previous frame. The augmented reality system can use any of the multiple techniques described herein to identify the movement of the virtual object to a new or different orientation in multiple images, wherein the movement is relative to the movement of the immediately previous frame or image presented to the end user. In this way, even if the virtual object has been previously presented to the end user at some location in some other part of the presentation, if a sufficient number of intermediary images have been presented, the virtual object can be identified as being moved or moving so that the reappearance of the virtual object at the previous location attracts the attention of the end user.

[0265] At 2806, the augmented reality system determines the location of virtual objects that have at least a defined minimum velocity in the end user's field of view. The augmented reality system can employ any of the various techniques described herein to determine the velocity of movement of the virtual object from image to image and compare the velocity to a defined or nominal velocity. The determined velocity can be relative to a fixed reference frame in the image or relative to other virtual objects and / or physical objects that appear in the image.

[0266] At 2808, the augmented reality system determines a portion of each image to present based at least in part on the position of the virtual object in the image. The augmented reality system may employ any of the various techniques described herein to determine the portion of each image to present. The determination may be based on any different factors. The factors may, for example, include factors or data indicating a position in an image or scene that the end user is paying attention to, focusing on, or has otherwise attracted the end user's attention. Factors may, for example, include factors or data indicating a position in an image or scene that the end user is predicted to pay attention to, focus on, or will otherwise attract the end user's attention. The augmented reality system may employ any of the various techniques described herein to identify a position that has attracted the end user's attention via actual detection or via prediction.

[0267] At 2810, the augmented reality system reads out a portion of the frame buffer for at least one subsequent frame. For example, the portion read out shifts the center of the image at least toward the portion of the respective image that has been determined to be presented. The augmented reality system can employ any of the various techniques described herein to read out a portion of the frame from the frame buffer that moves the center or border of the image based on the location of the actual or predicted center of attention of the end user.

[0268] Fig.29 A method 2900 of operating in an augmented reality system is shown according to one illustrated embodiment. Fig.26 Method 2800 may be employed when performing method 2600. In particular, method 2900 may be used to determine which portion of a frame is to be read out based on a predicted end-user head movement.

[0269] At 2902, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) predicts the occurrence of head motion of an end user. The augmented reality system can employ any of the various techniques described herein to predict head motion. Such techniques include, but are not limited to, detecting the appearance of new virtual objects, moving virtual objects, fast-moving virtual objects, previously selected virtual objects, and / or visually attractive virtual objects in an image.

[0270] At 2904, the augmented reality system determines a portion of each frame or image to present based at least in part on the predicted head movement. The augmented reality system can employ any of the multiple techniques described herein to determine the portion of the frame to be used. For example, the augmented reality system can select the portion so that the boundary encompasses the location of the predicted end point of the predicted head movement. When the head movement prediction is based on the appearance of a virtual object (e.g., newly introduced, moving, attractive appearance, previously selected by the end user), the end point can coincide with the location of the virtual object in a subsequent frame or image.

[0271] Fig.30 A method 3000 of operating in an augmented reality system is shown according to one illustrated embodiment.

[0272] The augmented reality system may overrender the frame, producing a larger frame than required for the maximum area and maximum resolution for a given display technology. For example, in a head-mounted or head-mounted augmented reality system, the area available for display or projection may be set by various parameters of the device. Likewise, although the augmented reality system may be able to operate at a number of different resolutions, the device will set an upper limit or maximum resolution. The overrendered frame includes pixel information for a group of pixels that exceeds the maximum area displayed at the maximum resolution. This can advantageously allow the augmented reality system to read out only a portion of a frame (e.g., a portion of each field of the frame if it is not interrupted). This can allow the augmented reality system to move the image presented to the user.

[0273] At 3002, an augmented reality system (e.g., a controller subsystem and / or a processor thereof) over-renders each of a plurality of frames for a defined field of view. This requires more pixel information required to generate a maximum area of ​​maximum resolution. For example, the area of ​​a frame can be increased by a percentage of the maximum area, such as increasing the pixel information in a horizontal, vertical, or diagonal direction defined by the frame. The larger the size of the frame, the more freedom the augmented reality system has to move the boundaries of the image presented to the user.

[0274] At 3004, the augmented reality system determines a portion of each image to present. The augmented reality system may determine the portion based on a variety of different factors. For example, the factor may indicate a position in an image or scene where the end user is paying attention, focusing, or has otherwise attracted the end user's attention. Once again, a variety of techniques may be employed, including but not limited to eye tracking. For another example, the factor may indicate a position in an image or scene where the end user is predicted to be paying attention, focusing, or will otherwise attract the end user's attention. Once again, a variety of techniques may be employed, including but not limited to: identifying newly appearing virtual objects, fast or rapidly moving virtual objects, visually attractive virtual objects, previously designated virtual objects (e.g., designated by the end user or by the interaction of a previously tracked end user) and / or virtual objects that attract attention based on the inherent properties of vertical objects. Virtual objects that attract attention based on the inherent properties of vertical objects may, for example, include virtual objects that visually represent objects or items of concern or worry (e.g., an impending threat) for a broad end user or a specific end user.

[0275] At 3006, the augmented reality system dynamically addresses one or more determined portions of the rendered frame into a buffer. The determined portion may, for example, have a center that is moved to be close to or even match or co-align with an identified location that is of interest, interest, or focus to the end user. The identified location may, for example, be a location in a previous image or frame that has attracted the end user's attention. The identified location may, for example, be a location in a subsequent frame that the augmented reality system has predicted will attract the end user's attention. Some implementations employ multiple frame buffers. As described elsewhere herein, this may facilitate interruption of the presentation of frames.

[0276] At 3008, the augmented reality system reads the determined portion of the rendered frame from the frame buffer.

[0277] Various exemplary embodiments of the present invention are described herein. These embodiments are cited in a non-restrictive sense. They are provided to more broadly illustrate the applicable aspects of the present invention. Various changes may be made to replace the description and equivalents of the present invention without departing from the true spirit and scope of the present invention. In addition, many modifications may be made to adapt to specific situations, materials, compositions, processes, treatments or steps to achieve the goals, spirit or scope of the present invention. Further, as will be understood by those skilled in the art, each individual difference described and illustrated herein has discrete components and features, which may be easily separated from or combined with any other several embodiments without departing from the scope or spirit of the present invention from any other several embodiments. All such modifications are intended to fall within the scope of the claims associated with the present disclosure.

[0278] The present invention includes methods that can be performed using subject devices. The method may include the act of providing such appropriate equipment. Such provision may be performed by the end user. In other words, the "providing" behavior only requires the end user to obtain, contact, process, place, set, activate, power on or other actions to provide the necessary equipment in the subject method. The methods listed herein can be implemented in any logically possible order of the listed events and in the order of the listing of events.

[0279] The exemplary aspects of the present invention and the details of material selection and manufacturing are described above. As for other details of the present invention, they can be recognized in combination with the patents and publications cited above and those skilled in the art who have been widely understood or recognized. The same principle also applies to the method-based aspects of the present invention according to the additional actions that are usually or logically adopted.

[0280] In addition, although the present invention has been described with reference to several examples that optionally combine multiple features, the present invention is not limited to these described or indicated situations with respect to each variant of the present invention. Various changes can be made to replace the description of the present invention and equivalents (whether they are listed herein or not included herein for the sake of brevity) without departing from the true spirit and scope of the present invention. In addition, when a numerical range is provided, it should be understood that every intermediate between the upper and lower limits of the range and any other claimed or intermediate values ​​in the declared range are all included in the present invention.

[0281] In addition, it is contemplated that any optional features of the variants of the present invention may be set forth and claimed independently, or in combination with any one or more features described herein. References to singular items include the possibility of the presence of multiple identical items. More specifically, as used herein and in the accompanying related claims, the singular forms "one", "an", "said" and "the" include multiple referents unless specifically stated to the contrary. In other words, the use of this article in the above specification and in the claims related to the present disclosure takes into account "at least one" of the subject items. It is further noted that such claims may be written to exclude any optional components. In this regard, this statement is intended to serve as a pre-conditional basis for the use of such exclusive terms, such as "only", "only", etc., in conjunction with the narration of the claim elements, or the use of "negative" limitations.

[0282] If such exclusive terms are not used, the term "comprising" in the claims related to the present disclosure shall allow the inclusion of any additional components - regardless of whether a given number of components are listed in such claims, or the added features may be considered to transform the nature of the components set forth in such claims. Unless specifically defined herein, all technical and scientific terms used herein shall be given the broadest commonly understood meaning possible while maintaining the validity of the claims.

[0283] The breadth of the present invention is not limited by the examples and / or subject description provided, but is limited only by the scope of the claim language associated with this disclosure.

Claims

1. An operation method in a virtual image presentation system, the method comprising: rendering a first complete frame having a first field and a second field to an image buffer by at least one or more optical fibers, wherein the first field includes at least a first Lissajous scan line and the second field includes at least a second Lissajous scan line interleaved with the at least first Lissajous scan line; accessing a frame buffer storing at least a first complete frame; predicting at least one predicted head movement of a user by using historical data or attributes of the user using at least one control subsystem including one or more sensors and at least one microprocessor in a virtual image presentation system; as well as Based at least in part or in whole on predicted head movement of the user predicted to exceed a nominal head movement value, before the full retrieval of the first full frame is completed, at least by dynamically interrupting the full retrieval of the first full frame, and at least by presenting an update of a smaller portion of the first full frame corresponding to the updated full frame in the display device of the virtual image presentation system to the user, wherein the size of the smaller portion is smaller than the overall size of the first full frame, and the update includes pixel information that has changed from the first full frame.

2. The method of claim 1, further comprising replacing the updated second Lissajous scan line with the second Lissajous scan line of the first complete frame.

3. The method of claim 1, further comprising phase shifting the second Lissajous scan line relative to the first Lissajous scan line to interleave the first Lissajous scan line and the second Lissajous scan line.

4. The method of claim 1, further comprising phase shifting a third Lissajous scan line relative to the second Lissajous scan line to interleave the first Lissajous scan line, the second Lissajous scan line, and the third Lissajous scan line.

5. The method of claim 4, further comprising phase shifting a fourth Lissajous scan line relative to the third Lissajous scan line to interleave the first, second, third, and fourth Lissajous scan lines.

6. The method of claim 1, wherein dynamically interrupting retrieval of the first complete frame is based at least in part on a relationship between head movement and eye movement of the user.

7. The method according to claim 1, further comprising: When the detected head movement exceeds a first head movement value, and when the predicted head movement is determined to exceed a nominal head movement value, the display of the virtual content is adjusted by blanking at least a portion of the display of the virtual content.

8. The method of claim 1 , wherein predicting the at least one predicted head movement further comprises: detecting movement of at least one eye of the user; determining an indication of attention or concentration of the user based in part or in whole on the detected motion; associating a virtual object or virtual event with an indication of the user's attention or gathering; predicting a transition of the user aggregation; as well as The predicted head movement is determined in part or in whole based on the transition of the user's focus.

9. The method of claim 1 , wherein predicting the at least one predicted head movement further comprises: identifying one or more virtual objects in the virtual content; identifying, from the one or more virtual objects, a virtual object predicted to attract the user's attention; identifying a center of one or more subsequent frames of the first complete frame; shifting the center to a position proximate to the virtual object; as well as The field of view of one or more subsequent frames presented to the user is adjusted in part or in whole based on the position of the virtual object.

10. The method of claim 1, wherein predicting the at least one predicted head movement further comprises: Identify the upper resolution limit of the virtual image rendering system; Identifying an upper display area or field of view of a virtual image presentation system; Rendering a first complete frame at an upper resolution limit of the virtual image presentation system with a set of pixels exceeding an upper display area or field of view; as well as At least one subsequent frame to the first complete frame is retrieved from the image buffer.

11. The method according to claim 1, further comprising: projecting light of a first geometric shape into at least one eye of the user using a light source in a virtual image rendering system; receiving, at a reflected light receiver, reflected light of a second geometry from the at least one eye of the user; as well as One or more movement characteristics of at least one eye of the user are determined based in part or in whole on the first geometry and the second geometry.

12. A virtual image presentation device, comprising: a microprocessor coupled to at least one or more optical fibers and configured to render a first complete frame having a first field and a second field to an image buffer, wherein the first field includes at least a first Lissajous scan line and the second field includes at least a second Lissajous scan line interleaved with the at least first Lissajous scan line; The microprocessor is further configured to access a frame buffer storing at least the first complete frame; a control subsystem comprising a microprocessor coupled to one or more sensors in the virtual image presentation device and configured to predict at least one predicted head motion of the user by using historical data or attributes of the user; and The virtual image presentation device is further configured to: Dynamically interrupting the entire retrieval of the first complete frame before the retrieval of the first complete frame is fully completed, at least in part or in whole based on predicted head movement of the user predicted to exceed a nominal head movement value, and displaying virtual content in the updated complete frame to the user in the display device of the virtual image presentation device by at least presenting an updated presentation of a smaller portion of the first complete frame corresponding to the updated complete frame in the display device, wherein the size of the smaller portion is smaller than the full size of the first complete frame and the update includes pixel information that has been changed from the first complete frame.

13. The virtual image presentation device according to claim 12, wherein: The microprocessor is also configured to phase shift the second Lissajous scan line relative to the first Lissajous scan line to interleave the first Lissajous scan line and the second Lissajous scan line.

14. The virtual image presentation device according to claim 13, wherein: The microprocessor is also configured to phase shift the third Lissajous scan line relative to the second Lissajous scan line to interleave the first Lissajous scan line, the second Lissajous scan line, and the third Lissajous scan line.

15. The virtual image presentation device according to claim 14, wherein: The microprocessor is further configured to phase shift the fourth Lissajous scan line relative to the third Lissajous scan line to interleave the first Lissajous scan line, the second Lissajous scan line, the third Lissajous scan line, and the fourth Lissajous scan line.

Images