Virtual and augmented reality systems and methods

The virtual or augmented reality display system addresses the challenge of providing comfortable and natural depth perception by using multiple depth planes and embedded control information to adjust image positions and depths, resulting in a more realistic and efficient 3D simulation.

JP7595201B2Active Publication Date: 2024-12-05MAGIC LEAP INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024014037
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2015-08-18
Filing Date
2024-02-01
Publication Date
2024-12-05
Estimated Expiration
2036-08-17

AI Technical Summary

Technical Problem

Existing virtual and augmented reality systems struggle to provide a comfortable and natural-feeling presentation of virtual image elements due to the complexity of the human visual perceptual system, often leading to discomfort and ineffective depth perception.

Method used

A virtual or augmented reality display system that includes a display configured to display images of multiple depth planes and a display controller to control the display based on embedded control information, allowing shifts and adjustments in image position, scale, and depth without re-rendering, using depth plane indicator data and head pose information to enhance depth perception.

Benefits of technology

The system provides a more realistic and comfortable simulation of three-dimensional environments by accurately adjusting image positions and depths based on user head pose, reducing computational load and power consumption while enhancing depth perception.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595201000001
    Figure 0007595201000001
  • Figure 0007595201000002
    Figure 0007595201000002
  • Figure 0007595201000003
    Figure 0007595201000003
Patent Text Reader

Abstract

To provide virtual and augmented reality imaging and visualization systems.SOLUTION: A virtual or augmented reality display system controls a display using control information included with virtual or augmented reality imagery that is intended to be shown on the display. The control information can be used to specify one of multiple possible display depth planes. The control information can also specify pixel shifts within a given depth plane or between depth planes. The system can also enhance head pose measurement from a sensor by using gain factors which vary based upon the user's head pose position within a physiological range of movement.SELECTED DRAWING: None
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 62 / 206,765, filed August 18, 2015, and entitled “VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS,” which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates to virtual and augmented reality imaging and visualization systems. [Background technology]

[0003] Modern computing and display technology has facilitated the development of virtual reality and augmented reality systems. Virtual reality or "VR" systems create a simulated environment for a user to experience. This can be done by presenting computer-generated images to the user through a display. The images generate a sensory experience that immerses the user in the simulated environment. Virtual reality scenarios typically involve not only the presentation of computer-generated images, but also the inclusion of real-world images.

[0004] Augmented reality systems generally complement real-world environments with simulated elements. For example, an augmented reality or "AR" system may provide a user with a view of the surrounding real-world environment via a display. However, computer-generated images may also be presented on the display to augment the real-world environment. The computer-generated images may include elements that are contextually relevant to the real-world environment. Such elements may include simulated text, images, objects, etc. The simulated elements may often be interactive in real time. FIG. 1 depicts an exemplary augmented reality scene 1 in which a user of AR technology views a real-world park-like setting 6 featuring people, trees, buildings in the background, and a concrete platform 1120. In addition to these items, computer-generated images are also presented to the user. The computer-generated images may include, for example, a robotic figure 1110 standing on the real-world platform 1120, and a flying cartoon-like avatar character 2 that appears to be an anthropomorphic bumblebee, even though these elements 2, 1110, do not actually exist in the real-world environment.

[0005] The human visual perception system is complex, making it difficult to create VR or AR techniques that promote comfortable, natural-feeling, and rich presentation of virtual, among other real-world, image elements. Summary of the Invention [Means for solving the problem]

[0006] In some embodiments, a virtual or augmented reality display system comprises a display configured to display images of multiple depth planes; and a display controller configured to receive rendered virtual or augmented reality image data from a graphics processor and to control the display based at least in part on control information embedded in the rendered image, the embedded control information indicating a shift to apply to at least a portion of the rendered image when displaying the image.

[0007] In some embodiments, the shift changes the displayed position of one or more virtual or augmented reality objects compared to the position of the one or more objects in a rendered image.

[0008] In some embodiments, the shift comprises a lateral shift of at least a portion of the image by a defined number of pixels in the same depth plane.

[0009] In some embodiments, the shift comprises a longitudinal shift of at least a portion of the image from one depth plane to another.

[0010] In some embodiments, the display controller is further configured to scale at least a portion of the image in conjunction with the vertical shift from one depth plane to another.

[0011] In some embodiments, the shift comprises a longitudinal shift of at least a portion of the image from one depth plane to a virtual depth plane, the virtual depth plane comprising a weighted combination of at least two depth planes.

[0012] In some embodiments, the shift is based on information about the user's head pose.

[0013] In some embodiments, the shifting is performed by the display controller without re-rendering the image.

[0014] In some embodiments, a method in a virtual or augmented reality display system includes receiving rendered virtual or augmented reality image data from a graphics processor and displaying images of multiple depth planes based at least in part on control information embedded in the rendered images, the embedded control information indicating a shift to apply to at least a portion of the rendered images when displaying the images.

[0015] In some embodiments, the method further includes shifting a displayed position of the one or more virtual or augmented reality objects compared to a position of the one or more objects in the rendered image.

[0016] In some embodiments, the method further comprises laterally shifting at least a portion of the image by a prescribed number of pixels in the same depth plane based on the control information.

[0017] In some embodiments, the method further includes vertically shifting at least a portion of the image from one depth plane to another depth plane based on the control information.

[0018] In some embodiments, the method further includes scaling at least a portion of the image in conjunction with vertically shifting the image from one depth plane to another.

[0019] In some embodiments, the method further includes vertically shifting at least a portion of the image from one depth plane to a virtual depth plane, the virtual depth plane comprising a weighted combination of at least two depth planes.

[0020] In some embodiments, the shift is based on information about the user's head pose.

[0021] In some embodiments, the method further comprises shifting the image without re-rendering the image.

[0022] In some embodiments, a virtual or augmented reality display system comprises a display configured to display virtual or augmented reality images for a plurality of depth planes, the images comprising a series of images composed of rows and columns of pixel data, and a display controller configured to receive the images from a graphics processor and to control the display based at least in part on control information embedded in the images, the embedded control information comprising depth plane indicator data indicating in which of the plurality of depth planes to display at least a portion of the image.

[0023] In some embodiments, the control information does not change the number of rows and columns of pixel data in the sequence of images.

[0024] In some embodiments, the control information comprises rows or columns of information that are substituted for rows or columns of pixel data in one or more of the series of images.

[0025] In some embodiments, the control information comprises rows or columns of information that are appended to pixel data of one or more of the series of images.

[0026] In some embodiments, the pixel data comprises a plurality of color values, and the depth plane indicator data is substituted for one or more bits of at least one of the color values.

[0027] In some embodiments, the depth plane indicator data is substituted for one or more least significant bits of at least one of the color values.

[0028] In some embodiments, the depth plane indicator data is substituted for one or more bits of the blue color value.

[0029] In some embodiments, each pixel is provided with depth plane indicator data.

[0030] In some embodiments, the display controller is configured to sequence the sequence of images based at least in part on the depth plane indicator data.

[0031] In some embodiments, a method in a virtual or augmented reality display system includes receiving a virtual or augmented reality image from a graphics processor, the image comprising a series of images composed of rows and columns of pixel data in a plurality of depth planes, and displaying the image based at least in part on control information embedded in the image, the embedded control information comprising depth plane indicator data indicating in which of the plurality of depth planes at least a portion of the image is to be displayed.

[0032] In some embodiments, the control information does not change the number of rows and columns of pixel data in the sequence of images.

[0033] In some embodiments, the control information comprises rows or columns of information that are substituted for rows or columns of pixel data in one or more of the series of images.

[0034] In some embodiments, the control information comprises rows or columns of information that are appended to pixel data of one or more of the series of images.

[0035] In some embodiments, the pixel data comprises a plurality of color values, and the depth plane indicator data is substituted for one or more bits of at least one of the color values.

[0036] In some embodiments, the depth plane indicator data is substituted for one or more least significant bits of at least one of the color values.

[0037] In some embodiments, the depth plane indicator data is substituted for one or more bits of the blue color value.

[0038] In some embodiments, each pixel is provided with depth plane indicator data.

[0039] In some embodiments, the method further comprises ordering the sequence of images based at least in part on the depth plane indicator data.

[0040] In some embodiments, a virtual or augmented reality display system comprises a first sensor configured to provide measurements of a user's head pose over time; and a processor configured to estimate a user's head pose based on the at least one head pose measurement and based on the at least one calculated predicted head pose, the processor configured to combine the head pose measurements and the predicted head pose using one or more gain factors, the one or more gain factors varying based on the user's head pose position within a physiological range of motion.

[0041] In some embodiments, the first sensor is configured to be head-mounted.

[0042] In some embodiments, the first sensor comprises an inertial measurement unit.

[0043] In some embodiments, one or more gain factors emphasize predicted head pose over head pose measurements when the user's head pose is in the central portion of the physiological range of motion.

[0044] In some embodiments, one or more gain factors emphasize predicted head pose over head pose measurements when the user's head pose is closer to the middle of the user's physiological range of motion than to the limits of the user's physiological range of motion.

[0045] In some embodiments, one or more gain factors emphasize head pose measurements over predicted head poses when the user's head pose approaches the limits of the physiological range of motion.

[0046] In some embodiments, one or more gain factors emphasize head pose measurements over predicted head poses when the user's head pose is closer to the limits of the physiological range of motion than to the middle of the physiological range of motion.

[0047] In some embodiments, the first sensor is configured to be head-mounted and further comprises a second sensor configured to be body-mounted, and the at least one head pose measurement is determined based on measurements from both the first sensor and the second sensor.

[0048] In some embodiments, the head pose measurement is determined based on a difference between measurements from the first sensor and the second sensor.

[0049] In some embodiments, a method of estimating head pose in a virtual or augmented reality display system includes receiving measurements of a user's head pose over time from a first sensor, and estimating, using a processor, a head pose of the user based on the at least one head pose measurement and based on the at least one calculated predicted head pose, where estimating the user's head pose includes combining the head pose measurements and the predicted head pose using one or more gain factors, where the one or more gain factors vary based on the user's head pose position within a physiological range of motion.

[0050] In some embodiments, the first sensor is configured to be head-mounted, and the method further includes receiving a body orientation measurement from a second sensor configured to be body-mounted, and estimating a head pose of the user based on the at least one head pose measurement and based on the at least one calculated predicted head pose, where the at least one head pose measurement is determined based on measurements from both the first sensor and the second sensor.

[0051] In some embodiments, a virtual or augmented reality display system comprises a sensor configured to determine one or more characteristics of ambient lighting, a processor configured to adjust one or more characteristics of a virtual object based on the one or more characteristics of the ambient lighting, and a display configured to display the virtual object to a user.

[0052] In some embodiments, the one or more characteristics of the ambient lighting comprise the brightness of the ambient lighting.

[0053] In some embodiments, the one or more characteristics of the ambient lighting comprise the hue of the ambient lighting.

[0054] In some embodiments, the one or more characteristics of the virtual object comprise the brightness of the virtual object.

[0055] In some embodiments, the one or more characteristics of the virtual object comprise the hue of the virtual object.

[0056] In some embodiments, a method in a virtual or augmented reality display system includes receiving one or more characteristics of ambient lighting from a sensor, adjusting, using a processor, one or more characteristics of a virtual object based on the one or more characteristics of the ambient lighting, and displaying the virtual object to a user.

[0057] In some embodiments, a virtual or augmented reality display system comprises a processor configured to compress virtual or augmented reality image data, where the images comprise images of a plurality of depth planes, the processor configured to compress the image data by reducing redundant information between the depth planes of the images, and a display configured to display the images of the plurality of depth planes.

[0058] In some embodiments, the image of a depth plane is represented differentially relative to adjacent depth planes.

[0059] In some embodiments, the processor encodes the movement of the object between the depth planes.

[0060] In some embodiments, a method in a virtual or augmented reality display system includes compressing virtual or augmented reality image data using a processor, where the images comprise images of a plurality of depth planes, the processor being configured to compress the image data by reducing redundant information between the depth planes of the images, and displaying the images of the plurality of depth planes.

[0061] In some embodiments, the image of a depth plane is represented differentially relative to adjacent depth planes.

[0062] In some embodiments, the method further comprises encoding the movement of the object between the depth planes.

[0063] In some embodiments, a virtual or augmented reality display system comprises a display configured to display virtual or augmented reality images of multiple depth planes, and a display controller configured to control the display, where the display controller dynamically configures subportions of the display to refresh every display cycle.

[0064] In some embodiments, the display comprises a scanning display and the display controller dynamically configures the scanning pattern to omit areas of the display where the image does not need to be refreshed.

[0065] In some embodiments, the display cycle comprises frames of a video image.

[0066] In some embodiments, the display controller increases the video frame rate when a subportion of the display being refreshed decreases in size.

[0067] In some embodiments, the display controller reduces the video frame rate when a subportion of the display being refreshed increases in size.

[0068] In some embodiments, a method in a virtual or augmented reality display system includes using a display to display virtual or augmented reality images of multiple depth planes and dynamically configuring a subportion of the display to refresh every display cycle.

[0069] In some embodiments, the display comprises a scanning display and the method further includes dynamically configuring the scanning pattern to omit areas of the display where the image does not need to be refreshed.

[0070] In some embodiments, the display cycle comprises frames of a video image.

[0071] In some embodiments, the method further includes increasing the video frame rate if a subportion of the display that is refreshed reduces in size.

[0072] In some embodiments, the method further includes decreasing the video frame rate if the subportion of the display being refreshed increases in size.

[0073] In some embodiments, a virtual or augmented reality display system comprises a transmitter that transmits an electric or magnetic field that varies in space, and a tangible object that allows a user to interact with a virtual object or scene, the tangible object comprising a sensor that detects the electric or magnetic field from the transmitter, and measurements from the sensor are used to determine the position or orientation of the tangible object relative to the transmitter.

[0074] In some embodiments, the transmitter is integrated with a head-mounted portion of a virtual or augmented reality display system.

[0075] In some embodiments, a method in a virtual or augmented reality display system includes transmitting an electric or magnetic field that varies in space using a transmitter, detecting the electric or magnetic field using a sensor, and using measurements from the sensor to determine a position or orientation of the sensor relative to the transmitter.

[0076] In some embodiments, the transmitter is integrated with a head-mounted portion of a virtual or augmented reality display system.

[0077] In some embodiments, a virtual or augmented reality display system comprises a display configured to display images of multiple depth planes, and a display controller configured to receive rendered virtual or augmented reality image data and to control the display based at least in part on control information embedded in the rendered image, the embedded control information indicating a desired brightness or color to apply to at least a portion of the rendered image when displaying the image. The desired brightness or color can change a displayed position of one or more virtual or augmented reality objects compared to a position of the one or more objects in the rendered image. The desired brightness or color can comprise a vertical shift of at least a portion of the image from one depth plane to a virtual depth plane, the virtual depth plane comprising a weighted combination of at least two depth planes.

[0078] In some embodiments, a virtual or augmented reality display system comprises a display configured to display images of multiple depth planes; and a display controller configured to receive rendered virtual or augmented reality image data and to control the display based at least in part on control information, the control information indicating that at least one depth plane is inactive, and the display controller configured to control the display based on an indication that the at least one depth plane is inactive, thereby reducing power consumption.

[0079] In some embodiments, the indication that at least one depth plane is inactive comprises control information comprising depth plane indicator data defining a plurality of active depth planes in which to display the image.

[0080] In some embodiments, the indication that at least one depth plane is inactive comprises control information comprising depth plane indicator data that specifies that the at least one depth plane is inactive.

[0081] In some embodiments, the control information is embedded in the rendered image.

[0082] In some embodiments, the display controller causes the power of one or more light sources to be reduced as a result of the control information indicating that at least one depth plane is inactive, thereby reducing power consumption.

[0083] In some embodiments, a method in a virtual or augmented reality display system includes receiving rendered virtual or augmented reality image data for displaying images on a plurality of depth planes, receiving control information indicating that at least one depth plane is inactive, and displaying images in the plurality of depth planes based at least in part on the control information indicating that at least one depth plane is inactive, thereby reducing power consumption.

[0084] In some embodiments, the control information comprises depth plane indicator data that defines a number of active depth planes in which to display the image.

[0085] In some embodiments, the control information comprises depth plane indicator data that defines at least one depth plane that is inactive.

[0086] In some embodiments, the control information is embedded in the rendered image.

[0087] In some embodiments, as a result of the control information indicating that at least one depth plane is inactive, one or more light sources are powered down, thereby reducing power consumption.

[0088] In some embodiments, a virtual or augmented reality display system comprises a display configured to display images of multiple depth planes having multiple color fields; and a display controller configured to receive rendered virtual or augmented reality image data and to control the display based at least in part on control information, the control information indicating that at least one color field is inactive, and the display controller configured to control the display based on an indication that the at least one color field is inactive, thereby reducing power consumption.

[0089] In some embodiments, the indication that at least one color field is inactive comprises control information comprising color field indicator data defining a plurality of active color fields for displaying the image.

[0090] In some embodiments, the indication that the at least one color field is inactive comprises control information comprising color field indicator data that defines the at least one color field to be inactive.

[0091] In some embodiments, the control information is embedded in the rendered image.

[0092] In some embodiments, the display controller causes the power of one or more light sources to be reduced as a result of the control information indicating that at least one color field is inactive, thereby reducing power consumption.

[0093] In some embodiments, a method in a virtual or augmented reality display system includes receiving rendered virtual or augmented reality image data for displaying images on multiple depth planes having multiple color fields; receiving control information indicating that at least one color field is inactive; and displaying images of the multiple color fields in the multiple depth planes based at least in part on the control information indicating that at least one color field is inactive, thereby reducing power consumption.

[0094] In some embodiments, the control information comprises color field indicator data defining a number of active color fields in which to display the image.

[0095] In some embodiments, the control information comprises color field indicator data that defines at least one color field that is inactive.

[0096] In some embodiments, the control information is embedded in the rendered image.

[0097] In some embodiments, as a result of the control information indicating that at least one color field is inactive, one or more light sources are powered down, thereby reducing power consumption. (Item 1) a display configured to display images of the multiple depth planes; and a display controller configured to receive rendered virtual or augmented reality image data from a graphics processor and to control the display based at least in part on control information embedded in the rendered image; Equipped with 2. A virtual or augmented reality display system, wherein the embedded control information indicates a shift to apply to at least a portion of the rendered image when the image is displayed. 2. The virtual or augmented reality display system of claim 1, wherein the shift changes the displayed position of one or more virtual or augmented reality objects compared to the position of the one or more objects in the rendered image. (Item 3) 2. The virtual or augmented reality display system of claim 1, wherein the shift comprises a lateral shift of at least a portion of the image by a defined number of pixels within the same depth plane. (Item 4) 2. The virtual or augmented reality display system of claim 1, wherein the shift comprises a vertical shift of at least a portion of the image from one depth plane to another. (Item 5) 5. The virtual or augmented reality display system of claim 4, wherein the display controller is further configured to scale at least a portion of the image in conjunction with a vertical shift from one depth plane to another. (Item 6) 2. The virtual or augmented reality display system of claim 1, wherein the shift comprises a vertical shift of at least a portion of the image from one depth plane to a virtual depth plane, the virtual depth plane comprising a weighted combination of at least two depth planes. (Item 7) 2. The virtual or augmented reality display system of claim 1, wherein the shift is based on information about the user's head pose. (Item 8) 2. The virtual or augmented reality display system of claim 1, wherein the shifting is performed by the display controller without re-rendering the image. (Item 9) 1. A method in a virtual or augmented reality display system, comprising: receiving rendered virtual or augmented reality image data from a graphics processor; displaying images of a plurality of depth planes based at least in part on control information embedded in the rendered images; Including, A method according to claim 1, wherein the embedded control information indicates a shift to apply to at least a portion of the rendered image when displaying the image. (Item 10) 10. The method of claim 9, wherein the step of displaying the image includes shifting a displayed position of one or more virtual or augmented reality objects compared to a position of the one or more objects in the rendered image. (Item 11) 10. The method of claim 9, wherein shifting the displayed position comprises laterally shifting at least a portion of the image by a prescribed number of pixels within the same depth plane based on the control information. (Item 12) 10. The method of claim 9, wherein shifting the displayed position comprises vertically shifting at least a portion of the image from one depth plane to another depth plane based on the control information. (Item 13) Item 13. The method of item 12, further comprising scaling at least a portion of the image in conjunction with vertically shifting the image from one depth plane to another. (Item 14) 10. The method of claim 9, wherein the shifting of the displayed position includes vertically shifting at least a portion of the image from one depth plane to a virtual depth plane, the virtual depth plane comprising a weighted combination of at least two depth planes. (Item 15) 10. The method of claim 9, wherein the shift is based on information about a user's head pose. (Item 16) 10. The method of claim 9, further comprising shifting the image without re-rendering the image. (Item 17) a sensor configured to determine one or more characteristics of the ambient lighting; a processor configured to adjust one or more characteristics of a virtual object based on the one or more characteristics of the ambient lighting; a display configured to display the virtual object to a user; and A virtual or augmented reality display system comprising: (Item 18) 20. The virtual or augmented reality display system of claim 17, wherein the one or more characteristics of the ambient lighting comprise a brightness of the ambient lighting. (Item 19) 20. The virtual or augmented reality display system of claim 17, wherein the one or more characteristics of the ambient lighting comprise a hue of the ambient lighting. (Item 20) Item 18. The virtual or augmented reality display system of item 17, wherein the one or more characteristics of the virtual object comprise a brightness of the virtual object. (Item 21) Item 18. The virtual or augmented reality display system of item 17, wherein the one or more characteristics of the virtual object comprise a hue of the virtual object. (Item 22) 1. A method in a virtual or augmented reality display system, comprising: receiving one or more characteristics of the ambient lighting from a sensor; adjusting, using a processor, one or more characteristics of a virtual object based on the one or more characteristics of the ambient lighting; displaying the virtual object to a user; A method comprising: [Brief description of the drawings]

[0098] [Figure 1] FIG. 1 illustrates a user's view of an augmented reality (AR) scene using an exemplary AR system.

[0099] [Diagram 2] FIG. 2 illustrates an embodiment of a wearable display system.

[0100] [Diagram 3] FIG. 3 illustrates a conventional display system for simulating a three-dimensional image for a user.

[0101] [Figure 4] FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes.

[0102] [Diagram 5] 5A-5C illustrate the relationship between the radius of curvature and the radius of focus.

[0103] [Figure 6] FIG. 6 illustrates an embodiment of a waveguide stack for outputting image information to a user.

[0104] [Figure 7] FIG. 7 shows an example of an output beam output by a waveguide.

[0105] [Figure 8] FIG. 8 illustrates an example design of a waveguide stack in which each depth plane has three associated waveguides, each outputting a different color of light.

[0106] [Figure 9] FIG. 9 illustrates an example timing scheme for a virtual or augmented reality system that displays light field images.

[0107] [Figure 10] FIG. 10 illustrates an exemplary format of a frame of video data, including accompanying control data.

[0108] [Figure 11]FIG. 11 illustrates another example format of a frame of video data, including control data.

[0109] [Figure 12] FIG. 12 illustrates an example format for a pixel of video data, including embedded control data.

[0110] [Figure 13] FIG. 13 illustrates how a frame of video can be separated into color components that can be displayed sequentially.

[0111] [Figure 14] FIG. 14 illustrates how a frame of light field video data can be separated into multiple depth planes, each of which can be divided into color component sub-frames for display, using depth plane indicator data.

[0112] [Figure 15] FIG. 15 illustrates an example where the depth plane indicator data of FIG. 12 indicates that one or more depth planes of a frame of light field video data are inactive.

[0113] [Figure 16] FIG. 16 illustrates an exemplary drawing area for a frame of a computer-generated image in an augmented reality system.

[0114] [Figure 17] FIG. 17 illustrates diagrammatically possible movements of the user's head about two axes of rotation.

[0115] [Figure 18] FIG. 18 illustrates how a user's head pose can be mapped onto a three-dimensional surface.

[0116] [Figure 19]FIG. 19 illustrates diagrammatically different head pose regions that may be used to define gain factors for improving head pose tracking. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0117] The virtual and augmented reality systems disclosed herein may include a display that presents computer-generated images to a user. In some embodiments, the display system is wearable, which may advantageously provide a more immersive VR or AR experience. FIG. 2 illustrates an example of a wearable display system 80. The display system 80 includes a display 62 and various mechanical and electronic modules and systems that support the functionality of the display 62. The display 62 may be coupled to a frame 64 that is wearable by a display system user or viewer 60 and configured to position the display 62 directly in front of the user's 60 eye. In some embodiments, a speaker 66 is coupled to the frame 64 and positioned adjacent to the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). The display 62 is operably coupled to a local data processing module 70, which may be mounted in a variety of configurations, such as fixedly attached to the frame 64, such as by a wired or wireless connection 68, fixedly attached to a helmet or hat worn by the user, embedded within headphones, or otherwise removably attached to the user 60 (e.g., in a backpack-type configuration, a belt-attached configuration, etc.).

[0118] The local processing and data module 70 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to assist in processing and storing data. This includes data captured from sensors, such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. The sensors may be operatively coupled to the frame 64, for example, or otherwise attached to the user 60. Alternatively, or in addition, the sensor data may be acquired and / or processed using a remote processing module 72 and / or a remote data repository 74, perhaps for passing to the display 62 after such processing or readout. The local processing and data module 70 may be operatively coupled to these remote processing modules 72 and remote data repository 74 by communications links (76, 78), such as via wired or wireless communication links, such that the remote modules (72, 74) are operatively coupled to each other and available as resources to the local processing and data module 70.

[0119] In some embodiments, remote processing module 72 may include one or more processors configured to analyze and process data (e.g., sensor data and / or image information). In some embodiments, remote data repository 74 may comprise a digital data storage facility that may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, allowing for fully autonomous use from the remote module.

[0120] In some embodiments, the computer-generated images provided via display 62 can create the impression of being three-dimensional. This can be done, for example, by presenting a stereoscopic image to the user. In some conventional systems, such images can include separate images of a scene or object from slightly different perspectives. The separate images can be presented to the user's right and left eyes, respectively, thus simulating binocular vision and its associated depth perception.

[0121] FIG. 3 illustrates a conventional display system for simulating a three-dimensional image for a user. Two distinct images 74 and 76, one for each eye 4 and 6, are output to the user. The images 74 and 76 are spaced from the eyes 4 and 6 by a distance 10 along an optical axis or z-axis parallel to the viewer's line of sight. The images 74 and 76 are flat, and the eyes 4 and 6 may focus on the images by assuming a single accommodated state. Such a system relies on the human visual system to combine the images 74 and 76 to provide the perception of depth of the combined image.

[0122] However, it should be appreciated that the human visual system is more complex and providing a realistic perception of depth is more difficult. For example, many viewers of conventional 3D display systems may find such systems uncomfortable or may not perceive any sensation of depth at all. Without being limited by theory, it is believed that a viewer of an object may perceive the object as "three-dimensional" due to a combination of vergence and accommodation. The vergence movement of the two eyes relative to each other (i.e., the rolling movement of the pupils toward or away from each other to cause the gaze of the eyes to converge and fixate on the object) is closely associated with the focusing of the eye's lenses (or "accommodation"). Under normal conditions, changing the focus of the eye's lenses, or accommodating the eyes to change focus from one object to another at different distances, will automatically cause a corresponding change in vergence to the same distance, under a relationship known as the "accommodation-vergence-divergence reflex." Similarly, changes in vergence-divergence will induce corresponding changes in accommodation under normal conditions. As described herein, many stereoscopic display systems display a scene using a slightly different presentation (and therefore a slightly different image) to each eye so that a three-dimensional perspective is perceived by the human visual system. However, such systems are uncomfortable for many viewers because they merely provide a different presentation of the scene but the eyes view the entire image information in a single accommodated state and therefore work against the accommodation-vergence-divergence reflex. Display systems that provide better matching between accommodation and vergence-divergence may produce a more realistic and comfortable simulation of a three-dimensional image.

[0123] For example, light field images can be presented to a user to simulate a three-dimensional view. Light field images can mimic the light rays that enter the viewer's eyes in a real-world environment. For example, when displaying a light field image, light rays from objects simulated to be perceived at a distance are more collimated when entering the viewer's eyes, while light rays from objects simulated to be perceived close are more diverged. Thus, the angle at which light rays from objects in a scene enter the viewer's eyes depends on the simulated distance of these objects from the viewer. Light field images in virtual or augmented reality systems can include multiple images of a scene or object from different depth planes. The images can be different for each depth plane (e.g., providing a slightly different presentation of the scene or object) and focused separately by the viewer's eyes, thereby helping to provide a comfortable depth perception to the user.

[0124] When these multiple depth plane images are presented to a viewer, either simultaneously or in succession, the result is interpreted by the viewer as a three-dimensional image. When a viewer experiences this type of light field image, the eyes accommodate to focus the different depth planes much like they would when experiencing a real-world scene. These focusing cues can provide a more realistic simulated three-dimensional environment.

[0125] In some configurations, at each depth plane, a full-color image may be formed by superimposing constituent images, each having a particular constituent color. For example, red, green, and blue images may each be output separately to form a respective full-color depth plane image. As a result, each depth plane may have multiple constituent color images associated with it.

[0126] FIG. 4 illustrates aspects of an approach for simulating a three-dimensional image using multiple depth planes. With reference to FIG. 4A, objects at various distances from eyes 4 and 6 on the z-axis are accommodated by eyes (4, 6) such that the objects are in focus. Eyes 4 and 6 assume specific accommodated states and focus on objects at different distances along the z-axis. As a result, a specific accommodated state may be said to be associated with a particular one of depth planes 14 such that an object or part of an object in a particular depth plane is in focus when the eye is in an accommodated state for that depth plane. In some embodiments, a three-dimensional image may be simulated by providing different presentations of images for each eye (4, 6) and by providing different presentations of images corresponding to each of the depth planes.

[0127] The distance between an object and an eye (4 or 6) can change the amount of divergence of light from that object as viewed by that eye. Figures 5A-5C illustrate the relationship between distance and divergence of light rays. The distance between an object and the eye 4 is represented in the order of decreasing distances R1, R2, and R3. As shown in Figures 5A-5C, the light rays become more divergent as the distance to the object decreases. As the distance increases, the light rays become more collimated. In other words, the light field generated by a point (an object or part of an object) can be said to have a spherical wavefront curvature that is a function of the distance the point is away from the user's eye. The curvature increases with decreasing distance between the object and the eye 4. As a result, at different depth planes, the divergence of the light rays is also different, and the divergence increases with decreasing distance between the depth plane and the viewer's eye 4. Only a single eye 4 is illustrated in FIGS. 5A-5C and other figures herein for clarity of illustration, but it should be understood that the discussion regarding eye 4 may apply to both eyes (4 and 6) of a viewer.

[0128] Without being limited by theory, it is believed that the human eye is typically capable of interpreting a finite number of depth planes to provide depth perception, such that a highly realistic simulation of perceived depth can be achieved by providing the eye with a different presentation of an image corresponding to each of these limited number of depth planes.

[0129] FIG. 6 illustrates an example of a waveguide stack for outputting image information to a user. The display system 1000 includes a stack of waveguides or a stacked waveguide assembly 178 that can be utilized to provide a three-dimensional perception to the eye / brain using multiple waveguides (182, 184, 186, 188, 190). In some embodiments, the display system 1000 is the system 80 of FIG. 2, and FIG. 6 shows several portions of the system 80 in more detail. For example, the waveguide assembly 178 may be integrated into the display 62 of FIG. 2.

[0130] Continuing with reference to FIG. 6, the waveguide assembly 178 may also include a number of features (198, 196, 194, 192) between the waveguides. In some embodiments, the features (198, 196, 194, 192) may be lenses. The waveguides (182, 184, 186, 188, 190) and / or the number of lenses (198, 196, 194, 192) may be configured to transmit image information to the eye with various levels of wavefront curvature or light beam divergence. Each waveguide level may be associated with a particular depth plane and may be configured to output image information corresponding to that depth plane. The image injection devices (200, 202, 204, 206, 208) may be utilized to inject image information into the waveguides (182, 184, 186, 188, 190), each of which may be configured to disperse incident light across each individual waveguide for output towards the eye 4, as described herein. Light exits output surfaces (300, 302, 304, 306, 308) of the image injection devices (200, 202, 204, 206, 208) and is injected into corresponding input edges (382, 384, 386, 388, 390) of the waveguides (182, 184, 186, 188, 190). In some embodiments, a single light beam (e.g., a collimated beam) may be launched into each waveguide to output a total field of cloned collimated beams directed toward eye 4 at a particular angle (and divergence) corresponding to the depth plane associated with the particular waveguide.

[0131] In some embodiments, the image input devices (200, 202, 204, 206, 208) are discrete displays, each of which generates image information for input into a corresponding waveguide (182, 184, 186, 188, 190, respectively). In some other embodiments, the image input devices (200, 202, 204, 206, 208) are the output of a single multiplexed display, which may, for example, send image information to each of the image input devices (200, 202, 204, 206, 208) via one or more optical conduits (such as fiber optic cables).

[0132] The controller 210 controls the operation of the stacked waveguide assembly 178 and the image input devices (200, 202, 204, 206, 208). In some embodiments, the controller 210 includes programming (e.g., instructions in a non-transitory computer-readable medium) that coordinates the timing and provisioning of image information to the waveguides (182, 184, 186, 188, 190), for example, according to any of the various schemes disclosed herein. In some embodiments, the controller may be a single integrated device or a distributed system connected by wired or wireless communication channels. The controller 210 may be part of the processing module (70 or 72) (FIG. 2) in some embodiments.

[0133] The waveguides (182, 184, 186, 188, 190) may be configured to propagate light within each individual waveguide by total internal reflection (TIR). Each of the waveguides (182, 184, 186, 188, 190) may be planar or curved, with major top and bottom surfaces and edges extending between the top and bottom surfaces. In the illustrated configuration, the waveguides (182, 184, 186, 188, 190) may each include a light redirecting element (282, 284, 286, 288, 290) configured to redirect light propagating within each individual waveguide out of the waveguide and output image information to the eye 4. A beam of light is output by the waveguide where the light propagating within the waveguide strikes the light redirecting element. The light redirecting elements (282, 284, 286, 288, 290) may be reflective and / or diffractive optical features. Although shown disposed on the bottom major surface of the waveguides (182, 184, 186, 188, 190) for ease of explanation and clarity of the drawings, in some embodiments the light redirecting elements (282, 284, 286, 288, 290) may be disposed on the top and / or bottom major surfaces and / or directly within the volume of the waveguides (182, 184, 186, 188, 190). In some embodiments, the light redirecting elements (282, 284, 286, 288, 290) may be formed in a layer of material that is attached to a transparent substrate to form the waveguides (182, 184, 186, 188, 190). In some other embodiments, the waveguides (182, 184, 186, 188, 190) may be a monolithic piece of material and the light redirecting elements (282, 284, 286, 288, 290) may be formed on and / or within that piece of material.

[0134] With continued reference to FIG. 6, as discussed herein, each waveguide (182, 184, 186, 188, 190) is configured to output light and form an image corresponding to a particular depth plane. For example, the waveguide (182) closest to the eye may be configured to deliver collimated light to the eye 4 as it is launched into such waveguide 182. The collimated light may represent an optical infinity focal plane. The next upper waveguide 184 may be configured to send collimated light that passes through a first lens (192; e.g., a negative lens) before it can reach the eye 4. Such first lens 192 may be configured to generate a slight convex wavefront curvature such that the eye / brain interprets the light originating from the next upper waveguide 184 as originating from a first focal plane closer inward from optical infinity toward the eye 4. Similarly, the third upper waveguide 186 passes its output light through both the first lens 192 and the second lens 194 before reaching the eye 4. The combined refractive powers of the first lens 192 and the second lens 194 may be configured to produce another incremental amount of wavefront curvature such that the eye / brain interprets the light originating from the third waveguide 186 as originating from a second focal plane that is closer inward from optical infinity towards the person than the light from the next upper waveguide 184.

[0135] The other waveguide layers (188, 190) and lenses (196, 198) are similarly configured, with the highest waveguide 190 in the stack sending its output through all of the lenses between it and the eye for an aggregate focal power that represents the focal plane closest to the person. To compensate the stack of lenses (198, 196, 194, 192) when viewing / interpreting light originating from the world 144 on the other side of the stacked waveguide assembly 178, a compensating lens layer 180 may be placed on top of the stack to compensate for the aggregate power of the lower lens stacks (198, 196, 194, 192). Such a configuration provides as many perceived focal planes as there are available waveguide / lens pairs. Both the light redirecting elements of the waveguides and the focusing sides of the lenses may be static (i.e., not dynamic or electroactive). In some alternative embodiments, they may be dynamic using electroactive features.

[0136] With continued reference to FIG. 6, the light redirecting elements (282, 284, 286, 288, 290) may be configured for a particular depth plane associated with a waveguide to both redirect light out of its respective waveguide and output the light with an appropriate amount of divergence or collimation. As a result, waveguides with different associated depth planes may have different configurations of light redirecting elements (282, 284, 286, 288, 290) that output light with different amounts of divergence depending on the associated depth plane. In some embodiments, as discussed herein, the light redirecting elements (282, 284, 286, 288, 290) may be volume or surface features that may be configured to output light at specific angles. For example, the light redirecting elements (282, 284, 286, 288, 290) may be volume holograms, surface holograms, and / or diffraction gratings. Light redirecting elements such as diffraction gratings are described in U.S. Patent Application Serial No. 14 / 641,376, filed March 7, 2015, which is incorporated by reference in its entirety. In some embodiments, the features (198, 196, 194, 192) may not be lenses; rather, they may simply be spacers (e.g., cladding layers and / or structures for forming air gaps).

[0137] In some embodiments, the light redirecting elements (282, 284, 286, 288, 290) are diffractive features that form a diffraction pattern or "diffractive optical element" (also referred to herein as "DOE"). Preferably, the DOE has a relatively low diffraction efficiency so that only a portion of the light of the beam is deflected towards the eye 4 with each intersection of the DOE, while the remainder continues to travel through the waveguide via total internal reflection. The light carrying the image information is thus split into several related exit beams that exit the waveguide at various locations, the result being a very uniform pattern of exit emission towards the eye 4 for this particular collimated beam that reflects around in the waveguide.

[0138] In some embodiments, one or more DOEs may be switchable between an "on" state in which they actively diffract and an "off" state in which they do not significantly diffract. For example, a switchable DOE may comprise a layer of polymer dispersed liquid crystal in which the microdroplets comprise a diffractive pattern in a host medium, and the refractive index of the microdroplets can be switched to substantially match the refractive index of the host material (in which case the pattern does not significantly diffract incident light), or the microdroplets can be switched to a refractive index that does not match that of the host medium (in which case the pattern actively diffracts incident light).

[0139] FIG. 7 shows an example of an output beam output by a waveguide. Although one waveguide is shown, it should be understood that other waveguides in the stack of waveguides 178 may function similarly. Light 400 is launched into the waveguide 182 at the input edge 382 of the waveguide 182 and propagates within the waveguide 182 by TIR. At the point where the light 400 impinges on the DOE 282, a portion of the light exits the waveguide as an output beam 402. The output beams 402 are shown as substantially parallel, but as discussed herein, they may also be redirected to propagate to the eye 4 at an angle (e.g., forming a diverging output beam) depending on the depth plane associated with the waveguide 182. It should be understood that a substantially parallel output beam may be indicative of a waveguide that corresponds to a depth plane at a large simulated distance from the eye 4 (e.g., optical infinity). Other waveguides may output a more divergent exit beam pattern, which would require the eye 4 to accommodate to focus on a closer simulated distance and would be interpreted by the brain as light from a distance closer to the eye 4 than optical infinity.

[0140] FIG. 8 illustrates a schematic of an exemplary design of a stacked waveguide assembly in which each depth plane has three associated waveguides, each outputting a different color of light. A full color image may be formed in each depth plane by overlaying images in each of multiple constituent colors, e.g., three or more constituent colors. In some embodiments, the constituent colors include red, green, and blue. In some other embodiments, other colors, including magenta, yellow, and cyan, may be used in conjunction with or replace one of red, green, or blue. Each waveguide may be configured to output a particular constituent color, such that each depth plane may have multiple waveguides associated with it. Each depth plane may have, for example, three waveguides associated with it, i.e., one waveguide for outputting red light, a second waveguide for outputting green light, and a third waveguide for outputting blue light.

[0141] Continuing with reference to FIG. 8, depth planes 14a-14f are shown. In the illustrated embodiment, each depth plane has three constituent color images associated with it: a first image of a first color G, a second image of a second color R, and a third image of a third color B. As a convention herein, the number following each of these letters indicates the diopter (1 / m), or the inverse of the apparent distance of the depth plane from the viewer, and each box in the figure represents an individual constituent color image. In some embodiments, G is green, R is red, and B is blue. As discussed above, the perceived distance of the depth plane from the viewer may be established by light redirecting elements (282, 284, 286, 288, 290), such as diffractive optical elements (DOEs), and / or lenses (198, 196, 194, 192) that diverge light at an angle associated with the apparent distance.

[0142] In some arrangements, each constituent color image may be output by a different waveguide in the stack of waveguides. For example, each depth plane may have three constituent color images associated with it: a first waveguide that outputs a first color G, a second waveguide that outputs a second color R, and a third waveguide that outputs a third color B. In an arrangement in which waveguides are used to output constituent color images, each box in the diagram may be understood to represent an individual waveguide.

[0143] It should be understood that while the waveguides associated with each depth plane are shown adjacent to one another in this schematic diagram for ease of illustration, in a physical device the waveguides may all be arranged in a stack with one waveguide per level. The different depth planes are indicated in the diagram by different numbers of diopters following the letters G, R, and B. (Display Timing Method)

[0144] In some embodiments, a virtual or augmented reality system provides a light field image by successively displaying multiple different depth planes for a given frame of video data. The system then updates to the next frame of video data and successively displays multiple different depth planes for that frame. For example, a first frame of video data may actually include three separate sub-frames of data: a far-field frame D0, a mid-field frame D1, and a near-field frame D2. D0, D1, and D2 may be displayed successively. Subsequently, a second frame of video data may be displayed. The second frame of video data may also include a far-field frame, a mid-field frame, and a near-field frame, etc., displayed successively. Although this embodiment uses three depth planes, the light field image is not so limited. Rather, any multiple depth planes may be used, depending, for example, on the desired video frame rate and the capabilities of the system.

[0145] Because each frame of light field video data includes multiple subframes for different depth planes, systems that provide light field images may benefit from a display panel capable of high refresh rates. For example, if the system displays video at a frame rate of 120 Hz but includes images from multiple different depth planes, the display would need to be capable of a refresh rate greater than 120 Hz to accommodate the multiple depth plane images for each frame of video. In some embodiments, a liquid crystal on silicon (LCOS) display panel is used, although other types of display panels (including color sequential and non-color sequential displays) can also be used.

[0146] 9 illustrates an example timing scheme for a virtual or augmented reality system displaying a light field image. In this example, the video frame rate is 120 Hz and the light field image includes three depth planes. In some embodiments, the green, red, and blue components of each frame are displayed sequentially rather than simultaneously.

[0147] A video frame rate of 120 Hz allows 8.333 milliseconds to display all of the depth planes for a single frame of video. As illustrated in FIG. 9, each frame of video data includes three depth planes, each depth plane including green, red, and blue color components. For example, depth plane D0 includes a green subframe G0, a red subframe R0, and a blue subframe B0. Similarly, depth plane D1 includes green, red, and blue subframes G1, R1, and B1, respectively, and depth plane D2 includes green, red, and blue subframes G2, R2, and B2, respectively. Considering that each video frame includes three depth planes, each depth plane having three color components, the allocated 8.333 milliseconds is divided into nine partitions of 0.926 milliseconds each. As illustrated in FIG. 9, the green subframe G0 for the first depth plane is displayed during the first time segment, the red subframe R0 for the first depth plane is displayed during the second time segment, and so on. The total green on time for each frame of video is 2.778 milliseconds. The same applies to the total red on time and blue on time for each video frame. However, it should be understood that other video frame rates may also be used, in which case the specific time intervals illustrated in FIG. 9 may be adjusted accordingly. Although the individual color components are illustrated as having equal display times, this is not required, and the ratio of display times between color components may be varied. Furthermore, the blinking order illustrated in FIG. 9 for the depth planes and color component subframes is only one example. Other blinking orders may also be used. Also, while FIG. 9 illustrates an embodiment using color sequential display technology, the techniques described herein are not limited to color sequential displays.

[0148] Other display timing schemes are also possible. For example, the frame rate, number of depth planes, and color components may vary. In some embodiments, the frame rate of a virtual or augmented reality system as described herein is 80 Hz and there are three depth planes. In some embodiments, different depth planes may be displayed in different frames. For example, a light field video with four depth planes may be displayed at an effective frame rate of 60 Hz by displaying two depth planes per frame at a frame rate of 120 Hz (depth planes D0 and D1 may be displayed in the first 8.33 ms, depth planes D2 and D3 may be displayed in the next 8.33 ms, with the full depth information provided in 16.7 ms for an effective frame rate of 60 Hz). In some embodiments, the number of depth planes shown may vary spatially on the display. For example, a greater number of depth planes may be shown in a lower portion of the display within the user's line of sight, and a smaller number of depth planes may be shown in a lower portion of the display located within the user's peripheral vision. In such an embodiment, an eye-tracking device (eg, a camera and eye-tracking software) can be used to determine what part of the display the user is looking at. (Control data for video data)

[0149] FIG. 10 illustrates an example format of a frame of video data, including accompanying control data. As illustrated in FIG. 10, each frame of video data may comprise an array of pixel data formatted into rows and columns. In the illustrated example, there are 1280 columns and 960 rows of pixel data forming an image. FIG. 10 also illustrates that control data 1010 may be appended to the frame of video data. In this example, the control packet 1010 may be appended to the frame of video data, for example, as an extra row. The first row (row 000) comprises control information, while rows 1-960 contain the actual image. Thus, in this embodiment, the host transmits a resolution of 1280×961 to the display controller.

[0150] The display controller reads the attached control information and uses it, for example, to compose image information 1020 that is sent to one or more display panels (e.g., left-eye and right-eye display panels). In this embodiment, no rows of control information 1010 are sent to the display panel. Thus, the host transmits information to the display controller at a resolution of 1280×961, but the display controller removes the control information 1010 from the stream of data and transmits only video information 1020 at a resolution of 1280×961 to the display panel. Image data can be transmitted to the display panel (e.g., an LCOS display panel) in, for example, a Display Serial Interface (DSI) format. Although FIG. 10 illustrates that the attached control information 1010 comprises a single row that is appended to the beginning of each frame of video data, other amounts of control information can alternatively be appended. Additionally, the control information 1010 does not necessarily have to be appended to the beginning of each frame of video data, but can instead be inserted into the video data at other locations. However, attaching the control information to the beginning of a frame may allow the controller to more easily act on the control information at the beginning of a frame of a rendered image prior to displaying the image data.

[0151] Figure 11 illustrates another example format of a frame of video data including control data. Figure 11 is similar to Figure 10, except that the control information 1110 is inserted in place of the first row of video data, rather than being appended to the frame of video data before the first row. Thus, the first row of the frame (row 000) comprises the control information, while the remaining 959 rows comprise the actual image data 1120.

[0152] In this example, the host transmits information at a resolution of 1280x961 to the display controller. The display controller can use the control data 1110 to configure the image information to be sent to the display panel. The display controller then transmits the frame of video data illustrated in Figure 11 to the display panel. However, in some embodiments, before transmitting the frame of video data illustrated in Figure 11 to the display panel, the display controller can remove the control information 1110, for example, by setting that row of video data to zero. This causes the first row of each frame of video data to appear as a dark line on the display.

[0153] Using the scheme illustrated in Figure 11, control information 1110 can be included with a frame of video data without changing the resolution of the information sent to the display controller. However, the tradeoff in this embodiment is that the effective display resolution is reduced due to the fact that some image data is replaced by the control data. Although Figure 11 illustrates that the control data 1110 is inserted in place of the first row of image data, the control data could alternatively be inserted in place of another row in the frame.

[0154] For example, the control data illustrated in FIGS. 10 and 11 (and subsequently in FIG. 12) can be used for a number of different purposes. For example, the control data can indicate whether a frame of video data should be displayed on a left eye video panel or a right eye video panel. The control data can indicate which of a number of depth planes the frame of video data corresponds to. The control data can indicate a blinking order of light field video data. For example, the control data can indicate an order to display each depth plane and an order to display color component sub-frames for each depth plane. In addition, it may be necessary to shift pixels left / right or up / down after content for a display has already been generated by the host. Rather than adjusting and re-rendering image data, the control data can include pixel shift information that specifies the direction and magnitude of pixel shifting to be performed by the display controller.

[0155] Such pixel shifting can be implemented for several reasons. Pixel shifting can be done when image content needs to be moved on the display, for example, due to a user's head movement. In such cases, the content may be the same, but its location within the viewing area on the display may need to be shifted. Rather than re-rendering the image content in the GPU and sending the entire set of pixels again to the display controller, pixel shifting can be applied to the image data using pixel shift control information. As illustrated in Figures 10 and 11, pixel shift control information can be included at the beginning of a frame. Alternatively and / or in addition, a late update control data packet can be sent within the frame (e.g., after the first row) to perform the appropriate pixel shifting based on the updated head pose mid-frame. This can be done, for example, using a Mobile Industry Processor Interface (MIPI) Display Serial Interface (DSI) virtual channel.

[0156] Pixel shifting can also be done when the user moves their head and a more accurate representation of the pixels is desired. Rather than having the GPU re-render the image, late shifting on the display can be applied using a pixel shifting approach. Any pixel shifting described herein can affect a single depth plane or multiple depth planes. As already discussed herein, in some embodiments, there are time differences when the various depth planes are displayed. During these time differences, the user may shift their eyes such that the viewing frustum may need to be shifted. This can be achieved using pixel shifting on any of the depth planes.

[0157] The pixel shift control information may indicate pixel shift in the XY direction within a frame of a single depth plane. Alternatively and / or in addition, the pixel shift control information may indicate a shift in the Z direction between depth plane buffers. For example, an object previously displayed in one or more depth planes may be moved to another depth plane set using the Z-pixel shift. This type of shift may also include a scaler that expands or shrinks the partial image by depth. For example, assume that a displayed character is floating between two depth planes and there is no obstruction of the character by another object. The apparent movement of the character in depth may be achieved by redrawing the character forward or backward in one or more depth planes using the Z-pixel shift and the scaler. This may be achieved without re-rendering the character and sending frame updates to the display controller, resulting in smoother movement performance at a lower computational expense.

[0158] Scalers can also be used to compensate for magnification effects that occur in the display, for example, as a result of lenses 192, 194, 196, 198. Such lenses may create a virtual image that is observable by a user. When a virtual object moves from one depth plane to another, the optical magnification of the virtual image may actually be the opposite of what would be expected in the physical world. For example, in the physical world, when an object is located in a depth plane further away from the viewer, it appears smaller than if it were located in a closer depth plane. However, when a virtual object moves from a closer to a farther depth plane in the display, the lenses may actually magnify the virtual image of the object. Thus, in some embodiments, a scaler is used to compensate for optical magnification effects in the display. A scaler can be provided per depth plane to correct for magnification effects caused by optical elements. In addition, a scaler can be provided per color if there are any scaling issues to be addressed on a per color basis.

[0159] In some embodiments, the maximum horizontal pixel shift can correspond to the entire panel width, while the maximum vertical pixel shift can correspond to the entire panel height. Both positive and negative shifts can be indicated by the control data. Using this pixel shift information, the display controller can shift frames of video data left or right, up or down, and forward or backward between depth planes. The pixel shift information can also fully or partially shift frames of video data from the left eye display panel to the right eye display panel, or vice versa. The pixel shift information can be included in the light field video data for each depth plane.

[0160] In some embodiments, such as those where a scanning-based display is used, incremental distributed pixel shifting can be provided. For example, an image for a frame of video can be incrementally shifted in one or more depth planes until the end of the image (e.g., the bottom) is reached. The first displayed pixels can be shifted more or less than the pixels displayed later in the frame to compensate for head movement or to simulate object movement. Additionally, there can be incremental pixel shifting on a plane-by-plane basis. For example, pixels in one depth plane can be shifted more or less than pixels in another depth plane. In some embodiments, eye-tracking techniques are used to determine which part of the display screen the user is looking at. Objects in different depth planes, or even at different locations within a single depth plane, can be pixel shifted (or not) depending on where the user is looking. If there are objects the user is not looking at, the pixel shift information for these objects can be ignored to improve performance for pixel shifting in the image the user is looking at. Again, an eye-tracking device can be used to determine where on the display the user is looking.

[0161] The control information can also be used to define and / or adjust one or more virtual depth planes. A virtual depth plane can be provided at a desired interval between two defined depth planes in a virtual or augmented reality system by blending two depth plane images with appropriate weighting to maintain a desired brightness of the image. For example, if a virtual depth plane is desired between depth plane D0 and depth plane D1, a blending unit can weight the pixel values ​​of the D0 image data by 50%, while also weighting the pixel values ​​of the D1 image data by 50%. (As long as the weightings sum to 100%, then the apparent brightness of the image can be maintained.) The result would be a virtual depth plane that appears to be located halfway between D0 and D1. The apparent depth of the virtual depth plane can be controlled by using different blending weightings. For example, if it is desired that the virtual depth plane appears closer to D1 than to D0, the D1 image can be weighted more heavily. One or more scalers can be used to ensure that the virtual object is substantially the same size in both of the depth planes being blended such that similar portions of the virtual object are combined during the blending operation. The control data can specify when the virtual depth plane image is calculated, and the control information can also include blending weights for the virtual depth planes. In various embodiments, the weights can be stored in a programmable look-up table (LUT). The control information can be used to select appropriate weights from the LUT that will provide the desired virtual depth plane.

[0162] The control information may also indicate whether an image frame for one of the two stereoscopic displays should be copied to the other. For example, in the case of the furthest simulated depth plane (e.g., background image), there may be a relatively small difference (e.g., due to parallax shift) between the left and right eye images. In such a case, the control information may indicate that the image for one of the stereoscopic displays is copied to the other display for one or more depth planes. This may be accomplished without re-rendering image data in the GPU for both the left and right eye displays or re-transferring data to the display controller. If there is a relatively small difference between the left and right eye images, pixel shift may also be used to compensate without re-rendering or re-transferring image data for both eyes.

[0163] The control data illustrated in Figures 10 and 11 can also be used for other purposes than those specifically recited here.

[0164] Although Figures 10 and 11 show that rows of control data may be included with the video data, the control data may also (or alternatively) be embedded into individual pixels of the video data. This is illustrated in Figure 12, which illustrates an example format of a pixel 1200 of the video data, including embedded control data 1240. Figure 12 illustrates that a pixel of the video data comprises a blue color value 1230 (byte 0), a green color value 1220 (byte 1), and a red color value 1210 (byte 2). In this embodiment, the color values ​​each have a color depth of 8 bits. In some embodiments, one or more of the bits corresponding to one or more of the color values ​​may be replaced by the control data 1240, at the expense of the bit depth of the color value. Thus, the control data may be embedded directly into a pixel of the video data, at the expense of the dynamic range of the color value of the pixel. For example, as illustrated in Figure 12, the highlighted two least significant bits of the blue color value may be dedicated as the control data 1240. Although not shown, bits of other color values ​​can also be dedicated as control data, and a different number of pixel bits can be dedicated as control data.

[0165] In some embodiments, the control data 1240 incorporated into the pixels of the video data can be depth plane indicator data (although the control data incorporated into the pixels can also be any other type of control data, including other types described herein). As discussed herein, light field video data can include several depth planes. The bit depth for one or more pixels in a video frame can be reduced, and the resulting available bits can be used to indicate the depth plane to which the pixel corresponds.

[0166] As a specific example, consider the 24-bit RGB pixel data illustrated in FIG. 12. The red, green, and blue values ​​each have a bit depth of 8 bits. As already discussed, the bit depth of one or more of the color components can be sacrificed and replaced by depth plane indicator data. For example, the blue component can be represented by 6 bits (bits B3-B8 in FIG. 12) instead of 8 bits because the eye is less sensitive to blue. The resulting extra two bits (bits B1 and B2) can be used to specify which of up to four depth planes the pixel corresponds to. If there are more or fewer depth planes, more or fewer color bits can be sacrificed. For example, if the bit depth is reduced by one bit, up to two depth planes can be defined. If the bit depth is reduced by three bits, up to eight depth planes can be defined, and so on. In this way, the dynamic range of the color values ​​can be traded off for the ability to encode the depth plane indicator data directly within the image data itself.

[0167] In some embodiments, the depth plane indicator data 1240 is encoded in all pixels of the video data. In other embodiments, the depth plane indicator data 1240 may be encoded in one pixel per frame, or one pixel per line, one pixel per virtual or augmented reality object, etc. Additionally, the depth plane indicator data 1240 can be encoded in only a single color component, or in multiple color components. Similarly, the technique of encoding the depth plane indicator data 1240 directly in the image data is not limited to only color images. The technique can be practiced for grayscale images, etc. as well.

[0168] FIG. 12 illustrates one technique for encoding depth plane indicator data in image data. Another technique is to employ chroma subsampling and use the resulting available bits as depth plane indicator data. For example, image data can be represented in a YCbCr format, where Y represents the luma component (which may or may not be gamma corrected), Cb represents the blue-difference chroma component, and Cr represents the red-difference chroma component. Because the eye is not as sensitive to chroma resolution as to luma resolution, chroma information can be provided with less resolution than luma information without unduly degrading image quality. In some embodiments, a YCbCr 4:2:2 format is used, where a Y value is provided for every pixel, but Cb and Cr values ​​are each provided for only every other pixel in an alternating fashion. If a pixel (without chroma sampling) would normally consist of 24 bits of information (an 8-bit Y value, an 8-bit Cb value, and an 8-bit Cr value), then after employing chroma subsampling, each pixel would require only 16 bits of information (an 8-bit Y value and an 8-bit Cb or Cr value). The remaining 8 bits can be used as depth plane indicator data. The depth plane indicator data can be used to separate pixels into the appropriate depth plane to be displayed at the appropriate time.

[0169] 12 and the chroma subsampling embodiment, the depth plane indicator data may define an actual depth plane supported by the virtual or augmented reality system and / or a virtual depth plane as discussed herein. If the depth plane indicator data defines a virtual depth plane, it may also define the weighting of the depth planes to be blended, as discussed herein.

[0170] The use of embedded depth plane indicator data in a display controller is illustrated in FIG. 14. But first, FIG. 13 is provided as background to show the operation of a display controller when only a single depth plane is present. FIG. 13 illustrates how a frame of video can be separated into color components that can be displayed sequentially. The left panel 1310 of FIG. 13 shows an image comprising one frame of a 120 frames per second video. As shown by the right panel 1330 of FIG. 13, the image is separated into red, green, and blue color components that are flashed on the display by the display controller for 1 / 120 of a second (8.33 milliseconds). For simplicity, FIG. 13 shows that the color components are each flashed once, and that the color components are each active for the same amount of time. The human visual system then blends the individual color component sub-frames into the original color image shown in the left panel of FIG. 13. FIG. 14 shows how this process can be adapted when each frame of video data includes multiple depth planes.

[0171] FIG. 14 illustrates how a frame of light field video data can be separated into multiple depth planes, each of which can be divided into color component subframes for display, using depth plane indicator data. In some embodiments, a host transmits a stream of light field video data to a display controller. This stream of video data is represented by the image in the left panel 1410 of FIG. 14. The display controller can use the embedded depth plane indicator data 1240 to separate the stream of video data into multiple RxGxBx sequences, where the R0G0B0 sequence corresponds to a first depth plane, the R1G1B1 sequence corresponds to a second depth plane, and the R2G2B2 sequence corresponds to a third depth plane. As illustrated in FIG. 13, this depth plane separation can be done based on the two least significant blue bits in each pixel. The result is shown in the center panel 1420 of FIG. 14, which shows three separate depth plane images. Finally, each of the three separate depth plane images shown in the center panel 1420 of Figure 14 can be separated into its constituent color component sub-frames. The color component sub-frames of each depth plane can then be flashed sequentially on the display as illustrated by the right panel 1430 of Figure 14. The sequence order can be, for example, R0G0B0R1G1B1R2G2B2 as illustrated in Figure 14, or G0R0B0G1R1B1G2R2B2 as illustrated in Figure 9.

[0172] The depth plane indicator data 1240 can be used by the display controller to determine the number of RxGxBx sequences to use and which pixels correspond to which sequences. Control data can also be provided to define the order in which the RxGxBx color sequences are flashed to the display. For example, for video data including three depth planes (D0, D1, D2), there are six possible orders in which the individual RxGxBx sequences can be flashed to the display panel: D0, D1, D2; D0, D2, D1; D1, D0, D2; D1, D2, D0; D2, D0, D1; and D2, D1, D0. If the order defined by the control data is D0, D1, D2, then the pixel with the blue LSB bit 0b00 corresponding to the first depth plane D0 can be selected as the first outgoing RxGxBx color sequence image. The pixel with blue LSB bits 0b01 corresponding to the second depth plane D1 may be selected as the outgoing second RxGxBx color sequence image, and so on.

[0173] Figure 15 illustrates an example in which the depth plane indicator data of Figure 12 indicates that one or more depth planes of a frame of light field video data are inactive. Figure 15 is similar to Figure 14 in that it shows a stream of video data (represented by the left panel 1510 of Figure 15) being separated into depth planes (represented by the center panel 1520 of Figure 15) which are then each separated into color component sub-frames (represented by the right panel 1530 of Figure 15). However, Figure 15 is distinct from Figure 14 in that only a single depth plane is shown as being active.

[0174] As already discussed, the depth plane indicator data 1240 of FIG. 12 comprises the two least significant bits of a blue color value in each pixel. These two bits can define up to four depth planes. However, the light field video data may include fewer than four depth planes. For example, in the previous embodiment, the light field video data includes only three depth planes. In such cases where the video data includes fewer than the maximum number of definable depth planes, the depth plane indicator data can specify that one or more depth planes are inactive. For example, continuing with the previous embodiment, if the two blue LSB bits in a pixel are set to 0b11, the pixel can be assigned to the inactive fourth depth plane D3. As shown in FIG. 15, only one of the three RxGxBx color sequences is activated in the output sequence, and the inactive depth planes are shown as black frames. As before, control data can be provided to specify the order in which the depth planes are displayed. As shown in the center panel 1520 of FIG. 15, in the illustrated embodiment, the control data specifies that the inactive depth plane D3 is shown first and last in the sequence. Thus, only the center frame in the sequence comprises the actual image data that is flashed to the display. (Other sequences can also be used. For example, the active depth plane can be ordered first or last in the sequence, or repeated more than once in the sequence.) If the display controller finds that a pixel is assigned to an inactive depth plane, the display controller can simply ignore the pixel and not flash it to the display. For example, when the control data indicates that a depth plane is inactive, power to a light source that provides light to the display for that particular depth plane can be reduced (e.g., shut off), thereby reducing power consumption. This can save switching power in the display driver.Thus, a power saving mode can be implemented by designating one or more depth planes of the video data as inactive. Similarly, in some embodiments, the control data can indicate that one or more color fields are inactive in a depth plane while one or more other color fields in the depth plane are active. Based on this control data, the display controller can control the display to ignore the inactive one or more color fields and display images from one or more active color fields without the inactive color fields. For example, when the control data indicates that a color field is inactive, power to a light source that provides light to the display for that particular color field can be reduced (e.g., shut off), thereby reducing power consumption. Thus, light sources such as light emitting diodes (LEDs), lasers, etc. that provide illumination to the display can be shut off or have their power reduced for the inactive depth planes and / or the inactive color fields. (Multiple Depth Plane Image Compression)

[0175] In some embodiments, image compression techniques are applied across multiple depth planes to reduce the amount of video image data by removing information redundancy between depth planes. For example, rather than transmitting an entire frame of image data for each depth plane, some or all of the depth planes may instead be represented in terms of changes relative to adjacent depth planes. (This can also be done on a temporal basis between frames at adjacent instants.) The compression techniques may be lossless or lossy, such that changes between adjacent depth plane frames, or between temporally adjacent frames that are less than a given threshold, may be ignored, thus resulting in a reduction in image information. In addition, the compression algorithm may use motion vectors to encode the motion of an object within a single depth plane (XY motion) and / or between depth planes (Z motion). Rather than requiring image data of a moving object to be repeatedly transmitted over time, the motion of the object may be achieved fully or partially using pixel shift control information, as discussed herein. (Dynamically configurable display drawing area)

[0176] In systems that display light field images, achieving high video frame rates can be difficult due to the relatively large amount of information contained per video frame (e.g., multiple depth planes, each with multiple color components). However, video frame rates can be improved, particularly in augmented reality modes, by recognizing that the computer-generated light field image may occupy only a portion of the display at a time, as shown in FIG.

[0177] FIG. 16 illustrates an exemplary drawing area for a frame of a computer-generated image in an augmented reality system. FIG. 16 is similar to FIG. 1, except that it shows only a portion of the display in which the augmented reality image is drawn. In this case, the augmented reality image includes a robot figure 1110 and a bumblebee character 2. The remaining areas of the display in the augmented reality mode may simply be a view of the real-world environment surrounding the user. Thus, it may not be necessary to draw computer-generated images in these areas of the display. In many cases, it may be the case that a computer-generated image occupies only a relatively small portion of the display area at a time. Video frame rates can be improved by dynamically reconfiguring the specific drawing area that is refreshed every frame to exclude areas where the computer-generated image does not need to be shown.

[0178] A computer-generated augmented reality image may be represented as a number of pixels, each having, for example, an associated brightness and color. A frame of video data may comprise an m×x array of such pixels, where m represents the number of rows and n represents the number of columns. In some embodiments, the display of the augmented reality system is at least partially transparent so as to be able to provide a view of the user's real-world surroundings in addition to showing the computer-generated image. If the brightness of a given pixel in the computer-generated image is set to zero or a relatively low value, the viewer will see the real-world environment at that pixel location. Alternatively, if the brightness of a given pixel is set to a higher value, the viewer will see the computer-generated image at that pixel location. For any given frame of the augmented reality image, the brightness of many of the pixels may be below a specified threshold value such that they do not need to be shown on the display. Rather than refreshing the display for every pixel below these thresholds, the display can be dynamically configured to not refresh these pixels.

[0179] In some embodiments, the augmented reality system includes a display controller for controlling the display. The controller can dynamically configure a drawing area for the display. For example, the controller can dynamically configure which of the pixels in a frame of video data are refreshed during any given refresh cycle. In some embodiments, the controller can receive computer-generated image data corresponding to a first frame of the video. As discussed herein, the computer-generated image may include several depth planes. Based on the image data for the first frame of the video, the controller can dynamically determine which of the display pixels to refresh for each depth plane. For example, if the display utilizes scanning display technology, the controller can dynamically adjust the scanning pattern to omit areas where the augmented reality image does not need to be refreshed (e.g., areas of the frame where there is no augmented reality image or where the brightness of the augmented reality image is below a specified threshold).

[0180] In this manner, based on each frame of video data received, the controller can identify sub-portions of the display where the augmented reality image should be shown. Each such sub-portion may include a single contiguous area or multiple non-contiguous areas (as shown in FIG. 16) on the display. Such sub-portions of the display can be determined for each depth plane in the light field image data. The display controller can then only cause the display to refresh the identified sub-portions of the display for that particular frame of the video. This process can be performed for each frame of the video. In some embodiments, the controller dynamically adjusts the area of ​​the display that will be refreshed at the beginning of each frame of video data.

[0181] If the controller determines that the area of ​​the display that needs to be refreshed is becoming smaller over time, the controller may increase the video frame rate since less time will be required to render each frame of augmented reality data. Alternatively, if the controller determines that the area of ​​the display that needs to be refreshed is becoming larger over time, the video frame rate may be decreased to allow sufficient time to render each frame of augmented reality data. The change in video frame rate may be inversely proportional to the percentage of the display that needs to be filled with the image. For example, the controller may increase the frame rate by a factor of ten if only one-tenth of the display needs to be filled.

[0182] Such video frame rate adjustments may be made on a frame-by-frame basis. Alternatively, such video frame rate adjustments may be made at prescribed time intervals or when a subportion of the display being refreshed increases or decreases in size by a prescribed amount. In some cases, depending on the particular display technology, the controller may also adjust the resolution of the augmented reality image shown on the display. For example, if the size of the augmented reality image on the display is relatively small, the controller may cause the image to be displayed at an increased resolution. Conversely, if the size of the augmented reality image on the display is relatively large, the controller may cause the image to be displayed at a reduced resolution. (Enhanced Head Pose Estimation)

[0183] As discussed herein, virtual and augmented reality systems can include body mounted displays such as helmets, glasses, goggles, etc. In addition, virtual and augmented reality systems can include sensors such as gyroscopes, accelerometers, etc. that make measurements that can be used to estimate and track the position, orientation, velocity, and / or acceleration of the user's head in three dimensions. The sensors can be provided in an inertial measurement unit worn by the user on his / her head. In this manner, the user's head pose can be estimated. The head pose estimate can be used as a means to allow the user to interact with the virtual or augmented reality scene. For example, if the user turns or tilts his / her head, the virtual or augmented reality scene can be adjusted in a corresponding manner (e.g., the view of the scene can be shifted or tilted).

[0184] FIG. 17 illustrates diagrammatically possible movements of a user's head about two axes of rotation. As illustrated, the user can rotate his / her head about a vertical axis and a horizontal axis perpendicular to the page. Although not illustrated, the user can also rotate his / her head about a horizontal axis that lies in the plane of the page. In some embodiments, it may be useful to define the direction of the user's gaze as the head pose direction. (Such a definition of head pose would not necessarily consider lateral tilt of the head, but may also consider other definitions of head pose.) FIG. 18 illustrates how a user's head pose can be mapped onto a three-dimensional surface 1810. FIG. 18 includes a surface normal vector 1820 that indicates the user's head pose. Each possible surface normal vector 1820 on the three-dimensional surface corresponds to a distinctly different head pose. In FIG. 18, a surface normal vector pointing straight up would correspond to the user's neutral head pose when the user is looking straight ahead.

[0185] Various algorithms can be used to estimate and track the user's head pose based on sensor measurements from a head-mounted inertial measurement unit. These include, for example, Kalman filters and other similar algorithms. These types of algorithms typically generate estimates that are based on sensor measurements over time, rather than just at any single instant in time. A Kalman filter, for example, includes a prediction phase in which the filter outputs a predicted estimate of the head pose based on the head pose estimate at a previous instant in time. Then, during an update phase, the filter updates the head pose estimate based on the current sensor measurements. Such algorithms can improve the accuracy of the head pose estimate, reducing errors in properly displaying virtual or augmented reality images in response to head movements. An accurate head pose estimate can also reduce the latency of the system.

[0186] Typically, a Kalman filter or similar algorithm produces the most accurate head pose estimates for head poses close to the user's neutral head pose (corresponding to the vertical surface normal vector 1820 in FIG. 18). Unfortunately, such algorithms do not consider movement limitations imposed by human physiology or the movement of the user's head relative to the body, and therefore may fail to properly estimate head pose movement as the head pose deviates further from the neutral head pose. However, various adaptations can be made to reduce the impact of these weaknesses on head pose tracking.

[0187] In some embodiments, head pose estimation and tracking using a Kalman filter or similar algorithm can be improved by using a variable gain factor that varies depending on the location of the current head pose within the limits of physiologically possible head poses. FIG. 18 illustrates a three-dimensional surface 1810 that corresponds to such limits of physiologically possible head poses. FIG. 18 shows that the user's head has a range of motion in any direction (e.g., left / right or up / down) of about 180° or less. The current head pose within the physiological limits can be used to adjust the variable gain factor estimated by the Kalman filter. In areas close to the center of the limits (i.e., neutral head pose), the gain factor can be set to emphasize predicted head poses over measured head poses, since the Kalman filter prediction error may be lower due to the higher linearity of head movement in this region. This can reduce the latency of the system without unduly affecting the accuracy of the head pose estimation. When the head pose approaches the limit boundaries of physiological head movement, then the algorithm can use a gain factor that is set to reduce the filter's reliance on predicted head poses or to emphasize measured head poses over predicted head poses in order to reduce error.

[0188] In some embodiments, each location on the physiological head pose limits illustrated in Figure 18 may correspond to a different gain. In other embodiments, the physiological head pose limits may be divided into separate regions and a different gain value may be associated with each of the different regions. This is illustrated in Figure 19.

[0189] FIG. 19 illustrates diagrammatically various head pose regions that may be used to define gain factors for improving head pose tracking. FIG. 19 shows a central region 1910 that corresponds to a relatively neutral head pose. It also includes an outer region 1930 that corresponds to a head pose close to physiological boundaries, and an intermediate region 1920 between the central and outer regions. In some embodiments, a different set of gain factors may be defined for each head pose region. The central region 1910 indicates an area with a higher linearity of movement that will have a higher accuracy of predictions produced by the Kalman filter algorithm. When the head pose is within the central region 1910, the gain factor of the Kalman filter may be set to emphasize the predicted head pose over the measured head pose, or otherwise reduce reliance on the measured head pose. When the head pose exits the central region and enters the intermediate or outer regions (1920, 1930, respectively), the movement may be further constrained by physiological factors that would adversely affect the Kalman predicted head pose if not taken into account by the algorithm. Thus, in these regions (especially the outer region 1930), the Kalman filter gain values ​​can be set to reduce the filter's reliance on predicted head poses and increase its reliance on measured head poses. For example, if it is known that head acceleration will cease near a limit boundary, then it would be inaccurate to rely too heavily on predicted head poses too far into the future. Although three head pose regions are illustrated in Figure 19, a different number of head pose regions can be used in other embodiments.

[0190] In some embodiments, head pose estimation and tracking can also be improved by sensing the position, orientation, velocity, and / or acceleration of the user's head relative to the user's body, rather than sensing head movement in an absolute sense. This can be done by providing an additional inertial measurement unit that is worn by the user on his or her body (e.g., on the torso or waist). It is important to note that head pose is a function of head and body movements. The limits of physiologically possible head poses are not fixed in space, but move, for example, with body rotation. If the user moved his or her head while keeping the body stationary and was sitting in a chair, the physiological limits would be relatively constrained such that a very good head pose estimate could be achieved by considering only the head movements. However, when the user is actually wearing a virtual or augmented reality head mounted display and moving around, then the physiological limits of possible head poses vary with body movements.

[0191] A second inertial measurement unit worn on the body (e.g., on board with a battery pack and / or processor for a virtual or augmented reality system) can help provide additional information to track movement of the physiological limits of head pose. Instead of fixing the limits in space, the second inertial measurement unit can allow head movement to be determined with respect to the body. For example, if the body rotates to the right, the physiological limits can be correspondingly rotated to the right to more accurately determine head pose within the physiological limits and avoid overly constraining the operation of the Kalman filter.

[0192] In some embodiments, head motion determined using a head-mounted inertial measurement unit can be subtracted from body motion determined using a body-mounted inertial measurement unit. For example, the absolute position, orientation, velocity, and / or acceleration of the body can be subtracted from the absolute position, orientation, velocity, and / or acceleration of the head to estimate the position, orientation, velocity, and / or acceleration of the head relative to the body. Once the orientation or motion of the head relative to the body is known, then the actual head pose location within physiological limits can be more accurately estimated. As discussed herein, this allows Kalman filter gain coefficients to be determined to improve head pose estimation and tracking. (Improved "Totem" localization)

[0193] In some virtual or augmented reality systems, a defined tangible object can be used as a "totem" that allows the user to interact with a virtual object or scene. For example, a tangible block that a user holds in his or her hand can be recognized by the system as an interactive device such as a computer mouse. The system can include, for example, a camera that tracks the movement of the tangible block in the user's hand and then adjusts the virtual pointer accordingly. A possible drawback of using computer vision to track a totem in space is that the totem may in some cases be outside the field of view of the camera or may be otherwise obscured. It would therefore be beneficial to provide a system for robustly tracking the position and movement of a totem in three dimensions using six degrees of freedom.

[0194] In some embodiments, the system for tracking the position and movement of the totem includes one or more sensors in the totem. These one or more sensors may be accelerometers and / or gyroscopes that independently determine the position and movement of the totem in space. This data can then be transmitted to a virtual or augmented reality system.

[0195] Alternatively, one or more sensors in the totem can be coupled with a transmitter to determine the totem and its position and movement in space. For example, the transmitter can generate a spatially varying electric and / or magnetic field in space, and the totem can include one or more sensors that repeatedly measure the electric or magnetic field at the totem's location, thereby allowing the totem's position and movement to be determined. In some embodiments, such a transmitter can be advantageously incorporated into a head-mounted display of a virtual or augmented reality system. Alternatively, the transmitter can be incorporated into a body-mounted puck. In this way, the totem's location and / or movement relative to the user's head or body, respectively, can be determined. This may be more useful information than if the transmitter were simply located in a fixed location (e.g., on a nearby table), since the totem's location and / or movement can be determined with respect to the user's head or body. (Adjusting image color based on ambient lighting)

[0196] In some embodiments, the virtual and augmented reality systems described herein include one or more sensors (e.g., cameras) to detect the brightness and / or hue of the ambient lighting. Such sensors can be included, for example, in the display helmet of the virtual or augmented reality system. The sensed information about the ambient lighting can then be used to adjust the brightness or hue of the generated pixels for the virtual object. For example, if the ambient lighting has a yellowish tint, the computer-generated virtual object can be modified to have a yellowish tint that more closely matches the color tone of the real object in the room. Such pixel adjustments can be made when the image is rendered by the GPU. Alternatively and / or in addition, such pixel adjustments can be made after rendering by using the control information discussed herein.

[0197] For purposes of summarizing the disclosure, certain aspects, advantages, and features of the present invention are described herein. It should be understood that not all such advantages may be achieved in accordance with any particular embodiment of the present invention. Thus, the present invention may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein, without necessarily achieving other advantages as may be taught or suggested herein.

[0198] The embodiments are described in conjunction with the accompanying drawings. However, it should be understood that the figures are not drawn to scale. Distances, angles, etc. are merely illustrative and do not necessarily bear precise relationship to the actual dimensions and layout of the devices shown. In addition, the foregoing embodiments are described with a level of detail that enables one skilled in the art to produce and use the devices, systems, methods, etc. described herein. A wide variety of modifications are possible. Components, elements, and / or steps may be altered, added, removed, or rearranged.

[0199] The devices and methods described herein can advantageously be implemented, at least in part, using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. A software module can comprise computer executable code stored in a computer's memory to perform the functions described herein. In some embodiments, the computer executable code is executed by one or more general-purpose computers. However, those skilled in the art will understand in light of this disclosure that any module that can be implemented using software executed on a general-purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such modules can be implemented entirely in hardware using a combination of integrated circuits. Alternatively, or in addition, such modules can be implemented completely or in part not by a general-purpose computer, but using a specialized computer designed to perform the specific functions described herein. Additionally, where methods are described that are or may be implemented at least in part by computer software, it will be understood that such methods may be provided on a non-transitory computer-readable medium (e.g., an optical disk such as a CD or DVD, a hard disk drive, a flash memory, a diskette, or the like) which, when read by a computer or other processing device, causes the method to be implemented.

[0200] While certain embodiments are explicitly described, other embodiments will be apparent to those of skill in the art based on this disclosure.

Claims

1. A virtual or augmented reality display system, comprising: a first sensor configured to provide a head pose measurement of a head of a user of the display system; A processor communicatively coupled to the first sensor, the processor comprising: receiving a plurality of head pose measurements produced by the first sensor, the plurality of head pose measurements being of the head poses at different times during a period of time; calculating a predicted head pose of the user based on the plurality of head pose measurements; determining an estimated head pose of the user at a particular time based at least in part on at least one of the plurality of head pose measurements and the predicted head pose, wherein determining the estimated head pose includes combining the head pose measurements and the predicted head pose at the particular time using one or more gain factors, the one or more gain factors indicating a relative emphasis of the head pose measurements and the predicted head pose in the combining; a processor and wherein the physiological range of motion of the head in at least two dimensions is divided into a plurality of non-overlapping regions, each of the plurality of non-overlapping regions being associated with one or more different gain factors; The one or more gain factors used in the combining are determined as the one or more gain factors associated with a particular region in which the head pose measurement falls at the particular time.

2. The system described in claim 1, wherein the first sensor is configured to be head-mounted.

3. The system described in claim 2, wherein the first sensor comprises an inertial measurement unit.

4. The system of claim 1, wherein the one or more gain factors emphasize the predicted head posture over the head posture measurement when the user's head posture is in a central portion of the physiological range of motion.

5. The system of claim 1, wherein the one or more gain factors emphasize the predicted head pose over the head pose measurement when the user's head pose is closer to the middle of the user's physiological range of motion than to the limits of the user's physiological range of motion.

6. The system of claim 1, wherein the one or more gain factors emphasize the head posture measurement over the predicted head posture when the user's head posture approaches the limits of the physiological range of motion.

7. The system of claim 1, wherein the one or more gain factors emphasize the head pose measurement value over the predicted head pose when the user's head pose is closer to the limit of the physiological range of motion than to the middle of the physiological range of motion.

8. The system of claim 1, wherein the first sensor is configured to be head-mounted, the system further comprising a second sensor configured to be body-mounted, and the at least one head pose measurement is determined based on measurements from both the first sensor and the second sensor.

9. The system of claim 8, wherein the head posture measurement is determined based on a difference between measurements from the first sensor and the second sensor.

10. A method for estimating head pose in a virtual or augmented reality display system, the method being performed by at least one processor, the method comprising: receiving a plurality of head pose measurements of a head of a user of the display system from a first sensor, the plurality of head pose measurements being of the head pose at different times during a period of time; calculating a predicted head pose of the user based on the plurality of head pose measurements; determining an estimated head pose of the user at a particular time based at least in part on at least one of the plurality of head pose measurements and the predicted head pose; and determining the estimated head pose comprises combining the head pose measurements and the predicted head pose at the particular time using one or more gain factors, the one or more gain factors indicating a relative emphasis of the head pose measurements and the predicted head pose in the combining; a physiological range of motion of the head in at least two dimensions is divided into a plurality of non-overlapping regions, each of the plurality of non-overlapping regions being associated with one or more different gain factors; A method according to claim 1, wherein the one or more gain factors used in said combining are determined as the one or more gain factors associated with the particular region in which the head pose measurement falls at the particular time.

11. The first sensor is configured to be head-mounted, and the method further comprises: receiving a body orientation measurement from a second sensor configured to be body mounted; estimating a head pose of the user based on the at least one head pose measurement and based on the at least one calculated predicted head pose; 11. The method of claim 10, further comprising: wherein the at least one head pose measurement is determined based on measurements from both the first sensor and the second sensor.

12. The method described in claim 10, wherein the one or more gain factors emphasize the predicted head posture over the head posture measurement when the user's head posture is in a central portion of the physiological range of motion.

13. The method described in claim 10, wherein the one or more gain factors emphasize the predicted head pose over the head pose measurement when the user's head pose is closer to the center of the user's physiological range of motion than to the limits of the user's physiological range of motion.

14. The method of claim 10, wherein the one or more gain factors emphasize the head pose measurement over the predicted head pose when the user's head pose approaches the limits of the physiological range of motion.

15. The method described in claim 10, wherein the one or more gain factors emphasize the head pose measurement value over the predicted head pose when the user's head pose is closer to the limit of the physiological range of motion than to the middle of the physiological range of motion.

Citation Information

Patent Citations

  • Image generation apparatus and image generation method

    JP2015095045A

  • Motion modeling in visual tracking

    US20150228118A1