Steerable foveal display

Abstract

To provide a display system comprising a foveal display having a monocular field of view.SOLUTION: A display system comprising a foveal display having a monocular field of view of at least 1 degree is positioned within a scannable field of view of at least 20 degrees, the foveal display being positioned for a user. In one embodiment, the foveal display is positioned for the fovea of a user.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Provisional Application No. 62 / 477,404, filed March 27, 2017, and U.S. Provisional Application No. 62 / 575,354, filed October 20, 2017, and incorporates both applications in their entireties. [Background technology]

[0002] Near-eye displays have competing requirements to display images at high resolution over a large field of view (FOV). For many applications in virtual and augmented reality, the field of view must be greater than 90 degrees, and ideally the binocular field of view would be greater than 180 degrees. At the same time, the resolution of the display must match the resolution of the human visual system, so that little to no pixelation is perceived in the virtual image. Combining these two requirements in one system poses several challenges. To avoid pixelation, the resolution needs to be on the order of 0.01-0.02 degrees per pixel. For a 90-degree square field of view, this corresponds to more than 4.5kx4.5k pixels per eye. Achieving such a resolution is difficult at the level of the panel, drive electronics, and rendering pipeline.

[0003] Additionally, optical systems that can project a wide FOV image to a user with sufficiently high resolution across the entire field of view are also difficult to design. System architectures that can provide users with high-resolution images across a wide field of view while simultaneously reducing rendering, data rates, and panel requirements would enable new applications for augmented and virtual reality systems. [Brief description of the drawings]

[0004] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numbers refer to similar elements and in which:

[0005] [Figure 1A] FIG. 1 shows an eye with the fovea centralis shown.

[0006] [Figure 1B] FIG. 1 is a diagram showing the visual field range of the eye.

[0007] [Figure 1C] FIG. 1 is a diagram showing the relationship between distance from the fovea and visual acuity.

[0008] [Figure 1D] FIG. 1 illustrates an exemplary vertical field of view.

[0009] [Figure 1E] FIG. 1 illustrates an exemplary horizontal field of view.

[0010] [Figure 1F] FIG. 2 illustrates an eye with a first exemplary gaze vector.

[0011] [Figure 1G] FIG. 1C illustrates the position of the steerable foveated display for the first exemplary gaze vector shown in FIG. 1F.

[0012] [Figure 1H] FIG. 1 illustrates an eye with a second exemplary gaze vector.

[0013] [Figure 1I] FIG. 1C illustrates the position of the steerable foveated display relative to the second exemplary gaze vector shown in FIG. 1H.

[0014] [Figure 1J] FIG. 1 illustrates an embodiment of a binocular display showing a foveated display and a field display for each eye.

[0015] [Figure 1K]FIG. 1 illustrates an embodiment of a binocular display showing a foveated display and a shared field display for each eye.

[0016] [Diagram 2] FIG. 1 is a block diagram of one embodiment of the present system.

[0017] [Diagram 3] FIG. 1 illustrates one embodiment of movement of a high-resolution region over time in a moveable foveated display.

[0018] [Figure 4A] 1 is a flow chart of one embodiment of utilizing a display. [Figure 4B] 1 is a flow chart of one embodiment of utilizing a display.

[0019] [Figure 5A] FIG. 1 illustrates an embodiment of a hybrid display.

[0020] [Figure 5B] FIG. 1 illustrates an embodiment of a display using roll-off magnification.

[0021] [Figure 6] FIG. 1 illustrates an embodiment of a hybrid display.

[0022] [Figure 7] FIG. 1 illustrates an embodiment of a hybrid display.

[0023] [Figure 8] FIG. 1 illustrates an embodiment of a hybrid display.

[0024] [Figure 9] FIG. 1 illustrates an embodiment of a hybrid display.

[0025] [Figure 10] FIG. 1 illustrates an embodiment of a hybrid display.

[0026] [Figure 11] FIG. 1 illustrates an embodiment of a hybrid display.

[0027] [Figure 12] FIG. 1 illustrates an embodiment of a hybrid display using time division multiplexing.

[0028] [Figure 13A] FIG. 1 illustrates an embodiment of a hybrid display using time division multiplexing. [Figure 13B] FIG. 1 illustrates an embodiment of a hybrid display using time division multiplexing.

[0029] [Figure 14] 14A and 14B are diagrams of one embodiment of a waveguide-based foveated display that can be used in the systems described above.

[0030] [Figure 15] 15A and 15B are diagrams of one embodiment of a waveguide-based field display that can be used in the systems described above.

[0031] [Figure 16A] FIG. 2 is a diagram of another embodiment of a hybrid display system.

[0032] [Figure 16B] 1 is an illustration of another embodiment of a hybrid display system.

[0033] [Figure 17] 1 is a flow chart of one embodiment of using a foveated display in conjunction with an external display.

[0034] [Figure 18]1 is a flow chart of an embodiment using a foveated display without an associated larger display.

[0035] [Figure 19] 1 is a flow chart of one embodiment of edge blending for a foveated display.

[0036] [Figure 20] 1 is a flow chart of one embodiment of using eye movement classification.

[0037] [Figure 21] 1 is a table showing exemplary types of eye movements.

[0038] [Figure 22] 1 is a flow chart of one embodiment of smart positioning.

[0039] [Figure 23] FIG. 1 is a block diagram of one embodiment of a computer system that can be used with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] Detailed Description This application discloses an operable foveated display device, referred to as a foveal display. The foveal display in one embodiment is positioned to provide a high resolution region where the user's fovea is located. The "fovea" is a small depression in the retina of the eye where visual acuity is highest. FIG. 1A illustrates an eye, showing the retina and fovea. The center of visual field is focused in this region where the cones are particularly concentrated. The center of the fovea is the region of the retina with the greatest resolution, but has a field of view of about 2 degrees. FIG. 1B shows the region of visual acuity from the foveal region with the greatest resolution to the far peripheral region with the least resolution. The resolution of the eye drops by almost an order of magnitude more than 20 degrees away from the center of the fovea. FIG. 1C shows the drop in sensitivity (Snellen ratio) based on distance from the center of the fovea (eccentricity).

[0041] In one embodiment, the system takes advantage of this by providing a steerable foveated display oriented to be aligned with the center of the user's field of view or another calculated location. In one embodiment, the field display provides a lower resolution field display image over a larger field of view. This means that the user perceives the image with peripheral vision and line of sight. In one embodiment, the system provides a high resolution image using a foveated display oriented primarily to the center of the field of view of the user's eye, and utilizes a second field display to provide a field display image over a wide field of view. This means that the user perceives the image with peripheral vision and line of sight. In one embodiment, the system uses a high pixel density display per eye to provide a high resolution image over a small field of view and a low resolution image over a large field of view, filling both eyes and the peripheral regions. In one embodiment, the resolution of the foveated display is between 0.2 and 3 arc minutes per pixel. In one embodiment, the resolution of the field display is between 1 and 20 arc minutes per pixel. In one embodiment, the field display and the foveated display can be combined into a single variable pixel display. In one embodiment, the system uses a variable pixel density display for each eye to present a high resolution image over a small field of view to the foveal region of each eye and a lower resolution image over a larger field of view to fill the binocular and peripheral regions. In one embodiment, the variable pixel density display can be a standard display processed at a variable density.

[0042] Such a system provides the perception of a high-resolution image with a wide field of view, while requiring only a fraction of the pixel count or processing power of a conventional near-eye display with comparably high perceived resolution. In one embodiment, such a system also significantly reduces the power consumption of the rendering system by reducing the number of pixels rendered.

[0043] The system may include three or more displays in one embodiment. In one embodiment, there may be three levels of resolution covering the foveal region, the binocular overlap region, and the peripheral region of each eye. In one embodiment, video images for multiple displays and resolutions may be aggregated together. In another embodiment, video images for multiple displays and resolutions may be separate.

[0044] Figure ID shows an exemplary vertical field of view, showing a 55 degree focal or comfort zone, as well as a peripheral region. The symbol recognition area is approximately 30 degrees vertically.

[0045] Figure 1E shows an exemplary horizontal field of view, showing a 60 degree focal region, a 30 degree symbol recognition zone, and a peripheral vision region, and a total binocular range of 135 degrees. Beyond that, there are monocular ranges (for the right and left eye) and a transient range that is only visible when the user moves their eyes.

[0046] In one embodiment, the steerable foveated display is positioned within a 30 degree vertical and horizontal symbol recognition region, in another embodiment, the steerable foveated display is positioned within a 55 degree vertical and 60 degree horizontal focal region / comfort zone region.

[0047] 1F and 1G show the field of view of a foveated display for one eye. In one embodiment, the foveated display 110 is positioned to be centered about a gaze vector 105. The gaze vector defines the center of the eye's field of view.

[0048] In one embodiment, the field of view of the foveated display 110 is a monocular field of view with a minimum field of view of 1 degree and a maximum field of view of 20 degrees. The field of view of the field display 120 in one embodiment has a monocular field of view of 40 degrees, up to the full monocular range. The full monocular range of view is typically considered to be 60 degrees towards the nose, 107 degrees from the nose, 70 degrees above horizontal, and 80 degrees below horizontal.

[0049] In one embodiment, the field display 120 may provide image data outside the range of the foveal display 110. FIG. 1F shows a top view illustrating the eye and the field of view of the foveal display 110 centered on the gaze vector 105. FIG. 1G shows a front view illustrating an exemplary location of the field of view of the foveal display 110. In one embodiment, the foveal display 110 has a total scannable field of view 160 between 20 and 160 degrees and can be positioned within that range. As discussed above, the foveal display 110 has a monocular field of view of at least 1 degree. In one embodiment, the foveal display fovea 110 has a monocular field of view of 10 degrees and the total scannable field of view 160 of the foveal display is 60 degrees. This allows the foveal display 110 to be positioned in the correct position.

[0050] 1H and 1I show the field of view of a foveated display 110 positioned in different locations when the user is looking to the upper left. As can be seen in this configuration, the field of view of the foveated display 110 is displaced and the portions of the field of view of the field displays above and below the field of view of the foveated display 110 are not uniform. FIG. 1I shows an example positioning of the display's field of view from the front.

[0051] A system including a foveated display combined with a field display allows the perception of high-resolution images over a wide field of view with only a fraction of the pixel count and processing power of conventional near-eye displays. In one embodiment, such a system also significantly reduces the power consumption of the rendering system by reducing the number of pixels rendered.

[0052] In one embodiment, the system may include three or more displays per eye. In one embodiment, there may be three levels of resolution covering the foveal, binocular overlap, and peripheral regions of each eye. In another embodiment, the system may include only a steerable foveal display, with the field display provided by an external system. In another embodiment, the system may consist of only a steerable foveal display with no associated field display.

[0053] 1J shows an embodiment of a binocular display including a right eye foveated display field of view 110A and a left eye foveated display field of view 110B. There is also a field display with a larger field of view 120A and 120B for each of the right eye 150A and left eye 150B, respectively. In one embodiment, the fields of view 120A, 120B of the field displays extend at least through the region of focus.

[0054] FIG. 1K illustrates an embodiment of a binocular display including a right eye foveated display field of view 110A and a left eye foveated display field of view 110B. In this configuration, however, the field display 170 is a single display that spans the user's entire field of view. In one embodiment, the foveated display and the field display may be displays integrated into a wearable display such as goggles. In another embodiment, the foveated display may be part of a wearable device while the field display is a separate display such as a projector or screen.

[0055] FIG. 2 illustrates an embodiment of an exemplary optical system 210, 280 and associated processing system 238. In an embodiment, the processing system may be implemented in a computer system including a processor. In an embodiment, the processing system 238 may be part of a display system. In another embodiment, the processing system 238 may be remote. In an embodiment, the optical system 210, 280 may be implemented in a wearable system, such as a head mounted display. The foveal image is presented to the user's eyes through a right eye foveal display 220 and a left eye foveal display 230 that orient the foveal display. In an embodiment, the foveal displays 220, 230 direct the foveal display image primarily to the center of the field of view of the user's eyes. In another embodiment, the images may be directed to different locations, as described below.

[0056] A foveal image for the right eye is created using a first display element 222. In one embodiment, the display element is a digital micromirror device (DMD). In one embodiment, the display element 222 is a scanning micromirror device. In one embodiment, the display element 222 is a scanning fiber device. In one embodiment, the display element is an organic light emitting diode (OLED). In one embodiment, the display element 222 is a liquid crystal on silicon (LCOS) panel. In one embodiment, the display element 222 is a liquid crystal display (LCD) panel. In one embodiment, the display element 222 is a micro LED or micro light emitting diode (μLED) panel. In one embodiment, the display element is a scanned laser system. In one embodiment, the system is a hybrid system with an off-axis holographic optical element (HOE). In one embodiment, the system includes a waveguide. In one embodiment, the waveguide is a multi-layer waveguide. In one embodiment, the display element can include a combination of such elements. The display elements are described in more detail in Figures 5-16 below.

[0057] In one embodiment, the first display element 222 is located in a near-eye device such as glasses or goggles.

[0058] The focus and field of view of the foveated display is set using intermediate optics 224, which may include, but are not limited to, lenses, mirrors, and diffractive optical elements. In one embodiment, the focus of the virtual image is set at infinity. In another embodiment, the focus of the virtual image is set short of infinity. In one embodiment, the focus of the virtual image can be changed. In one embodiment, the virtual image can have two or more focal lengths that are perceived simultaneously.

[0059] In one embodiment, the foveal display image is directed primarily at the center of the user's eye's field of view. In one embodiment, the field of view (FOV) of the foveal display image is greater than 1 degree. In one embodiment, the FOV of the foveal display image is between 1 degree and 20 degrees. In one embodiment, the foveal display image may be 6 degrees or greater, which addresses inaccuracies in eye tracking, provides the necessary area for successful blending so that the user cannot perceive the blend, and accounts for the time of foveal display repositioning for various types of eye movements.

[0060] In one embodiment, the system further includes a low resolution field display image having a field of view of 20 to 220 degrees.

[0061] In one embodiment, the foveated display image is projected directly to the user's eye using a set of one or more fully or partially transparent positioning elements 226. In one embodiment, the positioning elements 226 include steerable mirrors. In one embodiment, the positioning elements 226 include curved mirrors. In one embodiment, the positioning elements 226 include Fresnel reflectors. In one embodiment, the positioning elements 226 include diffractive elements. In one embodiment, the diffractive elements are surface relief gratings. In one embodiment, the diffractive elements are volume holograms. In one embodiment, the display 220 includes a focus adjuster 223 that allows the display to display image elements at multiple focal distances within the same frame. In one embodiment, the focus adjuster 223 may be an optical path length extender, as described in U.S. Patent Application Serial No. 15 / 236,101, filed August 12, 2016.

[0062] A similar set of elements is present in the left-eye foveated display 230. In one embodiment, the right-eye foveated display 220 and the left-eye foveated display 230 are coincident. In another embodiment, they may include different elements.

[0063] In one embodiment, the eye tracking device 240 tracks the user's gaze vector, such as where the eyes are looking. In one embodiment, the eye tracking system is a camera-based eye tracking system 240. In one embodiment, the eye tracking system 240 is an infrared scanning laser with a receiving sensor. Other eye tracking mechanisms may be used. The fovea position calculator 245 determines the center of the user's field of view based on data from the eye tracking system 240.

[0064] In one embodiment, the foveated display 220, 230 is adjusted using an adjustable positioning element 226, 236 to position the foveated image primarily toward the center of the field of view of the user's eye. In one embodiment, the image direction is adjusted by changing the angle of a mirror, which is one of the position elements 226, 236. In one embodiment, an electromagnetic force is used to change the mirror angle. In one embodiment, an electrostatic force is used to change the mirror angle. In one embodiment, a piezoelectric force is used to change the mirror angle. In one embodiment, the adjustable element is the image source, or display element 222, 232, which is moved to position the image. In one embodiment, the foveated image is positioned to be directed toward the center of the field of view of the user's eye. In another embodiment, another position element 226, 236, such as a manipulation element 226, 236, can be changed.

[0065] The field display 280 communicates with the processing system 238 via communication logic 270, 290. In one embodiment, there may be multiple displays. Two field displays are shown here: field display 285 and peripheral display 288. Other levels of resolution may be displayed. In one embodiment, the field display 280 may include a single field display 285 that is viewed by both eyes of the user, or one field display per eye. In one embodiment, the field display 280 may have variable resolution.

[0066] In one embodiment, if the field display 280 is a separate system, a synchronization signal generator 292 is used to synchronize the display of the separate foveated display 210 with the display of the field display 280. In one embodiment, the synchronization signal generator 292 is used to synchronize an adjustable mirror, or other positioning element of the foveated display, with the field display, thereby synchronizing the displays. In one embodiment, the field display 280 includes a blender system 294 for blending edges of the foveated display image with edges of the field display image to ensure a smooth transition.

[0067] In one embodiment, the low resolution field display image is presented to the user with a fully or partially transparent optical system. In one embodiment, the partially transparent system includes a waveguide optical system. In one embodiment, the partially transparent system includes a partial mirror, which may be flat or have refractive power. In one embodiment, the partially transparent system includes a diffractive optical element. In one embodiment, the image is presented to the user via a direct view optical system. In one embodiment, the partially transparent system includes inclusions for reflecting or scattering light.

[0068] In one embodiment of the field display 280, an additional display subsystem is used to display images in the region of the peripheral field of view 288 when viewing monocularly. In one embodiment, this subsystem is an LED array. In one embodiment, this subsystem is an OLED array. In one embodiment, this display subsystem uses a scanning laser. In one embodiment, this subsystem uses an LCD panel. In one embodiment, this subsystem does not have intermediate optical elements to manipulate the FOV or focus of the image. In one embodiment, this subsystem has intermediate optical elements. In one embodiment, these intermediate optical elements include a microlens array.

[0069] Image data displayed by the operative foveated display 210 and the field display 280 is generated by a processing system 238. In one embodiment, the system includes an eye-tracking device 240. In one embodiment, the eye-tracking device 240 tracks the user's gaze vector, such as where the eyes are looking. In one embodiment, the eye-tracking system is a camera-based eye-tracking system 240. Alternatively, the eye-tracking system 240 may be based on an infrared laser. A foveal position calculator 245 determines the center of the user's field of view based on data from the eye-tracking system 240.

[0070] The processing system 238 in one embodiment further includes a foveal position verifier 247 that verifies the positioning of the positioning elements 226, 236 to ensure that the displays 220, 230 are properly positioned. In one embodiment, this involves re-evaluating the foveal display position relative to the center of the user's eye's field of view to account for movement of the foveal display. In one embodiment, the foveal position verifier 247 uses a sensing mechanism to provide feedback that confirms that the positioning element has reached its target position. In one embodiment, the sensing mechanism may be a camera. In one embodiment, the sensing mechanism may be a gear. The sensing mechanism may be another type of sensor that can determine the position of the optical element. In one embodiment, if the actual position of the foveal display is not the target position, the foveal position verifier 247 may modify the display to provide correct image data, as described in more detail below.

[0071] In one embodiment, the eye movement classifier 260 can be used to predict where the user's gaze vector will move. This data can be used by the predictive positioner 265 to move the foveated displays 220, 230 based on the next location of the user's gaze vector. In one embodiment, the smart positioner 267 can utilize user data such as eye movement classification and gaze tracking to predictively position the displays 220, 230. In one embodiment, the smart positioner 267 can further use data about future data in the displayed frame to identify optimal positioning of the displays 220, 230. In one embodiment, the smart positioner 267 can position the displays 220, 230 in a location not indicated by the gaze vector. For example, when the displayed frame data has less relevant data (e.g., a butterfly illuminated on a dark screen) or when the intent of the frame is to show the viewer a specific location.

[0072] The processing system 238 may further include cutout logic 250. The cutout logic 250 determines the location of the foveal displays 220, 230 and provides display information along with the cutout to an associated field display 280. The field display 280 renders this data to generate a low-resolution field display image that includes a cutout of the corresponding portion of the image in the field display. This ensures that there is no interference between the foveal image and the field image. In one embodiment, if there is a cutout, the blender logic 255 blends the edges of the cutout with the foveal image to ensure a smooth transition. In another embodiment, the foveal display may be used to display sprites or brighter elements overlaid on top of the lower resolution field image. In such a case, neither the cutout logic 250 nor the blender logic 255 is required. In one embodiment, the cutout logic 250 and the blender logic 255 may be selectively activated as needed.

[0073] In one embodiment, the system may synchronize the foveated display 210 with a separate field display 280. In this case, in one embodiment, the displays are synchronized by synchronization logic 272. In one embodiment, the separate field display 280 is synchronized with an adjustable mirror or other positioning element of the foveated display 210. This synchronizes the displays. The field display 280 may receive positioning data. In one embodiment, there may be no cutout in this case.

[0074] In one embodiment, the processing system 238 can include an optical distortion system 275 for a foveated display 210 in which distortion increases from the center to the edge of the image. This intentional distortion increases the perceived size of pixels moving from the center to the edge of the foveated image. This change in perceived resolution reduces the amount of processing required because fewer pixels are needed to cover the same angular area of ​​the foveated display image.

[0075] 5B shows an example of a distorted image with decreasing resolution as the angle from the optical axis increases. Optical distortion may aid in blending between the foveated display 210 and the field display 280. In another embodiment, the foveated display 210 including the optical distortion system 275 can be used without a field display. This also simplifies the optical design and saves processing for blending.

[0076] In one embodiment, the variable resolution highly distorted image has a large ratio between the center and the edges. The total FOV of this display will be large (up to 180 degrees).

[0077] In one embodiment, the roll-off logic 277 provides a roll-off at the edges of the display. The roll-off in one embodiment can include a resolution roll-off (reducing the resolution towards the edges of the display area). In one embodiment, this can be implemented by magnification with the optical distortion system 275. In one embodiment, the roll-off can include a brightness and / or contrast roll-off (reducing the brightness and / or contrast towards the edges). Such a roll-off is designed to reduce the sharpness of the edges of the display. In one embodiment, the roll-off can be designed to roll off to "nothing", in which case it gradually decreases from the highest brightness / contrast to gray or black or an ambient color. In one embodiment, the roll-off logic 277 can be used by the foveated display 210 if there is no associated field display. In one embodiment, the roll-off logic 297 can be part of the field display 280 if the system has a field display.

[0078] FIG. 3 shows one embodiment of the movement of the foveal image over time as the user's eyes move. In both cases, there is a small zone where the foveal image is displayed. The 5 degree display position (in this example) provides high resolution and is focused in the center of the user's field of view. The low resolution field image provides a large field of view. However, because the eye's relative resolution outside the foveal region is low, the user perceives this combined image, which includes a high resolution small foveal image and a low resolution large field image, as high resolution across a wide field of view.

[0079] 4A is a flow chart of one embodiment utilizing a foveated display. The process begins at block 410. In one embodiment, before the process begins, the display system is adapted to the user. This initial setup includes determining the interpupillary distance (IPD) and any necessary treatments to ensure the user's "baseline" view is accurate.

[0080] At block 415, the user's gaze is tracked. In one embodiment, an IR camera is used to track the gaze. In one embodiment, the gaze tracking identifies the user's gaze vector, such as where the user is focusing. The gaze tracking can identify the left and right gaze vectors / angles, and gaze center (derived from the left / right gaze vectors). The gaze tracking can determine the position (X, Y, Z) and orientation (roll, pitch, yaw) of the left and right eyes relative to a baseline frame of reference. In one embodiment, the baseline frame of reference is established when the display is initially adapted to the user and the user's interpupillary distance, diopter, and other relevant data are established.

[0081] At block 420, the location of the fovea is determined based on the gaze vector data. In one embodiment, the fovea location includes the coordinates (X, Y, Z) and orientation (roll, pitch, yaw) of each eye.

[0082] At block 425, the process determines whether the foveal display should be repositioned. This is based on a comparison of the current position of the foveal display with the user's gaze vector or the intended position of the foveal image. If they are misaligned, the system determines that the foveal display needs to be repositioned. If so, at block 430, the display is repositioned. In one embodiment, if the foveal display moves beyond a certain distance, the display is turned off during the movement. This ensures that the user does not perceive movement. In one embodiment, the certain distance is greater than 0.5 degrees. In one embodiment, the foveal display is not turned off if movement occurs while the user is blinking. Note that although the term "repositioning" is used, this does not generally imply that there is a physical movement of the eyepiece. In one embodiment, a mirror or other optical element that positions the display is used to change the positioning of the center of the foveal image. The process then proceeds to block 435, regardless of whether the display has been repositioned.

[0083] At block 435, the system optionally crops out the portion of the field display that is co-located with the foveal display. This prevents the field display from interfering with the foveal display. In one embodiment, the cutout is performed in the rendering engine. In another embodiment, the foveal image may be a sprite or other bright image element where the cutout does not need to be clear. In that case, this block is skipped. In one embodiment, the cutout is skipped if the user's eye tracking indicates that the user's gaze has moved significantly from a baseline reference. The baseline reference is the user's default gaze position from which the gaze movement is tracked. A significant movement from the baseline reference will prevent the system from determining the user's correct gaze position. In this case, in one embodiment, the foveal image can be dropped or the foveal display can be turned off momentarily.

[0084] At block 440, in one embodiment, the edges between the foveal and field images are blended. This ensures a smooth and imperceptible transition between the field and foveal images. At block 445, a hybrid image incorporating the foveal and field displays is displayed to the user. The process then returns to block 410 to continue tracking and display. Note that while the description refers to foveal and field images, contemplated images include sequential images of a video.

[0085] 4B illustrates one embodiment of corrective actions that may be taken if display position verification indicates that the actual position of the foveated display does not match the intended position. The process begins at block 450.

[0086] At block 452, positioning of the foveated display is initiated. In one embodiment, this corresponds to block 430 of FIG. 4A. Returning to FIG. 4B, at block 454, the actual position of the foveated display is confirmed. In one embodiment, one or more sensors are used to determine the position and orientation of the foveated display. In one embodiment, the sensors may include cameras, mechanical elements that detect the position of an adjustable mirror, or other positioning elements, etc.

[0087] At block 456, the process determines whether the foveal display is correctly positioned. If so, the foveal display is at the calculated location and the foveal image is displayed in the appropriate location for the user. If the foveal display is correctly positioned, the image is displayed at block 464. In one embodiment, this includes displaying a hybrid image that includes the foveal image at the calculated location and its associated field display image. The process then ends at block 475.

[0088] If, at block 456, the process determines that the foveated display is not correctly positioned, the process proceeds to block 458.

[0089] At block 458, the process determines whether there is enough time to reposition the foveal display. This determination is based on the distance that needs to be moved, the speed of movement, and the time until the next image is sent by the processing system. In one embodiment, it also depends on the user's eye movement. In one embodiment, the system preferentially moves the foveal display when the image is not perceived while the user is blinking. In one embodiment, the repositioning occurs within the blanking period of the display. For example, moving only once along one coordinate takes less time than moving the foveal display in a large, three-dimensional manner. If there is enough time, the process returns to block 452 to reposition the foveal display. If not, the process proceeds to block 460.

[0090] At block 460, the process determines whether the actual position of the foveated display is within range of the intended position. In one embodiment, "within range" in this context means that the system can adjust the display for the difference. If it is within range, the process proceeds to block 462.

[0091] At block 462, the foveated image is adjusted for rendering at the actual location and the image is displayed at block 464. For example, in one embodiment, if the location difference is very small, the original calculated foveated image may be rendered in the wrong place without causing visual artifacts. In another embodiment, the foveated image may be adjusted to properly render at the actual location. For example, the foveated image may be cropped, brightened, distorted, contrast adjusted, color coordinate (white point) adjusted, cropped, or shifted laterally to account for the location difference. In one embodiment, the radial location of the edge blending may be shifted or changed. In one embodiment, the system may over-render, rendering a 5.5 degree visual image for a 5 degree foveated display, allowing for a 0.5 degree shift without requiring re-rendering.

[0092] If the foveal display is not in range, then in one embodiment, the frame data is sent to the field display for rendering at block 466. At block 468, in one embodiment, the foveal image is not displayed. In one embodiment, the frame is dropped. In another embodiment, the foveal display is momentarily turned off. In one embodiment, the foveal display is not considered to be in range if the user's eye tracking indicates that the user's gaze has moved too far from the baseline reference.

[0093] The field display image is rendered without image cutout and without displaying or rendering the foveated image at block 470. The field display image is displayed at block 472. The process then ends.

[0094] 5A shows one embodiment of a display including a foveated display subsystem 510 and a field display subsystem 550. The foveated display subsystem 510, in one embodiment, includes a display panel 515 or another image source, and an intermediate optical element 520. The output of the intermediate optical element 520 is directed to an adjustable mirror 525 or other element that provides positioning. The adjustable mirror 525 directs the image to a partial mirror 530 and a curved partial mirror 535, which direct the image to a user's eye 590. In one embodiment, the adjustable mirror 525 is configured with an OPTOTUNE TM 5. The field display subsystem 550 in one embodiment includes a projection subsystem 555 and a light guide 560. Alternate embodiments may use a different projection method for the field display subsystem 550. In one embodiment, the field display subsystem 550 may be replaced with a tunable prism, such as a tunable prism TP-12-16 from IEEE, in which one face of the prism is moved to adjust the angle. In one embodiment, the adjustable mirror 525 may be replaced with an acousto-optic modulator and a mirror. In one embodiment, each of these elements may be replaced with a similar element, which allows selective movement of the high resolution display to be directed to center the field of view of the user's eye 590. The field display subsystem 550 in one embodiment includes a projection subsystem 555 and a light guide 560. Alternate embodiments may utilize different projection methods for the field display subsystem 550.

[0095] FIG. 5B illustrates one embodiment of a roll-off that can be used to blend the foveal image with the field image. In one embodiment, the system resolution roll-off involves magnifying the edges of the display to display lower resolution data outside the foveal region. This also increases the field of view. Magnification can be provided in a variety of ways using hardware, software, or a combination. FIG. 16B illustrates an example display 580 showing the distribution of pixel density as the resolution rolls off. Pixels seen in the center are of uniform size (shown by the center polygon 585). Towards the edge of the display region, pixel sizes become larger and distorted. This can be seen in the left polygon 595. As the distance between pixel edges increases both horizontally and vertically, in one embodiment, pixels removed horizontally and vertically from the center region become more distorted and larger, as seen in the lower polygon 1680. Note that FIG. 5B illustrates a relatively small display, and the ratio between the center polygon 585 and the corner polygon 595 can range from greater than 1 to less than or equal to 10.

[0096] Figure 6 shows another embodiment of a display that includes a foveal subsystem 610 and a field display subsystem 650. In addition to these two subsystems, the embodiment of Figure 6 includes a peripheral vision display 670. The peripheral vision display in one embodiment is an OLED display.

[0097] 7 shows another embodiment of a display that includes a foveated display subsystem 710 and a field display subsystem 750. The field display subsystem in this embodiment is an OLED 760 with a microlens array.

[0098] FIG. 8 illustrates another embodiment of a display including a foveated display subsystem 810 and an optional field display subsystem 850. In this embodiment, the foveated display subsystem 810 is implemented in glasses or goggles and can be worn by a user. The optional field display subsystem 850 in one embodiment can be a display screen such as a TV monitor 860. The field display subsystem 850 can be a modular element that is optionally attached to the glasses or goggles. In one embodiment, the system can provide high resolution images only through the foveated display subsystem 810. If the optional field display subsystem 850 is available to the user, a rendering system (not shown) can communicate with the foveated display subsystem 810, and the field display subsystem 850 provides a wider field of view. In one embodiment, in this configuration, the foveated display subsystem can provide a field of view of up to 20 degrees.

[0099] FIG. 9 illustrates another embodiment of a display including a foveated display subsystem 910 and a field display subsystem 950. In this embodiment, the foveated display subsystem 910 includes a light guide 930 with a FoV of 40-55 degrees and is coupled to a projector 920 that functions as a display panel, such as an OLED microdisplay. In one embodiment, instead of transmitting the entire 40-55 degree image, the display panel 920 transmits only a small image associated with the area that covers the foveal region of the user's field of view. The remainder of the light guide 930 outside the spot is transparent. Outside the foveal region can be filled with a lower resolution field display 950, such as an OLED display 960.

[0100] 10 illustrates another embodiment of a display including a foveated display subsystem 1010 and a field display subsystem 1050. In this embodiment, the foveated display subsystem 1010 includes a display panel 1015, an intermediate optical element 1020, and an adjustable mirror 1025 that directs light off-axis to a holographic optical element (HOE) 1030. The HOE 1030 directs light from the display 1015 to the user's eye. The movement provided by the adjustable mirror 1025 allows for correct positioning of the foveated display subsystem 1010. In one embodiment, the field display subsystem 1050 includes a projection subsystem 1055 and a light guide 1060.

[0101] 11 illustrates another embodiment of a display including a foveated display subsystem 1110 and a field display subsystem 1150. In this embodiment, the foveated display subsystem 1110 includes a display panel 1115, an intermediate optical element 1120, and an adjustable mirror 1125 that directs light to a prism with an embedded partial mirror 1130. Light from the embedded partial mirror 1130 in the prism is reflected by a curved partial mirror 1140 to the user's eye. The movement provided by the adjustable mirror 1125 allows for correct positioning of the foveated display subsystem 1110. In one embodiment, the field display subsystem 1150 includes a projection subsystem 1155 and a light guide 1160.

[0102] Figure 12 shows another embodiment of a display that provides spatially multiplexed high and low resolution displays. In the embodiment of Figure 12, light is provided by a single display panel 1210. Two separate images are displayed by the single display panel 1210: a foveal display portion and a field display portion. The foveal display portion passes through a foveal display intermediate optic 1220, an adjustable mirror 1230, a partial mirror 1240, and a curved partial mirror 1245. In an embodiment, the mirrors 1240, 1245 can be replaced with another mechanism for redirecting light.

[0103] In one embodiment, the field display image portion from the single display panel 1210 goes to the field display intermediate optics option 1250, which passes it to the light guide 1260. This allows the single display panel 1210 to use spatial multiplexing to provide data for both the foveated display and the field display. In one embodiment, the relative sizes of the images on the display panel 1210 are not the same for the foveated display portion and the field display portion. In one embodiment, the display sizes are the same, but the field display intermediate optics 1250 magnifies the portion of the image that is used as the field display.

[0104] 13A and 13B show one embodiment of a time-multiplexed display including a foveated image and a low-resolution field display image. The system utilizes a single display panel 1310 in conjunction with a color or polarization selective mirror 1325 that selectively transmits data through (in the case of foveated image data) or reflects it to intermediate optics 1355 of the field display. The display panel 1310 displays the foveated image data and the low-resolution field display data in a time-multiplexed manner, e.g., alternating frames at a rate sufficient to create two sets of images for human perception.

[0105] 13A shows the optical path of a foveated image frame. The data passes through foveated display intermediate optics 1320 and then through color or polarization sensitive mirror 1325. It is reflected by adjustable mirror 1330. In one embodiment, partial mirror 1340 and curved partial mirror 1345 are used to direct the image to the user's eye. In one embodiment, an additional foveated display intermediate optics 1320 may be placed after color or polarization selective mirror 1325. Alternative configurations for directing the image can be used.

[0106] FIG. 13B shows the optical path of the field display image data. Image data from the single panel display 1310 passes through the foveated display intermediate optics 1320 before being reflected by a color or polarization selective mirror 1325 towards the field display intermediate optics. In one embodiment, one or more redirecting mirrors 1350 can be used to direct the light. From the field display intermediate optics 1355, the light passes through a light guide 1360. The output then passes through a curved partial mirror 1345 and a partial mirror 1340 to the user's eye.

[0107] By rapidly switching the display between the foveal image and the field display image, the system displays two images in a time-multiplexed manner, allowing both to be perceived simultaneously by the user.

[0108] 14A and 14B show one embodiment of a foveated display subsystem using a waveguide. This configuration of a foveated display subsystem may be used in one embodiment with any of the above embodiments. In one embodiment, the foveated image utilizes a display panel 1410. The output from the display panel 1410 passes through an optical element 1420. Although the optical element 1420 is shown as a single lens, one skilled in the art will understand that any intermediate optical element may be included as the optical element 1420. The output of the optical element 1420 is sent to a manipulation element 1430, which directs it to an input coupler 1440 of the light guide. The manipulation element 1430 directs the light to the appropriate portion of the input coupler 1440 of the light guide. The image data then passes through the light guide 1450 and is output through the output coupler 1460 of the light guide to the user's eye. The manipulation element 1430 directs the light correctly to a foveated image aligned with the position of the user's eye.

[0109] Figures 15A and 15B show one embodiment of a field display image using a multi-layer light guide. This stacked waveguide can be used in the configuration described above for the field display. In this example, there are two waveguides, one for each portion of the field of view. In another embodiment, there may be four stacked waveguides.

[0110] Output from the display panel 1510 passes through the optical element 1520. Although the optical element 1520 is shown as a single lens, one skilled in the art will understand that any intermediate optical element may be included as the optical element 1520. Output from the optical element 1520 is sent to the light guide input couplers 1540, 1545. In one embodiment, the optical element 1520 splits the data from the display panel 1510 based on color or polarization and directs it to one of the light guide input couplers 1540, 1545. In this example, the top light guide 1550 is used for a first viewing portion of the image and the bottom light guide 1555 is used for a second viewing portion of the image. Output from the foveal light guides 1550, 1555 is directed to the user's eye by the light guide output couplers 1560, 1565.

[0111] Figure 16A shows another embodiment of a display including a foveated display subsystem 1610 and a field display subsystem 1640. This configuration is similar to that described above with respect to Figure 5, but instead of using an adjustable mirror, a movable display panel 1615 is used to position the foveated display relative to the user's eyes. This configuration of the movable elements of the foveated display subsystem can be utilized in the systems described above, replacing the adjustable mirror.

[0112] FIG. 16B shows another embodiment of a display including a foveated display subsystem 1650 and a field display subsystem 1690. This configuration is similar to that described above with respect to FIG. 5, but instead of using an adjustable mirror, a tunable prism 1665 is used to position the foveated display relative to the user's eye. In this embodiment, one face of the tunable prism is moved to adjust the angle to position the foveated image. The tunable prism is a foveated display subsystem 1690 that is similar to the foveated display subsystem 1650 described above with respect to FIG. TMThe tunable prism TP-12-16 of the present invention may be used in the systems described above for the movable element of the foveated display subsystem. This configuration for the movable element of the foveated display subsystem may be used in the systems described above, replacing the adjustable mirror with the tunable prism 1665. In another embodiment, the adjustable mirror 525 may be replaced with an acousto-optic modulator and mirror. This configuration for the movable element of the foveated display subsystem may be used in the systems described above, replacing the adjustable mirror.

[0113] It should be noted that the configurations shown in Figures 5-16A show optical elements and specific layouts, however, the design does not mandate those specific layouts and other optical elements can be used in the system. Additionally, elements can be mixed and matched between configurations.

[0114] 17 is a flow chart of an embodiment of using a foveated display with an external display. An external display is a display that is not controlled by the same system as the foveated display. For example, the external display may be a projection system, such as in a virtual reality (VR) CAVE or another environment that provides a field display. In one embodiment, a user may wear an augmented reality (AR) or virtual reality (VR) headset, which provides a hybrid display with interaction with the environment, where the AR / VR headset provides a foveated display in addition to the field display provided by the other system.

[0115] The process begins at block 1710. At block 1715, a handshake is performed between the foveated display system and the external display system. In one embodiment, the handshake establishes that both systems can cooperate to provide a combined display. In one embodiment, the handshake includes setting up a connection between the foveated display system and the field display system.

[0116] The synchronization data from the external display system is set at block 1720. In one embodiment, this synchronization signal provides frame data since the foveal system is designed to be fully synchronized with the external system.

[0117] The position of the foveated display is determined at block 1725. As described above, this determination may be based on the user's gaze vector, a predicted gaze, or smart positioning based on data in the displayed frame.

[0118] At block 1730, the process determines whether the foveated display should be repositioned to display at the selected location, and if so, at block 1735, positioning is triggered.

[0119] At block 1750, the foveated display is overlaid and augmented with the external display. In one embodiment, the external display is separate and therefore does not include cutout logic. In another embodiment, there may be cutout logic that prevents the system from rendering portions of the low resolution image from where the foveated display image is shown.

[0120] At block 1760, blurring is applied to blend edges between the foveal display image and the field display image. At block 1770, a hybrid image including the foveal image and the field image is displayed to the user. In this manner, the user can experience enhanced display quality when entering a VR CAVE or other display environment with a wide field of view but a field display. The process then loops back to block 1720 to continue the process until the video or other display is finished.

[0121] 18 is a flow chart of an embodiment of using a foveal display without an associated field display. In this case, the system provides only a foveal display and not the field display described above. However, in one embodiment, the foveal display may apply blending or magnification to expand the field of view.

[0122] At block 1820, the process determines the location of the foveated display based on user data or other data. User data includes gaze vectors, predicted gaze vectors, etc. External data may include information about the image data being displayed.

[0123] At block 1830, the process determines whether the foveal display should be repositioned. Because the user's line of sight does not change, there is no need to reposition the display for multiple frames. If the position needs to be changed, at block 1840, the foveal display is adjusted. In one embodiment, the adjustment may include an operable eyebox to correct eye position. In one embodiment, the adjustment may include shifting the position of the display relative to a foveal region of the user's field of view. In one embodiment, the foveal display is turned off during the movement if the movement is greater than a certain distance. In one embodiment, the distance is greater than 0.5 degrees. In one embodiment, the foveal display may not be turned off if the user blinks during the movement.

[0124] At block 1850, the foveated display is presented in an appropriate position for the user.

[0125] At block 1860, in one embodiment, a roll-off is provided at the edges of the display. In one embodiment, the roll-off includes a resolution roll-off (reducing the resolution towards the edges of the display area). In one embodiment, the roll-off includes a brightness and / or contrast roll-off (reducing the brightness and / or contrast towards the edges). Such a roll-off is designed to reduce the abruptness of the edges of the display. In one embodiment, the roll-off can be designed to roll off to "nothing", where there is a gradual decrease from maximum brightness / contrast to gray or black or an ambient color.

[0126] In one embodiment, the resolution roll-off involves enlarging the pixel size at the edge of the foveal display to better blend with the lower resolution field display image outside the foveal region. This also increases the field of view. The magnification can be provided in a variety of ways using hardware, software, or a combination. Figure 5B shows an example display showing the distribution of pixel density as the resolution rolls off.

[0127] At block 1870, an appropriate gaze angle-based correction is applied to the image. As the gaze vector varies from straight on, the overall field of view becomes distorted. The gaze angle-based correction utilizes the known gaze angle used for positioning to correct the distortion in software. The process then returns to block 1820. In this manner, a steerable foveated display can be used to provide a steerable foveated display that follows the user's gaze or other cues. In one embodiment, the foveated display can provide a variable field of view.

[0128] 19 is a flow chart of one embodiment of blending edges of a foveated display. The process begins at block 1910. As described above, when the foveated display is positioned with the field display, the edges between the displays are blended. This gives the user a continuous impression. In one embodiment, this process corresponds to block 440 of FIG. 4 and block 1760 of FIG. 17.

[0129] At block 1920, the process identifies edges of the foveated image. In one embodiment, the edges are defined by the field of view available to the foveated display. In another embodiment, the foveated display may display a field of view that is smaller than the maximum it can display.

[0130] The process determines and applies the best blending technique at block 1930. In one embodiment, the blending techniques may include blending using an alpha mask, dither blending, interlaced pixels, color-based alpha channel blending, pixel-based alpha channel blending, multi-sample anti-aliasing (MSAA), temporal filtering blending, and / or other blending techniques.

[0131] At block 1950, the process determines whether another technique should be applied. If so, at block 1960, the next technique is selected and the process returns to block 1940. If not, the process ends at block 1970. As mentioned above, in one embodiment, this process is called for each frame where a high resolution foveated display image and a low resolution field display image are overlaid. In one embodiment, blending may not be applied when the foveated display shows sprites or other image elements overlaid on a background.

[0132] FIG. 20 is a flow chart of an embodiment using eye movement classification. Eye movement classification is used to predict the future position of the user's eyes to position a foveated display. The process begins at block 2010. At block 2015, the location of the foveal region of the user's visual field is identified 2015. At block 2020, the user's eye movements are classified. FIG. 21 illustrates some exemplary eye movements that may be identified. Eye movements include fixations, blinks, microsaccades, slow pursuits, and fast movements / saccades. In one embodiment, in addition to eye movements, head position may be used to classify eye movements and make predictions. These types of eye movements are known in the art.

[0133] At block 2025, the process determines an appropriate response to the eye movement. The response may include changing the display position, changing the field of view, changing the resolution, changing the depth data (which may depend on the 3D gaze vector), changing the convergence point. This determination may be based on a prediction of the future position of the user's gaze vector based on a classification of the eye movement.

[0134] At block 2030, the process determines whether the foveated display should be modified. If so, the foveated display is modified at block 2035. As discussed above, modifications may include changes in position, field of view, resolution, etc.

[0135] At block 2040, the process determines whether the field display should be altered based on the analysis. If so, the field display is altered at block 2045. In one embodiment, the field display may be altered by changing the resolution, depth data, convergence point, etc. In one embodiment, the field display is not steerable, although other alterations may be made.

[0136] At block 2050, edge blurring is applied between the foveal display and field display images. At block 2060, the hybrid image is displayed to the user. The process then returns to block 2015 to continue processing the next image. Note that in one embodiment, this process occurs very quickly, so an evaluation is performed on each frame before it is displayed.

[0137] 22 is a flow chart of one embodiment of smart positioning. The process begins at block 2210. This process can be used if the system is designed to utilize positioning other than just based on the user's gaze vector.

[0138] At block 2215, the user's eyes are tracked. In one embodiment, the user's head movement may also be tracked. This helps predict the user's eye movements based on the vestibulo-ocular reflex. Head movement and eye movement may be combined to determine the position and orientation of each eye.

[0139] At block 2220, external data is received. This external data may include highlighted elements that need to be displayed in high resolution using the foveal display, the location where the user's eyes are directed, or another external element. In one embodiment, the foveal display may be directed to relevant elements that are not in the user's gaze vector. For example, if there is only one element of interest on a dark screen, a high resolution foveal display is best deployed to the element of interest. As another example, if most of the screen is intentionally blurred, but there are parts with text or other detailed content, this may be the place to deploy the foveal display. Other reasons for positioning the display may be brought forward.

[0140] At block 2225, an optimal placement and configuration of the foveated display is determined based on external data and user data. As discussed above, the user data includes the user's eye and head positions. In one embodiment, the external data is user independent and reflects information about the frame being displayed. In one embodiment, unless there is external data to retarget the foveated display, the default configuration is to position it at the center of the user's fovea. However, based on external information, this may be changed for a particular frame and content.

[0141] At block 2230, the process determines whether the foveated display should be modified. The modification may be a change in position, resolution, focal length, etc. If so, at block 2235, the display is modified.

[0142] At block 2240, the process determines whether the field display portion should be modified. The modification may be a change in resolution, brightness, contrast, etc. If so, at block 2245, the display is modified.

[0143] At block 2250, the edges between the foveal display and field display images are blended, and the combined image is displayed at block 2255. The process then returns to block 2215.

[0144] While the above process is shown in flow chart form, those skilled in the art will appreciate that this is done for simplicity. The order of the various elements need not remain the same unless there is a dependency between the elements. For example, adjustments to the foveal display and the field display can be performed in any order. Tracking of the user's eyes and head can be performed continuously. The system can receive external data when it is available, rather than continuously or at specific times in the process. Other such adjustments to the flow chart are within the scope of the present invention.

[0145] 23 is a block diagram of one embodiment of a computer system that can be used with the present invention, however, it will be apparent to one skilled in the art that other alternatives of various system architectures can also be used.

[0146] The data processing system shown in Figure 23 comprises a bus or other internal communication means 2340 for communicating information, and a processing unit 2310 coupled to the bus 2340 for processing information. The processing unit 2310 may be a central processing unit (CPU), a digital signal processor (DSP), or another type of processing unit 2310.

[0147] The system, in one embodiment, further comprises a random access memory (RAM) or other volatile storage device 2320 (referred to as memory), coupled to the bus 2340, for storing information and instructions to be executed by the processor 2310. The main memory 2320 may be used for storing temporary variables or other intermediate information during execution of instructions by the processing unit 2310.

[0148] The system also includes a read only memory (ROM) 2350 and / or static storage device 2350, in one embodiment, coupled to the bus 2340 for storing static information and instructions for the processor 2310. In one embodiment, the system also includes a data storage device 2330, such as a magnetic or optical disk and its corresponding disk drive, flash memory or other storage capable of storing data when power is not applied to the system. The data storage device 2330 in one embodiment is coupled to the bus 2340 and stores information and instructions.

[0149] The system may further be coupled to output device(s) 2370, such as a cathode ray tube (CRT) or liquid crystal display (LCD), coupled to bus 2340 via bus 2360 for outputting information. The output device(s) 2370 may be visual, audio, and / or tactile output devices (e.g., vibration).

[0150] An input device 2375 may be coupled to the bus 2360. The input device 2375 may be an alphanumeric input device, such as a keyboard including alphanumeric and other keys, for enabling a user to communicate information and command selections to the processing unit 2310. Other user input devices 2380 may also be included. One such user input device 2380 is a cursor control device 2380, such as a mouse, trackball, stylus, cursor direction keys, or touch screen, which is coupled to the bus 2340 via the bus 2360 and can communicate direction information and command selections to the processing unit 2310 and control movement on the display device 2370.

[0151] Another device that may optionally be coupled to computer system 2300 is a network device 2385 for accessing other nodes of a distributed system over a network. Communication device 2385 may include any of a number of commercially available networking peripheral devices, such as those used for coupling to the Internet or wide area networks, personal area networks, wireless networks, or other methods of accessing other devices. Communication device 2385 may also be a null modem connection, or any other mechanism that provides connectivity between computer system 2300 and the outside world.

[0152] It should be noted that any or all of the components of this system and associated hardware shown in FIG. 23 may be used in various embodiments of the present invention.

[0153] Those skilled in the art will appreciate that a particular machine embodying the present invention may be configured in a variety of ways according to the particular implementation. The control logic or software implementing the present invention may be stored in main memory 2320, mass storage device 2330, or other storage medium locally or remotely accessible to processor 2310.

[0154] It will be apparent to one skilled in the art that the systems, methods, and processes described herein may be implemented as software stored in the main memory 2320 or the read-only memory 2350 and executed by the processor 2310. This control logic or software may reside in an article of manufacture including a computer readable medium having computer readable program code embodied therein and readable by the mass storage device 2330 for causing the processor 2310 to operate according to the methods and teachings.

[0155] The invention may also be implemented in a handheld or portable device that includes a subset of the computer hardware components listed above. For example, a handheld device may be configured to include only the bus 2340, the processor 2310, and the memory 2350 and / or 2320.

[0156] The handheld device may be configured to include a set of buttons or input signal components that allow a user to select from a set of available options. These may be considered as input device #1 2375 or input device #2 2380. The handheld device may also be configured to include an output device 2370, such as a liquid crystal display (LCD) or display element matrix for displaying information to a user of the handheld device. Conventional methods may be used to implement such a handheld device. The implementation of the present invention on such devices will be apparent to one of ordinary skill in the art in view of the disclosure of the invention provided herein.

[0157] The present invention can also be implemented in special purpose equipment that includes a subset of the computer hardware components described above, such as a kiosk or vehicle. For example, the equipment can include a processing unit 2310, a data storage device 2330, a bus 2340, and a memory 2320, and no input / output mechanisms or only rudimentary communication mechanisms that allow a user to communicate with the device in a basic way, such as a small touch screen. In general, the more specialized the use of the device, the fewer elements the device needs to function. In some devices, communication with the user is via a touch-based screen or similar mechanism. In one embodiment, the device may not provide direct input / output signals, but can be configured and accessed from a website or other network-based connection via the network device 2385.

[0158] Those skilled in the art will appreciate that any configuration of a particular machine implemented as a computer system may be used according to a particular implementation. The control logic or software implementing the present invention may be stored on any machine-readable medium locally or remotely accessible to the processor 2310. A machine-readable medium has any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, machine-readable media include read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or other storage media that can be used for temporary or permanent data storage. In one embodiment, the control logic may be implemented as a transmittable data such as an electrical, optical, acoustic, or other form of propagated signal (e.g., carrier wave, infrared signal, digital signal).

[0159] In the foregoing specification the invention has been described with reference to specific exemplary embodiments thereof. It will, however, be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A hybrid display system, comprising: a field display having a monocular field of view of at least 40 degrees; a foveated display having a monocular field of view of at least one degree and movably positioned within a scannable field of view spanning at least 20 degrees, the foveated display image being positioned relative to a user within the field of view of the field display, thereby presenting a hybrid display including image data from each of the field display and the foveated display; and cutout logic for determining a position of the foveal display and providing position information to the field display; the field display rendering a cutout in the field display image corresponding to a location in the foveal display image; Hybrid display system.

2. The hybrid display system of claim 1 , wherein the foveated display is implemented in a foveated display of a wearable display.

3. 10. The hybrid display system of claim 1, further comprising a foveal position verifier to verify an actual position of the foveal display and adjust the foveal display image data if the actual position is not the intended position.

4. The hybrid display system of claim 1 , further comprising the foveated display being positioned based on a prediction of future eye positions.

5. An eye tracking device that tracks the position and orientation of a user's eyes; 4. The hybrid display system of claim 3, further comprising: an eye movement classifier for identifying an eye movement type, the eye movement type being used to provide a prediction for predictively positioning the foveated display.

6. 2. The hybrid display system of claim 1, wherein the blending includes one or more of an alpha mask, a dither blend, an interlaced pixel, a color-based alpha channel blend, a pixel-based alpha channel blend, multi-sample anti-aliasing (MSAA), and a temporal filtering blend.

7. The field display is an external display, The hybrid display system of claim 1 , further comprising synchronization logic for synchronizing displays between the foveal display and the field display.

8. The hybrid display system of claim 1 , further comprising a positioning element for the foveated display, enabling movement of the foveated display.

9. 9. The hybrid display system of claim 8, wherein the positionable element comprises one or more of an adjustable mirror, a tunable prism, an acousto-optic modulator, a tunable display panel, a curved mirror, a diffractive element, and a Fresnel reflector.

10. a field display having a monocular field of view of at least 30 degrees and generating a field display image; a foveated display having a monocular field of view of at least one degree and generating a foveated display image, the foveated display image being movably positioned within a scannable field of view of at least 20 degrees, the foveated display image being positioned relative to a user; and cutout logic for determining a position of the foveal display and cutting out a portion of a field display image from the field display at a position corresponding to the position of the foveal display image.

11. The display system of claim 10 , wherein the combination of the foveated display and the field display provides a wide field of view and a perceived high resolution.

12. The display system of claim 10 further comprising blending logic that blends edges where the foveal display image and the field display image overlap.

13. 13. The display system of claim 12, wherein the blending includes one or more of an alpha mask, a dither blend, an interlaced pixel, a color-based alpha channel blend, a pixel-based alpha channel blend, multi-sample anti-aliasing (MSAA), and a temporal filtering blend.

14. The display system of claim 10 , further comprising synchronization logic for synchronizing displays between the foveal display and an external display providing a field display.

15. 11. The display system of claim 10, further comprising an eye-tracking device for tracking a user's eye position and orientation, and an eye movement classifier for identifying eye movement types, the eye movement types being used to predictively position the foveated display.

16. The display system of claim 10 , further comprising a positioning element for the foveated display, enabling movement of the foveated display.

17. 17. The display system of claim 16, wherein the positioning element comprises one or more of an adjustable mirror, a tunable prism, an acousto-optic modulator, a tunable display panel, a curved mirror, a diffractive element, and a Fresnel reflector.

18. The display system of claim 10 , further comprising an optical distortion system for a foveated display that increases a size of pixels at the edges of the foveated display image compared to a center of the foveated display image.

19. 11. The display system of claim 10, further comprising roll-off logic that provides a roll-off that reduces luminance toward edges of the field display image.

20. The foveated display comprises: a positioning element adjustable relative to the foveal display for positioning the foveal display within the field of view of a user's eye; a display element that generates a foveated image after the adjustable positioning element positions the foveated display; and an intermediate optical element for generating a virtual image, the virtual image being directed towards the eye of the user by the adjustable positioning element; The display system of claim 1 , comprising:

Images