Apparatus and method for improving the performance of integral imaging based light field displays using time division multiplexing - Patents.com

Abstract

Light field display based on integral imaging using time division multiplexing.

Description

[Technical field]

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 158,707, filed March 9, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to light field displays, and more particularly, but not exclusively, to light field displays based on integral imaging using time division multiplexing. [Background technology]

[0003] Light field displays based on integral imaging (InI) offer a great opportunity to realize true 3D scenes with the correct focus cues to mitigate known vergence-accommodation conflicts. However, a major challenge that must be overcome is the trade-off between spatial and depth resolution. Increasing the depth resolution requires increasing the number of separate views, called the number of views, to render a 3D scene, but increasing the number of views often comes at the expense of the spatial resolution of the scene. In this disclosure, we describe the design of a time-multiplexed InI-based light field display in accordance with the present invention that can potentially increase the number of views, and therefore the depth resolution, while maintaining high spatial resolution.

[0004] Conventional stereoscopic displays enable the perception of a 3D scene through a pair of two-dimensional (2D) perspective images, one for each eye, with binocular disparity and other pictorial depth cues, but they generally lack the ability to render the correct retinal blur effect and stimulate the natural ocular accommodation response, which leads to the well-known vergence accommodation conflict (VAC) problem. Potential display methods that can render focus cues and overcome the VAC problem include stereoscopic displays, holographic displays, multifocal planar displays, Maxwellian vision displays, and light field displays. Among these methods, integral imaging-based (InI-based) light field displays can reconstruct a 3D scene by recreating directional light rays that appear to emanate from 3D points at different depths in the 3D scene, and thus render the correct focus cues similar to those of a natural viewing scene.

[0005] FIG. 1 shows the configuration of a typical InI-based head-mounted display (HMD) system, which consists of a microdisplay, a microlens array (MLA), and an eyepiece. The microdisplay displays a series of elemental images (EI) containing different perspective views of a 3D scene. Each lenslet of the MLA corresponds to an EI on the microdisplay and forms a conjugate image of the EI on a central depth plane (CDP) to create one directional sample of the reconstructed 3D scene. As used herein, the CDP refers to a plane optically conjugate to the plane of the microdisplay across the MLA. The reconstructed 3D scene is observed by an observer at a field of view window (also called the exit pupil of the eyepiece) through the eyepiece, which provides the appropriate depth information. A distinct feature of light-field 3D displays, in contrast to conventional 2D displays, is that multiple distinct elemental views rendering a point (e.g. P) of the 3D scene are observed by placing the pupil of the eye at the field of view window.

[0006] The accommodation state of the observer's eye plays an important role in the perceived image. For example, FIG. 1 shows the rendering of a 3D point O through three different pixels, O1, O2, and O3, on the corresponding elemental images. Imaged by three corresponding microlenses, the light beams from the corresponding points (pixels) on the different EIs converge to a point O, which is further projected to the pupil through the eyepiece. When the eyeball is accommodated at the depth of the reconstruction point O of the reconstructed 3D scene, the light beams from the corresponding points (pixels) on the different EIs converge to a focal image O' on the retina, as shown in FIG. 1. For reconstruction points at other depths (e.g., point P), the images of the individual pixels are spatially displaced from each other on the retina, resulting in retinal blurring. The level of said retinal blurring varies with the difference in reconstruction depth and the accommodation of the eye.

[0007] Such light-field rendering approaches have already been applied in HMD designs for immersive virtual reality (VR) and optical see-through augmented or mixed reality (AR / MR) applications. For example, Lanman and Luebke demonstrated a near-eye immersive light-field display by placing a microdisplay and a microlens array (MLA) in front of the viewer's eye. Hua and Javidi demonstrated an optical see-through LF-HMD system by combining a micro InI unit with a see-through freeform magnifying eyepiece (Figure 2A). More recently, Huang and Hua demonstrated an optical see-through LF-HMD system that provides a high spatial resolution of about 3 arcmin over a depth of field of more than 3 diopters (Figure 2B). Although such studies have successfully demonstrated the potential ability of LF-HMD systems to render focus cues and thus address the well-known VAC problem in conventional stereoscopic displays, none of the existing LF-HMD prototypes are able to provide a sufficiently high spatial resolution comparable to human vision using state-of-the-art microdisplay technology. The inventors recognize that a key challenge is that the spatial resolution of the reconstructed 3D scene must be compromised in order to achieve adequate field density and a reasonable eyebox. Summary of the Invention

[0008] In response to such unmet needs disclosed above, along with other considerations, the present disclosure describes an exemplary design of a time-multiplexed InI-based light field display that can increase the number of views and therefore the depth resolution while maintaining high spatial resolution. In one of its aspects, the present invention can incorporate a fast programmable switchable array (e.g., a shutter array or a switchable light source array, etc.) that operates in a time-multiplexed manner to synchronize the rendering of multiple elemental image sets on the display. In doing so, the exemplary apparatus and method of the present invention can rapidly switch between multiple elemental image sets that render a 3D scene from slightly different viewing angles. As a result, the number of views and the viewing density can be doubled without sacrificing the spatial resolution. In another of its aspects, the present invention can provide an apparatus and method that improves the spatial resolution without compromising the viewing density and eyebox size, and several such exemplary apparatus and methods have been implemented and experimentally verified as further disclosed herein. According to our calculations, by appropriately selecting the system parameters of the InI stem according to the present invention, a high spatial resolution system comparable to human vision can be achieved. A proof-of-concept system was constructed and validation of the proposed method was demonstrated.

[0009] Accordingly, in one aspect thereof, the invention provides a time-multiplexed integral imaging (InI) light field display comprising: a microdisplay including a plurality of pixels configured to render a set of elemental images, each elemental image providing a different perspective view of a 3D scene; a microlens array disposed in optical communication with the microdisplay at a selected distance from the microdisplay and configured to receive light from elemental images of the microdisplay, the microlens array having a central depth plane associated therewith that is optically conjugate with the microdisplay across the microlens array, the microlens array configured to receive bundles of light rays from the elemental images to create an integral image at a corresponding reconstruction point relative to the central depth plane to reconstruct a light field of the 3D scene; and a switchable array disposed in optical communication with the microlens array and configured to receive light transmitted by the microlens array and transmit the received light to the light field of the 3D scene. The switchable array may be configured to selectively direct light from selected ones of the element images to the central depth plane, and / or the microdisplay may be configured to synchronize the rendering of the element images on the microdisplay with switching of the switchable array to operate the microdisplay and the switchable array in a synchronized time division multiplexed manner. The microlens array may include an array of microlenses having the same focal length.

[0010] In further aspects, the switchable array may be disposed at a location between the microlens array and the central depth plane and / or may be disposed at a location between the microlens array and the microdisplay. The switchable array may include switchable elements that can be turned on to pass light rays from the microlens array or turned off to block the passage of light rays. The programmable switchable array may include a shutter array and / or a switchable light source array. The microdisplay may be self-emissive or transmissive and / or may include a spatial light modulator. An aperture size of each switchable element of the switchable array may be smaller than an aperture size of each lenslet of the microlens array, such that each lenslet covers a plurality of elements of the switchable array. The switchable array may include a plurality of pixelated elements, each of which is smaller than the aperture size of each switchable element. A barrier array may be disposed between the microdisplay and the microlens array in optical communication therewith.

[0011] Additionally, the present invention can include an eyepiece disposed at a distance z0 from the central depth plane for receiving light from the light field of the 3D scene. An aperture array may be disposed between the microdisplay and the microlens array in optical communication therewith. Let a denote the distance from the microdisplay to the aperture array, and d denote the diameter of the apertures in the aperture array. A This is expressed as:

[0012]

number

[0013] Here, p El is the size of an element image, g is the distance from the microdisplay to the microlens array, and p MLA is the MLA's pitch. [Brief description of the drawings]

[0014] The foregoing summary, as well as the following detailed description of illustrative embodiments of the invention, can be better understood when read in conjunction with the appended drawings. [Figure 1] FIG. 1 shows a schematic diagram of a head-mounted light field display based on integral imaging. [Diagram 2] 2A-2B show schematic optical layouts of a light field display based on optical see-through head-mounted integral imaging. [Figure 3A] FIG. 3A illustrates a schematic diagram of an exemplary time-multiplexed InI-based light field display (eg, 4x4 element field of view and four-phase time division multiplexing in accordance with the present invention). [Figure 3B] FIG. 3B shows a schematic layout of the 2D shutter array of FIG. 3A. [Figure 3C] FIG. 3C shows a schematic of the layout of a 2D viewing window rendered by the time division multiplexing scheme of FIG. 3A. [Figure 4] FIG. 4 shows a schematic diagram of the operating principle of a time-division multiplexed InI based light field display (eg 2x2EI and 4-phase time-division multiplexed in the figure) for a phase 1 state of the display cycle of the device of FIG. 3A. [Diagram 5] 5A-5D are schematic diagrams showing the state of components and footprints at important surfaces in Phase 1 of the present invention, where FIG. 5A shows a schematic diagram of an element image on a microdisplay, FIG. 5B shows a schematic diagram of the state of the shutter array, FIG. 5C shows a schematic diagram of an intermediate image on the central depth plane, and FIG. 5D shows a schematic diagram of the element field distribution in the field window. [Figure 6] FIG. 6 illustrates diagrammatically the state of the second phase of a display cycle in a four-phase time division multiplexed InI-based light field display according to the present invention. [Figure 7]7A-7D are schematic diagrams showing the state of components and footprints at important surfaces in Phase 2 of the present invention, where FIG. 7A shows a schematic diagram of an element image on a microdisplay, FIG. 7B shows a schematic diagram of the state of the shutter array, FIG. 7C shows a schematic diagram of an intermediate image on the central depth plane, and FIG. 7D shows a schematic diagram of the element field distribution in the field of view window. [Figure 8] Figure 8 shows a schematic layout of a conventional InI system, where 3D points are rendered (e.g., for a 2x2 elemental field of view in the figure) and reconstructed in the central depth plane (CDP), and depending on the shape and placement of the MLA, the elemental field distribution in the field of view window is also shown. [Figure 9] Figures 9A and 9B are schematic diagrams showing the layout of sub-openings of a shutter array composed of small pixel elements during time division multiplexing according to the present invention, where Figure 9A shows a schematic diagram of a shutter S1 set to on during phase 1, and Figure 9B shows a schematic diagram of a shutter S2 set to on during phase 1. [Figure 10] 10A-10B are schematic diagrams illustrating an exemplary time-multiplexed InI-based light field display for enhancing the viewing fill factor according to the present invention, where FIG. 10A illustrates a schematic diagram of a phase 1 state of the display cycle and FIG. 10B illustrates a schematic diagram of a phase 2 state of the display cycle. [Figure 11] FIG. 11 shows a schematic of a 2×2 field of view, four-phase, time division multiplexed InI system according to the present invention. [Figure 12] FIG. 12 illustrates a schematic diagram of an exemplary apparatus according to the present invention for mitigating crosstalk problems using an aperture array. [Figure 13] 13A-13B show schematic layouts of exemplary time division multiplexed InI display systems according to the present invention in which directional microdisplays are utilized (e.g., a 4x4 element field of view and four-phase time division multiplexing as shown), with FIG. 13A showing a schematic state of a first phase of the display cycle and FIG. 13B showing a schematic state of a second phase of the display cycle. [Figure 14]FIG. 14 is a schematic diagram illustrating the parameters of the directional backlighting scheme of FIGS. 13A-13B, which includes a microdisplay with a microlens array. [Figure 15] 15A-15B are schematic diagrams illustrating the fill factor by utilizing a light source array having multiple light source elements in a light source cell according to the present invention, where each light source cell includes 8x8 light source elements, and the overlap of illumination areas between different phases can make the fill factor greater than 1. [Figure 16] FIG. 16 shows a prototype constructed exemplary time-multiplexed InI-based light field system in accordance with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Referring now to the figures, where like elements are numbered like throughout, FIG. 3 is a circuit diagram of an exemplary time-multiplexed InI-based 3D light field display 100 according to the present invention, which can include a microdisplay 102, a microlens array (MLA) 104, a high-speed programmable switchable array such as a switchable shutter array (SA) 106, and an eyepiece optics 108. The microdisplay 102 can be a self-emissive display such as an organic light-emitting display (OLED) that emits light, or a spatial light modulator (SLM) such as a liquid crystal display (LCD) or digital mirror device (DMD) that modulates an illumination source. In the case of an SLM, the microdisplay 102 can function in either a transmissive or reflective mode by transmitting or reflecting its illumination source to create a 2D image pattern.

[0016] The microdisplay 102 can render a different set of elemental images (EI) 101, each of which provides a perspective view of a 3D scene. The microdisplay 102 can be positioned at a distance g away from the MLA 104. The MLA 104 can include an array of microlenses 105 with the same focal length. Each of the elemental images 101 rendered on the microdisplay 102 is imaged onto a central depth plane (CDP) 109 through a corresponding microlens of the MLA 104. Depending on the lateral magnification of the microlenses 105, conjugate images of the elemental images 101 may overlap on the CDP 109. The MLA 104 helps to generate directional sampling of a 3D light field. Bundles of rays from the elemental images 101 enter the corresponding microlenses 105 and are integrated at corresponding reconstruction points (e.g., point P) to reconstruct the light field of a 3D scene. By changing the perspective content of each EI 101, objects of different depths can be rendered. The switchable shutter array 106 can include an array of switchable elements that can be turned on to pass light rays from the microdisplay 102 or turned off to block light rays from passing through. (FIG. 3A) The shutter array 106 can be positioned adjacent to both sides of the MLA 104 (e.g., adjacent to the front and back). The gap between the shutter array 106 and the MLA 104 can be minimized to reduce the negative side effects associated with lossy compression. The shutter array 106 can provide rapid switching between different light ray paths (e.g., white hatched paths vs. gray hatched paths in FIG. 3A) through the MLA 104 such that different sets of EIs 101 can be rendered on the microdisplay 102 and imaged by the MLA 104 in a time-multiplexed manner in synchronization with the on and off states of selected aperture sets. The time-multiplexed sets of EIs 101 can effectively increase the number of perspective views rendered for a reconstructed 3D scene.For example, as shown in FIG. 3A, by simply time-multiplexing two sets of EI 101 and two states of the SA 106, the number of fields of view for a reconstruction point P is doubled.

[0017] The eyepiece 108 may be located at a distance Z0 from the CDP 109 and may magnify the reconstructed 3D scene formed by the integrated imaging unit (including the microdisplay 102 and the MLA 104) and image the reconstructed 3D scene into visual space. The eyepiece 108 may be provided in any suitable configuration, such as a conventional rotationally symmetric lens group, a singlet or doublet, or a monolithic freeform prism. The eyepiece 108 may project the ray bundles from the reconstructed 3D scene onto a field of view window 110 where an observer may place the pupil of his or her eye to observe the magnified virtual 3D scene. The trajectories of each ray bundle from an element field of view are conceptually illustrated by a small square in FIG. 3A. The actual shape of the ray bundles on the field of view window 110 depends mainly on the shape of the aperture of the microlens 105. For example, if the shape of the aperture of the microlens 105 is circular, the ray bundles will also be circular.

[0018] 3B shows a schematic layout of the 2D shutter array 106, and FIG. 3C shows a schematic layout of the 2D viewing window 110 rendered by a time division multiplexing scheme. The aperture size d SA is preferably the aperture size d of the lenslet 105 MLA, such that each lenslet 105 covers multiple elements of the shutter array 106. By turning on or off different shutter elements under each lenslet 105, different parts of the lenslet 105 are selected and ray bundles from pixels of different sets of EI 101 are imaged. For example, in Figs. 3A-3B, by turning on the white elements (S1) of the shutter array 106, the top half of each lenslet 105 is selected and ray bundles from pixels rendered by a first set of EI 101 (shown in solid lines in Fig. 3A) are imaged to reconstruct a part of the light field of the 3D scene (e.g., point P). Similarly, when the grey elements of the shutter array 106 are switched on, the bottom half of each lenslet 105 is selected and ray bundles from pixels rendered by a second set of EI 101 (shown in dashed lines and grey shading in Fig. 3A) are imaged to reconstruct a second part of the light field of the 3D scene (e.g., point P). The light beams rendered by the different sets of element images 101 through different parts of the lenslets 105 are projected to different positions on the field of view window 110, forming characteristic field of view entrance positions on the pupil of the eye of the field of view A. Depending on the ratio of the shutter size to the microlens array pitch, the proposed method of time division multiplexing can increase the field of view number and field of view density accordingly.

[0019] The shutter array 106 can be based on existing spatial light modulator (SLM) technology. However, the pixel structure of a typical SLM with a low pixel fill factor can induce severe diffraction effects. To minimize the diffraction effects due to the pixelated aperture structure, a pixel fill factor of 85% or more is required, but the fill factor of commercially available transmissive liquid crystal displays is far below this requirement. However, the fill factor and switching speed of some commercially available reflective spatial light modulators, such as Liquid Crystal on Silicon (LCoS) technology, can meet this requirement. However, because LCoS is reflective, relay optics are required to image it onto the MLA aperture plane, which significantly increases the system volume.

[0020] 4-7D further illustrate the working principle of an exemplary four-phase time-multiplexed InI-based light field display 100 according to the present invention, in which only 2x2 elemental images 101 are shown for illustration purposes. In this figure, the aperture size of the shutter array 106 may be half the lenslet 105 pitch, such that each lenslet 105 of the MLA 104 is divided into four sub-apertures, each corresponding to a phase of the four-phase time-division multiplexing cycle, and a corresponding set of EIs 101 is rendered for each sub-aperture or phase. FIG. 4 and FIG. 6 respectively show two different phases of a full display cycle. In each phase, only one set of the shutters (e.g., S1 in phase 1 and S2 in phase 2) is switched on to allow the light rays to pass through the corresponding sub-aperture of the lenslet 105. Meanwhile, one set of EIs 101 with a right perspective view corresponding to the opened shutter set is displayed on the microdisplay 102. For example, P 1,1 , P 1,2 , P 1,3 and P 1,4 5A, where P represents the pixels on four adjacent element images 101 of the EI 101 set 1 displayed on the microdisplay 102 to reconstruct a three-dimensional point P. These four points on the microdisplay 102 are imaged by the corresponding sub-apertures of the microlenslets 105 and are respectively represented as P' on the CDP 109. 1,1 , P' 1,2 , P' 1,3 , P' 1,44, 5C. Rays emitted from these four points pass through the corresponding open shutter sets and converge to a reconstructed point P, which then forms a four element field at the field window 110 by the eyepiece 108 (FIG. 4). 1) the pixel rendered on the microdisplay 102, 2) the corresponding open shutter sets, 3) the projection of the pixel on the CDP 109, and 4) their ray trajectories on the field window surface 110 for phase 1 are shown in solid lines in FIGS. 5A-5D, respectively. In these figures, the pixel rendering and ray trajectories for the other phases are also shown, but with dashed outlines. The first subscript indicates the phase number (1, 2, 3, 4) of the four phases of the display cycle, and the second subscript indicates the field number. For example, in phase 1 of the display cycle, shutter set S1 is turned on and P 1,1 , P 1,2 , P 1,3 , P 1,4 A set of EIs 101 including four intermediate images P' are displayed on the CDP 109 (FIG. 5A). 1,1 , P' 1,2 , P' 1,3 , P' 1,4 is formed, and the ray is divided into four element fields, V 1,1 , V 1,2 , V 1,3 , V 1,45D. Figures 7A-7D show the state of 1) the pixels on the microdisplay 102 surface, 2) the open shutter set, 3) the image on the CDP 109, and 4) the ray trajectory on the field of view window 110 in phase 2. In each phase, only the elemental fields corresponding to the open shutter set are rendered, shown as the white field of view (V1 and V2) in the field of view window 110 in Figures 4 and 6. By combining the shutter array 106 with different sets of elemental images 101 in a time-division multiplexing manner, all the elemental fields can be received collectively by the human eye during a full display cycle. The ratio of the shutter size to the pitch of the microlenses 105 depends on the number of phases in a display cycle. In the case shown in Figures 4 and 6, where the full display cycle may include four phases, the size of the shutter opening is equal to half the microlenses 105 pitch. It is clear that other ratios of the shutter size to the MLA 104 lens pitch can be chosen to achieve different field distributions according to the present invention.

[0021] It is worth noting that the scheme described in Figures 4-7D can also be easily adapted to M-phase time division multiplexing, where M is greater than 1. It is further worth noting that the aperture elements of the shutter array 106 may comprise multiple pixelated elements. For example, a spatial light modulator consisting of small pixels may be employed as the programmable shutter array 106, and thus each of the aperture elements may be composed of multiple pixelated elements. Such pixelated aperture elements allow the sub-apertures to have a large area and overlap with each other by grouping different sets of pixelated elements. Such overlapping of sub-apertures can result in overlapping of the time division multiplexed sub-field of view windows 110, resulting in potential improvements in spatial and depth resolution. This effect is further demonstrated below.

[0022] To show how the proposed time-division multiplexed InI-based light field display can enhance the display performance in different configurations according to the present invention, we first derive the main parametric relationships using a schematic layout 800 of a conventional InI-based light field display for a set of elemental images 101 without a shutter array 106, as shown in Figure 8. The MLA 804 has the same focal length f MLA The gap between the microdisplay 802 and the MLA 804 is denoted as g, and the distance from the MLA 804 to the CDP 809 is denoted as l. CDP The distance between the CDP 809 of the reconstructed scene and the eyepiece 808 is z0, and the field of view window 810 is at a distance Z XP The gap g is spaced apart by the focal length f MLA If the gap g is equal to or smaller than the focal length f, then a virtual CDP 809 is formed to the left of the microdisplay 802 and the light beams exiting the lenslet appear to diverge. MLA In the larger case, an actual CDP 809 is formed to the right of the microdisplay 802 and the light beams emerging from the lenslet appear to converge towards the CDP 809 as shown in FIG.

[0023] Considering that the CDP 809 is an optically conjugate image of the microdisplay 802 via the MLA 804, its position is given by:

[0024]

number

[0025] where m MLA is the lateral magnification of the MLA804, and m MLA is given by the following formula:

[0026]

number

[0027] As shown in FIG. 8, the ray trajectories of the element fields on the field window 810 and the distribution of all the element fields 801 on the field window 810 depend on the shape and arrangement of the MLA 804. The pitch of the MLA 804 is denoted as pMLA, and the diameter of the microlens is denoted as dMLA. MLA The locus size d of the element field of view on the surface of the field of view window 810 can be expressed by the following equation.

[0028]

number

[0029] where f eypiece is the focal length of the eyepiece.

[0030] The lateral displacement between two adjacent element fields, i.e., the pitch s of the element field distribution on the field window 810, can be expressed as:

[0031]

number

[0032] The trajectory curvature factor α of an element field is defined as the ratio of the ray trajectory size d of an element field to the pitch s between the trajectories of two adjacent fields.

[0033]

number

[0034] 8, the fill factor α is typically in the range of 0 to 8, as it is limited by the physical placement of the MLA 804 and the numerical aperture (NA) of the ray bundle from the microdisplay 802. Note that a low fill factor can result in significant diffraction effects and field discontinuity artifacts.

[0035] The field of view sampling characteristic of an InI-based light field display is the field of view density α, which is defined as the number of fields of view per unit area. view This can be determined by calculating the inverse of the area defined by the pitch of the element fields on the field window. For simplicity, it is assumed that the element fields are evenly distributed in a rectangular array on the field window 110, and the ray path for each element field is also a perfect square, as shown in Figures 5A-5D. If a circular aperture lenslet array or other circular aperture array is utilized, the ray path may be circular, as shown in Figure 8. In this case, the field density can be expressed by the following formula:

[0036]

number

[0037] The sum of the trajectory dimensions of all the elemental fields of view that render a reconstructed 3D scene is the viewing window 110 or eye box (D) of a display through which the light field of the 3D reconstructed scene can be observed. eyebox The total size of the eyebox in the horizontal or vertical direction can be approximated by integrating the ray bundles from different EIs 101 corresponding to the reconstructed points on the CDP 109, and is estimated as follows:

[0038]

number

[0039] where N is the number of horizontal or vertical views used to reconstruct the 3D origin, and m is the lateral magnification of the MLA 104 on the CDP 109. MLA is equal to.

[0040] Without considering image degradation due to diffraction, the angular resolution of the reconstructed 3D scene observed through the field of view window 110 can be expressed as:

[0041]

number

[0042] where p is the pixel size on the microdisplay 102. To achieve a display that provides high spatial resolution, a small value of β is desired.

[0043] To realize a light field display based on InI that can render 3D scenes with large depth of field, has high vertical depth resolution, low image artifacts, and provides accurate focusing cues to mitigate the well-known vergence-accommodation conflict (VAC) problem, we propose a high field of view density α view The analytical relationship between the line of sight density and these display performance metrics has been thoroughly studied by Huang and Hua. As shown in equation (6), the line of sight density α view is inversely proportional to the square of the pitch s of the element line of sight distribution. On the other hand, as suggested by formula (4), the pitch s of the ray trajectories between two adjacent element fields of view is inversely proportional to the optical magnification m MLA is inversely proportional to the pitch p MLA Therefore, it is desirable to achieve high field density by reducing the optical magnification of the lenslet. However, as suggested by equation (7), the eyebox size D eyebox is directly proportional to the ray path pitch of the element fields on the field window 101, which implies that a small eyebox is obtained if a low optical magnification of the lenslet is chosen. Furthermore, as implied by equation (8), a large magnification of the MLA 104 leads to a large value of angular resolution per pixel, which results in poor spatial resolution for the display and reduced image quality.

[0044] Similar parametric relationships are derived for the time-multiplexed system and method for rendering a 3D light field proposed according to the present invention, as shown in Fig. 3A. However, unlike the conventional InI-based display method shown in Fig. 8, the elemental views for rendering a 3D scene are not rendered simultaneously, but in a time-multiplexed manner. As shown in Fig. 3A, to reconstruct a 3D image point P through 4x4 different elemental views, these 16 elemental views are divided into four sets of elemental images 101, each set of elemental images 101 consisting of four elemental views. The four sets of elemental images 101 are rendered in a four-phase time-multiplexed manner, as illustrated by Figs. 4-7D.

[0045] In a generalized configuration, different sets of elemental fields for different phases are combined together to render all the elemental fields of a time-division multiplexed light field. As shown in Figures 3A-7D, in a given phase, the pitch between adjacent switched-on shutter elements (e.g., S1 in Figure 4) is p SA This determines the pitch of the elemental fields rendered with the same phase. However, the effective pitch of the elemental fields combined by time division multiplexing is much smaller. This is due to the number of phases, M, that can be combined between adjacent fields rendered with one phase each in the horizontal and vertical directions. H and M. V The total phase required, M, depends on M=M H *M V Without loss of generality, we assume that the same number of views are combined horizontally and vertically (i.e., M H =M V In an M-phase time division multiplexed system, the effective pitch of adjacent interdigitated subapertures on a shutter can be expressed as:

[0046]

number

[0047] As shown in FIG. 3A, in order to ensure an even distribution between the combined element fields, the lateral displacement Δd between the shutter elements switched on in the two subsequent phases is SA-H and Δd SA-V must be carefully selected in the horizontal and vertical directions, respectively. Without loss of generality, it is best to use the same lateral displacement (i.e., Δd SA-H = Δd SA-V = Δd SA ) between adjacent shutter elements switched on in different phases (e.g. S1 and S2 in FIG. 4). SA should satisfy the following relationship:

[0048]

number

[0049] In this case, as shown in FIG. 3C, the light ray trajectory size d TM is the aperture size d of the lenslet MLA Instead of the aperture size d of the shutter element, SA Similarly, the ray trajectory pitch sTM between two adjacent element fields combined on the field window 110 depends on the lenslet pitch p MLA Instead of the effective pitch of the shutter opening, pSA.eff The size of the ray trajectory on the field of view window 110, d TM and the lateral displacement Δd between two adjacent element fields corresponding to the same microlens in a time-division multiplexed InI system. TM can be expressed by the following equation:

[0050]

number

[0051]

number

[0052] Similar to equations (11) and (12), the effective pitch s of the element fields in the field window 110 of a time-division multiplexed InI system TM D depends on the effective pitch of adjacent interdigitated subapertures on the shutter, which can be expressed as:

[0053]

number

[0054] The fill factor of the element field of view in a time division multiplexed system is expressed as:

[0055]

number

[0056] Unlike conventional systems, the fill factor of an element field depends on the fill factor of the shutter array 106, rather than the fill factor of MLA 104. This allows a large range of fill factors of the element field since the subapertures can overlap each other as described above and in subsequent implementations and embodiments. This allows for fill factors greater than 1, overcoming the physical fill factor limitations of MLA 104 and potentially broadening the range of implementations of InI systems.

[0057] In the time-division multiplexed rendering method, each microlens is used to render M sets of elemental fields of M phases. This means that the number of fields used to reconstruct a 3D point is proportional to the lateral magnification m of the MLA104. MLA Therefore, the dimension of the field of view window 110 or eyebox in either the horizontal or vertical direction of a time division multiplexed system is given by:

[0058]

number

[0059] If diffraction effects are not taken into account, the angular resolution of the reconstructed scene observed through the field of view window 110 in a time-division multiplexed system can be expressed by the same equation (8).

[0060] Based on the parametric relationships characterized by equations (1)-(15), there are several different ways of implementing the embodiments of the time division multiplexing scheme proposed in accordance with the present invention, which can achieve different aspects of the overall display quality improvement according to the requirements of different applications.

[0061] Example 1: Visibility Density Priority Method Based on the parametric relationships described above, one possible implementation according to the present invention of a time-division multiplexed light field system is to increase the number of elemental fields and field density while maintaining a given eyebox size and using lenslets with the same pitch and the same optical magnification. A schematic layout of a four-phase time-division multiplexed system 100 for increasing field density is shown in FIG. 3A. As shown in equation (9), the total number of fields that can be rendered by the time-division multiplexing scheme is proportional to the effective pitch p of the shutter array 106. SA.eff Pitch of MLA104 against MLA It depends on the ratio of M to M. For a given MLA 104, choosing a smaller effective shutter pitch size provides more element fields 101, but requires a higher refresh rate for the shutter array 106 and the microdisplay to render the M sets of element fields 101 fast enough so that the eye can view them in a time-multiplexed manner without suffering from flicker. Considering the refresh rate limitations of modern display technologies, a ratio of 2 is recommended for a four-stage rendering process, as shown in FIG. 3A.

[0062] In the field density enhancement scheme, the optical specifications of the MLA 104, eyepiece 108, and their relative spacing are selected to be the same as those of the conventional non-multiplexed scheme, as shown in Figure 8. As a result, the total eyebox size given by equation (15) and the spatial resolution of the M-phase multiplexed system given by equation (8) are the same as those parameters of the non-multiplexed system, except that the total number of fields of the multiplexed system is M times that of the non-multiplexed system. The ray trajectory pitch of adjacent element fields 101 given by equation (13) is much smaller for the time-division multiplexed system than for the non-multiplexed system, and the corresponding field density is much higher.

[0063] For comparison, Table 1 shows the optical specifications of the four-phase time division multiplexing scheme shown in FIG. 3A, and the same optical specifications apply to the non-time division multiplexing scheme shown in FIG. 8, which does not use the shutter array 106. In this design example, the pixel pitch p of the microdisplay is 8 um. The focal length of all the microlenses of the MLA 104 is 3 mm. The microlens diameter d of the MLA 104 is 1. MLA and the small lens pitch are both 1 mm. The focal length of the eyepiece 108 is 18 mm, and the distance between the CDP 109 and the eyepiece 108 is 18 mm. The field of view window 110 is located 24 mm behind the eyepiece 108. The pitch p SA is 1 mm, the aperture size d of the shutter array 106 SA is 0.5 mm in the case of the time division multiplexing method.

[0064] [Table 1]

[0065] Table 2 shows a comparison of viewing parameters between the viewing density-priority time division multiplexed InI system 100 according to the present invention of Fig. 3A and the conventional InI display system 800 of Fig. 8. It can be seen that the time division multiplexed InI system 100 according to the present invention has the same eyebox size and angular resolution as the conventional non-time division multiplexed system, but the viewing number and the viewing density are four times higher in the time division multiplexed system according to the present invention.

[0066] [Table 2]

[0067] Example 2: Field of view with fill factor method The fill factor of elemental fields plays a very important role in both the spatial resolution and visual appearance of a light field display. In conventional InI-based displays, the fill factor of the elemental fields is limited by the physical constraints of the microlens arrangement, as suggested by equation (5), and is typically between 0 and 1. Based on the parametric relationship described above, another alternative configuration of a time-multiplexed light field system is to adapt the multiplexing specifications to increase the fill factor of the elemental fields while maintaining a given set of field parameters, such as the total number of fields or eyebox size and spatial resolution. As suggested by equation (14), the field fill factor in a time-multiplexed system scales with the sub-aperture size d SA and the effective sub-opening pitch pSA.eff As shown by Figures 9A-9B, the shutter array 106 of Figure 3A can be made of small programmable pixelated elements such as a liquid crystal display array or a digital mirror device array. In such pixelated devices, the on or off state of each pixel can be independently addressed. In such cases, the sub-aperture size d SA is the effective sub-aperture pitch p SA.effSince the fill factor of a time-multiplexed system can be greater than 1, allowing for ray trajectory overlap of adjacent element fields of view, thereby minimizing image artifacts due to field of view discontinuities. The overlap of the time-multiplexed sub-field of view windows can also provide potential improvements in spatial and depth resolution. The lateral displacement Δd of the second sub-aperture set from the first aperture set SA is the size of the sub-opening d SA If the ray trajectories of the corresponding element fields rendered by the two time multiplexed subaperture sets overlap on the field window and have a fill factor greater than 1. The lateral displacement Δd of the second subaperture set from the first aperture set is less than SA is the size of the sub-opening d SA , the ray traces of the corresponding element fields do not overlap on the field window and the fill factor is less than one.

[0068] 10A-10B show an exemplary schematic layout of a four-phase time division multiplexing system 1000 for enhancing the viewing fill factor of a light field display according to the present invention. FIG. 10A shows the ray paths of a first phase and the corresponding arrangement of a first active sub-aperture set S1, and FIG. 10B shows the ray paths of a second phase and the corresponding arrangement of a second active sub-aperture set S2. Here, a programmable shutter array 1006 made up of addressable sub-pixels is utilized to control the size of the opened sub-apertures d SA and effective pitch p SA.eff As shown in Figures 10A-10B, a larger subaperture size can be obtained by grouping more controllable pixels together so that the ray bundle size of each element field is larger.

[0069] As an example, for a four-phase time division multiplexing specification to increase the field fill factor and therefore the field density, the same optical specifications are used as those shown in Table 1 for the non-time division multiplexing scheme, except that a transmissive LC based pixel array 1006 is used as the shutter array 106. The LC pixel array 1006 has a pixel pitch of 125um and has a suitable pixel resolution so that its overall dimensions match the size of the MLA 1004. There are a total of 8x8 pixelated elements under each 1mm aperture of the microlens. Figures 9A-9B show the design parameters of the first and second set of sub-apertures, S1 and S2, respectively. The size d of each sub-aperture of the shutter is 125um. SA is set to 6 pixels, and the lateral shift Δd between the adjacent sub-openings corresponding to the same microlens is SA1 is set to 2 pixels. Overall, the first set of sub-openings S1 overlaps the second set of sub-openings by 4 pixels. In this case, the effective shutter pitch p SA.eff is 4 pixels. The ray paths of the element fields rendered by the first set of subapertures and the second set of subapertures overlap by 33%, resulting in an effective field fill factor of 1.5. Table 3 shows the field parameters and spatial resolution for this time division multiplexing scheme.

[0070] [Table 3]

[0071] Example 3: Spatial resolution priority method Based on the parametric relationships discussed above, a further exemplary alternative configuration of the time-division multiplexed light field system according to the present invention is to adapt the multiplexing scheme to increase the spatial resolution of the reconstructed 3D scene while maintaining a given set of field parameters such as the total number of fields and field density. As equation (8) suggests, the angular resolution per pixel of the reconstructed 3D scene scales with the optical magnification m of the MLA. MLA The optical magnification m of the MLA is directly proportional toMLA The lower the value of the angular resolution per pixel, the smaller the value of the angular resolution per pixel, resulting in a high spatial resolution and good image quality for the display. Equation (8) further defines that the angular resolution is a function of the pixel pitch p of the microdisplay, the distance z0 between the CDP and the eyepiece, and the focal length f of the eyepiece. eyepiece , and the distance z between the eyepiece and the field window surface. XP Thus, for a given set of field parameter specifications, such as total field number and eyebox, the optical specifications of a time-multiplexed system can be optimized differently from a non-multiplexed system, thereby improving spatial resolution yield while producing the same field parameters.

[0072] Figure 11 shows a schematic layout of a four-phase time division multiplexing system 1100 for enhancing the spatial resolution of a light field display while maintaining a total 2x2 field of view of the reconstructed 3D scene. For comparison, Figures 9A-9B show a schematic layout of a conventional InI-based display scheme without multiplexing, which results in the same 2x2 field of view. To achieve a total 2x2 field of view in both systems, the optical magnification m of the MLA in the four-phase multiplexing scheme is MLA is chosen to be half the optical magnification of a conventional non-multiplexed system. As suggested by equation (2), control of the optical magnification of the MLA 104 can be achieved by adjusting the gap between the microdisplay and the MLA 104, adjusting the focal length of the MLA 104, or adjusting both. For ease of comparison, in the example shown in FIG. 11, the optical specifications of the MLA 104, such as focal length, pitch, and microlens diameter, are the same as those of the MLA 104 used in a non-multiplexed system, while adjusting the gap between the microdisplay and the MLA 104. However, the time-division multiplexing scheme of FIG. 11 provides twice the spatial resolution compared to the non-multiplexing scheme of FIGS. 9A-9B. The magnitude of the improvement in resolution brought about by the M-phase time-division multiplexing scheme is determined by the effective pitch p of the shutter array. SA.eff Pitch of MLA104 against MLAThe higher the number of phases, the higher the refresh rate of the shutter array and the microdisplay must be to render the M sets of elemental fields of view fast enough to be viewed in a time-multiplexed manner without eye flicker. Given the refresh rate limitations of modern display technologies, a ratio of 2 is recommended for a 4x resolution improvement.

[0073] As described above, equation (8) is expressed by taking into account the microdisplay pixel pitch p of the angular resolution, the optical magnification of the MLA 104, the distance z between the CDP 109 and the eyepiece 108, the focal length of the eyepiece 108, f eyepiece , and the distance zXP between the eyepiece 108 and the field of view window surface 110. Thus, in a resolution-first approach, the optical specifications of the MLA 104, eyepiece 108, and their relative spacing must be jointly optimized to obtain a balance between the spatial resolution given by equation (8) and the field of view parameters given by equations (9) through (15). Thus, possible embodiments of the resolution-first approach should not be limited to the example shown in FIG. 11. Under the assumption that the pixel pitch of the microdisplay 102 is the same, both systems shown in FIGS. 9A, 9B, and 11 can each render a total of 2x2 fields of view.

[0074] [Table 4]

[0075] For comparison, two sets of optical specifications for the four-phase time division multiplexing method are shown in Table 4. The first set (Example 1) corresponds to the example shown in FIG. 11, employs a microdisplay 102, and adopts the same optical specifications as those shown in Table 1 for the non-time division multiplexing method, except that the object:image relationship is different. The gap g between the microdisplay 102 and the MLA 104 is changed from 4.5 mm to 6 mm, and the distance between the MLA 104 and the CDP 109 is changed from 9 mm to 6 mm. The second example (Example 2) in Table 4 corresponds to the system specifications to obtain the same spatial resolution and number of views as the first example, but the view density and eyebox size of the non-multiplexing method in FIGS. 9A-9B are the same. In the second design example, the pixel pitch p of the microdisplay 102 is 8 um, the same as the first example. All the microlenses 105 of the MLA 104 have the same focal length of 2.57 mm. The microlens diameter d of the MLA 104 is 1.0 mm. MLA and the small lens pitch are both 1 mm. The focal length of the eyepiece 108 is 24 mm, and the distance between the CDP 109 and the eyepiece 108 is 24 mm. The field window is located 24 mm behind the eyepiece 108. The pitch p SA is 1 mm, and the aperture size d of the shutter array 106 is SA is 0.5 mm. Table 5 shows a comparison of the field of view parameters and spatial resolution between the two different time division multiplexed configurations. Comparative data for the non-multiplexed system is in Table 5.

[0076] [Table 5]

[0077] Example 4: Time division multiplexing scheme with reduced viewing crosstalk In all the exemplary embodiments shown above, an aperture array 1207 consisting of an array of ray-limiting optical apertures with the same pitch as MLA 1204 can be inserted between the microdisplay 1202 and MLA 1204 to reduce the crosstalk problem. As shown in FIG. 12, the aperture corresponding to each microlens 1205 allows only the desired rays to propagate and reach the eyepiece, but blocks the undesired rays from the adjacent element image 1201 from reaching the corresponding microlens 1205. For example, the opaque area (shown as a solid black line) on the aperture array 1207 between EI1 and EI2 blocks the dashed rays emitted from the element image EI1 from reaching the microlens ML2 adjacent to the microlens ML1. These blocked rays are the main cause of crosstalk and ghost images commonly observed in InI display systems. The distance from the microdisplay 1202 to the aperture array 1207 is denoted as a, and the diameter of the aperture opening is d. A In order to reduce the crosstalk, these parameters must satisfy the following constraints:

[0078]

number

[0079]

number

[0080] Here, p EI is the size of the element image, and p MLA is the pitch of the MLA. The aperture array 1207 is a printed fixed aperture array 1207 or a spatial light modulator.

[0081] Example 5: Time division multiplexing with controllable directional light sources The time-multiplexing scheme of the InI-based light field display 100 shown in FIG. 3A utilizes a switchable shutter array 106 located adjacent to the MLA 104. By turning on or off different shutter elements under each microlens 105, different ray paths can be rapidly switched between through different parts of the lenslet 1005, allowing ray bundles from different sets of pixels of the elemental image 101 to be rendered on the microdisplay 102 and imaged in a time-multiplexed manner by the MLA 104. The use of a switchable shutter array 1207 can potentially lead to light loss, since the transmittance or reflectance of the shutter array can be limited (FIG. 12). If a self-emissive microdisplay 102 is utilized, a switchable shutter array 106 is used for the exemplary configuration 100 of FIG. 3A. The microdisplay 102 can be based on a liquid crystal display (LCD) device (reflective or transmissive) that modulates the illumination source, or a non-self-emissive SLM-type display technology such as a digital mirror device (DMD). Both of these non-emissive display technologies require an illumination source, such as a frontlight in the case of a reflective SLM or a backlight in the case of a transmissive SLM. Configurations employing such non-emissive display technologies can still use a switchable shutter array 106 as a mechanism for rapidly switching between different light paths. An alternative configuration in accordance with the invention is to create a microdisplay that can selectively output or "emit" light toward different portions of the microlens aperture of the MLA 104 in a time division multiplexed manner, hereafter referred to as a "directional microdisplay." Such a directional microdisplay in accordance with the invention can cooperate with the MLA 104 to rapidly switch between different light paths through the MLA 104 to achieve the same optical functionality as the use of a switchable aperture array 106.

[0082] For example, Figures 13A-13B show an exemplary optical layout of a four-phase time division multiplexed InI-based light field system 1300 according to the present invention, utilizing a directional microdisplay 1302 that generates directional illumination as described above to achieve functionality similar to that of the time division multiplexing illustrated in Figures 3A-3C. The directional microdisplay according to the present invention can include a switchable light source array 1310 (e.g., an LED array), a barrier array 1312, an MLA 1304, and a spatial light modulator (SLM 1314), Figures 13A-13B. It is worth noting that the MLA 1304 in Figures 13A-13B is labeled "MLA2" to distinguish it from the main MOA 1304 required for integral imaging purposes. Furthermore, the schematic layout of the microdisplay 1302 assumes the use of a transmissive SLM 1314 that requires a backlight source. The layout can be easily modified for a reflective SLM 1314 that requires a frontlight source. The light source arrays 1310 each have the same dimensions d cell 13, the light source cells 1316 may include a plurality of light source cells 1316 having a pitch p cell is the pitch p of the element images EIEach cell 1316 can be considered as an elementary light source cell 1316 and provides the necessary backlighting for the corresponding elementary image displayed on a portion of the SLM 1314. Each cell 1316 can include an array 1310 of light source cells 1316, shown as rectangles on the light source array 1310 in Figs. 13A-13B. These light source cells 1316 are individually controllable, for example individual light emitting diodes (LEDs). Each light source cell 1316 can be switched on or off independently of other units of the same cell 1316. The barrier array 1312 attached to each light source cell 1316 is provided to prevent the crosstalk between adjacent cells 1316 and to provide a mechanical mount for the microlens array 1313. Each microlenslet of the microlens array 1313 can modulate light from a light source to generate directional backlighting for its corresponding elementary image rendered on the SLM 1314. The SLM 1314 modulates the light from the light source cells 1316 to render the elemental image for integral imaging. As shown in Figs. 13A-13B, by switching the light source cells 1316 on and off within the light source cells 1316, the function of selecting the light ray path through the main MLA 1304 can be generated, similar to the shutter array of Fig. 3A. For example, in Figs. 13A-13B, to generate a four-phase time division multiplexed directional microdisplay 1302, each light source cell 1316 may include 2x2 light source cells 1316, each corresponding to a phase of the four-phase time division multiplexing cycle and the corresponding EI set to be rendered. Figs. 13A-13B show two different phases of a full display cycle. In each phase, only one set of the light source cells 1316 in the light source array 1310 is turned on, and the light emitted by the corresponding set illuminates the SLM 1314 with a light ray in a desired direction controlled by the optical properties of the microlens array 1313. These desired rays continue to propagate towards selected portions of the microlens aperture of the main MLA 1304 .The on-off state of the light source units can be synchronized with the rendering of the corresponding set of element images that represent the correct perspective view to reconstruct a target 3D scene. For example, in FIG. 13A, the light rays generated by the top unit of the light source (shown as a white rectangle) are modulated by the first set of element images rendered on the SLM 1314. The light rays modulated by the SLM 1314 are propagated towards the bottom of the microlens aperture to generate element view V1. In FIG. 13B, the light rays generated by the bottom unit of the light source are modulated by the second set of element images rendered on the SLM 1314. The light rays modulated by the SLM 1314 are propagated towards the top of the microlens aperture to generate element view V2. In this embodiment, the number of light source cells 1316 in the light source array 1310 determines the number of phases in a display cycle. For example, FIG. 14 shows a four-phase time division multiplexing system. Each of the source cells 1316 generates a sub-aperture that corresponds to each of the microlenses in the imaging MLA 1304 .

[0083] FIG. 14 shows the parameters of the directional microdisplay 1302 together with the main imaging MLA 1304. Let the distance between the light source cells 1316 and the microlens array 1313 be I BL It is desirable for the light source cells 1316 imaged by the microlens array 1313 to be optically conjugate to the plane of the main imaging MLA 1304. Therefore, the distance I BL It is desirable that the following formula be satisfied:

[0084]

number

[0085] where m MLA2 is the lateral magnification of the microlens array 1313, which can be expressed by the following equation:

[0086]

number

[0087] The fill factor of the element field of view projected onto the field of view window plane depends on the physical size of each light source unit and the pitch between adjacent light source cells 1316.

[0088]

number

[0089] To enable a crosstalk free system, the image of the source cell 1316 is preferably no larger than the size of the MLA 1304, given by:

[0090]

number

[0091] To achieve a field fill factor of 1 while keeping the system crosstalk-free, the focal length of the microlens array 1313 should be well selected so that the image of each source cell 1316 is exactly the same as the pitch of the main imaging MLA 1304. Therefore, the focal length of the MLA 1304 and the distance from source array 1310 to microlens array 1313 should be carefully selected according to equations (17)-(20).

[0092] It is further worth noting that each of the light source cells 1316 shown in FIG. 14 can include multiple light emitting elements similar to the pixels of a pixelated shutter array. Each of the light emitting elements can be individually turned on or off. FIGS. 15A-15B show a design of a light source array 1510 having 2x2 light source cells 1316 according to the present invention. Each of the cells 1316 illuminates a corresponding one portion of the SLM 1314 to render an elementary image pattern. The illuminated area on the SLM 1314 corresponds to the 2x2 elementary field of view to reconstruct a 3D light field in the four phases of the time-multiplexed InI according to the present invention. The dimensions of the light source cells 1316 are d cell Each of the light source cells 1316 can include a 2D array of individually controllable light emitting elements (e.g., an 8x8 array of LED elements shown as small squares in Figs. 15A-15B). In this array format, the light emitting units d are each individually controllable by grouping different numbers of the light source elements in each cell 1316 to define an effective light emitting area corresponding to the sub-openings in Figs. 9A-9B. unit In addition, the size of the light-emitting source pitch p can be controlled by adjusting the lateral spacing between adjacent light source cells 1316. cell Finally, the lateral displacement Δd between adjacent light source cells 1316 within a cell 1316 can also be controlled. unit The field fill factor can be controlled by adjusting p. In the example shown in Figures 15A-15B, each light emitting unit 1316 can include an array of 6x6 light emitting elements, and each cell 1316 can include 8x8 elements. Figure 15A shows the arrangement of the light emitting cells to render a first set of elemental images, while the pitch between the light source cells 1316 is p. cellFIG. 15B shows the arrangement of the light-emitting cells 1316 for rendering a second set of element images, where the light-emitting units are shifted by two elements from the first corresponding light-emitting units for rendering the first set of element images. The overlap of the illumination areas between different phases can be easily seen. This overlap allows the system to have a field fill factor of 1.5. The exemplary configuration of FIGS. 15A-15B can achieve similar effects to those shown in, for example, FIGS. 9A-9B.

[0093] Experimental Prototype Based on the circuit diagram in FIG. 3A, a proof-of-concept prototype of a time-multiplexed InI-based light field system 1600 was implemented as shown in FIG. 16. The microdisplay 1602 used in the prototype was a 0.7-inch organic light-emitting display (OLED) from Sony with a color pixel pitch of 8 μm and a pixel resolution of 1920x1080 (ECX355B). The MLA 1604 used was an MLA630 from Fresnel Technologies (https:www.fresneltech.com). The focal length was 3.3 mm and the lens pitch was 1 mm. The shutter array 1606 used a transmissive LCD from JHDLCM Electronics Company (JHD12864). The number of pixels was 128x64 and the pixel pitch was 0.5 mm. With such a large pixel size, one pixel on the LCD corresponds to one shutter, so the system was not affected by diffraction of the pixel structure. A full image cycle includes four phases. The lateral magnification of the MLA 1604 was set to 2, rendering a total of 4x4 elemental views. The eyepiece 1608 was a commercially available eyepiece 1608 with a focal length of 27 mm. Overall, the prototype system 1600 was designed to achieve a field window size of approximately 6 mmx6 mm, with a total of 4x4 fields of view rendered by the system for each point in the reconstructed 3D scene. The field density was approximately 0.44 mm. -2, which corresponds to an element field pitch of 1.5 mm in the field window. When combined with an eyepiece 1608 with a focal length of 27 mm, the angular resolution per display pixel was approximately 2.04 arcmin in visual space.

[0094] These and other advantages of the present invention will be apparent to those skilled in the art from the foregoing specification. It will therefore be appreciated by those skilled in the art that changes or modifications may be made to the above-described embodiments without departing from the broad inventive concept of the present invention. It is therefore to be understood that the present invention is not limited to the particular embodiments described herein, but is intended to cover all changes and modifications that are within the scope and spirit of the present invention as set forth in the appended claims.

[0095] References The following references are incorporated herein by reference in their entireties: 1. H. Hua, “Enabling focus cues in head-mounted displays,” Proceeding of the IEEE 105(5), 805-824(2017). 2. GE Favalora, “Volumetric 3D displays and application infrastructure,” Computer, 38(8), 37-44 (2005). 3. H. Yu, K, Lee, J. Park, and Y. Park, “Ultrahigh-definition dynamic 3D holographic display by active control of volume speckle fields,” Nature Photonics 11(3), 186 (2017) 4.G.Li,D.Lee,Y.Jeong,J.Cho,and B.Lee,「Holographic display for see-through augmented reality using mirror-lens holographic optic element」Opt.Letters 41(11),2486-2489(2016) 5,S.Liu and H.Hua,「A systematic method for designing depth-fused multi-focal plane three-dimensional displays,」Opt.Express 18(11),11562-11573(2010) 6.J.Rolland, M.Kruger,and A.Goon,「Multifocal Planes head-mounted displays,」Opt.39(19),3209-3215(2000). 7.S.B.Kim and J.H.Park,「Optical see-through Maxwellian near-eye display with an enlarged eyebox,」Opt.Letters 43(4),767-770(2018) 8.H.Hua and B.Javidi,「A 3D integral imaging optical see-through head-mouted display,」Opt.Express,22(11),13484-13491(2014). 9.G.Wetzstein,D.Lanman,M.Hirsch and R.Raskar,「Tensor displays:Tensor displays:Compressive light field synthesis using multilayar with directional backlights,」Proc.acm siggraph,(2012). 10.D.lanman and D.luebke,「Near-eye light field displays,」ACM Trans.Graph.32(6),1-10(2013). 11.H.Huang and H.Hua,「High-performance integral-imaging-based light field augmented display using freeform optics,」Opt.Express26(13),17578-17590(2018). 12.B.Liu,X.Sang,X.Yu,X.Gao,L.Liu,C.Gao,P.Wang,Y.Le,and J.Du,「Time-multipexed field display with 120-degree wide viewing angle」,Opt.Express27(24),pp.35728-35739(2019). 13.H.Huang and H.Hua,「Generalized methods and strategies for modeling and optimizing the optics 3D head-mouted light field displays,」Opt.Express27(18),25154-25171(2019). 14.H.Huang and H.Hua,「Systematic characterization and optimization of 3D light displays,」Opt.Express 25(16),18508-18525(2017). 15.J.H.Park,S.W.Min,S.Jung,and B.Lee.「Analysis of viewing parameters for two display methods based on integral photography.」Applied Optics 40,no.29 5217-5232(2001). 16.X.Wang,Y.Qin,H.Hua,Y.H.Lee,and T.Wu.「Digital swithcable multi-focal lens freeform optics.」Opt.Express 16.26(8):11007-17(2018). 17.X.Wang,and H.Hua.「Digitally Swithchable Mirolens Array for Integral Imaging.」SID Symposium Digest of Technical Papers.Vol.51.No.(2020) 18.M.Xu and H.Hua,「Finite depth and varifocal head-mouted display based on geometrical lightguide,」Opt.Express 28(8),12121-12137(2020).

Claims

1. 1. A time-multiplexed integral image (InI) light field display comprising: a microdisplay including a plurality of pixels configured to render a set of elemental images, each elemental image providing a different perspective view of a 3D scene; a microlens array disposed in optical communication with the microdisplay at a selected distance therefrom for receiving light from the elemental images of the microdisplay, the microlens array having a central depth plane associated therewith that is optically conjugate with the microdisplay across the microlens array, the microlens array configured to receive bundles of light rays from the elemental images to create an integrated image at a corresponding reconstruction point relative to the central depth plane to reconstruct a light field of the 3D scene; a switchable array disposed in optical communication with the microlens array and configured to receive light transmitted by the microlens array and transmit the received light to the light field of the 3D scene; A time-multiplexed integral image (InI) light field display having

2. A time-division multiplexed integral image (InI) light field display as claimed in the preceding claim, wherein the switchable array is configured to selectively direct light from selected ones of the component images to the central depth plane.

3. 13. A time division multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, wherein the microdisplay is configured to synchronize the rendering of the elemental images on the microdisplay with the switching of the switchable array, thereby operating the microdisplay and the switchable array in a synchronized time division multiplexed manner.

4. 10. A time-multiplexed integral image (InI) light field display according to any one of the preceding claims, wherein the microlens array comprises an array of microlenses having the same focal length.

5. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, wherein the switchable array is arranged at a position between the microlenses and the central depth plane.

6. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, wherein the switchable array is arranged at a position between the microlenses and the microdisplay.

7. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, wherein the switchable array comprises switchable elements which can be turned on to pass light rays from the microlens array or turned off to block the passage of light rays.

8. 10. A time-multiplexed integral image (InI) light field display according to any one of the preceding claims, wherein the programmable switchable array comprises a shutter array.

9. 10. A time-multiplexed integral image (InI) light field display according to any one of the preceding claims, wherein the programmable switchable array comprises a switchable light source array.

10. 10. A time-multiplexed integral image (InI) light field display according to any one of the preceding claims, wherein the microdisplays are self-emissive.

11. 10. A time-multiplexed integral image (InI) light field display according to any one of the preceding claims, wherein the microdisplays are transmissive.

12. 10. A time-multiplexed integral image (InI) light field display according to any one of the preceding claims, wherein the microdisplay comprises a spatial light modulator.

13. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, wherein the micro-display comprises one or more of a liquid crystal display and a digital mirror device.

14. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, comprising an eyepiece positioned at a distance z0 from the central depth plane to receive light from the light field of the 3D scene.

15. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, wherein an aperture size of each switchable element of the switchable array is smaller than an aperture size of each lenslet of the microlens array, each lenslet covering multiple elements of the switchable array.

16. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, wherein the switchable array comprises a plurality of pixelated elements of a size smaller than the aperture size of each switchable element.

17. 13. A time-multiplexed integral image (InI) light field display as claimed in any one of the preceding claims, comprising a barrier array arranged between the microdisplay and a microlens array and capable of optical communication.

18. A time-division multiplexed integral image (InI) light field display as described in any one of claims 1 to 16, which has an aperture array capable of optical communication arranged between the microdisplay and a microlens array.

19. 20. The time-multiplexed integral image (InI) light field display of claim 18, wherein a distance from the microdisplay to the aperture array is denoted as a and a diameter of an aperture of the aperture array is denoted as dA; [0023] A time-division multiplexed integral image (InI) light field display, where pEI is the dimension of the element image, g is the distance from the microdisplay to the microlens array, and pMLA is the pitch of the MLA.

Images